AI Data Center Interconnects: The Nervous System of the AI Boom

01 One GPU Is Never Enough

The public image of the AI boom is a chip: a slab of silicon bristling with stacked memory. But a single accelerator, no matter how fast, cannot train a frontier model. Modern language models hold hundreds of billions of parameters, and the activation values and optimizer states that surround them are larger still. Training is therefore sliced across thousands of accelerators, each owning a shard of the work, and every optimization step ends with those shards comparing notes.

That comparison is the catch. Each step of distributed training triggers collective operations such as all-reduce, in which every accelerator exchanges gradient data with every other. The volume is enormous and the deadline is immediate: the next step cannot begin until the exchange completes. A GPU can multiply matrices far faster than a typical network can move the results, so when the fabric is slow, the most expensive hardware in the building sits idle waiting for bytes.

This is why interconnect, not raw compute, is increasingly treated as the binding constraint on scale. Cluster architects now design the network first and the accelerator count second, and the industry has converged on a two-tier answer: a very fast fabric inside the rack, and a cheaper, broader fabric between racks.

02 NVLink and NVSwitch: The Fabric Inside the Rack

NVIDIA's answer to the intra-rack problem is NVLink, a proprietary high-bandwidth interconnect that first shipped on the P100 accelerator in 2016 with 160 GB/s of bidirectional bandwidth per GPU. Each generation has multiplied that figure. According to NVIDIA's published specifications, V100 reached 300 GB/s, A100 reached 600 GB/s, and H100 reached 900 GB/s. The current NVLink 5 generation on Blackwell parts lists 1.8 TB/s per GPU, roughly eleven times the original figure.

Raw link bandwidth is only half the story. NVSwitch chips sit between the GPUs in a rack and provide a switched all-to-all fabric, so any GPU can talk to any other at full NVLink speed rather than sharing a bus. In NVIDIA's GB200 NVL72 rack design, 72 Blackwell GPUs and their companion CPUs are wired into a single NVLink domain that presents itself to software as one very large accelerator. The point of the design is not just speed but symmetry: no GPU is a network hop further away than any other.

The result is a tier of the cluster where communication is nearly free, which is what makes aggressive model parallelism practical. Tensor parallelism, which splits individual matrix multiplications across GPUs, is only viable when the GPUs involved can exchange intermediate results within a single layer's computation, at speeds close to memory access.

03 Optical Interconnects and the Limits of Copper

Inside the rack, short copper traces can still carry tens of gigabits per second per lane. Beyond roughly a meter or two, electrical signaling struggles: attenuation rises with both distance and data rate, and every extra lane needs retimers that burn power. The standard fix is to convert to light. Pluggable optical transceivers, mounted on the switch faceplate, translate between the electrical and optical domains and carry 400- and 800-gigabit streams across the data center over fiber.

Pluggables work, but they impose a tax. The signal still has to travel electrically from the switch silicon to the faceplate, through packaging and a crowded board, and the module itself dissipates watts per port. The industry's next step is co-packaged optics, which moves the optical engine into or immediately beside the switch package, shrinking the electrical path to millimeters. Broadcom has demonstrated co-packaged switch silicon, and NVIDIA has announced Quantum-X and Spectrum-X photonics networking products aimed at AI clusters, with the pitch framed in joules per bit moved.

The roadmap matters because link rates keep climbing. As 800-gigabit ports become the default for AI back-end networks, 1.6-terabit optics are next in line, and every doubling makes the electrical-reach problem harder, pushing the industry toward optics earlier in the path.

04 Ethernet, InfiniBand, and the Ultra Ethernet Response

Between racks, the historical default for high-performance computing has been InfiniBand, a specialized interconnect with remote direct memory access and low, deterministic latency. NVIDIA acquired InfiniBand's leading vendor, Mellanox, in 2020, and the technology powers the back-end networks of many large AI clusters through the Quantum switch line.

InfiniBand's position is now contested by Ethernet, the default fabric of the broader data center. Lossless behavior for AI traffic can be approximated on Ethernet with RDMA over Converged Ethernet and careful congestion tuning. The more ambitious project is the Ultra Ethernet Consortium, founded in 2023 under the Linux Foundation with members including AMD, Arm, AWS, Microsoft, Meta, and Oracle, which published its 1.0 specification in 2025. It aims to define an Ethernet-based transport purpose-built for AI and HPC traffic, with explicit support for the collective communication patterns that dominate training.

The competition is healthy for buyers. A single vendor owning both the accelerator and the fabric simplifies procurement but concentrates pricing power, and the open-networking camp argues that Ethernet's manufacturing scale can eventually deliver AI-class performance at commodity economics.

05 The Economics of the Fabric

Interconnect is not a rounding error in cluster budgets. In a large AI training cluster, networking — switches, optics, copper, fiber, and the cabling labor to wire tens of thousands of ports — plausibly runs to a fifth of the capital cost, and the share grows as accelerators get faster and demand fatter pipes to keep up. The exact split varies by design, and N43 treats the figures in the chart below as illustrative rather than measured, but the direction is clear: as compute per dollar improves, the fabric becomes a larger fraction of what remains.

Power is the other ledger line. Every switch, retimer, and optical module draws electricity that produces no floating-point operations, and optics are a meaningful consumer in the back-end network. This is why co-packaged optics is marketed primarily on energy: shortening the electrical reach and integrating the laser are framed as joules-per-bit savings at cluster scale, where a watt saved in the network is a watt available to sell as compute.

06 What Saturation Looks Like

The evidence for the interconnect bottleneck shows up in step time. Large training runs report that a substantial fraction of wall-clock time goes to communication — collective calls in libraries such as NCCL, all-reduce bursts at the end of each step, and tensor-parallel exchanges that recur inside every transformer layer. Operators work hard to overlap communication with computation, prefetching what can be prefetched and hiding transfers behind the next batch of math, but overlap has limits: when the fabric saturates, the overlap breaks and step time inflates.

Saturation has visible symptoms. Collectives that should take microseconds start taking milliseconds; a single slow link or congested path creates stragglers that hold back every other accelerator; oversubscribed tiers in the network topology throttle exactly the traffic pattern, all-to-all, that AI training generates. Diagnosing this has become a sub-discipline of its own, with operators profiling per-collective timings the way an earlier generation of engineers profiled per-function runtimes.

Specific percentages in public discussion should be read as estimates rather than measurements, since operators rarely publish communication-time breakdowns. The figures vary with model, parallelism strategy, and fabric — and that variance is itself the point: communication overhead is not a fixed tax but a design variable, and clusters are judged by how small they can make it.

07 What to Watch Next

Three milestones will shape the next few years. First, 1.6-terabit optics moving from demonstrations to volume deployment, roughly doubling the traffic a single switch port can carry. Second, co-packaged optics crossing from announcements to shipping product, the true test of whether the joules-per-bit argument survives manufacturing at scale. Third, the first Ultra Ethernet-specification networks in production, which will show whether an open fabric can match InfiniBand-class behavior at Ethernet-class cost.

The deeper trend is architectural: a split between a scale-up tier, where bandwidth is abundant and latency is engineered to near zero, and a scale-out tier, where cost per bit rules. As accelerator generations continue to outpace networks, expect that split to harden — and expect the nervous system of the AI data center to keep drawing as much engineering attention as the muscles it connects.