High-Speed Interconnects on the Edge: NVLink and PCIe Gen 5 in a Box

A GPU's compute throughput is the number most spec sheets lead with, and the number most buyers compare first. It is also, on its own, a poor predictor of how a multi-accelerator system actually performs on a training or fine-tuning workload. The reason is unglamorous but decisive: once a model no longer fits comfortably on a single accelerator, the system's real ceiling is set by how fast the accelerators can talk to each other, not by how fast any one of them can compute. A cluster of fast GPUs joined by a slow link behaves like a fast car stuck behind a slow one. The bottleneck sets the pace for everyone.

This is not a niche concern that only shows up at hyperscale. It is present the moment a workload spans more than one accelerator, which is the normal case for any serious fine-tuning or continued-pretraining job. Getting interconnect right, and getting it right inside a sealed, single-chassis appliance rather than a datacenter rack with room to spread out, is one of the harder architectural problems in building sovereign AI hardware.

Why interconnect becomes the bottleneck

Training and fine-tuning are not embarrassingly parallel workloads where each accelerator can just grind on its own slice of the problem and report back at the end. Gradients computed on one GPU have to be synchronized with gradients computed on every other GPU in the job, repeatedly, once per training step. When a model is too large to fit in a single accelerator's memory, its layers or parameters get sharded across multiple GPUs, and now a single forward or backward pass requires those GPUs to exchange activations and gradients constantly, not just at the end of a batch.

Every one of those exchanges crosses whatever link connects the accelerators. If that link is slow relative to the accelerators' own compute speed, the GPUs spend a meaningful fraction of every training step idle, waiting on data from a neighbor rather than doing math. The practical effect is that a job's wall-clock time can be dominated by communication overhead rather than computation. Adding more accelerators to a poorly interconnected system can make this worse, not better, because each additional GPU adds more synchronization traffic without adding proportional communication bandwidth to carry it.

Two different links, two different jobs

Two connectivity layers matter in a multi-GPU system, and they solve different problems.

PCIe (PCI Express) is the general-purpose bus that connects a GPU to the rest of the system: the CPU, system memory, storage, and networking. Every accelerator in the chassis needs a PCIe connection regardless of anything else, because that is how the host system loads data onto the GPU, retrieves results, and orchestrates the job in the first place. Successive PCIe generations roughly double per-lane bandwidth over their predecessor, so a newer PCIe generation matters for how fast data gets onto and off of the accelerators (checkpoints, dataset shards, activations moving to and from host memory) even before GPUs start talking to each other.

GPU-to-GPU interconnect is a separate, purpose-built layer that exists specifically so accelerators can exchange data directly with each other at much higher bandwidth and lower latency than routing that traffic through the general-purpose PCIe fabric and the host CPU would allow. NVLink is the best-known example of this class of interconnect: a direct, high-bandwidth path between GPUs that lets them share memory and synchronize gradients without every exchange detouring through the CPU. The distinction matters because a system can have perfectly respectable PCIe connectivity and still bottleneck badly on multi-GPU training if the GPU-to-GPU path is left riding on that same general-purpose bus instead of a dedicated high-speed link.

What changes when the datacenter rack becomes a sealed box

In a hyperscale data hall, interconnect topology is largely a layout problem with generous tolerances: racks are wired with room to route cabling, cooling is provisioned at the room level, and adding bandwidth is often a matter of adding more of something, more racks, more switches, more cable runs. None of those assumptions survive the move into a sealed, single-chassis appliance built for a facility that cannot host a rack, let alone a room full of them.

Space is the first constraint. A sealed enclosure has a fixed internal volume, and every trace, connector, and cable run for a high-speed interconnect has to fit inside it without compromising the thermal and physical-security engineering the rest of the chassis depends on. High-speed digital signaling is also unforgiving about physical layout — trace length, routing symmetry, and signal integrity all degrade the interconnect's effective bandwidth if the board and backplane design does not account for them from the start, and there is far less room to compensate for a suboptimal layout than there would be with cable slack in a rack.

Thermal load compounds the problem rather than sitting next to it. The accelerators and the interconnect hardware that ties them together are both generating heat inside the same sealed volume discussed elsewhere in this series, and a densely packed multi-GPU interconnect topology concentrates that heat rather than spreading it across a room's worth of rack units. Power delivery follows the same logic: multiple accelerators wired together at datacenter-class bandwidth draw and switch significant power in a compact footprint, and the electrical design has to hold up under sustained load, not just survive a brief burst.

None of this is a reason multi-GPU interconnect is impractical in a sealed appliance. It is the reason it has to be engineered deliberately, as a first-order design constraint alongside compute and thermal budget, rather than treated as an afterthought once the accelerators are chosen.

Topology is a design decision, not a default

How accelerators within a chassis are wired to each other — which GPUs share a direct high-speed link, which pairs have to route through an intermediate hop, how PCIe lanes are partitioned across GPUs, storage, and host connectivity — is a topology decision made deliberately for the workload the appliance is built to run, not a default that falls out of picking components off a shelf. A topology tuned for a small number of tightly coupled accelerators working on one large sharded model looks different from one tuned for a larger number of accelerators each working on more-independent slices of a job. Getting this wrong produces a system where the accelerators themselves are fast but the training run is not, because the topology introduces exactly the kind of synchronization stalls that fast interconnect exists to eliminate.

This is also where the sealed form factor forces genuine engineering tradeoffs rather than theoretical ones. A datacenter design that under-provisions interconnect can sometimes paper over the gap by adding more switches or re-cabling a rack. A sealed appliance does not get that flexibility after the fact. The interconnect topology has to be right at the point the chassis is built, because it is not something a customer, or even a field technician, can meaningfully rewire once the unit is sealed and deployed.

Where this matters most across the lineup

Interconnect design is not a uniform concern across every E31 device, because not every workload stresses the GPU-to-GPU link the same way. Czar, built for training and fine-tuning on sensitive data, is where this engineering effort concentrates: multi-day fine-tuning runs on models sharded across several accelerators are exactly the workload class where interconnect bandwidth determines whether a job finishes in a reasonable window or stalls on communication overhead the whole way through. Forge's inference workload for a repo-aware coding assistant typically runs on a single accelerator or a small, less tightly coupled set of them, so while it still depends on solid PCIe connectivity to move data efficiently, it does not lean on the same class of dedicated GPU-to-GPU fabric that a training job demands. Chassis builds inherit the same underlying interconnect engineering discipline, but the specific topology in any given build is scoped to the workload of the deal it is built for, since an ISV shipping an inference-heavy product and one shipping a training-capable R&D tool are not asking the interconnect to do the same job.

The broader point holds across all of it: an accelerator's advertised compute throughput describes what one chip can do in isolation. What a sealed, multi-accelerator appliance can actually deliver on a real training or fine-tuning workload is set as much by the links between the chips as by the chips themselves. Inside a sealed enclosure, engineering those links well is a harder, more deliberate problem than it is in a data hall with room to spare.