High-Concurrency Networking: Serving Hundreds of API Calls on Local 100GbE

In a cloud deployment, serving hundreds of concurrent API callers is somebody else's problem first. A managed load balancer distributes connections across an elastic pool of backend instances, autoscaling adds capacity when queue depth rises, and the network fabric connecting all of it is operated by a hyperscaler with a research budget devoted to exactly this problem. A sealed appliance sitting inside a secure facility has none of that. There is no elastic pool, only the hardware that shipped. There is no autoscaling, only the accelerator memory and compute that's physically present in the chassis. And there is no hyperscaler-operated fabric, only whatever network sits between the appliance and the engineers, analysts, or services calling it, which for a Forge deployment is typically a local 100 gigabit Ethernet segment inside the customer's own facility.

That constraint sounds limiting, and in one sense it is: an appliance can't out-provision a traffic spike the way a cloud service can. But it also means every part of the request path is knowable in a way a multi-tenant cloud service isn't. There's no noisy neighbor, no cross-region routing, no dependency on infrastructure outside the sealed boundary. The engineering problem shifts from "how do we scale elastically" to "how do we make full, correct use of a fixed, known amount of hardware." That's a more tractable problem, provided the fan-out from network to compute is designed deliberately rather than assumed to just work.

Why hundreds of concurrent callers is a real design point, not an edge case

A Forge appliance exists to serve a repo-aware coding assistant to an entire engineering organization at once. That means the concurrency isn't hypothetical — it's the normal operating condition. At any given moment, dozens to hundreds of engineers can be issuing completion requests, running retrieval queries against the indexed codebase, or holding open a long-lived session while they work. Unlike a single-user local model running on a workstation, where "concurrency" means one request at a time, an organization-wide appliance has to behave like a shared service even though it's running on hardware with a fixed accelerator budget.

The naive failure mode is treating this as purely a compute problem — buy enough GPU to run every request as fast as possible — while ignoring that the requests have to arrive, get routed, get queued, and get answered over a network path that has its own limits. A 100GbE local fabric is fast, but "fast" doesn't mean infinite, and the way requests get batched, scheduled, and returned across that fabric determines whether hundreds of simultaneous callers experience a system that feels responsive or one that feels like it's fighting itself.

What actually saturates first

The instinct is to assume the accelerator is the bottleneck under load, and eventually it is. But it's rarely the first thing to saturate. Before compute becomes the constraint, a high-concurrency inference service tends to run into three earlier pressure points.

The first is connection handling at the network edge of the appliance. Hundreds of clients opening and holding connections, particularly with streaming responses that keep a socket open for the duration of a generation, puts real load on the request-handling layer well before it touches an accelerator. A design that assumes low concurrency and handles connections synchronously will queue callers at the front door long before the GPU is doing meaningful work.

The second is memory bandwidth and KV-cache pressure on the inference layer itself. Every concurrent session holds state — context, retrieved documents, the model's working memory for that conversation. As concurrent sessions climb, the constraint often isn't raw compute throughput but whether there's enough fast memory to hold everyone's context without evicting it prematurely, which would force expensive recomputation and show up to the caller as inconsistent latency rather than a clean slowdown.

The third, and the one specific to this topic, is the network fabric itself: how request and response traffic moves between wherever it originates on the local network and the node actually running inference, and, for appliances with more than one accelerator or more than one node, how traffic is distributed across them. A 100GbE link is generous bandwidth for API-scale traffic, but bandwidth headroom doesn't automatically translate into low, consistent latency under hundreds of simultaneous short-lived and long-lived connections if the routing and batching logic behind it wasn't built for that pattern.

Batching without punishing the first caller

The core tension in serving many concurrent inference requests efficiently is that accelerators achieve their best throughput when work is batched together, but batching means the first request in a batch has to wait for enough other requests to arrive before processing starts — or accept a small delay window deliberately introduced to let a batch form. Get that window wrong in either direction and something breaks: too eager, and the accelerator runs inefficiently on tiny batches, wasting capacity that could have served more callers; too patient, and individual callers feel the system hesitate before it starts responding at all.

Modern inference-serving approaches address this with continuous or in-flight batching, where new requests can join a batch that's already executing rather than waiting for a batch boundary, and where a long-running generation for one caller doesn't block a short one from a different caller from getting scheduled. That's an architectural choice in the serving layer, not something a fast network alone provides. But it's also a choice that only pays off if the network delivering requests to that scheduler isn't itself introducing jitter that undermines the scheduler's assumptions about when work arrives.

Why the fabric topology matters more once there's more than one node

A single well-specified inference node can absorb a meaningful amount of concurrent load on its own, and for many Forge deployments that's the right starting scope. But once a deployment's concurrency needs push past what a single node comfortably serves, the appliance is no longer just answering a networking question at its edge — it's answering one internally, between nodes.

Two nodes sharing inference load need a way to distribute incoming requests that doesn't itself become a serial bottleneck, and if any part of a workload requires coordination between nodes, such as shared retrieval state or a router making placement decisions per request, that coordination traffic rides the same local fabric as the API traffic itself. A 100GbE local network gives real headroom for this, but headroom is not the same as a topology that uses it well. A flat network where every node talks to every other node through a single shared switch behaves very differently under load than one where request routing, retrieval traffic, and inter-node coordination are given deliberate separation, whether that's through traffic prioritization, dedicated paths, or simply keeping chatty coordination traffic off the same path as large response payloads streaming back to callers.

This is also where the appliance's sealed nature is an advantage rather than a complication. Because the fabric is entirely local and entirely under one operator's control, there's no need to design for unpredictable multi-tenant traffic sharing the same links, and no need to account for a provider's routing decisions made outside the customer's visibility. The topology inside the sealed boundary can be scoped precisely to the one workload it's serving.

Latency consistency matters more than peak throughput

For a team of engineers relying on a coding assistant throughout the workday, the number that matters isn't peak requests-per-second under synthetic load. It's whether the hundredth concurrent request today feels the same as the fifth one did an hour ago. A system that's fast on average but occasionally stalls under bursty load is worse, from a working engineer's perspective, than one that's consistently a little slower. Bursty behavior is usually a symptom of one of the earlier pressure points (connection handling, cache eviction, batching windows) getting exposed by network-level jitter rather than smoothed out by it.

That's why appliance networking design for this use case treats consistency, not just raw throughput, as the target metric. A local 100GbE fabric sized and topologized for the actual concurrency profile of an engineering organization — not just for a benchmark's peak number — is what keeps hundreds of simultaneous callers each experiencing the appliance as their own responsive assistant, rather than as hundreds of people competing for one machine's attention.

The design principle underneath all of it

None of this is solved by bandwidth alone, and none of it is solved by compute alone. Serving hundreds of concurrent API callers on a sealed, fixed-hardware appliance is a systems problem that spans connection handling at the edge, memory-aware scheduling in the inference layer, and a local network fabric topologized for the specific pattern of traffic an organization actually generates: short requests, long streaming responses, and occasional inter-node coordination, all sharing the same sealed boundary. Getting each layer right independently isn't sufficient. They have to be scoped together, against the concurrency the deployment will actually see, rather than against a generic assumption that fast local networking makes the rest of the problem disappear.