Serving the Whole Team: High-Concurrency Inference for 100+ Local Developers

A pilot deployment is a forgiving environment. Ten or twenty engineers trying out a coding assistant generate bursty, forgiving traffic: someone types, waits, reads a suggestion, moves on. Whatever scheduling logic sits behind the appliance barely has to work for the pilot to feel fast. The real test comes later, when the pilot succeeds and the rollout scope changes from "a team" to "the organization." At that point the appliance isn't answering occasional questions from a friendly early-adopter group anymore. It's the shared coding assistant for a hundred or more developers, all working during the same overlapping hours, all expecting the same instant responsiveness they had when they were one of twenty.

Nothing about the hardware changed between those two moments. What changes is the shape of demand, and whether the system in front of that fixed hardware was built to stay fair and predictable under it, or was only ever exercised at pilot scale.

Concurrency at this scale is not "more of the same traffic"

It's tempting to model a hundred-developer rollout as a linear scale-up of a ten-developer pilot: five times the users, so roughly five times the load, handled by proportionally more capacity. That model undercounts the problem. A pilot group's usage is naturally spread out: different people, different tasks, different moments of needing an answer. An organization-wide rollout compresses that spread. Standups happen at the same time. Sprint deadlines land on the same day. A team debugging a shared incident all query the assistant about the same file within minutes of each other. Concurrency doesn't arrive as a smooth average load. It arrives in correlated bursts driven by the actual rhythm of how engineering teams work together.

That means the appliance has to be provisioned and scheduled for its peak overlapping demand, not its average demand across a day. A system tuned to look good on an average-throughput number can still leave real developers staring at a spinner during the ten minutes when everyone on the floor happens to be using it at once. And those ten minutes, for a working engineer, are the only ten minutes that define whether the tool feels trustworthy.

Every session is a citizen, not a request

The unit that matters for a coding assistant isn't the individual API call. It's the developer's session. A session carries open context: the files they're working in, the retrieval state built from their query history, sometimes a long-running conversation about a specific bug. That state has to live somewhere close to fast memory for the session to feel responsive, and it competes for that same fast memory with every other developer's active session.

At pilot scale, holding every active session's context resident is trivial: there simply aren't enough concurrent sessions to create pressure. At organizational scale, session memory becomes a genuinely scarce resource that has to be actively managed: which sessions stay warm, which get gracefully evicted after a period of inactivity, and how eviction is handled so that a developer returning to a stale session experiences a brief re-warm rather than a jarring loss of context mid-task. Getting this wrong doesn't fail loudly. It shows up as one specific developer's experience quietly degrading while the aggregate numbers still look fine, which is exactly the kind of problem that survives a demo and only surfaces once real people depend on the tool every day.

Fairness is a scheduling decision, not an accident

A single fast developer issuing a long, complex query (a large refactor suggestion, a broad repository-wide retrieval) can consume a disproportionate share of available compute if the scheduler treats all work as equal once it's admitted. On a shared appliance, an inference layer that doesn't actively manage this will let a handful of heavy users degrade the experience for everyone else without anyone doing anything wrong. Nobody violated a policy. The system just let one workload crowd out a hundred smaller ones because nothing was watching for that pattern.

Serving a whole team well means treating fairness as a deliberate scheduling property rather than an emergent one. Short, interactive completions, the bread-and-butter of a coding assistant, where a developer is waiting in the loop, generally need to be prioritized differently than longer background-style queries, so that one doesn't stall the other. Continuous batching, where new requests can join in-flight work rather than waiting behind whatever arrived first, is part of how this gets solved at the inference-engine level. But the scheduling policy on top of it, deciding how work gets ordered and how much of the accelerator's attention any one session is allowed to hold at once, is what actually determines whether a hundred concurrent developers each feel like they have the assistant's full attention.

What degrades first, and how it should degrade

Every fixed piece of hardware has a ceiling, and a rollout that grows past its pilot scope will eventually approach that ceiling during peak hours even when it's well provisioned for its intended headcount. The design question isn't whether that ever happens. It's what the system does when it does.

The failure mode to design against is silent, uneven degradation: some developers get normal latency, others get inexplicably slow responses, and nobody can tell whether it's their query, their network segment, or the system itself. That's the experience that erodes trust in the tool fastest, because it looks like unreliability rather than capacity. The better failure mode is visible and even: latency that stretches predictably and proportionally for everyone under genuine peak load, ideally with the system itself surfacing that it's under pressure rather than degrading invisibly. A hundred developers who occasionally see a consistent, explainable half-second of extra wait during the worst five minutes of the day will trust the tool far more than a hundred developers where ten of them, at random, quietly get a materially worse experience than everyone else.

Sizing for the org, not for the demo

Because Forge runs on a fixed hardware footprint rather than an elastic cloud pool, the sizing conversation has to happen honestly and up front, scoped to the actual organization the appliance will serve, not to whatever concurrency a pilot happened to exercise. That means characterizing, as concretely as the customer can, what "peak overlapping demand" actually looks like for their engineering org: team size, working-hour overlap across timezones if relevant, the mix of short interactive completions versus heavier retrieval-driven queries, and how correlated usage tends to get around standups, releases, and incidents. A deployment sized against average daily load will feel fine in a demo and strained in production; one sized against realistic peak concurrency is built for the moment that actually matters.

This is also why the conversation about growing past a single node's comfortable concurrency ceiling belongs early in planning rather than as a reactive fix once developers start complaining. Multi-node scaling for a Forge deployment is a known path, not an afterthought, but it works best when it's anticipated in the initial sizing rather than retrofitted under pressure from a hundred impatient engineers who just want their assistant back.

The point of a hundred-developer rollout

A coding assistant that only works well for a small pilot group hasn't actually solved the problem an engineering organization has. It's solved a smaller, easier version of it. The real measure of a Forge deployment is whether the five-hundredth developer to adopt it, opening a session during the busiest ten minutes of the sprint, gets the same responsive, private, on-enclave experience as the first developer who tried it in the pilot. That's a scheduling and capacity-planning problem as much as it is a hardware problem, and treating it as one from the start is what lets a sealed appliance actually serve as the assistant for an entire engineering team, rather than a demo that happened to work for a few.