Low Latency, High Output: How Forge Beats Cloud Time-to-First-Token

A developer waiting on a coding assistant is not waiting the way someone waits for a page to load. They are mid-thought, halfway through reasoning about a function signature, a test failure, a refactor, and the assistant's job is to arrive inside that window before the thought dissolves into something else. Time-to-first-token is the measure of whether it does. It is the single latency number that determines whether a suggestion lands as a continuation of the developer's own thinking or as an interruption to it.

That makes TTFT a different kind of metric for a coding copilot than it is for most other AI workloads. A batch summarization job or an overnight analysis pipeline can absorb a few extra seconds of queuing without anyone noticing. An interactive completion cannot. The developer is looking at the cursor. Every additional second of latency is a second in which the cost of switching away, to a search tab, to a different file, to just writing the code by hand, starts to look lower than the cost of waiting, and once that switch happens, the assistant has lost the moment it existed to serve.

Why cloud APIs struggle with this specific moment

A hosted, multi-tenant inference API is not built around any single developer's moment of thought. It is built to absorb aggregate demand from many customers at once, and that design goal shapes its latency behavior in ways that are largely invisible until a request happens to arrive at the wrong instant. A request from a developer's IDE has to cross the public internet or a dedicated link to reach the provider's infrastructure, clear TLS negotiation and provider-side routing, and then wait in a batching queue whose depth depends on how many other tenants happen to be active at that exact moment: a queue the requesting organization cannot see and cannot provision for.

None of that variability is a defect in cloud inference; it is the direct consequence of amortizing expensive compute across a large, unrelated customer base, and it works well for workloads that can tolerate it. But a coding assistant sitting inside a live editing loop cannot tolerate it gracefully, because the cost of unpredictable latency is not evenly distributed across the workday. A developer who has learned that "sometimes it's instant and sometimes it takes a while" stops trusting the tool to be there at the moment they need it, and starts treating every invocation as a small gamble rather than a reflex. The unpredictability, more than the average latency, is what breaks the loop.

What Forge removes from the path

Forge is a sealed, single-tenant appliance, which means the request path for a developer's completion never leaves the building it runs in and never shares compute with an unrelated organization's traffic. Two of the largest, least controllable contributors to cloud TTFT, public-network transit and cross-tenant queuing, are architecturally absent rather than merely reduced. The request travels a local-network hop from the IDE to the appliance, and the only workload contending for compute is this organization's own engineering traffic, which is a queue an operator can reason about, size for, and monitor directly.

What remains is the genuinely architectural part of latency: the prefill pass, where the model processes the full input context, the prompt, the surrounding code, whatever repository context has been retrieved, before it can emit a first token. That cost scales with how much context is fed in and with the model's own size, and it is shaped directly by the accelerator hardware and the inference engine scheduling work on top of it. Substrate's inference layer is tuned specifically for this case: a single organization's interactive, bursty completion traffic, where the objective is consistent low latency per request rather than maximum aggregate throughput across many strangers' workloads. Continuous batching lets new completions join in-flight work rather than queue behind whatever arrived first, which matters enormously for a usage pattern built from many short, overlapping requests rather than a few long ones.

Repo-aware context without a network round trip

A repo-aware coding assistant's prompts are rarely just the text a developer typed. They typically carry retrieved context, related files, prior conversation state, symbols pulled from the codebase, assembled specifically so the model can reason about a change in light of the surrounding system. On a cloud-hosted assistant, assembling that context and shipping it to a remote model means the retrieval step and the network transit step stack on top of each other, and each one adds to the interval before a token appears.

On Forge, retrieval happens on the same enclave that runs inference, against an index that lives on the appliance rather than behind an external call. That does not eliminate the cost of retrieval or of a long prefill (a large, well-populated context window is genuinely expensive to process, on any architecture), but it removes a layer of network overhead stacked on top of retrieval, and it means the entire path from keystroke to first token stays inside infrastructure the organization operates and can observe end to end. For a workload defined by short, frequent, context-heavy requests, that compounding matters more than it would for a workload made of a few long-running calls.

High output is a consequence of low, predictable latency

"High output" for a coding assistant is not primarily about raw tokens-per-second once generation is underway. It is about how many times per hour a developer actually reaches for the tool instead of working around it. A slow or unpredictable assistant trains its own users out of using it: developers route around it for the exact moments it would help most, the mid-thought completions and quick sanity checks, because the wait doesn't feel worth it. A fast, consistent one gets invoked constantly, because there is no real cost to asking. Output, in the aggregate, is downstream of trust in latency, and trust in latency is built request by request rather than declared by a single benchmark figure.

This is also why consistency matters as much as raw speed. A developer can adapt to an assistant that is reliably a little slower than instant. A developer cannot adapt to one that is instant nine times out of ten and inexplicably sluggish on the tenth, because the tenth is the one that erodes trust in the other nine. Removing cross-tenant queuing does not just lower average TTFT. It narrows the spread between best-case and worst-case latency, which is frequently the more important number for a tool meant to live inside a developer's flow.

What this does not claim

None of this is a claim that a sealed, single-tenant appliance is unconditionally faster than every cloud API under every condition, or that Forge produces a fixed TTFT independent of context length, model size, and concurrent developer load, all of which remain real variables that a deployment has to be sized for. A cloud provider with vastly larger aggregate compute can, for a given moment, bring more raw hardware to bear on an individual request than a fixed on-premises footprint can. What Forge changes is which variables are within an operator's control and which are not: prompt length, model choice, and hardware provisioning are levers a team can plan around, while cross-tenant queuing and public-network transit, on a shared cloud endpoint, are not.

For a coding assistant, that distinction is the whole point. The moment a developer is willing to wait for is short, and it recurs hundreds of times a day across an engineering organization. An architecture that removes the sources of latency an operator cannot see or control, and leaves only the ones a deployment can be sized and tuned for, is the more honest way to make that moment reliably fast.