Time-to-First-Token: What Actually Determines Latency in a Sealed Appliance

Time-to-first-token, or TTFT, is the interval between a request landing and the first token of a response arriving. It is the latency metric users feel most directly: the difference between an interface that feels responsive and one that feels like it is thinking about whether to respond at all. It is also one of the most frequently misunderstood numbers in inference infrastructure, because it gets treated as a single property of a model rather than as the output of an entire system.

That distinction matters more, not less, once inference moves off a shared cloud API and onto a sealed, single-tenant appliance. The variables that drive TTFT on each of those two architectures overlap only partially, and some of the biggest cloud-side latency sources simply do not exist in a sealed deployment, while a few new ones take their place. Understanding which is which is more useful than any single benchmark number, because the honest answer to "how fast is a sealed appliance" is "it depends on which part of the latency budget you are asking about."

What TTFT is actually made of

Before a first token can be generated, a request has to travel through several distinct stages, and each one contributes its own slice of latency. Network transit gets the request from wherever it originated to wherever inference happens. A request queue absorbs the gap between when a request arrives and when a compute slot is free to work on it. The prefill pass processes the entire input prompt through the model in one shot, building up the key-value cache the model needs before it can emit anything. Only after prefill completes does the first output token get generated and returned.

Each of those stages behaves differently depending on where the system sits and who else is using it at the same moment. That is the crux of why comparing TTFT across architectures is not a matter of comparing two numbers. It is a matter of comparing two very different sets of stages that happen to produce a number with the same name.

Where cloud API latency actually comes from

A hosted, multi-tenant API is, by design, sharing compute across many customers' requests simultaneously, and most of what makes its TTFT variable traces back to that sharing rather than to the model itself. Network transit is the first variable: a request from an on-premises environment to a cloud endpoint travels across the public internet or a dedicated link, through TLS negotiation and whatever routing sits between the caller and the provider's data center, and that round trip is pure overhead added before the model ever sees the prompt.

Queuing is the second, larger variable, and it is the one most responsible for the wide swings users notice in day-to-day cloud API latency. A shared endpoint is serving an unknown number of other tenants' requests at any given instant, and a request arriving during a demand spike waits behind however much load happens to be ahead of it in the batching queue. This is why the same prompt sent to the same hosted model can return a first token in what feels instant at one hour and noticeably slower at another. The model's own prefill cost has not changed, but the queue in front of it has. Rate limiting and provider-side load balancing add a further layer of variability on top, since a request can be routed to different underlying capacity depending on real-time demand across the provider's whole customer base.

None of this is a criticism of cloud inference as an architecture. It is a direct, and mostly reasonable, consequence of amortizing expensive compute across many tenants. But it does mean that cloud TTFT is, structurally, a function of aggregate demand as much as it is a function of the model and the prompt.

Where sealed-appliance latency actually comes from

A sealed appliance removes tenancy sharing and public-network transit from the equation entirely, because the hardware is dedicated to one deployment and typically sits on the customer's own network or air-gapped environment. That does not make TTFT zero or automatically low. It changes which variables are left to matter.

Network transit shrinks to whatever local-network hop separates the calling application from the appliance, which is a fraction of a public-internet round trip under ordinary conditions. Queuing still exists, but it is now a function of that single organization's concurrent request volume against that appliance's provisioned capacity, not the aggregate demand of an unrelated customer base, a queue an operator can reason about and provision for, rather than one shaped by traffic they cannot see.

What is left, largely, is prefill cost, and prefill cost is genuinely architectural rather than incidental. It scales with prompt length, since the model has to process every input token before it can emit the first output token, which is exactly why retrieval-augmented workloads that stuff long context windows into every request carry a heavier latency cost than short, targeted prompts. It scales with model size, since a larger parameter count means more computation per token processed. And it is shaped directly by the accelerator hardware doing the work and the software stack orchestrating it: memory bandwidth for moving weights and activations, compute throughput for the matrix operations prefill requires, and how well the inference engine batches and schedules concurrent requests on that hardware. Substrate's inference engine is engineered specifically around this last variable: batching and scheduling decisions inside the appliance are tuned for the single-tenant case, where the goal is consistent, predictable latency for one organization's traffic pattern rather than maximum aggregate throughput across many unrelated ones.

Why "which is faster" is the wrong first question

Put the two architectures side by side and the honest comparison is not a single winner. A sealed appliance structurally eliminates two of a cloud API's largest and most variable latency contributors, public-network transit and cross-tenant queuing, which is why its TTFT tends to be more consistent request to request, for a workload of comparable prompt length and model size. A cloud API, in exchange, can bring far larger aggregate compute capacity to bear on any single request, which can matter for very large models or very heavy prefill workloads where raw hardware scale outweighs the tenancy overhead.

What a buyer evaluating latency should actually be asking is narrower and more useful than "which architecture is faster": what does prefill cost look like for my typical prompt length and model size on the specific hardware in this appliance, and how much of my current cloud latency is coming from queuing and network transit rather than from the model doing genuine work. For workloads with long retrieved context, which describes a large share of enterprise RAG deployments, prefill can dominate the latency budget on either architecture, and no amount of eliminating network hops changes that; it is a reason to think about retrieval design and context length as a latency lever in its own right, not just a hardware question.

Benchmarking latency honestly

Any credible TTFT comparison has to hold prompt length, model size, and concurrency constant across both sides, because varying any of them changes which stage of the pipeline is actually being measured. A single-request, short-prompt benchmark mostly measures network transit and raw prefill speed. A benchmark run under realistic concurrent load measures queuing behavior as much as anything else, and that is the condition under which the two architectures diverge most, since a sealed appliance's queue depth is bounded by one organization's traffic while a shared API's is not. Reported latency numbers that do not specify concurrency, prompt length, and model configuration are not comparable to anything, including each other, and any claimed benchmark result, from a vendor, including this one, is only meaningful alongside the workload conditions it was measured under.

None of this is a case that a sealed appliance is unconditionally faster than a cloud API, or the reverse. It is a case that TTFT is a systems property, not a model property, and that the sealed architecture changes which parts of that system are within the operator's control. For a buyer whose real constraint is that source code, sensitive research data, or regulated records cannot leave the building in the first place, the latency conversation happens on the appliance side of that line by default. At that point the more useful engineering question is not whether it is fast, but which levers, prompt length, concurrency, hardware provisioning, actually govern how fast it is.