Benchmarking Local Llama-3-Code: Performance on Forge

Ask a team evaluating a coding copilot what "fast" means and you will get a cloud-shaped answer by default: time to first token, tokens per second, maybe a leaderboard score against a public benchmark suite. Those numbers are real and worth measuring. They are also incomplete for a sealed appliance, because they describe a model in isolation, running against a warm inference cluster with no repository context, no retrieval pass, and no concurrency from other engineers on the same box. Forge does not run that way. Evaluating a local code model like Llama-3-Code on Forge means measuring the whole path a real request takes: index lookup, context assembly, generation, and however many engineers are hitting the appliance at once, because that is the path every actual suggestion travels.

This is a framework for thinking about that measurement, not a scorecard. Any specific throughput or latency figure depends on the hardware configuration, the repository size, the concurrency profile, and the exact model checkpoint in use, all of which vary by deployment. What follows is architectural: which factors move the needle, in which direction, and what a team should actually instrument before drawing conclusions from a benchmark run.

Why leaderboard numbers don't transfer

Public benchmark results for code models are typically produced under conditions that have almost nothing in common with an air-gapped appliance serving a live engineering team. The benchmark harness issues one prompt at a time to a model with no additional context beyond the prompt itself, on infrastructure sized and warmed specifically for that run. Real usage on Forge looks different in every one of those dimensions. A request arrives already carrying repository context pulled from the local index: relevant files, recent commits, symbol references, because that context is most of what makes a suggestion useful rather than generic. Multiple engineers issue requests concurrently, competing for the same accelerator. And the appliance is provisioned once, as fixed hardware, rather than elastically scaled per query.

None of that makes leaderboard numbers meaningless. They are a reasonable proxy for raw model capability, and a useful filter when choosing which checkpoint to deploy in the first place. But treating a leaderboard's tokens-per-second figure as a prediction of what an engineering team will experience on a shared appliance is the most common mistake in evaluating local inference. The gap between the two is not noise; it is the exact set of things a sealed, multi-user, context-aware deployment adds on top of raw generation.

The path a real request actually takes

A useful benchmark for Forge has to account for four stages, because each one adds latency and each one is affected differently by hardware choices and repository size.

Retrieval comes first: before the model generates anything, Forge's retrieval layer resolves what context the request needs from the local repository index (relevant files, related symbols, prior related completions). This step scales with repository size and index freshness, not with model size, and on a large monorepo it can be a meaningful fraction of total latency in its own right. A benchmark that only times model generation and ignores retrieval is measuring a different, easier problem than the one engineers actually experience.

Context assembly follows: retrieved material gets packed into the model's context window, and how efficiently that packing happens affects both prompt length and generation quality. Longer, denser context tends to improve suggestion relevance but costs prefill time. The model has to process the entire prompt before producing a first output token, so a context-rich request measurably increases time to first token even before generation starts.

Generation is the stage most benchmarks actually measure: tokens produced per second once the model starts responding. It is real, and it does depend heavily on accelerator class, memory bandwidth, and quantization choices. But it is one stage among four, and on a context-heavy coding request it is frequently not the largest contributor to what an engineer experiences as "how long did that take."

Concurrency is the stage that disappears entirely from single-request leaderboard numbers and matters most in practice. A Forge appliance serving ten engineers simultaneously is not running ten independent copies of the benchmark. It is scheduling ten requests against shared compute, and how gracefully that scheduling degrades under load is arguably the single most important performance property for a real deployment. A model that looks excellent in a single-request benchmark and falls apart at five concurrent users is not actually fast for a team.

What to hold constant, what to vary

A benchmark run that is going to inform a real decision needs a methodology, not a single number. The variables worth isolating, deliberately, one at a time:

Concurrency level: measuring latency and throughput at one, several, and peak expected simultaneous users, because the interesting behavior is almost always in the degradation curve, not the single-user floor.

Repository size and index state: a freshly indexed small repository and a large, actively-changing monorepo will produce different retrieval-stage costs, and a team should benchmark against something close to its actual codebase, not a synthetic small one that flatters the numbers.

Context window utilization: short, narrowly-scoped completions versus longer, whole-file or cross-file suggestions exercise the prefill and generation stages very differently, and a team's actual usage pattern (autocomplete-style versus larger generative tasks) should drive which one gets weighted more heavily in the evaluation.

Quantization and precision choices: running a coding model at different precision levels trades some generation quality for throughput and memory footprint, and where that tradeoff should land depends on the sensitivity of the workload, not on a universal default.

Sustained load versus burst: a benchmark that runs for thirty seconds measures something different from one that runs for an hour, because thermal behavior, memory pressure, and cache effects on a fixed appliance only show up under sustained use, and a fixed-hardware deployment does not get to shrug off sustained load the way an elastically scaled cloud service can.

Reporting a single "tokens per second" figure without specifying which of these conditions produced it is not a benchmark result. It is a data point with the methodology stripped off, and it should be treated with the same skepticism a rigorous engineering team would apply to any unsourced performance claim.

What's actually fixed versus tunable on Forge

Some performance levers are properties of the deployment and worth understanding before a benchmark is designed. The accelerator class and memory configuration of the appliance are fixed at provisioning time. This is sealed, purpose-built hardware, not a cloud instance that can be resized between test runs, so a benchmark should reflect the actual unit a team will run on, not a different reference configuration. The retrieval index is local and incremental, updated as the repository changes rather than rebuilt per query, so index-warmup state at benchmark time should be disclosed alongside the results rather than assumed away. And because Forge runs as a sealed appliance rather than a metered API, there is no throttling or rate-limiting behavior to benchmark around. The ceiling a team hits under heavy concurrent load is the appliance's actual physical ceiling, not a policy limit, which makes concurrency testing more informative on Forge than it typically is against a cloud endpoint where provider-side rate limits obscure the hardware's real behavior.

Reading a benchmark result honestly

The purpose of benchmarking a local code model on Forge is not to produce a marketing number. It is to answer a specific operational question: will this configuration hold up under this team's actual concurrency and repository size, at a latency profile engineers will tolerate before they start working around the tool instead of with it. That answer is specific to a deployment, and it should stay specific. A number produced on a lightly loaded appliance against a small test repository does not generalize to a fifty-engineer team working a large monorepo, and treating it as if it does is the same error as trusting a cloud leaderboard number in the first place, just committed against your own hardware instead of someone else's.

The more durable habit is architectural: know which of the four stages, retrieval, context assembly, generation, concurrency, is the actual bottleneck for your usage pattern, because that determines what tuning effort will move the needle. A team bottlenecked on retrieval gains little from a faster accelerator. A team bottlenecked on concurrent generation gains little from re-indexing more aggressively. Benchmarking with that structure in mind turns a single throughput number into an actual diagnosis, which is the only version of a benchmark worth acting on inside a sealed environment where you cannot simply add more cloud capacity and watch the problem disappear.