Evaluation on Czar Forge: nothing is promoted on a feeling

A team that has started using agents on its code soon meets a question a demo cannot answer. A new prompt seems to produce better reviews. A tuned model seems to write tests more like the team's. A reworked agent seems to triage build failures faster. Seems is the problem. Someone tried it on the change in front of them, liked what came back, and the new version went into use. Evaluation on a Czar Forge is a harness on the appliance that scores models, prompts, and agents against the team's own test sets and its own resolved work, so that the version in use is the one that measured better rather than the one that felt better.

What is evaluated

Three kinds of thing are scored on the Forge, and the same harness handles all of them. The first is models: the open-weight base models the team serves and the adapters Adaptation has tuned on the team's own code and data. The second is prompts: the instructions and context that shape what an agent does with a model, which change far more often than the model and with the least ceremony. The third is agents themselves, as they are authored in Agency Studio: the whole configuration of model, prompt, tools, bounds, and the repositories or systems the agent is permitted to read.

Treating an agent as a thing to evaluate matters, because a team rarely changes one part at a time. A new adapter arrives alongside a revised prompt and a wider set of permitted repositories, and the question the team cares about is whether the agent, as it will actually run, does the job better than the one it replaces.

Test sets from the team's own resolved work

The best test set a team has is its own history. Every merged change carries a review that accepted it and often an earlier version that was sent back. Every resolved incident has a cause the team eventually found. Every closed ticket has a disposition someone decided on. Every question an engineer asked the repository index has an answer that was, or was not, in the passage it cited. The team already holds the answer key; it has simply never been assembled into one.

Evaluation assembles it on the appliance. A platform team can build a set of past build failures with their known causes and ask whether a triage agent finds them. A software team can hold back a set of resolved changes and ask whether a coding agent's proposal matches what the team accepted, and whether the tests it wrote pass against the real suite. A machine-learning team can keep a set of questions with known answers and ask whether a tuned model answers them without losing what the base model knew. The sets are the team's own, stay on the appliance beside the code they came from, and grow as the team resolves more work.

The harness on the appliance

The harness runs on the Forge, inside the perimeter, using Inference and Retrieval as the agents themselves do, so a candidate is scored under the conditions it would run in. Candidates are compared on the same set, side by side: the new adapter against the base model and the adapter it would replace, the revised prompt against the current one, the reworked agent against the version in use. The scores are written to the record with the set, the candidates, and the run, so the same comparison can be made again and give the same answer.

What the harness measures is what the team cares about rather than a general benchmark. Did the proposal match what the team accepted? Did the tests pass? Did the answer cite the passage it should have? Did the triage name the cause the team found? A team can add measures of its own and keep a standing set that every candidate is scored against first. Evaluation can also re-score a promoted candidate on that standing set on a cadence the team chooses, so a model whose answers have begun to drift as the codebase moves is noticed from the record rather than from a complaint.

What Watchman verifies

Evaluation says what ought to be running. Watchman, the Substrate service that verifies the appliance, confirms that it is. The model, the adapter, the prompt version, and the agent version serving the team at any moment are the ones a named engineer promoted, and Watchman records that they are. If something running differs from what the record says was promoted, the team sees it from the record. One service establishes that a candidate deserved promotion; the other establishes that the promoted candidate is the one at work.

Promotion is a decision a person makes

The harness reports; it does not decide. Agents on a Czar Forge prepare and propose; a named engineer merges, promotes, or releases. Nothing reaches production, a customer, or the outside without that person, and every step is on the record the team holds. For a candidate that has been evaluated, a named engineer reads the scores, promotes the candidate or sends it back, and that decision is written beside the scores in Agency Studio. A run can be replayed later to show what was compared and what the person saw when they decided.

For a very large evaluation sweep, the Compute network is being built so that a team could send the sweep out as a sealed overnight job and receive the scores as a signed bundle, read and acted on from the Forge. The routine evaluation that precedes every promotion stays on the appliance.

A team that evaluates before it promotes stops arguing about whether the new version is better and starts reading the record that says whether it is. On a Czar Forge the record is the team's, the test sets are its own history, and the decision belongs to a named engineer who can show how it was made.