Rolling back a release on the pool

Sometimes a release does not suit a floor. A task that used to run cleanly now falls back more often; a perception model behaves differently on the site's own lighting; the automation lead simply wants more time before committing. None of these are emergencies. All of them are reasons a site might want yesterday's release back, and a Czar Grid is built so that going back is the same controlled act as going forward.

This article covers the operational side: who decides, what the nodes do, and what the site has in hand afterwards.

Why rollback is possible at all

A Czar node runs a sealed, signed, read-only image. It does not accumulate drift between releases, because nothing on the image changes while it runs. The previous release is therefore not a half-remembered state; it is a specific signed bundle the site already received, verified, and applied once before.

That is what makes rollback honest. The site is not asking the appliance to undo a series of changes and hope the result matches. It is asking the node to load a known image whose signature it can check, which is exactly what the node did on the way up. If the site kept the previous bundle on controlled media, as the air-gapped update procedure recommends, nothing needs to be fetched.

Deciding to roll back

The decision belongs to the site, and on a well-run floor it goes through the same people who approve task versions in Agency Studio: the automation lead, the safety officer, and, where one is present, the integrator. The reason is written down. The record should later show not only that the Grid changed images twice but why the second change happened.

It helps to separate two kinds of reason. The first is behavior: a fleet is falling back more, a task's bounds no longer feel right against the new perception output, a language model's answers to the floor's own documents have changed tone. The second is caution: the site wants more observation time before standing on the new release. Both are legitimate. The record reads the same either way, and neither requires permission from anyone outside the building.

The procedure, node by node

Rollback is an update with an older bundle. The site takes one node out of service, presents the previous signed bundle, and the node verifies the signature. If it verifies, the node applies the image. It then attests what it is running and only after that resumes serving robots and accepting tasks. The site moves to the next node.

While each node is out, the rest of the Grid keeps serving. Robots whose questions would have gone to that node are served by the others or fall back to their on-board models within their own budgets. Safety was never on the pool: stops, speed limits, and motion limits stay on the robot as its vendor delivered them, through the update, through the rollback, and through every minute in between.

On a connected site the bundle may already be present from the earlier download, retrieved by the customer through the portal on a machine that is not the appliance. On an air-gapped site it is carried in on media exactly as before. In neither case does the appliance reach out to get it.

What survives a rollback

New operators sometimes worry that returning to an older image discards what the site has built. It does not. The image is the software. The site's accumulated material, meaning adapted model weights trained on its own captures, the task library with its versions and approvals, and the record of every run and every decision, is the operator's data on the operator's hardware. It stays where it is.

What a site should check after rolling back is whether any task version was approved against the newer release specifically. If a version was tuned to behavior that the older image does not produce, the site may want to return to the earlier task version as well, and that too is a recorded approval. Agency Studio keeps versions for exactly this reason.

Reading the record afterwards

When the Grid is back on one consistent release, the record for each node shows three attestations in sequence: the original image, the newer one, and the return. Beside them sit the approvals that authorized each step and the operator's note explaining why. An auditor, an insurer, or the site's own management can read that sequence and understand what happened without anyone narrating it.

That is the operational value of treating rollback as a first-class procedure rather than an emergency. The site keeps its options open, the floor keeps running, and the story of what the pool was running on any given day is in the record rather than in anyone's memory. The same discipline applies when hardware rather than software changes, which is covered in replacing a node in a Grid.