Replacing a node in a Grid

A Czar Grid is several Czars joined into one pool. That structure is what makes replacing a node an ordinary maintenance act rather than an outage. One node comes out, the rest keep serving, a sealed replacement arrives with its history already written down, and it proves what it is before the Grid lets it in. This article walks through that exchange from the operator's side.

Taking the node out

The decision to remove a node usually comes from the record. Watchman reports a node that cannot verify its image, or the fallback pattern isolates to one node, or the site simply chooses to rotate hardware on its own schedule. Whatever the reason, the automation team takes the node out of service deliberately and notes why.

From the moment it leaves the pool, the remaining nodes carry the floor. Substrate continues serving each robot against its latency budget across the nodes that remain. Where the smaller pool cannot answer in time, robots fall back to their on-board models and behave as their vendors built them to. Safety was never on the pool. The record shows exactly which fleets, if any, are falling back more while the Grid is short a node, and the site uses that to decide how urgently it wants the replacement racked.

What goes out the door

Before the old node leaves the building, the site decides what happens to anything it holds. The site's material, which includes captures, task records, adapted model weights, and the record of decisions, is the operator's and remains on the operator's hardware. Handling the departing node's storage is a policy matter the site settles according to its own rules, and the engineers involved follow that policy rather than impose one.

Sites with a sovereign posture tend to have strict answers here, and the procedure accommodates them. The point is that the decision is the site's, made in advance, and written down beside the removal in the record.

What arrives

The replacement is a sealed Czar. The enclosure is sealed and the software image is sealed, signed, and read-only. Its provenance, meaning the record of its integration, its sealing, and its software load, exists before it reaches the dock. When the safety officer or a procurement lead asks where this particular node came from and what was done to it, the answer is a record, not a reassurance from whoever delivered it.

Nobody on site opens it or loads software onto it. The site's work is physical: into the rack, onto the power and the network drop the old node used.

Attestation before joining

The new node does not simply appear in the pool. Before it joins the Grid or accepts a task, it attests what it is running. The Grid checks that attestation against the signed image it expects, and the result is recorded. Only then does the node begin serving robots.

This is the same gate every node passed on installation day, and it is applied to a replacement without exception. If the attestation does not match, the node does not join and the site has a question to answer before anything else happens. In the ordinary case it matches, the record gains one more attestation entry, and the pool is back to full strength.

Confirming the Grid is whole

The automation team reads the record after the replacement has served through at least one shift. Fallbacks that rose while the Grid was short should return to their earlier pattern. Budgets that were missed should be met again. Watchman's integrity result for the new node should read clean, and reading it again the next morning is a cheap habit.

If the Grid was mid-way through an update when the old node came out, the replacement should be brought to the same release as its peers. It arrives on a signed image; if that is not the release the rest of the Grid runs, the site applies the right signed bundle through the usual procedure and the node attests again. Updating an air-gapped Grid describes that procedure for sovereign sites, and it is the same steps on a connected one.

When the exchange is done, the record tells a complete story. One node was taken out, with a reason. The site decided what to do with its storage, and did it. A replacement arrived with provenance already on file. It attested, it joined, it served, and its integrity reads clean. Nobody has to remember any of this, because it is written in the same local record as every task run and every approval.

That is the operational point of provenance and attestation. They are not features the site admires once at installation; they are what makes the Grid changeable over a long service life without the site ever having to take anyone's word for what is in the rack. The same discipline applies when software rather than hardware changes, as rolling back a release on the pool describes.