Monitoring a pool that does not phone home

Most infrastructure is monitored by letting it talk to somewhere else. A Czar Grid does not do that. Camera feeds, captures, task records, model weights, and the record of every decision stay on the operator's hardware inside the operator's network. There is no telemetry stream to Element 31 and no vendor dashboard elsewhere. So the practical question for an operations team is how to watch something that will not report outward, and the answer is that the pool keeps an unusually complete record and the site reads it.

The record is the monitoring surface

Substrate writes down what the pool does. For each request it records which robot asked, what kind of answer it needed, which model on the pool served it, and whether the answer arrived inside that robot's latency budget. Agency Studio records every task run with its version, its approver, and what happened. Watchman records whether each node's running image matches what it attested.

This record is not a log in the sense of scrolling text nobody reads. It is the site's operating history, structured so that the automation team can ask ordinary questions of it: what did this fleet experience during the second shift, which tasks ran against which perception model, did any node fail integrity. The site owns it outright and reads it with its own people.

Budgets, misses, and the fallback pattern

Every robot on the pool carries a latency budget, the time inside which an answer is useful to that machine. A mobile robot crossing an aisle has a tighter budget than an arm waiting at a bench. Substrate serves each request against its budget, and the record shows whether the budget was met.

A missed budget is not a safety event. The robot falls back to its on-board model and does what its vendor built it to do. But a missed budget is information. It is the earliest and most honest signal the site has about the pool, because it is expressed in the terms that matter to the floor rather than in internal counters. A team that watches budgets met and missed, fleet by fleet and hour by hour, understands the health of its pool without needing anyone to translate.

Fallbacks are the operational heartbeat. The useful way to read them is by shape rather than by total. Fallbacks that cluster around one fleet point to that fleet's network path or configuration. Fallbacks that cluster at one time of day point to load, and the conversation goes to the integrator about what else the floor is asking of the pool then. Fallbacks that appear only on one node point to that node, and the integrity record for it should be read next.

A sudden absence of fallbacks is also worth noticing, especially after a release change. It may be good news. It may also mean a fleet stopped asking the pool at all, which the record will show as a drop in requests rather than a rise in misses. Reading both together is the habit that distinguishes an experienced operator from a new one.

Integrity, per node

Watchman verifies that each node is running the sealed, signed image it attested. The result is binary and it is in the record. Operations does not need to know how verification works; it needs to confirm each node reads clean, and to know that a node which cannot verify does not serve.

This check matters most after any change: an update, a rollback, a node replacement. Reading the integrity result immediately after and again the following morning is a small habit that closes most of the questions an auditor would later raise.

Feeding the site's own tools

Sites that already run monitoring for the floor often want the pool's record in the same place. On a connected site that is reasonable. The pool reaches only routes the site declared through Local Bridge, and a route from the record to the site's own monitoring system, inside the building, is the kind of declared connection the posture allows. It is still a decision, written down and approved, not a default.

On an air-gapped site the record is read where it lives, at the pool's console inside the isolated segment. Operators on those sites build the reading into the shift routine rather than into an alerting system, and many find the discipline useful rather than limiting. Day two describes that routine.

What not to expect

The pool will not phone home, and Element 31 is not watching it. There is no quiet connection for support purposes. When the site wants help, a session is customer-initiated through the portal, and on sovereign sites it happens only inside an approved uplink window with a hard expiry. Monitoring a pool that does not phone home therefore means exactly what it says: the site watches, from inside, with a record built to be watched. When something in that record needs a closer look, reviewing an incident from a local record is the next step.