Vision-language-action models on a facility pool

The most useful models in robotics right now are the ones that take an image, an instruction, and a sense of the task, and produce a proposal for what the robot should do next. They are commonly called vision-language-action models. They are also, almost without exception, too large to run on the robot that needs them.

That leaves the usual choice: shrink the model until it fits the robot and accept that it understands less, or send the robot's camera feed to a vendor's cloud and accept the round trip and the leak. A facility inference pool is the third place. It is in the building, close enough to answer within the robot's budget, and on hardware the operator owns.

What the model does and does not do

A vision-language-action model is a decision model. Given what the robot sees and what it has been asked, it proposes the next step: reach here, grasp this, move toward that shelf, hand this to the person. Its value is in understanding the scene and the instruction together, which is where a small on-board model falls short.

What it does not do, on a floor with a Czar Grid, is move the robot. The proposal goes back to the robot over the facility's network, and the robot's own controller carries it out, or declines to, within the stops, speed limits, and motion limits the vendor shipped. The model's output is a suggestion with a deadline. The controller holds the body.

This split is not a compromise forced by the architecture. It is the reason the architecture is safe to use. A large model that could directly command motion would be a large model inside the safety case, and no vendor would ship that. A large model that proposes, with the robot's own envelope between the proposal and the motor, keeps the safety case where the vendor put it.

Where it runs

A Czar is a sealed node in the rack. Several joined are a Grid, one pool for the whole facility. Substrate is the serving system on that pool: it runs inference across the nodes, holds a latency budget for each robot, and keeps the memory and retrieval the model may need to understand an instruction that refers to the site's own documents, such as a packing standard or a work instruction.

Every robot on the floor can ask the same model. A humanoid carrying a tote, a mobile manipulator at a workstation, and a pick arm at a cell can all be served from the same pool over the same network, each within its own budget. When the pool cannot answer a robot in time, that robot falls back to its on-board model or stops. It does not act on a proposal that has not arrived.

Instructions, in the site's own words

Part of what makes these models useful is that the instruction can be ordinary language. A supervisor can say what needs doing in the words the floor already uses. The model has to understand those words, and the words on a real floor are local: the names of zones, the nicknames for item types, the shorthand on a pick list.

Substrate's retrieval over the site's own documents gives the model that vocabulary from inside the building. Adaptation on the site's own captures gives it the floor's own objects and lighting. Both run on the pool, with nothing leaving. A model adapted to this floor understands this floor's instructions better than a general model ever will, and the weights that encode that understanding are the operator's.

The feed and the record

A vision-language-action model consumes the robot's camera feed and the words people say to it. Those are among the most sensitive things a facility produces. On a facility pool they go to the rack and no further. Camera frames, instructions, captures, task records, and adapted weights stay on the operator's hardware inside the operator's network, whether the rack is air-gapped or connected only to systems the site already secures.

Every proposal the model makes is also on the record, alongside what the robot did with it. When a task goes wrong, the site can read what the model saw, what it proposed, and whether the robot acted or fell back. That is a record an auditor can read, kept on site.

For the task layer that bounds what these models can be asked to do, see Agency Studio on the technology page, and for the safety reasoning in more depth, read why safety stays on the robot.