Inference pool
An inference pool is shared compute inside a facility that runs the perception, decision, and language models for every robot in the building, over the building's own network.
· 2 min read
IN SHORT
An inference pool is shared compute inside a facility that runs the perception, decision, and language models for every robot in the building over the building's own network. It exists because those models are too large to carry on a robot and too urgent to send to a distant cloud, so the pool sits in the rack where the robots already are.
An inference pool is shared compute installed inside a facility that runs the perception, decision, and language models for every robot in the building, over the building's own network. It replaces two poorer options: carrying every model on the robot, or sending every question to a distant cloud.
Why it matters
A robot has limited room for compute. The models that read a cluttered scene, decide the next move, or follow a spoken instruction are too large to carry and too urgent to send far away. Left to its own hardware, each robot either does less or waits longer.
A pool changes the arithmetic for the whole floor. Robots keep a small on-board model for reflexes and lean on the pool for the heavy work. When the pool is in the building, the round trip is short, and the camera feeds and task records never leave the operator's network.
It also matters for mixed fleets. A warehouse that runs mobile robots from one vendor, pick arms from another, and humanoids from a third does not want three separate stacks. One pool serves all of them.
How it works at Element 31
Element 31 builds the inference pool as an appliance called Czar, a sealed node that goes into the building's own rack. One Czar is a pool for a small floor. Several Czars joined together form a Czar Grid, one pool for a whole facility.
Substrate is the serving system on the pool. It schedules inference across the nodes, holds each robot to its latency budget, keeps memory and retrieval over the site's documents, and adapts models on the site's own captures without anything leaving the building.
Safety never moves into the pool. Stops, speed limits, reflexes, and motion limits stay on the robot as its vendor shipped them. If the pool cannot answer in time, the robot falls back to its on-board model or stops. A pool that fails leaves a robot that stops, not one that guesses.
A site chooses its posture. The pool can run air-gapped, or connected only to the systems the site already secures, such as the robot vendors' fleet software and the warehouse system. More robots on the pool do not change the subscription.
Related terms
Questions this raises
- Why not run every model on the robot itself?
- The perception, decision, and language models that make a robot useful on a varied floor are too large for the compute a robot can carry. A pool in the building holds those models once and serves every robot, while each robot keeps a smaller on-board model for reflexes.
- Why not use a cloud service instead of a pool in the building?
- A distant cloud adds a round trip that a robot in motion cannot wait for, and it means camera feeds and task records leave the building. A pool inside the facility answers over the local network and keeps that data on the operator's hardware.
- What happens to a robot when the pool is unavailable?
- The robot falls back to its on-board model, and if that model cannot proceed, the robot stops. Stops, speed limits, and motion limits are on the robot, not in the pool, so a pool failure never produces a guess.
KEEP READING