An inference pool versus a GPU on every robot
Putting a large accelerator on every robot pays for peak capacity that sits idle most of the shift. A shared pool in the rack sizes compute to the floor, not to the machine.
· 4 min read
IN SHORT
A GPU on every robot buys peak capacity for each machine and leaves most of it idle for most of the shift, while adding mass, heat, and battery draw the robot must carry. A shared inference pool in the facility rack sizes compute to the floor's combined demand, serves every vendor's robots, and is upgraded in one place.
There are two places to put the compute a robot fleet needs beyond its reflexes: on each robot, or in a rack the robots share. The first is the default because it is what robot vendors ship. The second is what a facility ends up wanting once the fleet is large enough, or mixed enough, to notice the cost of the first.
What a per-robot accelerator actually buys
An accelerator on a robot is sized for that robot's hardest moment: the unfamiliar object at the edge of the camera, the aisle with two people and a forklift in it, the label it has never seen. Those moments are real, and the model that handles them is large. But they are rare. For most of a shift the robot is driving a known route or repeating a known pick, and the large model does nothing.
Multiply that by the fleet. Every machine carries peak capacity, and at any instant only a few of them are using it. The facility has paid for the sum of the peaks and is using something close to the average. The gap between those two is the idle compute a per-robot design cannot avoid.
What the robot pays to carry it
Compute on a robot is not free even when idle. It has mass, which the drive system must move and the battery must pay for. It produces heat, which the enclosure must shed. It draws power whether or not it is answering a question, which shortens the shift between charges. On a humanoid, where every gram of mass and every minute of battery is contested, the cost is sharper still.
Compute in a rack carries none of these penalties on the machine. The rack has mains power, has cooling, and does not have to climb stairs. The robot carries only what it needs to be safe and to move, and reaches the rack for the rest.
Sizing to the floor instead of the machine
A pool is sized to the floor's combined demand rather than to the sum of every robot's worst case. Because different robots hit their hard moments at different times, the pool's peak is far below the sum of the individual peaks. The capacity that would have idled on each machine is instead shared, and most of it is doing work most of the time.
At Element 31 the pool is Czar, a sealed node in the building's own rack. One Czar serves a small floor. Several Czars joined into a Czar Grid serve a whole facility as one pool. Adding robots to a floor that already has a pool does not mean buying compute for each one; it means the pool is shared a little more widely. The commercial model follows the same shape: one subscription for the configured Grid, and more robots on the pool do not move it.
Upgrading in one place
Models get larger and better, and a model that fits an accelerator today will not fit the one that replaces it. On a per-robot design, a better model means a hardware change across the fleet, scheduled around the vendor's roadmap and the robot's warranty. On a pool, a better model means an update in the rack, delivered as a signed bundle, with the fleet untouched.
The same holds for adding a kind of model the fleet did not have. A floor that wants its robots to read work orders written for people adds a language model to the pool; it does not retrofit every robot.
What this does not change
The robot still carries its own model for what it does every second and for everything its vendor built in: stops, speed limits, reflexes, motion limits. The pool is not in the path of those. When the pool cannot answer inside the robot's latency budget, the robot uses the model it carries and acts conservatively.
So the comparison is not between a robot with compute and a robot without. It is between a fleet that carries every large model on every machine and a fleet that shares the large models from the rack. See Why robots outgrow their on-board compute for how fleets arrive at the second design.
Questions this raises
- Does a shared pool mean the robot has no compute of its own?
- No. Every robot keeps its on-board computer and its own model for reflexes, stops, and motion limits. The pool serves the larger perception, decision, and language models the robot would otherwise have to do without.
- What happens if the pool is busy when a robot asks?
- Each robot has a latency budget, and the pool serves within it or not at all. If the answer would arrive too late, the robot falls back to its on-board model and acts conservatively, which on a vendor-built robot means slowing or stopping rather than guessing.
- Can a pool serve robots that already carry a large accelerator?
- Yes. A robot with substantial on-board compute still benefits from models larger than it can hold, from retrieval over the site's documents, and from a shared record. The pool adds to what the robot carries; it does not require the robot to carry less.
KEEP READING