Heavy Metal on the Edge: Training on H100s Inside a Sealed Appliance

Training and fine-tuning workloads have a different relationship with hardware than inference does. An inference endpoint can be modest and still be useful; plenty of production copilots run comfortably on a handful of accelerators. A serious fine-tuning or continued-pretraining run on a customer's own sensitive corpus wants datacenter-class silicon: high memory bandwidth, fast interconnect between GPUs, and enough VRAM per card that a model doesn't have to be sharded across so many devices that communication overhead eats the gain. That silicon was designed to live in a hyperscale data center, with its own power plant, chilled water loop, and a facilities team whose entire job is keeping thousands of racks within their thermal envelope. Fitting a meaningful slice of that capability into a sealed enclosure that ships to a customer's own floor is not a rebadging exercise. It's a genuine mechanical and electrical engineering problem, and it's worth being specific about what that problem actually is.

The constraint that shapes everything else: heat has to leave the box

Training-class GPUs are dense heat sources by design. The same packaging that maximizes compute-per-rack-unit in a hyperscale hall concentrates thermal load into a small volume. In a hyperscale data center, that heat is somebody else's problem in a useful way: it leaves the chip, leaves the rack, and disappears into a facility built from the ground up around cooling at scale. A sealed appliance doesn't get to make that assumption. It has to reject the heat those same accelerators generate using thermal design built into the chassis itself, sized for the room it's actually going to sit in, which is far more likely to be a secure facility with conventional building HVAC than a purpose-built data hall.

That single constraint cascades into most of the other design decisions. Airflow path, fan placement, card spacing, and enclosure volume all get decided by the answer to "where does the heat go and how fast," not by how tightly the silicon could theoretically be packed. A design that chased maximum device density without respecting that constraint would either throttle under sustained training load (quietly turning a fine-tuning run that should take days into one that takes weeks) or fail outright. Getting this right generally means being honest about the density a sealed, facility-agnostic enclosure can sustain, rather than promising hyperscale-rack density in a box that has to breathe through a normal server room.

Power delivery is a design decision, not a footnote

The same silicon that concentrates heat also concentrates electrical draw. Training-class GPU nodes pull substantially more power than the inference-oriented hardware most sealed appliances have historically shipped with, and that has consequences upstream of the chassis: power supply sizing, the electrical circuit the customer's facility needs to provide, and how gracefully the system behaves under a brownout or a partial power event. A training appliance that assumes the customer's site has the same electrical headroom as a data hall is making an assumption that often doesn't hold in a government facility, a defense contractor's secure room, or a regulated enterprise's existing server closet. Part of the engineering work is making the power envelope legible up front, so a customer's facilities team can plan for it before the box arrives, not discover it during install.

Interconnect, sealed

Multi-GPU training lives or dies on the bandwidth between accelerators, not just the bandwidth of any single chip. That's true in a hyperscale cluster and it's equally true inside a sealed enclosure, arguably more constraining, because a sealed box has a fixed, finite number of accelerators and no ability to reach across a rack fabric to borrow more when a job would benefit from it. The interconnect topology inside the chassis has to be engineered so that the GPUs actually present as a coherent training cluster rather than a set of independently fast devices bottlenecked by whatever link happens to sit between them. This is invisible from the outside — nobody evaluating an appliance from a spec sheet sees the interconnect fabric — but it's the difference between a training job that scales close to linearly across the available GPUs and one that plateaus well short of it.

What sealing costs, and what it buys

None of this is free. A sealed, air-gapped enclosure gives up some of the tricks a hyperscale facility uses to push density and efficiency: free cooling tied to a specific climate, facility-scale liquid cooling loops, power infrastructure amortized across an entire campus. An honest comparison acknowledges that a training appliance sized to fit inside a secure room, running on standard facility power and cooling, is not going to match a purpose-built training cluster on raw density or on efficiency per watt at the facility level. That gap is real, and it is the price of the boundary.

What it buys in return is the reason the whole category exists: the training data, the intermediate checkpoints, and the resulting model weights never leave a controlled perimeter. For a workload built on classified, export-controlled, or otherwise regulated data, that property isn't a nice-to-have layered on top of a slower box. It's the precondition for the workload being permissible to run at all. The relevant comparison for that customer was never "sealed appliance versus a hyperscale cluster with unlimited budget." It's "sealed appliance versus training that cannot happen on this data in this environment by any other means," because the alternative was never actually available.

Where this sits inside Czar

This is the engineering brief Czar's enclosure design has to satisfy: enough training-class compute, correctly cooled, correctly powered, and correctly interconnected, to run real fine-tuning and R&D workloads against sensitive data, sized to the workload a customer actually describes during scoping rather than to a generic maximum. The Substrate platform sits above that hardware layer handling the data governance, retrieval, and integrity concerns that make a training run auditable inside a sealed boundary, but none of that software matters if the underlying enclosure can't sustain the thermal and electrical load of the silicon doing the actual training. Heavy metal at the edge is a hardware problem before it's a software one. Treating it as anything less is how sealed appliances earn a reputation for underperforming their spec sheets under sustained load.