Over-the-Air Updates Without the Air: Secure Sneakernet Patching Protocols

Every cloud service in the world takes over-the-air updates for granted. A vendor pushes a build, a fleet of machines pulls it down, a rollout dashboard shows the percentage climbing toward one hundred. None of that mechanism survives contact with a device that has no WAN-facing network hardware at all — not disabled, not firewalled, absent. That absence is the entire point of a sealed appliance, and it is also the reason patching has to be solved as a distinct engineering problem rather than inherited from the rest of the industry's tooling.

"Sneakernet" sounds like a joke about carrying a floppy disk across a room, and the term's informality tends to make people underestimate what a serious version of it requires. Moving a patch by physical media is the easy part — any USB drive can do that. The hard part is making sure the thing that arrives on the drive is exactly what the vendor built, that it hasn't been substituted or altered in transit, that it can't be replayed to roll a device backward into a known-vulnerable state, and that every step of that chain leaves a record a reviewer can check later. Physical delivery removes a network attack surface. It does not remove the need for an update protocol. It just relocates where that protocol has to enforce its guarantees.

What the network normally does that now has to happen another way

An internet-connected update pipeline typically leans on the network itself for several things at once: authenticating the source, protecting the transport, and providing a live channel back to the vendor for revocation if a build turns out to be bad after release. Take the network away and each of those has to be re-derived from something that doesn't depend on connectivity.

Authentication has to move from "this came over a TLS session with the vendor's server" to "this artifact is signed by a key the device already trusts, and the signature verifies against content the device hashes itself." Transport protection, which normally guards against interception and tampering in flight, has to be replaced by the fact that tampering with a signed artifact breaks its signature regardless of how it physically traveled. And the live revocation channel — the ability to tell a fleet "stop, that build has a problem" the moment something goes wrong — has no equivalent at all unless the appliance's own protocol builds one in deliberately, because there is no persistent connection to push a revocation notice down.

That last point is the one people most often miss when they first think through sneakernet patching. Signing is intuitive. An explicit answer to "how does a device that's never spoken to the internet find out a patch it already has staged should not be installed" is not.

The chain a patch has to survive

Think of the path a patch takes as a sequence of handoffs, each of which is a place where the guarantee can be either preserved or quietly broken.

Build and signing at the vendor. The artifact is produced in a controlled build environment and signed with a key held in hardware the vendor controls, ideally offline or in a dedicated signing appliance rather than a general-purpose build server. Everything downstream depends on this step being trustworthy, since no amount of protocol rigor later in the chain can compensate for a compromised signing key at the source.

Media preparation and its own integrity layer. The signed artifact goes onto physical media — a write-once optical disc or a controlled USB device — ideally itself accompanied by a manifest and hash that can be checked independently of the update payload's own signature, so a corrupted or tampered disc is detectable before the device even attempts to parse the patch inside it.

Transit and custody. This is the classic sneakernet weak point in the popular imagination — a courier, a mail service, a hand-carried case — and it deserves real controls: tamper-evident packaging, chain-of-custody logging, and ideally a media type that makes silent, undetectable alteration difficult. But this layer is not the last line of defense. If the cryptographic verification at the device is doing its job, a compromised courier can delay or deny a patch, but shouldn't be able to substitute a malicious one that verifies successfully. Custody controls exist to catch tampering early and to support the audit trail, not to be the only thing standing between a bad actor and a successful compromise.

Verification at the appliance. This is where the protocol earns its keep. The device checks the artifact's signature against a trust root that shipped with the hardware or was provisioned through an earlier, already-verified update — never against a key introduced by the update media itself, which would let a forged patch vouch for its own legitimacy. It checks the artifact's version against an anti-rollback counter stored in a way that survives reinstallation, so an old, previously-superseded build with a known issue cannot be reintroduced by presenting outdated but validly-signed media. And it checks freshness in whatever form the protocol defines, because a validly signed patch from eighteen months ago is not automatically a patch that should install today.

Local attestation of the result. Once installed, the device records what it installed, when, from what artifact hash, and under whose physical custody the media arrived — building the same kind of evidentiary trail an air-gap disconnection claim needs, applied here to the update history instead of the network posture.

A break at any single link undermines the whole chain, which is why "we sign our updates" is a necessary but incomplete answer. Signing without anti-rollback protection stops forged patches but not replayed old ones. Anti-rollback without a hardware-rooted trust anchor stops replay but not a forged trust root introduced at provisioning time. The chain is only as strong as its least-considered link, and sneakernet protocols get evaluated on exactly that basis.

Anti-rollback deserves its own attention

Rollback protection is worth dwelling on specifically, because it's the control most often bolted on as an afterthought and the one that matters most for a device that may go months between contact with anyone who can independently confirm its patch state.

A connected fleet can lean on a vendor's live inventory of what's current to make a stale build obvious. A sealed appliance in the field has no such reference except what it carries with it and what's stored in hardware that survives a reinstall attempt. If a monotonic counter is only stored in the same writable partition the update itself replaces, an attacker with physical access and a copy of an old, validly-signed patch can potentially reset the device's notion of "current" along with the software. Anti-rollback protection that actually holds has to live somewhere the update process can't touch — dedicated write-once or hardware-enforced counter storage — and the update protocol has to check it before installation, not just record the new value after.

This is also the layer that has to account for legitimate exceptions without weakening the general rule. There are real scenarios — a regression discovered after release, a deliberate downgrade during incident response — where installing an older build is the correct operational choice. A well-designed protocol handles that as an explicit, separately-authorized action requiring its own signed instruction, not as a gap in the rollback check that any old media happens to satisfy.

Why this is a protocol problem, not a courier problem

It's tempting to treat sneakernet patching as mostly a logistics and personnel-vetting exercise — trusted couriers, sealed bags, signature-on-delivery — and there's a real place for that discipline. But logistics controls answer "did this package travel safely." A mature patching architecture doesn't want the answer to "is this software safe to install" to depend on that first question at all. The cryptographic and anti-rollback layers exist precisely so that a lapse in physical custody is a containable incident — caught, logged, investigated — rather than an automatic path to a compromised appliance. Good logistics reduce how often the verification layer gets tested. They are not a substitute for it.

Where this sits across the lineup

Forge and Czar deployments both depend on this working reliably, for related but slightly different reasons. A Forge appliance sitting inside a defense contractor's development environment needs security patches on a cadence that keeps pace with disclosed vulnerabilities, without ever opening a path back to the internet to get them — which makes the update protocol, not the network, the thing standing between the device and known exploits. A Czar sandbox running long-lived training and fine-tuning workloads needs the same discipline applied to a broader surface, since the software stack supporting sensitive R&D work is typically larger and more frequently updated than a narrowly scoped coding assistant.

Chassis builds inherit the same Substrate-level update protocol, but the operational question shifts: a bespoke, per-deal appliance shipped under an ISV's own brand still needs a patching path back to that ISV, and the chain of custody has to be documented in a form that satisfies whatever regulatory framework governs that specific buyer, since a courier and signing process built for one deployment's compliance regime doesn't automatically transfer to another.

The underlying principle is the same across all three: removing the network doesn't remove the need for updates, and it doesn't lower the bar for what a legitimate update has to prove about itself before a device installs it. If anything the bar is higher, because there's no live channel to catch a mistake after the fact. The physical medium is the easy substitution. The cryptographic and anti-rollback discipline that has to travel with it is the actual engineering problem, and it's the part that determines whether "no network" stays a security property instead of becoming an update-integrity liability.