Patching an Air-Gapped AI Stack Safely

An air-gapped stack that cannot take a patch is not conservative. It is decaying. The first serious CVE in the OS, the container runtime, or a Python wheel will force a choice: open a hole, or go dark. Committees that never funded a patch path always open the hole, then keep the word air-gapped in the brochure.

Patching without a network is a logistics problem before it is a technical one. Someone has to build or obtain a bundle on a connected side, prove what the bundle is, carry it, admit it, apply it, and prove the enclave did not also admit a browser, a second NIC, or a vendor's 'helper' script. Skip any of those steps and you have either a hole or a rumour.

This guide is the process we walk with SIs who have to keep a disconnected agent alive for a year. It assumes you already decided the enclave is real. If the node can still reach a licence server, read that article first — you are not patching an air gap. You are maintaining a VPN.

Two rooms and one bag

Keep a connected build room and a disconnected run room. They may be two VLANs, two cages, or two buildings. They must not share a route. The build room pulls vendor charts, OS errata, CUDA userspace, Python wheels, model files and signature bundles. The run room only ever sees what the ceremony admits.

The bag is whatever media your security policy allows: a dedicated USB that never mounts on a personal laptop, an approved optical disk, or a data-diode drop into a staging vault. The bag has a written identity — serial number, custodian, seal. A personal SanDisk from a backpack is not a bag. It is an incident waiting for a label.

If a stage has no artefact, it did not happen.

Stage	What happens	Evidence you file
Build	Pull, scan, pin versions, sign the bundle	SBOM, hashes, scanner report, vendor advisory
Courier	Sealed media or diode drop, two-person rule	Bag log, seal photo or serial, time
Admit	Verify hashes and signatures on a staging host	Verification printout, who typed
Stage	Install on a non-prod clone or canary	Smoke test, gold-set delta for models
Apply	CAB window, drain, patch, health	Change ticket, before/after versions
Prove	No new routes, no new listening ports	Post-change scan, SIEM heartbeat

What you are actually patching

SIs underestimate the bill of materials. An 'AI box' is an OS, a kernel, NVIDIA userspace if you use those GPUs, a container runtime, a Kubernetes or VM layer, an ingress, a vector store, an object store, a Python environment, a tokenizer, weights, a prompt/tool package, and whatever licence daemon someone hid. A CVE in any of those is your CVE.

• Firmware and BMC — often forgotten, often networked. Disable outbound BMC. Patch it on the same calendar.
• OS and kernel — including the disconnected host that does the admitting. That host is high-value.
• GPU driver / CUDA userspace — pin and test; a 'latest' driver can stall inference.
• Cluster and ingress — Kubernetes, Helm, service mesh sidecars that phone home if you let them.
• Data plane — Postgres, object store, vector engine. These have their own CVEs and their own backups.
• Model artefacts — weights, adapters, tokenizers, safety classifiers. Different gate, same bag.
• Your own glue — the courier scripts. Unreviewed glue is how malware enters a careful room.

Pin, do not float

Floating tags — latest, stable, ubuntu22.04 — are how two rooms drift. Pin digests. The build room records digest in, digest out. The run room refuses anything that is not on the pin list. A vendor who cannot give you a digest is not ready to sell into an enclave.

The emergency path

A monthly window will not save you from a wormable runtime CVE. Write an emergency courier before you need it: who can convene, who can break a seal on a Saturday, how you shrink the bundle to the one package that matters, how you roll back if the patch bricks a driver. Emergency is not 'someone bring a 4G dongle'.

Practice it once on a dummy CVE. Time it. If the dummy takes four days because the signing officer is on tour, you do not have an emergency path. You have a hope.

Model updates are patches

Treat a new weight file like a new binary. It can change answers, leak more context, or ignore a refusal you relied on. The ceremony is the same bag. The gate is different: run the gold set, re-run prompt-injection cases, compare citation behaviour, and keep the previous hash mountable. 'The vendor said it is better' is not a gate.

Tokenizer and prompt-package updates deserve the same suspicion. A one-line change in a tool description can send a new field to a new API. Read the diff. If the vendor ships opaque tarballs with no diff, make a readable manifest a condition of the contract.

Objections you will hear — and what to do with them

Scanning on the connected side is enough.

It is necessary and not sufficient. The bag can be swapped. The admit host can be dirty. Verify on the high side. If you only trust the build room, an attacker only needs the build room.

We will patch when we reconnect for DR tests.

Then your patch latency is your DR calendar. Worms do not wait for the DR week. Build the bag or accept a written risk that a known CVE will sit for months.

Immutable appliances cannot be patched by the department.

Then the vendor must run this ceremony and you must watch it. Immutability is not a reason to skip hashes. It is a reason to demand a signed whole-image swap with rollback.

A six-week patch-path playbook

1. Week 1: inventory every component that can have a CVE, including BMC and licence files.
2. Week 2: stand up the build room and the admit host. No shared route. Write the bag SOP.
3. Week 3: pin current digests. Export a last-known-good bundle and store it as if it were a backup.
4. Week 4: run a dummy import on staging. Time it. Fix the missing signature or the missing officer.
5. Week 5: tabletop the emergency CVE. Name Saturday owners.
6. Week 6: put the calendar and the emergency path into CAB. Refuse go-live on personal data until both exist.

How this shows up in the file

Attach the BoM, the bag SOP, the last successful dummy import, and the emergency names. A later auditor will ask how a CVE was handled. 'We are air-gapped so we did not patch' is not an answer they will like, and 'we opened the firewall' is worse.

What belongs in the bundle — and what must not

A good bundle is boring: pinned images, a checksum manifest, a scanner report, a CVE note, a rollback pointer, and a signature. A bad bundle is a zip the vendor named latest.zip plus a README that says run install.sh. install.sh is where outbound curls hide. Read the script in the build room. If it contains a URL, it does not enter the bag until that URL is gone.

Keep weights out of OS bundles when you can. Mixing a 20-plus-gigabyte model file with a 40-megabyte nginx patch is how people skip hash checks 'because the copy takes too long'. Two bags, two manifests, one ceremony style.

1. Render the full image and package list before you copy.
2. Strip installer scripts of any network call. If you cannot, reject the installer.
3. Store last-known-good next to the new bundle until the soak ends.
4. Record who broke the seal and who typed the verify command.

After a successful production apply, do not delete the old bundle for at least one cycle. Rollback is a bag in the other direction, not a prayer to the vendor.

This article is a field guide, not legal, procurement, electrical or engineering advice. Confirm numbers, duties and designs against the current Gazette, CERT-In directions, your SDC / NIC / campus standards, a site survey and your counsel before you file them.