Air-Gapped Agent Debugging: A Field Guide

Connected teams debug by exporting. They ship the trace to a SaaS, they paste the prompt into another model, they open a support tunnel. Air-gapped teams who copy those habits will either stall or breach. The craft is older than LLMs: reproduce locally, reduce the case, record what you did, and change one thing.

This field guide is for the SI on a Saturday in an SDC cage, and for the campus engineer whose admissions agent started citing last year. It assumes you already have local logs. If you do not, you do not have a debug problem. You have an observability procurement problem.

CERT-In still wants logs you can produce. Debugging is one reason those logs exist. They exist for you, not for a vendor's portal.

The minimum debug pack

Signal	Why you need it	Common absence
Request id	Join across gateway, retrieve, generate	Three uncorrelated logs
Officer identity / role	ACL bugs look like model bugs	A shared service account
Model + prompt-pack hashes	Version drift	latest tag
Retrieval set + scores	Wrong circulars	Only the final prose
Tool names + args + results	Silent tool failure	Swallowed exceptions
Timings	Timeouts, thermal throttle	One wall-clock number
Resource: GPU/CPU/RAM	OOM that looks like 'hallucination'	No node metrics

A Saturday sequence

1. Freeze versions. Do not pull a 'quick patch' until you can reproduce.
2. Find the request id from the officer's screen or the gateway.
3. Pull the joined trace in your SIEM or local viewer. Do not screenshot fifteen panels into WhatsApp.
4. Classify: retrieval miss, ACL miss, generation miss, tool miss, resource miss, or operator miss.
5. Reduce: replay the same request in staging with the same hashes. If you cannot replay, your staging is a lie.
6. If retrieval is wrong, leave the model alone. Fix metadata, filters or the document.
7. If generation is wrong on good chunks, check the prompt pack and the refusal rules before you swap weights.
8. If you need the vendor, grant time-bound, recorded, watched access to the viewer. Do not mail the trace.
9. Write the change as a bag item even if it is 'only a prompt'. Prompt is production.

Classification saves hours

Officers report 'the AI is wrong'. That sentence has five meanings. A withdrawn circular is a records bug. A leak from another folder is an ACL bug. A fluent invention with no citation is a generation-policy bug. A hang at 30 seconds is often a timeout or a thermal limit. A wrong language is a routing bug. Write the class on the ticket before anyone says 'retrain'.

Tools that are allowed

• Local trace viewer and SIEM queries.
• A replay CLI that takes a request id and hits staging.
• Diff of prompt packs and tool manifests.
• Gold-set slice: run the ten nearest items after a change.
• A written vendor-view session, recorded.

Tools that are not allowed without a note: personal ChatGPT, a 4G puck, a vendor remote-control binary, exporting a raw trace to email, copying a production database to a laptop 'to work at home'.

Objections you will hear — and what to do with them

We cannot debug without their senior engineer in the US.

Then you cannot operate. Either train a local engineer to the first four classes, or do not take the workflow live. Time zones are not an architecture.

Logging retrieval sets will explode storage.

Sample, compress, retain 180 days as required, and drop raw chunks earlier only if counsel agrees and you still keep ids. Blindness is more expensive than disks.

The officer wants a fix now, not a class.

Give them the paper path now. Give the system a class. A fast wrong patch is how Saturday becomes Sunday.

A two-week debug-readiness playbook

1. Week 1: implement request ids, joined traces, and a replay CLI on staging.
2. Week 2: run three fire drills — stale circular, ACL leak, GPU cap — and write the class on each ticket. Add the forbidden-tools list to the SOP.

How this shows up in the file

The SOP should list the Saturday sequence and the forbidden hops. Incident tickets should show a class. When someone asks why you refused to mail a trace, the SOP is the answer, not a personality clash with the TAM.

Writing the incident so the next shift can read

Saturday work dies on Sunday if the ticket says 'fixed prompt'. Write the class, the request id, the hashes, the change, and the gold-slice result. The next officer should be able to replay without calling you. If they cannot, you performed heroics, not operations.

Field	Example
Class	Retrieval / stale in-force flag
Request id	a3f1…
Model / pack	sha256:… / sha256:…
Change	Flagged G.O. 12/2021 withdrawn; re-ingest id 8841
Gold slice	withdrawn-rule tag 12/12 pass
Hops	None; vendor not shown the trace

Keep a library of these tickets inside the enclave. They are better training than a vendor webinar. They also show a later auditor that you debug without exporting citizens.

If the same class repeats three times, it is not a Saturday problem. It is an ingest or ACL design problem. Promote it out of the incident queue into the backlog the process owner sees.

This article is a field guide, not legal, procurement, electrical or engineering advice. Confirm numbers, duties and designs against the current Gazette, CERT-In directions, your SDC / NIC / campus standards, a site survey and your counsel before you file them.

How to prove this on a rack, not on a slide

“Air-Gapped Agent Debugging: A Field Guide” only matters if a CISO can fail it. A P4 Systems Integrator should be able to point at a cable, a registry, a licence file, a PDU reading or a SIEM index and say: this is the control. If the only evidence is a brochure that mentions “debugging air gapped AI”, you do not have the control.

You cannot paste the trace into a chatbot. You cannot open a vendor tunnel 'for five minutes'. Debugging behind the gap is a craft: reproduce, reduce, record, and refuse to export the citizen. Air-gap and on-prem programmes die in the second month, when the first update, the first crash, or the first GPU lead-time slip arrives. Budget the boring path — media, offline licence, local registry, local traces — in the same note as the model name.

On-prem is not air-gapped. An India region is not either. Write the forbidden path (outbound HTTPS, licence phone-home, crash reporter, hidden model API) as a numbered list and test it with the internet off. Whatever still dies was a dependency you did not draw.

1. Draw the data path for one user-visible answer under “debugging air gapped AI”.
2. Disable outbound internet on staging and run the demo script.
3. List every remaining hop: update, licence, registry, NTP, DNS, SIEM.
4. Give each hop an owner inside the department, not only the SI.
5. Minute the restore or the media-transfer once before go-live.

Close this loop before the next CAB

Put “Air-Gapped Agent Debugging: A Field Guide” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 Systems Integrator, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “debugging air gapped AI” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.

What the next noting must contain

“Air-Gapped Agent Debugging: A Field Guide” belongs in a file, not only in a search result. A P4 Systems Integrator should be able to point at one artefact that proves “debugging air gapped AI”: a packet capture, a processing schedule, a scored evaluation row, a dated notice, or a refusal rule. If the only evidence is a slide, you have a heading.

You cannot paste the trace into a chatbot. You cannot open a vendor tunnel 'for five minutes'. Debugging behind the gap is a craft: reproduce, reduce, record, and refuse to export the citizen. DPDP 2023 does not define sovereign AI and does not write a blanket localisation rule for every model hop. CERT-In’s 28 April 2022 directions still set specified incident clocks and 180-day log retention in India for in-scope events. The November 2025 AI governance text is guidance, not a statute. A Proprietary Article Certificate, when it is lawful, lives in GFR Rule 166 — not Rule 161.

Write three dated sentences under C2 Air-Gapped & On-Prem: what was decided, which designation owns it after the next posting order, and when it will be re-checked. Unsigned sentences are souvenirs. Dated sentences are controls.

• Name the designation that owns “debugging air gapped AI”, plus a deputy.
• Attach one artefact a stranger can open next year.
• Name the instrument you are actually using — Act, direction, GFR clause, GeM term, or guideline paragraph.
• Leave unsourced percentages, GMV slides and house forecasts out of the noting.
• Revisit when the model, the SI, the notice, the region or the posting changes.