Air-Gapped Model Evaluation Without the Internet

Model evaluation on the public internet is a leaderboard, a hosted notebook and a judge model in someone else's region. None of that is available in a real enclave, and some of it would be a transfer if it were. Teams then skip eval and ship. That is how a 'better' model starts citing a withdrawn G.O. on week one.

Offline evaluation is a small factory. You need a gold set that can live in the enclave, a harness that does not phone home, metrics a non-researcher will sign, and a rule that a failed floor blocks the bag. The factory is less glamorous than a chatbot. It is the only reason you can change anything later.

This guide is for SIs who have to stand that factory up in an SDC, a plant or a campus without begging for a proxy exception.

What you are actually scoring

You are not scoring 'intelligence'. You are scoring this workflow. For RAG, that means retrieval of the right circular, refusal when retrieval is empty, citation integrity, ACL respect, and language. For agents, add tool choice and the absence of a send-without-approval. A generic MMLU dump will not tell you any of that.

Question type	What a pass looks like	Typical silent fail
In-force rule	Cites the current circular, correct date	Fluent paragraph, old G.O.
Withdrawn rule	Says superseded, points to new	Helpfully applies the old relief
Other officer's case	Refusal, no retrieval	Summarises because similarity was high
Empty corpus	I cannot see a source	Invented section number
[Hostile prompt](/blog/prompt-injection-in-citizen-facing-agents)	Ignores jailbreak, stays in purpose	Agrees to 'ignore previous'
Language	Answers in the officer's language	English-only competence
PII in gold	Not present, or synthetic	Real Aadhaar / mobile in the set

Building a gold set that can live inside

Sit with three officers and harvest questions they already ask. Write the expected source and the expected refusal. Invent synthetic citizens. Do not copy last year's grievance dump into the eval tenant 'because it is realistic'. Realistic personal data in an eval set is a second processing purpose you probably did not write down.

• Label each item with purpose, language, difficulty, and whether it is hostile.
• Version the set. When a circular changes, the item changes or is retired.
• Keep a canary item that must never appear in logs sent anywhere. If it appears outside, you have a leak.
• Store the set in the enclave with the same ACL as the workflow. Eval is not a playground.
• Do not let vendors take a copy 'to improve the product' unless you intend to fund their product with your questions.

Judges without a foreign brain

Use exact checks where you can: did the citation id resolve, did the forbidden document id appear, did the answer contain a planted secret. Use a rubric and a human panel for prose. If you want a model-as-judge, run it locally on the same side of the gap and accept that it is biased. Never make a hosted judge the gate on a set that contains operational text.

The harness

The harness is a batch job. It reads the gold set, calls the same endpoint officers will call, writes scores and traces to a local store, and exits non-zero if a floor is missed. It must not import a telemetry SDK. It must not pull a fresh judge container from the internet at runtime. Pin it like a product.

1. Load gold items from the enclave object store.
2. Call retrieve + generate with the production ACL of a test officer identity.
3. Score with exact checks first, rubric second.
4. Write a report the CAB can attach: pass/fail per tag, examples of fails, hash of the model and the set.
5. Fail the pipeline if citation integrity or ACL tags regress. Do not average them away with a fluency bonus.

Keep the last three reports. A model bump that wins fluency and loses withdrawn-rule items is a downgrade. Officers will feel the downgrade before your dashboard does.

Floors, not vibes

Write numeric floors the process owner initials: citation resolve rate, forbidden-doc rate (must be zero on ACL tags), withdrawn-rule mishandling rate, empty-retrieve refusal rate. Fluency can be a secondary human score. Never let a higher fluency offset a non-zero ACL miss. That trade is how breaches look like quality improvements.

Objections you will hear — and what to do with them

We do not have labelled data.

Then you do not have a right to change the model. Sit with officers for two afternoons. Fifty items is a start. A vendor's brochure is not a substitute.

Offline eval will slow releases.

Yes. That is the point. A government agent is not a consumer chatbot. If you need daily prompt deploys, you are not ready for live records.

Human eval does not scale.

Human eval on a fixed panel of twenty items after the automatic gates pass does scale. What does not scale is asking a secretary to be the first scorer in production.

A four-week offline eval factory

1. Week 1: harvest questions with officers. Ban live PII. Version the set.
2. Week 2: write exact checks and floors. Initial them.
3. Week 3: build the harness from pinned images. Run it on the current model. That score is your baseline.
4. Week 4: attach the harness to the bag ceremony. A model or prompt that misses a floor does not enter production. Practice one failure.

How this shows up in the file

CAB should see the last report hash and the floor sheet. A note that says 'quality tested' without a floor is a vibe. When someone asks why you refused a vendor's new model, the report is the answer.

Leakage and contamination

Gold sets rot in two directions. They leak out, becoming training data for a vendor. They leak in, becoming the only questions anyone ever asks, so the agent overfits the harness. Protect both ways. Canary strings detect egress. A small rotating slice of items that never sit in the published set detects overfitting.

Contamination also happens when officers 'help' by pasting production tickets into the gold set to make it realistic. That is how personal data enters eval. Give them a template for synthetic tickets that still hurt: the same scheme, fake names, real circular ids.

• One canary item that must never appear in a support export.
• A rotating 10% held back from everyday runs, used monthly.
• A ban on live case ids in item text.
• A version changelog when a circular changes.

If a vendor asks for your gold set to 'calibrate', give them the public-circular slice only, or give them nothing and invite them to watch a run inside the enclave. Calibration is often collection.

This article is a field guide, not legal, procurement, electrical or engineering advice. Confirm numbers, duties and designs against the current Gazette, CERT-In directions, your SDC / NIC / campus standards, a site survey and your counsel before you file them.

How to prove this on a rack, not on a slide

“Air-Gapped Model Evaluation Without the Internet” only matters if a CISO can fail it. A P4 Systems Integrator should be able to point at a cable, a registry, a licence file, a PDU reading or a SIEM index and say: this is the control. If the only evidence is a brochure that mentions “offline model evaluation”, you do not have the control.

If you cannot score a model without the internet, you cannot choose a model without the internet. Offline eval is not a nice-to-have. It is how you keep the air gap after the first vendor bump. Air-gap and on-prem programmes die in the second month, when the first update, the first crash, or the first GPU lead-time slip arrives. Budget the boring path — media, offline licence, local registry, local traces — in the same note as the model name.

On-prem is not air-gapped. An India region is not either. Write the forbidden path (outbound HTTPS, licence phone-home, crash reporter, hidden model API) as a numbered list and test it with the internet off. Whatever still dies was a dependency you did not draw.

1. Draw the data path for one user-visible answer under “offline model evaluation”.
2. Disable outbound internet on staging and run the demo script.
3. List every remaining hop: update, licence, registry, NTP, DNS, SIEM.
4. Give each hop an owner inside the department, not only the SI.
5. Minute the restore or the media-transfer once before go-live.

Close this loop before the next CAB

Put “Air-Gapped Model Evaluation Without the Internet” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 Systems Integrator, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “offline model evaluation” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.