Offline Observability: Logs Without Telemetry

After the air-gap, the SI lost their favourite dashboard. Latency was a feeling. Errors were rumours. An officer said the assistant had become stupid on Tuesdays. Nobody could prove whether Tuesdays were a load problem, a bad promotion, or a single broken tool. The department blamed the gap. The gap was not the problem. The missing local mirrors of the dashboard were.

Disconnecting a collector without replacing the questions it answered is how air-gaps get a reputation for blindness. The questions are ordinary. Is serve up? How long did prefill take? Which tool failed? Did we promote a worse artefact? Is someone scraping the API? None of those questions requires a foreign host to hold the prompt.

This guide is the local estate we ask SIs to stand up before they unplug the SaaS. It is observability as an internal product, not as a phone-home feature.

Three planes of local seeing

If a tool wants to mix these planes into one SaaS event, split it or drop it.

Plane	What it holds	Who may see content
Metrics	Counters and histograms: QPS, latency, queue depth, GPU memory, tool error rates	Almost anyone who operates the box; no content
Structured logs and traces	Request IDs, user IDs if needed, tool names, status, optional redacted payloads	Operators; content fields off by default
Quality and eval	Frozen eval scores after each promotion; sampled local reviews	Eval owners; samples treated as personal data if they are

Metrics should be Prometheus-shaped or equivalent, scraped locally, graphed locally. They survive an air-gap without argument. Traces should land on a local collector. Turn off request bodies until a named person turns them on for a named incident. Quality should be a job that runs on promotion, not a hidden stream of live conversations to a model vendor.

What to log when the principal is in the prompt

An agent log is often personal data. Design it like one. Default fields: timestamp, request id, workflow name, officer id if you must, latency, token counts, tool names, error class. Optional fields behind a flag: retrieved document ids, prompt, completion. The flag writes an audit row when it is turned on.

Retention is not forever. Align specified security logs with CERT-In’s 180-day Indian store. Align content-bearing debug logs with a shorter clock unless a legal hold says otherwise. Erasure must know this store exists. If your write-set list from the erasure article forgot observability, fix that before you congratulate yourself on local metrics.

Dashboards officers will actually use

A local estate that only SRE understands will be ignored, and the next outage will be reported as the model became stupid. Give the workflow owner three tiles: up or down, median latency against last week, and tool-error rate. Those tiles do not need content fields. They need to be on a wall or a page the owner already opens.

Promotion quality needs a fourth tile: eval score versus the previous artefact. If the tile is red, the owner can demand a rollback without reading a trace. That is how you replace the vendor’s magic quality dashboard without sending them conversations.

On-call runbooks should start with those tiles, then a short path to traces with content still off, then the break-glass. If the first step in the runbook is open the vendor console, you have not finished the build. You have a bookmark.

Objections

Our SRE team only knows the SaaS product. Then run the same product’s self-hosted edition if it can live without a licence ping, or train them on a local stack. Skill preference is not a transfer basis.

Without full prompts we cannot improve the model. Improvement is a purpose. If you have it, do it on a scrubbed eval set, locally. Live prompt streams to a vendor are how improvement becomes undeclared processing.

OpenTelemetry exporters are harmless. Exporters are pipes. Pipes go somewhere. Point them at your collector or do not install them.

Traces without novels

A useful trace is a skeleton. Request id, spans for retrieve, generate and each tool, timings, status. The skeleton tells you whether Tuesday was retrieval or decode. The novel — the prompt, the chunks, the draft — stays behind the flag.

Sample traces, not conversations. Keep one in a hundred skeletons for a week if you need to study shape. If you need to study quality, use the frozen eval, not a live stream. Mixing those jobs is how a quality programme becomes a second copy of the case file.

When a vendor insists their collector is required to compute embeddings of traces for anomaly detection, ask whether the same detector can run on skeletons in your VLAN. If the answer is no, you are being sold a transfer, not a detector. Anomaly detection on tool graphs and latency is local work.

Crash folders deserve the same discipline. A dump that includes the last request body is a personal-data store. Name an owner, a retention of days not months, and a rule that the dump is not copied to a laptop without a ticket. The Tuesday disk problem will show up in metrics long before you need to open a dump.

A 40-day local observability build

1. Days 1–10: list every exporter in the product. Kill foreign destinations.
2. Days 11–20: stand up local metrics and a dashboard that answers up, slow, and tool-fail.
3. Days 21–30: structured logs with content off; document the break-glass.
4. Days 31–40: promotion eval job; add the log store to the erasure write-set.

What goes in the file

The exporter inventory, the dashboard screenshots from a soak, the retention table, the break-glass procedure, and the name of the person who can enable content fields. If those exist, you may unplug the SaaS without going blind.

Add the on-call runbook that starts from the three owner tiles, not from a vendor URL. Add the last promotion-eval screenshot. Add the date you last proved that content fields were still off. Observability that is only configured is not observability. Observability that has been used in a dull week is.

If a support bundle ever leaves, file the ticket, the scrub checklist, and the certificate that the foreign copy died. That bundle is a transfer. Treat it with the same seriousness as a prompt to a hosted model, because that is what it often contains.

Prcept should expose metrics and structured logs you can scrape inside the VLAN. If we need a debug artefact, it should be a file you export, not a daemon we left behind.

How to prove this on a rack, not on a slide

“Offline Observability: Logs Without Telemetry” only matters if a CISO can fail it. A P4 System Integrator should be able to point at a cable, a registry, a licence file, a PDU reading or a SIEM index and say: this is the control. If the only evidence is a brochure that mentions “air-gapped AI observability”, you do not have the control.

You still need to know when an agent is failing. You do not need to ship the prompt to a SaaS to find out. Build a local observability estate. Air-gap and on-prem programmes die in the second month, when the first update, the first crash, or the first GPU lead-time slip arrives. Budget the boring path — media, offline licence, local registry, local traces — in the same note as the model name.

On-prem is not air-gapped. An India region is not either. Write the forbidden path (outbound HTTPS, licence phone-home, crash reporter, hidden model API) as a numbered list and test it with the internet off. Whatever still dies was a dependency you did not draw.

1. Draw the data path for one user-visible answer under “air-gapped AI observability”.
2. Disable outbound internet on staging and run the demo script.
3. List every remaining hop: update, licence, registry, NTP, DNS, SIEM.
4. Give each hop an owner inside the department, not only the SI.
5. Minute the restore or the media-transfer once before go-live.

Close this loop before the next CAB

Put “Offline Observability: Logs Without Telemetry” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 System Integrator, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “air-gapped AI observability” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.