Backup and DR for On-Prem AI Workloads

On-prem AI inherits a bad habit from data-science laptops: the work is the model, so people back up the model. Then a PDU dies and they discover the system was the index, the chunk store, the ACL map, the prompt package, the licence file, the gold set and the officer identities. The weights restore in an hour. The service does not.

Disaster recovery for an agent is closer to a database estate than to a workstation image. You must know what is authoritative, what is derived, what is a secret, and in what order to bring them back. You must also know which citizen process runs on paper while you do it.

NIST SP 800-34 is a useful reminder that contingency is a documented process, not a storage SKU. Use it as orientation. Your file still needs your halls, your RTOs and your names.

Classify the stores before you buy a tool

Store	Class	DR implication
System of record (files, cases)	Authoritative	Already in departmental backup; do not invent a second truth
Chunk text + metadata	Authoritative for RAG	Protect like a database; this rebuilds vectors
Vector index	Derived	Snapshot if RTO is tight; else rebuild
Model weights / adapters	Artefacts	Two hashed copies in the artefact library
Prompts, tools, policies	Config as code	Versioned; restore with the model hash
Gold set + harness	Quality control	If you lose it, you cannot safely change anything
Traces / logs	Evidence	[CERT-In](/blog/cert-in-reporting-when-an-agent-misbehaves) 180-day duty; restore to SIEM rules
Keys / licence files	Secrets	Separate ceremony; never only on the GPU node
Local NVMe scratch	Ephemeral	Do not pretend it is backed up

RTO, RPO, and the paper path

Ask the process owner what happens if the agent is dark for four hours, one day, one week. If the answer is 'the counter stops', you are in second-site territory. If the answer is 'we revert to the existing file movement', write that path and test that it still exists. Many departments deleted the old desk when the chatbot launched. That is not digital transformation. That is burning the boats.

• RPO for chunks: how many hours of ingest can you replay from the record system?
• RTO for inference: do you need a warm GPU or is a day's delay acceptable?
• RTO for traces: can you lose a morning of logs, or is that a CERT-In and audit problem?
• People RTO: who is allowed to restore keys? If they are on tour, the disk is a paperweight.

Order of restore

1. Identity and time. A restored cluster with the wrong clock will wreck licences and log correlation.
2. Keys and licence files, by ceremony.
3. Object store / chunk store.
4. System of record connectivity.
5. Vector rebuild or snapshot restore.
6. Model artefacts (verify hashes).
7. Prompt / tool package matching those hashes.
8. Gold-set smoke: a handful of items before officers return.
9. SIEM heartbeat. If you cannot see, you are not up.

Second-site options that are not a foreign region

• Warm: a second node in another Indian government hall, artefacts already hashed there, DNS or a runbook cutover.
• Cold: artefacts and encrypted chunk replicas in a second hall; GPU borrowed or hired when needed.
• Paper: a documented manual process with staff who still remember it.
• Degraded: a smaller CPU model that answers a subset — public FAQs only — while the GPU hall recovers.

Choose in a meeting with the process owner. Do not let the SI choose 'warm everywhere' as a way to double a BoQ, or 'paper' as a way to skip work. Write the RTO next to the choice.

Objections you will hear — and what to do with them

The hall already does DR.

Ask whether their DR includes NVMe, GPU nodes, licence files and a vector rebuild. Hall DR is usually VMs and databases. Extend it explicitly or it will miss you politely.

We cannot copy personal embeddings to a second site.

Then either encrypt and contract that site as a processor, or accept a rebuild-from-records RTO. Do not copy 'just the vectors' into an unnamed bucket and call it safer. It is not.

Snapshots will hurt GPU performance.

Then snapshot the chunk store and accept rebuild time, or schedule snapshots in a quiet window. Performance is not an argument for zero recoverability.

A 45-day DR playbook

1. Days 1–10: inventory stores and classify authoritative vs derived. Find the NVMe nobody listed.
2. Days 11–20: write RTO/RPO and the paper path. Get the process owner to initial.
3. Days 21–35: implement chunk replicas, artefact dual copies, key ceremony, restore order.
4. Days 36–45: pull the plug on staging. Time the restore. Fix the first missing hash. File the minutes.

How this shows up in the file

The DR annexure is a restore order, a site name, an RTO, and the date of the last drill. A sentence that says 'backup as per SDC policy' is how NVMe disappears. When the PDU dies, that annexure is the only document anyone should need.

A ninety-minute tabletop that is not theatre

Hand the room a card: the GPU row lost power at 10:10; UPS held twelve minutes; the node did not shut clean; the on-call is on a train. They must produce either a restored staging-quality service or a citizen-facing message the secretary will sign, plus a time.

1. Minute 0–15: classify what is likely intact (source store in the hall) versus suspect (NVMe index).
2. Minute 15–35: start the restore order. Who has the key ceremony? Who has the artefact hashes?
3. Minute 35–60: if restore will miss RTO, open the paper path. Write the public sentence.
4. Minute 60–90: list the missing artefact. Assign an owner. That list is the real output.

If the missing artefact is 'the only copy of the adapter was on the dead node', you just learned why dual copies exist. If the missing artefact is a phone number, you learned that DR is a directory problem. Both are cheaper in a conference room.

Run the tabletop before peak season. Exam weeks and scheme launches are bad moments to discover that the restore order was optimistic.

This article is a field guide, not legal, procurement, electrical or engineering advice. Confirm numbers, duties and designs against the current Gazette, CERT-In directions, your SDC / NIC / campus standards, a site survey and your counsel before you file them.

How to prove this on a rack, not on a slide

“Backup and DR for On-Prem AI Workloads” only matters if a CISO can fail it. A P1 CIO/CTO should be able to point at a cable, a registry, a licence file, a PDU reading or a SIEM index and say: this is the control. If the only evidence is a brochure that mentions “AI disaster recovery on premise”, you do not have the control.

An on-prem agent without a restore drill is a single crate with a fan. Back up the derived stores, the keys and the gold set — and decide which hall is the second site. Air-gap and on-prem programmes die in the second month, when the first update, the first crash, or the first GPU lead-time slip arrives. Budget the boring path — media, offline licence, local registry, local traces — in the same note as the model name.

On-prem is not air-gapped. An India region is not either. Write the forbidden path (outbound HTTPS, licence phone-home, crash reporter, hidden model API) as a numbered list and test it with the internet off. Whatever still dies was a dependency you did not draw.

1. Draw the data path for one user-visible answer under “AI disaster recovery on premise”.
2. Disable outbound internet on staging and run the demo script.
3. List every remaining hop: update, licence, registry, NTP, DNS, SIEM.
4. Give each hop an owner inside the department, not only the SI.
5. Minute the restore or the media-transfer once before go-live.

Close this loop before the next CAB

Put “Backup and DR for On-Prem AI Workloads” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P1 CIO/CTO, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “AI disaster recovery on premise” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.