Storage Design for Departmental Document Corpora

SIs like to start RAG with a crawler. Records officers like to start with a series. Only one of those people can keep you out of trouble. A departmental corpus is not 'all PDFs we could find'. It is a set of series with owners, retention, language and an in-force status. Storage design is how that set stays true after the first enthusiastic intern.

This guide sits under the RAG architecture article and above the vector-database comparison. Here we talk about bytes: what to keep, how to name it, how to tier it, and how to delete it. If this layer is sloppy, no index will save you.

DPDP storage limitation and ordinary records law both push in the same direction: do not keep personal or closed material only because a crawler already paid for the embedding.

Three tiers, one identifier

Tier	Contents	Who may write
Source	Original PDF/scan/office file, checksum, MIME	Ingest job + records officer
Text	OCR/extract, tables, language, quality score	Ingest job; human correctors
Derived	Chunks, vectors, thumbnails	Pipeline only; rebuildable

Give every source object a stable id from the records system or a minted id that is written back to the records system. Derived objects carry that id. If you cannot go from a citation to the source bytes in one hop, you do not have a corpus. You have a haystack.

What to refuse at the door

• Objects with no series and no owner.
• Personal mailboxes and 'misc' folders.
• Password-protected files nobody can open in the enclave.
• Scans below the OCR confidence floor, unless queued for humans.
• Duplicates by checksum — point, do not copy, unless the series requires the copy.
• Anything the records officer marks out of scope, even if similarity would love it.

Formats

Prefer the format the department already treats as authentic. Do not convert a signed PDF to a Word file to make parsing easier and then throw away the PDF. Keep office files if that is the original; produce a PDF/A for human viewing if your policy says so. Write the conversion as a derived object, not as a replacement.

Retention and erasure

1. Map each series to a retention rule the department already has.
2. When the source hits the rule, delete or archive per that rule, then delete derived chunks and vectors by id.
3. Certificate the date derived stores are gone. Backups expire on their cycle — write that honestly.
4. Do not use the AI store as a way to keep a record the paper schedule already killed.

Capacity planning without fake precision

Scans are large. Text is small. Vectors sit in between and multiply when you re-embed. Plan source storage from the series you chose, not from 'the whole NAS'. Give yourself room for one full re-embed. Measure a sample of 100 documents and extrapolate as a range. Then buy hall storage, not a surprise NVMe on the GPU box.

Objections you will hear — and what to do with them

The officers will not use it if every document is not in.

They will not trust it if the wrong document is in. Start with the in-force series. Expand on request with an owner.

Object storage is cloud thinking.

Checksums and ids are records thinking. You can do them on a NAS. The word object is not a region.

OCR can wait.

Then those scans are not in the corpus. Do not embed empty shells and pretend. Queue them.

A three-week corpus playbook

1. Week 1: sit with the records officer. List series, owners, retention, in-force flags.
2. Week 2: stand the three tiers with ids and checksums. Ingest a sample. Refuse the junk.
3. Week 3: erasure drill on a planted document, lifecycle on a closed dummy case, and a citation hop from answer to source bytes. Then scale ingest.

How this shows up in the file

Attach the series list, the tier diagram, and the retention map. A crawler setting has no place on page one. When someone asks why a popular pamphlet is absent, the series list is the answer.

Naming and prefixes that a later SI can inherit

Use prefixes that match series, not teams: state/revenue/go/2024/… not ramesh/tmp/new. Team prefixes die when Ramesh is transferred. Series prefixes match the records officer's language and the retention map.

Write checksums at ingest and verify them on restore. Bit rot and silent NFS faults exist. A citation to a PDF that no longer hashes the same is a dispute waiting for a citizen.

• id, series, source_checksum, ingested_at, in_force, language, owner, retention_code.
• A tombstone object when you delete, so a dangling chunk is a bug you can find.
• A ban on personal desktops as a source path, even for 'one important file'.

If two departments share a hall store, prefix at the department first. Shared buckets with good intentions become cross-citations. Storage is a tenancy control, not only a disk.

This article is a field guide, not legal, procurement, electrical or engineering advice. Confirm numbers, duties and designs against the current Gazette, CERT-In directions, your SDC / NIC / campus standards, a site survey and your counsel before you file them.

How to prove this on a rack, not on a slide

“Storage Design for Departmental Document Corpora” only matters if a CISO can fail it. A P4 Systems Integrator should be able to point at a cable, a registry, a licence file, a PDU reading or a SIEM index and say: this is the control. If the only evidence is a brochure that mentions “document storage AI government”, you do not have the control.

The vector index is not the archive. Design a document store a records officer will sign: tiers, identifiers, retention, and a path back to the file that still wins. Air-gap and on-prem programmes die in the second month, when the first update, the first crash, or the first GPU lead-time slip arrives. Budget the boring path — media, offline licence, local registry, local traces — in the same note as the model name.

On-prem is not air-gapped. An India region is not either. Write the forbidden path (outbound HTTPS, licence phone-home, crash reporter, hidden model API) as a numbered list and test it with the internet off. Whatever still dies was a dependency you did not draw.

1. Draw the data path for one user-visible answer under “document storage AI government”.
2. Disable outbound internet on staging and run the demo script.
3. List every remaining hop: update, licence, registry, NTP, DNS, SIEM.
4. Give each hop an owner inside the department, not only the SI.
5. Minute the restore or the media-transfer once before go-live.

Close this loop before the next CAB

Put “Storage Design for Departmental Document Corpora” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 Systems Integrator, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “document storage AI government” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.

Store classes you must name

Hot corpus, cold archive, chunk store, vector index, eval set, backup. Each class needs a retention clock and an owner. If two classes share a disk with no access split, purpose limitation is a story, not a control.

Do not put vigilance files and citizen schemes on one index because retrieval quality improved. Quality is not a lawful basis.

What the next noting must contain

“Storage Design for Departmental Document Corpora” belongs in a file, not only in a search result. A P4 Systems Integrator should be able to point at one artefact that proves “document storage AI government”: a packet capture, a processing schedule, a scored evaluation row, a dated notice, or a refusal rule. If the only evidence is a slide, you have a heading.

The vector index is not the archive. Design a document store a records officer will sign: tiers, identifiers, retention, and a path back to the file that still wins. DPDP 2023 does not define sovereign AI and does not write a blanket localisation rule for every model hop. CERT-In’s 28 April 2022 directions still set specified incident clocks and 180-day log retention in India for in-scope events. The November 2025 AI governance text is guidance, not a statute. A Proprietary Article Certificate, when it is lawful, lives in GFR Rule 166 — not Rule 161.

Write three dated sentences under C2 Air-Gapped & On-Prem: what was decided, which designation owns it after the next posting order, and when it will be re-checked. Unsigned sentences are souvenirs. Dated sentences are controls.

• Name the designation that owns “document storage AI government”, plus a deputy.
• Attach one artefact a stranger can open next year.
• Name the instrument you are actually using — Act, direction, GFR clause, GeM term, or guideline paragraph.
• Leave unsourced percentages, GMV slides and house forecasts out of the noting.
• Revisit when the model, the SI, the notice, the region or the posting changes.