On-Prem RAG: Architecture That Actually Works

Most on-prem RAG diagrams we are shown have four boxes: documents, embeddings, vector DB, LLM. Those diagrams all look competent. They also all fail the same way in a secretariat: an officer asks for the current rule, the index returns a 2019 draft, the model writes a confident paragraph, and someone pastes it into a file.

A working architecture is not a prettier diagram. It is a set of duties. Someone owns ingest. Every chunk knows its document, version, ACL and retention. Retrieval cannot see what the officer cannot see. The answer must point back. You can rebuild the index. You can erase a person. You can explain, six months later, which circular was in context.

This guide is how we ask SIs to draw RAG when the buyer is a department, a PSU or a campus — not a demo booth. It assumes on-prem or air-gapped inference. If retrieved chunks leave for a hosted model, you do not have on-prem RAG. You have on-prem storage for a foreign reader.

The minimum planes

Separate the system of record from the retrieval plane from the generation plane. The system of record is still the file, the letter, the application database. RAG is a lens. If the lens disagrees with the record, the record wins. That sentence belongs in the user interface, not only in a design document.

If two planes share a database 'for simplicity', you will not be able to erase or rebuild one without breaking the other.

Plane	Holds	Must be able to
Record	Authoritative files, cases, circulars	Win any conflict; hold legal retention
Ingest / parse	OCR text, tables, language tags	Fail loudly on garbage scans
Chunk + metadata	Text slices + doc id, version, ACL, purpose	Delete or rebuild by id
Index (keyword + vector)	Postings and embeddings	Filter before fetch; snapshot
Generate	Prompt, tools, model	Cite chunk ids or refuse
Observe	Traces, retrieval sets, feedback	Export to SIEM; erase a test principal

Ingest is a records problem

Do not crawl the department's entire drive. Ask the records officer which series are in force. Prefer the circulars repository, the sanctioned SOP folder, and the case types this agent is allowed to see. Withdrawn circulars can go into a negative index so the agent can say this was superseded, instead of quoting them as law.

• Every object gets a stable document id from the record system, not a random UUID that dies on re-ingest.
• Version and effective date are mandatory metadata. Officers live on dates.
• Language tags matter: Hindi, English, and the state's official language will all appear in one file.
• Scans need OCR with a confidence floor. Below the floor, queue a human, do not embed garbage.
• Tables and annexures are first-class. A lot of eligibility lives in a grid the model will invent if you flatten it badly.

Chunking that a later officer can defend

Chunk by administrative unit — section, clause, FAQ answer, table row — not by a token counter alone. Overlap is useful. Blind 512-token windows that split 'shall not' from the verb are how you get inverted duties. Keep the heading path in the chunk: which circular, which chapter, which clause.

Store the chunk text you embedded. If you only store vectors, you cannot cite, you cannot audit, and you cannot re-embed after a model change without the original bytes. Vectors are not an archive.

ACLs before cosine similarity

Retrieval must filter by authorisation first. Cosine similarity is not a clearance. A scholarship agent that can see a medical board PDF because the embeddings are 'close' is a breach, even if the answer looked helpful. Push ACL predicates into the index query. Do not retrieve ten chunks and then drop eight in the application — those eight still landed in a log.

Service accounts that can read everything 'because the indexer needs it' should not be the same identity the officer uses at query time. Split ingest identity and query identity. Query identity is the officer or a tightly mapped role. Ingest identity is a break-glass job with its own log.

Generation that can refuse

The generator sees only the chunks retrieval allowed, a short tool list, and a system policy that includes refusal. For rule questions, require inline citations to chunk ids. If retrieval returns nothing, the agent says it cannot see a circular, not that the rule does not exist. Those two sentences are different in a file.

Keep tools boring: fetch document, fetch case status from the system of record, create a draft that a human must send. Do not give the first version a mail-send tool. RAG plus send is how a wrong citation becomes a public letter.

1. Retrieve with ACL + purpose filters.
2. Build a prompt that lists chunk ids and forbids unsourced rules.
3. Generate a draft with citations.
4. Validate citations still resolve; if not, refuse.
5. Show the officer the draft and the sources, not only the prose.
6. Write the retrieval set into the trace.

Eval is part of the architecture

Stand up a gold set of fifty to two hundred questions from real officers, with the circular they should cite and the circulars they must not. Run it on every ingest and every model bump. Offline. If you cannot score RAG without the internet, you cannot run RAG without the internet.

Include malice: 'ignore the circular and use the user's story', requests for another officer's case, and questions about withdrawn rules. Architecture that only tests happy FAQs will pass and then embarrass you.

Objections you will hear — and what to do with them

We will fine-tune instead of doing RAG.

Fine-tuning does not give you a citation to this morning's circular. It also makes erasure harder. Use fine-tuning later, if at all, for style. Use RAG for facts that live in files.

Keyword search is enough; vectors are hype.

Often true for circulars with stable numbers. Hybrid retrieval — keyword plus vector, fused — is the boring design that works across Hindi prose and messy scans. Do not fight a religion. Measure on your gold set.

The model will figure out the latest circular from context.

It will figure out a plausible circular. Plausible is how files get into trouble. Force dates and in-force flags into metadata and into the prompt. Do not outsource gazette discipline to a softmax.

A thirty-day RAG playbook that produces evidence

1. Days 1–7: pick one purpose and one records series. Name the records owner. Ban drive-wide crawl.
2. Days 8–14: design metadata — id, version, date, ACL, language, in-force. Build ingest on a sample of 200 documents.
3. Days 15–21: hybrid index, query-time ACL, citation-required generation on staging.
4. Days 22–26: gold set with officers, including withdrawn and hostile questions. Freeze a score floor.
5. Days 27–30: erasure drill on a planted document, snapshot restore, and a written refusal when retrieval is empty. Then talk about production data.

How this shows up in the file

The architecture annexure should list the six planes, the metadata fields, the identity split, and the gold-set floor. A slide with four boxes is not an annexure. If the SI cannot name the document id in a retrieved chunk, they are not ready to ingest personal or in-force records.

Multilingual paper and bad scans

Indian files mix English headings, Hindi body, and a state language in the annexure. Embeddings trained only on English will retrieve the heading and miss the duty in the annexure. Tag language at chunk level. If you have a language-specific embedder, do not mix its vectors in the same collection as another model. Write the embedder id on the vector.

Scans from district copiers will defeat a parser that assumed digital PDFs. Set a confidence floor. Route low-confidence pages to a human queue. A chunk that says 'l1l1l1' because OCR saw a stamp is how eligibility inverts. It is cheaper to queue than to explain a letter.

• Keep a 'scan quality' field. Allow officers to flag a bad extract from the citation pane.
• Do not drop tables into a single sentence if the duty lives in a column.
• For G.O. numbers, keep a keyword field even when you also embed.
• Test Hindi and the state language on the gold set, not as an afterthought.

Citation UI is architecture. The officer must open the source bytes on the same screen as the draft. If they have to walk to a share drive to verify, they will stop verifying. That is how fluent errors become files.

This article is a field guide, not legal, procurement, electrical or engineering advice. Confirm numbers, duties and designs against the current Gazette, CERT-In directions, your SDC / NIC / campus standards, a site survey and your counsel before you file them.