Running Agents on CPU-Only Government Hardware

The default Indian government AI indent now contains a GPU even when the workflow is draft a reply from this circular. GPUs are useful. They are also scarce, hot, slow to buy, and embarrassing when they sit at 4% utilisation behind a collectorate helpdesk that sees forty queries a day. CPU-only inference is not a consolation prize. For many public workflows it is the design that can actually be staffed.

This is a field guide for programme owners who have a rack of older Xeon or EPYC blades, a political request for an agent, and no honest GPU date. It is not a claim that a 70-billion-parameter chat model will feel like a hosted demo on a 2018 blade. It is a claim that you can choose a smaller local model, a tight corpus, and a human-in-the-loop desk, and still retire the unofficial public-chatbot habit.

If your use case is nightly batch summarisation of scanned dak, CPUs are often enough. If your use case is a public multimodal assistant with sub-second tokens, they are not. Write which one you are actually doing.

What CPU can honestly do in 2026

Retrieval: embedding a circular corpus and fetching the right standing order is light work. Do it on CPU without apology. Most the model is dumb complaints in districts are retrieval and corpus problems, not missing tensor cores. Drafting: a small or medium instruction-tuned model, quantized, with a 4–8k context and a retrieval pack in front, can draft a grievance reply or a meeting brief at a latency an officer will tolerate if they were going to spend twelve minutes anyway. They will not tolerate it if you promised a magical typewriter.

Classification and routing: label this ticket as revenue / welfare / police, or send it to the right section. This is often a classic model or a tiny transformer. It does not need a data-centre GPU. What CPU will not honestly do: large-context read this 200-page DP and chat, high-concurrency public portals, speech-to-speech desks, and image-heavy pipelines. If leadership wants those, either wait for hardware or refuse the scope. A slow hallucination is not a sovereign success.

Match the workflow to the metal you have, not the metal in the keynote.

Workflow	CPU-only posture	What you must change vs the hosted demo
Circular Q&A for section officers	Usually yes	Smaller model, ruthless corpus, cite the circular
Grievance draft for a human to edit	Usually yes at desk volume	Queue, not instant for 200 concurrent citizens
Overnight dak / file briefs	Yes, batch	Run at night; do not pretend it is interactive
Public chatbot, open domain	No, not honestly	Refuse or move to a larger shared service
Speech / vision desks	Rarely, except tiny models	Budget accelerators or do not announce the desk
Fine-tune every week on new tickets	Painful	Prefer prompts and retrieval; fine-tune rarely

Engineering habits that make CPU survivable

Quantize and then eval. Do not ship an untested 4-bit model because a blog said it was fine. Score your refusal tests and your citation tests on the quantized build. If citations break, go up a precision or down a workflow. Cap concurrency. A collectorate does not need 128 parallel generations. It needs two, with a queue and a message that says the brief will be ready in eight minutes. Queues are architecture. They are also manners.

Keep context short. Officers will paste entire files. Truncate with a visible note: only the first N pages were read; attach the note if that is unsafe. A CPU box that tries to swallow a 200-page PDF will look broken. A CPU box that refuses clearly looks honest. Separate embedders and generators. Cache embeddings of circulars. Circulars do not change hourly. Prefer batch windows for anything that is not a citizen at the counter. Night jobs hide latency. Interactive desks cannot.

When to stop and ask for a card

When a scored eval shows officers editing more than they type from scratch. When queue time exceeds the statutory or political clock. When you are refusing half the queries because context will not fit. Those are hardware or scope problems. Do not keep optimising a 2016 blade forever to avoid a note to finance. When you ask, ask for a class and a route, not a brand from a conference. And keep the CPU path as the fallback so a slipped allocation does not darken the desk.

• Write latency as a percentile you measured on your metal, not as a vendor slide.
• Show officers the queue. Hidden queues feel like outages.
• Re-eval after every quantization or model swap.
• Do not steal cycles from the district MIS host. Buy or borrow a separate box.
• Document power and cooling even for CPUs — a sealed NIC room in May is hostile.

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

CPU means we are not serious about AI.

Serious is a scored workflow on metal you can keep up in May. A dark GPU is not serious. It is interior decoration with a PSU.

Quantization will wreck Hindi and the state language.

It might. That is why you eval in those languages on the quantized build before you promise a desk. If it wrecks them, raise precision, shrink the model family, or wait for a card. Do not discover this on the collector's first Gujarati file.

We will start on CPU and silently swap the model later.

Swaps need a re-eval and a change ticket. Silent swaps are how citations rot. Plan the swap as a version, not as a surprise.

Thirty days to a CPU desk that officers will use

1. Days 1–5: pick one draft-only workflow and a tiny corpus you already have the right to use.
2. Days 6–12: choose two candidate small models, quantize, and score on twenty real (redacted) items.
3. Days 13–20: put the winner on a dedicated box, not the MIS host; add a visible queue.
4. Days 21–30: sit with two officers, measure edit distance and complaints, write whether you still need a GPU.

How this shows up in the file

File the measured latency, the language-wise eval, the decision to stay on CPU or to indent a card, and a sentence that the MIS host will not share this load. A later engineer should not have to rediscover why the small model won. If you cannot measure, you cannot claim CPU is enough. You can only claim it is cheaper.

How to talk about tokens without lying to officers

Officers do not want tokens per second. They want to know whether the brief will be ready before the DM's meeting. Translate measurements. On this blade, a two-page brief from a four-page file is typically ready in six to ten minutes, and we show a queue. That sentence can go in a note. We achieve 18 tokens/s cannot, unless the reader is the implementer. Measure on the live corpus language. A model that is fine in English bureaucratic prose can collapse on a mix of Hindi and a state language with legal nouns.

Batch windows as a political tool

Overnight briefs are a way to make CPU look generous. The reader arrives to a pile of drafts. Interactive chat at the public counter is a way to make the same CPU look broken. Split the workflows in the note so nobody demands counter-speed from a night job.

Power, dust and the 2018 blade

CPU-only is not maintenance-free. Dust filters, UPS, and a rule that the MIS host is not the inference host still apply. A blade that also runs the property-tax database will be blamed for both slowness and hallucinations. Separate them even if both are boring Intel. When you finally indent a GPU, keep the CPU path as the documented fallback. The first week of a new card is an RMA week more often than slideware admits.

What the next noting must contain

“Running Agents on CPU-Only Government Hardware” belongs in a file, not only in a search result. A P4 Programme/Implementation should be able to point at one artefact that proves “CPU inference government”: a packet capture, a processing schedule, a scored evaluation row, a dated notice, or a refusal rule. If the only evidence is a slide, you have a heading.

Most district workflows are retrieval, draft and refuse — not image generation. That work can live on boring CPUs if you choose models, batch sizes and honesty about latency. DPDP 2023 does not define sovereign AI and does not write a blanket localisation rule for every model hop. CERT-In’s 28 April 2022 directions still set specified incident clocks and 180-day log retention in India for in-scope events. The November 2025 AI governance text is guidance, not a statute. A Proprietary Article Certificate, when it is lawful, lives in GFR Rule 166 — not Rule 161.

Write three dated sentences under C2 Air-Gapped & On-Prem: what was decided, which designation owns it after the next posting order, and when it will be re-checked. Unsigned sentences are souvenirs. Dated sentences are controls.

• Name the designation that owns “CPU inference government”, plus a deputy.
• Attach one artefact a stranger can open next year.
• Name the instrument you are actually using — Act, direction, GFR clause, GeM term, or guideline paragraph.
• Leave unsourced percentages, GMV slides and house forecasts out of the noting.
• Revisit when the model, the SI, the notice, the region or the posting changes.

This article is informational field guidance for Indian public institutions, not legal, procurement or engineering advice. Confirm against the current Gazette, GFR, GeM portal terms, OEM quotations and your counsel before you file it.