Quantisation Trade-offs for Government Accuracy

Quantisation is how a model that wanted 80 GB of VRAM is made to sit in 24. The method is older than this year's vendors: store the weights in fewer bits, keep a few scales, hope the next token does not notice. On a public English quiz the hope often holds. On a scholarship file where the difference between 18,000 and 81,000 is a para, the hope is not a control.

This is a data-study method, not a leaderboard. We will not invent a table of real Indian deployments and pretend it is a census. We will tell you what Q4 and Q8 typically do to different error classes, how to measure them on your own gold set, and which government fields are known to be brittle. The study you can defend is the one you ran on your cases.

It is not legal advice and it is not a warranty. Quantisation formats (GGUF k-quants, GPTQ, AWQ, bitsandbytes) are not interchangeable. A Q4_K_M is not a Q4_0. Write the format, the calibration set and the hash, or you have not written a finding. A slide that says 4-bit is 95 percent is a marketing artefact until it names those three things and your fields.

What the bits actually are

An unquantised instruct checkpoint is usually FP16 or BF16: two bytes per parameter. Q8-style integer quantisation stores about one byte per parameter plus scales. Q4-style stores about half a byte plus more aggressive grouping. The VRAM win is real and roughly linear in the weight tensor. Activations, KV cache and context length still sit on the card. A Q4 70B that you then run at 32k context is not a 24 GB toy.

Quality is not linear. Research and operator experience — including the llama.cpp community's own notes on k-quants — converge on a boring pattern: 8-bit is close to 16-bit on most short-form tasks; 5-bit and well-tuned 4-bit k-quants are often acceptable on fluent English prose; 3-bit and naive 4-bit drop facts. Close is not identical on amounts. Fluency survives quantisation longer than arithmetic, copying, and morphology.

Government work is heavy on the things quantisation hurts first: copying identifiers, adding figures, declining Indic names, keeping a section number intact, and not inventing a gazette date. If your demo is a poem in English, you have not tested the product. If your demo is a GSTIN, a scholarship amount and a section of a 2013 circular, you have begun to test the product.

Typical qualitative pattern — not a scored Indian census. Confirm on your gold set.

Error class	Q8 vs 16-bit	Well-tuned Q4/Q5 vs Q8	What to do
Fluent English draft	Usually indistinguishable in officer review	Often acceptable	Do not use this as your only score
Amount, date, file number copy	Small residual risk	Material risk — digit swaps, dropped commas	Validator; refuse free-text for these fields
Indic morphology / code-mix	Usually holds if the base model could	Often degrades; endings and sandhi break	Gold set in the live language mix
Tool / JSON schema adherence	Usually holds	More malformed calls	Constrained decoding, not hope
Long-context retrieval fidelity	Mild loss	Larger loss; mid-document facts drop	Shorter chunks; do not just add context

How to run the study so finance will accept it

Build a gold set from closed departmental files. Fifty is a start. Two hundred is better. Stratify: languages, districts, form versions, handwritten versus printed, the ugly scans. If the vendor built the set, it is a brochure. If the set is only English, it is a brochure with better typography.

Freeze the prompt, the retrieval pack, the decoding settings and the model hash. Change one thing: the quant. Score field-level exact match on identifiers and amounts, a human rubric on prose, and schema-validity on tool calls. Report confidence intervals even if they are ugly. A single-number 94.6 percent without a denominator is a marketing artefact.

Publish the failures. The interesting rows are the ones where Q4 invented a section, swapped a leading digit, or dropped a matra and changed a name. Those rows decide whether Q4 is allowed near money. A mean score can hide a lethal tail. The file should attach the miss list, not only the mean.

Do not average across workflows. A Q4 that is fine on FAQ chat can be unacceptable on DBT list generation. The file should name the workflow the quant is blessed for, not the platform. A platform blessing is how a later SI copies Q4 onto a payment proposal because last quarter's FAQ looked fine.

1. Name the workflow and the legally material fields.
2. Draw the gold set from your MIS, not from the internet.
3. Pin model, prompt, retrieval and decoder. Vary only the quant and record the format string.
4. Score exact-match, rubric and schema separately. Show the misses.
5. Bless a quant per workflow, with a re-test trigger when the base model or tokenizer changes.

What we will not claim, even if a competitor will

We will not tell you that Q4 is 95 percent of Q8 on Indian government work. That number has been circulating in decks without a corpus, a language mix, or a field definition. It may be true of a particular English chat eval. It is not a statute and it is not your scholarship file.

We will not invent a 28-department table of production accuracies. If we have run a gold set with you, that result lives in your file, under your cases, with your hash. It is not a market index we can republish as if it were a census.

We will not treat a public leaderboard as a substitute. Leaderboards are useful for screening. They are not DPDP, they are not CAG, and they are not your circular's definition of income. Transfer from MMLU to a GSTIN is a hypothesis. Write it as one.

The operational cost of the bits

Q4 saves VRAM and can raise concurrency. That is the whole economic case. If the card is already bought and idle, Q4 saves you nothing until you put a second workflow on it. If the card is not yet bought, Q4 may let you buy a smaller SKU or skip a node. Write which of those two files you are in.

Q4 can raise SI cost. More evals, more validators, more incidents where an officer says the model has become stupid after a rebuild. Budget the re-eval when you change quant format. A silent Q4 rebuild after a CVE is how digits start slipping in October.

On an air-gapped rack the rebuild path matters more than the bit-width. A Q8 you can rebuild from a signed internal registry is safer than a Q4 you can only obtain from a public CDN. Bits do not travel; bags do. See the air-gap ops notes on headcount if that sentence is new. The economic saving of Q4 dies if the only artefact you can bag is a public Q4 someone else built on an unknown calibration set.

Calibration is part of the artefact

A Q4 built on a general crawl is not the same artefact as a Q4 calibrated on your gold set. Write the calibration source. If the SI downloaded a public GGUF and called it yours, the blessing is theirs, not yours. Rebuild from a pinned 16-bit or Q8 checkpoint you hold, or do not claim the study.

When the tokenizer or chat template changes, the quant is a new artefact. The re-test trigger is not bureaucracy. It is how you avoid an October in which amounts slip and nobody can say which bag introduced the bits.

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

The vendor's model card says Q4 is lossless

Ask for the eval name, the language mix, and the field-level misses. If the card reports a single English score, it is not evidence for your MIS. Lossless is a word from compression theory, not from scholarship files.

We do not have time to build a gold set

Then you do not have time to change the bits. Stay on the higher-precision quant you already tested, or do not deploy. Speed is not a waiver from exact-match on money fields.

Q8 will not fit, so Q4 is mandatory

Then shrink the model, shrink the context, or buy the card. Mandatory Q4 on a money field without a validator is a choice to accept digit risk. Write that sentence in the file if you still want it.

A public paper showed Q4 within 1 percent

Cite the paper, the task, and why it matches your field. A 1 percent drop on MMLU is not a 1 percent drop on GSTIN. Transfer is a hypothesis. Test it.

A three-week quant study

1. Week 1: freeze the workflow, pull 50–200 gold cases, name the legally material fields, pin the unquantised or Q8 baseline hash.
2. Week 2: run Q8 and the candidate Q4/Q5 format on the same packs. Score fields separately. Print the misses. Do not average them away.
3. Week 3: bless per step. Add validators where Q4 is allowed near identifiers. Write the re-test trigger. Put the sheet in the purchase or change file.

How this shows up in the file

Subject: Quantisation blessing for [workflow] — Q8 versus [format].

Baseline model [name, hash]. Candidate quant [format string, hash]. Gold set [N] departmental cases, languages [list]. Exact-match on [fields] was [q8] versus [q4]. Misses attached. [Format] is blessed only for [steps]. Money and identity fields remain on [Q8 / validator / human]. Re-test is mandatory on any tokenizer, prompt or base-model change.

This note is not a general accuracy certificate and not legal advice.

This article is informational field guidance for Indian public institutions, not legal, procurement, tax, accounting, tariff or engineering advice. Confirm against the current Gazette, GFR, GeM term, SERC tariff order, IndiaAI portal rule, CAG mandate, DPDP text, departmental finance manual and your counsel before you file it. Figures are methods and order-of-magnitude illustrations, not a dataset of real deployments and not a substitute for a live quote.