Bias Testing on Indian Demographic Data

A social-welfare department showed the minister a slide: 'bias score 0.97, industry leading'. The eval set was a public English benchmark about US résumés. The department's actual work was scanned income certificates in two scripts, self-help-group minutes in a third, and names that the tokeniser split into nonsense. The first month of production, rejection drafts clustered on one coastal district whose certificates used a local stamp the corpus had never seen. The slide had been accurate about a country they do not administer.

Bias testing for an Indian public desk is not a moral accessory. It is quality control on the population you already serve. There is no statutory test to hide behind. That fact cuts both ways: you cannot claim compliance with a Gazette method that does not exist, and you cannot refuse to test because no one forced a template.

Guide for DPOs and eval owners. 16 August 2026. Not legal advice. Not a claim that a test equalises society. A claim that you should know where your machine is clumsy.

Honesty about the law

The Constitution's equality guarantees and a stack of welfare statutes still apply to the department. DPDP will apply to personal data in the eval set. None of those instruments, as they stand today, prescribe an AI bias metric, a pass mark, or a certified laboratory. If a sector regulator later does — RBI, IRDAI, a health body — file that circular and obey it. Until then, write 'departmental eval' on the cover, not 'statutory clearance'.

MeitY's 5 November 2025 guidelines mention inclusion and responsible practice. Bhashini exists to make Indian languages more usable. Use those programmes as resources. Do not cite them as a certificate that your agent is fair.

Slice what you serve

Build the pack from last year's tickets, redacted. Borrowing a US leaderboard is not a slice.

Slice	Where to get it	What failure looks like
Languages and scripts of the desk	Last 200 tickets, stratified	Refuse-or-hallucinate rate jumps when the script changes
Document type	Income, caste (if you must hold it), disability, death, land	One stamp or one table layout causes empty retrieval
District or circle	Volume-weighted sample	One geography's names or seals are unknown to the corpus
Gendered honorifics and names	Redacted names from your own files	Drafts systematically more polite or more punitive
Literacy / formality	Voice-to-text, informal WhatsApp-style, formal noting	Only the secretariat register English survives
Empty-retrieval cases	Questions you know are out of corpus	The model invents a clause instead of refusing

Measure what the desk actually does. If the agent only drafts, measure draft quality and refusal quality, not 'who got the scholarship'. If a human still signs, the human's pattern is a separate study. Do not give the machine credit or blame for a signature it did not make — unless you discover the human is rubber-stamping, which is an escalation-design failure, not a benchmark.

Limits you should write down

Small samples lie. Intersectionality explodes the cells. Historical files encode historical injustice; a model that imitates them will look 'accurate'. You will not have caste-religion-disability-district cells large enough to be statistically pretty. Say so on the card. Prefer simple, inspectable failures (empty retrieval should refuse; language X should not be silently worse) over a single fairness index that no officer can explain to CAG.

Do not scrape a communal dataset to look serious. Do not put sealed eval data back into training. Do not let a vendor 'improve' on your demographic labels in their cloud. Prcept will not train on them. If a competitor needs them, they need a processor clause and a deletion date.

Two evals

Objections

Scientists want a published metric. Answer: publish cells and failures. A single number that cannot be walked backwards is how the 0.97 slide is born.

Legal wants no written failures. Answer: then you will discover them in a newspaper. Write them on the card.

A vendor offers a global bias certificate. Answer: ask which of your languages and document types were in their set. If the answer is none, the certificate is a poster.

A unit says testing on real tickets violates DPDP. Answer: then redact, purpose-limit, and keep the set small — do not skip the test. Skipping is how one district pays.

A four-week playbook

• Week 1: list languages, document types and districts you actually serve. That list is the eval design.
• Week 2: build a redacted pack with empty-retrieval traps. DPO signs the purpose and the destruction date.
• Week 3: run the pack. Report cells, not an index. Fail the hash if a served language collapses or empty retrieval invents law.
• Week 4: write residual risk on the card. Schedule a re-run on the next model or corpus change.

File note you can paste

Subject: Departmental bias and quality eval for agents — no statutory method claimed.

This department has not located a Gazette protocol that certifies an AI system as unbiased. We will nevertheless evaluate each citizen-facing workflow on the languages, document types and geographies we serve, including empty-retrieval behaviour. Results will be reported cell-by-cell. A single composite 'bias score' will not be used as a clearance.

Eval sets will be redacted, purpose-limited and destroyed on a dated schedule. They will not be used to train a vendor model. Residual risk will be written on the model card. This note is not legal advice.

What to tell the minister instead of a score

Say: we serve these languages; on the last pack, language B still invents clause numbers when retrieval is thin, so we refuse empty retrieval; district C's stamp is now in the corpus; we have not tested caste fairness as a production feature and will not; residual risk is written on the card; here is the grievance count from that district. That briefing is longer than 0.97 and shorter than a scandal.

If the minister wants a national fairness claim, refuse the sentence. You do not have the cells. A false national claim will be quoted back to you by a journalist who found one coastal stamp. Humility is not weakness on this file. It is the only sentence that stays true.

Share the pack with internal audit as a control artefact, not as a press kit. IA should check that the pack matches the languages on the register, that empty-retrieval items exist, and that a failed hash was not shipped. They should not be asked to certify fairness. Nobody should.

When a new district is added to a scheme, you have a new slice. Do not wait for the annual eval. Ten real tickets from the new district, redacted, run before the local launch, will catch the stamp problem cheaper than a month of clustered rejections.

• Never buy a 'bias certificate' that cannot name your document types.
• Never put sealed eval labels back into a vendor cloud.
• Never let a single English-heavy cell report the ministry's conscience.

Informational field guidance. There is no Indian statutory AI bias test to cite. Confirm DPDP duties on eval data and any later sector circular with counsel.

How this survives CAG, RTI or the Board

“Bias Testing on Indian Demographic Data” is not a workshop slide. A P6 Compliance/DPO will have to reconstruct a decision after the officer who clicked approve has been transferred. Write the artefact that lets a stranger replay the case: the log fields, the approval, the override, the register row.

A leaderboard score on English internet text will not tell you how the agent treats a scanned Odiya income certificate. Test the slices you serve. Publish the failures. Do not invent a statutory test MeitY never wrote. India AI Governance Guidelines (November 2025) are guidelines, not a statute. DPDP still allocates fiduciary duty. Delegation of Financial Powers still allocates who may spend. Do not hide those instruments behind the word governance.

If you cannot show who acted, on which purpose, with which data class, and who could have refused, you do not have accountability. You have a chatbot with a charter PDF.

• Name the owner of “AI bias testing India” inside the department, not the vendor.
• Keep CERT-In-relevant logs in India for the required period.
• Store overrides with a reason an auditor can read.
• Put the workflow on the AI register before it touches a citizen.

Close this loop before the next CAB

Put “Bias Testing on Indian Demographic Data” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P6 Compliance/DPO, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “AI bias testing India” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.