Speech Recognition Accuracy Across Dialects

The integrator played a Delhi studio clip. The recogniser was tidy. The steering committee nodded. The same week, a caller from a district two states away said a crop name the studio clip had never heard, on a handset that compressed the vowels, with a television on. The transcript was an English scheme the caller had not asked about.

We are not going to invent the size of that gap. Anyone who quotes a single Indic ASR percentage as if it were a government fact is selling. The useful work is to measure your gap and to decide what the agent does when the gap appears.

This data-study is a method. It has no leaderboard. It has a sampling plan, a split you should report, and a refusal to let studio audio stand in for a dialect.

What actually moves error on a public helpline

Dialect and accent. Code-mix inside a spoken turn. Scheme nicknames. Place names. Handset and codec. Noise. Barge-in. People who speak far from the mic. People who repeat because the IVR was slow. Children or relatives speaking for the beneficiary.

If your report does not split at least dialect-or-region, device class, and noise, you do not have a dialect study. You have an average that will flatter the capital.

A sampling plan you can run without a lab

Draw from real calls, with a lawful basis and a listening-access list. Stratify by the regions you advertise. Include the worst lines, not only the intelligible ones. Keep a studio pack if you wish, as a ceiling, not as the score.

Transcribe gold with two listeners who know the dialect. A single listener from another region will 'correct' the caller into the prestige dialect and punish the model for hearing the truth — or the reverse.

Score word error if you want a continuous number. Also score whether operative tokens survived: scheme name, village, identifier, yes/no on consent. Operative token error is what the file should care about.

Report these cells. Leave a cell empty if you have no sample — do not fill it with the studio number.

Cell	What goes in	What you must not do
Region / dialect	Your advertised catchment, labelled as callers label it	Collapse all Hindi into one cell
Device	Feature phone vs smartphone, if you can see it	Use only the vendor's headset clips
Noise	Field vs indoor, tagged by the listener	Denoise before scoring and forget to say so
Operative tokens	Names, places, schemes, yes/no	Hide a token fail inside a mild word-error average

What the agent does with the gap

Measurement without behaviour is a paper. When operative tokens are unsure, the agent reads back, offers a short menu, or transfers. It does not guess a village because the phonetics were close.

If one advertised dialect is far worse than the others, either staff humans for that geography, narrow the voice channel, or stop advertising that dialect. A dashboard that hides the cell is a political fact waiting for a newspaper.

What we will not claim

We will not claim a national word-error rate for Hindi, Tamil, or 'Indic speech'. We will not reprint a vendor's percentage as a finding. We will not treat a BHASHINI or any other public demo as your dialect result. Public missions can be part of a stack. They are not your sample.

If someone in the room says 'industry standard is X percent', ask for the set. If there is no set, there is no standard.

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

We cannot collect dialect audio; it is too sensitive.

Then you cannot honestly claim dialect coverage. Run a smaller consented sample or keep voice out of that geography. Sensitivity is not a reason to reuse a studio clip.

Splitting by dialect will look like we are profiling communities.

You are measuring whether a public service works where you said it would. Report region of the call, not caste. Do not build a speaker-identity system and call it a dialect eval.

One model must serve the whole state.

Then the worst cell is your number. Averages are how capitals buy tools that districts cannot use.

We will fine-tune on the failing dialect and close the gap.

Good. Re-measure on a held-out slice of that dialect, on the same handsets. Fine-tune is a hypothesis until the cell moves.

Do not invent a percentage

There will be pressure to put a single word-error number in the RFP because other RFPs have numbers. Resist it unless the number is tied to your sample and your path. A copied percentage is a fiction that vendors will happily 'meet' on studio audio.

If you need a number at all, make it operational: share of operative tokens confirmed, share of calls transferred on low confidence, share of advertised hours actually staffed. Those numbers describe a service. A studio word-error does not.

A month to a dialect gap you can show

This is a measurement sprint, not a speech lab.

1. Week 1: map advertised languages and regions to the actual call volume.
2. Week 1: write the lawful basis and the listening-access list.
3. Week 2: draw a stratified sample. Include bad lines.
4. Week 2: double-listen gold with people who know the dialect.
5. Week 3: score the production path. Fill the cells. Leave empties empty.
6. Week 3: score operative tokens separately.
7. Week 4: write the behaviour — read-back, menu, transfer — for weak cells.
8. Week 4: decide whether to keep advertising the weak dialect on voice.

How this shows up in the file

The note should say: we have not adopted a national ASR percentage. We measured the production path on the attached sample, split by region, device and noise. Weak cells will transfer, not guess. Studio numbers are not in the selection table.

Attach the sampling plan, the access list, and the behaviour for low confidence.

This article is informational field guidance for Indian public institutions, not legal, procurement, security-accreditation, linguistics or engineering advice. Confirm against the current Gazette, Official Languages Act and Rules, state official-language law, GIGW, RPwD Act, DPDP text and Rules, CERT-In directions, departmental manual and your counsel before you file it.

How to test this with real speech, not staff English

“Speech Recognition Accuracy Across Dialects” fails in the field if you only tested officers. A P4 Programme/Implementation should hear a first-generation student, a rural caller, or a Hinglish grievance before claiming “ASR dialect accuracy India”.

There is no honest single ASR accuracy number for India. There is a gap between the dialect you demoed and the dialect that calls after lunch. Measure that gap on your lines. Twenty-two scheduled languages is a Constitution fact, not a model fact. Script support is not language support. Official language rules may require bilingual output even when the model prefers one script.

• Name the languages and scripts in the eval set.
• Include code-mix and scheme-name tests.
• Measure comprehension, not BLEU alone.
• Design a human fallback when language fails.

Close this loop before the next CAB

Put “Speech Recognition Accuracy Across Dialects” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 Programme/Implementation, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “ASR dialect accuracy India” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.

What must be true before you file this

If “Speech Recognition Accuracy Across Dialects” is only a heading, it will not survive a file inspection. A P4 Programme/Implementation should be able to attach one artefact that proves “ASR dialect accuracy India”: a log export, a clause, a scored row, a dated notice, or a refusal rule.

Write three dated sentences: what was decided, who owns it, and when it will be re-checked. Unsigned sentences are souvenirs. Dated sentences are controls.

• Name the owner of “ASR dialect accuracy India” inside the institution.
• Attach one artefact a stranger can open next year.
• Revisit when the model, the notice, or the SI changes.
• Do not treat a vendor slide as evidence.

What the next file must contain

“Speech Recognition Accuracy Across Dialects” earns a line in the noting only if a P4 Programme/Implementation can attach proof of “ASR dialect accuracy India.” A heading is not proof. A vendor slide is not proof. A workshop photograph is not proof.

Write three dated sentences: what was decided, who owns it after the next posting order, and when it will be re-checked. If you cannot write the three sentences, you are not ready to buy, to sell, or to go live.

Leave unsourced percentages out of the note. DPDP is not a blanket localisation statute. The November 2025 AI governance text is guidance, not an Act. CERT-In’s 28 April 2022 directions still set specified incident and log clocks. A PAC, when lawful, lives in GFR Rule 166.

• Name the designation that owns “ASR dialect accuracy India.”
• Attach one artefact a stranger can open next year.
• Record the instrument you are actually using.
• Revisit when the model, the SI, the notice or the posting changes.