Indic Tokenisation and Why It Costs You

An SI sized a four-GPU node from an English tokens-per-second slide. Production was 70 percent Hindi and Odia, with the rest code-mix. The context window filled on documents that looked short to the officer. Retrieval chopped words in the middle of a conjunct. The bill, where a hosted fallback still existed, was a multiple of the pilot. The steering committee was told the users were 'verbose'. The users were speaking normally. The tokeniser was English-shaped.

This teardown is for the integrator who has to explain why the Hindi desk needs more silicon than the English demo. It is not a paper on byte-pair encoding. It is the operational meaning of tokenisation for Indian public text.

We will not publish a universal multiplier. Different checkpoints, different tokenisers, different scripts. A blog that says 'Hindi is 2.7 times English' is performing science. Sit the count on your corpus and your hash.

Not a quote. Not a rate card. Measure, then size.

What a token is doing to you

A token is the atom the model bills, caches, and stuffs into a fixed window. English-heavy tokenisers often split Indic words into more atoms because the merge table saw less of that script. Morphologically rich words and conjuncts become shards. You pay per shard.

The window is a hard political object. Officers want the whole circular in context. If the Hindi circular tokenises fat, you either drop pages, summarise, or buy a longer-context model that may not fit the rack. That choice should be in the design note, not in a midnight config.

Retrieval chunkers that split on whitespace or on a token budget will cut inside an Indic word more often. The chunk the agent sees is then an injured fragment. Relevance drops. The officer says the model is weak. The chunker is weak.

Measure on the real mix, not on a novel

Take 100 real tickets, 20 circulars, and 10 forms in each script you claim. Run the exact tokeniser of the hash you will deploy. Record tokens per character, tokens per word as a clerk would count words, and tokens per document. Do the same for an English paraphrase of a sample. The ratio is your local tax. It is not a law of nature.

Code-mix will not sit neatly between the two monolingual ratios. A Hinglish SMS can tokenise worse than either. Roman-script Indian languages can look 'cheap' in tokens and still be semantically hard. Do not use token cheapness as a quality score.

Numerals, Aadhaar groups, and file numbers can explode into surprising token counts depending on spaces and Unicode blocks. That is another reason the numeral contract belongs upstream of the model.

What to log in the sizing file. Fill with your runs. We will not pre-fill multipliers.

Corpus slice	What you log	Decision it feeds
Citizen SMS / chat, per language	Tokens per message, p50 and p95	Concurrency and latency SLO
Scheme circulars	Tokens per page, pages that exceed window	Chunking and summary policy
Uploaded PDFs	OCR text tokens vs original estimate	Document pipeline cost
Code-mix	Ratio vs monolingual Hindi and English	Whether mix is its own eval row
Tool traces + retrieval	Tokens that never reach the citizen	Hidden cost of agent steps

What you can change without a new religion

Pick a checkpoint whose tokeniser was trained with your scripts in mind, then verify. Do not believe the model card's 'multilingual' line without the count.

Normalise Unicode before tokenising. NFC versus NFD can change counts and retrieval matches. Put normalisation in the pipeline you own.

Chunk on sentence or clause boundaries that work for the script, not on 512 English tokens copied from a blog. Overlap enough that a split conjunct has a second chance.

Do not translate to English 'to save tokens' unless you have written the hop, the quality loss, and the sovereignty cost. Cheap tokens that leave the perimeter are not cheap.

Cache what you can: unchanged circulars, unchanged scheme FAQs. Indic tax on a static circular should be paid once.

How this hits the rack and the bill

Tokens per second on a GPU slide is usually measured on English. Your effective citizen-per-second is that number divided by your local tax, further divided by retrieval and tool overhead. Size from the effective number.

If you use a billed API for any hop, ask for the tokeniser documentation and run the same pack. Vendors sometimes count tokens differently for Indic scripts. The contract should say whose counter wins.

We still will not print a rupee per 1,000 tokens. Take the live rate card if you are on one. On-prem, the cost is utilisation and power — see the idle-cluster and TCO pieces.

Two sizing errors

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

This is a research detail. Officers do not care.

Officers care about latency and about dropped circulars. Tokenisation is how those failures arrive. Translate it for them: 'the Hindi file uses more of the machine's short-term memory'.

A 128k context model makes this obsolete.

Longer context is slower and fatter. It also still chunks badly if you stuff injured fragments. Long context is a tool, not an absolution.

We will just buy more GPUs.

Sometimes you must. Measure first. Many files need chunking and caching more than they need another card.

Give us the multiplier for Hindi.

No. Run the pack on your hash. We will sit that run with you. We will not tattoo a fake constant on your finance note.

Ten days to a token tax you can file

This is a script and a spreadsheet, not a research project.

1. Day 1–2: export the corpus slices in the table, de-identified.
2. Day 3–4: run the production tokeniser. Log p50 and p95. Do not use a different tokenizer from a laptop notebook.
3. Day 5–6: run the same on English paraphrases for a sample. Compute local ratios. Mark pages that explode the window.
4. Day 7–8: change one chunker setting and one cache. Re-measure. Keep the win only if eval quality does not drop.
5. Day 9–10: file the numbers next to the GPU count. If they disagree with the vendor slide, the slide loses.

File note you can paste

Subject: Tokenisation measurements for [workflow] — sizing input.

The attached pack was run on model hash [insert] and its tokeniser on [date]. Tokens per document for the production language mix are reported at p50 and p95. No universal Indic multiplier is adopted. Sizing and any API bills will use these counts. Translation hops to reduce tokens are not approved by this note.

This is not a commercial quote.

What we will count in front of you

Prcept AI will run your pack on the hash in your rack and leave the counts on your disk. We will not hide an English-only throughput slide in the annexure. If our tokeniser is worse than a competitor's on your Odia circulars, that is a fair fail.

Sovereignty includes knowing why the window filled. 'The users are verbose' is not an explanation.

This article is informational field guidance for Indian public institutions, not legal, language-policy, procurement, finance or engineering advice. Confirm against the live Gazette, Official Languages Act and Rules, your State's official-language law, MeitY / IndiaAI notices, GFR, GeM terms, DPDP text, departmental manuals and your counsel before you file it.

How to test this with real speech, not staff English

“Indic Tokenisation and Why It Costs You” fails in the field if you only tested officers. A P4 System Integrator should hear a first-generation student, a rural caller, or a Hinglish grievance before claiming “Indic tokenisation cost”.

The same grievance in Hindi can consume well more tokens than the English paraphrase. You pay in context window, latency, retrieval fragmentation and — if you are on a billed API — money. Tokenisers are not neutral. Measure yours. Twenty-two scheduled languages is a Constitution fact, not a model fact. Script support is not language support. Official language rules may require bilingual output even when the model prefers one script.

• Name the languages and scripts in the eval set.
• Include code-mix and scheme-name tests.
• Measure comprehension, not BLEU alone.
• Design a human fallback when language fails.

Close this loop before the next CAB

Put “Indic Tokenisation and Why It Costs You” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 System Integrator, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “Indic tokenisation cost” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.