Evaluating Vendor Demos: A Scoring Sheet

A committee in a coastal collectorate watched a twenty-minute demo that drafted a flawless English noting, complete with a polite paragraph on citizen-centricity. There was applause. There was tea. There were no Odia tickets, no denied outbound path, no log that showed which circular had been retrieved, and no moment where an officer refused the draft. The minutes said demonstration successful. The hash that later arrived on a disk was a smaller model. The English noting never came back.

This is a scoring sheet you can print and put on the table before the laptop opens. It is for technical demonstrations that already sit in a tender, and for the unofficial 'just come and show us' meetings that quietly become the specification. The rule is the same. Score the artefact you will buy, on the data you will process, under the network you claimed, with the human gate you will live with.

Not a GFR form and not legal advice. Fit the weights to your published evaluation matrix so you do not invent a new scoring system after seeing who is good at slides.

Before the laptop opens

Freeze the model hash, the language pack, and whether any hosted hop is allowed. Tell the vendor the room's network will be yours. Hand them the public sample of your eval. Keep the held-out slice in an envelope. Appoint a timekeeper. Appoint someone who will press 'reject' on a draft.

If the vendor says the model must download weights at the start, the isolation row is already in trouble. Start the clock anyway. Failure is information.

The sheet

Use the same rows for every bidder. Do not add a charm row. Do not subtract marks because the salesperson was blunt. Blunt and isolated beats charming and hosted.

Print this. Each row is 0 / 1 / 2 unless you published different weights. Zero means the artefact was missing, not that you disliked the font.

Row	2 — pass	0 — fail
Same hash as the bid	Hash on screen matches the sealed bid	A larger or hosted model 'just for demo'
Your held-out set	Marked on items they had not seen	Only their screenshots or English news
Language / code-mix rows	Year-one languages as specified	English with three transliterated names
Isolation	Runs on the room's deny-outbound path	Needs their travel SIM, garden, or CDN
Audit trail	Shows who, tool, data class, why, output	A chat window with no reconstructable log
Human gate	An officer can reject; the reject is logged	Send is the only button
Tool allow-list	Undeclared tool cannot be called	The agent 'helpfully' opens a new API
Failure behaviour	Abstains, escalates, does not invent a scheme	Hallucinated circular, confident tone

How to mark without a fight

Two officers mark independently. A third deals with disagreements on the same afternoon, not a week later when memory has been replaced by the best lunch. Write a one-line reason on every zero. 'Did not like the UI' is not a reason. 'Held-out item 14 invented a scheme' is a reason.

If the published matrix said the demo is only twenty marks, do not let a dazzling demo overturn a failed mandatory isolation row. Mandatory is mandatory. Charm is not a corrigendum.

• Same room conditions for every bidder.
• Held-out set unseen.
• Hash check first, tea later.
• A forced reject and a forced unknown item in every demo.

Unofficial demos still write the file

If a vendor demoed before the RFP, record it as vendor material. Do not let that memory become a hidden scoring row. The published sheet is the only sheet. Officers who 'already know who is good' should still mark zeros when the hash is wrong.

A district NIC lead in Palakkad once kept a private score in a notebook from three hotel demos. When the formal demonstration day arrived, two of those firms sent different engineers and a different hash. The notebook scores were about people, not products. The published sheet, filled the same afternoon, is what the file still holds. Private memory is how a charming visit becomes an undeclared eligibility cut.

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

We will look hostile if we cut the WAN.

You will look like a buyer. A vendor who needs the WAN to show an air-gap product has told you the truth early. That is courtesy, not hostility.

There is no time for held-out marking.

Then do not score language or quality. Score only what you have time to test, and keep the rest as a later pilot gate. Inventing marks from applause is worse than leaving a row unscored.

All demos are theatre, so we should skip them.

Theatre with a sheet is still better than theatre with tea. The sheet is how you stop the minutes from saying successful when the hash was wrong.

The vendor asked for the questions in advance.

Give the public sample. Keep the held-out pack. A vendor who will only perform on known items is asking to be scored on rehearsal.

Score the failure path or you have scored a commercial

Every polished demo is a commercial. The part that is not a commercial is what happens when the input is poor. Put three forced failures on the sheet and refuse to skip them for time. One: a ticket with a missing mandatory field. Two: a request the department does not run as a scheme. Three: a code-mix string from your year-one language row. An agent that invents a ward, invents a scheme, or silently hops to English has told you how it will behave on a public desk at 4 p.m.

Log those three items with the same care as the pretty ones. The minutes should quote the invented scheme name. A later bidder who complains that you were harsh can be shown the quote. Harsh is not the word. Reconstructable is the word.

If the vendor asks to retry after 'a small configuration', allow one retry on the same hash, same path, same items. A retry on a new hash is a new product. Write which attempt you marked. Committees that average the pretty first take and the patched second take are scoring hope.

A demo day that produces a file

Send the sheet with the bid. Do not surprise bidders with secret rows. Surprise them with held-out items, which is the point of held-out items.

1. T-minus 14 days: publish the sheet and the public sample. Freeze hashes in the bid.
2. T-minus 2 days: prepare the room path, the deny rule, the envelope.
3. Demo day: hash, held-out, forced reject, forced unknown, pcap if isolation is claimed.
4. Same afternoon: two marks, one reconciliation, reasons on every zero.
5. Next morning: the sheet goes behind the note, not into a drawer.

How this shows up in the file

Attach the marked sheets, the hash printouts, and the held-out item that failed or passed. A later complaint should be answerable with paper, not with 'the committee was impressed'.

If a row was not tested, write not tested. Do not gift a two.

This article is informational field guidance for Indian public institutions, not legal, procurement, security-accreditation or engineering advice. Confirm against the current Gazette, GFR, GeM term, CVC instruction, CERT-In direction, DPDP text, departmental manual and your counsel before you file it.

How to put this in the RFP, not the preamble

A P2 Procurement who searches “vendor demo evaluation” is usually drafting or scoring a bid. “Evaluating Vendor Demos: A Scoring Sheet” belongs in eligibility, the evaluation matrix, or a numbered annexure. If it only lives in the covering note, L1 will ignore it.

If the demo runs on the vendor's network, the vendor's English set, and the vendor's larger model, you have scored a play. Use this sheet on your set, your isolation, and the hash you will buy. QCBS weights are a choice you must publish before opening. Accuracy is a task plus a dataset, not a slogan. SLAs for agents must name tool-calls, human gates and log export — uptime alone is a hosting metric.

Do not let a vendor write the specification and then bid on it. Record unsolicited proposals. Pay for pilots that touch personal data. Write exit before you write go-live.

1. Move the control from the preamble into a scored or eligibility row.
2. Attach a one-page definition (accuracy, SLA, language, data handling).
3. Require an artefact in the technical bid, not a slide.
4. Extend the bid date if a corrigendum is material.
5. Minute the demo on your data, offline if you claimed air-gap.

Close this loop before the next CAB

Put “Evaluating Vendor Demos: A Scoring Sheet” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P2 Procurement, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “vendor demo evaluation” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.