Reference-Check Questions for AI Vendors

A bid named a Central ministry as a live deployment. The committee was impressed. A section officer in that ministry, reached on a working number, said they had hosted a two-week demo on synthetic tickets in a conference room, after which the vendor sent a case-study PDF. No sanction, no DPA, no production connector, no AMC. The logo was real. The reference was not.

This is a checklist for the unglamorous call that should sit between the technical marks and the commercial opening, or as a condition before LoI. It is not a smear sheet. Good vendors survive it. Inflated logos do not. In a market that still sells pilots as go-lives, the call is part of due diligence, not an insult.

Not legal advice. Do not record people against their organisation's rules. Do not treat a bitter ex-customer as gospel. Two independent facts beat one rant.

Who you call

The named officer in the bid, and one user who is not the officer who signed the case study. If the vendor will only offer a marketing lead, you do not have a reference. You have a speaker bureau.

Stay inside government-to-government courtesy. Ask for fifteen minutes. Send the questions in advance if their vigilance culture requires it. Write a file note after, not a tweet.

The questions

Ask them in this order. The first three decide whether the rest of the call is about a system or about a visit.

Print this. Tick what you heard, not what the brochure said.

Ask	A live system sounds like	A logo sounds like
Was this sanctioned, piloted, or only demonstrated?	A file number, a paid value, a term	'They came and showed the team'
Did production personal data move? Was there a DPA?	Yes, and a named schedule — or a clear no	'Just a trial, not really processing'
Was the architecture the one in our bid (on-prem / air-gap / hosted)?	The same object, or a listed difference	Hosted demo, on-prem promise
Which languages and mixes actually worked?	Named languages, named failures	'It handles Indic'
What broke in the first month?	A specific break and a fix	'Nothing, they were very professional'
Who holds keys and where do logs land?	A named store and a named officer	'The vendor handles security'
Would you buy the same object again?	A conditional yes, or a no with a reason	A long pause, a transfer of the call

What to do with the answers

If the 'deployment' was a demo, strike the experience mark that depended on it. If the architecture differs, treat the reference as a related but different object. If two references cannot name a failure, you may have been routed to a script. Call the user.

Do not disqualify a vendor for a messy first month if the fix is real. Mess is information. Perfection is often rehearsal.

• File number or it is a demo.
• Architecture match or it is a cousin.
• Named failure or it is a script.
• User voice or it is a spokesperson.

Foreign and private references

A bank in another country running a hosted assistant is weak evidence for your air-gapped grievance desk. A private campus helpdesk is weak evidence for a treasury workflow. Comparable data class, comparable isolation, comparable language load — or the reference is colour, not marks.

Ask also how the reference paid. A complimentary campus sandbox, a CSR-labelled visit, and a sanctioned GeM order are three different objects. If the cited body cannot name a file number or a DPA, write demo on the sheet even if the logo is famous. Famous is not comparable.

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

Vendors will only give happy names.

Yes. That is why you ask for a user, ask for a failure, and treat architecture mismatch as a miss. Happy names still leak facts if you ask about DPAs and hashes.

We cannot call a ministry informally.

Then write a formal request, or ask the vendor to set a recorded joint call. Informality is optional. The questions are not.

Startups will have no government references.

Then do not write a three-ministry eligibility cut. Accept comparable isolation work, paid private work with a DPA, or a paid pilot as the first reference you will help them earn. Do not invent logos, and do not punish honesty.

Reference checks delay L1.

They delay a bad LoI. Put them on the calendar when you publish the bid, not as an afterthought when someone smells a PDF.

What the call note must contain so it is evidence

A later complaint will say you were unfair to a logo. The only defence is a note that could have been written by someone who was not in the room. Date and time. Officers on the line, with designations. The number you dialled and how you got it — bid annexure, ministry directory, vendor-arranged bridge. The seven answers in the cited officer's words, not in yours. The one-line conclusion. Whether a user was reached. Whether a file number was offered. Whether the architecture matched. That page is dull. Dull is how you keep the mark.

If the vendor sits on the call, say so. Their presence changes what a civil servant will admit. Prefer a government-to-government line. If that is impossible, record that the vendor arranged the bridge and treat glowing adjectives as weaker evidence. A user who names a tokenizer failure in front of the vendor is unusually credible. A user who says 'they were very professional' on a vendor-hosted line is a brochure.

Do not put raw personal data from the reference department into your file. You are verifying a buy, not collecting their grievances. File numbers, architecture words, and 'production data moved / did not' are enough. If they email you a case-study PDF you already have, do not treat the email as a new fact.

• Write what was said the same day. Memory becomes marketing overnight.
• If two references contradict the bid's architecture sentence, reopen technical marks.
• If nobody will take the call, write could not verify — that is a mark, not a void.
• Never let the vendor draft your call note 'for convenience'.

Five days of calls that change marks

Two officers, three references per shortlisted bidder, one page per call. That is enough.

1. Day 1: extract names, numbers, claimed architecture, claimed data classes.
2. Day 2–3: make the calls. Write what was said, not what you wished was said.
3. Day 4: compare across bidders. Strike marks that sat on demos.
4. Day 5: chair reviews the strikes. File the pages.

How this shows up in the file

Each call note has the date, the officers on the line, the seven questions, and a one-line conclusion: live comparable system / live different system / demo / could not verify. Attach it to the evaluation sheet.

If you did not call, write that you accepted a logo. That sentence is often enough to make someone pick up the phone.

This article is informational field guidance for Indian public institutions, not legal, procurement, security-accreditation or engineering advice. Confirm against the current Gazette, GFR, GeM term, CVC instruction, CERT-In direction, DPDP text, departmental manual and your counsel before you file it.

How to put this in the RFP, not the preamble

A P2 Procurement who searches “AI vendor reference check” is usually drafting or scoring a bid. “Reference-Check Questions for AI Vendors” belongs in eligibility, the evaluation matrix, or a numbered annexure. If it only lives in the covering note, L1 will ignore it.

A logo is not a reference. Call the officer. Ask if it was a paid production system, what failed, and whether the architecture you are buying is the one they actually ran. QCBS weights are a choice you must publish before opening. Accuracy is a task plus a dataset, not a slogan. SLAs for agents must name tool-calls, human gates and log export — uptime alone is a hosting metric.

Do not let a vendor write the specification and then bid on it. Record unsolicited proposals. Pay for pilots that touch personal data. Write exit before you write go-live.

1. Move the control from the preamble into a scored or eligibility row.
2. Attach a one-page definition (accuracy, SLA, language, data handling).
3. Require an artefact in the technical bid, not a slide.
4. Extend the bid date if a corrigendum is material.
5. Minute the demo on your data, offline if you claimed air-gap.

Close this loop before the next CAB

Put “Reference-Check Questions for AI Vendors” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P2 Procurement, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “AI vendor reference check” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.