Acceptance Testing Criteria for AI Deliverables

UAT was scheduled for Friday because the inauguration was Monday. Officers clicked through six happy paths the SI had bookmarked. Someone asked a genuine question from last week's file. The agent retrieved the wrong scheme and sounded sure. The room laughed. The minutes said user acceptance completed with minor observations. The final invoice went in on Tuesday.

Acceptance for an agent is not a user liking a UI. It is a decision that the department will now process real work — sometimes personal data — through a system that can retrieve, draft and call tools. If your criteria cannot fail, your payment milestone cannot protect you.

This template is for technical committees and programme owners writing the acceptance schedule. It is not a laboratory protocol. It is the minimum a later auditor should find behind the word accepted.

Five gates, all of them written

Functional gate: each year-one workflow card has a hold-out set the department owns. Score the published metric. Include items the agent must refuse. A demo script written by the SI is not a hold-out set.

Tool gate: force a tool failure and a dry-run. See whether the officer gets a classified error or a fluent shrug. See whether a retry would double-write.

Control gate: prove the human gate cannot be switched off in config by the vendor's standing account. Prove identity is the department's IdP.

Isolation and logs gate: run the no-undeclared-egress check you put in the bid. Export traces into the store you will actually use. If CERT-In duties apply to you, this is also how you show you can keep ICT logs.

Exit rehearsal gate: load the promised export of prompts, evals and (if any) adapters on department hardware, even if you do not plan to leave. Acceptance is the last moment you have money behind a ZIP file.

Acceptance rows you can paste into the schedule.

Gate	Artefact	Who signs	Fail means
Hold-out evaluation	Scored sheet on department-owned items	Indentor + independent member	No go-live; fix or reduce workflow
Refusal items	List of must-not-answer / must-not-act	DPO / programme owner	No production personal data
Tool failure drill	Minutes + screenshots of classified error	MIS owner	Write tools stay disabled
Gate immutability	Attempt to disable gate is logged and fails	CISO nominee	No write-back
Egress / licence isolation	Capture or allow-list audit on your VLAN	CISO	No acceptance of isolation claim
Log export	File arrives in SIEM in named format	SOC / records	Hold payment on that milestone
Exit load	Export loads without vendor laptop	Technical chair	Hold last tranche

Who is the user in UAT

Not the SI. Not the vendor engineer. Not only the champion officer who has lived with the PoC. Include one officer who was not in the workshops. Include the records or SOC person for the log gate. Include the DPO for refusal items. A single happy user is how inauguration theatre becomes a certificate.

Give them a script of tasks, not a tour. Tasks come from the workflow cards. If they cannot complete the task without the engineer, you have accepted a service bureau, not a departmental tool.

Payment, defects, and conditional acceptance

Do not accept with a laundry list of defects that include isolation, gates or log export. Those are not punch-list paint. They are the product.

Conditional acceptance is allowed if you write the conditions, the date, and what happens to payment if the date is missed. An open-ended accepted subject to observations is accepted.

Keep a portion of the implementation fee behind the exit-load gate even if functional UAT passed. Vendors who cannot load the export after they have been paid will not load it later.

What UAT is not

It is not a press note. It is not a comparison with a public chatbot. It is not a training attendance sheet. It is not the PoC all over again on production data. If you never ran a clean PoC, do not launder that failure through UAT.

Objections you will hear — and what to do with them

These are the lines that stall the file. Answer them in the room, then put the answer in the note. A spoken answer without paper will be forgotten by the next officer.

Hold-out sets will delay political go-live.

Then reduce the workflow, do not reduce the truth. A smaller accepted card is better than a large unaccepted one with a ribbon.

We cannot write gold answers. That is why we bought AI.

If you cannot recognise a good draft, you cannot supervise the agent. Build a modest gold set from last month's real (redacted) work. That labour is the department's, not a vendor favour.

The vendor says UAT on synthetic data is meaningless.

Synthetic or redacted data is what you use until acceptance. Production personal data is the prize after the gates pass. Do not reverse the order.

Independent members do not understand AI.

They understand missing files, disabled gates, and empty SIEMs. That is most of this UAT.

Build the UAT pack while the bid is still open

1. During bid period: assemble hold-out items and refusal items per card. Keep them off the bidders.
2. At kickoff: share the task list, not the gold answers.
3. Before UAT week: run a dry isolation and log-export rehearsal so UAT is not the first time the cable is pulled.
4. UAT week: five gates, written fails, no inauguration inside the sitting.
5. After sign-off: release the milestone that matches the gates you actually passed.

How this shows up in the file

Subject: Acceptance schedule for the agent platform.

Acceptance requires the five gates in the annexure: hold-out score, refusal behaviour, tool-failure drill, gate immutability and identity, isolation and log export, and a load test of the exit package. Conditional acceptance, if any, will state dates and payment consequences. Inauguration is not a gate.

This note is not legal advice.

Shadow mode is not acceptance

Running the agent beside the human process for a fortnight is useful. It is not UAT. Shadow mode has no payment consequence, no failed gate, and no incentive for the vendor to surface the ugly tool path. Officers treat it as a toy. The SI treats it as a rehearsal they already passed.

If you want a shadow period, write it as a named phase with a start, an end, and a rule: no production write-back, no production personal data unless the processing schedule already allows it, and scores from the shadow week do not replace the sealed hold-out. The hold-out is the exam. Shadow is extra homework.

After a failed gate, do not rename the failure as extended shadow. That sentence is how inauguration week swallows the schedule. Either the gate passes, or the workflow shrinks, or the milestone stays unpaid. Three honest options. A fourth option called we will watch it in production is how the municipal bye-law story repeats.

This article is a field guide for Indian public buyers, not legal, procurement, financial or audit advice. Confirm every citation against the live GFR compilation on doe.gov.in, the relevant DoE procurement manual, GeM terms, CVC guidance and your own counsel before a sentence enters a tender file.

How to put this in the RFP, not the preamble

A P2 Procurement who searches “UAT criteria AI project” is usually drafting or scoring a bid. “Acceptance Testing Criteria for AI Deliverables” belongs in eligibility, the evaluation matrix, or a numbered annexure. If it only lives in the covering note, L1 will ignore it.

Acceptance is not a photo of the secretary with a chatbot. It is a scored hold-out set, a failed-tool drill, a gate drill, an isolation drill, and a log export that your SIEM can read. QCBS weights are a choice you must publish before opening. Accuracy is a task plus a dataset, not a slogan. SLAs for agents must name tool-calls, human gates and log export — uptime alone is a hosting metric.

Do not let a vendor write the specification and then bid on it. Record unsolicited proposals. Pay for pilots that touch personal data. Write exit before you write go-live.

1. Move the control from the preamble into a scored or eligibility row.
2. Attach a one-page definition (accuracy, SLA, language, data handling).
3. Require an artefact in the technical bid, not a slide.
4. Extend the bid date if a corrigendum is material.
5. Minute the demo on your data, offline if you claimed air-gap.

Close this loop before the next CAB

Put “Acceptance Testing Criteria for AI Deliverables” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P2 Procurement, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “UAT criteria AI project” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.