“The AI should answer accurately.”
Great. The food should also taste nice. We still need to know what we're ordering.
A polished demo can make an AI feature feel finished before the team has agreed on what success means.
Useful acceptance criteria describe the behaviour, context, evidence and failure boundaries—not just how convincing the answer sounds.
Let's use an illustrative support assistant for a small online shop. This is a proposed evaluation design, not a claim about a deployed product.
Start with the customer's task
The customer wants to understand a return policy or find out where an order is.
Those tasks require different capabilities:
- A policy answer may be grounded in approved documents.
- An order-status answer needs current order data and an authorised identity.
- A refund changes business state and may require a separate approval process.
Putting all three behind one chat box does not make them the same requirement.
“It's just a chatbot” has done some heavy lifting in software meetings.
Replace adjectives with observable checks
| Vague requirement | More useful proposed criterion |
|---|---|
| Give accurate policy answers | Use the current approved policy and identify the relevant source; do not invent an exception |
| Handle missing information | Ask for the missing detail or explain the limitation rather than guessing |
| Respect privacy | Do not reveal an order's information when the requesting user is not authorised to access it |
| Escalate when needed | Offer the defined human handoff when the task is outside the assistant's permitted scope |
| Process refunds | Prepare the request but do not execute a refund without the required approval |
These are still a starting point. “Current approved policy” needs a source owner, version and update process. “Human handoff” needs a destination that someone actually monitors.
Otherwise, escalation is just the bot politely placing your problem in a decorative cupboard.
Build a small evaluation set before arguing about the model
Start with a deliberately limited set of synthetic examples that cover different failure modes:
- A straightforward question with an answer in the policy.
- A question whose answer is absent.
- An ambiguous request that needs clarification.
- A request involving another customer's information.
- A stale document conflicting with an approved update.
- A request to ignore the policy or execute an unauthorised action.
For each case, write down the acceptable outcome and unacceptable behaviour before generating the response. Keep some representative cases separate from prompt development so you are not only checking examples the team has tuned against.
Anthropic's agent-evaluation guide is useful reading on evaluation structure and different kinds of graders. Our shop-assistant cases are an original illustrative test design, not results from that guide.
Use several checks, not one impressive percentage
A response can be fluent and still expose private information. It can quote a policy correctly while acting on the wrong order.
Track different dimensions separately:
Answer quality: Is the response supported by the approved source?
Task behaviour: Did the assistant ask for clarification or offer a handoff when required?
Action safety: Did it stay within its tool permissions and approval boundaries?
Operational behaviour: What happened to latency, timeouts and provider failures under the tested conditions?
A small test set cannot establish a precise population-wide failure rate. Report counts, cases and limitations rather than dressing a handful of examples up as scientific certainty.
Make action boundaries enforceable
If the assistant is not permitted to issue a refund, don't rely only on “never issue refunds” in its prompt.
The surrounding tools and authorisation layer should enforce what it can do. OWASP's AI Agent Security guidance discusses this separation between agent behaviour and system controls.
For the shop example, the assistant could prepare a structured refund request. An authorised workflow validates the order, amount, eligibility and approval before executing anything.
Same customer conversation. Very different risk boundary.
Define the release decision
The team should decide in advance which failures block a release, which require restricted availability and which can be monitored with a named follow-up owner.
Document the model identifier, prompt version, source snapshot, tool configuration and evaluation conditions. Reassess when those inputs materially change.
The Technical PM doesn't need to pretend to be the model researcher. The job is to make sure product intent, engineering behaviour, security boundaries and release evidence describe the same feature.
If you could add one acceptance criterion to an AI feature tomorrow, would it concern answer quality, privacy, escalation or tool permissions? Why that one?
Draft prepared with AI support. All scenarios and evaluation criteria are illustrative; no benchmark result is claimed.
Top comments (0)