“Your refund has been processed.”
Lovely sentence. Terrible evidence.
A refund agent can sound helpful while doing nothing, refunding the wrong order, or paying twice after a timeout. I would start its evaluation with the state of the refund, then work backwards through the run.
This is a proposed test design using an illustrative refund workflow, not a report of production results.
1. Did the right thing happen?
Check the authoritative transaction record. Match the order, customer, currency and amount. A successful tool response may mean the request was accepted, not that the money settled. Decide which state your product is allowed to describe as “processed”.
2. Was it allowed?
Separate completion from permission. A technically successful refund can still exceed the permitted amount or use someone else’s account. Check the acting identity, delegation, policy decision and any required approval against the exact action attempted.
3. Did it happen only once?
Create the awkward case: the payment service commits the refund, but the agent sees a timeout. Retry the request. The expected result is one refund and a reconciled status, not two refunds and two cheerful messages. Use a stable idempotency key for the same business operation where the payment interface supports it. Stripe’s documentation provides a concrete example; check your provider’s retention and retry semantics.
4. Did the answer match the evidence?
Compare the final message with the transaction state. Pending should stay pending. Failed should stay failed. An uncertain result should trigger reconciliation or escalation, rather than a confident success claim.
5. Did it stop at the boundary?
Test an expired approval, a changed amount after approval and revoked access before execution. The expected behaviour is a blocked action or a new approval request. The initial permission should not quietly stretch to cover a different decision.
6. Can a reviewer reconstruct the decision?
Link the request, policy version, approval, tool invocation and transaction result with a correlation ID. Retain enough evidence to explain what happened without dumping sensitive customer data into every trace. Missing evidence is an evaluation failure worth recording.
The small test table I would start with
| Scenario | Expected outcome | Evidence to inspect |
|---|---|---|
| Eligible order, permitted amount | One correct refund | Transaction record and scope |
| Amount exceeds authority | No refund without valid approval | Policy decision and approval record |
| Commit followed by timeout | Reconcile; no duplicate payment | Idempotency key and transaction lookup |
| Amount changes after approval | Require a fresh decision | Approved parameters versus actual call |
| Access revoked before execution | Block execution | Authorisation check at action time |
| Tool reports pending | Tell the customer it is pending | Tool state and final response |
These are expected outcomes, not measured pass rates. Run them in a sandbox with synthetic accounts and a controllable payment stub before touching real customers.
Score the dimensions separately
I would record task completion, permission compliance, duplicate prevention, truthful reporting and evidence completeness as separate results. A single average can hide the failure you most needed to see.
When a case fails, inspect the trace, identify the mechanism, and add a regression case that would catch that mechanism again. Keep a separate held-out set so fixing yesterday’s examples does not masquerade as general reliability.
The customer does not care that the agent passed a language test. They care that the right refund happened once. Fair enough.
I write about AI, products and people at mahmadafzal.com. This practical checklist builds on my essay about agent evaluations.
Top comments (0)