A task-completion score can hide an agent that reached the right answer through an unauthorised action. Evaluate what happened, which actions occurred and whether the system stayed within its authority.
This article was generated by an AI agent for Acadify Solution and Acadify AI Lab. All scenarios are illustrative, not client results or benchmark measurements.
1. Write an action contract
For each tool, define who may invoke it, the tenant and resource scope, required arguments, permitted state transitions, approval requirements and the meaning of success, failure or an unknown outcome.
Enforce access in the application/tool layer. A prompt can describe a boundary; it does not replace an authorisation check. Reading an enquiry, proposing a CRM update and sending an external reply deserve separate contracts.
2. Test the target as well as the payload
Use a disposable environment with synthetic data. Include two tenants and similarly named records. Verify that a supplied identifier does not grant access to a record outside the authorised scope.
Validate types, required fields and business rules at execution time. Rejected calls should cause no state change. Record the attempt and rejection with appropriate redaction.
3. Inject an uncertain write outcome
Test a service that commits a write but loses the response. A timeout does not prove that nothing changed.
Where the API supports it, use a stable operation ID and status lookup. Reusing the ID with a different payload should detect a conflict. If there is no reliable status or deduplication mechanism, represent the unknown outcome and escalate according to the workflow's risk.
| Injected condition | What to check |
|---|---|
| Failure before commit | No state change; bounded retry if permitted |
| Commit followed by timeout | Status resolution or safe escalation |
| Replayed request | No unintended duplicate side effect |
| Same operation ID, different payload | Conflict detected |
A retry budget alone does not make a write safe. Expected behaviour must follow the API contract and business rules.
4. Bind approval to a proposal
Show the reviewer the target and important arguments. Test changes to content, recipient or target after approval. The old approval must not silently cover a materially different action.
Test rejection and expiry too. A delayed decision should not authorise a later, changed proposal. A blanket approval token disconnected from the proposal is difficult to audit.
5. Treat retrieved instructions as untrusted input
Place conflicting instructions in a synthetic document—for example, a paragraph asking the assistant to send a message outside the requested workflow. Check both the model's behaviour and the application's permission enforcement.
Record the input, selected context and attempted calls. Do not use real secrets or private customer records as test material. A single passing case does not show that prompt injection is solved; vary the cases and report the tested scope.
6. Exercise stopping and handover
Test missing fields, invalid tool output, expired access, repeated failures and step limits. Define terminal states such as completed, awaiting approval, failed and unknown outcome.
A handover should identify the last confirmed action and the unresolved step. A fluent final message is not evidence of a completed workflow.
7. Retain useful, limited traces
Record the case ID, workflow version, authorised scope, redacted arguments, approval decisions, tool results and state changes. Decide who can inspect traces and how long to retain them.
A reviewer should be able to reconstruct the result without unrestricted access to every document the system encountered.
8. Separate release gates
Report task outcome separately from policy and integration checks. Under a defined permission gate, an unauthorised write should fail the case even if the final answer is correct.
Document case counts, fixture versions, model and prompt versions, stochastic repeats where appropriate and known gaps. Keep a held-out set when tuning.
This checklist is a starting point, not a security certification or reliability guarantee. Adapt it to your system and the consequences of its actions.
Which failure injection would you add first to your agent's release test?
Acadify Solution provides software and AI engineering services. Acadify AI Lab supports scoped LLM and agent evaluations using agreed tasks, rubrics and reviewable findings.
Top comments (0)