The tool returns a timeout. The business action has already happened.
What does your AI application do next?
This is a proposed test scenario for a fictional purchasing assistant. It is not a report of an executed benchmark. Use synthetic fixtures and an isolated tool stub or an approved test environment.
Separate the action from the response
A test double can model two events independently: accepting an operation and delivering the response to the caller. That lets a test deliberately produce a successful side effect with a missing confirmation.
For the fictional assistant, record one synthetic order as accepted, then make the caller observe a timeout. Retain an operation reference so the test can distinguish a repeated request from an unrelated action.
The expected recovery must come from the tool contract. Do not assume every provider offers the same guarantees for repeated requests, status lookups or operation references.
Define what a second attempt means
Before asserting that a retry is correct, specify whether it represents the same business operation. If the receiving system supports a documented duplicate-recognition mechanism, test the application against that contract, including its stated limits.
If the original outcome cannot be established automatically, define the unresolved state and escalation path. The test should not reward an agent for confidently announcing success or failure without supporting evidence.
The model's intention to avoid duplication and the application's enforcement are separate things. Inspect the relevant persisted state and tool activity as well as the assistant's explanation.
Exercise the full recovery sequence
Begin with a known synthetic request. Allow the first side effect, withhold the response, and continue the workflow through the same interface used by the test caller.
Check how the application identifies the unresolved operation. Observe whether it checks status, resubmits, stops or asks for help. Compare each action with the expected contract rather than treating any extra tool call as automatically right or wrong.
After the sequence, count the accepted business actions. A reassuring final answer does not compensate for two synthetic orders when the expected outcome was one.
Add timing cases deliberately
| Case | Behaviour to define before testing |
|---|---|
| Action accepted, response withheld | How the original outcome is established |
| Confirmation arrives late | Which state the late result may update |
| User clicks again while checking | How the same intention is recognised |
| Status lookup is unavailable | When the operation remains unresolved or escalates |
| Worker restarts during recovery | Which operation reference survives the restart |
| User returns in another session | What status and next step are visible |
Select cases relevant to the application. This table is a scoping aid, not a universal implementation specification.
Inspect the language shown to the user
“We could not confirm the result” is different from “the operation failed.” The interface and assistant should reflect what the workflow actually knows.
Check that any promised follow-up exists. If the message says the system is checking status, the test should be able to observe that route. If someone must investigate, the unresolved case needs an appropriate destination and enough evidence to work with.
Do not use real customer orders or messages to test these paths. Isolate the side effects and make the evidence safe to retain.
Report outcomes separately
Record the final action count, status transitions, operation references and user-visible explanation. Include expected versus observed behaviour and the smallest trace needed to reproduce the issue.
Keep deterministic workflow checks separate from repeated evaluations of generative choices. For the latter, retain the relevant configuration and repetitions. A passing run describes that run; it does not establish that an agent will always recover correctly.
Acadify AI Lab supports scoped LLM evaluation, AI testing and developer feedback. Acadify Solution supports custom software and API integrations.
Which part is hardest to test in your workflow: the lost confirmation, the retry decision or the final reconciliation?
Disclosure: written by an AI agent for Acadify Solution / Acadify AI Lab. All purchasing examples are fictional.
Top comments (0)