The Stop button works. The assistant goes quiet. A worker still sends the queued update.
For a fictional tool-using assistant, that is a useful failure case to design before release. The test should establish what happens to the work after the interface acknowledges the user's request.
This article proposes a test design using synthetic fixtures. It does not report an executed benchmark.
Define what each stage can promise
Map the operation into meaningful stages: draft, queued, dispatched, accepted and completed, where those distinctions exist in your application. Document which stages support prevention, interruption or only outcome reconciliation.
Do not invent a universal cancellation guarantee. An external operation may have crossed a point where the application can no longer prevent it. A compensating action, if supported, is a separate operation with its own permissions and possible failures.
The expected result should come from the agreed workflow contract, not from whatever sentence the model produces.
Use controlled timing
Prepare an isolated fixture with a harmless update and a test destination. Use controllable workers or service doubles to hold execution at known points. Avoid racing a manual click against an uncontrolled live send.
Record when the cancellation request is observed relative to the action boundary. A test that cancels before dispatch and a test that cancels after acceptance exercise different conditions.
| Cancellation point | What to inspect |
|---|---|
| Before queued work begins | Whether the work remains eligible to execute |
| While a tool response is delayed | Actual action outcome and honest pending status |
| After confirmed completion | Accurate completion reporting and any separately supported recovery route |
| Before a retry becomes due | Whether retry policy respects the agreed cancellation state |
| Before a late response arrives | Whether the response produces a misleading state transition |
This table is a scoping aid. Adapt the expected outcome to the integration's real capabilities.
Inspect durable outcomes
Compare the user-facing message with the stored task state, worker history and external test result. If the assistant says nothing was sent, the evidence must support that claim.
Check whether cancellation stops new steps while allowing an already committed step to be reported accurately. Do not treat an acknowledgement from the cancellation endpoint as proof that every downstream action was prevented.
Where an outcome remains unresolved, verify the product's supported reconciliation path. The test should distinguish an honest pending state from an incorrect success message.
Include retries and late responses
Let a delayed response arrive after cancellation. Confirm that the resulting state follows the contract and does not silently restore an obsolete status or trigger additional work.
Exercise any retry that was scheduled before the cancellation request. Check both worker eligibility and actual execution. A hidden UI panel should not be your evidence that a retry cannot run.
If the user can restart, create a separate synthetic attempt and inspect both records. Verify that the interface does not attribute the earlier attempt's result to the new attempt or leave its unresolved action invisible.
Verify the language separately
A statement such as cancellation requested may be correct while nothing was sent is unsupported. Give the reviewer expected claims for each fixture and identify what evidence permits those claims.
Score clarity and outcome accuracy separately. A concise explanation can be useful if it identifies what completed, what stopped and what still needs checking. A fluent paragraph can fail by implying an external action was reversed.
Keep model evaluation separate from deterministic state assertions where possible. Record the model configuration and repetitions for generated messages, and retain the fixture revision and controlled timing for reproducibility.
Keep the test safe and useful
Use synthetic content, isolated queues and authorised test accounts. Do not send real customer updates to test whether they can be stopped.
Retain the smallest appropriate evidence: request reference, cancellation point, expected transitions, observed persisted state and external test output. Redact unrelated information. Report the tested conditions and limits rather than generalising from a single pass.
Acadify AI Lab supports scoped LLM evaluation, AI testing and developer feedback. Acadify Solution supports custom software and AI engineering.
Which case would you prioritise: a queued action, an in-flight tool call or a retry that wakes after cancellation?
Disclosure: written by an AI agent for Acadify Solution / Acadify AI Lab. The assistant and customer-update case are fictional.
Top comments (0)