“It gave the wrong answer yesterday” is useful feedback, but it is not yet a regression test.
You need enough context to reproduce the problem, a reviewed definition of acceptable behaviour and a record of the version that produced it. Without those, a prompt edit can appear to fix the report simply because the next run used different evidence.
Here is a small workflow for turning an AI bug report into a test case. The refund example is fictional. This article was generated by an AI agent for Acadify Solution and Acadify AI Lab.
Start with the user's task
Ask what the person was trying to do. Separate that from the answer they disliked.
For example: “Find out whether a purchase is eligible for a refund” describes a task. “The answer sounded too confident” describes an observation. Both belong in the investigation, but neither establishes the expected result on its own.
Capture only the inputs you are authorised to retain. Do not ask for passwords, tokens or a dump of unrelated customer records. If you replace private details with synthetic ones, record the transformation and review whether the failure still reproduces.
Freeze the conditions you can control
Record the application build, prompt version, model identifier and configuration. For retrieval-based features, retain an authorised snapshot of the relevant context or a reproducible corpus/index reference.
If your environment cannot reproduce a provider-side model version exactly, say so. “Same model name” may not establish an identical system. Record the date and the available version information, and limit your conclusions accordingly.
A case file might look like this:
{
"case_id": "refund-clarification-001",
"origin": "synthetic example",
"task": "Explain refund eligibility",
"input": "Can I return this purchase?",
"fixture_version": "refund-fixture-v1",
"context": "Refund eligibility requires the purchase date and item condition.",
"expected_behaviour": [
"Ask for the missing purchase date and item condition",
"Do not claim that the customer is eligible yet"
],
"review_status": "needs human review"
}
This is an illustrative schema, not a complete evaluation format. Your system may also need conversation history, tool outputs, permission scope and language or accessibility requirements.
Review the expectation before scoring
Do not turn the reported answer into the ground truth automatically. Review the governing source and the actual task.
In the example, asking for the missing details is acceptable. Several phrasings could do that well. Exact string matching would reject many reasonable answers and may reward one that merely copies a reference response.
Use explicit criteria: did the response request the needed information, avoid an unsupported eligibility claim and remain within the available policy? If reviewers disagree, resolve the ambiguity or label it. A disputed case should not quietly become a release gate.
Locate the failure stage
For a RAG workflow, inspect the retrieved context before changing the generation prompt. Was the relevant policy present? Was it current? Did a filter exclude it? Did the application truncate the passage?
For a tool-using workflow, inspect the arguments and result. A correct explanation of a failed lookup does not mean the underlying operation completed successfully.
Give the case a failure category that helps the next investigation. “Bad answer” is less useful than “missing evidence,” “unsupported conclusion” or “tool target mismatch.” Avoid pretending the category proves a cause before you have checked it.
Compare versions on the same reviewed cases
Run the current version and proposed version under stated conditions. Retain their outputs and the reviewer decisions. Add cases representing previously working tasks, rather than checking only the reported failure.
Where output is stochastic, predefine an appropriate repeat count and report the individual outcomes as well as a summary. A single good response is weak evidence of a reliable fix.
Do not repeatedly tune on every test you intend to use as an independent check. Keep a held-out set, and track which cases have influenced development.
Keep the result separate from the release decision
A useful comparison table is small:
| Case | Current version | Proposed version | Review note |
|---|---|---|---|
| Reported failure | Pending review | Pending review | Expected behaviour agreed first |
| Previously working task | Pending review | Pending review | Check for a meaningful regression |
| Missing information | Pending review | Pending review | Clarification may be the correct outcome |
These cells intentionally contain no invented scores. Fill them with observations from your own authorised test run.
Your release gate should reflect the product's consequences. An access violation or incorrect side effect may deserve a different gate from a wording issue. An aggregate average can hide a failure on a critical task.
Make the case useful next month
Give the case an owner and a reason it exists. When the policy or workflow changes, review whether the expected behaviour remains valid. Version the revision instead of changing the label invisibly.
Set access and retention rules for stored traces. A regression suite should preserve useful evidence, not become an ungoverned archive of user data.
The goal is a report that another engineer can investigate and a test that still means something after the immediate fix. That is a more useful handover than “we changed the prompt and it seems better.”
What detail is most often missing from the AI bug reports your team receives?
Acadify Solution provides software and AI engineering services. Acadify AI Lab supports scoped LLM evaluation and developer feedback with agreed tasks and rubrics.
Top comments (0)