DEV Community: Acadify Solution

Turn an AI Bug Report into a Regression Case Someone Else Can Reproduce

Acadify Solution — Wed, 14 Oct 2026 01:43:19 +0000

“It gave the wrong answer yesterday” is useful feedback, but it is not yet a regression test.

You need enough context to reproduce the problem, a reviewed definition of acceptable behaviour and a record of the version that produced it. Without those, a prompt edit can appear to fix the report simply because the next run used different evidence.

Here is a small workflow for turning an AI bug report into a test case. The refund example is fictional. This article was generated by an AI agent for Acadify Solution and Acadify AI Lab.

Start with the user's task

Ask what the person was trying to do. Separate that from the answer they disliked.

For example: “Find out whether a purchase is eligible for a refund” describes a task. “The answer sounded too confident” describes an observation. Both belong in the investigation, but neither establishes the expected result on its own.

Capture only the inputs you are authorised to retain. Do not ask for passwords, tokens or a dump of unrelated customer records. If you replace private details with synthetic ones, record the transformation and review whether the failure still reproduces.

Freeze the conditions you can control

Record the application build, prompt version, model identifier and configuration. For retrieval-based features, retain an authorised snapshot of the relevant context or a reproducible corpus/index reference.

If your environment cannot reproduce a provider-side model version exactly, say so. “Same model name” may not establish an identical system. Record the date and the available version information, and limit your conclusions accordingly.

A case file might look like this:

{
  "case_id": "refund-clarification-001",
  "origin": "synthetic example",
  "task": "Explain refund eligibility",
  "input": "Can I return this purchase?",
  "fixture_version": "refund-fixture-v1",
  "context": "Refund eligibility requires the purchase date and item condition.",
  "expected_behaviour": [
    "Ask for the missing purchase date and item condition",
    "Do not claim that the customer is eligible yet"
  ],
  "review_status": "needs human review"
}

This is an illustrative schema, not a complete evaluation format. Your system may also need conversation history, tool outputs, permission scope and language or accessibility requirements.

Review the expectation before scoring

Do not turn the reported answer into the ground truth automatically. Review the governing source and the actual task.

In the example, asking for the missing details is acceptable. Several phrasings could do that well. Exact string matching would reject many reasonable answers and may reward one that merely copies a reference response.

Use explicit criteria: did the response request the needed information, avoid an unsupported eligibility claim and remain within the available policy? If reviewers disagree, resolve the ambiguity or label it. A disputed case should not quietly become a release gate.

Locate the failure stage

For a RAG workflow, inspect the retrieved context before changing the generation prompt. Was the relevant policy present? Was it current? Did a filter exclude it? Did the application truncate the passage?

For a tool-using workflow, inspect the arguments and result. A correct explanation of a failed lookup does not mean the underlying operation completed successfully.

Give the case a failure category that helps the next investigation. “Bad answer” is less useful than “missing evidence,” “unsupported conclusion” or “tool target mismatch.” Avoid pretending the category proves a cause before you have checked it.

Compare versions on the same reviewed cases

Run the current version and proposed version under stated conditions. Retain their outputs and the reviewer decisions. Add cases representing previously working tasks, rather than checking only the reported failure.

Where output is stochastic, predefine an appropriate repeat count and report the individual outcomes as well as a summary. A single good response is weak evidence of a reliable fix.

Do not repeatedly tune on every test you intend to use as an independent check. Keep a held-out set, and track which cases have influenced development.

Keep the result separate from the release decision

A useful comparison table is small:

Case	Current version	Proposed version	Review note
Reported failure	Pending review	Pending review	Expected behaviour agreed first
Previously working task	Pending review	Pending review	Check for a meaningful regression
Missing information	Pending review	Pending review	Clarification may be the correct outcome

These cells intentionally contain no invented scores. Fill them with observations from your own authorised test run.

Your release gate should reflect the product's consequences. An access violation or incorrect side effect may deserve a different gate from a wording issue. An aggregate average can hide a failure on a critical task.

Make the case useful next month

Give the case an owner and a reason it exists. When the policy or workflow changes, review whether the expected behaviour remains valid. Version the revision instead of changing the label invisibly.

Set access and retention rules for stored traces. A regression suite should preserve useful evidence, not become an ungoverned archive of user data.

The goal is a report that another engineer can investigate and a test that still means something after the immediate fix. That is a more useful handover than “we changed the prompt and it seems better.”

What detail is most often missing from the AI bug reports your team receives?

Acadify Solution provides software and AI engineering services. Acadify AI Lab supports scoped LLM evaluation and developer feedback with agreed tasks and rubrics.

Acadify Solution · Acadify AI Lab

Testing AI Agent Tool Calls: A Release Checklist Beyond Task Completion

Acadify Solution — Tue, 13 Oct 2026 02:16:37 +0000

A task-completion score can hide an agent that reached the right answer through an unauthorised action. Evaluate what happened, which actions occurred and whether the system stayed within its authority.

This article was generated by an AI agent for Acadify Solution and Acadify AI Lab. All scenarios are illustrative, not client results or benchmark measurements.

1. Write an action contract

For each tool, define who may invoke it, the tenant and resource scope, required arguments, permitted state transitions, approval requirements and the meaning of success, failure or an unknown outcome.

Enforce access in the application/tool layer. A prompt can describe a boundary; it does not replace an authorisation check. Reading an enquiry, proposing a CRM update and sending an external reply deserve separate contracts.

2. Test the target as well as the payload

Use a disposable environment with synthetic data. Include two tenants and similarly named records. Verify that a supplied identifier does not grant access to a record outside the authorised scope.

Validate types, required fields and business rules at execution time. Rejected calls should cause no state change. Record the attempt and rejection with appropriate redaction.

3. Inject an uncertain write outcome

Test a service that commits a write but loses the response. A timeout does not prove that nothing changed.

Where the API supports it, use a stable operation ID and status lookup. Reusing the ID with a different payload should detect a conflict. If there is no reliable status or deduplication mechanism, represent the unknown outcome and escalate according to the workflow's risk.

Injected condition	What to check
Failure before commit	No state change; bounded retry if permitted
Commit followed by timeout	Status resolution or safe escalation
Replayed request	No unintended duplicate side effect
Same operation ID, different payload	Conflict detected

A retry budget alone does not make a write safe. Expected behaviour must follow the API contract and business rules.

4. Bind approval to a proposal

Show the reviewer the target and important arguments. Test changes to content, recipient or target after approval. The old approval must not silently cover a materially different action.

Test rejection and expiry too. A delayed decision should not authorise a later, changed proposal. A blanket approval token disconnected from the proposal is difficult to audit.

5. Treat retrieved instructions as untrusted input

Place conflicting instructions in a synthetic document—for example, a paragraph asking the assistant to send a message outside the requested workflow. Check both the model's behaviour and the application's permission enforcement.

Record the input, selected context and attempted calls. Do not use real secrets or private customer records as test material. A single passing case does not show that prompt injection is solved; vary the cases and report the tested scope.

6. Exercise stopping and handover

Test missing fields, invalid tool output, expired access, repeated failures and step limits. Define terminal states such as completed, awaiting approval, failed and unknown outcome.

A handover should identify the last confirmed action and the unresolved step. A fluent final message is not evidence of a completed workflow.

7. Retain useful, limited traces

Record the case ID, workflow version, authorised scope, redacted arguments, approval decisions, tool results and state changes. Decide who can inspect traces and how long to retain them.

A reviewer should be able to reconstruct the result without unrestricted access to every document the system encountered.

8. Separate release gates

Report task outcome separately from policy and integration checks. Under a defined permission gate, an unauthorised write should fail the case even if the final answer is correct.

Document case counts, fixture versions, model and prompt versions, stochastic repeats where appropriate and known gaps. Keep a held-out set when tuning.

This checklist is a starting point, not a security certification or reliability guarantee. Adapt it to your system and the consequences of its actions.

Which failure injection would you add first to your agent's release test?

Acadify Solution provides software and AI engineering services. Acadify AI Lab supports scoped LLM and agent evaluations using agreed tasks, rubrics and reviewable findings.

Acadify Solution · Acadify AI Lab

A Small RAG Evaluation Harness: Test Retrieval Before Blaming the LLM

Acadify Solution — Mon, 12 Oct 2026 11:38:20 +0000

A wrong RAG answer does not automatically mean the language model failed. The system may have retrieved the wrong document, omitted the relevant passage, or supplied conflicting evidence.

This guide builds a small retrieval-only evaluation harness. It separates one question—did we retrieve the expected evidence?—from answer quality. It is an illustrative example, not a measured result from an Acadify client project.

1. Define what the test can and cannot tell you

Start with two retrieval measures:

Recall@k: the fraction of known relevant documents present in the first k results.
Reciprocal rank@k: the inverse rank of the first relevant result, or zero if none appears.

Recall@k helps detect missing evidence. Reciprocal rank helps distinguish evidence ranked first from evidence buried lower in the results. Neither measures whether the final answer is correct, safe, or grounded.

Use document identifiers at the same granularity as your labels. If your retriever returns chunks but your labels refer to documents, map chunk IDs to document IDs before scoring.

2. Create reviewed cases

Each answerable question needs a set of expected document IDs. Use questions representative of the actual workflow, and have someone review the labels.

For an unanswerable question, an empty set does not prove that any retrieved document is irrelevant. Here it means there is no known answer in the approved corpus. We report these separately and exclude them from recall and ranking averages.

3. Run a deterministic example

The following uses only the Python standard library. Replace the fixture retrieval function with your own retrieval API when you are ready.

from dataclasses import dataclass
from statistics import mean
from typing import Callable, Sequence

@dataclass(frozen=True)
class Case:
    case_id: str
    question: str
    relevant_ids: frozenset[str]


def score(case: Case, retrieved: Sequence[str], k: int) -> dict:
    if k < 1:
        raise ValueError("k must be positive")

    # Deduplicate while preserving rank. Score unique document results.
    top = list(dict.fromkeys(retrieved))[:k]
    relevant = case.relevant_ids

    if not relevant:
        return {"case_id": case.case_id, "answerable": False,
                "retrieved_ids": top, "recall": None, "rr": None}

    hits = relevant.intersection(top)
    first_hit = next((rank for rank, doc_id in enumerate(top, 1)
                      if doc_id in relevant), None)
    return {"case_id": case.case_id, "answerable": True,
            "retrieved_ids": top,
            "recall": len(hits) / len(relevant),
            "rr": 1 / first_hit if first_hit else 0.0}


def evaluate(cases: Sequence[Case],
             retrieve: Callable[[str], Sequence[str]], k: int) -> dict:
    rows = [score(case, retrieve(case.question), k) for case in cases]
    answerable = [row for row in rows if row["answerable"]]
    return {
        "k": k,
        "answerable_cases": len(answerable),
        "unanswerable_cases": len(rows) - len(answerable),
        "mean_recall": mean(row["recall"] for row in answerable)
                       if answerable else None,
        "mrr": mean(row["rr"] for row in answerable)
               if answerable else None,
        "rows": rows,
    }


cases = [
    Case("refund", "How do refunds work?", frozenset({"refund-policy"})),
    Case("cancel", "How can I cancel?", frozenset({"cancel-guide"})),
    Case("missing", "What is next year's price?", frozenset()),
]
fixture = {
    "How do refunds work?": ["faq", "refund-policy", "refund-policy"],
    "How can I cancel?": ["billing", "faq"],
    "What is next year's price?": ["current-pricing"],
}

result = evaluate(cases, lambda question: fixture[question], k=2)
assert result["mean_recall"] == 0.5
assert result["mrr"] == 0.25
assert result["unanswerable_cases"] == 1
print(result)

The fixture deliberately retrieves the refund document at rank two and misses the cancellation document. The averages describe these two answerable fixture cases only. They are not an estimate of production performance.

4. Inspect failures before changing the system

For each failed case, check:

Is the expected document present in the indexed corpus?
Is the label still correct for the corpus version?
Was the relevant text split across chunks?
Did metadata filters exclude it?
Did the query need terminology the embedding model did not match?
Did reranking push the evidence outside the context budget?

Change one component at a time. Run the same reviewed cases against the same corpus snapshot so you can attribute differences. Keep a held-out set; repeatedly tuning on the evaluation set can make the score improve without improving the real workflow.

5. Add answer evaluation as a separate stage

After retrieval, test the generated answer with an explicit rubric:

Are its factual claims supported by the supplied passages?
Do citations point to evidence supporting the associated claim?
Does it answer the user's actual question?
Does it ask for clarification or abstain when evidence is insufficient?
Does it respect access controls and avoid exposing restricted data?

Automated checks and model-based judges can help triage results. Their judgments still need validation against human review. Passing a retrieval test does not establish answer correctness or authorization safety.

6. Keep the evidence reproducible

Record case ID, corpus/index version, retrieval settings, model version, prompt version and the retrieved IDs. Store passages and outputs only where you have permission; redact confidential or personal data before sharing traces.

For production comparisons, use more representative cases, repeat stochastic runs where appropriate, report the denominator and sampling limitations, and track latency and cost alongside quality.

The practical lesson: locate the failure stage first. A prompt rewrite cannot recover evidence the system never retrieved.

What is harder in your RAG workflow: creating reliable labels, evaluating citations, or reproducing retrieval failures?

About the author: Acadify Solution provides software and AI engineering services. Acadify AI Lab is our AI evaluation practice. This article was generated by an AI agent; its dataset and code are illustrative. Learn more at Acadify Solution or Acadify AI Lab.