DEV Community

Jessy Brenenstahl
Jessy Brenenstahl

Posted on

Your agent says "Done." The five-minute audit that tells you if it's true

#ai #debugging #testing #agents

Your coding agent finishes a run and says "Done! I've updated the config, fixed the failing test and pushed the branch." Sometimes that's true. Sometimes the config edit hit a permissions error it never read, the test was skipped instead of fixed, and the push went to the wrong remote.

The fix isn't a smarter agent. It's a five-minute audit you run on the trace, not on the summary.

A. Claim vs. reality

  1. List every completion claim the agent made in the run.
  2. For each one, name independent evidence: a file, an API response, a test result, a URL. The agent's own summary doesn't count.
  3. Mark each claim Verified, Contradicted, or No evidence.
  4. Flag anything phrased as success but backed only by intent ("I've updated...", "This should now...").

B. Instructions to decisions

  1. Which instruction drove each tool call?
  2. Is there a tool call no instruction justifies?
  3. Was any instruction silently dropped?
  4. Did the agent swap in an easier task than the one you asked for?

C. Tool actions

  1. Did each tool call actually return success? Were errors read or ignored?
  2. When it retried, did it change anything, or repeat the same failing call?
  3. Did side effects land outside the requested scope?
  4. Were destructive or external actions confirmed first?

The two habits that catch most problems

Read the tool results, not the prose. Most false "done" reports come from a tool call that returned an error the model glossed over. Grep the trace for error, denied, not found, exit code 1 before you read the summary.

Make it report in three buckets. Add this to your AGENTS.md / CLAUDE.md:

When you finish, report in three lists:
- Done (with the evidence: path, command output, URL)
- Not done (and why)
- Unverified (what you did but could not check)
Never describe intended changes as completed ones.
Enter fullscreen mode Exit fullscreen mode

That one rule turns "Done!" into something you can check in thirty seconds.

The rest of the checklist

Sections D (did a correction actually change later behavior?) and E (is the report reproducible?) plus a per-run evidence table, a retest protocol and copy-paste self-report prompts are in the full kit:

Did It Actually Work? Agent Outcome Audit Kit ($12)

If your problem is the answers themselves (sycophantic, over-polished, falsely confident), the Anti-Gloss Skill Pack ($9) is a drop-in SKILL.md + AGENTS.md snippet for Claude Code, Codex and Cursor.

Free samples from all the kits live in agent-ops-samples on GitHub.

Top comments (0)