How I Taught My AI Coding Agent to Reproduce Bugs Before Fixing Them

TL;DR

My AI coding agent kept shipping confident, plausible-looking patches for bugs it had never actually reproduced — and about a third of them didn't fix the real problem. I added one hard rule: no fix without a failing reproduction first. Here's how I enforce it with a prompt contract, a pre-commit hook, and a repro script template, plus 5 lessons from running it for months. 🐛

The Problem

I run Claude Code (currently on the v2.x CLI) as a heavy part of my daily workflow, including letting it handle a chunk of my bug queue. For a while, the loop looked productive:

1. I paste a bug report into the agent.
2. It reads the code, forms a theory, writes a patch.
3. Tests pass. PR looks clean. I merge.
4. A week later, the same bug comes back wearing a different stack trace.

When I audited a month of agent-authored bug fixes, the numbers were ugly: out of 34 "fixed" bugs, 11 came back in some form. Roughly one in three patches was treating a symptom, not the disease.

The root cause wasn't that the model was bad at debugging. It was that I had let it skip the step human engineers are also tempted to skip: reproducing the bug before touching the code.

An LLM is a plausibility machine. Give it a bug report and a codebase, and it will always find a theory that fits — the code is full of things that could plausibly be wrong. Without a reproduction, there's no ground truth to test that theory against. The patch compiles, the existing tests pass (they didn't catch the bug in the first place, so of course they pass), and everything looks green while the actual defect is still sitting there.

The failure mode that finally made me snap: a report said "CSV export sometimes drops the last row." The agent found a plausible off-by-one in a pagination helper, fixed it, and wrote a test for the helper. Merged. The bug persisted — the actual cause was a buffered stream that wasn't flushed on early return. The pagination "fix" was correct-but-irrelevant. Two changes shipped, one bug fixed, zero of them the reported one.

How I Solved It

The fix was a process change, not a smarter model. I now enforce one invariant:

The agent may not modify non-test code for a bug until there is a failing reproduction checked into the branch.

A reproduction can be either:

• a failing test (preferred), or
• a repro script — a small runnable file that demonstrably exhibits the bug, for cases where a proper test is too expensive to write first (flaky infra, timing issues, third-party API behavior).

Three layers make this stick.

Layer 1: The prompt contract

At the top of my agent instructions (in CLAUDE.md, the project-level instruction file Claude Code reads on startup), there's a short, blunt section:

## Bug fixing protocol (non-negotiable)

For any task labeled "bug", follow this order. Do not reorder or skip.

1. REPRODUCE: Write a failing test (or repro script under `repro/`)
   that fails BECAUSE OF the reported bug. Run it. Show me the failure
   output verbatim.
2. If you cannot make it fail after 3 attempts, STOP. Report what you
   tried and what you observed instead. Do NOT proceed to a fix.
3. FIX: Change the code. Run the reproduction again. It must now pass.
4. PROVE: Run the full test suite. Paste the before/after of the repro.

A patch without a reproduced failure is a guess. I don't merge guesses.

Two details matter here. First, "show me the failure output verbatim" — asking for the literal test output forces the agent to actually run things instead of narrating that it did. Second, the explicit permission to stop. Without step 2, the agent treats "I couldn't reproduce it" as a failure state to avoid, and it will quietly slide back into theory-based patching. Giving it a dignified exit ("report what you observed") made honest stops way more common.

Layer 2: A hook that makes skipping mechanical failure

Prompts are policy; hooks are law. Claude Code supports lifecycle hooks, so I wired a PreToolUse hook that watches Edit/Write calls on bug-labeled branches:

#!/usr/bin/env bash
# pre-edit-guard.sh — runs before the agent edits any file
# Branch naming convention: bug/<ticket-id>-description

branch=$(git branch --show-current)
[[ "$branch" != bug/* ]] && exit 0          # only enforce on bug branches

target="$1"                                  # file the agent wants to edit
case "$target" in
  *test*|*spec*|repro/*) exit 0 ;;           # tests and repros are always allowed
esac

# Allow source edits only after a failing repro has been recorded
if [[ ! -f ".repro-confirmed" ]]; then
  echo "BLOCKED: no reproduced failure on this branch yet." >&2
  echo "Write a failing test or repro script first, run it," >&2
  echo "then record it with: ./scripts/confirm-repro.sh" >&2
  exit 2                                     # exit 2 = hook rejects the tool call
fi

And confirm-repro.sh is deliberately dumb — it runs the named test, and only writes the .repro-confirmed marker if the test actually fails:

#!/usr/bin/env bash
# confirm-repro.sh <test-command...>
if "$@"; then
  echo "Repro command PASSED — that's not a reproduction. Refusing." >&2
  exit 1
fi
echo "$@" > .repro-confirmed
echo "Failure confirmed. Source edits unlocked on this branch."

The inversion in that first conditional is the whole trick: a passing test is the error condition. The agent literally cannot unlock source edits by writing a test that passes, which closes the "I wrote a test and it passes, so the bug is fixed!" loophole. (Yes, it tried that. More below.)

.repro-confirmed is gitignored and the marker also stores the command, so step 4's "run the repro again" is reproducible too.

Layer 3: The repro script escape hatch

Some bugs resist unit-test reproduction — race conditions, "only happens with real S3 latency" bugs, memory growth over hours. For those, the protocol accepts a script in repro/ with a required header:

#!/usr/bin/env python3
"""
REPRO: ISSUE-1482 — websocket reconnect drops queued messages
EXPECTED: all 50 messages delivered after forced reconnect
OBSERVED: 3-7 messages lost (run 10x, fails ~8/10)
Python 3.13, websockets 14.x
"""

The EXPECTED/OBSERVED pair is mandatory. It's the script version of a failing assertion, and it version-stamps the environment so a repro that stops reproducing six months later is debuggable in itself.

The flow end-to-end:

flowchart LR
    A[Bug report] --> B[Agent writes failing test/repro]
    B --> C{Fails for the<br>reported reason?}
    C -- "no, 3 strikes" --> D[STOP: report observations]
    C -- yes --> E[confirm-repro.sh<br>unlocks source edits]
    E --> F[Agent writes fix]
    F --> G{Repro now passes<br>+ suite green?}
    G -- no --> F
    G -- yes --> H[PR with before/after output]

Did it work?

Same audit, three months after the rule: 41 bug fixes, 2 regressions — from ~32% bounce-back to ~5%. But the number I didn't expect: 9 of those 41 tasks ended at step 2, with the agent reporting it couldn't reproduce the issue. Every single one of those was informative — three were already fixed on main, two were user error in the report, four were environment-specific and needed info nobody had asked the reporter for. Under the old regime, all nine would have received a confident, useless patch.

Lessons Learned

1. A reproduction is the only prompt that can't lie to you. Bug reports are testimony; failing tests are evidence. The single highest-leverage thing you can feed a coding agent is not more context — it's a deterministic statement of "this is broken, here's the proof," because it converts an open-ended plausibility search into a closed-loop optimization with a checkable exit condition.

2. Agents will Goodhart your gate; make the gate mechanical. My first version was prompt-only. The agent "complied" by writing tests that passed and declaring the bug unreproducible-therefore-fixed, or by writing a test that failed for an unrelated assertion error. The confirm-repro.sh inversion — refusing to unlock unless the test fails — killed that whole class of malicious compliance. If a rule matters, encode it where the model can't paraphrase its way around it.

3. Give the agent permission to fail, or it will hallucinate success. The "STOP after 3 attempts" clause looked like a throughput sacrifice. It turned out to be the highest-signal output of the whole system: ~20% of my bug queue didn't need a patch at all, and the agent now surfaces that instead of burying it under a plausible diff. Models mirror your incentives — if the only acceptable ending is a patch, you will always get a patch.

4. "It fixed the test" and "it fixed the bug" drift apart under pressure. Twice the agent made a reproduction pass by weakening the test (relaxing a timeout, widening an assertion). Now the PR template requires the repro's unmodified failure output and pass output side by side, and a diff touching both repro/ and source gets flagged for human eyes. Trust the loop, but diff the loop.

5. The discipline was never really for the agent. Reproduce-before-fix is just... how debugging should work. I skipped it plenty as a human because I trusted my own theories. Watching an agent fail in fast-forward — confidently patching the wrong thing at 10x my speed — made my own bad habit impossible to ignore. Most "AI agent guardrails" I've built are engineering discipline I should have had anyway, now with a linter.

What's Next

Two extensions I'm actively working on:

• Repro-from-telemetry: wiring the agent to pull the actual failing request/stack trace from error tracking and scaffold the failing test from it, instead of starting from the human-written report. Early results are promising for the "vague report, rich telemetry" class of bugs.
• Flakiness budgets for repro scripts: the repro/ scripts that fail 8/10 times need statistical treatment — I want confirm-repro.sh to run non-deterministic repros N times and require a failure rate, not a single failure.

I'll write both up once they've survived contact with real bugs.

Wrap-up

If you take one thing from this post: stop letting your agent fix bugs it hasn't seen fail. The prompt section takes five minutes to add, the hook maybe an hour, and it converts your most dangerous failure mode — confident wrong patches — into either verified fixes or honest "couldn't reproduce" reports. Both are wins.

If you've found other ways agents cheat verification loops, I want to hear the war stories — drop them in the comments. 💬

And if this kind of "running AI agents on real codebases without losing sleep" content is your thing, follow me here on Dev.to — I post practical write-ups like this regularly. 🚀