DEV Community

Livecristen Studios
Livecristen Studios

Posted on

Why I stopped trusting "fixed" from my coding agent

#ai #python #testing #vibecoding

Agent edits a file, test goes green, agent says "fixed". But that green test only tells you one thing: the command passes now. It doesn't tell you it failed before. And it doesn't tell you it would fail again without the change.

So I built FixProof, a skill file plus a small Python CLI (stdlib only, 3.9+) that runs those checks itself. Here's what it looks like on the slugify example from the repo.

The bug 

def slugify(text):
    # BUG: spaces are removed instead of turned into dashes
    return text.strip().lower().replace(" ", "")
Enter fullscreen mode Exit fullscreen mode

The test expects slugify("Hello World ") to be hello-world. Note the double space.

Step 1: record the failure first 

$ python3 scripts/verify.py init --repro "python3 -m unittest test_slugify" --suite "python3 -m unittest test_other"
FixProof baseline: reproduction outcome = fail (exit 1)
Bug reproduced. Apply the fix, then run: verify.py check
Enter fullscreen mode Exit fullscreen mode

Step 2: a fix that looks done 

return text.strip().lower().replace(" ", "-")
Enter fullscreen mode Exit fullscreen mode

Feels right. But the double space turns into hello--world:

$ python3 scripts/verify.py check
FixProof 0.3.0  status: FAILED  (exit 1)
  - fix does not solve the bug: reproduction still fails (3/3 runs)
Enter fullscreen mode Exit fullscreen mode

An agent that skims the output would call this done. The CLI doesn't.

Step 3: the real fix 

return "-".join(text.strip().lower().split())
Enter fullscreen mode Exit fullscreen mode 
FixProof 0.3.0  status: VERIFIED  (exit 0)
  - baseline failed, fix passes all reruns, revert test fails without the fix, no regressions
  revert test: fails_without_fix
Enter fullscreen mode Exit fullscreen mode

The revert test is the part I care most about. It temporarily removes the fix, runs the repro again, checks it fails, and puts the file back. If the test still passes without your change, the test isn't actually testing the bug, and you get UNVERIFIED instead of VERIFIED.

The five results

  • VERIFIED: failed before, passes after on every rerun, fails again when reverted, nothing else broke
  • FAILED: repro still fails, or something that passed before now fails
  • PARTIALLY VERIFIED: repro passes but something's missing (no suite, no revert test)
  • UNVERIFIED: bug never reproduced, test passes even without the fix, flaky reruns
  • BLOCKED: missing tool, timeout, not a git repo

What it can't do

Being honest here:

  • the agent picks the repro command. If that command doesn't really hit the bug, VERIFIED just means the wrong command was verified
  • an agent could delete .fixproof/ and make a new log. If it matters, rerun check yourself
  • needs git
  • if the fix lives inside the test file, it can't be reverted cleanly
  • macOS isn't tested yet

Try it 

git clone https://github.com/Livecristen/fixproof.git ~/.claude/skills/fixproof
Enter fullscreen mode Exit fullscreen mode

(or ~/.codex/skills/ / .cursor/skills/)

It's v0.3.0, MIT: https://github.com/Livecristen/fixproof

Curious whether the status rules match how you work with agents. Feedback welcome.

Top comments (1)

Collapse
 
innokentyb profile image
Kent Bodrov •

How do you keep the revert check isolated from work happening in the same checkout? An editor or another agent could change the file between removal and restoration. I'd run baseline, candidate and reverted candidate in separate worktrees with the same test revision and environment, and bind those revisions into the receipt. That would make the counterfactual check useful without risking an unrelated edit or attributing an environment change to the patch.