Last week's eval on a truncated JSON bug showed us something uncomfortable: DebugAI can diagnose a bug correctly and still hand you a fix that fails 100% of the time, at 92% claimed confidence.
One data point isn't a pattern. So we built a harness to run that check automatically, across a real corpus, every time we touch the engine.
Before the harness produced one number we could trust, it produced three bugs in itself. Here they are, because the failure modes are generic enough that anyone building test tooling will hit them too.
The Setup
The harness spawns our own MCP server (npx @debugai/mcp), sends real bugs through the debug_error tool, and grades the response two ways: does the root cause mention what actually broke, and does the returned fix, applied to a clean copy of the source and run through a real test, actually pass. Confidence gets bucketed against pass rate. No eyeballing.
Corpus is 11 bugs. Eight are fixtures across common failure classes. Three are real DebugAI production incidents, ported straight from our own commit history, not reconstructed.
The Key That Wasn't There
The grader's output gets written to a JSONL file, one JSON object per line, so results survive after the terminal scrolls away. First real run, we went to read fixes[i].reason on a failed fix and got a KeyError. The key wasn't there.
JSON.stringify drops any object key whose value is undefined. Our grading function set reason on one code path (a fix that couldn't be applied at all) and never set it on the other (a fix that applied fine but failed the test). Both paths looked complete in the source. Only one of them survived serialization.
Fix: every return path now sets every key explicitly, null instead of leaving it unset. JSON.stringify keeps null. It only drops undefined.
The Patcher That Only Spoke One Language
Half the corpus is Python, ported from our own Python engine code. The other half is JavaScript. The harness applies a fix two ways depending on what the model returns: replace an exact line range, or, when no line number is given, find the named function and replace the whole thing.
The function-finder only recognized JavaScript syntax: function foo, const foo = (...) =>. Nothing for Python's def or async def. Every Python fix that arrived without a line number was silently unrunnable, from the first run. It didn't error. It just reported "could not apply," which reads exactly like an ordinary miss.
Worse: even after teaching it to find def foo, the code that figures out where the function ends counted curly braces. Python doesn't have any. We had to swap in indentation-based detection: a function ends at the first line that dedents back to or past where it started.
This is the dangerous kind of bug. It doesn't crash. It doesn't log anything wrong. It just quietly reports a plausible-looking result that happens to be worse than useless, because it looks identical to a real failure.
Today Ate Yesterday
Results write to a folder named by date. Run the harness twice in one day, chasing down a specific bug, and the second run overwrites the first. The "compare against the last run" feature then reports "no previous run found," on a day that very obviously had one.
Fix: folder names now include the time, not just the date. Old date-only folders still sort correctly as older, so nothing needed migrating.
What This Actually Taught Us
None of these three bugs are exotic. An undefined value vanishing from JSON. A regex written for one language and never revisited for the second. A cache key that didn't account for reruns. Individually, forgettable.
Together, they would have let us publish a confidence-calibration finding while half our own corpus was silently ungradable. The number would have looked exactly as trustworthy as a correct one.
If your eval tooling can fail silently, it will, exactly when you're not looking. Build in a self-test that replays one case with a known, already-documented answer, and check the harness reproduces that answer before trusting it on anything new. Ours does. It's how we caught bug two in the first place.
We're not publishing the calibration numbers yet. Three runs so far were mostly cache hits, replaying the same underlying model calls rather than fresh trials, and that's not a strong enough basis for a public claim. Next step is a clean run against a fresh cache window. When that lands, it gets its own post, numbers and all.
FAQ
Q: Why publish a post about bugs in your own test tooling instead of the actual eval results?
A: Because the tooling bugs are the more useful lesson, and the eval results aren't ready to stand behind publicly yet. We'd rather ship the honest version of this post now than sit on it until the bigger number is ready.
Q: Is the eval harness open source?
A: Not yet. It's a standalone repo, separate from the main product, built specifically to grade debug_error against a known corpus. If there's interest, we'll consider publishing it.
DebugAI reads your codebase, not just your stack trace, and hands you a fix you can apply in one click. We grade our own fixes the same way we're asking you to trust them: by running them.
Originally published at debugai.io.
Top comments (0)