DEV Community: Mohi Uddin

The Hardest Bug to Find Was a False Positive.

Mohi Uddin — Sun, 02 Aug 2026 01:00:00 +0000

The Benchmark Behind Proactive Scan

On Tuesday we shipped Proactive Scan: one command that ranks your riskiest files and reads each with cross-file context, before anything crashes.

Today we're publishing the benchmark behind it. All of it: the numbers, the methodology, the part where our first version cried wolf, and the kinds of bugs it still misses.

The Result

28 test cases, written fresh for this benchmark. 20 files with planted defects, 8 clean files as controls.

Category	Score	What was planted
Single-file defects	8/8	SQL injection, resource leak, division by zero, mutable default arg, bare except, missing await, off-by-one, hardcoded secret
Cross-file defects	12/12	wrong argument count, missing export, return-shape mismatch, type mismatch, zero-value config, interface drift, async race, renamed function, generic type misuse, optional and null flow, config modulo, callback arity
False alarms on clean files	0/8	error handling, executemany, type narrowing, async patterns, optional chaining, class design

Run on July 2, 2026 against our staging API, which runs the same pipeline as production: Voyage embeddings for the codebase index, Claude Haiku 4.5 for analysis.

This is our corpus, scored by us, with n=28. It says the scan does what we designed it to do. It does not say we beat anyone. We ran no competitors, so you will find no competitor numbers here.

Why We Wrote the Cases From Scratch

The lazy way to build this benchmark is to grab known bugs from popular open-source repos. We had exactly that lying around from an internal regression test. We didn't publish it, for one reason: models have read those repos. A detector that recognizes a famous React bug from training data is doing recall, not analysis.

So every one of the 28 cases was written fresh in late June. Original code, original planted defects, patterned after real-world failure classes but not copied from anything public. No model has seen them.

What a Cross-File Case Looks Like

The single-file cases are table stakes. Any decent linter flags a bare except. The 12 cross-file cases are the reason the scan exists, because they're invisible unless the tool knows the rest of your project.

Here is XF-05, two files, trimmed:

# limits.py
# requests per window; 0 disables (but callers divide by it)
WINDOW_SLOTS = 0

# rate.py
from limits import WINDOW_SLOTS

def per_slot(total):
    return total / WINDOW_SLOTS

rate.py is flawless on its own. Every line is correct. The bug lives in the relationship: a config value in another file is zero, and this file divides by it. Point a single-file tool at rate.py and it shrugs. The scan indexes both files, pulls limits.py in as context when it reads rate.py, and reports the ZeroDivisionError waiting to happen.

That's the pattern for all 12 cross-file cases: the buggy file is locally clean and the defect only exists given a fact from elsewhere in the project.

How Scoring Works

Each case declares an expected signal: keywords and an issue class the finding must mention (for XF-05: "division by zero" or the offending constant's name). A case counts as detected when the scan reports a finding on the planted file matching that signal. A clean file passes when the scan returns nothing.

Lenient keyword matching means we're grading "did it see the problem," not "did it phrase the problem the way we would." We wrote both the cases and the grader, which is a real conflict, and the mitigation is transparency: the corpus, the expected signals, and the runner are published, so you can re-run it and disagree with our scoring.

The False Positive War

The headline number everyone asks about is detection. The number that decides whether anyone keeps the tool installed is false alarms. A scan that flags healthy code trains you to ignore it, and then it's worse than no scan.

Our git history tells the story honestly. Three shipped prompt versions in five days.

scan-v3. First version with the benchmark in place. Found everything, and also "found" style preferences in clean files. A clean error-handling file got flagged for not re-raising. Wolf, cried.

scan-v4. Carved out an entire false-positive class: code that handles external or unknown input defensively was being flagged for the input being unknown. The fix was prompt discipline: a finding must name a concrete failure, with concrete inputs, reachable from the code as written. "This could be risky" is not a finding.

scan-v5. One control file (CLEAN-06, async patterns) still produced an occasional hedge-flag. Hardened the instruction: if the code is correct, say nothing. Zero findings is a valid, good answer.

That last sentence took the longest to make the model believe. Language models want to be helpful, and "I found nothing" doesn't feel helpful. Getting a model to stay quiet on 8 clean files was harder than getting it to talk about 20 buggy ones.

What It Misses

The benchmark tests what we planted. It says nothing about several bug classes we know are out there, and honesty about the boundary is the point of publishing.

Bugs that need runtime data. A race that only appears under load, a leak that needs a traffic pattern. Static reading won't catch these; that's what the debugger side of DebugAI is for, after they fire.

Spec bugs. Code that does exactly what it says, where what it says is not what the business needed. No tool without your intent can catch these.

Scale. Fixture files are small. Production files with 2,000 lines and 40 imports are a harder retrieval problem. We test that separately and it is not in this number.

Languages. The corpus is Python and JavaScript/TypeScript, which is what DebugAI supports. Nothing here generalizes beyond that.

28 cases is a benchmark, not a proof. A perfect score on 28 cases means the mechanism works, not that the mechanism is perfect.

Reproduce It

The corpus (all 28 cases with expected signals), the runner, and the raw results JSON are public: github.com/1shizaan/debugai-scan-benchmark. Point the runner at your own API key and re-run:

python3 run.py --api-key dbg_your_key_here

If you find a case where our scoring flatters us, open an issue. We'll take the hit in public.

FAQ

Q: Why not compare against Copilot or Semgrep?

A: Because a fair head-to-head is its own project: same corpus, each tool run the way a real user runs it, documented versions, blind scoring, published losses. Rushing that for a launch week would produce exactly the kind of number we don't trust from other vendors. If we do it, it gets its own post.

Q: Doesn't writing your own test cases guarantee a good score?

A: It's a real risk, and the honest answer is partially. We designed cases the scan should catch. The controls are the check on it: a scanner tuned to flag everything would ace detection and fail all 8 clean files. Zero false positives is the half of the score we couldn't game without publishing the evidence.

Q: Which model runs the scan?

A: Claude Haiku 4.5 with our scan-v5 prompt, over a Voyage-embedded index of the project. Same pipeline on the free tier.

Originally published at debugai.io.

Three Silent Bugs That Broke Our AI Evaluation Harness

Mohi Uddin — Sat, 01 Aug 2026 01:00:00 +0000

Last week's eval on a truncated JSON bug showed us something uncomfortable: DebugAI can diagnose a bug correctly and still hand you a fix that fails 100% of the time, at 92% claimed confidence.

One data point isn't a pattern. So we built a harness to run that check automatically, across a real corpus, every time we touch the engine.

Before the harness produced one number we could trust, it produced three bugs in itself. Here they are, because the failure modes are generic enough that anyone building test tooling will hit them too.

The Setup

The harness spawns our own MCP server (npx @debugai/mcp), sends real bugs through the debug_error tool, and grades the response two ways: does the root cause mention what actually broke, and does the returned fix, applied to a clean copy of the source and run through a real test, actually pass. Confidence gets bucketed against pass rate. No eyeballing.

Corpus is 11 bugs. Eight are fixtures across common failure classes. Three are real DebugAI production incidents, ported straight from our own commit history, not reconstructed.

The Key That Wasn't There

The grader's output gets written to a JSONL file, one JSON object per line, so results survive after the terminal scrolls away. First real run, we went to read fixes[i].reason on a failed fix and got a KeyError. The key wasn't there.

JSON.stringify drops any object key whose value is undefined. Our grading function set reason on one code path (a fix that couldn't be applied at all) and never set it on the other (a fix that applied fine but failed the test). Both paths looked complete in the source. Only one of them survived serialization.

Fix: every return path now sets every key explicitly, null instead of leaving it unset. JSON.stringify keeps null. It only drops undefined.

The Patcher That Only Spoke One Language

Half the corpus is Python, ported from our own Python engine code. The other half is JavaScript. The harness applies a fix two ways depending on what the model returns: replace an exact line range, or, when no line number is given, find the named function and replace the whole thing.

The function-finder only recognized JavaScript syntax: function foo, const foo = (...) =>. Nothing for Python's def or async def. Every Python fix that arrived without a line number was silently unrunnable, from the first run. It didn't error. It just reported "could not apply," which reads exactly like an ordinary miss.

Worse: even after teaching it to find def foo, the code that figures out where the function ends counted curly braces. Python doesn't have any. We had to swap in indentation-based detection: a function ends at the first line that dedents back to or past where it started.

This is the dangerous kind of bug. It doesn't crash. It doesn't log anything wrong. It just quietly reports a plausible-looking result that happens to be worse than useless, because it looks identical to a real failure.

Today Ate Yesterday

Results write to a folder named by date. Run the harness twice in one day, chasing down a specific bug, and the second run overwrites the first. The "compare against the last run" feature then reports "no previous run found," on a day that very obviously had one.

Fix: folder names now include the time, not just the date. Old date-only folders still sort correctly as older, so nothing needed migrating.

What This Actually Taught Us

None of these three bugs are exotic. An undefined value vanishing from JSON. A regex written for one language and never revisited for the second. A cache key that didn't account for reruns. Individually, forgettable.

Together, they would have let us publish a confidence-calibration finding while half our own corpus was silently ungradable. The number would have looked exactly as trustworthy as a correct one.

If your eval tooling can fail silently, it will, exactly when you're not looking. Build in a self-test that replays one case with a known, already-documented answer, and check the harness reproduces that answer before trusting it on anything new. Ours does. It's how we caught bug two in the first place.

We're not publishing the calibration numbers yet. Three runs so far were mostly cache hits, replaying the same underlying model calls rather than fresh trials, and that's not a strong enough basis for a public claim. Next step is a clean run against a fresh cache window. When that lands, it gets its own post, numbers and all.

FAQ

Q: Why publish a post about bugs in your own test tooling instead of the actual eval results?

A: Because the tooling bugs are the more useful lesson, and the eval results aren't ready to stand behind publicly yet. We'd rather ship the honest version of this post now than sit on it until the bigger number is ready.

Q: Is the eval harness open source?

A: Not yet. It's a standalone repo, separate from the main product, built specifically to grade debug_error against a known corpus. If there's interest, we'll consider publishing it.

DebugAI reads your codebase, not just your stack trace, and hands you a fix you can apply in one click. We grade our own fixes the same way we're asking you to trust them: by running them.

Originally published at debugai.io.

The Higher-Ranked AI Fix Failed. The Lower-Ranked One Passed.

Mohi Uddin — Fri, 31 Jul 2026 14:55:00 +0000

Confidence Numbers Weren't Tracking Correctness. Here's What We Built Instead.

Two fixes came back from the same call last week, ranked by confidence like they always are. The higher-ranked one failed the moment we ran it against the real bug. The lower-ranked one passed.

We'd been trusting that ranking, the same way anyone reading a DebugAI response trusts it: a bigger number means the model is more sure. So we stopped trusting it and started checking.

Two weeks ago we found three bugs in our own eval harness before we'd let it grade a single real result. This is what the harness found once we trusted it, and what we built because we couldn't unsee it.

What the Numbers Said

We bucket every fix by its reported confidence, then run it against the real bug and record whether it actually passed. Here's the split from the run we're standing behind:

Reported confidence	Fixes that passed
80% to 90%	0%
90% and above	38%

Read that twice. Fixes the model was most sure about passed less often than a coin flip. Fixes in the 80 to 90 percent band, still a band most people would read as "pretty confident," passed exactly none of the time in this run. The number was going up while the odds of it being right were not.

"Passed" here means the fix, applied to a clean copy of the source, made the actual test pass.

Not "looked reasonable." Not "the model sounded sure." Ran, on real code, and worked.

Same Shape, Three Times

The first sign of this wasn't the table above. It was a single bug, checked by hand, where two ranked fixes came back at 92% and 85% confidence and both of them failed. That one case could have been noise, so we went looking for more of it, on two separate runs of the harness we'd built specifically to check.

All three looks found the same shape: confidence going up did not mean correctness going up. Then we rebuilt the harness from scratch in a new home, ran it a fourth time on a fresh 11-bug corpus, and got the table above. Same shape again, on code the earlier runs had never touched.

Four separate looks, at different times, on different bugs, with different versions of the checking tool. If this were a one-off measurement error, at least one of those runs should have come back clean. None did.

What We Built Because of It

A confidence number that doesn't track correctness isn't just unhelpful, it's actively misleading, because it looks exactly like a helpful one. So instead of asking the model to grade its own homework, we built a grader that isn't the model.

For two classes of bug where the check is cheap and deterministic, parse errors and import errors, DebugAI now runs the proposed fix through a real mechanical check before it ever reports a confidence number.

Parse Check

Does the fixed code actually parse.

Not "does it look like valid syntax to a language model," but a real parser, sandboxed, with a hard CPU limit.

Import Check

Does the import the fix relies on match a file DebugAI actually retrieved for this request.

Worth being precise about what this claims: it means the import resolves against the context we pulled, not that the package is installed or the function exists at runtime. We say exactly that, nowhere more.

If the check fails, confidence gets capped at 15, no matter what the model originally reported.

min(fix.confidence, 15)

when verified comes back false.

The model doesn't get a vote on its own grade once we've actually run the check.

Both checks run sandboxed, with a five-second budget shared across all candidate fixes for a request. If a check can't finish in time, it doesn't guess. It reports that it didn't finish.

A check that didn't finish and a check that failed are different claims, and we don't let them collapse into each other.

verified: null means "we didn't check," either because this bug class isn't covered yet or because the five-second budget ran out.

verified: false means "we checked, and it failed."

Rounding null up to false, or down to a silent pass, would make the label lie in exactly the way the original confidence number did.

Seeing It Happen for Real

This isn't theoretical. The first time the check fired against a real request, this is what came back:

{
  "rank": 1,
  "confidence": 15,
  "verified": false,
  "verification_reason": "import(s) not found in retrieved context: billing.pricing (checked against files DebugAI retrieved, not installed packages)"
}

The model didn't know the exact name of the function it needed to import. Rather than confidently inventing one, it generated a placeholder and flagged that it needed more context.

The old behavior would have reported whatever confidence the model felt like reporting on a guess.

Instead, the check ran, found the import didn't resolve against anything we'd actually retrieved, and the confidence landed exactly where min(fix.confidence, 15) puts it: 15.

Not a dramatic catch, the model was already being honest here, but it's the first real proof the wiring works end to end, on a live request, not a fixture.

What This Doesn't Fix Yet

Parse and import checks cover exactly two bug classes.

The bucket table above (0% at 80 to 90% confidence, 38% at 90%+) was measured on a corpus that's mostly outside those two classes, and outside those classes, nothing has changed.

The inversion is still there for TypeError, RuntimeError, and everything else that needs a live environment or real data to actually verify, not just parse.

Type-class checking, the kind that would catch more of what's left, is built behind the same interface and not shipped. It's gated on whether it can run inside the five-second budget without becoming the slow part of every response.

We're not claiming this closes the gap.

We're claiming it closes it for the two classes cheap enough to check honestly today, and that the rest stays visibly unverified rather than quietly assumed fine.

Every fix DebugAI returns now carries one of three states:

true
false
null

true means we ran a real check and it held.

false means we ran it and it failed, and the confidence number reflects that whether the model likes it or not.

null means we didn't check, and we say so instead of letting a high number stand in for an answer we don't have.

FAQ

Q: Does this mean DebugAI's fixes are usually wrong?

A: No.

It means the confidence number attached to a fix wasn't a reliable signal of whether that specific fix would work, for the classes we measured.

The fixes themselves aren't graded here; the correlation between "how sure the model sounded" and "was it right" is.

Those are different questions, and conflating them is exactly the mistake this whole feature exists to stop making.

Q: Will this cover more bug classes over time?

A: Parse and import were the two cheap enough to check mechanically without adding real latency.

Type checking is built and waiting on a latency decision.

Anything that needs live execution against real state, most runtime errors, needs a different approach than a five-second sandboxed check, and we're not going to claim we've solved that until we actually have.

DebugAI reads your codebase, not just your stack trace, and hands you a fix you can apply in one click.

Now some of those fixes come with a number we checked ourselves, and the ones we haven't checked yet say so.

Originally published at debugai.io.

Your AI gave that fix 92% confidence. Nothing checked it.

Mohi Uddin — Thu, 30 Jul 2026 19:38:00 +0000

Your agent hands you a fix with 92% confidence.

Ask yourself what produced the 92. Not what it means. What produced it.

A model wrote that number about its own output. Nothing ran. Nothing parsed. No test executed. It is a language model's estimate of how a language model feels about a language model's suggestion. And it renders in the same font, the same shade of green, as a number that came out of a compiler.

That is the part I could not stop thinking about while building a debugging tool. Not that AI fixes are wrong. Most are fine. The problem is that the good ones and the bad ones arrive looking identical, so you have to read every one carefully, which is most of the time you were trying to save.

Stack Overflow's 2025 Developer Survey found 84% of developers using AI tools, 45% saying debugging AI-written code takes longer than expected, and 66% naming the top frustration as answers that are "almost right, but not quite".

Almost right is the expensive failure. Obviously wrong costs five seconds. Almost right costs twenty minutes, three files away, after you have already built on top of it.

A field instead of a feeling

So DebugAI returns a verified field on every proposed fix, and it has three states, not two.

{
  "rank": 1,
  "title": "Add the missing import",
  "confidence": 88,
  "verified": true,
  "verification_reason": "ast.parse succeeded"
}

true means a mechanical check ran and passed.
false means a check ran and failed. The fix still comes back, with its confidence capped, because a failed check is information you want.
null means nothing checked it. The confidence is the model's own estimate and should be read as exactly that.

The rule I hold is that null never renders as false, and never renders as nothing at all. "Not checked" and "checked and fine" are different claims. Collapsing them is how a tool starts lying without anyone deciding to lie.

Here is the part where I lose some of you.

We check two kinds of bug. Out of all of them.

Version one of this covers exactly two classes.

Parse. SyntaxError and IndentationError. Python goes through ast.parse in-process. JavaScript goes to node --check in a temp directory, under a subprocess with a 2 second CPU limit and a 2 second wall clock. This proves the fix's syntax is valid. It proves nothing about whether the fix is correct.

Import. ImportError and ModuleNotFoundError. This one resolves the named import against the files DebugAI already retrieved for the request.

And now the sentence that belongs in every tool like this and appears in almost none of them. The import check does not prove the package is installed. The engine has no access to your real dependency graph. A verified: true on an import check means "this import matches the context we retrieved", not "this import will work when you run it".

That limitation is written in the source file, above the function, so nobody maintaining it can pretend otherwise later.

Everything outside those two classes returns null. Your TypeError on line 42 gets a diagnosis, ranked fixes, exact edits, and a null verification, because nothing mechanical checked it. Type-level checking is planned. It is not built, so it does not claim to be.

Total budget for all of this is 5 seconds across every candidate fix in one request. A check that does not finish returns null, not false. Failing to finish is not a verdict.

Why ship something this narrow

Because the alternative was shipping something broad that guesses, and a guess wearing a checkmark is worse than no checkmark.

A narrow honest signal is usable. You learn quickly that syntax and import fixes come back green, and that a hard concurrency bug comes back grey, and you calibrate. A broad dishonest signal teaches you nothing except to ignore the badge, which is where most confidence scores have already landed.

There is a second reason, less noble. Narrow is testable. Two classes with deterministic checkers can be verified in CI. "Our AI validates your fix" cannot.

The sandbox, and two things that did not work

Running node --check on model-written code means running a subprocess on input you did not write. That gets bounded.

What bounds it today: RLIMIT_CPU at 2 seconds, plus a wall-clock timeout on the subprocess call, plus a temp directory it cannot see out of.

What we tried and reverted, on 2026-07-09, empirically rather than by assumption:

RLIMIT_AS (address space) made node --check hang or abort with a uv_thread_create assertion failure on a trivial file, at every value from 128MB to 2048MB. V8 reserves a large virtual address space at startup regardless of actual heap use, so a hard AS cap fights Node's own initialization rather than the candidate code.

RLIMIT_NPROC is a per-real-UID limit on Linux, not a per-subprocess one. Setting it there capped process and thread creation for the entire user account the engine runs as. At 16 it broke Node's worker thread startup outright, with the same abort. At any value it is the wrong instrument: it throttles the whole process, not the sandboxed child.

There is no syscall-level network block, which is a real gap and is written down as one. It is acceptable only because node --check parses without executing the candidate's code, so nothing on this path can make an outbound call. The note in the source says plainly that if a future check needs to actually execute candidate code, resource limits alone are not sufficient and it needs a network namespace or a dedicated low-privilege UID first.

I am telling you the limits of my own sandbox because you are about to send it your stack traces, and you should know what it does before you do.

Using it

DebugAI runs as an MCP server, so any MCP client can call it.

npx -y @debugai/mcp setup

Browser sign-in, no key to copy. It writes config for every MCP client it finds (Claude Code, Claude Desktop, Cursor, Windsurf, Zed, Gemini CLI, Cline), backs up each file first, then checks the wiring actually works.

Look before it writes:

npx -y @debugai/mcp install --dry-run

Remove every trace:

npx -y @debugai/mcp uninstall

There is also a VS Code extension, which registers the same server, which is why the CLI skips VS Code by default rather than giving you every tool twice.

Free tier is 10 debugs a day, no card. Pro is $12/mo. A founding rate of $9/mo stays $9 for life and is claimable until September 1, 2026.

What I actually want you to take from this

Not the product. The field.

If you build anything that hands a developer a suggestion, the useful question is not "how confident is the model". It is "what, if anything, checked this", and the honest answer is frequently "nothing". Say that. A grey "not checked" next to a fix is worth more than a green 92% that means nobody looked.

I would rather show you a null than dress a guess as a result.

Originally published at debugai.io.