I ran my code analyzer on 9 codebases it had never seen. It found 3 real bugs, and embarrassed me first.

moussaoutlook — Thu, 16 Jul 2026 14:31:31 +0000

A maintainer of Apache Superset merged one of my fixes the same day I filed it. "I can verify the fix is correct… CI's green now. LGTM." It was a one-line change: a TYPE_CHECKING import pointing at a module path that had quietly moved during a test-tree refactor, so every type-check of that file had been silently failing since. I didn't find it by reading Superset's code. A tool I've been building found it, and I'd never seen that repo before in my life.

That tool is DACIP, and this is the honest story of how it did on nine codebases I picked precisely because I had no idea what was inside them, including the part where it made me look foolish first.

The problem I actually have

I build with AI coding agents every day. The failure mode that kept burning me was never bad code, the agents write fine code, though. It was contract breaks. The agent renames a Flask route from /api/health to /api/healthz, the Python tests pass, the TypeScript compiles, CI goes green, then a React call site three files away now 404s in production. Nothing caught it, because nothing was looking at both sides at once.

Unit tests check one side. Type checkers check one language. AI reviewers have opinions about the diff in front of them, but they don't hold a model of "every place the frontend calls the backend" and check it against "every route the backend actually serves." That specific class of bug sails through green CI.

So DACIP does exactly that and nothing else: it extracts the routes your backend serves and the calls your frontend makes, from source, from both git refs of a pull request, and reports only what the change broke. Every finding names the orphaned caller at file:line and ships a command that reproduces it: fails on the branch, passes when fixed. No language model anywhere in the pipeline. Same two commits in, byte-identical report out, every single time.

That determinism is the whole pitch. But determinism is worthless if the output is wrong. So I needed to know: on code I'd never seen, how often does it cry wolf?

The claim that died on contact

My claim was "zero false positives." I'd tuned it on my own repos, where I knew every answer. Confident, I pointed it at redash (a BI platform), ~123 routes, a big React frontend, and it asserted 8 defects.

Seven were wrong.

Not subtly wrong. Wrong because redash wraps its route registration in a helper (add_org_resource) my extractor didn't understand, inherits HTTP handlers from local base classes I didn't follow, and builds URLs through wrapper calls I read as opaque. My tool didn't understand redash's dialect of Flask, so it hallucinated breaks that weren't there.

This is the moment most tools quietly fudge. You tune the thresholds until the false positives disappear on this repo, ship it, and call it "zero false positives", until the next repo, where it lies again. I did the opposite. Every false positive was a gap class, a specific pattern the extractor didn't model, and each one got a fix with a regression test pinning it shut. Nineteen of them, across the nine repos:

Monorepo source roots (Django living under api/, so dotted import paths resolve to a root that isn't the repo root)
Express servers living inside frontend/, app.post('/x', handler) is a route definition, not a frontend call
urlpatterns built by list concatenation, i18n_patterns(*urls), and non-constant mounts
drf-nested-routers, Flask-AppBuilder's @expose classes, PostHog's entirely custom RouterRegistry
…fourteen more, each dull, each real, each now regression-tested.

The rule I held: when the tool can't extract enough to be sure, it must say so, not guess. A repo where DACIP reports little is a repo where it tells you it sees little. That "coverage" line is part of every report, and it's the product as much as the findings are.

Where it landed: nine repos, zero false positives, three real bugs

After the gap classes were fixed, here's the final scorecard, every asserted defect hand-verified against the actual source by tracing the real route table and the real call site:

Repo	Routes extracted	Asserted defects
getredash/redash	149	1 (real)
apache/superset	307	1 (real, merged)
HumanSignal/label-studio	185	1 (real)
posthog/posthog	1,920	0
getsentry/sentry	151	0
zulip/zulip	445	0
netbox-community/netbox	393	0
Flagsmith/flagsmith	616	0
mathesar/mathesar	24	0

These nine repos don't just live in this table, they're re-scanned every night at pinned commits, with every scan run twice and the artifacts byte-compared. Tonight's numbers are live on the proof dashboard. (https://dacip.dev/proof/)

The three real bugs, all disclosed upstream before this post:

redash ships a client method calling GET /api/dashboards/recent with no backend route serves it. (Latent: the method has no callers today, which the issue says plainly. #7769)
superset's stale TYPE_CHECKING import, above. (#41972, merged same day.)
label-studio ships io_storages/filesystem.py importing a .base module that doesn't exist, ModuleNotFoundError on first import, zero current importers. A landmine, not a live crash. (#9815)

The part I want you to poke at

"Zero false positives" is a strong claim and you should be suspicious of it. So here's exactly what it means and doesn't:

It means every asserted defect, the P1/P2 findings with a full evidence chain, survived hand-verification against the code. It does not mean "found every bug." Coverage varies enormously: 99% of comparable calls matched on redash, near-zero comparable surface on mathesar and netbox. Custom frontend HTTP clients blind the linker until I teach it that idiom. There's a public gap list, Next.js Pages Router is partial, and some custom router registries each need their own extractor. (FastAPI landed after this batch, validated on the full-stack-fastapi-template.)

The distinction between "asserted defect" and "observation" isn't editorial taste, it's enforced by promotion gates in the pipeline. A finding only gets asserted if extraction was trustworthy enough in that area to stand behind it. Everything below that bar is labelled an observation and surfaced separately. An LLM can't tell you when it's guessing. A deterministic analyzer has no excuse not to.

That's the actual thesis. Not "our AI is smarter." The opposite: no AI, no temperature, no drift, and a hard rule that when it can't see, it says so. The false-positive count is zero because of that rule, not in spite of it.

Try to break it

DACIP runs as a GitHub App (a check on your PRs) and as a guardrail inside your coding agent (it denies contract-breaking edits before they land, with the blast radius in the denial). It's free for public repos and one private repo.

# inside any public GitHub repo clone — macOS arm64 / Linux x86_64
uvx dacip investigate "api contract audit"

If you run it on your codebase and it asserts something false, that's a bug in my tool and I want the report, the zero-FP claim is falsifiable on purpose. And if you maintain one of the repos above and think I got a finding wrong, tell me publicly and I'll correct the record. That's not a risk to the brand. That is the brand.