AI writes more and more of our code, and code review can't keep up. So I built BuildProof: a tool that looks at a code change and tries to prove a bug, not just guess one.
How it works
- The AI reads the change and suspects a bug.
- It writes a test, and that test runs on the old code AND the new code.
- If the test passes on the old code and fails on the new one, that's a real behaviour change.
- A separate AI "judge" decides whether that change was intended or a bug.
- You only get bugs that come with a failing test as proof.
The test
BugsJS is a public dataset of real bugs from real projects (Express, ESLint, Hexo, Karma, Shields, Bower, hessian.js). I re-introduced each bug and let the hunter look at the change, with no hints.
Results
- Caught 32 of 44 real bugs.
- In the reverse direction (the actual fix, which is correct code), it raised a false alarm 4 times out of 44.
- Cost: about $0.15 per check.
It also knows when to stay quiet: I ran it on 4 open pull requests in lodash, koa and axios. It found nothing, and it was right not to. It doesn't invent bugs to look busy.
Your code runs in a locked container with no network and is deleted when the check ends. There's also a free mode that uses your own local model (Ollama).
Try it
30-second demo, no project needed:
npx buildproof try
On your own git project (3 free cloud checks):
npx buildproof hunt --base main --cloud
I'd really like feedback, especially on the misses. Which kinds of bugs should it catch that it doesn't?
Top comments (0)