burhanarif472-lgtm

Posted on Oct 23

I built FixProof to verify whether AI-generated bug fixes are actually proven

#ai #testing #opensource #showdev

AI coding tools can generate a fix and tell us the bug is solved.

But there is an important question:

Did the regression test actually prove the fix?

That question led me to build FixProof, an MIT-licensed open-source developer tool for checking the evidence behind AI-generated bug fixes.

The idea

FixProof follows a simple proof chain:

Fail before → pass after → inspect test-integrity warnings → verdict

It runs the selected regression test against the committed code before the fix and again against the committed code after the fix.

A passing test after the change is not enough by itself.

FixProof also looks for warning signs such as:

tests being removed
tests being skipped
assertions being changed or potentially weakened
incomplete or ambiguous test evidence

The goal is to make it harder for a change to appear verified when the regression test never properly reproduced the original problem.

Four possible results

FixProof reports one of four outcomes:

verified — the regression was reproduced before the fix and passes afterward with no integrity warnings
failed — the final test still fails
inconclusive — there is not enough reliable evidence to prove the fix
needs-review — fail-before/pass-after evidence exists, but test-integrity warnings need human review

Supported test runners

FixProof v1.0 currently supports:

Node.js node:test
Jest
Vitest
pytest
explicitly configured trusted project commands

It also includes GitHub Actions integration and a browser-based report viewer.

Example

A basic verification looks like this:

node bin/fixproof.mjs verify \
  --repo ../my-project \
  --base HEAD~1 \
  --head HEAD \
  --test tests/regression.test.mjs \
  --output fixproof-report.json

FixProof creates a report describing what happened before and after the change and any integrity warnings it found.

The report can then be opened in the browser viewer.

What FixProof does not claim

FixProof does not claim that a verified result means the entire application is bug-free.

It verifies a much narrower question:

Does the selected regression evidence reliably demonstrate this particular fix within the tested scope and environment?

I think keeping that boundary explicit is important, especially when AI coding agents are involved.

Try it

The project has now reached v1.0.0.

GitHub:
https://github.com/burhanarif472-lgtm/fixproof

Live demo:
https://burhanarif472-lgtm.github.io/fixproof/

The project is open source under the MIT License.

Feedback wanted

I’m now looking for developers who use AI-assisted coding tools to try FixProof on real bug fixes.

I’d especially like to learn:

where verification becomes inconclusive
which real-world project setups are currently difficult
whether the reports explain the evidence clearly
which integrations would make the tool more useful

Finding the limitations is more valuable to me right now than simply collecting stars.

If you try it, I’d appreciate hearing what works, what breaks, and what you would change.

DEV Community