DEV Community

iwadjp
iwadjp

Posted on AI-assisted

When AI writes the fix and the test together, is PASS enough?

#testing #node #git #ai

The problem

When an AI coding assistant writes a fix and a regression test in the same change, "the test passes now" doesn't tell you the test would have failed before the fix. By the time you're looking at the result, the working tree is already fixed — there's no easy way to go back and check.

Why PASS is not enough

git stash, checking out an old commit, or working from a second clone all work, but they're manual, easy to skip, and don't handle a dirty tree well (the fix may not even be committed yet). What you actually want is simple to state and annoying to do by hand: run this exact test against the code as it was before the fix, and against the code as it is now, and see if it goes FAIL then PASS.

What Timewitness does

Timewitness is a small Node.js CLI that does that. It takes your current working tree — dirty, uncommitted, whatever state it's in — and reconstructs an isolated copy of it as it was before you started (arm), then re-runs the test you specify against both that "before" copy and the current "after" state (prove). No commit, stash, or second clone required; your real working tree is never touched.

It reports one of three outcomes:

  • PROVEN — the test failed on the "before" copy and passed on the current one, with matching environment/dependencies and repeated runs agreeing.
  • NOT_PROVEN — before/after don't show the FAIL-then-PASS pattern (e.g. it already passed before the fix).
  • INCONCLUSIVE — something about the comparison couldn't be trusted (skipped tests, timeouts, environment drift, and so on), so it refuses to guess.

Small example 

node timewitness.cjs arm

# edit code and add/change a regression test as usual

node timewitness.cjs prove --test test/foo.test.js
Enter fullscreen mode Exit fullscreen mode 
TIMEWITNESS: PROVEN

Before: FAIL 2/2
After : PASS 2/2
Enter fullscreen mode Exit fullscreen mode

Real validation

The README documents a run against a real bug fix from a separate private project of mine (source and tests extracted into a synthetic, private-data-free reproduction environment):

TIMEWITNESS: PROVEN
Before: FAIL 3/3
After : PASS 3/3
Enter fullscreen mode Exit fullscreen mode

Alongside that, a synthetic negative control (a test unrelated to the fix) correctly comes back NOT_PROVEN:

TIMEWITNESS: NOT_PROVEN
Before: PASS 2/2
After : PASS 2/2
Reason: BEFORE_ALSO_PASSES
Enter fullscreen mode Exit fullscreen mode

Across those two runs, false PROVEN (a case where it should not have said PROVEN but did) occurred 0 times.

What PROVEN does not mean

PROVEN is a narrow claim, not overall correctness:

  • It doesn't prove the fix as a whole is correct.
  • It doesn't measure whether test coverage is adequate.
  • It doesn't mean the bug is completely fixed — only that this specific test distinguishes the before/after states, under the conditions actually tested.

Scope right now is Windows, Git, and Node's built-in node:test runner. 49 automated tests cover the isolation and comparison logic itself.

Source

GitHub: https://github.com/iwadjp/timewitness

I built Timewitness myself, mainly for my own use. Feedback and criticism — especially on the isolation approach — are welcome. More background (and four other small tools built for related "the final state alone doesn't tell you what happened" problems) is on my blog.

Disclosure: AI tools assisted with drafting and review of this post and parts of the project; I made the final editorial, code, and publication decisions.

Top comments (0)