Your AI-written tests pass. That doesn't mean they work.

#javascript #testing #ai #webdev

Here's a function. Three branches, two thresholds.

export function classify(score) {
  if (score >= 80) return "healthy";
  if (score >= 50) return "warning";
  return "critical";
}

And here's a test suite for it, of the kind an AI assistant will happily produce:

test("returns a value for a high score", () => {
  expect(classify(95)).toBeDefined();
});

test("returns a value for a mid score", () => {
  expect(classify(60)).toBeDefined();
});

test("returns a value for a low score", () => {
  expect(classify(10)).toBeDefined();
});

Three tests. Every line executed. Coverage: 100%.

Now change >= to > on line 2. Change it to <. Replace the whole condition with true. Delete the second branch entirely.

The tests still pass. All of them. Every time.

Because toBeDefined() doesn't check what the function returned — only that it returned something. The tests execute the code. They never verify it.

Coverage measures the wrong thing

Coverage answers: did a test run this line?

The question you actually care about: would a test fail if this line were wrong?

Those are not the same question, and the gap between them is where bugs live. A line can be covered by twenty tests and still be completely unprotected.

This has always been true. What changed is the volume. When a human writes a weak assertion, it's usually one, and usually because they were tired. When an assistant generates a suite, the weak assertions arrive by the file — and they arrive looking correct. toBeDefined(), expect(fn).toHaveBeenCalled(), expect(result).toBeTruthy(). These read like tests. They pass code review. They light up the coverage report green.

Mutation testing asks the right question

Mutation testing changes your source code on purpose — flips a >= to >, turns a condition into true, negates a boolean — and re-runs your suite against each change.

If a test fails, good: something was actually checking that behaviour. The mutant is killed.

If every test still passes, that behaviour is executed but unverified. The mutant survived, and you've found a real gap.

It's a decades-old technique. The reason it isn't standard practice is cost: mutating a whole repository means running your test suite hundreds or thousands of times.

Scope it to the diff

You don't need to mutate the whole repository. You need to know whether the code in this pull request is actually tested.

That's the idea behind aitg: take the changed lines from git diff, mutate only those, and report on them. Minutes become seconds. It runs on StrykerJS, which does the actual mutation work.

npm install --save-dev @aitg/cli @stryker-mutator/core
npx aitg init --local   # no account, nothing uploaded
npx aitg scan

On the classifier above:

Mutation testing complete (vitest, 67 mutants).

Results
  Mutation score: 0%
  Killed:       0
  Survived:     8

Surviving mutants
  ! classify.js:2 (ConditionalExpression)
  ! classify.js:2 (EqualityOperator)
  ! classify.js:3 (ConditionalExpression)
  ! classify.js:3 (EqualityOperator)

100% coverage. 0% mutation score. Eight ways to break the code that no test would catch.

The output has to be actionable

A list of surviving mutants isn't useful on its own. So the tool writes a prompt you can paste into whatever assistant you use, with each gap explained:

- if (score >= 80) return "healthy";
+ score > 80

A comparison boundary was shifted (inclusive/exclusive) and no test noticed, so the exact edge value is untested. Assert the boundary itself, not just values on either side.

versus, on the same line:

- if (score >= 80) return "healthy";
+ score < 80

A comparison was reversed and no test noticed, so the direction of the check is unverified.

Those need different tests. >= to > is only distinguishable at exactly 80. >= to < is distinguishable almost anywhere. Telling a developer "an equality check was inverted" for the first case sends them to write the wrong test.

The explanation is derived from the actual mutation, not its category. This matters more than it sounds: an explanation that visibly contradicts the diff printed above it costs you trust in every other line of the report.

Uncovered code is a different problem

If nothing reaches a line, no mutant on that line can survive — it never runs. So a file with zero tests can look cleaner than a file with weak ones.

That's backwards, so uncovered code gets its own section, and the distinction is spelled out: a survivor means tighten an assertion that already exists; no-coverage means write a test that reaches the code at all.

What this doesn't do

It won't tell you your architecture is wrong, or that you're testing the wrong things entirely. It measures one specific property: whether your assertions would notice if the code changed.

It also needs a passing test suite to start from. Mutation testing compares mutated runs against a green baseline. No baseline, no signal.

And it's slower than coverage. Scoped to a diff it's seconds, but it's not free.

On testing the tool itself

A tool that measures test quality had better have decent tests. It has 54.

More to the point, those tests were validated by mutation rather than by counting coverage: six defects fixed during development were deliberately reintroduced, and each had to make tests fail.

One didn't. It checked that the diff engine compares against the merge-base rather than the branch tip — and it passed under both. The divergent commit in the fixture added a file, which shows up as a deletion when diffing against the tip, and deletions get skipped anyway. Both strategies produced identical visible results.

The test looked correct. It passed. It verified nothing.

It now modifies a shared file instead, and catches the mutation. But that's exactly the failure mode described at the top of this post, found in the suite of the tool built to detect it. If it can hide there, it can hide anywhere.

Try it

npx aitg init --local
npx aitg scan

Run it on your most recent PR. If the mutation score comes back near your coverage number, your tests are doing their job. If there's a gap, that gap is where the bugs get through.