My AI Tests Passed While Half the Mutants Went Undetected

If someone asked me how I verified code written by AI, I used to have a simple answer: I ran the tests.

My project had 451 tests across 84 files. Claude Code had written them, and I had reviewed and merged them. A green test run felt reassuring.

Then a friend mentioned mutation testing. I tried it on ten files in src/lib that already had tests. The tool generated 550 small changes to the production code and checked whether the tests noticed.

Scope	Mutants	Killed	Survived	No coverage
Ten tested files in src/lib	550	273	161	116

The tests caught 49.6% of those mutants. Another 161 survived despite running tests, and 116 had no test coverage.

That does not mean half of my 451 tests were worthless. It means roughly half of the generated mutations in this sample went undetected. That was enough to make me inspect what the tests actually asserted.

Some tests only checked the source text

Here is an excerpt from my project, with the test description translated:

function source(relativePath: string): string {
  return readFileSync(
    fileURLToPath(new URL(`../${relativePath}`, import.meta.url)),
    'utf8',
  )
}

it('separates new time credits from the legacy user baseline', () => {
  const codingTimeRepository = source(
    'src/server/repository/coding-time-credit/index.ts',
  )

  expect(codingTimeRepository).toContain('userCodingTimeCreditLedger')
  expect(codingTimeRepository).not.toContain('projectId')
})

This reads a file as a string and checks for words. It does not execute the credit calculation or verify its result.

I found 62 tests like this across seven files. They can enforce a convention, but they cannot establish that the program behaves correctly. The implementation could return the wrong value while those checks stay green.

Other patterns deserved a closer look too:

• Copying the implementation's current output directly into the expected value, without checking whether that output matches the requirement.
• Reimplementing the production algorithm inside the test to calculate the expected result.
• Checking only that a mock was called. My project had 139 toHaveBeenCalled() assertions; the useful question was what each one established about behavior.
• Treating test-file count or coverage as proof that important decisions have assertions.

None of this is unique to AI. But when the same agent writes the implementation and its tests in one session, both can inherit the same misunderstanding.

A 2026 study, All Smoke, No Alarm, examined 86,156 test-file patches from 33,596 agent-authored pull requests. Its syntactic classification found weak or absent explicit oracle signals in 80.2% of patches. That is a result about observable assertion patterns in that dataset, not proof that 80.2% of all AI tests are useless. It does reinforce why counting tests is a poor substitute for inspecting their checks. Study

Mutation testing checks whether a change makes a test fail

A mutation tool changes the production code in small ways: replacing < with <=, changing + to -, forcing a condition to false, or replacing a return value.

Each modified version is a mutant. The tool runs the relevant tests against it.

Result	Meaning
Killed	At least one test failed with the mutation active
Survived	The tests still passed
No coverage	No test covered the mutated code

These are the three results I focused on. Tools also report timeouts, invalid mutations, and other states. Stryker's overall score counts detected mutants against valid mutants; its covered-code score uses a different denominator. Stryker states and metrics

Consider this function:

function isUserOldEnough(user: { age: number }) {
  return user.age >= 18
}

If a mutant changes >= 18 to > 18, tests using ages 17 and 19 still pass. A test for exactly 18 can distinguish the two implementations.

Coverage can tell me that a test executed the comparison. Mutation testing helps me find whether the assertion notices a meaningful change to it.

Eleven tests missed six boundary changes

My first run used StrykerJS with Vitest in a Next.js project. The configuration was small:

{
  "testRunner": "vitest",
  "plugins": ["@stryker-mutator/vitest-runner"],
  "mutate": ["src/lib/**/*.ts"],
  "disableTypeChecks": false,
  "incremental": true,
  "reporters": ["clear-text", "html"]
}

With Stryker and its Vitest runner installed and the test configuration available, I ran:

npx stryker run

That initial run took 1 minute and 5 seconds in my project.

The disableTypeChecks: false setting kept Stryker from inserting its usual @ts-nocheck directives. It did not make Vitest perform TypeScript checking; type validation still needs its own checker or command. Stryker configuration

The most memorable result came from kstRelativeTime, a utility that produces labels such as “just now,” “three hours ago,” and “yesterday.” Eleven tests exercised it. All six mutants that changed < to <= survived.

I had tests around the date boundaries, but not at the boundaries. None asked what should happen at exactly 48 hours. A branch that produced a number of weeks had no coverage at all.

Other results were harder to ignore:

• Changing + 1 to - 1 in a sequence number used for concurrent editing still passed.
• Replacing if (local === remote) with if (false) in text-merging logic still passed.
• The Markdown image parser killed only six out of 109 mutants.

Every one of those files already had tests. The CI result was green.

The source-text tests caused a different problem. Instrumentation changes the source file, so tests that inspect its literal text can fail because the tool inserted scaffolding. I excluded those tests from the mutation run. I wanted failures caused by changed behavior, not by the instrumentation's spelling.

Why this matters when agents write tests

Tests give a coding agent feedback. Weak assertions can tell the agent that it has finished even when an important behavior remains unchecked.

Mutation testing provides a more specific follow-up than “write more tests.” I can point to a surviving change and ask what input would distinguish the original behavior from the mutated behavior. Then I can review whether that expected result actually follows from the requirement.

Meta's Automated Compliance Hardener uses a related approach at a different scale. It generates concern-specific mutants with an LLM, then generates tests to catch faults that the existing suite misses. The reported deployment produced 9,095 mutants and 571 tests; engineers accepted 73% of tests in its Messenger and WhatsApp test-a-thons. The useful idea is that a new test can improve fault detection even when it exercises lines that existing tests already cover. Meta paper

The agent can help improve the tests, but I still need to decide what the program should do. Otherwise it can simply encode the same wrong assumption more thoroughly.

Keep the feedback close to the change

The obvious concern is runtime. A naive process that runs a five-minute suite for each of 1,000 mutants would take about 83 hours.

Modern tools avoid much of that repetition. Stryker can select tests using per-test coverage, and incremental mode reuses previous results where appropriate. That makes a small run on relevant logic a different proposition from mutating an entire repository every time. Stryker configuration, incremental mode

For Comwit and its project template, I adopted a few practical rules:

1. Start with application logic: utilities, server logic, APIs, and state. I excluded pages, routing, UI components, and database schemas from this particular setup.
2. Run mutation tests on changed files during normal work. Use a broader run occasionally to establish a baseline.
3. When an agent changes tests, have it run the changed-file mutation command and inspect survivors.
4. Review survivors by cause. A boundary change suggests an exact-value test; a skipped branch suggests a missing input path. Equivalent behavior or irrelevant logging changes need a different decision.

The command in my repository is pnpm test:mutation:changed. It is a project-specific script, not a built-in Stryker command.

I also ran a larger baseline: 902 logic files, 41,343 mutants, and about 49 minutes in my environment.

Area	Total mutants	Covered mutants	Survived	Score on covered mutants
All targeted logic	41,343	10,230	3,799	62.9%
src/server/repository	12,516	5,121	2,021	60.5%
src/server/domain-jobs	2,490	1,084	459	57.7%

The denominator matters. The 62.9% figure describes the covered mutants, not all 41,343 generated mutants. Only about a quarter of the generated mutants had coverage, and more than a third of those covered mutants survived.

Those numbers gave me places to investigate. They did not give me a reliable estimate of how many production bugs the application had.

The score is a prompt to investigate

A surviving mutant is not automatically a missing requirement. Some mutations leave observable behavior unchanged. Those equivalent mutants cannot be killed by a test that correctly checks the same behavior.

Mutation testing also cannot prove that the implementation matches the intended specification. It checks sensitivity to the generated changes. A test suite and implementation can agree with each other and still agree on the wrong behavior.

That is why I do not want the mutation score to become a team KPI. It would be easy to optimize the number and miss the reason for running the tool.

I still ask AI to write tests. What changed is the next question I ask: if this code were slightly wrong, which test would notice?

In my first run, one minute of mutation testing gave me a much more useful review queue than another reassuring green check.

---

I build Comwit, where this workflow is part of the project template. Adapted from my original Korean article on Velog, published October 7, 2026.

Contact: X @meursyphus.