An agent wrote 14 tests for a module I knew was broken. Mutation score came back 41%

Last month I ran a small experiment on a checkout module I already knew was broken.

The module applies tiered discounts to an order total. The bug is boring: the tier discount gets applied twice above a certain subtotal, and the coupon argument is read but never used. It survived two code reviews because the output still looks like money and the difference only shows up on bigger carts.

I gave an agent read access to that one file and asked for a test suite. Nothing else in the prompt, no examples, no description of expected behavior. It came back with 14 tests. All green on the first run. Line coverage for the file went from nothing to 92%. The bug shipped anyway.

Martin Fowler's site published a piece last week called "TDD inside the agent loop - theater or actual value?", and reading it is what pushed me to stop arguing from feeling and write down my own numbers. So here they are.

The file under test

const TIERS = [
  { min: 0, off: 0 },
  { min: 500, off: 0.05 },
  { min: 1000, off: 0.12 },
];

export function orderTotal(items: Item[], coupon?: Coupon) {
  const subtotal = items.reduce((acc, i) => acc + i.price * i.qty, 0);
  const tier = TIERS.filter((t) => subtotal >= t.min).pop()!;

  let total = subtotal * (1 - tier.off);

  if (coupon && subtotal >= 1000) {
    total = total * (1 - tier.off); // the tier lands a second time
  }

  return Math.round(total * 100) / 100;
}

What the agent gave me

Two of the 14 tests were fine. The rest looked like this:

describe('orderTotal', () => {
  it('returns a number', () => {
    expect(typeof orderTotal([{ price: 10, qty: 1 }])).toBe('number');
  });

  it('applies the 5% tier', () => {
    const items = [{ price: 500, qty: 1 }];
    expect(orderTotal(items)).toBe(500 * (1 - 0.05));
  });

  it('handles a coupon', () => {
    const items = [{ price: 1200, qty: 1 }];
    expect(orderTotal(items, { code: 'X10' })).toBe(929.28);
  });
});

Look at the last one. 929.28 is the double discount. The agent read the implementation, computed what the code currently does, and froze it as the expected value. The test is green because the bug is now documented as a requirement.

This is the part that no coverage report will ever tell you. A test written from the implementation can only confirm the implementation.

The mutation run

npx stryker run --mutate 'src/checkout/order-total.ts'

-----------------|---------|----------|----------|-----------|---------|
File             | % score | # killed | # surviv | # timeout | # error |
-----------------|---------|----------|----------|-----------|---------|
order-total.ts   |   41.37 |       12 |       17 |         0 |       0 |
-----------------|---------|----------|----------|-----------|---------|

92% of the lines executed, 41.37% of the mutants killed. Twelve dead, seventeen alive. Two of the survivors say the whole story:

Survived mutant #7 (ConditionalExpression)
-  if (coupon && subtotal >= 1000) {
+  if (coupon && true) {

Survived mutant #12 (EqualityOperator)
-  TIERS.filter((t) => subtotal >= t.min)
+  TIERS.filter((t) => subtotal > t.min)

Mutant #12 flips >= into > and nothing in the suite notices, which means no test ever hits a subtotal of exactly 500 or exactly 1000. Fourteen tests and not one boundary. Mutant #7 removes the subtotal condition entirely and the suite stays green, because the coupon test asserts the broken output.

Two things I tried that went nowhere

First I did the obvious thing and asked for more. "Improve the test suite, target 95% coverage." It wrote six extra tests, coverage went to about 96%, mutation score moved to roughly 47%. More assertions on shape, more toBeDefined, same blind spots. Raising the coverage gate rewarded exactly the behavior I was trying to kill.

Then I put the theory in the prompt: "write boundary tests, this suite has weak mutation coverage." The agent produced tests that look like boundary tests, with subtotals of 499, 500 and 501, and it filled every expected value by running the current code in its head. Green again. The vocabulary changed, the epistemology did not.

What moved the number

Two changes, both of them about what the agent is allowed to read.

Feed the surviving mutants back in as the task. Stryker writes a JSON report, so the loop becomes "here is mutant #12, write a test that kills it" instead of "write tests". Two rounds of that took the score from 41% to somewhere near 78%.
Make the red step mandatory and machine-checked. The agent may not see the implementation when writing the test, and the test has to fail against the current code before anyone looks at it.

The second one is three lines in a pre-commit hook and it is the only part of this I would call TDD:

# a new test must be able to fail on the unfixed file
git stash push -- src/checkout/order-total.ts
if npx jest src/checkout/order-total.spec.ts --silent; then
  echo "test passes on the broken implementation - rejected"
  git stash pop && exit 1
fi
git stash pop

Crude, and it only works when the fix and the test arrive in the same change. But it caught four tests in the following week that were green against code they were supposed to break.

The cost, honestly

Mutation runs are slow. That single file takes about 40 seconds, the full suite takes something close to 11 minutes on our runner, so it only runs on changed files in CI and on a nightly job for the payment paths. I have no idea whether the mutants-as-input loop holds up on a codebase with heavy mocking, because ours has fairly little of it. If your suite mocks the module under test, most mutants die for the wrong reason and the score lies in the other direction.

What I stopped believing is the green board. An agent writing tests after reading the implementation produces a very convincing photograph of the bug you already have. Coverage measures which lines ran. Mutation measures whether anyone would have noticed if those lines were wrong, and the gap between 92 and 41 is where the money leaked.

So, a real question: for those of you running agents inside a TDD loop, how do you stop the model from deriving the expected value from the code it is looking at? Hiding the implementation behind an interface helped a bit here, and I would like to hear what else works before I turn this into a rule for the whole repo.

I write about this kind of thing on the Revin blog: https://revin.com.br/en

DEV Community