DEV Community

Aleksandr Argunov
Aleksandr Argunov

Posted on Originally published at argbe.tech AI-assisted

AI-generated unit tests: two patterns that pass with the bug, and a red-green fix

#testing #ai #tdd #typescript

Two tests can pass while applyDiscount(100) returns 100 instead of the ticket's 90. One replaces the function with a mock. The other copies its boundary bug into the expected value.

Agents love to create a thousand unit tests. All green. Then review smashes the whole thing.

The agent already implemented a trash solution, and only then wrote generalized unit tests for it. Those tests describe the code it just wrote. So they all pass, bugs included.

When an agent writes both the code and its tests, don't count the green checks. Look at where each expected value comes from. If it comes from the agent's code, the test can repeat the agent's mistake instead of catching it. If it comes from the ticket, the spec or a real example, and the test runs the real code and failed because the behavior was missing, it checks something.

Two patterns that pass with the bug

Here are both in TypeScript with Vitest. Suppose an agent writes this for a discount ticket:

// pricing.ts. The ticket: orders of 100 or more get 10% off.
export function applyDiscount(total: number): number {
    return total > 100 ? total * 0.9 : total // bug: should be >=
}
Enter fullscreen mode Exit fullscreen mode

A test that mocks the function it's supposed to test:

// mocked.test.ts
import { expect, it, vi } from 'vitest'
import { applyDiscount } from './pricing'

vi.mock('./pricing', () => ({ applyDiscount: () => 90 }))

it('gives 10% off at 100', () => {
    expect(applyDiscount(100)).toBe(90) // checks the mock, not the code
})
Enter fullscreen mode Exit fullscreen mode

A test that calculates the expected value with the code's formula:

// same-formula.test.ts
import { expect, it } from 'vitest'
import { applyDiscount } from './pricing'

it('applies the discount', () => {
    const total = 100
    // the code's own formula, bug included
    const expected = total > 100 ? total * 0.9 : total
    expect(applyDiscount(total)).toBe(expected)
})
Enter fullscreen mode Exit fullscreen mode

Both pass, and the code still charges 100 for an order of exactly 100. With the mock, the real applyDiscount never runs. Mocking a dependency, like a payment service, is a different and legitimate use (Software Engineering at Google). Copying the formula copies the bug. Google's testing blog shows the same trap in Don't Put Logic in Tests.

Researchers have measured a version of this. In a July 2026 study, 11 models in 13 model and mode setups wrote tests for 318 Java methods behind 233 real bugs. Averaged across those setups, 3.84% of the tests written from the buggy code passed on the bug and failed on the fix. Written from the fixed code, 0.46% did (Zhao, Zhou and Cohen).

More tests don't fix it

In a study first posted in February 2026 on 500 SWE-bench Verified tasks, pushing agents to write more tests barely changed how many they solved (Chen et al.). GPT-5.2 solved 359 tasks either way, and Gemini 3 Pro Preview went from 371 to 366. Telling Kimi K2 Thinking and DeepSeek v3.2 Reasoner not to write tests moved them from 317 to 304 and from 300 to 291. None of these changes was statistically significant.

The cost depended on the model: GPT-5.2 made more API calls and used more tokens without solving more, and the two models told to skip tests used fewer (results). It's one benchmark and one agent scaffold, mini-SWE-agent, and the study didn't test TDD or the workflow below.

Red and green slices, then review

In my experience, TDD with small red and green slices, and reviewers after, works much better:

  1. Define what done means, in a way that can fail.
  2. Take the expected values from something the agent didn't write: the ticket, the spec, a real example.
  3. Write one failing test (red). It has to fail because the behavior is missing.
  4. Write only enough code to pass it (green).
  5. Next slice. Reviewers come in after green.

For the discount, the test takes its expected value straight from the ticket:

// spec.test.ts
import { expect, it } from 'vitest'
import { applyDiscount } from './pricing'

// From the ticket: an order of exactly 100 pays 90.
it('gives 10% off an order of exactly 100', () => {
    expect(applyDiscount(100)).toBe(90)
})
Enter fullscreen mode Exit fullscreen mode

Run against the agent's code, it fails red with expected 100 to be 90. Change > to >= and it goes green. The mock test still passes, because it never ran the code. The formula test now fails: it was guarding the bug. Checked with Vitest 4.1.12: save the four blocks under their file names and run npx vitest run.

Matt Pocock's /tdd skill is his version of this workflow. One slice at a time, and writing all the tests up front is an anti-pattern there. Refactoring belongs to review. That part is a choice: Kent Beck's Canon TDD lets you refactor inside each cycle, after green.

Lauren Tan (poteto), the author of pstack, builds step 1 into it. Its /figure-it-out skill asks the agent to state what done means, in a way that can fail, before it starts the run.

TDD and /figure-it-out can work magic when you understand what you are doing.

What each check covers

Even good unit tests are one row of this chart. Here's what every row covers. The When column is my rough rhythm, not a rule.

Animated chart titled What your 1,000 tests actually check. Ten rows show when each check runs and what it covers, across the screen, server and database of your app and an outside payment service. Unit, every edit: one function, alone. Mocks, inside unit tests: a fake stand-in for a real service. Smoke, every build: does it even start? Integration, every PR: do the parts work together? Regression, every PR: did the change break something old? Review, after green: a second pair of eyes, human or model. End-to-end, before release: a real user flow, start to finish. Security, before release: can someone get what isn't theirs? Mutation, rarely: plant a bug, see if any test notices. Set apart in gold at the bottom, Manual, by a human: actually uses it.

Check When What it covers Source
Unit Every edit A small unit of behavior, run fast. The unit can be one function or a few that work together. Fowler
Mocks In unit tests and narrow integration tests Stand-ins for real collaborators, like a payment service. A mock also checks how it was called. Fowler
Smoke Every build Does it start, and do the critical basics work, before deeper testing? ISTQB
Integration Every PR Do separately built parts work together once connected? Fowler
Regression Every PR Did the change break something that used to work? Unit, integration and end-to-end tests can all do this job. ISTQB
Review After green Someone other than the author reads the change: a person or a review model. Google, GitHub
End-to-end Before release A whole user flow, start to finish, in production-like conditions. ISTQB
Security Throughout development Whether the security controls hold. "Can someone get what isn't theirs?" is one question of many. OWASP, framework
Mutation Rarely Change the code on purpose and see whether any test fails. A surviving change isn't always a real bug. Stryker
Manual By a human A person uses the product and checks that it behaves as expected. Atlassian

The maintained article, with the animated chart, source links and the last-checked date, is on argbe.tech.

Top comments (1)

Collapse
 
launchgatecheck profile image
Launch Gate •

A cheap guard for the same-formula pattern: a mutation check. Flip the boundary in the code (> to >=, or change 0.9 to 0.91) and run the suite. If it stays green, the tests aren't pinning the behavior. Doing that by hand on the two or three functions that touch money takes minutes and shows quickly which green checks count. The other half is the boundary table itself: write the spec's values (99.99, 100, 100.01) as literal inputs and literal expected outputs taken from the ticket, so the expected column never passes through the code under test.