AI-Generated Tests: How Many Actually Test Different Behaviors?

Why I ran this

A coding agent can write a hundred passing tests before you finish your coffee. The test count goes up, the coverage report turns green, and the pull request looks thorough. What nobody tells you is how many of those tests check something the others don't.

I maintain vitest-dedup, a static analyzer that looks for duplicate Vitest tests, so I have an obvious interest here. That is also why I wanted to measure instead of assume. The hard part of this problem is not finding tests that look alike. It is not mistaking similar tests for equivalent ones, because the remedies are opposite: you delete a duplicate, but deleting the "similar" test that actually covers a different code path is how you lose coverage while the suite stays green.

So I set up a small experiment. A coding agent wrote 139 tests for two small modules, in two separate sessions. I classified every suspicious pair by hand and compared three detectors: a naive token-based clone detector, vitest-dedup 0.4.0, and vitest-dedup 0.5.0, which I released on October 17. Everything is reproducible; the commands are at the end. The answer was not the one I went in expecting.

Four ways two tests can be "the same"

Before counting anything, it helps to name what we are counting. Two tests can relate in four ways, and each one calls for a different action.

The first is an identical test. Same body, same context, maybe a different title or renamed variables. One copy can go.

The second is a near-duplicate. The bodies differ, but they perform the same behavior in a way a tool cannot prove equivalent. Someone has to look.

The third is a parameterization candidate. Same structure, different literal values. Both cases are probably needed, and they belong in one test.each table. Deleting one loses coverage.

The fourth is the one that causes trouble: tests that look similar, or even textually identical, but exercise different behavior. A tool that flags these is not just noisy. It is suggesting a deletion that removes a real check.

vitest-dedup encodes this split in its output. Every finding carries a verdict: redundant (one copy can go), consolidatable (merge into a table, keep every row), or a review prompt. That verdict matters more than the finding count, as the experiment shows.

Where a naive detector gets it wrong

The classic approach to clone detection works on tokens. Type-1 clones are token-for-token identical; type-2 clones are identical after replacing identifiers and literals with placeholders. I wrote a 60-line detector that does both on test callbacks, ignoring titles. It is a fair stand-in for what you get if you point a generic clone detector, or a quick script, at a test folder.

The examples in this section are hand-written traps, built to separate "similar" from "equivalent". The agent-written corpus comes later.

Here is the first trap. Two tests with identical text:

describe('frozen clock', () => {
  const clock = { now: () => 0 }

  it('keeps the bucket full', () => {
    const bucket = new TokenBucket(3, 1, clock)
    expect(bucket.available()).toBe(3)
  })
})

describe('clock that jumps an hour per read', () => {
  let t = 0
  const clock = { now: () => (t += 3_600_000) }

  it('keeps the bucket full', () => {
    const bucket = new TokenBucket(3, 1, clock)
    expect(bucket.available()).toBe(3)
  })
})

The first test checks that nothing refills when time stands still. The second checks that a huge refill is clamped at capacity. Both assert 3, for entirely different reasons. The naive detector reports them as exact duplicates. The bodies are the same text, but clock is a different binding in each one, and that binding is the test.

The same thing happens with hooks. Two it blocks with identical bodies, one under a beforeEach that creates new Cart(0.1) and one under new Cart(), are different tests. Type-1 flags them as well.

vitest-dedup 0.5.0 reports neither pair. Captured variables participate in the comparison at the place they are used, and they keep track of which enclosing variable they came from, even when the initial values happen to match. Hooks, mocks and fixtures are part of a test's context, so identical bodies under different contexts are not compared as equal.

The only pair vitest-dedup reports is the real duplicate: identical tests where one renamed cart to basket.

The opposite failure is just as common. Agents rename things. Here is a pair where everything except the behavior changed:

it('emits one event per order', () => {
  const events: string[] = []
  class SpyBus {
    publish(name: string) {
      events.push(name)
    }
  }
  const bus = new SpyBus()
  bus.publish('order.placed')
  expect(events).toEqual(['order.placed'])
})

it('publishes the order.placed event', () => {
  const published: string[] = []
  class RecordingBus {
    publish(topic: string) {
      published.push(topic)
    }
  }
  const recorder = new RecordingBus()
  recorder.publish('order.placed')
  expect(published).toEqual(['order.placed'])
})

A type-1 comparison misses this completely. vitest-dedup 0.4.0 saw the same sequence of behavior but could not prove the local classes equivalent, so it filed a near-duplicate-test warning for review. 0.5.0 models local classes: fields, methods, constructors, private names and what each instance captures. Consistent renaming stays comparable, so the verdict becomes a proven duplicate-test:

Why local classes and closures are the hard part

Test code is full of small local classes and closures: fake clocks, spy buses, recording repositories. They are exactly where "looks the same" and "is the same" drift apart.

A local class is not just its methods. Two classes with identical method text are still two different class objects, and a test that checks identity or instanceof can tell them apart.

A constructor can have effects at definition time and at construction time, and those are different moments. Collapsing them makes two tests look equal when one of them observes a side effect the other never triggers.

Property kinds matter too. These two objects print almost the same:

const fromGetter = { get limit() { return 3 } } // fromGetter.limit === 3
const fromMethod = { limit() { return 3 } } // fromMethod.limit is a function

Two tests that differ only in which of these they build were a duplicate-test error in 0.4.0. 0.5.0 keeps property descriptor kinds apart and reports nothing.

Closures add mutable state. If a test captures an array and a helper pushes into it, the comparison has to follow the history of that array, not only its initializer. If the captured object escapes into code the analyzer cannot see, any claim about it would be a guess.

That last point is the design decision I care most about. When the analyzer cannot model something, such as a prototype patched after an instance exists, a decorator, or reflective access, the dependent test is marked unresolved and excluded from comparison instead of being compared on surface text. The run is flagged as incomplete when that happens, because a missing finding is then no evidence of a missing duplicate. Unrelated tests in the same file still compare normally.

The experiment

Two small TypeScript modules: a shopping cart (items, coupons, tax, rounding to cents) and a token bucket rate limiter with an injectable clock. Each is 40 to 60 lines.

For each module, a fresh coding agent session (Claude) got a deliberately ordinary prompt: write a comprehensive Vitest suite for this module and make sure it passes. Then a second fresh session got the follow-up every team has written at some point: a reviewer says the tests are not thorough enough, add edge cases, keep the existing tests. I did not edit the generated files.

The result was 139 passing tests: 93 from the first sessions, 46 from the second. I added 10 hand-written trap tests, like the ones above, in a separate file.

Then I classified every pair that any detector flagged, and read both suites end to end looking for redundancy that nothing flagged. The question for each pair was simple: if one test were deleted, would any behavior lose its only check?

What the detectors found

On the 139 agent-written tests:

	Naive type-1	Naive type-2	vitest-dedup 0.4.0	vitest-dedup 0.5.0
Exact or renamed duplicates reported	0	can't tell them apart from literal variants	0	0
Similarity groups or findings	0	11 groups (34 tests)	19 findings	18 findings
Groups that mix unrelated tests	0	3 (they call different methods)	1 (a near-duplicate that isn't)	0
Table suggestions I would reject	n/a	n/a	2	2

The first surprise: the agent did not copy and paste. There was not a single exact or renamed duplicate in 139 tests. Its repetition was shape repetition, the same arrange, act, assert with different numbers.

vitest-dedup 0.5.0 reported 18 parameterization-candidate findings covering 31 tests, all as warnings with the remedy "consolidate into one test.each table that keeps every case as its own row". I agreed with 15, found one debatable, and rejected two.

A typical accepted one: three tax tests that differ only in rate, price and expected total.

// before: three tests with the same shape
it('applies tax to the subtotal', () => {
  const cart = new Cart(0.1)
  cart.addItem(item('a', 100))
  expect(cart.total()).toBe(110)
})
// ...'doubles the amount at a tax rate of 1', 'rounds the final total to cents'

// after: one table, every case kept as a named row
it.each([
  { case: 'applies tax to the subtotal', taxRate: 0.1, price: 100, expected: 110 },
  { case: 'doubles the amount at a tax rate of 1', taxRate: 1, price: 25, expected: 50 },
  { case: 'rounds the final total to cents', taxRate: 0.0825, price: 19.99, expected: 21.64 },
])('$case', ({ taxRate, price, expected }) => {
  const cart = new Cart(taxRate)
  cart.addItem(item('a', price))
  expect(cart.total()).toBe(expected)
})

The suite still runs 149 tests, all passing, and the two findings that involved these tests disappear.

The two I rejected are instructive. In one pair the cart receives SKUs 'a' and 'a'; in the other, 'a' and 'b'. Structurally that is a literal difference. Behaviorally it is the difference between merging two lines and summing two lines. A table would technically work, but it would hide the fact that the SKU column selects a code path. No finding claimed either test was deletable, so the cost of being wrong here is a bad refactor suggestion, not lost coverage.

0.4.0 reported the same 18, plus one near-duplicate between "accepts fractional capacity" and "can take exactly the full capacity". They are different behaviors; 0.5.0 no longer reports the pair.

The naive type-2 detector found roughly the same clusters, but replacing every identifier with a placeholder made cart.total() and cart.discount() look identical, so three of its eleven groups mixed tests that call different methods. It also cannot tell "these differ in a literal" from "these differ in which function they call", which is the difference between consolidate and leave alone.

The redundancy nobody flagged

By hand, I counted 10 tests that could be deleted without losing a check, about 7% of the suite. That number is smaller than I expected, and where those tests come from is more interesting than the count.

Some are subsumption: a later test performs the same steps with more checkpoints. The second session wrote a test for a clock moving backwards that walks through four timestamps, which makes two earlier backwards-clock tests from the first session redundant. They use different numbers and a different order, so no structural comparison connects them.

Some are degenerate dimensions. Asked for "interactions between features", the second session wrote tests like "is 0 with a 100% coupon and a tax rate of 1". Tax multiplies a total that is already zero, so the extra dimension cannot change the result; the test re-checks the 100% coupon case. Three tests followed this pattern, two of them from the second session. Seeing that requires knowing the arithmetic, which is outside what a static comparison of test bodies can claim. A rule of thumb for review: if the extra variable only multiplies or offsets a value the test pins at zero, the test re-checks its base case.

And two came from vitest-dedup itself, through a rule I ship off by default. subsumed-test reports when one test's behavior is a strict ordered subsequence of another's. With it enabled, it flagged three pairs. Two were correct and I had missed both by hand: "adds a new item" performs exactly the add and the two assertions that "does not change the cart when an add is rejected" also performs, so it adds nothing. The third was wrong: it treated validation of a negative fixed coupon as covered by validation of a negative percent coupon, which are different input classes. One wrong out of three is exactly why I ship it off by default. It is a good prompt for a review session and a bad CI gate.

In this run, five of the seven redundancy groups I found by hand involve a test from the second session. "Add more edge cases" produced as much re-testing as new testing. If you run agents in multiple passes over the same file, that is where to look.

So, how many?

About 93% of the agent's tests checked a behavior no other test covered, in this run. The number I would actually watch is a different one: 31 tests that wanted to be tables, and 10 that a reviewer would delete only after reading them carefully. A tool can find the first group on every pull request. The second still needs a person, and the most useful thing a tool can do there is say clearly which findings are proofs and which are questions.

Limitations worth knowing

vitest-dedup is static. It reads configuration and test files without importing or running them, which is what makes it safe to run on any pull request, and also what bounds it. When I cut 0.5.0, I chose to ship it with three known false-positive cases still open rather than hold back the constructor and captured-state work: asynchronous iteration, captured code that is opaque or too deep to follow, and the way parameterized object keys are encoded. I listed them in the 0.5.0 release notes so nobody has to discover them the hard way. The first one, for await loops, is already fixed on main and will ship in the next version.

My experiment is small and has an obvious bias: one agent, two modules, and I wrote both the tool and the classification. Treat the percentages as a description of this corpus, not a law. The raw outputs and my per-pair verdicts are in the repository so you can disagree with specific calls.

Gating only on new duplication

An existing suite usually cannot turn on a zero-tolerance gate at once. The baseline workflow records today's findings once with --update-baseline, then every later run with --baseline dedup-baseline.json gates only on what is added.

To test it, I recorded a baseline for the experiment and applied a change of the kind an agent produces when it does not see an earlier test: a renamed copy of an existing cart test in a new describe block. The gate failed on that one new duplicate and kept the 19 recorded findings visible but out of the decision:

The jsonl reporter is designed to be read by an agent, which closes the loop: the agent that wrote the tests can be the one that consolidates them. To run the gate in CI, the GitHub Action takes the same baseline as an input and verifies the binary's checksum and build provenance before running it.

Reproducing it

npm ci                                            # Node 24 LTS
npx vitest run                                    # 149 tests
npx vitest-dedup . --no-metrics                   # default rules
npx vitest-dedup . --no-metrics --rule subsumed-test=warning
node scripts/naive-dedup.mjs tests/*.test.ts      # naive token detector
npx vitest-dedup demo --no-metrics                # renamed local classes, 0.5.0
npx vitest-dedup@0.4.0 demo --no-metrics          # same pair, 0.4.0

The modules, the generated tests, the prompts, the naive detector and every raw output are in vitest-dedup-materials. vitest-dedup is MIT licensed and written in Rust; npx vitest-dedup . runs it without installing anything.

If you run it on a suite your agents wrote, I would like to know which of the four kinds you find most. My guess, after this run, is that the tables will outnumber the copies.