DEV Community: Luiz Figueiredo

AI-Generated Tests: How Many Actually Test Different Behaviors?

Luiz Figueiredo — Sun, 18 Oct 2026 11:57:00 +0000

Why I ran this

A coding agent can write a hundred passing tests before you finish your coffee. The test count goes up, the coverage report turns green, and the pull request looks thorough. What nobody tells you is how many of those tests check something the others don't.

I maintain vitest-dedup, a static analyzer that looks for duplicate Vitest tests, so I have an obvious interest here. That is also why I wanted to measure instead of assume. The hard part of this problem is not finding tests that look alike. It is not mistaking similar tests for equivalent ones, because the remedies are opposite: you delete a duplicate, but deleting the "similar" test that actually covers a different code path is how you lose coverage while the suite stays green.

So I set up a small experiment. A coding agent wrote 139 tests for two small modules, in two separate sessions. I classified every suspicious pair by hand and compared three detectors: a naive token-based clone detector, vitest-dedup 0.4.0, and vitest-dedup 0.5.0, which I released on October 17. Everything is reproducible; the commands are at the end. The answer was not the one I went in expecting.

Four ways two tests can be "the same"

Before counting anything, it helps to name what we are counting. Two tests can relate in four ways, and each one calls for a different action.

The first is an identical test. Same body, same context, maybe a different title or renamed variables. One copy can go.

The second is a near-duplicate. The bodies differ, but they perform the same behavior in a way a tool cannot prove equivalent. Someone has to look.

The third is a parameterization candidate. Same structure, different literal values. Both cases are probably needed, and they belong in one test.each table. Deleting one loses coverage.

The fourth is the one that causes trouble: tests that look similar, or even textually identical, but exercise different behavior. A tool that flags these is not just noisy. It is suggesting a deletion that removes a real check.

vitest-dedup encodes this split in its output. Every finding carries a verdict: redundant (one copy can go), consolidatable (merge into a table, keep every row), or a review prompt. That verdict matters more than the finding count, as the experiment shows.

Where a naive detector gets it wrong

The classic approach to clone detection works on tokens. Type-1 clones are token-for-token identical; type-2 clones are identical after replacing identifiers and literals with placeholders. I wrote a 60-line detector that does both on test callbacks, ignoring titles. It is a fair stand-in for what you get if you point a generic clone detector, or a quick script, at a test folder.

The examples in this section are hand-written traps, built to separate "similar" from "equivalent". The agent-written corpus comes later.

Here is the first trap. Two tests with identical text:

describe('frozen clock', () => {
  const clock = { now: () => 0 }

  it('keeps the bucket full', () => {
    const bucket = new TokenBucket(3, 1, clock)
    expect(bucket.available()).toBe(3)
  })
})

describe('clock that jumps an hour per read', () => {
  let t = 0
  const clock = { now: () => (t += 3_600_000) }

  it('keeps the bucket full', () => {
    const bucket = new TokenBucket(3, 1, clock)
    expect(bucket.available()).toBe(3)
  })
})

The first test checks that nothing refills when time stands still. The second checks that a huge refill is clamped at capacity. Both assert 3, for entirely different reasons. The naive detector reports them as exact duplicates. The bodies are the same text, but clock is a different binding in each one, and that binding is the test.

The same thing happens with hooks. Two it blocks with identical bodies, one under a beforeEach that creates new Cart(0.1) and one under new Cart(), are different tests. Type-1 flags them as well.

vitest-dedup 0.5.0 reports neither pair. Captured variables participate in the comparison at the place they are used, and they keep track of which enclosing variable they came from, even when the initial values happen to match. Hooks, mocks and fixtures are part of a test's context, so identical bodies under different contexts are not compared as equal.

The only pair vitest-dedup reports is the real duplicate: identical tests where one renamed cart to basket.

The opposite failure is just as common. Agents rename things. Here is a pair where everything except the behavior changed:

it('emits one event per order', () => {
  const events: string[] = []
  class SpyBus {
    publish(name: string) {
      events.push(name)
    }
  }
  const bus = new SpyBus()
  bus.publish('order.placed')
  expect(events).toEqual(['order.placed'])
})

it('publishes the order.placed event', () => {
  const published: string[] = []
  class RecordingBus {
    publish(topic: string) {
      published.push(topic)
    }
  }
  const recorder = new RecordingBus()
  recorder.publish('order.placed')
  expect(published).toEqual(['order.placed'])
})

A type-1 comparison misses this completely. vitest-dedup 0.4.0 saw the same sequence of behavior but could not prove the local classes equivalent, so it filed a near-duplicate-test warning for review. 0.5.0 models local classes: fields, methods, constructors, private names and what each instance captures. Consistent renaming stays comparable, so the verdict becomes a proven duplicate-test:

Why local classes and closures are the hard part

Test code is full of small local classes and closures: fake clocks, spy buses, recording repositories. They are exactly where "looks the same" and "is the same" drift apart.

A local class is not just its methods. Two classes with identical method text are still two different class objects, and a test that checks identity or instanceof can tell them apart.

A constructor can have effects at definition time and at construction time, and those are different moments. Collapsing them makes two tests look equal when one of them observes a side effect the other never triggers.

Property kinds matter too. These two objects print almost the same:

const fromGetter = { get limit() { return 3 } } // fromGetter.limit === 3
const fromMethod = { limit() { return 3 } } // fromMethod.limit is a function

Two tests that differ only in which of these they build were a duplicate-test error in 0.4.0. 0.5.0 keeps property descriptor kinds apart and reports nothing.

Closures add mutable state. If a test captures an array and a helper pushes into it, the comparison has to follow the history of that array, not only its initializer. If the captured object escapes into code the analyzer cannot see, any claim about it would be a guess.

That last point is the design decision I care most about. When the analyzer cannot model something, such as a prototype patched after an instance exists, a decorator, or reflective access, the dependent test is marked unresolved and excluded from comparison instead of being compared on surface text. The run is flagged as incomplete when that happens, because a missing finding is then no evidence of a missing duplicate. Unrelated tests in the same file still compare normally.

The experiment

Two small TypeScript modules: a shopping cart (items, coupons, tax, rounding to cents) and a token bucket rate limiter with an injectable clock. Each is 40 to 60 lines.

For each module, a fresh coding agent session (Claude) got a deliberately ordinary prompt: write a comprehensive Vitest suite for this module and make sure it passes. Then a second fresh session got the follow-up every team has written at some point: a reviewer says the tests are not thorough enough, add edge cases, keep the existing tests. I did not edit the generated files.

The result was 139 passing tests: 93 from the first sessions, 46 from the second. I added 10 hand-written trap tests, like the ones above, in a separate file.

Then I classified every pair that any detector flagged, and read both suites end to end looking for redundancy that nothing flagged. The question for each pair was simple: if one test were deleted, would any behavior lose its only check?

What the detectors found

On the 139 agent-written tests:

	Naive type-1	Naive type-2	vitest-dedup 0.4.0	vitest-dedup 0.5.0
Exact or renamed duplicates reported	0	can't tell them apart from literal variants	0	0
Similarity groups or findings	0	11 groups (34 tests)	19 findings	18 findings
Groups that mix unrelated tests	0	3 (they call different methods)	1 (a near-duplicate that isn't)	0
Table suggestions I would reject	n/a	n/a	2	2

The first surprise: the agent did not copy and paste. There was not a single exact or renamed duplicate in 139 tests. Its repetition was shape repetition, the same arrange, act, assert with different numbers.

vitest-dedup 0.5.0 reported 18 parameterization-candidate findings covering 31 tests, all as warnings with the remedy "consolidate into one test.each table that keeps every case as its own row". I agreed with 15, found one debatable, and rejected two.

A typical accepted one: three tax tests that differ only in rate, price and expected total.

// before: three tests with the same shape
it('applies tax to the subtotal', () => {
  const cart = new Cart(0.1)
  cart.addItem(item('a', 100))
  expect(cart.total()).toBe(110)
})
// ...'doubles the amount at a tax rate of 1', 'rounds the final total to cents'

// after: one table, every case kept as a named row
it.each([
  { case: 'applies tax to the subtotal', taxRate: 0.1, price: 100, expected: 110 },
  { case: 'doubles the amount at a tax rate of 1', taxRate: 1, price: 25, expected: 50 },
  { case: 'rounds the final total to cents', taxRate: 0.0825, price: 19.99, expected: 21.64 },
])('$case', ({ taxRate, price, expected }) => {
  const cart = new Cart(taxRate)
  cart.addItem(item('a', price))
  expect(cart.total()).toBe(expected)
})

The suite still runs 149 tests, all passing, and the two findings that involved these tests disappear.

The two I rejected are instructive. In one pair the cart receives SKUs 'a' and 'a'; in the other, 'a' and 'b'. Structurally that is a literal difference. Behaviorally it is the difference between merging two lines and summing two lines. A table would technically work, but it would hide the fact that the SKU column selects a code path. No finding claimed either test was deletable, so the cost of being wrong here is a bad refactor suggestion, not lost coverage.

0.4.0 reported the same 18, plus one near-duplicate between "accepts fractional capacity" and "can take exactly the full capacity". They are different behaviors; 0.5.0 no longer reports the pair.

The naive type-2 detector found roughly the same clusters, but replacing every identifier with a placeholder made cart.total() and cart.discount() look identical, so three of its eleven groups mixed tests that call different methods. It also cannot tell "these differ in a literal" from "these differ in which function they call", which is the difference between consolidate and leave alone.

The redundancy nobody flagged

I counted 11 tests that could be deleted without losing a check, about 8% of the suite. That number is smaller than I expected, and where those tests come from is more interesting than the count.

Some are subsumption: a later test performs the same steps with more checkpoints. The second session wrote a test for a clock moving backwards that walks through four timestamps, which makes two earlier backwards-clock tests from the first session redundant. They use different numbers and a different order, so no structural comparison connects them.

Some are degenerate dimensions. Asked for "interactions between features", the second session wrote tests like "is 0 with a 100% coupon and a tax rate of 1". Tax multiplies a total that is already zero, so the extra dimension cannot change the result; the test re-checks the 100% coupon case. Three tests followed this pattern, two of them from the second session. Seeing that requires knowing the arithmetic, which is outside what a static comparison of test bodies can claim. A rule of thumb for review: if the extra variable only multiplies or offsets a value the test pins at zero, the test re-checks its base case.

And three came from vitest-dedup itself, through a rule I ship off by default. subsumed-test reports when one test's behavior is a strict ordered subsequence of another's. With it enabled, it flagged three pairs, and all three hold up. Two I had missed by hand: "adds a new item" performs exactly the add and the two assertions that "does not change the cart when an add is rejected" also performs, so it adds nothing. The third I first rejected, because a negative fixed coupon and a negative percent coupon looked like different input classes. They reach the same value < 0 branch, and the percent test also asserts the message, so by my own criterion the fixed one is redundant. A reader pointed this out in the comments; I had judged that pair by the spec and the others by code path. The rule stays off by default because its findings are often distinct tests rather than redundant ones; this small corpus happened to be clean. It is a good prompt for a review session and a bad CI gate.

In this run, five of the seven redundancy groups I found by hand involve a test from the second session. "Add more edge cases" produced as much re-testing as new testing. If you run agents in multiple passes over the same file, that is where to look.

So, how many?

About 92% of the agent's tests checked a behavior no other test covered, in this run. The number I would actually watch is a different one: 31 tests that wanted to be tables, and 11 that a reviewer would delete only after reading them carefully. A tool can find the first group on every pull request. The second still needs a person, and the most useful thing a tool can do there is say clearly which findings are proofs and which are questions.

Limitations worth knowing

vitest-dedup is static. It reads configuration and test files without importing or running them, which is what makes it safe to run on any pull request, and also what bounds it. When I cut 0.5.0, I chose to ship it with three known false-positive cases still open rather than hold back the constructor and captured-state work: asynchronous iteration, captured code that is opaque or too deep to follow, and the way parameterized object keys are encoded. I listed them in the 0.5.0 release notes so nobody has to discover them the hard way. The first one, for await loops, is already fixed on main and will ship in the next version.

My experiment is small and has an obvious bias: one agent, two modules, and I wrote both the tool and the classification. Treat the percentages as a description of this corpus, not a law. The raw outputs and my per-pair verdicts are in the repository so you can disagree with specific calls.

Gating only on new duplication

An existing suite usually cannot turn on a zero-tolerance gate at once. The baseline workflow records today's findings once with --update-baseline, then every later run with --baseline dedup-baseline.json gates only on what is added.

To test it, I recorded a baseline for the experiment and applied a change of the kind an agent produces when it does not see an earlier test: a renamed copy of an existing cart test in a new describe block. The gate failed on that one new duplicate and kept the 19 recorded findings visible but out of the decision:

The jsonl reporter is designed to be read by an agent, which closes the loop: the agent that wrote the tests can be the one that consolidates them. To run the gate in CI, the GitHub Action takes the same baseline as an input and verifies the binary's checksum and build provenance before running it.

Reproducing it

npm ci                                            # Node 24 LTS
npx vitest run                                    # 149 tests
npx vitest-dedup . --no-metrics                   # default rules
npx vitest-dedup . --no-metrics --rule subsumed-test=warning
node scripts/naive-dedup.mjs tests/*.test.ts      # naive token detector
npx vitest-dedup demo --no-metrics                # renamed local classes, 0.5.0
npx vitest-dedup@0.4.0 demo --no-metrics          # same pair, 0.4.0

The modules, the generated tests, the prompts, the naive detector and every raw output are in vitest-dedup-materials. vitest-dedup is MIT licensed and written in Rust; npx vitest-dedup . runs it without installing anything.

If you run it on a suite your agents wrote, I would like to know which of the four kinds you find most. My guess, after this run, is that the tables will outnumber the copies.

Your duplication report says 0%. Your Cucumber steps disagree.

Luiz Figueiredo — Wed, 14 Oct 2026 18:31:44 +0000

Every Cucumber suite I've reviewed grows the same way. A new scenario needs a step, nobody is quite sure whether a matching one already exists, and so a slightly different one gets added. Multiply that by a few engineers and a few months, and the step library quietly fills up with near-copies.

After years of pointing this out in code reviews, I built a tool to do it for me. This post covers why I built CukeDedup, what it checks, and what running it against 92 open-source repositories taught me about my own assumptions.

Three ways to log in

Here's a pattern I've repeatedly encountered:

Given('the user is logged in', async () => {
  await login();
});

Given('the user is authenticated', async () => {
  await login();
});

Given('an authenticated user', async () => {
  await login();
});

All three can work. A generic copy-paste detector might report no duplicated blocks. And yet, do you really need three of them?

Meet CukeDedup

CukeDedup is an open-source static analyzer that finds duplicate, ambiguous, reusable and unused Cucumber step definitions. No config file, one command:

npx cuke-dedup .

It works with Cucumber.js, Playwright-BDD, Cypress Cucumber, Jest-Cucumber and the Vitest Cucumber Plugin, and reads step definitions written in JavaScript or TypeScript.

Why I built it

As a frontend lead, I helped introduce end-to-end testing. We picked Cucumber and Gherkin on purpose: I wanted QA and product people to read, write and argue about scenarios without touching the automation code.

That collaboration model worked in my experience. But an old problem remained: writing a new test is easy, keeping a large suite organized is not.

People pick up Playwright or Cypress quickly. What takes time is knowing what already exists. Before adding a step, you need to know whether a similar one is already there, or whether two existing ones could become one parameterized step. Experienced engineers can often spot this quickly. Someone new to the codebase may not, so the first time they hear about it is in code review.

As the reviewer, that's where my time went. I'm a big believer in linters for exactly this reason: if a rule matters, a tool should check it before the pull request, not a person after it. It's faster for the author, less tedious for the reviewer, and rules stop depending on who happens to review.

Code duplication is not behavior duplication

On one project I combined ESLint, Danger.js and jscpd. jscpd was great at catching copy-pasted code. It could not catch this:

When('I create a red project', async () => {
  await createProject('red');
});

When('I create a blue project', async () => {
  await createProject('blue');
});

which might be better expressed as:

When('I create a {word} project', async color => {
  await createProject(color);
});

The opposite happens too: two steps can read almost the same and still mean different things in your domain. Generic duplicate detectors look at code. I needed something that looks at both the step text and the implementation behind it.

It started as a pile of scripts on my machine. When I joined the effort to build a new E2E setup at my current job, I knew I'd hit the same problems again, so I turned the scripts into a proper tool. I wanted it open source from day one, and I wanted to finally build something non-trivial in Rust: a tool you run all the time has to be fast and light on memory.

Three things I cared about

1. Zero config. jscpd gets this right: you type jscpd . and you learn something. Nobody looking for duplicates wants to learn a new configuration format first. CukeDedup respects .gitignore, recognizes common project layouts and the Cucumber.js, Playwright-BDD and Cypress configs. Configuration exists for exceptions, not as an entry fee.

2. Evidence, not verdicts. "These two look similar" isn't useful. Findings include source locations, relevant matcher or handler evidence, and a suggested action when applicable. The decision stays with you: similar code can represent different domain concepts, and not every abstraction is a good one. To be clear, it doesn't understand English. It doesn't know logged in means authenticated; it can report that their handlers are equivalent under its supported static comparison rules.

3. Saying "I don't know." Static analysis can't resolve everything: dynamic config, wrappers, imports it can't follow. When that happens CukeDedup marks the run as incomplete instead of reporting a clean result. Zero findings is not zero problems, and --fail-on-incomplete lets CI treat that difference seriously.

The day-to-day loop is the same command again:

What 92 open-source repos taught me

A tool tested only on examples you wrote yourself will always look correct. So I ran it against 92 public JavaScript/TypeScript repositories that use Cucumber-style tests. This was not a representative sample of the ecosystem; it was to meet code I'd never seen. Many of the most useful discoveries were limitations in CukeDedup itself, rather than confirmed defects in the projects I scanned.

Consistent naming looks like duplication. In one project, every near-duplicate-step finding had matcher similarity above 0.90, because the team used a tidy naming convention for steps acting on different elements. Text similarity alone wasn't enough, so I raised the required handler similarity from 0.50 to 0.70 (configurable). Findings for that rule dropped by about half in the measured corpus, while the expected cases in the regression suite continued to pass. That is a reduction in findings, not a measured 50% improvement in precision.
Erasing literals makes unrelated code look alike. Tiny handlers that just call cy.log(...) were being suggested as parameterization candidates. Requiring compatible matchers, no negation conflict (not, un-…) and a single contiguous difference reduced that rule's findings by about 40% in the measured corpus, with expected regression cases preserved. I don't want a tool that pushes abstraction for its own sake.
"Ambiguous" was really a scope problem. Two projects produced 2,263 and 1,563 ambiguous-step findings when scanned at a root that combined separate runtime suites. They ran their suites as separate runtime units; I was analyzing them as one registry. When the affected suites were scanned within their actual runtime boundaries, those cross-suite ambiguities disappeared. The fix wasn't an algorithm, it was understanding how the tests actually run.
A bundled jQuery made it hang for 20 minutes. File size wasn't the cause: an 89 KB real file hung while a 1 MB synthetic one finished in seconds. The culprit was superlinear work in the analysis itself. After the algorithmic fix, that jQuery input took 0.6 seconds in the recorded measurement. Separately, files excluded by the generated-content filter are reported, and the run is marked incomplete.

When an issue can be isolated, I try to turn it into a minimal regression case in a corpus of expected behavior, with both "must find" and "must not find" cases. It describes what the tool should do rather than freezing whatever it currently does, and I deliberately break the code to check that each test actually fails. A test that stays green when you break things isn't protecting anything.

Using it on a real project

Install from npm (prebuilt native binaries) or build from source with Cargo:

npm install --save-dev cuke-dedup
# or
cargo install cuke-dedup

Reports come as terminal output, HTML (searchable, side-by-side comparison), JSON, JSONL and SARIF for GitHub code scanning.

Legacy suites: start with a baseline. Nobody can clean up years of duplication before adopting a tool. Record what exists today, then configure the gate to fail on new findings:

cuke-dedup . --baseline baseline.json --update-baseline
cuke-dedup . --baseline baseline.json --fail-on-new

The first goal isn't paying off the debt. It's not adding to it.

In CI, there's a GitHub Action that downloads the native binary and verifies its checksum and build provenance before running:

# .github/workflows/cuke-dedup.yml
name: CukeDedup
on: pull_request

permissions:
  contents: read

jobs:
  cuke-dedup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: figueiredoluiz/cuke-dedup@v0.12.0
        with:
          threshold: 5

What about AI agents?

Coding agents are now good at writing tests and proposing refactors. That makes cheap, deterministic guardrails more valuable, not less. There's no reason to have an agent read hundreds of files to find a duplicate the static analyzer can identify without invoking a model. Let the tool produce the evidence and let the agent spend its effort on the judgment calls.

CukeDedup can stream findings as JSONL, and ships an Agent Skill that tells compatible agents to treat findings as evidence to investigate, not as orders to merge everything that looks similar:

npx skills add figueiredoluiz/cuke-dedup --skill cuke-dedup

What's next

At the time of writing, CukeDedup is at v0.12.0. JavaScript and TypeScript are the main focus, and Cucumber-Ruby support is available as an experimental opt-in. Next up: more frameworks, possibly Java and PHP, fewer false positives, and publishing what I learn from scanning open-source suites.

I'd love for this to become a community project rather than just mine. False-positive reports, step-definition patterns it doesn't understand yet, feature ideas and pull requests are all welcome. Even a case where it gets things wrong helps.

I'm curious: how does your team keep step definitions from multiplying? Naming conventions, review checklists, a shared steps library, or just living with it? Tell me in the comments.

GitHub: https://github.com/figueiredoluiz/cuke-dedup
npm: https://www.npmjs.com/package/cuke-dedup
Rules reference: https://github.com/figueiredoluiz/cuke-dedup/blob/main/docs/rules.md

CukeDedup is an independent project, not affiliated with or endorsed by the Cucumber project.