DEV Community

Emery Yang
Emery Yang

Posted on

A 90-Minute Spike to Kill Vacuous AI Unit Tests

#testing #javascript #ai #webdev

AI-written tests often assert nothing useful. A 90-minute mutation spike can kill them. Treat this protocol as a proposal, not a measured benchmark.

Cheap generation shifts the bottleneck to review. Token volume does not buy coverage. A green suite can still hide a flipped branch.

The failure mode

Generated tests copy shape, not behavior. They invoke the function once. They check a type or a mock call.

Three patterns show up in review.

  • expect(result).toBeDefined()
  • expect(Array.isArray(list)).toBe(true)
  • expect(fn).toHaveBeenCalled()

None of those catch a wrong formula. None of those catch a swapped boolean. CI stays green while the module is wrong.

Vacuous tests are technical debt with a passing badge. They slow later changes. They train the team to trust noise.

One hypothesis

Hypothesis: AI tests on one module fail a two-part gate.

  1. Assertion density below 1.0 means kill.
  2. Any surviving source mutation means kill.

Ship only if both gates pass. Write the kill in the PR. Do not negotiate extra time.

This is a ship-or-kill spike. One module. Ninety minutes. No framework migration. No coverage manifesto.

Pick the module first

Choose a pure function with branches. Pricing, parsing, and policy checks work. Skip React trees for this timer.

The module must run under npm test already. If setup takes more than ten minutes, pick another file. The spike tests the suite, not the toolchain.

Avoid network calls. Avoid clocks. Avoid filesystem writes. Mutations need a deterministic fail.

Clock

Stay in one checkout. Do not refactor product code.

  1. Minutes 0–10: select one branched function.
  2. Minutes 10–25: generate tests from current source only.
  3. Minutes 25–40: run the density script below.
  4. Minutes 40–70: apply five one-line mutations.
  5. Minutes 70–85: fill the decision table.
  6. Minutes 85–90: record ship or kill in the PR.

If generation needs a hosted agent, use a scratch box. Do not wait on shared CI minutes. Do not extend the timer for “one more prompt.”

Artifact A: assertion density

Label: unexecuted example. Adapt paths before any run.

The script walks *.test.js files. It counts it( / test( versus expect( / assert(.

// tools/assert-density.mjs
// Proposed harness. Not executed in this article.

import fs from "node:fs";
import path from "node:path";

const TEST_RE = /\.(test|spec)\.[cm]?js$/;
const CASE_RE = /\b(?:it|test)\s*\(/g;
const ASSERT_RE = /\b(?:expect|assert)\s*\(/g;

function walk(dir, acc = []) {
  for (const name of fs.readdirSync(dir)) {
    if (name === "node_modules") continue;
    const p = path.join(dir, name);
    const st = fs.statSync(p);
    if (st.isDirectory()) walk(p, acc);
    else if (TEST_RE.test(name)) acc.push(p);
  }
  return acc;
}

function score(src) {
  const tests = (src.match(CASE_RE) || []).length;
  const asserts = (src.match(ASSERT_RE) || []).length;
  const ratio = tests === 0 ? 0 : asserts / tests;
  return { tests, asserts, ratio };
}

const root = process.argv[2] || "./src";
const rows = walk(root).map((file) => {
  const s = score(fs.readFileSync(file, "utf8"));
  return { file, ...s };
});

for (const r of rows) {
  console.log(`${r.ratio.toFixed(2)}\t${r.tests}\t${r.asserts}\t${r.file}`);
}

const tests = rows.reduce((n, r) => n + r.tests, 0);
const asserts = rows.reduce((n, r) => n + r.asserts, 0);
const ratio = tests === 0 ? 0 : asserts / tests;
console.log(`TOTAL\t${tests}\t${asserts}\t${ratio.toFixed(2)}`);
process.exit(ratio < 1 ? 2 : 0);
Enter fullscreen mode Exit fullscreen mode

Run it like this.

node tools/assert-density.mjs ./src
echo $?
Enter fullscreen mode Exit fullscreen mode

Exit code 2 means kill. Ratio under 1.0 is the default floor. Raise the floor for parsers. Lower it only with a written reason.

Weak matchers still inflate the ratio. Density is a filter, not a proof. Read any file that scores high with toBeDefined only.

Vacuous matcher catalog

Flag a test as vacuous when the sole matcher is in this list.

  • toBeDefined / toBeUndefined
  • toBeTruthy / toBeFalsy on objects
  • toBeInstanceOf without field checks
  • toHaveBeenCalled without arguments
  • toEqual(expect.any(Object))

Count those as zero real asserts in the PR note. The script cannot do that job. A human skim still belongs in minutes 70–85.

If every case uses one vacuous matcher, kill the suite. Do not average them with stronger files.

Artifact B: five mutations

Label: unexecuted example. Mutate source, not tests.

Copy the module aside if you want a diff. Apply one change per run. Re-run only the generated file.

# Proposed commands. Stop at five mutations.

# M1: flip a comparison
sed -i 's/===/!==/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M2: off-by-one
sed -i 's/+ 1/+ 0/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M3: drop a negation
sed -i 's/!isEligible/isEligible/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M4: constant to zero
sed -i 's/TAX_RATE = 0.08/TAX_RATE = 0/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js

# M5: early return
sed -i 's/return total;/return 0;/' src/pricing.js
npm test -- src/pricing.test.js
git checkout -- src/pricing.js
Enter fullscreen mode Exit fullscreen mode

Score a mutation as killed only if tests fail. A pass is a survivor. One survivor kills the suite for this spike.

Do not edit tests to chase the mutant. That hides the signal. Do not mutate comments or strings. That wastes a slot.

If sed misses the token, pick a real operator by hand. Log the exact line. Keep the command in the note.

Interpreting a surviving mutant

A survivor is evidence, not a mystery. Classify it in one line.

  • Oracle copied from source: tests echo the formula.
  • Branch never entered: fixture data is too clean.
  • Matcher too wide: equality on a wrapper object.
  • Test targets mocks: production path is untouched.

That one-line class is enough. Do not open a rewrite. The kill already answers the hypothesis.

If all five mutants die, still check density. A brittle suite can fail on any edit. That is not the same as specifying behavior.

Decision table

Fill this table in the PR. Do not add rows during the spike.

Signal Threshold Continue Kill
Assertion ratio >= 1.0 Next row Stop
Vacuous-only cases Zero Next row Stop
Mutations killed 5 of 5 Next row Stop
Branch-hitting fail M3 or M4 fails Next row Stop
Timer <= 90 minutes Record Incomplete

Proposed rule: any Kill cell wins. Incomplete also means kill. Ship needs a clean table and a human skim of matchers.

Store the table as spikes/vacuous-tests.md. Date the file. Paste the five commands. Paste the density totals.

Where a scratch model and server fit

The spike needs two cheap resources. A model to draft tests from source. A machine to run density and mutations without burning paid CI.

MonkeyCode provides free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the model only against the module under test. Paste source, not secrets. Paste no production payloads. Ask for tests that fail on wrong results.

Use the free server as a scratch runner. Clone the repo. Run both artifacts. File the note. Tear the machine down.

Do not assume quota, hardware, model names, or uptime. Those claims are out of scope here. The protocol must still run on a laptop.

If the product is unavailable, run locally. The gates do not depend on a vendor. The decision table is the deliverable.

What this does not prove

Regex is not an AST. expect.assertions(1) can fool the counter. Snapshot tests look dense and still miss logic.

Five mutations are not Stryker. They miss equivalent mutants. They miss IO boundaries. They miss concurrency bugs.

A model can copy the source into the test. That inflates density. Read the file. Delete copied oracles before scoring.

Green CI after this spike is not a release gate. It is evidence for one module on one day. Do not publish the ratios as product benchmarks.

Who should not use this

Skip the spike in these cases.

  • You already run Stryker, PIT, or mutmut on every PR.
  • The module is not unit-testable in ten minutes.
  • The code is safety-critical or moves money in production.
  • You need statistical claims for a paper or bake-off.
  • The only goal is to advertise a coding assistant.

In those cases, use a real mutation framework. Budget more than 90 minutes. Keep vendor evaluation off this timer.

Close the loop

Write one of two sentences in the PR.

  • Ship: density 1.4, 5/5 mutants killed.
  • Kill: density 0.6, mutant M4 survived.

Then stop. Do not generate more tests to rescue a kill. That spends the next hour hiding the same failure.

Keep the decision table in git. That file is the artifact. A free model and free server are optional runners, not the proof.

Top comments (0)