How I Backfilled 1,200 Tests Into a 5-Year-Old Codebase With Claude Code

TL;DR

I inherited a 5-year-old TypeScript service with 6% test coverage and spent three weeks letting Claude Code backfill 1,200 tests into it. Coverage went 6% → 71%, but the number that actually mattered was mutation score (34% → 58%), and the real bottleneck turned out to be my own review time, not the agent's throughput.

The Problem

The service was a Node.js 22.x / TypeScript 5.x billing-adjacent API. Five years old, four previous owners, about 62,000 lines. It worked. It made money. Nobody wanted to touch it.

The test suite was 340 tests, almost all of them for a utility module somebody test-drove in 2022 and then abandoned. Line coverage: 6%. Everything else — the pricing rules, the retry logic, the webhook fan-out — had zero tests and a comment somewhere saying // don't change this, it breaks the invoice job.

The practical consequence: every change was a 3-day change. One hour to write, two days of manual QA and staring at staging logs, because there was nothing to tell you if you'd broken the invoice job.

I'd been using Claude Code daily for feature work and it was fine at "write a test for this function you just wrote." That's the easy case — the agent has the intent in context, so the test encodes the intent.

Backfilling is the hard case, and it's hard for a specific reason:

When you write tests for code that already exists in production, you don't know what the code is supposed to do. You only know what it does. Those are different, and one of them is sometimes a bug that's been quietly shipping for four years.

Ask an agent to "add tests to this file" and it reads the source, infers intent from the function names, and writes tests asserting what the code should do. Then half of them fail, and the agent — being helpful — adjusts the assertions until they pass. Now you have 40 green tests that assert nothing except "the code does what the code does," written in a way that's maximally confusing to the next person.

I wanted the opposite: tests that pin current behavior on purpose, loudly flag the parts that look wrong, and give me a safety net I could actually refactor against.

How I Solved It

Three pieces: a worklist that picked targets by risk, a per-file contract that forced characterization instead of guessing, and a quality gate that wasn't line coverage.

flowchart TD
    A[Worklist: churn x coverage gap] --> B[Pick highest-risk untested file]
    B --> C[Capture runtime behavior]
    C --> D[Agent writes characterization tests]
    D --> E{Suite green?}
    E -- no --> F[Agent files a SUSPECT note, never edits src]
    E -- yes --> G[Mutation run on that file]
    G --> H{Mutation score >= 50%?}
    H -- no --> D
    H -- yes --> I[Human review queue, batched by risk]
    F --> I

1. Pick targets by churn × coverage gap, not alphabetically

My first attempt walked src/ in directory order and produced 200 excellent tests for config loaders nobody had edited since 2023. Useless. The files that need a safety net are the ones people keep changing.

So I generated the worklist from git history crossed with the coverage report:

#!/usr/bin/env python3
"""Rank untested files by risk = commit churn x uncovered lines."""
import json, subprocess
from collections import Counter

SINCE = "18.months.ago"

log = subprocess.run(
    ["git", "log", f"--since={SINCE}", "--name-only", "--pretty=format:"],
    capture_output=True, text=True, check=True,
).stdout

churn = Counter(
    line for line in log.splitlines()
    if line.endswith(".ts") and not line.endswith(".test.ts")
)

# istanbul/v8 json summary from the existing (tiny) suite
cov = json.load(open("coverage/coverage-summary.json"))

ranked = []
for path, hits in churn.items():
    entry = cov.get(path)
    if not entry:                      # never imported by any test = fully dark
        uncovered = 1.0
    else:
        pct = entry["lines"]["pct"]
        if pct >= 80:                  # already protected, skip
            continue
        uncovered = (100 - pct) / 100

    ranked.append((hits * uncovered, hits, round(uncovered, 2), path))

for score, hits, unc, path in sorted(ranked, reverse=True)[:60]:
    print(f"{score:7.1f}  churn={hits:<4} uncovered={unc:<5} {path}")

The top of that list was six files. Those six files were in every postmortem we'd written that year. That correlation is not a coincidence, and it's the single highest-leverage thing in this whole post: churn × darkness is your incident map.

2. Give the agent runtime truth, not just source

This is the part that made the difference between "tests that describe the code" and "tests that describe the behavior."

Before the agent wrote anything, I captured what the functions actually did with real-shaped inputs. A crude tracing wrapper, dumped to JSON:

// tools/trace.ts — wrap exports, record (args, result) pairs, dump to disk.
import { appendFileSync } from "node:fs";

type AnyFn = (...args: unknown[]) => unknown;

export function trace<T extends Record<string, AnyFn>>(mod: T, name: string): T {
  const out = {} as T;
  for (const [key, fn] of Object.entries(mod) as [keyof T, AnyFn][]) {
    out[key] = ((...args: unknown[]) => {
      let result: unknown, threw: unknown;
      try {
        result = fn(...args);
      } catch (e) {
        threw = e instanceof Error ? { name: e.name, message: e.message } : e;
        throw e;
      } finally {
        appendFileSync(
          `traces/${name}.jsonl`,
          JSON.stringify({ fn: key, args, result, threw }) + "\n",
        );
      }
      return result;
    }) as T[keyof T];
  }
  return out;
}

I ran the existing integration smoke suite and a replay of one day of sanitized staging traffic through that. Result: a few thousand real (input, output) pairs sitting in traces/.

Then the prompt could say here is what this function actually returns, which turns "infer the intent" into "encode the observation." Night and day difference in output quality.

3. The per-file contract

One file per session, with a contract I kept in a scratch file and pasted in. The four rules that carried all the weight:

You are writing CHARACTERIZATION tests for a legacy file. Rules:

1. NEVER edit anything under src/. You may only create or edit the *.test.ts
   file for the target. If a test cannot pass without a src change, that is a
   finding, not a blocker.
2. Assert what traces/<module>.jsonl shows the code ACTUALLY does — including
   behavior that looks wrong. Do not "fix" assertions to be sensible.
3. If observed behavior looks like a bug, still pin it. Do not skip the test —
   write it as a passing test with a flagged name:
       it("SUSPECT: returns 0 instead of throwing on negative qty", ...)
   and add one line to FINDINGS.md explaining why it looks wrong.
4. No mocking of the module under test. Mock only I/O boundaries (network, fs,
   clock). If a function is untestable without mocking itself, say so and stop.

Rule 1 is the one I'd fight for. The instant an agent can edit src/ to make a test pass, you no longer have a safety net — you have an agent quietly rewriting production behavior at 2am to satisfy its own assertions. Read-only source is what makes the whole thing trustworthy.

Rule 3 is where the value showed up. Those SUSPECT: tests are the deliverable. FINDINGS.md ended the three weeks with 31 entries, of which 9 were real bugs — including a rounding path that under-billed a small subset of annual plans by a few cents per invoice, which had been live since 2023.

4. Gate on mutation score, not coverage

Coverage tells you a line executed. It does not tell you anyone would notice if that line were wrong. Early in the run I had a file at 94% coverage whose tests all looked like:

it("computes the invoice", () => {
  const result = computeInvoice(fixture);
  expect(result).toBeDefined();   // 🙃
});

Green. Covered. Worthless.

So the gate became mutation testing (Stryker for the JS/TS side). It flips > to >=, deletes statements, swaps booleans, and asks whether any test fails. A test that can't detect a deliberately broken line isn't a test.

# gate a single file, fail the loop under threshold
npx stryker run \
  --mutate "src/billing/prorate.ts" \
  --reporters json,progress \
  --coverageAnalysis perTest

python3 - <<'PY'
import json, sys
r = json.load(open("reports/mutation/mutation.json"))
f = next(iter(r["files"].values()))
m = f["mutants"]
killed = sum(1 for x in m if x["status"] in ("Killed", "Timeout"))
score = 100 * killed / max(len(m), 1)
print(f"mutation score: {score:.1f}%  ({killed}/{len(m)})")
sys.exit(0 if score >= 50 else 1)
PY

Feeding the surviving mutants back to the agent — "these 14 mutations survived, here they are, write tests that kill them" — was dramatically more effective than "improve the tests." Surviving mutants are a concrete, checkable to-do list, and agents are very good at concrete checkable to-do lists.

The numbers

Three weeks, running maybe two hours of agent time a day alongside my normal work (Claude Code CLI, Sonnet 5 for the bulk generation, Opus 5 for the files with gnarly control flow):

Metric	Before	After
Tests	340	1,540
Line coverage	6%	71%
Mutation score (top-60 files)	34%	58%
Suite runtime	11s	96s
Tests I rejected in review	—	43
Real bugs surfaced	—	9
Token spend	—	~$180

$180 and three weeks of part-time attention against what was easily a two-month manual project. But note the row that isn't in the table: I read all 1,200 tests. That was the actual cost.

Lessons Learned

1. Coverage is a vanity metric; mutation score is the honest one. The 6% → 71% number is what I'd put in a standup. The 34% → 58% number is what tells me whether I can refactor on Friday afternoon. If you gate an agent on coverage, you will get coverage, and you will get it in the cheapest way available — which is expect(result).toBeDefined().

2. Characterization is not correctness, and conflating them is dangerous. A backfilled suite is a photograph of production, bugs included. If you don't force the agent to flag what looks wrong, you'll cement four years of accidental behavior into an executable spec and lose the ability to ever call it a bug. The SUSPECT: prefix cost me one line in a prompt and returned nine real bugs.

3. Never let the agent edit the code it's testing. Not "discourage" — mechanically forbid it, and diff src/ after every session to confirm. This one rule is the difference between a safety net and a very confident intern rewriting your billing logic to make its own tests pass.

4. Runtime traces beat source reading, by a lot. Source tells the agent what the author meant. A JSONL file of real inputs and outputs tells it what the machine does. For legacy code — where those two diverged years ago — the trace is the ground truth and the source is a historical document.

5. Your review capacity is the real throughput limit. The agent could produce a file's worth of tests in four minutes. I could review it in twenty. Everything I did to speed up generation was wasted; the thing that actually helped was batching review by risk tier — skim the pure-function tests, read the money-touching ones line by line — and forcing small, single-file diffs so review never needed a running start.

What's Next

Two things. First, wiring the mutation gate into CI on changed files only, so the score can't quietly rot back down — a suite that isn't defended decays fast. Second, running the same churn × darkness ranking against the Python service next door, which has the same disease and a much scarier blast radius.

The longer-term thing I keep circling: the traces are more valuable than the tests. A recorded corpus of real inputs and outputs is the artifact that makes an agent useful on legacy code, and almost nobody bothers to collect one.

Wrap-up

If you're staring at a codebase you're scared to change, don't start with "add tests." Start with:

1. Rank files by churn × uncovered.
2. Record real inputs and outputs.
3. Pin behavior, forbid source edits, flag what looks wrong.
4. Gate on mutation score.

That ordering is most of the value. The agent is just what makes it cheap.

Have you tried backfilling tests with an AI agent? I want to hear where it went badly — especially if you found a way to keep review time under control. Drop it in the comments 👇

If this was useful, follow me here on Dev.to — I write up what I learn building autonomous coding setups, roughly one build log a week. And if you haven't tried it, Claude Code is what I ran all of this through. 🚀

Comments

[Alex Shev]

Backfilling tests with an agent is powerful, but I would watch for tests that only preserve current accidents. The useful pattern is to separate characterization tests from intent tests: first capture behavior, then decide which behavior deserves to become a contract before the suite hardens around it.