Why Legacy Rewrites Need a Judge, Not a Better Prompt

Lately I keep getting asked about legacy replacements.

It's become one of the more common conversations I have, and the framing is almost always the same: AI can rewrite this now, can't it? The demand is clearly rising, and I don't think the premise is wrong. A coding agent can rewrite a legacy application startlingly fast. Producing the new code has stopped being the bottleneck.

What hasn't moved is the ability to say the new system behaves like the old one. That turns out to be a claim nobody has the evidence to make — about either system.

So when one of these comes up now, analyzing the old code is no longer where I'd start. That's not the obvious move: the default is to understand the old system first, then rebuild from that understanding. It's the order every plan I've seen begins in, and it's the order we tried, with a coding agent doing the analysis. The honest summary is that it didn't converge.

The approach I've drifted into instead:

Record the old system's observable behavior as an executable suite, accept that suite as the specification for the migration scope, and let coding agents implement against it until it's green.

Test-driven modernization, with the legacy system acting as the oracle — not supplying the tests. The scenarios, the observation points, and the normalization rules are all still mine to design. What the old system supplies is the expected values.

This isn't advice, and it isn't a claim that any of this is solved. It's the most practical hypothesis I've got after watching the obvious approach fail to converge.

Where I've landed for now (and expect to keep revising):

• Source analysis produces interpretations. Execution produces comparable evidence.
• For a declared scope, the coverage contract plus the recorded results plus the tests are the specification — not because they're complete, but because they were accepted.
• The coding agent is the replaceable part. The harness that judges it is the asset.
• The deliverable that matters isn't the new code. It's the evidence someone can sign a cutover against.

First I tried just pointing the agent at it

The version I built before this one had no harness in it at all.

Claude Code, pointed straight at the legacy codebase. I put real effort into the scaffolding — skills for the recurring analysis moves, prompt structures that held up across sessions, work plans that decomposed the system into ordered units, and a loop: analyze a region of the old code, write the equivalent in the new stack, review, move on. Not a naive setup, and I still think it was the right thing to try first.

It worked, partially. That's the awkward part — it produces something. You get a new implementation that is recognizably the same application, and on a good day a reviewer nods at it.

What it didn't produce was a reason to believe it. Three things kept going wrong, and they turned out to be the same thing wearing different clothes.

The token cost per unit of confirmed behavior was miserable. The loop spent most of its budget re-reading code it had already read, re-deriving conclusions it had already reached, and writing long analysis prose so the next step could consume it. Enormous input, and the output was a paragraph of interpretation — expensive tokens buying a plausible reading of a stored procedure.

Fidelity was something we checked by eye. Two screens side by side, two sets of records lined up, a squint. "Looks right." There is no number in that sentence. Nobody could say which side effects were verified and which weren't, or whether yesterday's slice was still passing. And volume defeats attention long before it defeats the agent: nobody eyeballs the four hundredth order-status transition carefully.

Nothing ratcheted. A slice that had been reviewed and accepted could quietly regress while the next one was in progress, and we'd find out at integration, or later. Manual review doesn't leave anything behind that can re-run itself.

The diagnosis I eventually landed on: I was trying to solve a measurement problem with better prompts. Every iteration of skills and plans was an attempt to make the agent's interpretation more convincing. What I needed wasn't a more convincing interpretation. It was a mechanism that could disagree with the agent without a human squinting at two screens.

So I stopped improving the analysis and started building the measurement. What follows isn't a design reasoned out in advance — it's what the failure of the first approach pointed at.

Behavior was never in one place

The analysis-first approach doesn't converge, and it isn't because the agent reads badly. It's that the behavior isn't gathered anywhere.

Some is in application code. The rest is in stored procedures, triggers, nightly batch jobs, a permissions table that three screens read differently, a scheduler entry nobody owns, an external service returning a shape the docs don't mention, and operational rules that live in a person. A Java / JSP + Oracle + nightly batch + mail system doesn't have a specification; it has an accumulated set of effects. You can read all of it and still not know the ordering, or which branch fires in production, or that invoice rounding differs by one path.

But you can run it — drive a business scenario against the running system and record what comes out the other side. Recording alone doesn't make that authoritative, though. It needs provenance (which build, which data set, which run), stable semantics across repeated runs, and an accepted policy that lets that run serve as the baseline. Human judgment belongs at the scope, policy, and cutover boundaries, not on every field of every run — putting a person in the inner loop is how this stops scaling at exactly the volume where it starts to matter.

A green suite also proves only the scenarios it contains, which is the obvious objection and deserves answering here rather than at the end. So the loop needs a second gate: evidence that the surfaces, journeys, data effects, error paths, and boundaries the scope requires are actually represented. Coverage here isn't a percentage, it's a contract about what had to be present. Which makes "the suite is the specification" a slogan; the accurate version is that for a declared scope, the coverage contract, the tests, and the attested legacy results together form the executable specification.

The comparator may normalize representation. It may not forgive behavior.

The harness runs each scenario against both systems and compares. Which sounds obvious right up until you try to compare two systems that were supposed to differ. IDs are new. Timestamps are new. Row ordering is unspecified. The new UI's implementation and DOM are deliberately different, even where the user-visible behavior has to stay equivalent.

So a comparator that demands equality is useless. But the fix is not a comparator that forgives things — this is what I got wrong at first, and it's the distinction the whole apparatus rests on.

Representation is generated IDs, request timestamps, encodings, the order of unordered collections, and metadata proven to be non-semantic. Note the proven: "this field looked incidental" is how a business rule gets discarded. What the comparator cannot do is decide that a behavior may change or disappear. That decision exists and is frequent, but it's a migration-policy decision that moves part of the scope outside strict translation, recorded with what was decided and by whom.

Layer	What it's allowed to decide
Normalization	representation only — the same behavior, differently expressed
Migration policy	intentional change or removal, outside strict translation mode

Two rows, deliberately. A permissive mode can carry a narrowly scoped exception — but never as a normal path to green. The moment "ignore this difference" becomes routine vocabulary, a green result turns into something you can negotiate your way into, and the apparatus is just an expensive way to feel reassured.

One scenario, end to end

Abstractly this all sounds tidy, so here's a single slice as it actually moves. Order creation, standard path, item in stock.

The legacy run, recorded:

scenario  order.create / standard / in-stock
legacy    build=legacy-2019.11  data=masked-set-3  run=8f2c

  screen      list refreshed, new row at top, status "pending approval"
  api         201  {order_no: "O-2024-000123", total: 150000}
  db          orders +1, order_lines +1, inventory_reservations +1 (qty 10)
  permission  approver check evaluated → allow
  events      OrderCreated
  batch       nightly invoice job picks it up (t+1)
  mail        order-received → customer address

The new implementation, first pass:

PASS  screen, api status, orders, order_lines, permission, batch, mail
DIFF  api.order_no              expected O-2024-000123   actual O-2026-000001
DIFF  inventory_reservations    expected 1 (qty 10)      actual 0

Two diffs, and they are not the same kind of thing:

order_no   generated id       → normalized (shape asserted: O-YYYY-NNNNNNN)
qty on     inventory reserve  → semantic; NOT normalizable

The first resolves. The second is behavior, so it becomes a task rather than a rule:

Expected:  inventory reservation created: 1 (qty 10)
Actual:    inventory reservation created: 0
Repair:    Add inventory reservation to order creation.
           Preserve idempotency.
           Do not modify billing or shipping modules.

Agent implements, harness re-runs, and the slice closes with something a human can sign:

order.create / standard / in-stock          PRESERVED
  evidence     legacy run 8f2c, new run c41a, 7 observed effects, 0 unresolved
  normalized   order_no, created_at, DOM structure
  policy       none applied

That last block is the actual product of the exercise. Not the code — the row that says which run, how many effects, what was normalized away, and whether anyone changed the rules to get here.

Rigor compresses, again

Note what the agent received: not a description of a system, but a diff and a scope constraint. In the first approach the equivalent input was paragraphs about how the legacy inventory module probably works — long, hedged, expensively derived, partly wrong. Here it's two numbers: shorter, and grounded in an observed run.

Same aphorism as the graph post, arriving from another direction. The more precisely the behavior was captured, the fewer tokens it takes to say what's broken. Capture costs up front and then stops costing on every iteration — the opposite of re-reading the codebase each round.

Which leaves the loop itself unremarkable, and that's the point: red-green-refactor, except the green came from a system that already exists.

The agent writes the code; the harness decides if it's right

The discipline is a trust-boundary question: the agent must not be the thing that judges its own work.

Claude Code, Codex, whatever comes next — interchangeable. The stable asset is everything around it: scenario definitions, execution adapters per stack, both runners, result normalization, behavior comparison, repair-task generation, evidence storage. Agents are strikingly good at producing a plausible "all tests pass," so the comparator, the normalization rules, and the evidence store are the pieces I want deliberately authored, small enough to review, and boring. The bespoke, agent-assisted part stays on the side where a mistake surfaces as a failing scenario rather than a silent pass — otherwise you've built an unsoundness that reads as safety.

The capture layers are mostly plumbing: browser driving for screen behavior (Playwright), boundary capture for APIs, before/after snapshots for the database, tracing for execution paths and side effects, stubbed sinks for batch and mail and events, JSONL and Parquet for evidence, SQLite and DuckDB for working state and diff analysis. Swap any of it for your equivalent; the tools matter less than they look like they do. What matters is that the output is a normalized record that can be replayed and compared — and that, rather than the language or the stack, is what decides whether a system can be migrated this way.

Two honest caveats there. A stubbed sink proves the system issued the right request at the boundary; it does not prove the mail arrived or the settlement cleared. And traces are not proofs of equivalence — internal routes should change in a rewrite. What traces buy is explanation: why a diff happened, which observations were missing, which side effects nobody thought to record.

What "complete" has to mean

No suite proves everything a system can do, so "complete" isn't available in the honest sense. What's available is an operational definition: for this scope, these surfaces and these effects had to be present, and the coverage contract says so explicitly. Everything else gets a bucket rather than silence:

PRESERVED     verified equivalent
INTENTIONAL   changed by policy, recorded and attributed
REMOVED       withdrawn by policy, recorded and attributed
UNVERIFIED    known unknown, still open

The middle two are outcomes of policy decisions, not comparator settings — the same separation as before, arriving at the sign-off document. And UNVERIFIED is the bucket that earns the whole scheme: a migration always has behavior nobody captured, and the difference between a good and a bad project is whether that list exists and gets looked at. It's also where faithfully-reproduced legacy bugs go to be argued about. You will reproduce them; the answer is to move them into INTENTIONAL deliberately rather than discover them later.

This is also why the unit of work is a slice — customer registration, order creation, invoice generation — rather than the system. Two hundred red scenarios at once tell you nothing, and progress becomes a count of slices with evidence attached, which is a harder number to fool yourself with than percent-complete.

What I'd defend

AI made implementation cheap. It did not make correctness cheap, and it made proving nothing easier at all. So the artifact I care about on these projects isn't the generated code. It's the executable specification that can reject the generated code — and the useful property is that nobody in the loop is judging their own output.

I'd hold the specifics loosely. The first approach didn't fail because it was stupid; it was reasonable given what the tools did at the time, and the tools moved. That'll keep happening. If agents get materially better at holding a whole system's behavior in view, or capturing behavior gets cheaper than I think it can, parts of the structure above stop earning their keep and I'll be building something else. What I'd defend is narrower than the design: the correctness of a rewrite has to be decided by something other than the thing that wrote it.

What I want is for a cutover to stop being a judgment call made on confidence and become something the evidence says — what was preserved, what changed by decision, and what nobody has checked yet.

If the behavior is captured properly, you don't have to ask anyone to be brave about the release.