Four reconstructions, four kinds of missing documentation, and the defects landed in the same place every time.
In HAL/S it was the characters the scanner destroyed. In the Nautilus simulator it was the parameters Verne never wrote down. In Wikicarta it was the desktop window no headless harness can open. In the compressor it was the part of the input space the corpus never sampled.
Different projects, different languages, different failure symptoms. One shape: the defects clustered wherever no automated check could reach, and in every case the code looked fine. Tests green, types satisfied, output plausible.
What was actually being varied
These weren't four projects that happened to be about old software. They were one experiment with a controlled variable: how degraded is the source of truth the model works from, and can it be caught being wrong?
| arm | source of truth | oracle | outcome |
|---|---|---|---|
| HAL/S | degraded formal spec + preserved code | strong, independent | faithful reconstruction; 12/12 preserved programs run |
| Nautilus | a novel | none — one was built | 5 findings; the legend fails on every axis |
| Wikicarta | memory of a product | none — human judgement | shippable app; every defect at the runtime seam |
| compressor | behaviour only, key sealed | strong, opened after freeze | decoder 100%, encoder 8.3% |
The intuition going in was that performance would degrade smoothly as the source got worse. It didn't. Something else predicted the results, and it cut across all four.
Determined beats hard
The reconstruction quality tracked whether the available evidence determines the answer — not how difficult the task was.
The compressor makes this measurable, because both halves ran under identical conditions. The decoder recovered the bitstream format at 100%, because a format is one-way determined: if you want to decode the stream, exactly one answer works. The encoder scored 8.3% on the same format, because its heuristics — match-finder strategy, tie-breaking, greedy versus lazy — are choices that leave no trace in the decompressed bytes. Same model family, same oracle, same week. A 12× gap, entirely explained by whether the evidence pinned the answer.
The other three say the same thing in their own terms:
- HAL/S. The 485 grammar productions were determined by the page images, and the reconstruction got them. The characters the scan physically destroyed were determined by nothing, and no amount of model capability recovers them. That limit is information-theoretic. Better OCR narrows it and never closes it.
- Nautilus. The physics is determined; the equations produce the same crush depth for anyone. The parameters Verne omitted are determined by nothing, so submerged endurance ranges across 6 to 211 hours — a 35× spread — and battery capacity across 100×. Those spreads aren't noise around a finding. They are the finding.
- Wikicarta. Parsing, backoff, cache naming and database ordering are determined by their inputs, and compiled correct on the first or second attempt, consistently. Whether the category wheel feels like Encarta is determined by nothing you can write down, and stayed a human call from start to finish.
The practical version: before handing a legacy task to a model, ask whether the evidence you have actually pins the answer. If it does, expect the model to find it and build a cheap mechanical check to confirm. If it doesn't, expect a confident, plausible, unmarked invention — and note that the model will not tell you which mode it's in, because from the inside they feel identical.
A strong oracle is not the protection you think
The compressor was built to demonstrate that a strong oracle masks confabulation. It mostly refuted that.
The reconstruction's central error — assigning Huffman code lengths by symbol index rather than in heap-extraction order — failed the byte-match test loudly. The oracle caught it. What the oracle did hide were two mechanisms the corpus never triggered: a block-splitting path that needs inputs larger than 8 KiB to fire, and a match-finder chain cap that needs input more self-similar than anything tested.
So the masking risk is not a property of the oracle's strength. It's a property of the input space you sampled. A strong checker over a narrow corpus hides precisely the mechanisms your corpus never exercises, and hides them silently, because everything it can see is green.
Strengthening the checker does nothing for this. Widening the corpus does.
That generalises past compression. It's the same failure as a test suite with excellent coverage of the paths anyone thought to write a test for.
What the human actually contributed
I expected to spend supervision effort reviewing code. Across all four projects, almost none of it went there. The load-bearing human contributions were:
- HAL/S — deciding not to vote between OCR engines, and then reading 485 grammar productions against page images by eye.
- Nautilus — signing the invariants and making them append-only, so the model could patch the physics but was structurally unable to weaken the checks that judge it.
-
Wikicarta — being the runtime. Clicking the button in the real window, on the real WebKit build, where the blocking dialog freeze and the cross-realm
instanceoffailure lived. - compressor — holding the seal. Not reading the answer key until the reconstruction was frozen at a tag.
Two categories, and neither is code review. One is supplying what no code can check — an eye on a page image, a hand on a mouse. The other is designing the checks that are possible and then putting them structurally out of the model's reach.
The OCR decision is the sharpest of the four, because it's a judgement call that separates a good result from a worthless one that looks identical. Running three OCR engines and taking the majority is the obvious move and it is wrong: the errors are correlated, so a vote launders error rather than cancelling it, and no fusion can recover a character that no engine produced. Survivable in prose, fatal in a grammar where the metacharacters are the content. Take the vote and you get a clean, plausible, silently corrupt grammar — and every downstream artifact inherits it.
The Nautilus invariant signing is the most reusable. Twelve invariants live in version control with a manifest checksum only a human re-signs. When the fuzzer finds a violation, the repair workflow requires patching the core and forbids touching the invariant. An agent optimising against a check it can also edit is not being checked.
Build the oracle before the artifact
Ranked by how much verification apparatus existed before the real work started, the four projects come out in the same order as how much I trust their conclusions.
Nautilus had determinism, golden-stream replay and malformed-input fuzzing green in its check harness before a single equation was written, plus a frozen protocol and a rule that every spec parameter traces to either a sourced claim or a documented gap range. Its numbers are the most defensible in the series. The compressor's seal, cold-recall protocol, freeze tag and grading harness are most of that repository; the reconstruction is the small part. HAL/S inherited an independent interpreter, which is a luxury and which is why it is the easiest arm.
Wikicarta had none of this available, and it shows — its conclusions are qualitative, argued from patterns across features rather than measured. They're still the most immediately useful findings in the series, and they're the least provable. Both things are true because of the same missing oracle.
There's a specific trap worth naming. A verification harness that finds nothing is indistinguishable from a broken one. Nautilus's 24-hour soak ran 879,643 ticks and found zero invariant violations, which is only meaningful because the harness was first validated against a deliberately poisoned invariant — which it caught, minimised from seven commands to two, and packaged. Test the test.
Three results that went the wrong way
A series that only reports confirmations is a demo reel. These four produced three clean refutations, and they were the most informative outputs:
- The compressor predicted the classic overfitting curve — strong on trained data, weak on held-out. It came out inverted, 8% trained against 71% held-out, because the split tracked difficulty rather than seen-versus-unseen.
- It also predicted the oracle would mask confabulation. The oracle caught the main error; a narrow corpus hid the rest.
- Nautilus's fuzzer was built to find physics exploits. It found none. The only finding in the whole loop was a flaw in the experiment design — the genetic algorithm had been optimising survival rather than performance, because the controller pinned its own objectives.
Each of those changed the conclusion. None would have surfaced from reasoning about the problem.
Where this doesn't reach
Four projects, one operator, no control group. There is no human-only reconstruction to compare against, so nothing here says AI was faster or better than the alternative — only what it did and did not manage. The projects were chosen partly for being interesting, which is a selection effect. Model versions moved during the work. Three of the four are Rust, which is a small language sample.
The one number I'd most like and don't have is a like-for-like human baseline on the same corpus. That would need someone to transcribe 485 productions and reconstruct an encoder without a model, which is a real cost for a comparison nobody has funded yet.
What I'd tell someone starting
Handing legacy code to a model works, and it works well enough that the interesting question is where to spend your own attention. Four projects say:
Not on the code. The logic was right nearly every time. Spend it at the host, runtime and OS boundary, where every real defect in the desktop project lived and where no harness reached.
On knowing which mode you're in. Where the evidence determines the answer, expect success and build the cheap mechanical check — production numbering, reference integrity, a verbatim grounding gate. Where it doesn't, expect confident invention, and force it into the open: a divergence log, a gap inventory with plausible ranges, an ambiguity recorded rather than resolved inline. Every one of these projects has a file whose entire job is holding the things that were not derivable, and those files are what make the rest credible.
On the checks the model can't edit. Sign them, hash them, make weakening them require a human.
On widening the input space rather than tightening the oracle. The gaps that survive strong testing are exactly the ones your corpus never triggers.
All four repositories are public, with the divergence logs, gap inventories, transcription artifacts and grading harnesses included — those are the parts worth reading, more than the reconstructions themselves.
Top comments (1)
the line that stands out is that the model won't tell you which mode it's in, confident-and-right or confident-and-made-up, they feel identical from the inside. that's the real problem reviewing ai code day to day too, not just old system archaeology. you can't tell by reading a diff. the fix you landed on, an invariant the model can't edit, is the only real answer. build the check before you trust the artifact, not after.