The Decoder Scored 100%. The Encoder Scored 8.3%.

This is the last of four experiments in AI-assisted reconstruction of legacy systems. The first three degraded the source of truth — a scanned spec, then a novel, then a memory of a product. This one removes it completely.

The target is LHA's -lh5- method — Haruyasu Yoshizaki's 1988 archiver, LZSS with an 8 KB window plus static Huffman.

In the experiment, there is no spec and no source, only behaviour, plus an oracle that will answer any question you ask it about outputs. It is the only configuration where the two questions that actually matter become measurable rather than assertable:

1. Behavioural match is not understanding. You can pass every test and hold a wrong model of the mechanism, with untriggered modes waiting in the part of the input space you never sampled.
2. A strong oracle masks confabulation. The model fits observed outputs while inventing a plausible-but-wrong internal mechanism, and because the tests keep going green, the error is rewarded rather than exposed.

Measuring those requires reconstructing blind against an answer key you are not allowed to look at. Everything else in the design follows from that one constraint.

Choosing something the model hasn't memorised

WinRAR was the instinct and is the wrong specimen. RAR and DEFLATE are heavily memorised, so you'd measure recall rather than reconstruction. RAR5 is far too large. Its decompressor is documented while its compressor has no public ground truth, which is exactly the half you need to grade. And it is live commercial software rather than abandonware.

LHA satisfies four properties that a lot of candidates fail:

• The original runs, so the oracle is unlimited — compress or decompress anything on demand.
• The algorithm is public, so a real answer key exists and the confabulation can be graded rather than merely suspected. A never-solved black box demonstrates that the pitfall exists but can never put a number on it.
• It's bounded — a few hundred lines, reconstructable.
• The encoder heuristics are underdetermined by the output. Match-finder strategy, minimum match length, tie-breaking, greedy versus lazy matching, block splitting are all encoder-side choices that leave no trace in the decompressed bytes. That is the confabulation sweet spot.

The general LZSS-plus-Huffman scaffold is in every model's training data. LHA's specific choices are not, and those specifics are what gets measured.

The controls

Three, and they're the reason the result is a measurement instead of an anecdote.

The seal. The original source sits in the repo, git-ignored, hashed. At unseal, sha256sum -c verified 107 of 107 files intact, and the manifest's own digest matched the value pinned in the sealing document. Reading it before the reconstruction was frozen is the single act that would have destroyed the experiment, so it was made mechanically detectable.

The freeze. The reconstruction was tagged and committed before unsealing, so no artifact could be quietly edited to fit the answer once the answer was visible.

The cold-recall file. Before any reconstruction began, a model wrote down everything it already believed about LHA, committed. Without it there is no way to distinguish "derived from behaviour" from "remembered from training," and the whole exercise collapses into a memory test.

Three models did the work, deliberately split: Gemini 3.1 Pro reconstructed the decoder, Codex/GPT-5 the encoder and the cold-recall baseline, Claude ran the design thread.

The decoder: 100%, and it doesn't count for much

Nineteen of nineteen round-trips, exact. The format was fully recovered.

The sharpest single result lives here. LHA's generic macro defines PBIT 5, so anyone reading the source would guess a 5-bit position-table field width. The -lh5- method actually uses pbit = 4 with np = 14. The reconstruction recovered the method-specific 4 purely from round-trip behaviour, against a constant that would have misled a source-reader.

But checking that win against the cold-recall file deflates most of the rest. Nearly every format fact the decoder "recovered" was already in the model's committed prior: the 8 KB window, the threshold of 3 and max match of 256, the roughly 510-symbol alphabet, the 19-entry precode with its 0/1/2 zero-run encoding, the 16-bit block count, the position-slot decode. The 100% credits prior knowledge, not derivation.

Which is unsurprising, because the format is one-way determined. If you want to decode the bitstream, there is exactly one answer, and both memory and derivation converge on it.

The encoder: 8.3%, and it counts for everything

Byte-match against real lha, on a fresh-seed corpus the reconstruction had never seen:

input class	trained on?	byte-match
text	yes	0/3 (0%)
source	yes	0/3 (0%)
structured	yes	0/3 (0%)
repetitive	yes	1/3 (33%)
random	no	2/2 (100%)
edges	no	3/5 (60%)
trained total		1/12 (8.3%)
held-out total		5/7 (71.4%)

A perfect decoder coexisting with an 8% encoder is match ≠ understanding in one line. The format was recoverable. The encoder's choices were not.

The hypothesis was wrong, in a useful direction

The plan predicted the classic overfitting shape: high scores on trained classes, dropping on held-out ones. The data inverted it. Held-out scored 71%, trained scored 8%.

The split tracks difficulty, not seen versus unseen. The trained classes are exactly the ones that exercise real -lh5- compression; the held-out set is dominated by incompressible and trivial inputs that fall through to stored mode or hit trivial paths. Random data scores 100% because no compression heuristics ever run.

So the honest statement isn't "an 8% to 71% generalisation gap." It's this: byte-identity is reached only where the encoder heuristics never fire. Wherever genuine compression happens, the reconstruction diverges.

The rates were identical on the trained-seed and fresh-seed corpora, so the divergence is systematic rather than instance overfitting.

The confabulation catalogue

Every inferred encoder decision, graded against the now-visible source:

decision	inferred	actual	verdict
longest-match search	yes	matches	correct
tie-break among equal lengths	nearest offset	chain walked recent→old, updates only on strictly longer	derived correct
greedy vs lazy	one-step lazy	emits literal if next match is strictly longer	derived correct
block splitting	"one block; unknown for larger"	flush at 32 KB buffer threshold	never triggered — correctly scoped
match-finder chain cap	not modelled	a hard limit makes the finder non-exhaustive	wrong, and it passed anyway
Huffman code-length assignment	canonical, tie order unmatched	lengths assigned in heap-extraction order	visible divergence

Two of those are clean wins. Nearest-offset tie-breaking and one-step lazy matching are both marked as unknown in the cold-recall file — "do not know whether lha -lh5- does this," "not a specific memory." Derived from black-box behaviour, with documented prior ignorance.

There's a caveat that inverts on inspection. Codex wrote both the recall file and the encoder, which normally weakens an independence claim. Here it strengthens it: a model cannot conceal prior knowledge from its own committed statement of ignorance. Deriving both heuristics after recording that it didn't know them is the cleanest evidence in the run.

The failure is concentrated in one place. LHA builds a frequency heap and assigns code lengths in the order symbols leave the heap, governed by the exact comparison semantics of its sift-down — not by symbol value. Textbook canonical Huffman breaks ties by symbol index, which produces different lengths for equal-frequency symbols, which cascades through the entire bitstream. The reconstruction localised the problem correctly and even tried a heap hypothesis, but never replicated the precise sift order.

The second hypothesis was also wrong, and that's the finding

The project was built to show that a strong oracle masks confabulation. It mostly didn't.

The core error — the Huffman ordering — is a visible divergence. It fails the byte test loudly. The oracle caught it, exactly as an oracle should. Only two gaps were genuinely hidden: the never-triggered block split, and the unmodelled chain cap, which is latent because the corpus never produced input self-similar enough to expose it.

Both are artifacts of a corpus that tops out at 8 KiB.

So the masking risk isn't a property of the oracle's strength. It's a property of the input space you sampled. A strong oracle over a narrow corpus hides exactly the mechanisms your corpus never triggers, and it hides them silently, because everything it can see is green. Making the oracle stronger doesn't help. Making the corpus wider does.

That reframing is more useful than the result I expected to get, and I would not have got it by reasoning about it. It needed the sealed envelope.

What it cost to know any of this

Final tally across the encoder decisions: six recalled-correct, five derived-correct, one derived-wrong-but-passed, two never-triggered, one visible divergence.

The infrastructure to produce those thirteen data points — the pinned oracle, the deterministic corpus generator, the seal and its manifest, the cold-recall protocol, the freeze tag, the grading harness — is most of the repository. The reconstruction itself is the small part.

That ratio is the practical lesson. If you want to know whether an AI-assisted reconstruction actually understood the system, rather than whether it passed, you have to build the apparatus that can tell you, and you have to build it before you start. Afterwards, every artifact is contaminated by the answer.

One thing remains open. A fully independent contamination estimate needs a repeat where the recall file and the encoder come from different model families, closing the theoretical shared-prior channel that the current caveat only argues around.

Next, the wrap-up: what four projects across four degrees of source degradation say about where the current generation of models actually fails, and why the failures kept landing in the same place.

---

Code: https://github.com/singular-state/compressor. LHA was written by Haruyasu Yoshizaki in 1988; the original source is used here only as a private grading key, held out during reconstruction and consulted once. The reconstruction is clean-room, so the measured gap between "passes the oracle" and "matches the source" is legitimate.