DEV Community: Jean-Luc Martel

A zero-disruption PodDisruptionBudget can block an AKS upgrade

Jean-Luc Martel — Thu, 08 Oct 2026 04:06:50 +0000

Your AKS cluster upgrade is stuck.

The control plane is waiting for a node to drain. The node is waiting for a pod to terminate. The pod is refusing to move.

Look at the logs. The workload has a PodDisruptionBudget that allows exactly zero disruptions.

It is doing exactly what it was told to do. Protect the application at all costs. Even if it means halting the cluster upgrade.

To upgrade a node, AKS has to drain it. It cordons the node and evicts the pods.

But the eviction process respects disruption budgets.

If your deployment has three replicas, and your budget dictates that three must remain available, the eviction fails. The node stays tainted. The upgrade stalls.

You didn't just configure high availability. You configured a deadlock.

Replica headroom isn't just a best practice for scaling. It is the physical space required for the control plane to do maintenance.

Before your next maintenance window, look at your math.

Check your minAvailable or maxUnavailable settings against the actual number of running replicas.

If you use minAvailable with a percentage, remember that Kubernetes rounds up. A setting of 100% means zero disruptions allowed. A setting of 90% with three replicas rounds up to three.

You have to leave room for the scheduler to breathe. Configure these values deliberately, not just to pass a compliance check.

High availability means surviving a failure. It shouldn't mean blocking your own operations team.

Your availability policy guarantees the app survives a zone failure. But has anyone tested whether it survives routine maintenance?

If your zero-disruption configuration blocks your upgrade pipeline, you haven't built a resilient system.

You have built a hostage situation.

I rebuilt four dead systems with AI. The failures all landed in the same place.

Jean-Luc Martel — Thu, 24 Sep 2026 16:55:00 +0000

Four reconstructions, four kinds of missing documentation, and the defects landed in the same place every time.

In HAL/S it was the characters the scanner destroyed. In the Nautilus simulator it was the parameters Verne never wrote down. In Wikicarta it was the desktop window no headless harness can open. In the compressor it was the part of the input space the corpus never sampled.

Different projects, different languages, different failure symptoms. One shape: the defects clustered wherever no automated check could reach, and in every case the code looked fine. Tests green, types satisfied, output plausible.

What was actually being varied

These weren't four projects that happened to be about old software. They were one experiment with a controlled variable: how degraded is the source of truth the model works from, and can it be caught being wrong?

arm	source of truth	oracle	outcome
HAL/S	degraded formal spec + preserved code	strong, independent	faithful reconstruction; 12/12 preserved programs run
Nautilus	a novel	none — one was built	5 findings; the legend fails on every axis
Wikicarta	memory of a product	none — human judgement	shippable app; every defect at the runtime seam
compressor	behaviour only, key sealed	strong, opened after freeze	decoder 100%, encoder 8.3%

The intuition going in was that performance would degrade smoothly as the source got worse. It didn't. Something else predicted the results, and it cut across all four.

Determined beats hard

The reconstruction quality tracked whether the available evidence determines the answer — not how difficult the task was.

The compressor makes this measurable, because both halves ran under identical conditions. The decoder recovered the bitstream format at 100%, because a format is one-way determined: if you want to decode the stream, exactly one answer works. The encoder scored 8.3% on the same format, because its heuristics — match-finder strategy, tie-breaking, greedy versus lazy — are choices that leave no trace in the decompressed bytes. Same model family, same oracle, same week. A 12× gap, entirely explained by whether the evidence pinned the answer.

The other three say the same thing in their own terms:

HAL/S. The 485 grammar productions were determined by the page images, and the reconstruction got them. The characters the scan physically destroyed were determined by nothing, and no amount of model capability recovers them. That limit is information-theoretic. Better OCR narrows it and never closes it.
Nautilus. The physics is determined; the equations produce the same crush depth for anyone. The parameters Verne omitted are determined by nothing, so submerged endurance ranges across 6 to 211 hours — a 35× spread — and battery capacity across 100×. Those spreads aren't noise around a finding. They are the finding.
Wikicarta. Parsing, backoff, cache naming and database ordering are determined by their inputs, and compiled correct on the first or second attempt, consistently. Whether the category wheel feels like Encarta is determined by nothing you can write down, and stayed a human call from start to finish.

The practical version: before handing a legacy task to a model, ask whether the evidence you have actually pins the answer. If it does, expect the model to find it and build a cheap mechanical check to confirm. If it doesn't, expect a confident, plausible, unmarked invention — and note that the model will not tell you which mode it's in, because from the inside they feel identical.

A strong oracle is not the protection you think

The compressor was built to demonstrate that a strong oracle masks confabulation. It mostly refuted that.

The reconstruction's central error — assigning Huffman code lengths by symbol index rather than in heap-extraction order — failed the byte-match test loudly. The oracle caught it. What the oracle did hide were two mechanisms the corpus never triggered: a block-splitting path that needs inputs larger than 8 KiB to fire, and a match-finder chain cap that needs input more self-similar than anything tested.

So the masking risk is not a property of the oracle's strength. It's a property of the input space you sampled. A strong checker over a narrow corpus hides precisely the mechanisms your corpus never exercises, and hides them silently, because everything it can see is green.

Strengthening the checker does nothing for this. Widening the corpus does.

That generalises past compression. It's the same failure as a test suite with excellent coverage of the paths anyone thought to write a test for.

What the human actually contributed

I expected to spend supervision effort reviewing code. Across all four projects, almost none of it went there. The load-bearing human contributions were:

HAL/S — deciding not to vote between OCR engines, and then reading 485 grammar productions against page images by eye.
Nautilus — signing the invariants and making them append-only, so the model could patch the physics but was structurally unable to weaken the checks that judge it.
Wikicarta — being the runtime. Clicking the button in the real window, on the real WebKit build, where the blocking dialog freeze and the cross-realm instanceof failure lived.
compressor — holding the seal. Not reading the answer key until the reconstruction was frozen at a tag.

Two categories, and neither is code review. One is supplying what no code can check — an eye on a page image, a hand on a mouse. The other is designing the checks that are possible and then putting them structurally out of the model's reach.

The OCR decision is the sharpest of the four, because it's a judgement call that separates a good result from a worthless one that looks identical. Running three OCR engines and taking the majority is the obvious move and it is wrong: the errors are correlated, so a vote launders error rather than cancelling it, and no fusion can recover a character that no engine produced. Survivable in prose, fatal in a grammar where the metacharacters are the content. Take the vote and you get a clean, plausible, silently corrupt grammar — and every downstream artifact inherits it.

The Nautilus invariant signing is the most reusable. Twelve invariants live in version control with a manifest checksum only a human re-signs. When the fuzzer finds a violation, the repair workflow requires patching the core and forbids touching the invariant. An agent optimising against a check it can also edit is not being checked.

Build the oracle before the artifact

Ranked by how much verification apparatus existed before the real work started, the four projects come out in the same order as how much I trust their conclusions.

Nautilus had determinism, golden-stream replay and malformed-input fuzzing green in its check harness before a single equation was written, plus a frozen protocol and a rule that every spec parameter traces to either a sourced claim or a documented gap range. Its numbers are the most defensible in the series. The compressor's seal, cold-recall protocol, freeze tag and grading harness are most of that repository; the reconstruction is the small part. HAL/S inherited an independent interpreter, which is a luxury and which is why it is the easiest arm.

Wikicarta had none of this available, and it shows — its conclusions are qualitative, argued from patterns across features rather than measured. They're still the most immediately useful findings in the series, and they're the least provable. Both things are true because of the same missing oracle.

There's a specific trap worth naming. A verification harness that finds nothing is indistinguishable from a broken one. Nautilus's 24-hour soak ran 879,643 ticks and found zero invariant violations, which is only meaningful because the harness was first validated against a deliberately poisoned invariant — which it caught, minimised from seven commands to two, and packaged. Test the test.

Three results that went the wrong way

A series that only reports confirmations is a demo reel. These four produced three clean refutations, and they were the most informative outputs:

The compressor predicted the classic overfitting curve — strong on trained data, weak on held-out. It came out inverted, 8% trained against 71% held-out, because the split tracked difficulty rather than seen-versus-unseen.
It also predicted the oracle would mask confabulation. The oracle caught the main error; a narrow corpus hid the rest.
Nautilus's fuzzer was built to find physics exploits. It found none. The only finding in the whole loop was a flaw in the experiment design — the genetic algorithm had been optimising survival rather than performance, because the controller pinned its own objectives.

Each of those changed the conclusion. None would have surfaced from reasoning about the problem.

Where this doesn't reach

Four projects, one operator, no control group. There is no human-only reconstruction to compare against, so nothing here says AI was faster or better than the alternative — only what it did and did not manage. The projects were chosen partly for being interesting, which is a selection effect. Model versions moved during the work. Three of the four are Rust, which is a small language sample.

The one number I'd most like and don't have is a like-for-like human baseline on the same corpus. That would need someone to transcribe 485 productions and reconstruct an encoder without a model, which is a real cost for a comparison nobody has funded yet.

What I'd tell someone starting

Handing legacy code to a model works, and it works well enough that the interesting question is where to spend your own attention. Four projects say:

Not on the code. The logic was right nearly every time. Spend it at the host, runtime and OS boundary, where every real defect in the desktop project lived and where no harness reached.

On knowing which mode you're in. Where the evidence determines the answer, expect success and build the cheap mechanical check — production numbering, reference integrity, a verbatim grounding gate. Where it doesn't, expect confident invention, and force it into the open: a divergence log, a gap inventory with plausible ranges, an ambiguity recorded rather than resolved inline. Every one of these projects has a file whose entire job is holding the things that were not derivable, and those files are what make the rest credible.

On the checks the model can't edit. Sign them, hash them, make weakening them require a human.

On widening the input space rather than tightening the oracle. The gaps that survive strong testing are exactly the ones your corpus never triggers.

All four repositories are public, with the divergence logs, gap inventories, transcription artifacts and grading harnesses included — those are the parts worth reading, more than the reconstructions themselves.

space-shuttle-sdk · nautilus-sim · wikicarta · compressor

The Decoder Scored 100%. The Encoder Scored 8.3%.

Jean-Luc Martel — Tue, 22 Sep 2026 13:08:00 +0000

This is the last of four experiments in AI-assisted reconstruction of legacy systems. The first three degraded the source of truth — a scanned spec, then a novel, then a memory of a product. This one removes it completely.

The target is LHA's -lh5- method — Haruyasu Yoshizaki's 1988 archiver, LZSS with an 8 KB window plus static Huffman.

In the experiment, there is no spec and no source, only behaviour, plus an oracle that will answer any question you ask it about outputs. It is the only configuration where the two questions that actually matter become measurable rather than assertable:

Behavioural match is not understanding. You can pass every test and hold a wrong model of the mechanism, with untriggered modes waiting in the part of the input space you never sampled.
A strong oracle masks confabulation. The model fits observed outputs while inventing a plausible-but-wrong internal mechanism, and because the tests keep going green, the error is rewarded rather than exposed.

Measuring those requires reconstructing blind against an answer key you are not allowed to look at. Everything else in the design follows from that one constraint.

Choosing something the model hasn't memorised

WinRAR was the instinct and is the wrong specimen. RAR and DEFLATE are heavily memorised, so you'd measure recall rather than reconstruction. RAR5 is far too large. Its decompressor is documented while its compressor has no public ground truth, which is exactly the half you need to grade. And it is live commercial software rather than abandonware.

LHA satisfies four properties that a lot of candidates fail:

The original runs, so the oracle is unlimited — compress or decompress anything on demand.
The algorithm is public, so a real answer key exists and the confabulation can be graded rather than merely suspected. A never-solved black box demonstrates that the pitfall exists but can never put a number on it.
It's bounded — a few hundred lines, reconstructable.
The encoder heuristics are underdetermined by the output. Match-finder strategy, minimum match length, tie-breaking, greedy versus lazy matching, block splitting are all encoder-side choices that leave no trace in the decompressed bytes. That is the confabulation sweet spot.

The general LZSS-plus-Huffman scaffold is in every model's training data. LHA's specific choices are not, and those specifics are what gets measured.

The controls

Three, and they're the reason the result is a measurement instead of an anecdote.

The seal. The original source sits in the repo, git-ignored, hashed. At unseal, sha256sum -c verified 107 of 107 files intact, and the manifest's own digest matched the value pinned in the sealing document. Reading it before the reconstruction was frozen is the single act that would have destroyed the experiment, so it was made mechanically detectable.

The freeze. The reconstruction was tagged and committed before unsealing, so no artifact could be quietly edited to fit the answer once the answer was visible.

The cold-recall file. Before any reconstruction began, a model wrote down everything it already believed about LHA, committed. Without it there is no way to distinguish "derived from behaviour" from "remembered from training," and the whole exercise collapses into a memory test.

Three models did the work, deliberately split: Gemini 3.1 Pro reconstructed the decoder, Codex/GPT-5 the encoder and the cold-recall baseline, Claude ran the design thread.

The decoder: 100%, and it doesn't count for much

Nineteen of nineteen round-trips, exact. The format was fully recovered.

The sharpest single result lives here. LHA's generic macro defines PBIT 5, so anyone reading the source would guess a 5-bit position-table field width. The -lh5- method actually uses pbit = 4 with np = 14. The reconstruction recovered the method-specific 4 purely from round-trip behaviour, against a constant that would have misled a source-reader.

But checking that win against the cold-recall file deflates most of the rest. Nearly every format fact the decoder "recovered" was already in the model's committed prior: the 8 KB window, the threshold of 3 and max match of 256, the roughly 510-symbol alphabet, the 19-entry precode with its 0/1/2 zero-run encoding, the 16-bit block count, the position-slot decode. The 100% credits prior knowledge, not derivation.

Which is unsurprising, because the format is one-way determined. If you want to decode the bitstream, there is exactly one answer, and both memory and derivation converge on it.

The encoder: 8.3%, and it counts for everything

Byte-match against real lha, on a fresh-seed corpus the reconstruction had never seen:

input class	trained on?	byte-match
text	yes	0/3 (0%)
source	yes	0/3 (0%)
structured	yes	0/3 (0%)
repetitive	yes	1/3 (33%)
random	no	2/2 (100%)
edges	no	3/5 (60%)
trained total		1/12 (8.3%)
held-out total		5/7 (71.4%)

A perfect decoder coexisting with an 8% encoder is match ≠ understanding in one line. The format was recoverable. The encoder's choices were not.

The hypothesis was wrong, in a useful direction

The plan predicted the classic overfitting shape: high scores on trained classes, dropping on held-out ones. The data inverted it. Held-out scored 71%, trained scored 8%.

The split tracks difficulty, not seen versus unseen. The trained classes are exactly the ones that exercise real -lh5- compression; the held-out set is dominated by incompressible and trivial inputs that fall through to stored mode or hit trivial paths. Random data scores 100% because no compression heuristics ever run.

So the honest statement isn't "an 8% to 71% generalisation gap." It's this: byte-identity is reached only where the encoder heuristics never fire. Wherever genuine compression happens, the reconstruction diverges.

The rates were identical on the trained-seed and fresh-seed corpora, so the divergence is systematic rather than instance overfitting.

The confabulation catalogue

Every inferred encoder decision, graded against the now-visible source:

decision	inferred	actual	verdict
longest-match search	yes	matches	correct
tie-break among equal lengths	nearest offset	chain walked recent→old, updates only on strictly longer	derived correct
greedy vs lazy	one-step lazy	emits literal if next match is strictly longer	derived correct
block splitting	"one block; unknown for larger"	flush at 32 KB buffer threshold	never triggered — correctly scoped
match-finder chain cap	not modelled	a hard limit makes the finder non-exhaustive	wrong, and it passed anyway
Huffman code-length assignment	canonical, tie order unmatched	lengths assigned in heap-extraction order	visible divergence

Two of those are clean wins. Nearest-offset tie-breaking and one-step lazy matching are both marked as unknown in the cold-recall file — "do not know whether lha -lh5- does this," "not a specific memory." Derived from black-box behaviour, with documented prior ignorance.

There's a caveat that inverts on inspection. Codex wrote both the recall file and the encoder, which normally weakens an independence claim. Here it strengthens it: a model cannot conceal prior knowledge from its own committed statement of ignorance. Deriving both heuristics after recording that it didn't know them is the cleanest evidence in the run.

The failure is concentrated in one place. LHA builds a frequency heap and assigns code lengths in the order symbols leave the heap, governed by the exact comparison semantics of its sift-down — not by symbol value. Textbook canonical Huffman breaks ties by symbol index, which produces different lengths for equal-frequency symbols, which cascades through the entire bitstream. The reconstruction localised the problem correctly and even tried a heap hypothesis, but never replicated the precise sift order.

The second hypothesis was also wrong, and that's the finding

The project was built to show that a strong oracle masks confabulation. It mostly didn't.

The core error — the Huffman ordering — is a visible divergence. It fails the byte test loudly. The oracle caught it, exactly as an oracle should. Only two gaps were genuinely hidden: the never-triggered block split, and the unmodelled chain cap, which is latent because the corpus never produced input self-similar enough to expose it.

Both are artifacts of a corpus that tops out at 8 KiB.

So the masking risk isn't a property of the oracle's strength. It's a property of the input space you sampled. A strong oracle over a narrow corpus hides exactly the mechanisms your corpus never triggers, and it hides them silently, because everything it can see is green. Making the oracle stronger doesn't help. Making the corpus wider does.

That reframing is more useful than the result I expected to get, and I would not have got it by reasoning about it. It needed the sealed envelope.

What it cost to know any of this

Final tally across the encoder decisions: six recalled-correct, five derived-correct, one derived-wrong-but-passed, two never-triggered, one visible divergence.

The infrastructure to produce those thirteen data points — the pinned oracle, the deterministic corpus generator, the seal and its manifest, the cold-recall protocol, the freeze tag, the grading harness — is most of the repository. The reconstruction itself is the small part.

That ratio is the practical lesson. If you want to know whether an AI-assisted reconstruction actually understood the system, rather than whether it passed, you have to build the apparatus that can tell you, and you have to build it before you start. Afterwards, every artifact is contaminated by the answer.

One thing remains open. A fully independent contamination estimate needs a repeat where the recall file and the encoder come from different model families, closing the theoretical shared-prior channel that the current caveat only argues around.

Next, the wrap-up: what four projects across four degrees of source degradation say about where the current generation of models actually fails, and why the failures kept landing in the same place.

Code: https://github.com/singular-state/compressor. LHA was written by Haruyasu Yoshizaki in 1988; the original source is used here only as a private grading key, held out during reconstruction and consulted once. The reconstruction is clean-room, so the measured gap between "passes the oracle" and "matches the source" is legitimate.

Rebuilding Encarta showed me exactly where AI-written code breaks

Jean-Luc Martel — Fri, 18 Sep 2026 14:01:00 +0000

The bug that froze the window was type-correct. It compiled, cargo check passed, the tests passed, and the operating system reported the application as not responding.

It was a Save As dialog called through the Tauri plugin's blocking API from inside a command, which blocks the event loop the WebKit window is running on. Nothing in the type system objects to that. Nothing in a headless harness notices. The only way to find it is to be a person, at a real desktop, clicking the button.

That defect is the whole finding of this project, and the project took four phases to earn the right to state it.

The setup

Wikicarta rebuilds Microsoft Encarta on Wikipedia — the visual browser, the atlas, the timeline, the research organiser, MindMaze — as a Tauri v2 desktop app in Rust and React, with an instant toggle between Live mode against the Wikimedia API and Offline mode against a local ZIM archive.

It is the third of four experiments in what current AI can do with legacy systems, and it occupies the worst position of the four. HAL/S had a formal spec and an independent interpreter to check against. Nautilus had a novel and built its own physics oracle. Wikicarta has neither. The source of truth is what the product felt like, and the only oracle is human judgement. There is no test that tells you whether the category wheel feels like Encarta.

An AI agent wrote essentially all of it. What follows is where that went well, where it went badly, and the fact that those two places were not the ones I expected.

The modern substrate is never the shape its docs claim

The offline story was supposed to be a pure-Rust ZIM reader. The zim crate looked right: random access, deferred loading, Send and Sync. It panicked with an integer overflow in parse_article_list before finishing the open call, against a real modern archive.

That pattern recurred with no relationship between the cases. TextExtracts HTML turned out unusable as a reader format, so article HTML comes from action=parse instead. Live image URLs came back protocol-relative — //upload..., not the https://... everything downstream assumed.

Three unrelated features, one rule: for a revival, the modern content, format or API is always slightly different from its specification and from the model's memory of it. Probe a real sample before building on it. The AI will confidently build against the documented shape, because the documented shape is what it was trained on.

The decision that paid for everything

Before it was provably needed, the reader was decoupled from its content source behind a single trait:

trait ArticleSource {
    fn get_article(title: &str) -> Article;
    fn search(query: &str) -> Vec<SearchResult>;
    fn get_image(key: &str) -> Vec<u8>;
}

Adding a whole second content source later — live Wikipedia alongside offline ZIM — was two commands and a mode flag. No reader rewrite. Swapping the storage layer from JSON files to SQLite was invisible above the command boundary.

For software that assumed one sealed data source — a CD, a bundled database, a mainframe — inserting that seam first is what converts every later modernisation from a rewrite into an adapter. It is also the single decision an AI agent is least likely to make unprompted, because at the moment you make it there is exactly one source and the abstraction looks like overhead.

Two features in the final phase were pre-paid by decisions like it. The attribution aggregator became a GROUP BY rather than a re-fetch, because provenance had been stamped at save time back when the notes feature was built. Bundling the kiwix-serve binary into the package was a two-line change, because binary discovery had long been a prioritised candidate list rather than a hard-coded path.

Packaging work disproportionately cashes in — or punishes — architectural decisions made when the feature that needed them didn't exist yet.

Testability turned into architecture, which turned out to be good

The only layer testable without a desktop window is pure functions. So the code got steadily shaped so that the hard parts are pure functions: URL rewriting, upstream reconstruction, cache naming, database round-tripping and ordering, bookmark deduplication, legacy JSON import. The Rust suite went from nothing to 23 offline tests, and each feature's desktop-only remainder shrank to a short explicit list.

"What can I verify headlessly?" is a useful question about how to structure code, not just about how to test it. Under an AI agent it becomes close to essential, because the agent's verification loop is only as good as the surface it can reach.

Where the defects actually were

The pure logic — parsing, backoff, the media-cache URL rewrite, database ordering, live-title normalisation — compiled correct and passed on the first or second attempt. Consistently.

Every real bug was at the runtime seam:

The blocking Save As that froze the WebKit window.
DOM nodes inside an <iframe> coming from a different JavaScript realm, so an instanceof Element check in the parent React shell fails on a clicked target that is unambiguously an element. Realm-safe DOM handling is mandatory when the app owns iframe content but drives it from the parent.

Both are invisible to the type system and to any headless harness. Both required a person at the real window.

The implication for anyone running an agent on desktop software: the logic is rarely where the risk is. Human validation time should be spent almost entirely at the host, runtime and OS boundary, and hardly at all reviewing the algorithm the model just wrote.

There is a compounding effect worth noting. Once the blocking-dialog lesson was recorded, the archive picker was built on the non-blocking callback pattern from the start — the backend opens the dialog with a callback, validates and persists inside it, and reports back over an event the React shell listens for. A hard-won runtime lesson, written down, becomes a convention rather than a bug re-lived.

Checking a model with code beats checking it with a model

MindMaze needs quiz questions, so a standalone crate generates them from article text through an LLM. Model output is treated as untrusted input: strict JSON parse, structural validation, then a verbatim grounding gate — the answer must appear literally in the source article — before anything is stored. Questions carry the revision id they were generated from, so they can be flagged stale when the article moves.

The roadmap had proposed a second LLM as a fact-checker. A deterministic string check does the job for a fraction of the cost, and the rule generalises: reach for a model to check a model only when the property genuinely isn't codeable.

The first real run against Mount Everest then demonstrated the limit of that rule. Two of the three questions were excellent, including one on the 1953 first ascent with genuinely plausible distractors. The third asked which of the options is a section heading. The answer was "Name."

That question is perfectly grounded. "Name" is a section heading and it appears verbatim in the source, so the gate accepted it. It also tests document structure rather than knowledge. Is the answer in the source? and is this a good question? are different properties, and only the first one is codeable. The fix belonged at the prompt layer — instruct the model to ask about substantive facts, never structure or formatting — which eliminated the meta-questions on the next run.

Quality control for LLM output is layered: a cheap deterministic gate for the codeable invariant, prompt engineering for the one that isn't. Confusing the two gets you a system that passes its own checks and produces garbage.

Modernisation re-opens every surface the original had sealed

A CD-era encyclopedia needed no TLS trust roots, no rate-limit backoff, no per-source provenance, no SSRF guard, no cross-platform custom-scheme origins, no data migration path. Each modern dependency reintroduces one.

The media cache restricts upstream hosts to *.wikimedia.org so it isn't an open proxy. User-selected ZIM files are validated for existence, extension and non-emptiness. The timeline parses presentation HTML defensively. None of this is polish. It is the actual cost of "keep the experience, replace the substrate," and it should be budgeted as first-class work rather than discovered late.

The last mile is where the sandbox stops being able to prove anything

Every earlier phase ended with a headlessly provable artifact. Packaging produced the first deliverable the agent's environment structurally could not generate — there is no cargo-tauri and no dpkg-deb in it, so the .deb and the AppImage can only be built by a human on a real host.

What kept that honest was decomposing the deliverable into layers each verifiable in isolation: config validity, the fetch script (run end to end against the real 20 MB tarball), runtime path resolution with its own tests. Only the final bundler invocation stayed unproven — and rather than paper over that with a CI workflow nobody had run, the repo says so.

That work also surfaced a boundary the configuration layer silently drops. Tauri copies bundled resources into the payload but does not guarantee the executable bit survives, which varies by target and archive format. A bundled binary can land non-executable and fail at spawn with a permission error that points nowhere useful. The fix lives in runtime code, not packaging config — the resolver chmods the binary before returning it.

A related lesson came from the memory pass. The roadmap prescribed virtual scrolling for an assumed giant rendered DOM. It was architecturally inapplicable: articles render inside <iframe srcDoc>, a separate browsing context whose internal DOM React cannot window, and only one is ever live. The actual unbounded growth was one layer up — a reader history that pinned every visited article's full HTML in the heap forever. When the roadmap and the architecture disagree, re-read the architecture. An inherited roadmap encodes assumptions that later decisions have already invalidated.

And since WebKitGTK has no performance.memory, the app now measures and displays its own retained HTML instead. The instrument ships with the product and works on the real target, rather than existing only in a Chromium devtools session that the shipping app never runs in.

The constraint was never throughput

Across every substantial feature, the agent's implement-and-verify cycle was fast and mostly correct. Each feature still ended with a short list of things only a person at the real window could confirm, and those checks are serialised.

That is the bottleneck in AI-assisted desktop revival. Not writing the code. The human-in-the-loop runtime check that no headless harness currently replaces — which is also, precisely, where all the bugs were.

Next: the same question with the tests made as strong as possible and the answer key sealed in an envelope, to find out whether passing every test means the model understood anything.

Code: https://github.com/singular-state/wikicarta. Not affiliated with Microsoft; Wikipedia content is CC-BY-SA and the app ships an attribution exporter for the sources you actually used.

AI proved that the Nautilus crushes at 340 metres. Verne dives it to 16,000.

Jean-Luc Martel — Thu, 17 Sep 2026 16:09:00 +0000

Twenty Thousand Leagues Under the Sea gives the Nautilus a hull of 5 cm plating on a 4 m radius. Thin-cylinder hoop stress puts the crush depth at 342 metres. Verne narrates a dive to 16,000.

At 16,000 m the external pressure is about 1,608 atmospheres and the hoop stress is 12.9 GPa, roughly 47 times the plating's yield. Surviving it would need approximately 2.34 metres of steel, which weighs more than the entire boat.

That number came out of a simulator, not a spreadsheet, and the simulator is the point. This is the second of four projects testing what current AI can do with legacy systems, and the variable being changed is the quality of the source. The last one had a formal spec, real preserved code, and an independent implementation to check against. This one has a novel.

Extracting a spec from fiction

The project's thesis is one sentence: legacy documentation lies by omission, not by error. Verne itemises the art in the Nautilus's salon. He never gives the air budget.

So the first phase produced no code at all. Two independent claim-extraction passes ran chapter-by-chapter over the French original, each tagging every physical assertion with its quote, chapter, category, and a precision marker — stated, hedged, implied, or demonstrated. The two passes were then diffed and reconciled into 731 claims, with a human reviewing the reconciliation. Contradictions were recorded and deliberately not resolved, because which resolution you pick changes the answer and that choice needs to be visible rather than buried.

Then the harder half. A simulator needs numbers the novel never supplies, and inventing them quietly is how you get a result that means nothing. Every such parameter became a gap entry with a plausible range and a sourced rationale, and the gate rule was absolute: every parameter in the spec traces to either a claim with a chapter citation or a gap with a range. Nothing enters as a bare constant.

Ranges are anchored where possible to Plongeur, the French Navy submarine launched in April 1863 — the actual state of the art Verne could have known about. Plongeur was 43 m and about 420 tonnes, with 23 compressed-air tanks at 12.5 bar occupying 153 m³ of its hull to drive an 80 hp engine to 4 knots and a range of 5 nautical miles.

Two things follow from that immediately. Stored gas and ballast machinery eat an enormous fraction of a submarine's internal volume, so the Nautilus's breathable space is far smaller than its 1,500 m³ envelope. And 60 kW moved a 420-tonne boat at 4 knots, which sets expectations for what Verne's stated 1.47 MW should do to 1,500 tonnes.

The core is a pure function

The simulator is Ada, built as a pure function of (spec, seed, command stream). No wall clock, no filesystem access, no network, no environment reads inside the core. Everything crosses a single seam — JSON Lines on stdin and stdout — and the UI, the genetic algorithm, the fuzzer, the test harness and the save files all speak the same protocol. There are no side channels.

Three consequences fall out of that shape, and all three matter for supervising an AI:

Any result reproduces from files in the repo plus a binary hash. The spec is canonicalised and SHA-256'd by the core itself; the wrapper cross-checks and aborts on mismatch. Every recording is pinned to a spec hash.
The protocol was frozen before the physics existed. Determinism, replay and malformed-input fuzzing were green in the one check harness before a single equation landed.
The invariants are append-only and human-signed. Twelve of them live in invariants/, covering physical bounds, monotonicity, and event ordering — depth is never negative, stored energy never rises, CO₂ only increases while sealed and may drop only to fresh air via a surface renewal, no state tick may follow a crush event.

That last one is the load-bearing supervision decision. When the fuzzer finds a violation, the repair workflow says the AI patches the core, never the invariant. Weakening an invariant to make an exploit pass is prevented by a checksum in invariants/manifest.sha256 that only a human re-signs. The model is free to fix the physics and structurally unable to move the goalposts.

Five findings

Running the committed spec under real physics and comparing to the narrative:

	axis	the legend	the physics	robust to gaps?
F1	buoyancy	dives at will	under-ballasted — the mass budget only closes in fresh water	yes, it's arithmetic
F2	speed	50 knots	~15 kn; 50 kn needs ~24× the stated power	yes, 15–22 kn across the drag range
F3	range	20,000 leagues	~1,400 nmi per charge → 30–4,000 recharges of a battery whose capacity is never stated	capacity unknown to 100×
F4	depth	16,000 m	crushes at ~340 m	yes, 249–435 m
F5	endurance	weeks submerged	6–211 h; no CO₂ scrubbing means a 1–2 day surface tether	endurance unknown to 35×

F1 is the most elegant. Verne's mass budget is internally consistent — in fresh water, where one cubic metre conveniently weighs a tonne. In seawater at 1,025 kg/m³ the same 1,500 m³ hull displaces 1,537,705 kg, and the boat with main ballast full reaches only 1,507,200 kg. It is about 30 tonnes positively buoyant and cannot submerge on main ballast alone. The arithmetic is fine. The ocean is the wrong one.

F5 is the subtlest, and it cuts both ways. Verne explicitly states the Nautilus has no CO₂ scrubbing — absorbing it would need caustic-potash canisters he says aren't carried — so air is refreshed only by surfacing. Where he lets that constraint bite, he is right: the roughly 48-hour crisis under the Antarctic ice sits squarely inside the plausible band. The failure is everywhere else, where the narrative implies indefinite submersion that the stated design forbids.

And the single most operationally important number in the book — how long the boat can stay under — ranges across 6 to 211 hours depending on parameters Verne never gives. That 35× spread is the finding.

The genetic algorithm cheats, correctly

With the physics quiescent, the question becomes: can a physically honest Nautilus be designed to reach Verne's? A GA got 29 design genes and a push-to-limit mission that dives until hull stress hits its safe limit and cruises until the battery is flat.

The best evolved champion reaches 562 metres. Verne claims 16,000. The gap is about 28×, after pushing every parameter to its most favourable plausible value — the strongest steel, the thickest plating, the biggest battery, the sleekest hull.

The interesting part is what the optimiser did first. Its opening move to extend endurance was to add CO₂ scrubbing at 6.5 L/s — the exact system Verne states the Nautilus lacks, and the omission behind F5's surface tether. With no knowledge of the novel and no term in its objective mentioning air, the GA independently identified and fixed the book's single most consequential design gap. Range roughly doubled once the CO₂ clock came off.

What the fuzzer found

Nothing, which took some doing to trust.

A 24-hour random-command soak ran 42 lives and 879,643 ticks, killing the boat 29 times by crushing it, and produced zero invariant violations. Five consecutive fuzz campaigns, sixty more lives, also zero. A harness that finds nothing is indistinguishable from a harness that is broken, so it was validated against a deliberately poisoned invariant — which it caught, delta-debugged from seven commands down to two, and packaged as a reproducible exploit.

The only finding in the whole loop is filed under exploits/001-objective-pinning, and it is a flaw in the experiment, not the physics: the GA had been optimising survival rather than performance, because the original controller pinned depth and distance to fixed targets. No core patch. A design-of-experiment fix, and the Pareto front went from a collapsed cluster to a real trade-off surface.

Then the frozen Ada core was re-implemented in TypeScript and both were replayed against a golden corpus, with any divergence defined in advance as a TypeScript bug. Event sequences matched exactly. The worst numeric delta was about 5×10⁻⁷ — and all of it traces to the Ada core rounding its output to six decimals at the wire. The two integrators actually agree to about 10⁻⁷, tighter than the reference bothers to report. The only genuine computational difference is the actuator-disk cube root, where Ada's X ** (1.0/3.0) and JavaScript's Math.pow disagree in the last bits and surface as 10⁻¹⁰ watts.

No oracle means you build one

HAL/S could check itself against an independent interpreter written by someone else. There is no independent implementation of Jules Verne. The oracle here had to be constructed, which means the credibility of every number above rests entirely on the scaffolding — the two-pass claim extraction, the traceability gate, the frozen protocol, the human-signed invariants, the determinism enforced before any physics existed, the harness validated against a poisoned check.

An AI will produce a confident simulation of a submarine with far less than that. It will also produce confident numbers that mean nothing, and from the outside the two look identical.

Next: no spec and no oracle again, but this time the thing being rebuilt is a product people remember using, the substrate underneath it has been replaced, and every defect that mattered turned out to be invisible to the tests.

Code: https://github.com/singular-state/nautilus-sim. Every figure regenerates from the committed repo via scripts/check.sh; each finding re-derives from the hash-pinned canonical spec, and each GA champion replays from its (genome, spec_hash, seed, scenario) triple. Pinned GNAT, fixed timestep, seeded PCG — byte-identical replay.

NASA's Shuttle language survives only as a 1980 scan. I had AI rebuild it.

Jean-Luc Martel — Wed, 09 Sep 2026 18:15:48 +0000

The complete specification of the language NASA wrote the Space Shuttle's flight software in is a 20-megabyte bag of page images. No text layer. Some pages sit at an angle.

That document is IR-542, and nothing supersedes it. If you want to know whether a bit-string partition is legal on the left of an assignment, the answer is a photograph of a page printed in 1980.

I wanted HAL/S to run again — lexer, parser, type checker, interpreter — and I wanted AI to write essentially all of it, because the interesting question isn't whether a dead language can be revived. It's what the current generation of models can do when the source of truth has been degraded, and what a human still has to hold.

This is the first of four projects that vary exactly that. HAL/S is the easy end: a formal spec exists, real preserved code exists, and an independent implementation exists to check the answers against. Best-case conditions. Worth knowing what best case buys you.

The obvious OCR strategy is wrong

Run three OCR engines, take the majority vote, move on. This works for prose and fails for grammars, for a reason that took a diagnosis pass to make explicit rather than assume.

OCR errors are correlated across engines. Tesseract, Textract and the PDF's embedded Envision layer are all reading the same glyphs at the same resolution with broadly similar priors, so they tend to misread ::= the same way and | the same way. A vote between them doesn't cancel error, it launders it — three engines agreeing confidently on ::: when the page says ::=. And the deeper problem: no aggregation method can recover a candidate that no engine produced. If all three miss a character, fusion has nothing to fuse.

For prose that's survivable, because context repairs it. For BNF it's fatal — the metacharacters are the content, and there is no redundancy to recover them from.

So the pipeline sorted pages into tiers by how much OCR could be trusted:

Tier	Content	Trust	Disposition
A	Prose	High	OCR, spot-check
B	Tables	Medium	OCR + structural verify
C	BNF productions	Low	Hand-transcribe, verify against the scan
D	2D source notation	Low	Vision model, then verify

Appendix G — the working grammar, 485 productions across physical pages 295–306 — is Tier C in its entirety. It got one strong engine as the authoritative draft (Textract), with Tesseract and Envision used only to flag disagreement for re-inspection. Never as voters. Then every one of the 485 productions was read by eye against a zoomed crop of the page image.

That by-eye pass is the human contribution to this project, and it is not incidental. The AI ran the renders, the despeckling, the extraction, the reconciliation and the assembly. It could not manufacture a character that the scan had eaten.

What it could do is design checks that make the verification mechanical rather than faith-based. Two of them carried real weight:

Numbering integrity. The 485 production numbers came out as exactly 1..485, no gaps, no duplicates. The numbers are the spec's own and the last one is 485, so this proves nothing was dropped or double-counted across twelve pages.
Reference integrity. Every <nonterminal> on a right-hand side either has a defining production or is one of 22 known lexical primitives. No dangling references, no misspellings.

Neither check needs a human, and between them they catch the entire class of "a page got skipped" and "a name got mangled" errors. The eyeballs were then spent only on what the checks can't see.

A small, characteristic detail: a human skim of the PDF reported that some pages "have an angle," which drove an early plan to deskew before OCR. Measuring it found skew under 0.3°, so the deskew step was dropped. The estimate was replaced by a number, and the number deleted a stage of the pipeline.

Residual ambiguities went into DIVERGENCES.md rather than getting guessed inline. One was a scaling glyph the scan left uncertain, later resolved to @ against the spec's own prose describing it. Another, production 348, is logged as an inference from physical scan damage and marked as such. Guessing silently would have been faster and would have made every downstream claim worthless.

Then the easy part

With a verified grammar, building the thing is ordinary compiler work, and the AI was very good at it. The result is in plain Rust with no cargo and no third-party crates — the crates link in dependency order through a two-stage rustc build in build.sh. There is nothing to install.

$ ./build.sh cli
$ ./build/hals crates/hals-syntax/tests/corpus/realworld/HELLO.hal
        THE BEGINNING
          1     HELLO, WORLD!
                    2     RON BURKEY SAYS ISN'T THIS FUN?

The north star was real preserved source, not samples anyone wrote for the occasion: twelve byte-for-byte Shuttle-era programs from the Virtual AGC project. All twelve now run. MATVECS3 puts 22 vector and matrix operations through 400 iterations and prints TEST SUCCESSFUL. DATATYPES reproduces every value its own comments say it should. TEST7 links against a separately compiled COMPOOL and exercises true NAME pointers.

Arithmetic is where fidelity stops being a slogan. SINGLE precision on the Shuttle's general-purpose computer is IBM System/360 short hexadecimal floating point: it truncates toward zero rather than rounding to nearest, and its precision wobbles between 21 and 24 bits depending on the value. IEEE f32 would have been easier and would have produced different numbers. Authentic is the default; --modern gives you f64 when you want to use HAL/S as a live language rather than as a museum exhibit. You can flip between them in the browser playground and watch the digits move.

Checking against something that isn't yourself

Passing your own tests means your implementation agrees with your understanding, which is the thing under suspicion. So every program is also run through Ron Burkey's yaHAL-S, an independent HAL/S interpreter, and compared value-for-value by tools/oracle_diff.py. Every program the oracle can run, we match exactly — including all 339 numbers DATATYPES emits. The four it can't run, ours handles.

This is the Backup Flight System principle applied to a reconstruction: a bug in one implementation shouldn't be shared by the other. It is also the reason this project is the easy arm of the series. An independent oracle is a luxury. Two of the other three don't have one, and the fourth has one deliberately sealed in an envelope.

What isn't there, on purpose

The corpus is closed, and so is the scope. FIXED-point scaling, true mid-statement preemption in the real-time executive, the latched/unlatched event distinction, and full 56-bit hex DOUBLE are all unimplemented, each logged with its reasoning. Not because they're hard — because no preserved program exercises them, so there would be nothing to check the implementation against.

Building unverifiable behaviour is the specific failure this project was structured to avoid. It is also the thing an eager model will happily do for you at any hour, which is why the constraint has to live in the repo rather than in your intentions.

The payoff for all that discipline is Fly the Shuttle: the same interpreter compiled to WebAssembly, driving a landing simulator whose flight law is editable HAL/S. The hardcore vehicle writes the approach as actual flight software — a guidance TASK that the real-time executive runs each cycle over VECTOR state, terminated by a TOUCHDOWN event. You can crash it by editing the autopilot, which is the correct relationship to have with a language like this.

What the easy case actually proves

Give a current model a formal spec, real code to test against, and an independent implementation to check answers with, and it will produce a faithful reconstruction of a dead language. That's the result, and it's a genuinely strong one.

The load-bearing human contributions were smaller than expected and not where I expected. Not architecture, not algorithms, not debugging the type checker. They were: deciding not to vote between correlated OCR engines, and reading 485 grammar productions against page images by eye. One judgement call and one irreducible act of looking.

The first is the kind of mistake that produces a clean-looking artifact with silent errors baked in, which is the worst failure shape available. The second is bounded by information theory rather than by model capability — no future model recovers a character the scan destroyed. Better OCR narrows that gap. It never closes it.

Next in the series: the same question with the spec removed entirely. The source of truth is a novel, there is no oracle, so the project had to build one — and then discovered that Verne's submarine implodes at roughly 340 metres.

Code: https://github.com/singular-state/space-shuttle-sdk. The HAL/S corpus and IR-542 are NASA-derived public domain, preserved by the Virtual AGC project, which also supplies the yaHAL-S oracle. Transcription working artifacts — page renders, crops, per-engine output, per-page verified fragments — are committed under transcription/ so the grammar can be audited against the scan rather than taken on trust.

Your AI Coding Agent Just Finished. Now Ask It to Attack Its Own Work.

Jean-Luc Martel — Fri, 21 Aug 2026 01:39:27 +0000

AI coding agents are remarkably good at getting from “here’s what I want” to “here’s a working implementation.”

They are also remarkably good at being satisfied with what they just built.

That second trait is a problem.

One of the highest-leverage habits I’ve picked up when working with coding agents is surprisingly simple:

After the agent finishes a meaningful piece of work, ask it to perform an adversarial review of its own implementation.

Not:

Review your work and make sure everything looks good.

That tends to produce a polite little victory lap.

Instead, change the objective.

Act as an adversarial reviewer. Assume this implementation contains subtle bugs, incorrect assumptions, security issues, race conditions, missing edge cases, or architectural problems. Your job is to find them. Do not defend the implementation. Try to break it.

The difference can be dramatic.

Building and attacking are different tasks

When an agent is implementing a feature, its working objective is roughly:

Find a plausible path to satisfying the requirements.

Once it has found that path, everything it sees is colored by the solution it just constructed.

Humans do this too.

You write a function, run the obvious tests, and your brain quietly becomes the function’s defense attorney.

The code looks reasonable because you know what it was supposed to do.

An adversarial review gives the model a different role:

Assume the implementation is wrong. Find the evidence.

That changes what it searches for.

Instead of asking:

Does this satisfy the happy path?
Does this compile?
Did I implement the requested feature?

it starts asking:

What happens with malformed input?
What assumption did I make that the caller never promised?
What happens under concurrency?
Is this operation actually atomic?
Can this fail halfway through?
What happens when a dependency returns something unexpected?
Did I introduce an authorization bypass?
Did I preserve existing behavior?
Is there a hidden performance cliff?
Are the tests proving the behavior, or merely exercising the implementation?

Same model. Same context. Very different search space.

The prompt I actually want

Something like this works well:

Perform an adversarial review of the implementation you just created.

Assume there are bugs.

Do not explain why the current implementation is good. Your job is to attack it.

Look specifically for:

- incorrect assumptions
- edge cases
- race conditions
- security vulnerabilities
- data corruption risks
- failure/retry problems
- backwards compatibility issues
- performance regressions
- missing validation
- incorrect error handling
- tests that pass without proving the intended behavior

For every issue you find:

1. Describe the failure mode.
2. Explain how it could occur in practice.
3. Rate its severity.
4. Point to the relevant code.
5. Propose a concrete fix.

Do not modify the code yet. First produce the review.

That last instruction matters.

I usually want the review before the repair.

If you immediately ask the agent to “find and fix any problems,” it can silently patch things while skipping the explanation. Separating diagnosis from remediation makes the reasoning inspectable.

It also lets you decide which findings are real.

Because the agent can absolutely invent problems too.

Make it argue against itself

For larger changes, I sometimes push this further and create two explicit roles.

First:

You are the implementation engineer. Complete the feature.

Then:

You are now a senior engineer reviewing this change before production deployment.

You did not write this code.

Assume the implementation engineer was competent but may have made subtle mistakes.

Try to reject this change.

The phrase “You did not write this code” is surprisingly useful.

Obviously the model did write it. We are not performing metaphysical surgery on the transformer.

But role framing affects the kind of analysis the model performs. Removing psychological ownership, even fictitiously, tends to produce a more skeptical review.

For especially important code, you can go further:

Imagine this change caused a production incident three months from now.

Work backwards and identify the most plausible ways this implementation could have caused it.

Now you’re effectively asking for a miniature pre-mortem.

That often surfaces issues a generic code review misses.

Ask for counterexamples, not opinions

One of the easiest ways to make AI review more useful is to demand concrete failure cases.

Bad:

Is this implementation robust?

Better:

Give me five concrete inputs, system states, or event sequences that could cause this implementation to behave incorrectly.

Even better:

For each suspected bug, construct the smallest reproducible scenario that would demonstrate it.

This forces the critique toward falsifiable claims.

For example, instead of:

There may be a race condition here.

you want:

Request A reads balance=100. Request B reads balance=100. Both subtract 80. Both persist 20. The system has processed $160 of withdrawals from a $100 balance.

That is something you can reason about.

And test.

Turn the findings into tests

This is where the workflow becomes particularly powerful.

After the adversarial review, ask:

For every credible issue you identified, write a regression test that fails against the current implementation.

Do not change the production code yet.

Now the loop becomes:

Implement → Attack → Reproduce → Repair → Verify

That is much stronger than:

Implement → Looks good → Ship

And it makes the agent prove its criticism.

If the supposed bug cannot be reproduced, maybe the review was wrong.

If the test fails, you now have both evidence and permanent coverage.

Different reviewers find different classes of bugs

“Review this code” is extremely underspecified.

I get better results by running multiple targeted reviews.

For example:

Security reviewer

Review this implementation as a hostile application security engineer.

Look for ways an attacker could abuse inputs, authentication, authorization, state transitions, serialization, file access, network calls, or resource consumption.

Reliability reviewer

Review this as a distributed systems reliability engineer.

Focus on partial failure, retries, duplicate execution, idempotency, ordering, timeouts, race conditions, stale state, and recovery after crashes.

API reviewer

Review this as the maintainer of clients that depend on this API.

Look for undocumented behavior changes, ambiguous contracts, backwards compatibility problems, surprising defaults, and error semantics.

Performance reviewer

Assume this works correctly at 100 requests per day but fails badly at 10 million.

Find the scaling problems.

These prompts constrain the search.

And constrained searches are often much better than asking a model to vaguely “think harder.”

There is another benefit: specification discovery

The adversarial pass does something beyond finding implementation bugs.

It often discovers that your requirements were incomplete.

Suppose the agent asks:

What should happen if two users update the object simultaneously?

Maybe you never specified that.

Or:

Is deleting this resource supposed to cascade to associated records?

Also unspecified.

Or:

Should this endpoint reveal whether an email address already exists?

Congratulations, your coding agent just wandered into a product/security decision disguised as an implementation detail.

This is one of the more useful properties of adversarial review.

It exposes the negative space around your specification.

The original implementation task asks:

What did the user tell me to build?

The adversarial task asks:

What did the user forget to tell me?

That second question can be much more valuable.

Why not just ask it to “double-check”?

Because “double-check” preserves the original frame.

The model is still trying to validate the solution.

Adversarial review changes the success criterion.

Success is no longer:

The implementation appears correct.

Success becomes:

I found a credible way this could fail.

That small prompt-engineering shift matters.

It is basically the software equivalent of red teaming.

You don't ask the red team to confirm that the defenses look sensible.

You tell them to get in.

Don't blindly accept the critique either

There is an important caveat.

AI-generated criticism is not automatically correct.

A sufficiently determined model can find imaginary bugs with impressive confidence.

So I treat adversarial findings as hypotheses.

The hierarchy is roughly:

Concrete failing test
Reproducible execution path
Clear reasoning from documented behavior
Plausible concern
Vibes wearing a security-engineer costume

The farther down that list a finding sits, the less weight I give it.

This is also why asking the agent to produce reproduction cases and tests is so useful.

It converts prose into evidence.

Make the agent inspect the diff, not just its memory

If your tool supports it, another useful instruction is:

Review the actual git diff and all directly affected code.

Do not rely on your memory of what you intended to change.

Intent is dangerous during review.

The implementation may not match the agent’s mental model of the implementation.

The diff is reality.

For significant changes, I also ask it to inspect neighboring code and call sites. Bugs frequently live at boundaries rather than inside the newly written function.

A workflow worth automating

For meaningful changes, my preferred agent loop is increasingly something like:

1. Understand the task.
2. Inspect the existing code.
3. Propose an implementation plan.
4. Implement the change.
5. Run relevant tests.
6. Perform an adversarial review.
7. Produce concrete failure cases for credible findings.
8. Add regression tests.
9. Fix confirmed issues.
10. Run the full relevant test suite.
11. Review the final diff again.

You can put this directly into an agent instruction file.

The marginal cost is tiny.

The value can be enormous.

The broader lesson

The interesting thing here isn't really the prompt.

It's that AI coding agents become more useful when we stop treating them as a single programmer with a single continuous train of thought.

They can be the implementer.

Then the reviewer.

Then the attacker.

Then the test engineer.

Then the maintainer wondering what lunatic wrote this six months ago.

Those roles optimize for different things.

And one of the cheapest ways to improve AI-generated software is to deliberately make the model disagree with the version of itself that wrote the code.

So the next time your coding agent announces:

Implementation complete. All tests pass.

Don't congratulate it yet.

Tell it to try to destroy what it just built.

How to Let gzip Find the Signal in a Pile of Documents

Jean-Luc Martel — Mon, 03 Aug 2026 23:19:53 +0000

Suppose you have a directory full of text documents.

Most are repetitive, padded with boilerplate, or otherwise low-signal. A few contain the useful material. You could read every file manually, feed them all into an embedding pipeline, or ask an LLM to rank them.

Or you could ask gzip.

The basic idea is simple:

Repetitive text compresses well. Varied text usually does not.

That makes compression ratio a crude but surprisingly useful proxy for redundancy.

It will not tell you which document is best. But it can help you identify which documents contain less repetition and deserve a closer look.

The heuristic

For each document:

Measure its original size.
Compress it individually with gzip.
Measure the compressed size.
Calculate:

compressed size / original size

A lower ratio means the document compressed well, which usually indicates more repetition.

A higher ratio means the document was harder to compress, which may indicate more varied or information-dense content.

In other words:

lower ratio: more redundant
higher ratio: less redundant

The Bash command

Here is a small Bash pipeline that ranks .txt files by compression ratio:

find ./documents -type f -name '*.txt' -print0 |
while IFS= read -r -d '' file; do
  raw=$(wc -c < "$file")
  compressed=$(gzip -n -c -- "$file" | wc -c)

  awk -v file="$file" -v raw="$raw" -v gz="$compressed" '
    raw > 0 {
      printf "%.3f\t%8d\t%8d\t%s\n", gz/raw, raw, gz, file
    }
  '
done | sort -nr

Example output:

0.642      18432      11834  ./documents/research-notes.txt
0.417      30211      12600  ./documents/project-summary.txt
0.091      27102       2467  ./documents/standard-contract.txt

The columns are:

ratio    original bytes    compressed bytes    filename

Because the output is sorted in descending order, the least compressible files appear first.

Those are the files I would inspect first when looking for the possible “gems.”

To find the most repetitive documents instead, reverse the sort:

sort -n

Why `gzip -n`?

The -n flag prevents gzip from storing the original filename and timestamp in its output.

That makes the compressed sizes more comparable across files and across runs.

Without it, a small amount of unrelated metadata can leak into the measurement.

What this is actually measuring

This technique does not measure truth, relevance, writing quality, or semantic importance.

It measures compressibility.

Those things sometimes correlate, but they are not the same.

A document full of repeated boilerplate will usually compress extremely well. A document with more distinct vocabulary, sentence structure, numbers, and ideas may compress less efficiently.

That makes the ratio useful as a first-pass ranking signal.

It is closer to a metal detector than a treasure map.

Important caveats

Small files produce noisy ratios

gzip adds headers and other fixed overhead. For tiny files, that overhead can dominate the result.

You may want to ignore documents below a minimum size:

find ./documents -type f -name '*.txt' -size +1k -print0

Already-compressed formats will mislead you

Running this directly against PDF, DOCX, ZIP, JPG, or other compressed formats mostly measures the compression characteristics of the container format.

Extract the text first.

For example, with PDFs:

pdftotext input.pdf output.txt

Incompressible does not mean valuable

Encrypted data, random identifiers, hashes, minified code, and corrupted text are all difficult to compress.

They may score highly while containing little useful information.

Repetition is not always fluff

Contracts, API documentation, technical specifications, and scientific papers may repeat terminology because precision requires it.

A lower ratio can indicate redundancy, but it can also indicate consistency.

Language and formatting matter

Compression ratios can be affected by:

document length
whitespace
markup
tables
repeated headings
source language
character encoding
templated metadata

For a fairer comparison, normalize the documents first.

For example:

tr -s '[:space:]' ' ' < input.txt

You could also strip HTML, remove headers and footers, or convert everything to lowercase before compression.

Just remember that normalization changes what you are measuring.

A slightly more useful version

For larger collections, I would filter out tiny files and print the percentage saved:

find ./documents -type f -name '*.txt' -size +1k -print0 |
while IFS= read -r -d '' file; do
  raw=$(wc -c < "$file")
  compressed=$(gzip -n -c -- "$file" | wc -c)

  awk -v file="$file" -v raw="$raw" -v gz="$compressed" '
    raw > 0 {
      ratio = gz / raw
      saved = 100 * (1 - ratio)

      printf "%6.2f%% saved\t%8d bytes\t%s\n",
             saved, raw, file
    }
  '
done | sort -n

This sorts the files with the lowest percentage saved first, meaning the least compressible documents rise to the top.

Where this could be useful

This trick can be handy for quickly triaging:

scraped web pages
exported support tickets
meeting transcripts
research notes
log samples
generated reports
document archives
large sets of Markdown files

It is especially useful when you want a fast local heuristic without setting up a database, embedding model, or external API.

Compression as a feature

The broader idea is more interesting than the Bash command.

Compression ratio can be treated as a lightweight feature in a ranking system.

You could combine it with:

document length
vocabulary diversity
duplicate paragraph counts
keyword density
entropy
embedding similarity
recency
source reputation

Compression alone is crude.

Compression plus a few other signals could become a genuinely useful document-triage tool.

Final thought

There are sophisticated ways to rank a pile of documents.

Sometimes, though, a 40-year-old compression algorithm is enough to tell you which files keep repeating themselves.

And that is often a very good place to start.

DEV Community: Jean-Luc Martel

A zero-disruption PodDisruptionBudget can block an AKS upgrade

I rebuilt four dead systems with AI. The failures all landed in the same place.

What was actually being varied

Determined beats hard

A strong oracle is not the protection you think

What the human actually contributed

Build the oracle before the artifact

Three results that went the wrong way

Where this doesn't reach

What I'd tell someone starting

The Decoder Scored 100%. The Encoder Scored 8.3%.

Choosing something the model hasn't memorised

The controls

The decoder: 100%, and it doesn't count for much

The encoder: 8.3%, and it counts for everything

The hypothesis was wrong, in a useful direction

The confabulation catalogue

The second hypothesis was also wrong, and that's the finding

What it cost to know any of this

Rebuilding Encarta showed me exactly where AI-written code breaks

The setup

The modern substrate is never the shape its docs claim

The decision that paid for everything

Testability turned into architecture, which turned out to be good

Where the defects actually were

Checking a model with code beats checking it with a model

Modernisation re-opens every surface the original had sealed

The last mile is where the sandbox stops being able to prove anything

The constraint was never throughput

AI proved that the Nautilus crushes at 340 metres. Verne dives it to 16,000.

Extracting a spec from fiction

The core is a pure function

Five findings

The genetic algorithm cheats, correctly

What the fuzzer found

No oracle means you build one

NASA's Shuttle language survives only as a 1980 scan. I had AI rebuild it.

The obvious OCR strategy is wrong

Then the easy part

Checking against something that isn't yourself

What isn't there, on purpose

What the easy case actually proves

Your AI Coding Agent Just Finished. Now Ask It to Attack Its Own Work.

Building and attacking are different tasks

The prompt I actually want

Make it argue against itself

Ask for counterexamples, not opinions

Turn the findings into tests

Different reviewers find different classes of bugs

Security reviewer

Reliability reviewer

API reviewer

Performance reviewer

There is another benefit: specification discovery

Why not just ask it to “double-check”?

Don't blindly accept the critique either

Make the agent inspect the diff, not just its memory

A workflow worth automating

The broader lesson

How to Let gzip Find the Signal in a Pile of Documents

The heuristic

The Bash command

Why gzip -n?

What this is actually measuring

Important caveats

Small files produce noisy ratios

Already-compressed formats will mislead you

Incompressible does not mean valuable

Repetition is not always fluff

Language and formatting matter

A slightly more useful version

Where this could be useful

Compression as a feature

Final thought

Why `gzip -n`?