DEV Community: Jean-Luc Martel

AI proved that the Nautilus crushes at 340 metres. Verne dives it to 16,000.

Jean-Luc Martel — Thu, 17 Sep 2026 16:09:00 +0000

Twenty Thousand Leagues Under the Sea gives the Nautilus a hull of 5 cm plating on a 4 m radius. Thin-cylinder hoop stress puts the crush depth at 342 metres. Verne narrates a dive to 16,000.

At 16,000 m the external pressure is about 1,608 atmospheres and the hoop stress is 12.9 GPa, roughly 47 times the plating's yield. Surviving it would need approximately 2.34 metres of steel, which weighs more than the entire boat.

That number came out of a simulator, not a spreadsheet, and the simulator is the point. This is the second of four projects testing what current AI can do with legacy systems, and the variable being changed is the quality of the source. The last one had a formal spec, real preserved code, and an independent implementation to check against. This one has a novel.

Extracting a spec from fiction

The project's thesis is one sentence: legacy documentation lies by omission, not by error. Verne itemises the art in the Nautilus's salon. He never gives the air budget.

So the first phase produced no code at all. Two independent claim-extraction passes ran chapter-by-chapter over the French original, each tagging every physical assertion with its quote, chapter, category, and a precision marker — stated, hedged, implied, or demonstrated. The two passes were then diffed and reconciled into 731 claims, with a human reviewing the reconciliation. Contradictions were recorded and deliberately not resolved, because which resolution you pick changes the answer and that choice needs to be visible rather than buried.

Then the harder half. A simulator needs numbers the novel never supplies, and inventing them quietly is how you get a result that means nothing. Every such parameter became a gap entry with a plausible range and a sourced rationale, and the gate rule was absolute: every parameter in the spec traces to either a claim with a chapter citation or a gap with a range. Nothing enters as a bare constant.

Ranges are anchored where possible to Plongeur, the French Navy submarine launched in April 1863 — the actual state of the art Verne could have known about. Plongeur was 43 m and about 420 tonnes, with 23 compressed-air tanks at 12.5 bar occupying 153 m³ of its hull to drive an 80 hp engine to 4 knots and a range of 5 nautical miles.

Two things follow from that immediately. Stored gas and ballast machinery eat an enormous fraction of a submarine's internal volume, so the Nautilus's breathable space is far smaller than its 1,500 m³ envelope. And 60 kW moved a 420-tonne boat at 4 knots, which sets expectations for what Verne's stated 1.47 MW should do to 1,500 tonnes.

The core is a pure function

The simulator is Ada, built as a pure function of (spec, seed, command stream). No wall clock, no filesystem access, no network, no environment reads inside the core. Everything crosses a single seam — JSON Lines on stdin and stdout — and the UI, the genetic algorithm, the fuzzer, the test harness and the save files all speak the same protocol. There are no side channels.

Three consequences fall out of that shape, and all three matter for supervising an AI:

Any result reproduces from files in the repo plus a binary hash. The spec is canonicalised and SHA-256'd by the core itself; the wrapper cross-checks and aborts on mismatch. Every recording is pinned to a spec hash.
The protocol was frozen before the physics existed. Determinism, replay and malformed-input fuzzing were green in the one check harness before a single equation landed.
The invariants are append-only and human-signed. Twelve of them live in invariants/, covering physical bounds, monotonicity, and event ordering — depth is never negative, stored energy never rises, CO₂ only increases while sealed and may drop only to fresh air via a surface renewal, no state tick may follow a crush event.

That last one is the load-bearing supervision decision. When the fuzzer finds a violation, the repair workflow says the AI patches the core, never the invariant. Weakening an invariant to make an exploit pass is prevented by a checksum in invariants/manifest.sha256 that only a human re-signs. The model is free to fix the physics and structurally unable to move the goalposts.

Five findings

Running the committed spec under real physics and comparing to the narrative:

	axis	the legend	the physics	robust to gaps?
F1	buoyancy	dives at will	under-ballasted — the mass budget only closes in fresh water	yes, it's arithmetic
F2	speed	50 knots	~15 kn; 50 kn needs ~24× the stated power	yes, 15–22 kn across the drag range
F3	range	20,000 leagues	~1,400 nmi per charge → 30–4,000 recharges of a battery whose capacity is never stated	capacity unknown to 100×
F4	depth	16,000 m	crushes at ~340 m	yes, 249–435 m
F5	endurance	weeks submerged	6–211 h; no CO₂ scrubbing means a 1–2 day surface tether	endurance unknown to 35×

F1 is the most elegant. Verne's mass budget is internally consistent — in fresh water, where one cubic metre conveniently weighs a tonne. In seawater at 1,025 kg/m³ the same 1,500 m³ hull displaces 1,537,705 kg, and the boat with main ballast full reaches only 1,507,200 kg. It is about 30 tonnes positively buoyant and cannot submerge on main ballast alone. The arithmetic is fine. The ocean is the wrong one.

F5 is the subtlest, and it cuts both ways. Verne explicitly states the Nautilus has no CO₂ scrubbing — absorbing it would need caustic-potash canisters he says aren't carried — so air is refreshed only by surfacing. Where he lets that constraint bite, he is right: the roughly 48-hour crisis under the Antarctic ice sits squarely inside the plausible band. The failure is everywhere else, where the narrative implies indefinite submersion that the stated design forbids.

And the single most operationally important number in the book — how long the boat can stay under — ranges across 6 to 211 hours depending on parameters Verne never gives. That 35× spread is the finding.

The genetic algorithm cheats, correctly

With the physics quiescent, the question becomes: can a physically honest Nautilus be designed to reach Verne's? A GA got 29 design genes and a push-to-limit mission that dives until hull stress hits its safe limit and cruises until the battery is flat.

The best evolved champion reaches 562 metres. Verne claims 16,000. The gap is about 28×, after pushing every parameter to its most favourable plausible value — the strongest steel, the thickest plating, the biggest battery, the sleekest hull.

The interesting part is what the optimiser did first. Its opening move to extend endurance was to add CO₂ scrubbing at 6.5 L/s — the exact system Verne states the Nautilus lacks, and the omission behind F5's surface tether. With no knowledge of the novel and no term in its objective mentioning air, the GA independently identified and fixed the book's single most consequential design gap. Range roughly doubled once the CO₂ clock came off.

What the fuzzer found

Nothing, which took some doing to trust.

A 24-hour random-command soak ran 42 lives and 879,643 ticks, killing the boat 29 times by crushing it, and produced zero invariant violations. Five consecutive fuzz campaigns, sixty more lives, also zero. A harness that finds nothing is indistinguishable from a harness that is broken, so it was validated against a deliberately poisoned invariant — which it caught, delta-debugged from seven commands down to two, and packaged as a reproducible exploit.

The only finding in the whole loop is filed under exploits/001-objective-pinning, and it is a flaw in the experiment, not the physics: the GA had been optimising survival rather than performance, because the original controller pinned depth and distance to fixed targets. No core patch. A design-of-experiment fix, and the Pareto front went from a collapsed cluster to a real trade-off surface.

Then the frozen Ada core was re-implemented in TypeScript and both were replayed against a golden corpus, with any divergence defined in advance as a TypeScript bug. Event sequences matched exactly. The worst numeric delta was about 5×10⁻⁷ — and all of it traces to the Ada core rounding its output to six decimals at the wire. The two integrators actually agree to about 10⁻⁷, tighter than the reference bothers to report. The only genuine computational difference is the actuator-disk cube root, where Ada's X ** (1.0/3.0) and JavaScript's Math.pow disagree in the last bits and surface as 10⁻¹⁰ watts.

No oracle means you build one

HAL/S could check itself against an independent interpreter written by someone else. There is no independent implementation of Jules Verne. The oracle here had to be constructed, which means the credibility of every number above rests entirely on the scaffolding — the two-pass claim extraction, the traceability gate, the frozen protocol, the human-signed invariants, the determinism enforced before any physics existed, the harness validated against a poisoned check.

An AI will produce a confident simulation of a submarine with far less than that. It will also produce confident numbers that mean nothing, and from the outside the two look identical.

Next: no spec and no oracle again, but this time the thing being rebuilt is a product people remember using, the substrate underneath it has been replaced, and every defect that mattered turned out to be invisible to the tests.

Code: https://github.com/singular-state/nautilus-sim. Every figure regenerates from the committed repo via scripts/check.sh; each finding re-derives from the hash-pinned canonical spec, and each GA champion replays from its (genome, spec_hash, seed, scenario) triple. Pinned GNAT, fixed timestep, seeded PCG — byte-identical replay.

NASA's Shuttle language survives only as a 1980 scan. I had AI rebuild it.

Jean-Luc Martel — Wed, 09 Sep 2026 18:15:48 +0000

The complete specification of the language NASA wrote the Space Shuttle's flight software in is a 20-megabyte bag of page images. No text layer. Some pages sit at an angle.

That document is IR-542, and nothing supersedes it. If you want to know whether a bit-string partition is legal on the left of an assignment, the answer is a photograph of a page printed in 1980.

I wanted HAL/S to run again — lexer, parser, type checker, interpreter — and I wanted AI to write essentially all of it, because the interesting question isn't whether a dead language can be revived. It's what the current generation of models can do when the source of truth has been degraded, and what a human still has to hold.

This is the first of four projects that vary exactly that. HAL/S is the easy end: a formal spec exists, real preserved code exists, and an independent implementation exists to check the answers against. Best-case conditions. Worth knowing what best case buys you.

The obvious OCR strategy is wrong

Run three OCR engines, take the majority vote, move on. This works for prose and fails for grammars, for a reason that took a diagnosis pass to make explicit rather than assume.

OCR errors are correlated across engines. Tesseract, Textract and the PDF's embedded Envision layer are all reading the same glyphs at the same resolution with broadly similar priors, so they tend to misread ::= the same way and | the same way. A vote between them doesn't cancel error, it launders it — three engines agreeing confidently on ::: when the page says ::=. And the deeper problem: no aggregation method can recover a candidate that no engine produced. If all three miss a character, fusion has nothing to fuse.

For prose that's survivable, because context repairs it. For BNF it's fatal — the metacharacters are the content, and there is no redundancy to recover them from.

So the pipeline sorted pages into tiers by how much OCR could be trusted:

Tier	Content	Trust	Disposition
A	Prose	High	OCR, spot-check
B	Tables	Medium	OCR + structural verify
C	BNF productions	Low	Hand-transcribe, verify against the scan
D	2D source notation	Low	Vision model, then verify

Appendix G — the working grammar, 485 productions across physical pages 295–306 — is Tier C in its entirety. It got one strong engine as the authoritative draft (Textract), with Tesseract and Envision used only to flag disagreement for re-inspection. Never as voters. Then every one of the 485 productions was read by eye against a zoomed crop of the page image.

That by-eye pass is the human contribution to this project, and it is not incidental. The AI ran the renders, the despeckling, the extraction, the reconciliation and the assembly. It could not manufacture a character that the scan had eaten.

What it could do is design checks that make the verification mechanical rather than faith-based. Two of them carried real weight:

Numbering integrity. The 485 production numbers came out as exactly 1..485, no gaps, no duplicates. The numbers are the spec's own and the last one is 485, so this proves nothing was dropped or double-counted across twelve pages.
Reference integrity. Every <nonterminal> on a right-hand side either has a defining production or is one of 22 known lexical primitives. No dangling references, no misspellings.

Neither check needs a human, and between them they catch the entire class of "a page got skipped" and "a name got mangled" errors. The eyeballs were then spent only on what the checks can't see.

A small, characteristic detail: a human skim of the PDF reported that some pages "have an angle," which drove an early plan to deskew before OCR. Measuring it found skew under 0.3°, so the deskew step was dropped. The estimate was replaced by a number, and the number deleted a stage of the pipeline.

Residual ambiguities went into DIVERGENCES.md rather than getting guessed inline. One was a scaling glyph the scan left uncertain, later resolved to @ against the spec's own prose describing it. Another, production 348, is logged as an inference from physical scan damage and marked as such. Guessing silently would have been faster and would have made every downstream claim worthless.

Then the easy part

With a verified grammar, building the thing is ordinary compiler work, and the AI was very good at it. The result is in plain Rust with no cargo and no third-party crates — the crates link in dependency order through a two-stage rustc build in build.sh. There is nothing to install.

$ ./build.sh cli
$ ./build/hals crates/hals-syntax/tests/corpus/realworld/HELLO.hal
        THE BEGINNING
          1     HELLO, WORLD!
                    2     RON BURKEY SAYS ISN'T THIS FUN?

The north star was real preserved source, not samples anyone wrote for the occasion: twelve byte-for-byte Shuttle-era programs from the Virtual AGC project. All twelve now run. MATVECS3 puts 22 vector and matrix operations through 400 iterations and prints TEST SUCCESSFUL. DATATYPES reproduces every value its own comments say it should. TEST7 links against a separately compiled COMPOOL and exercises true NAME pointers.

Arithmetic is where fidelity stops being a slogan. SINGLE precision on the Shuttle's general-purpose computer is IBM System/360 short hexadecimal floating point: it truncates toward zero rather than rounding to nearest, and its precision wobbles between 21 and 24 bits depending on the value. IEEE f32 would have been easier and would have produced different numbers. Authentic is the default; --modern gives you f64 when you want to use HAL/S as a live language rather than as a museum exhibit. You can flip between them in the browser playground and watch the digits move.

Checking against something that isn't yourself

Passing your own tests means your implementation agrees with your understanding, which is the thing under suspicion. So every program is also run through Ron Burkey's yaHAL-S, an independent HAL/S interpreter, and compared value-for-value by tools/oracle_diff.py. Every program the oracle can run, we match exactly — including all 339 numbers DATATYPES emits. The four it can't run, ours handles.

This is the Backup Flight System principle applied to a reconstruction: a bug in one implementation shouldn't be shared by the other. It is also the reason this project is the easy arm of the series. An independent oracle is a luxury. Two of the other three don't have one, and the fourth has one deliberately sealed in an envelope.

What isn't there, on purpose

The corpus is closed, and so is the scope. FIXED-point scaling, true mid-statement preemption in the real-time executive, the latched/unlatched event distinction, and full 56-bit hex DOUBLE are all unimplemented, each logged with its reasoning. Not because they're hard — because no preserved program exercises them, so there would be nothing to check the implementation against.

Building unverifiable behaviour is the specific failure this project was structured to avoid. It is also the thing an eager model will happily do for you at any hour, which is why the constraint has to live in the repo rather than in your intentions.

The payoff for all that discipline is Fly the Shuttle: the same interpreter compiled to WebAssembly, driving a landing simulator whose flight law is editable HAL/S. The hardcore vehicle writes the approach as actual flight software — a guidance TASK that the real-time executive runs each cycle over VECTOR state, terminated by a TOUCHDOWN event. You can crash it by editing the autopilot, which is the correct relationship to have with a language like this.

What the easy case actually proves

Give a current model a formal spec, real code to test against, and an independent implementation to check answers with, and it will produce a faithful reconstruction of a dead language. That's the result, and it's a genuinely strong one.

The load-bearing human contributions were smaller than expected and not where I expected. Not architecture, not algorithms, not debugging the type checker. They were: deciding not to vote between correlated OCR engines, and reading 485 grammar productions against page images by eye. One judgement call and one irreducible act of looking.

The first is the kind of mistake that produces a clean-looking artifact with silent errors baked in, which is the worst failure shape available. The second is bounded by information theory rather than by model capability — no future model recovers a character the scan destroyed. Better OCR narrows that gap. It never closes it.

Next in the series: the same question with the spec removed entirely. The source of truth is a novel, there is no oracle, so the project had to build one — and then discovered that Verne's submarine implodes at roughly 340 metres.

Code: https://github.com/singular-state/space-shuttle-sdk. The HAL/S corpus and IR-542 are NASA-derived public domain, preserved by the Virtual AGC project, which also supplies the yaHAL-S oracle. Transcription working artifacts — page renders, crops, per-engine output, per-page verified fragments — are committed under transcription/ so the grammar can be audited against the scan rather than taken on trust.

Your AI Coding Agent Just Finished. Now Ask It to Attack Its Own Work.

Jean-Luc Martel — Fri, 21 Aug 2026 01:39:27 +0000

AI coding agents are remarkably good at getting from “here’s what I want” to “here’s a working implementation.”

They are also remarkably good at being satisfied with what they just built.

That second trait is a problem.

One of the highest-leverage habits I’ve picked up when working with coding agents is surprisingly simple:

After the agent finishes a meaningful piece of work, ask it to perform an adversarial review of its own implementation.

Not:

Review your work and make sure everything looks good.

That tends to produce a polite little victory lap.

Instead, change the objective.

Act as an adversarial reviewer. Assume this implementation contains subtle bugs, incorrect assumptions, security issues, race conditions, missing edge cases, or architectural problems. Your job is to find them. Do not defend the implementation. Try to break it.

The difference can be dramatic.

Building and attacking are different tasks

When an agent is implementing a feature, its working objective is roughly:

Find a plausible path to satisfying the requirements.

Once it has found that path, everything it sees is colored by the solution it just constructed.

Humans do this too.

You write a function, run the obvious tests, and your brain quietly becomes the function’s defense attorney.

The code looks reasonable because you know what it was supposed to do.

An adversarial review gives the model a different role:

Assume the implementation is wrong. Find the evidence.

That changes what it searches for.

Instead of asking:

Does this satisfy the happy path?
Does this compile?
Did I implement the requested feature?

it starts asking:

What happens with malformed input?
What assumption did I make that the caller never promised?
What happens under concurrency?
Is this operation actually atomic?
Can this fail halfway through?
What happens when a dependency returns something unexpected?
Did I introduce an authorization bypass?
Did I preserve existing behavior?
Is there a hidden performance cliff?
Are the tests proving the behavior, or merely exercising the implementation?

Same model. Same context. Very different search space.

The prompt I actually want

Something like this works well:

Perform an adversarial review of the implementation you just created.

Assume there are bugs.

Do not explain why the current implementation is good. Your job is to attack it.

Look specifically for:

- incorrect assumptions
- edge cases
- race conditions
- security vulnerabilities
- data corruption risks
- failure/retry problems
- backwards compatibility issues
- performance regressions
- missing validation
- incorrect error handling
- tests that pass without proving the intended behavior

For every issue you find:

1. Describe the failure mode.
2. Explain how it could occur in practice.
3. Rate its severity.
4. Point to the relevant code.
5. Propose a concrete fix.

Do not modify the code yet. First produce the review.

That last instruction matters.

I usually want the review before the repair.

If you immediately ask the agent to “find and fix any problems,” it can silently patch things while skipping the explanation. Separating diagnosis from remediation makes the reasoning inspectable.

It also lets you decide which findings are real.

Because the agent can absolutely invent problems too.

Make it argue against itself

For larger changes, I sometimes push this further and create two explicit roles.

First:

You are the implementation engineer. Complete the feature.

Then:

You are now a senior engineer reviewing this change before production deployment.

You did not write this code.

Assume the implementation engineer was competent but may have made subtle mistakes.

Try to reject this change.

The phrase “You did not write this code” is surprisingly useful.

Obviously the model did write it. We are not performing metaphysical surgery on the transformer.

But role framing affects the kind of analysis the model performs. Removing psychological ownership, even fictitiously, tends to produce a more skeptical review.

For especially important code, you can go further:

Imagine this change caused a production incident three months from now.

Work backwards and identify the most plausible ways this implementation could have caused it.

Now you’re effectively asking for a miniature pre-mortem.

That often surfaces issues a generic code review misses.

Ask for counterexamples, not opinions

One of the easiest ways to make AI review more useful is to demand concrete failure cases.

Bad:

Is this implementation robust?

Better:

Give me five concrete inputs, system states, or event sequences that could cause this implementation to behave incorrectly.

Even better:

For each suspected bug, construct the smallest reproducible scenario that would demonstrate it.

This forces the critique toward falsifiable claims.

For example, instead of:

There may be a race condition here.

you want:

Request A reads balance=100. Request B reads balance=100. Both subtract 80. Both persist 20. The system has processed $160 of withdrawals from a $100 balance.

That is something you can reason about.

And test.

Turn the findings into tests

This is where the workflow becomes particularly powerful.

After the adversarial review, ask:

For every credible issue you identified, write a regression test that fails against the current implementation.

Do not change the production code yet.

Now the loop becomes:

Implement → Attack → Reproduce → Repair → Verify

That is much stronger than:

Implement → Looks good → Ship

And it makes the agent prove its criticism.

If the supposed bug cannot be reproduced, maybe the review was wrong.

If the test fails, you now have both evidence and permanent coverage.

Different reviewers find different classes of bugs

“Review this code” is extremely underspecified.

I get better results by running multiple targeted reviews.

For example:

Security reviewer

Review this implementation as a hostile application security engineer.

Look for ways an attacker could abuse inputs, authentication, authorization, state transitions, serialization, file access, network calls, or resource consumption.

Reliability reviewer

Review this as a distributed systems reliability engineer.

Focus on partial failure, retries, duplicate execution, idempotency, ordering, timeouts, race conditions, stale state, and recovery after crashes.

API reviewer

Review this as the maintainer of clients that depend on this API.

Look for undocumented behavior changes, ambiguous contracts, backwards compatibility problems, surprising defaults, and error semantics.

Performance reviewer

Assume this works correctly at 100 requests per day but fails badly at 10 million.

Find the scaling problems.

These prompts constrain the search.

And constrained searches are often much better than asking a model to vaguely “think harder.”

There is another benefit: specification discovery

The adversarial pass does something beyond finding implementation bugs.

It often discovers that your requirements were incomplete.

Suppose the agent asks:

What should happen if two users update the object simultaneously?

Maybe you never specified that.

Or:

Is deleting this resource supposed to cascade to associated records?

Also unspecified.

Or:

Should this endpoint reveal whether an email address already exists?

Congratulations, your coding agent just wandered into a product/security decision disguised as an implementation detail.

This is one of the more useful properties of adversarial review.

It exposes the negative space around your specification.

The original implementation task asks:

What did the user tell me to build?

The adversarial task asks:

What did the user forget to tell me?

That second question can be much more valuable.

Why not just ask it to “double-check”?

Because “double-check” preserves the original frame.

The model is still trying to validate the solution.

Adversarial review changes the success criterion.

Success is no longer:

The implementation appears correct.

Success becomes:

I found a credible way this could fail.

That small prompt-engineering shift matters.

It is basically the software equivalent of red teaming.

You don't ask the red team to confirm that the defenses look sensible.

You tell them to get in.

Don't blindly accept the critique either

There is an important caveat.

AI-generated criticism is not automatically correct.

A sufficiently determined model can find imaginary bugs with impressive confidence.

So I treat adversarial findings as hypotheses.

The hierarchy is roughly:

Concrete failing test
Reproducible execution path
Clear reasoning from documented behavior
Plausible concern
Vibes wearing a security-engineer costume

The farther down that list a finding sits, the less weight I give it.

This is also why asking the agent to produce reproduction cases and tests is so useful.

It converts prose into evidence.

Make the agent inspect the diff, not just its memory

If your tool supports it, another useful instruction is:

Review the actual git diff and all directly affected code.

Do not rely on your memory of what you intended to change.

Intent is dangerous during review.

The implementation may not match the agent’s mental model of the implementation.

The diff is reality.

For significant changes, I also ask it to inspect neighboring code and call sites. Bugs frequently live at boundaries rather than inside the newly written function.

A workflow worth automating

For meaningful changes, my preferred agent loop is increasingly something like:

1. Understand the task.
2. Inspect the existing code.
3. Propose an implementation plan.
4. Implement the change.
5. Run relevant tests.
6. Perform an adversarial review.
7. Produce concrete failure cases for credible findings.
8. Add regression tests.
9. Fix confirmed issues.
10. Run the full relevant test suite.
11. Review the final diff again.

You can put this directly into an agent instruction file.

The marginal cost is tiny.

The value can be enormous.

The broader lesson

The interesting thing here isn't really the prompt.

It's that AI coding agents become more useful when we stop treating them as a single programmer with a single continuous train of thought.

They can be the implementer.

Then the reviewer.

Then the attacker.

Then the test engineer.

Then the maintainer wondering what lunatic wrote this six months ago.

Those roles optimize for different things.

And one of the cheapest ways to improve AI-generated software is to deliberately make the model disagree with the version of itself that wrote the code.

So the next time your coding agent announces:

Implementation complete. All tests pass.

Don't congratulate it yet.

Tell it to try to destroy what it just built.

How to Let gzip Find the Signal in a Pile of Documents

Jean-Luc Martel — Mon, 03 Aug 2026 23:19:53 +0000

Suppose you have a directory full of text documents.

Most are repetitive, padded with boilerplate, or otherwise low-signal. A few contain the useful material. You could read every file manually, feed them all into an embedding pipeline, or ask an LLM to rank them.

Or you could ask gzip.

The basic idea is simple:

Repetitive text compresses well. Varied text usually does not.

That makes compression ratio a crude but surprisingly useful proxy for redundancy.

It will not tell you which document is best. But it can help you identify which documents contain less repetition and deserve a closer look.

The heuristic

For each document:

Measure its original size.
Compress it individually with gzip.
Measure the compressed size.
Calculate:

compressed size / original size

A lower ratio means the document compressed well, which usually indicates more repetition.

A higher ratio means the document was harder to compress, which may indicate more varied or information-dense content.

In other words:

lower ratio: more redundant
higher ratio: less redundant

The Bash command

Here is a small Bash pipeline that ranks .txt files by compression ratio:

find ./documents -type f -name '*.txt' -print0 |
while IFS= read -r -d '' file; do
  raw=$(wc -c < "$file")
  compressed=$(gzip -n -c -- "$file" | wc -c)

  awk -v file="$file" -v raw="$raw" -v gz="$compressed" '
    raw > 0 {
      printf "%.3f\t%8d\t%8d\t%s\n", gz/raw, raw, gz, file
    }
  '
done | sort -nr

Example output:

0.642      18432      11834  ./documents/research-notes.txt
0.417      30211      12600  ./documents/project-summary.txt
0.091      27102       2467  ./documents/standard-contract.txt

The columns are:

ratio    original bytes    compressed bytes    filename

Because the output is sorted in descending order, the least compressible files appear first.

Those are the files I would inspect first when looking for the possible “gems.”

To find the most repetitive documents instead, reverse the sort:

sort -n

Why `gzip -n`?

The -n flag prevents gzip from storing the original filename and timestamp in its output.

That makes the compressed sizes more comparable across files and across runs.

Without it, a small amount of unrelated metadata can leak into the measurement.

What this is actually measuring

This technique does not measure truth, relevance, writing quality, or semantic importance.

It measures compressibility.

Those things sometimes correlate, but they are not the same.

A document full of repeated boilerplate will usually compress extremely well. A document with more distinct vocabulary, sentence structure, numbers, and ideas may compress less efficiently.

That makes the ratio useful as a first-pass ranking signal.

It is closer to a metal detector than a treasure map.

Important caveats

Small files produce noisy ratios

gzip adds headers and other fixed overhead. For tiny files, that overhead can dominate the result.

You may want to ignore documents below a minimum size:

find ./documents -type f -name '*.txt' -size +1k -print0

Already-compressed formats will mislead you

Running this directly against PDF, DOCX, ZIP, JPG, or other compressed formats mostly measures the compression characteristics of the container format.

Extract the text first.

For example, with PDFs:

pdftotext input.pdf output.txt

Incompressible does not mean valuable

Encrypted data, random identifiers, hashes, minified code, and corrupted text are all difficult to compress.

They may score highly while containing little useful information.

Repetition is not always fluff

Contracts, API documentation, technical specifications, and scientific papers may repeat terminology because precision requires it.

A lower ratio can indicate redundancy, but it can also indicate consistency.

Language and formatting matter

Compression ratios can be affected by:

document length
whitespace
markup
tables
repeated headings
source language
character encoding
templated metadata

For a fairer comparison, normalize the documents first.

For example:

tr -s '[:space:]' ' ' < input.txt

You could also strip HTML, remove headers and footers, or convert everything to lowercase before compression.

Just remember that normalization changes what you are measuring.

A slightly more useful version

For larger collections, I would filter out tiny files and print the percentage saved:

find ./documents -type f -name '*.txt' -size +1k -print0 |
while IFS= read -r -d '' file; do
  raw=$(wc -c < "$file")
  compressed=$(gzip -n -c -- "$file" | wc -c)

  awk -v file="$file" -v raw="$raw" -v gz="$compressed" '
    raw > 0 {
      ratio = gz / raw
      saved = 100 * (1 - ratio)

      printf "%6.2f%% saved\t%8d bytes\t%s\n",
             saved, raw, file
    }
  '
done | sort -n

This sorts the files with the lowest percentage saved first, meaning the least compressible documents rise to the top.

Where this could be useful

This trick can be handy for quickly triaging:

scraped web pages
exported support tickets
meeting transcripts
research notes
log samples
generated reports
document archives
large sets of Markdown files

It is especially useful when you want a fast local heuristic without setting up a database, embedding model, or external API.

Compression as a feature

The broader idea is more interesting than the Bash command.

Compression ratio can be treated as a lightweight feature in a ranking system.

You could combine it with:

document length
vocabulary diversity
duplicate paragraph counts
keyword density
entropy
embedding similarity
recency
source reputation

Compression alone is crude.

Compression plus a few other signals could become a genuinely useful document-triage tool.

Final thought

There are sophisticated ways to rank a pile of documents.

Sometimes, though, a 40-year-old compression algorithm is enough to tell you which files keep repeating themselves.

And that is often a very good place to start.

DEV Community: Jean-Luc Martel

AI proved that the Nautilus crushes at 340 metres. Verne dives it to 16,000.

Extracting a spec from fiction

The core is a pure function

Five findings

The genetic algorithm cheats, correctly

What the fuzzer found

No oracle means you build one

NASA's Shuttle language survives only as a 1980 scan. I had AI rebuild it.

The obvious OCR strategy is wrong

Then the easy part

Checking against something that isn't yourself

What isn't there, on purpose

What the easy case actually proves

Your AI Coding Agent Just Finished. Now Ask It to Attack Its Own Work.

Building and attacking are different tasks

The prompt I actually want

Make it argue against itself

Ask for counterexamples, not opinions

Turn the findings into tests

Different reviewers find different classes of bugs

Security reviewer

Reliability reviewer

API reviewer

Performance reviewer

There is another benefit: specification discovery

Why not just ask it to “double-check”?

Don't blindly accept the critique either

Make the agent inspect the diff, not just its memory

A workflow worth automating

The broader lesson

How to Let gzip Find the Signal in a Pile of Documents

The heuristic

The Bash command

Why gzip -n?

What this is actually measuring

Important caveats

Small files produce noisy ratios

Already-compressed formats will mislead you

Incompressible does not mean valuable

Repetition is not always fluff

Language and formatting matter

A slightly more useful version

Where this could be useful

Compression as a feature

Final thought

Why `gzip -n`?