DEV Community: Jean-Luc Martel

PHP: 30 Years of Making Developers Say "Wait, What?" (Part 2: It Gets Worse)

Jean-Luc Martel — Wed, 28 Oct 2026 07:00:00 +0000

Apparently, three examples of PHP weirdness weren't enough.

In the original post, we explored how PHP considers "0" to be false, how array keys can silently overwrite one another, and how an innocent-looking foreach loop can unexpectedly modify your data.

Those were fun.

But PHP has been around for three decades, and you don't accumulate that much history without developing a few interesting personality traits.

So here are four more examples of PHP doing exactly what it was designed to do, even when that seems deeply unreasonable.

4. Filtering an Array Can Turn It Into a JSON Object

Let's say you're building an API.

You have an array of fruit. You want to remove one item and return the rest as JSON.

Simple enough.

$items = ['apple', 'banana', 'cherry'];

$filtered = array_filter(
    $items,
    fn($item) => $item !== 'banana'
);

echo json_encode($filtered);

What do you expect?

Probably this:

["apple","cherry"]

What you actually get:

{"0":"apple","2":"cherry"}

Wait. What?

You started with an array. You filtered an array. You encoded an array.

And somehow, you got a JSON object.

Here's why.

PHP's array_filter() preserves the original keys.

After filtering, the array looks like this:

[
    0 => 'apple',
    2 => 'cherry'
]

When json_encode() encounters a PHP array whose keys aren't consecutive integers starting at zero, it serializes it as a JSON object rather than a JSON array.

Your perfectly reasonable filtering operation has silently changed the shape of your API response.

Imagine the delight of the frontend developer expecting an array.

The fix:

echo json_encode(array_values($filtered));

Which gives you:

["apple","cherry"]

The lesson: If you're filtering arrays for JSON responses, remember that array_filter() preserves keys. Use array_values() when you need a proper JSON array.

5. A Variable Can Exist and Still Not Be Set

Let's talk about isset().

The name seems reasonably self-explanatory.

Surely it tells you whether something exists?

Well, not exactly.

$config = [
    'timeout' => null
];

var_dump(isset($config['timeout']));

Result:

bool(false)

But the key is right there!

You can see it. I can see it. PHP can presumably see it.

And yet isset() returns false.

That's because isset() doesn't actually mean "does this variable or key exist?"

It means something closer to "does this exist AND contain a value other than null?"

So we have two functions:

isset($config['timeout']);                // false

array_key_exists('timeout', $config);     // true

Same array. Same key. Different answers.

And there's a practical consequence.

Suppose null means "disable the timeout," while a missing key means "use the default timeout."

You might write:

$timeout = $config['timeout'] ?? 30;

But the null coalescing operator also falls back when the value is null.

So $timeout becomes 30, even though the configuration explicitly contained null.

If those two states matter, you need to distinguish them:

$timeout = array_key_exists('timeout', $config)
    ? $config['timeout']
    : 30;

Now an explicitly configured null stays null.

The lesson: In PHP, "missing" and "present but null" are different states, but isset() treats them the same. Sometimes that's convenient. Sometimes it's a bug waiting for the right configuration file.

6. Two Different Strings Can Be Equal Because They're Both Zero

Here's a fun one.

var_dump("0e123" == "0e999");

What would you expect?

They're clearly different strings.

Surely the answer is false.

bool(true)

Of course it is.

The explanation involves PHP's numeric-string handling.

Both strings are valid representations of numbers in scientific notation.

0e123 means zero multiplied by ten to the power of 123.

0e999 means zero multiplied by ten to the power of 999.

Both evaluate numerically to zero.

And because PHP's loose comparison operator (==) recognizes both strings as numeric, it compares their numeric values.

So:

"0e123" == "0e999"    // true
"0e123" === "0e999"   // false

This particular behavior has a wonderful nickname: magic hashes.

Certain hash outputs happen to consist entirely of a zero, followed by e, followed by digits.

When two such hashes are compared using loose equality, PHP can interpret them as equal numeric values even though their string contents differ.

Historically, this has contributed to security vulnerabilities in applications that compared hash strings incorrectly.

And no, switching from PHP 7 to PHP 8 doesn't eliminate this particular issue. Both operands here are valid numeric strings, so the loose comparison still considers them equal.

The fix is straightforward:

"0e123" === "0e999"  // false

For security-sensitive hash comparisons, use the appropriate cryptographic comparison functions, such as hash_equals() when comparing authentication-related strings, or password_verify() for password hashes.

The lesson: Loose equality isn't merely a convenience feature. When values originate from external input, it can become a security problem.

7. `switch` and `match` Disagree About Reality

PHP 8 introduced match, a more modern alternative to switch.

And naturally, they don't quite agree on what constitutes a match.

Consider:

$value = "10";

switch ($value) {
    case 10:
        echo "Integer ten";
        break;

    case "10":
        echo "String ten";
        break;
}

Output:

Integer ten

The value is a string.

The first case is an integer.

But PHP's switch uses loose comparisons, so "10" matches 10.

The first matching case wins.

Now let's rewrite that using match:

$value = "10";

echo match ($value) {
    10   => "Integer ten",
    "10" => "String ten",
};

Output:

String ten

Same input. Same alternatives. Different result.

Why?

Because match uses strict comparisons (===), while switch uses loose comparisons (==).

And match has some other differences, including no fall-through and a requirement to handle unmatched values or provide a default.

To be fair, match is arguably the better-designed construct.

But imagine converting an old switch statement to match during a refactoring session, assuming that you're just modernizing the syntax.

You could change application behavior without changing any of the values being compared.

The lesson: When replacing switch with match, you're not merely changing syntax. You're changing comparison semantics.

PHP Is Consistent. That's Not Always Comforting.

Here's the interesting thing about all seven examples we've now explored across these two articles.

None of them are random.

PHP has rules governing truthiness, type coercion, array keys, references, JSON serialization, and comparisons.

The interpreter follows those rules consistently.

The problem is that those rules don't always line up with what developers intuitively expect.

And sometimes two individually reasonable rules interact to produce something spectacularly unreasonable.

That's what makes PHP such a fascinating language to revisit.

It has evolved a lot since the 1990s. Modern PHP is faster, more expressive, and substantially better engineered than the language many developers remember.

But it still has a few surprises hiding behind perfectly ordinary syntax.

And after thirty years, I suspect it's earned the right to be a little eccentric.

PHP: 30 Years of Making Developers Say "Wait, What?"

Jean-Luc Martel — Sun, 25 Oct 2026 01:57:59 +0000

With PHP suddenly back in the spotlight, I figured this was a good time to revisit one of the web's most successful, most criticized, and occasionally most bewildering programming languages.

PHP has been powering websites for decades. It has evolved a lot, particularly through PHP 7 and 8, but it still carries some fascinating quirks.

And I'm not talking about obscure syntax or features nobody uses. I'm talking about perfectly ordinary code that does something completely unexpected.

Here are three of my favorites.

1. The String "0" Is False. The String "00" Is True.

Let's start with something simple.

var_dump((bool) "0");      // false
var_dump((bool) "00");     // true
var_dump((bool) "false");  // true
var_dump((bool) "");       // false

Isn't that beautiful?

The string "0" evaluates to false, but "00" evaluates to true.

Even the literal string "false" evaluates to true.

PHP considers an empty string and the specific string "0" to be falsy. Other nonempty strings, including "00", are truthy.

Now consider:

$password = "0";

if (!$password) {
    echo "No password provided!";
}

Except a password was provided. PHP just decided it didn't count.

And if you're thinking that empty() might save you:

empty("0");    // true
empty("00");   // false

Nope.

The lesson: Don't use truthiness to determine whether a string contains meaningful input. Be explicit about what you're checking.

2. PHP Arrays Can Silently Overwrite Your Data

PHP arrays aren't really conventional arrays. They're ordered maps that support both integer and string keys.

Which sounds perfectly reasonable until you do this:

$data = [
    "8"  => "eight",
    8    => "EIGHT",
    "08" => "zero eight"
];

print_r($data);

You might expect three entries.

You get two.

Array
(
    [8] => EIGHT
    [08] => zero eight
)

PHP automatically converts the string key "8" to the integer 8.

So the second entry overwrites the first.

But "08" stays a string because it isn't a valid decimal integer representation for this conversion.

And it gets better:

$data = [
    1    => "integer",
    "1"  => "string",
    true => "boolean"
];

print_r($data);

Result:

Array
(
    [1] => boolean
)

Three apparently different keys. One surviving value.

The first two collide because "1" becomes integer 1. The third collides because true also becomes integer 1.

The lesson: PHP array keys undergo implicit type conversion. If your application depends on distinguishing keys by type, you're in for an unpleasant surprise.

3. A `foreach` Loop Can Quietly Corrupt Your Array

This is my favorite, because it's the kind of bug that can survive a code review.

Consider:

$numbers = [1, 2, 3];

foreach ($numbers as &$number) {
    // Do some processing
}

foreach ($numbers as $number) {
    // Do some more processing
}

print_r($numbers);

Nothing particularly exotic here.

The first loop iterates by reference. The second iterates normally.

What does the final array contain?

Array
(
    [0] => 1
    [1] => 2
    [2] => 2
)

Wait. What happened to the 3?

The culprit is that innocent-looking ampersand in the first loop:

foreach ($numbers as &$number)

When that loop finishes, $number remains a reference to the last element of the array.

The reference doesn't magically disappear when the loop ends.

Then the second loop reuses $number.

Each iteration assigns another array value to a variable that's still referencing the last element.

By the time the second loop finishes, the last element has been overwritten with the second-to-last value.

The truly delightful part?

This behavior is explicitly documented in the PHP manual.

The fix is simple:

foreach ($numbers as &$number) {
    // Do some processing
}

unset($number); // Break the reference

But the fact that you need to remember to do this is what makes the behavior so interesting.

The lesson: References in PHP can outlive the control structures that introduced them. Reusing a variable after a by-reference loop can have side effects that are surprisingly difficult to spot.

So, Is PHP Broken?

Not really.

All three behaviors follow documented rules. They're not random, and they're not bugs in the PHP interpreter.

But they illustrate something interesting about programming languages.

A language can be internally consistent while still producing results that surprise the people using it.

PHP has spent decades balancing developer convenience, backward compatibility, and language evolution. Sometimes those priorities have produced genuinely useful features. Sometimes they've produced behavior that makes you stare at your screen for a few minutes.

And that's part of what makes exploring programming languages so entertaining.

This was fun so I think I'll make it a series looking at the stranger corners of familiar languages.

Migrating AKS workloads from AAD Pod Identity to Entra Workload ID

Jean-Luc Martel — Thu, 22 Oct 2026 02:49:22 +0000

You're looking at an AKS cluster for a client.

Somewhere in it, a pod is authenticating to Azure using AAD Pod Identity. It relies on AzureIdentity and AzureIdentityBinding custom resources. It depends on the Node Managed Identity (NMI) pods intercepting metadata requests on port 2579.

It is deprecated. Microsoft pulled the plug a year ago. The NMI intercept is a tax on the cluster's CPU and a liability in the threat model.

You need to move it to Entra Workload ID. It uses OIDC federation. It is cleaner. But cleaner does not mean easy.

Inventory the plumbing

Do not just generate new manifests and hope for the best.

Inventory what you actually have. Every AzureIdentityBinding is a dependency. Every AzureIdentity is a managed identity mapped to a specific set of pod labels.

Map each workload to its intended federated Kubernetes Service Account. If you do not know exactly what the old identity does, you will not know what permissions the new one needs.

Treat the old bindings as a map of your actual access patterns, not just a list of resources to delete.

Validate the exact strings

Entra Workload ID does not guess. It matches strings.

When you create the federated identity credential in Entra ID, the issuer is your AKS OIDC endpoint. The subject must be exactly system:serviceaccount:<namespace>:<name>.

Miss a colon. Add a trailing space. Use the wrong namespace. It will not give you a helpful error. It will just give you a 401 Unauthorized.

The pod also requires the label azure.workload.identity/use: "true". Without it, the Entra workload identity mutating webhook ignores the pod. No environment variables get injected. No service account token gets projected. The pod starts, but it cannot authenticate.

Test the token exchange before you move production traffic.

Exec into a debug pod. Request the token from the projected volume. Decode it. Check the aud (audience) and sub (subject) claims. Verify the exact string matches what you configured in Entra ID.

Stage the migration

Do not do a cutover. Do a migration.

Deploy the new service account and the federated identity credential. Update the workload definition to use the new service account and apply the required pod label.

Watch the logs. Verify the token exchange succeeds.

Once the new identity is serving traffic and passing health checks, kill the old AzureIdentityBinding and AzureIdentity resources.

Do not leave the old plumbing in the cluster "just in case." Deprecated resources left behind become permanent attack surface. If you do not need it, delete it.

Prove it works

You are not done when the pod starts. You are done when you prove it can do its job.

Check the Azure Activity Log. Look for the API calls.

See the new managed identity making the requests. See the old one go silent.

That is the only evidence that matters. The new identity reaches the required services with the intended permissions, and the old plumbing is gone.

The Green Padlock That Almost Cost Me a Contract

Jean-Luc Martel — Mon, 19 Oct 2026 20:15:12 +0000

Encryption is supposed to answer one question for the person sitting at the browser: am I really talking to who I think I'm talking to? On a corporate network running TLS inspection, the honest answer is often "not necessarily" — and most of the people relying on that network have no idea.

There are serious risks if the setup isn't done properly, but raising that flag can lead to uncomfortable discussions. I found that out the hard way, when demonstrating the problem nearly got my contract terminated.

The setup

The client ran a TLS inspection appliance on their internal network — the kind that decrypts outbound HTTPS so it can be scanned for malware and data loss before being re-encrypted and sent on. These devices are common and, in principle, defensible: most malware command-and-control and data exfiltration now hides inside TLS, so without inspection, a lot of security tooling is blind.

The way these appliances work is by man-in-the-middling every HTTPS connection. The appliance holds its own certificate authority. When you visit a site, it generates a certificate for that site on the fly, signs it with its own CA, and presents it to your browser. For this to work without warnings, that CA's root certificate is installed in the trust store of every managed machine — usually pushed automatically through group policy or device management.

Once that root is trusted, the browser stops complaining. The certificates the appliance generates look completely valid. No warning. No broken padlock. Nothing to tell the user their "encrypted" session is being read in clear text in the middle.

On top of that, the certificate issuance on this network was far more permissive than it should have been. The practical effect was that the internal CA could produce a trusted certificate for essentially any domain name — including public ones like google.com.

The problem I wanted to show

The thing I wanted the security team to see is subtle, and it's the part people consistently underestimate: once a root is trusted, the browser's special protections for high-value sites quietly stop applying.

Browsers do a lot to protect domains like google.com:

Certificate pinning ties those domains to a specific set of public CAs.
Certificate Transparency requires publicly trusted certificates to be logged in public, auditable logs.
HSTS forces connections to use HTTPS.

Every one of these makes an exception for locally installed roots. Pinning is skipped when the chain ends at an admin-installed CA. CT isn't required for private roots. HSTS is satisfied by any cert the browser considers valid. These exceptions exist on purpose — they're what stops a legitimate inspection appliance from breaking Google for the whole company. But they also mean that anyone who can issue from the trusted internal CA inherits all of that stand-aside treatment.

In other words: the protections you'd assume would catch an impersonated Google simply don't fire inside that environment.

The demonstration

To make this concrete rather than theoretical, I built a tiny proof of concept on my own machine:

I created a web page that said, in plain text, "I'm a fake Google page." No login form, no attempt to harvest anything — the content existed only to make the point unmistakable.
I added an entry to my own machine's hosts file pointing google.com at the IP serving that page.
I generated a certificate for google.com from the environment's trusted CA.
I loaded google.com in the browser.

The page loaded as google.com, with a green padlock, no warnings, and a certificate the browser reported as valid.

That was the whole point. The browser was not malfunctioning. It was doing exactly what the trust configuration told it to do. Every safeguard that should have flagged an impersonated Google had been designed to stand aside for a locally trusted root, and that's exactly what happened.

The hosts file was the deliberately contained part. It only affected my own machine — nobody else could reach that page. In a real attack, the hosts file stands in for an attacker's network position: a poisoned entry on the internal DNS resolver, ARP spoofing on a flat LAN, a rogue DHCP server, or a tampered proxy configuration. On a lot of internal networks, getting into that position is a lower bar than people like to assume. The demo swapped that step out for a one-line, single-machine change so there was no collateral effect at all.

The reaction

I sent the example to the security team as part of a technical debate about the risks of their setup. Their response was not to engage with the finding. It was to try to have my contract terminated, on the grounds that I had been "phishing."

I had not shared the fake page with anyone. There was no victim, no credential harvesting, no email, no deception of a single other person. The entire thing lived on one machine — mine — and demonstrated a property of their infrastructure. Calling it phishing was a category error, and I think a revealing one.

Why the reaction matters more than the bug

Phishing is the use of deception to trick a person into giving something up. A proof of concept that impersonates a site to no one but yourself, to demonstrate a flaw in a system you were engaged to assess, is the opposite of that: it's disclosure. The deception, such as it was, was aimed at a browser I controlled, to reveal something true about the environment.

What the reaction demonstrated is a failure mode I've seen more than once: when a finding is uncomfortable, it's easier to attack the method than to sit with the conclusion. The conclusion here was genuinely uncomfortable. It wasn't "the appliance has a bug." It was "the way trust is configured on this network means the guarantees your users assume they have — that a green padlock on google.com means they're talking to Google — don't hold, and an insider with issuance rights can impersonate anything to anyone, invisibly."

That's not a comfortable thing to hear about a system you own and defend. But shooting the person who showed it to you doesn't make it less true. It just guarantees the next person won't tell you.

What should have happened

The appropriate response to that demo was a conversation about controls, not contracts:

Keep the interception CA separate from the internal issuing CA, so a compromise of one doesn't hand over the other.
Apply name constraints to internal roots so they can only sign the domains they're supposed to.
Lock certificate profiles so requested names have to match the requester's identity, instead of allowing arbitrary domains.
Restrict and audit issuance — know who can request certificates and log everything that gets signed.
Protect the CA key properly, ideally in an HSM, because whoever holds it can impersonate anything to every managed device.
Be honest with users that inspection is happening, and keep sensitive categories like banking and health out of scope.

None of that is exotic. It's the standard hygiene for running TLS inspection without turning it into a single, silent point of total compromise.

The takeaway

The green padlock is a promise, and TLS inspection quietly rewrites the terms of that promise for everyone on the network. That can be done responsibly. But the first step is being willing to look at what your own configuration actually allows — especially when someone hands you an uncomfortable demonstration of it on a single contained machine, and asks you to think about it.

Attacking the messenger is a choice. It's just not a security control.

BigQuery BI Engine deserves a benchmark before you dismiss Looker Studio as slow

Jean-Luc Martel — Sun, 18 Oct 2026 07:00:00 +0000

Someone is complaining that Looker Studio is slow.

A dashboard takes forty seconds to load.

The immediate reaction is to blame the tool.

"Looker is too slow."
"We need to migrate."
"We need a custom frontend."

Look at the query logs.

The dashboard is running a full table scan on a 500GB fact table every time a user changes a date filter.

Looker Studio isn't the bottleneck. The query path is.

Blaming the visualization layer for a warehouse execution problem is like blaming the paint for a crumbling foundation.

Before you throw money at BigQuery BI Engine, isolate the variables.

Performance hypotheses usually mash three distinct things together.

First, connector caching. Is the dashboard just failing to cache the extract?

Second, query shape. Are you running a distinct count on a high-cardinality string?

Third, BI Engine support. Does the query use supported data types and operations?

BI Engine can't fix a query it doesn't support.

If your query falls back to the main BigQuery engine, the BI Engine reservation sits idle while you pay for it.

Don't guess. Benchmark.

Take the five most-used, slowest queries from the dashboard.

Run them without BI Engine. Record the latency. Record the bytes billed.

Turn on a BI Engine reservation. Run the exact same queries. Record the latency. Record the bytes billed.

Calculate the cost of the reservation against the savings in bytes billed and the value of the latency reduction.

If the query isn't supported, latency won't drop.

If the query is just badly shaped, latency might drop, but the cost won't justify the reservation.

A reservation is a blunt instrument. It solves a specific class of problem and ignores everything else.

Before you provision one, ask yourself: which exact metric will prove the bottleneck is in the compute layer, and not in the query design?

If you can't answer that, you haven't fixed a performance problem.

You've just bought a faster way to run a bad query.

Azure subscription Owner is an access-delegation privilege

Jean-Luc Martel — Tue, 13 Oct 2026 07:00:00 +0000

You hand someone the Owner role on an Azure subscription.

You think you're giving them the keys to manage resources. VMs, databases, networking. You think you've made them a builder.

You look at the role name. "Owner." It sounds like they just own the stuff inside the box.

They don't. They own the box. And they own the locks.

It’s not tenant-wide administration. A subscription Owner isn't a Global Administrator. They can't rewrite the rules for the entire Azure tenant. They can't even remove protected deny assignments if someone locked a resource down properly.

But they don't need to.

Because within the boundary of that subscription, Owner is an access-delegation privilege.

They can assign powerful data-access roles. They can hand out the keys to the data plane. They can change legacy Key Vault access policies. They can grant someone else the right to read the secrets, even if they themselves were never explicitly granted that data access.

The control plane becomes a backdoor to the data plane.

We treat subscription management and data access as two different conversations.

They aren't.

If you want to protect your data, you need to look at who holds the Owner role. Review who can assign those powerful data-access roles. Review who can quietly flip a legacy Key Vault access policy to let themselves in.

When that Owner grant was approved, what was the justification? "Needs to manage the dev environment." "Needs to deploy the infrastructure."

But was the grant approved with those delegation powers in view?

If not, you haven't granted resource administration.

You have granted data access. And you didn't even realize it.

The data engineering reality of GNNs in production

Jean-Luc Martel — Thu, 08 Oct 2026 20:32:16 +0000

Your offline GNN looks perfect. The AUC is high. The precision is great.

You ship it to production to catch fraud or optimize a supply chain.

Then the metrics tank.

The model isn't broken. The data is just too old.

It is making real-time decisions using neighborhoods and embeddings from yesterday, or last week. In fraud, a node's connections change in milliseconds. In supply chain, they change in hours.

The model is evaluating a ghost.

GNNs don't just need features. They need fresh ones. They need neighborhoods that reflect the current state of the world.

This means your graph storage and feature-update pipelines cannot be an afterthought. They must be engineered to the exact freshness the use case demands.

A batch job running at midnight is fine for predicting customer churn. It is catastrophic for blocking a fraudulent transaction at checkout.

Data engineering must run on the same clock as the business decision.

Don't trust your architecture diagram. Trust a trace.

Pick one production decision and trace it backward. Start at the served model input.

Look at the embedding it consumed. Look at the neighborhood it aggregated.

Trace those back to the graph storage layer. Trace that back to the feature-update pipeline. Trace that back to the source system change.

If a user changes their email, or a supplier misses a shipment, how long does it take for that reality to reach the model?

If you can't measure the lag, you can't trust the prediction.

Offline evaluation assumes the model has access to the exact state of the graph at the moment of the event. Production rarely offers that luxury.

A zero-disruption PodDisruptionBudget can block an AKS upgrade

Jean-Luc Martel — Thu, 08 Oct 2026 04:06:50 +0000

Your AKS cluster upgrade is stuck.

The control plane is waiting for a node to drain. The node is waiting for a pod to terminate. The pod is refusing to move.

Look at the logs. The workload has a PodDisruptionBudget that allows exactly zero disruptions.

It is doing exactly what it was told to do. Protect the application at all costs. Even if it means halting the cluster upgrade.

To upgrade a node, AKS has to drain it. It cordons the node and evicts the pods.

But the eviction process respects disruption budgets.

If your deployment has three replicas, and your budget dictates that three must remain available, the eviction fails. The node stays tainted. The upgrade stalls.

You didn't just configure high availability. You configured a deadlock.

Replica headroom isn't just a best practice for scaling. It is the physical space required for the control plane to do maintenance.

Before your next maintenance window, look at your math.

Check your minAvailable or maxUnavailable settings against the actual number of running replicas.

If you use minAvailable with a percentage, remember that Kubernetes rounds up. A setting of 100% means zero disruptions allowed. A setting of 90% with three replicas rounds up to three.

You have to leave room for the scheduler to breathe. Configure these values deliberately, not just to pass a compliance check.

High availability means surviving a failure. It shouldn't mean blocking your own operations team.

Your availability policy guarantees the app survives a zone failure. But has anyone tested whether it survives routine maintenance?

If your zero-disruption configuration blocks your upgrade pipeline, you haven't built a resilient system.

You have built a hostage situation.

I rebuilt four dead systems with AI. The failures all landed in the same place.

Jean-Luc Martel — Thu, 24 Sep 2026 16:55:00 +0000

Four reconstructions, four kinds of missing documentation, and the defects landed in the same place every time.

In HAL/S it was the characters the scanner destroyed. In the Nautilus simulator it was the parameters Verne never wrote down. In Wikicarta it was the desktop window no headless harness can open. In the compressor it was the part of the input space the corpus never sampled.

Different projects, different languages, different failure symptoms. One shape: the defects clustered wherever no automated check could reach, and in every case the code looked fine. Tests green, types satisfied, output plausible.

What was actually being varied

These weren't four projects that happened to be about old software. They were one experiment with a controlled variable: how degraded is the source of truth the model works from, and can it be caught being wrong?

arm	source of truth	oracle	outcome
HAL/S	degraded formal spec + preserved code	strong, independent	faithful reconstruction; 12/12 preserved programs run
Nautilus	a novel	none — one was built	5 findings; the legend fails on every axis
Wikicarta	memory of a product	none — human judgement	shippable app; every defect at the runtime seam
compressor	behaviour only, key sealed	strong, opened after freeze	decoder 100%, encoder 8.3%

The intuition going in was that performance would degrade smoothly as the source got worse. It didn't. Something else predicted the results, and it cut across all four.

Determined beats hard

The reconstruction quality tracked whether the available evidence determines the answer — not how difficult the task was.

The compressor makes this measurable, because both halves ran under identical conditions. The decoder recovered the bitstream format at 100%, because a format is one-way determined: if you want to decode the stream, exactly one answer works. The encoder scored 8.3% on the same format, because its heuristics — match-finder strategy, tie-breaking, greedy versus lazy — are choices that leave no trace in the decompressed bytes. Same model family, same oracle, same week. A 12× gap, entirely explained by whether the evidence pinned the answer.

The other three say the same thing in their own terms:

HAL/S. The 485 grammar productions were determined by the page images, and the reconstruction got them. The characters the scan physically destroyed were determined by nothing, and no amount of model capability recovers them. That limit is information-theoretic. Better OCR narrows it and never closes it.
Nautilus. The physics is determined; the equations produce the same crush depth for anyone. The parameters Verne omitted are determined by nothing, so submerged endurance ranges across 6 to 211 hours — a 35× spread — and battery capacity across 100×. Those spreads aren't noise around a finding. They are the finding.
Wikicarta. Parsing, backoff, cache naming and database ordering are determined by their inputs, and compiled correct on the first or second attempt, consistently. Whether the category wheel feels like Encarta is determined by nothing you can write down, and stayed a human call from start to finish.

The practical version: before handing a legacy task to a model, ask whether the evidence you have actually pins the answer. If it does, expect the model to find it and build a cheap mechanical check to confirm. If it doesn't, expect a confident, plausible, unmarked invention — and note that the model will not tell you which mode it's in, because from the inside they feel identical.

A strong oracle is not the protection you think

The compressor was built to demonstrate that a strong oracle masks confabulation. It mostly refuted that.

The reconstruction's central error — assigning Huffman code lengths by symbol index rather than in heap-extraction order — failed the byte-match test loudly. The oracle caught it. What the oracle did hide were two mechanisms the corpus never triggered: a block-splitting path that needs inputs larger than 8 KiB to fire, and a match-finder chain cap that needs input more self-similar than anything tested.

So the masking risk is not a property of the oracle's strength. It's a property of the input space you sampled. A strong checker over a narrow corpus hides precisely the mechanisms your corpus never exercises, and hides them silently, because everything it can see is green.

Strengthening the checker does nothing for this. Widening the corpus does.

That generalises past compression. It's the same failure as a test suite with excellent coverage of the paths anyone thought to write a test for.

What the human actually contributed

I expected to spend supervision effort reviewing code. Across all four projects, almost none of it went there. The load-bearing human contributions were:

HAL/S — deciding not to vote between OCR engines, and then reading 485 grammar productions against page images by eye.
Nautilus — signing the invariants and making them append-only, so the model could patch the physics but was structurally unable to weaken the checks that judge it.
Wikicarta — being the runtime. Clicking the button in the real window, on the real WebKit build, where the blocking dialog freeze and the cross-realm instanceof failure lived.
compressor — holding the seal. Not reading the answer key until the reconstruction was frozen at a tag.

Two categories, and neither is code review. One is supplying what no code can check — an eye on a page image, a hand on a mouse. The other is designing the checks that are possible and then putting them structurally out of the model's reach.

The OCR decision is the sharpest of the four, because it's a judgement call that separates a good result from a worthless one that looks identical. Running three OCR engines and taking the majority is the obvious move and it is wrong: the errors are correlated, so a vote launders error rather than cancelling it, and no fusion can recover a character that no engine produced. Survivable in prose, fatal in a grammar where the metacharacters are the content. Take the vote and you get a clean, plausible, silently corrupt grammar — and every downstream artifact inherits it.

The Nautilus invariant signing is the most reusable. Twelve invariants live in version control with a manifest checksum only a human re-signs. When the fuzzer finds a violation, the repair workflow requires patching the core and forbids touching the invariant. An agent optimising against a check it can also edit is not being checked.

Build the oracle before the artifact

Ranked by how much verification apparatus existed before the real work started, the four projects come out in the same order as how much I trust their conclusions.

Nautilus had determinism, golden-stream replay and malformed-input fuzzing green in its check harness before a single equation was written, plus a frozen protocol and a rule that every spec parameter traces to either a sourced claim or a documented gap range. Its numbers are the most defensible in the series. The compressor's seal, cold-recall protocol, freeze tag and grading harness are most of that repository; the reconstruction is the small part. HAL/S inherited an independent interpreter, which is a luxury and which is why it is the easiest arm.

Wikicarta had none of this available, and it shows — its conclusions are qualitative, argued from patterns across features rather than measured. They're still the most immediately useful findings in the series, and they're the least provable. Both things are true because of the same missing oracle.

There's a specific trap worth naming. A verification harness that finds nothing is indistinguishable from a broken one. Nautilus's 24-hour soak ran 879,643 ticks and found zero invariant violations, which is only meaningful because the harness was first validated against a deliberately poisoned invariant — which it caught, minimised from seven commands to two, and packaged. Test the test.

Three results that went the wrong way

A series that only reports confirmations is a demo reel. These four produced three clean refutations, and they were the most informative outputs:

The compressor predicted the classic overfitting curve — strong on trained data, weak on held-out. It came out inverted, 8% trained against 71% held-out, because the split tracked difficulty rather than seen-versus-unseen.
It also predicted the oracle would mask confabulation. The oracle caught the main error; a narrow corpus hid the rest.
Nautilus's fuzzer was built to find physics exploits. It found none. The only finding in the whole loop was a flaw in the experiment design — the genetic algorithm had been optimising survival rather than performance, because the controller pinned its own objectives.

Each of those changed the conclusion. None would have surfaced from reasoning about the problem.

Where this doesn't reach

Four projects, one operator, no control group. There is no human-only reconstruction to compare against, so nothing here says AI was faster or better than the alternative — only what it did and did not manage. The projects were chosen partly for being interesting, which is a selection effect. Model versions moved during the work. Three of the four are Rust, which is a small language sample.

The one number I'd most like and don't have is a like-for-like human baseline on the same corpus. That would need someone to transcribe 485 productions and reconstruct an encoder without a model, which is a real cost for a comparison nobody has funded yet.

What I'd tell someone starting

Handing legacy code to a model works, and it works well enough that the interesting question is where to spend your own attention. Four projects say:

Not on the code. The logic was right nearly every time. Spend it at the host, runtime and OS boundary, where every real defect in the desktop project lived and where no harness reached.

On knowing which mode you're in. Where the evidence determines the answer, expect success and build the cheap mechanical check — production numbering, reference integrity, a verbatim grounding gate. Where it doesn't, expect confident invention, and force it into the open: a divergence log, a gap inventory with plausible ranges, an ambiguity recorded rather than resolved inline. Every one of these projects has a file whose entire job is holding the things that were not derivable, and those files are what make the rest credible.

On the checks the model can't edit. Sign them, hash them, make weakening them require a human.

On widening the input space rather than tightening the oracle. The gaps that survive strong testing are exactly the ones your corpus never triggers.

All four repositories are public, with the divergence logs, gap inventories, transcription artifacts and grading harnesses included — those are the parts worth reading, more than the reconstructions themselves.

space-shuttle-sdk · nautilus-sim · wikicarta · compressor

The Decoder Scored 100%. The Encoder Scored 8.3%.

Jean-Luc Martel — Tue, 22 Sep 2026 13:08:00 +0000

This is the last of four experiments in AI-assisted reconstruction of legacy systems. The first three degraded the source of truth — a scanned spec, then a novel, then a memory of a product. This one removes it completely.

The target is LHA's -lh5- method — Haruyasu Yoshizaki's 1988 archiver, LZSS with an 8 KB window plus static Huffman.

In the experiment, there is no spec and no source, only behaviour, plus an oracle that will answer any question you ask it about outputs. It is the only configuration where the two questions that actually matter become measurable rather than assertable:

Behavioural match is not understanding. You can pass every test and hold a wrong model of the mechanism, with untriggered modes waiting in the part of the input space you never sampled.
A strong oracle masks confabulation. The model fits observed outputs while inventing a plausible-but-wrong internal mechanism, and because the tests keep going green, the error is rewarded rather than exposed.

Measuring those requires reconstructing blind against an answer key you are not allowed to look at. Everything else in the design follows from that one constraint.

Choosing something the model hasn't memorised

WinRAR was the instinct and is the wrong specimen. RAR and DEFLATE are heavily memorised, so you'd measure recall rather than reconstruction. RAR5 is far too large. Its decompressor is documented while its compressor has no public ground truth, which is exactly the half you need to grade. And it is live commercial software rather than abandonware.

LHA satisfies four properties that a lot of candidates fail:

The original runs, so the oracle is unlimited — compress or decompress anything on demand.
The algorithm is public, so a real answer key exists and the confabulation can be graded rather than merely suspected. A never-solved black box demonstrates that the pitfall exists but can never put a number on it.
It's bounded — a few hundred lines, reconstructable.
The encoder heuristics are underdetermined by the output. Match-finder strategy, minimum match length, tie-breaking, greedy versus lazy matching, block splitting are all encoder-side choices that leave no trace in the decompressed bytes. That is the confabulation sweet spot.

The general LZSS-plus-Huffman scaffold is in every model's training data. LHA's specific choices are not, and those specifics are what gets measured.

The controls

Three, and they're the reason the result is a measurement instead of an anecdote.

The seal. The original source sits in the repo, git-ignored, hashed. At unseal, sha256sum -c verified 107 of 107 files intact, and the manifest's own digest matched the value pinned in the sealing document. Reading it before the reconstruction was frozen is the single act that would have destroyed the experiment, so it was made mechanically detectable.

The freeze. The reconstruction was tagged and committed before unsealing, so no artifact could be quietly edited to fit the answer once the answer was visible.

The cold-recall file. Before any reconstruction began, a model wrote down everything it already believed about LHA, committed. Without it there is no way to distinguish "derived from behaviour" from "remembered from training," and the whole exercise collapses into a memory test.

Three models did the work, deliberately split: Gemini 3.1 Pro reconstructed the decoder, Codex/GPT-5 the encoder and the cold-recall baseline, Claude ran the design thread.

The decoder: 100%, and it doesn't count for much

Nineteen of nineteen round-trips, exact. The format was fully recovered.

The sharpest single result lives here. LHA's generic macro defines PBIT 5, so anyone reading the source would guess a 5-bit position-table field width. The -lh5- method actually uses pbit = 4 with np = 14. The reconstruction recovered the method-specific 4 purely from round-trip behaviour, against a constant that would have misled a source-reader.

But checking that win against the cold-recall file deflates most of the rest. Nearly every format fact the decoder "recovered" was already in the model's committed prior: the 8 KB window, the threshold of 3 and max match of 256, the roughly 510-symbol alphabet, the 19-entry precode with its 0/1/2 zero-run encoding, the 16-bit block count, the position-slot decode. The 100% credits prior knowledge, not derivation.

Which is unsurprising, because the format is one-way determined. If you want to decode the bitstream, there is exactly one answer, and both memory and derivation converge on it.

The encoder: 8.3%, and it counts for everything

Byte-match against real lha, on a fresh-seed corpus the reconstruction had never seen:

input class	trained on?	byte-match
text	yes	0/3 (0%)
source	yes	0/3 (0%)
structured	yes	0/3 (0%)
repetitive	yes	1/3 (33%)
random	no	2/2 (100%)
edges	no	3/5 (60%)
trained total		1/12 (8.3%)
held-out total		5/7 (71.4%)

A perfect decoder coexisting with an 8% encoder is match ≠ understanding in one line. The format was recoverable. The encoder's choices were not.

The hypothesis was wrong, in a useful direction

The plan predicted the classic overfitting shape: high scores on trained classes, dropping on held-out ones. The data inverted it. Held-out scored 71%, trained scored 8%.

The split tracks difficulty, not seen versus unseen. The trained classes are exactly the ones that exercise real -lh5- compression; the held-out set is dominated by incompressible and trivial inputs that fall through to stored mode or hit trivial paths. Random data scores 100% because no compression heuristics ever run.

So the honest statement isn't "an 8% to 71% generalisation gap." It's this: byte-identity is reached only where the encoder heuristics never fire. Wherever genuine compression happens, the reconstruction diverges.

The rates were identical on the trained-seed and fresh-seed corpora, so the divergence is systematic rather than instance overfitting.

The confabulation catalogue

Every inferred encoder decision, graded against the now-visible source:

decision	inferred	actual	verdict
longest-match search	yes	matches	correct
tie-break among equal lengths	nearest offset	chain walked recent→old, updates only on strictly longer	derived correct
greedy vs lazy	one-step lazy	emits literal if next match is strictly longer	derived correct
block splitting	"one block; unknown for larger"	flush at 32 KB buffer threshold	never triggered — correctly scoped
match-finder chain cap	not modelled	a hard limit makes the finder non-exhaustive	wrong, and it passed anyway
Huffman code-length assignment	canonical, tie order unmatched	lengths assigned in heap-extraction order	visible divergence

Two of those are clean wins. Nearest-offset tie-breaking and one-step lazy matching are both marked as unknown in the cold-recall file — "do not know whether lha -lh5- does this," "not a specific memory." Derived from black-box behaviour, with documented prior ignorance.

There's a caveat that inverts on inspection. Codex wrote both the recall file and the encoder, which normally weakens an independence claim. Here it strengthens it: a model cannot conceal prior knowledge from its own committed statement of ignorance. Deriving both heuristics after recording that it didn't know them is the cleanest evidence in the run.

The failure is concentrated in one place. LHA builds a frequency heap and assigns code lengths in the order symbols leave the heap, governed by the exact comparison semantics of its sift-down — not by symbol value. Textbook canonical Huffman breaks ties by symbol index, which produces different lengths for equal-frequency symbols, which cascades through the entire bitstream. The reconstruction localised the problem correctly and even tried a heap hypothesis, but never replicated the precise sift order.

The second hypothesis was also wrong, and that's the finding

The project was built to show that a strong oracle masks confabulation. It mostly didn't.

The core error — the Huffman ordering — is a visible divergence. It fails the byte test loudly. The oracle caught it, exactly as an oracle should. Only two gaps were genuinely hidden: the never-triggered block split, and the unmodelled chain cap, which is latent because the corpus never produced input self-similar enough to expose it.

Both are artifacts of a corpus that tops out at 8 KiB.

So the masking risk isn't a property of the oracle's strength. It's a property of the input space you sampled. A strong oracle over a narrow corpus hides exactly the mechanisms your corpus never triggers, and it hides them silently, because everything it can see is green. Making the oracle stronger doesn't help. Making the corpus wider does.

That reframing is more useful than the result I expected to get, and I would not have got it by reasoning about it. It needed the sealed envelope.

What it cost to know any of this

Final tally across the encoder decisions: six recalled-correct, five derived-correct, one derived-wrong-but-passed, two never-triggered, one visible divergence.

The infrastructure to produce those thirteen data points — the pinned oracle, the deterministic corpus generator, the seal and its manifest, the cold-recall protocol, the freeze tag, the grading harness — is most of the repository. The reconstruction itself is the small part.

That ratio is the practical lesson. If you want to know whether an AI-assisted reconstruction actually understood the system, rather than whether it passed, you have to build the apparatus that can tell you, and you have to build it before you start. Afterwards, every artifact is contaminated by the answer.

One thing remains open. A fully independent contamination estimate needs a repeat where the recall file and the encoder come from different model families, closing the theoretical shared-prior channel that the current caveat only argues around.

Next, the wrap-up: what four projects across four degrees of source degradation say about where the current generation of models actually fails, and why the failures kept landing in the same place.

Code: https://github.com/singular-state/compressor. LHA was written by Haruyasu Yoshizaki in 1988; the original source is used here only as a private grading key, held out during reconstruction and consulted once. The reconstruction is clean-room, so the measured gap between "passes the oracle" and "matches the source" is legitimate.

Rebuilding Encarta showed me exactly where AI-written code breaks

Jean-Luc Martel — Fri, 18 Sep 2026 14:01:00 +0000

The bug that froze the window was type-correct. It compiled, cargo check passed, the tests passed, and the operating system reported the application as not responding.

It was a Save As dialog called through the Tauri plugin's blocking API from inside a command, which blocks the event loop the WebKit window is running on. Nothing in the type system objects to that. Nothing in a headless harness notices. The only way to find it is to be a person, at a real desktop, clicking the button.

That defect is the whole finding of this project, and the project took four phases to earn the right to state it.

The setup

Wikicarta rebuilds Microsoft Encarta on Wikipedia — the visual browser, the atlas, the timeline, the research organiser, MindMaze — as a Tauri v2 desktop app in Rust and React, with an instant toggle between Live mode against the Wikimedia API and Offline mode against a local ZIM archive.

It is the third of four experiments in what current AI can do with legacy systems, and it occupies the worst position of the four. HAL/S had a formal spec and an independent interpreter to check against. Nautilus had a novel and built its own physics oracle. Wikicarta has neither. The source of truth is what the product felt like, and the only oracle is human judgement. There is no test that tells you whether the category wheel feels like Encarta.

An AI agent wrote essentially all of it. What follows is where that went well, where it went badly, and the fact that those two places were not the ones I expected.

The modern substrate is never the shape its docs claim

The offline story was supposed to be a pure-Rust ZIM reader. The zim crate looked right: random access, deferred loading, Send and Sync. It panicked with an integer overflow in parse_article_list before finishing the open call, against a real modern archive.

That pattern recurred with no relationship between the cases. TextExtracts HTML turned out unusable as a reader format, so article HTML comes from action=parse instead. Live image URLs came back protocol-relative — //upload..., not the https://... everything downstream assumed.

Three unrelated features, one rule: for a revival, the modern content, format or API is always slightly different from its specification and from the model's memory of it. Probe a real sample before building on it. The AI will confidently build against the documented shape, because the documented shape is what it was trained on.

The decision that paid for everything

Before it was provably needed, the reader was decoupled from its content source behind a single trait:

trait ArticleSource {
    fn get_article(title: &str) -> Article;
    fn search(query: &str) -> Vec<SearchResult>;
    fn get_image(key: &str) -> Vec<u8>;
}

Adding a whole second content source later — live Wikipedia alongside offline ZIM — was two commands and a mode flag. No reader rewrite. Swapping the storage layer from JSON files to SQLite was invisible above the command boundary.

For software that assumed one sealed data source — a CD, a bundled database, a mainframe — inserting that seam first is what converts every later modernisation from a rewrite into an adapter. It is also the single decision an AI agent is least likely to make unprompted, because at the moment you make it there is exactly one source and the abstraction looks like overhead.

Two features in the final phase were pre-paid by decisions like it. The attribution aggregator became a GROUP BY rather than a re-fetch, because provenance had been stamped at save time back when the notes feature was built. Bundling the kiwix-serve binary into the package was a two-line change, because binary discovery had long been a prioritised candidate list rather than a hard-coded path.

Packaging work disproportionately cashes in — or punishes — architectural decisions made when the feature that needed them didn't exist yet.

Testability turned into architecture, which turned out to be good

The only layer testable without a desktop window is pure functions. So the code got steadily shaped so that the hard parts are pure functions: URL rewriting, upstream reconstruction, cache naming, database round-tripping and ordering, bookmark deduplication, legacy JSON import. The Rust suite went from nothing to 23 offline tests, and each feature's desktop-only remainder shrank to a short explicit list.

"What can I verify headlessly?" is a useful question about how to structure code, not just about how to test it. Under an AI agent it becomes close to essential, because the agent's verification loop is only as good as the surface it can reach.

Where the defects actually were

The pure logic — parsing, backoff, the media-cache URL rewrite, database ordering, live-title normalisation — compiled correct and passed on the first or second attempt. Consistently.

Every real bug was at the runtime seam:

The blocking Save As that froze the WebKit window.
DOM nodes inside an <iframe> coming from a different JavaScript realm, so an instanceof Element check in the parent React shell fails on a clicked target that is unambiguously an element. Realm-safe DOM handling is mandatory when the app owns iframe content but drives it from the parent.

Both are invisible to the type system and to any headless harness. Both required a person at the real window.

The implication for anyone running an agent on desktop software: the logic is rarely where the risk is. Human validation time should be spent almost entirely at the host, runtime and OS boundary, and hardly at all reviewing the algorithm the model just wrote.

There is a compounding effect worth noting. Once the blocking-dialog lesson was recorded, the archive picker was built on the non-blocking callback pattern from the start — the backend opens the dialog with a callback, validates and persists inside it, and reports back over an event the React shell listens for. A hard-won runtime lesson, written down, becomes a convention rather than a bug re-lived.

Checking a model with code beats checking it with a model

MindMaze needs quiz questions, so a standalone crate generates them from article text through an LLM. Model output is treated as untrusted input: strict JSON parse, structural validation, then a verbatim grounding gate — the answer must appear literally in the source article — before anything is stored. Questions carry the revision id they were generated from, so they can be flagged stale when the article moves.

The roadmap had proposed a second LLM as a fact-checker. A deterministic string check does the job for a fraction of the cost, and the rule generalises: reach for a model to check a model only when the property genuinely isn't codeable.

The first real run against Mount Everest then demonstrated the limit of that rule. Two of the three questions were excellent, including one on the 1953 first ascent with genuinely plausible distractors. The third asked which of the options is a section heading. The answer was "Name."

That question is perfectly grounded. "Name" is a section heading and it appears verbatim in the source, so the gate accepted it. It also tests document structure rather than knowledge. Is the answer in the source? and is this a good question? are different properties, and only the first one is codeable. The fix belonged at the prompt layer — instruct the model to ask about substantive facts, never structure or formatting — which eliminated the meta-questions on the next run.

Quality control for LLM output is layered: a cheap deterministic gate for the codeable invariant, prompt engineering for the one that isn't. Confusing the two gets you a system that passes its own checks and produces garbage.

Modernisation re-opens every surface the original had sealed

A CD-era encyclopedia needed no TLS trust roots, no rate-limit backoff, no per-source provenance, no SSRF guard, no cross-platform custom-scheme origins, no data migration path. Each modern dependency reintroduces one.

The media cache restricts upstream hosts to *.wikimedia.org so it isn't an open proxy. User-selected ZIM files are validated for existence, extension and non-emptiness. The timeline parses presentation HTML defensively. None of this is polish. It is the actual cost of "keep the experience, replace the substrate," and it should be budgeted as first-class work rather than discovered late.

The last mile is where the sandbox stops being able to prove anything

Every earlier phase ended with a headlessly provable artifact. Packaging produced the first deliverable the agent's environment structurally could not generate — there is no cargo-tauri and no dpkg-deb in it, so the .deb and the AppImage can only be built by a human on a real host.

What kept that honest was decomposing the deliverable into layers each verifiable in isolation: config validity, the fetch script (run end to end against the real 20 MB tarball), runtime path resolution with its own tests. Only the final bundler invocation stayed unproven — and rather than paper over that with a CI workflow nobody had run, the repo says so.

That work also surfaced a boundary the configuration layer silently drops. Tauri copies bundled resources into the payload but does not guarantee the executable bit survives, which varies by target and archive format. A bundled binary can land non-executable and fail at spawn with a permission error that points nowhere useful. The fix lives in runtime code, not packaging config — the resolver chmods the binary before returning it.

A related lesson came from the memory pass. The roadmap prescribed virtual scrolling for an assumed giant rendered DOM. It was architecturally inapplicable: articles render inside <iframe srcDoc>, a separate browsing context whose internal DOM React cannot window, and only one is ever live. The actual unbounded growth was one layer up — a reader history that pinned every visited article's full HTML in the heap forever. When the roadmap and the architecture disagree, re-read the architecture. An inherited roadmap encodes assumptions that later decisions have already invalidated.

And since WebKitGTK has no performance.memory, the app now measures and displays its own retained HTML instead. The instrument ships with the product and works on the real target, rather than existing only in a Chromium devtools session that the shipping app never runs in.

The constraint was never throughput

Across every substantial feature, the agent's implement-and-verify cycle was fast and mostly correct. Each feature still ended with a short list of things only a person at the real window could confirm, and those checks are serialised.

That is the bottleneck in AI-assisted desktop revival. Not writing the code. The human-in-the-loop runtime check that no headless harness currently replaces — which is also, precisely, where all the bugs were.

Next: the same question with the tests made as strong as possible and the answer key sealed in an envelope, to find out whether passing every test means the model understood anything.

Code: https://github.com/singular-state/wikicarta. Not affiliated with Microsoft; Wikipedia content is CC-BY-SA and the app ships an attribution exporter for the sources you actually used.

AI proved that the Nautilus crushes at 340 metres. Verne dives it to 16,000.

Jean-Luc Martel — Thu, 17 Sep 2026 16:09:00 +0000

Twenty Thousand Leagues Under the Sea gives the Nautilus a hull of 5 cm plating on a 4 m radius. Thin-cylinder hoop stress puts the crush depth at 342 metres. Verne narrates a dive to 16,000.

At 16,000 m the external pressure is about 1,608 atmospheres and the hoop stress is 12.9 GPa, roughly 47 times the plating's yield. Surviving it would need approximately 2.34 metres of steel, which weighs more than the entire boat.

That number came out of a simulator, not a spreadsheet, and the simulator is the point. This is the second of four projects testing what current AI can do with legacy systems, and the variable being changed is the quality of the source. The last one had a formal spec, real preserved code, and an independent implementation to check against. This one has a novel.

Extracting a spec from fiction

The project's thesis is one sentence: legacy documentation lies by omission, not by error. Verne itemises the art in the Nautilus's salon. He never gives the air budget.

So the first phase produced no code at all. Two independent claim-extraction passes ran chapter-by-chapter over the French original, each tagging every physical assertion with its quote, chapter, category, and a precision marker — stated, hedged, implied, or demonstrated. The two passes were then diffed and reconciled into 731 claims, with a human reviewing the reconciliation. Contradictions were recorded and deliberately not resolved, because which resolution you pick changes the answer and that choice needs to be visible rather than buried.

Then the harder half. A simulator needs numbers the novel never supplies, and inventing them quietly is how you get a result that means nothing. Every such parameter became a gap entry with a plausible range and a sourced rationale, and the gate rule was absolute: every parameter in the spec traces to either a claim with a chapter citation or a gap with a range. Nothing enters as a bare constant.

Ranges are anchored where possible to Plongeur, the French Navy submarine launched in April 1863 — the actual state of the art Verne could have known about. Plongeur was 43 m and about 420 tonnes, with 23 compressed-air tanks at 12.5 bar occupying 153 m³ of its hull to drive an 80 hp engine to 4 knots and a range of 5 nautical miles.

Two things follow from that immediately. Stored gas and ballast machinery eat an enormous fraction of a submarine's internal volume, so the Nautilus's breathable space is far smaller than its 1,500 m³ envelope. And 60 kW moved a 420-tonne boat at 4 knots, which sets expectations for what Verne's stated 1.47 MW should do to 1,500 tonnes.

The core is a pure function

The simulator is Ada, built as a pure function of (spec, seed, command stream). No wall clock, no filesystem access, no network, no environment reads inside the core. Everything crosses a single seam — JSON Lines on stdin and stdout — and the UI, the genetic algorithm, the fuzzer, the test harness and the save files all speak the same protocol. There are no side channels.

Three consequences fall out of that shape, and all three matter for supervising an AI:

Any result reproduces from files in the repo plus a binary hash. The spec is canonicalised and SHA-256'd by the core itself; the wrapper cross-checks and aborts on mismatch. Every recording is pinned to a spec hash.
The protocol was frozen before the physics existed. Determinism, replay and malformed-input fuzzing were green in the one check harness before a single equation landed.
The invariants are append-only and human-signed. Twelve of them live in invariants/, covering physical bounds, monotonicity, and event ordering — depth is never negative, stored energy never rises, CO₂ only increases while sealed and may drop only to fresh air via a surface renewal, no state tick may follow a crush event.

That last one is the load-bearing supervision decision. When the fuzzer finds a violation, the repair workflow says the AI patches the core, never the invariant. Weakening an invariant to make an exploit pass is prevented by a checksum in invariants/manifest.sha256 that only a human re-signs. The model is free to fix the physics and structurally unable to move the goalposts.

Five findings

Running the committed spec under real physics and comparing to the narrative:

	axis	the legend	the physics	robust to gaps?
F1	buoyancy	dives at will	under-ballasted — the mass budget only closes in fresh water	yes, it's arithmetic
F2	speed	50 knots	~15 kn; 50 kn needs ~24× the stated power	yes, 15–22 kn across the drag range
F3	range	20,000 leagues	~1,400 nmi per charge → 30–4,000 recharges of a battery whose capacity is never stated	capacity unknown to 100×
F4	depth	16,000 m	crushes at ~340 m	yes, 249–435 m
F5	endurance	weeks submerged	6–211 h; no CO₂ scrubbing means a 1–2 day surface tether	endurance unknown to 35×

F1 is the most elegant. Verne's mass budget is internally consistent — in fresh water, where one cubic metre conveniently weighs a tonne. In seawater at 1,025 kg/m³ the same 1,500 m³ hull displaces 1,537,705 kg, and the boat with main ballast full reaches only 1,507,200 kg. It is about 30 tonnes positively buoyant and cannot submerge on main ballast alone. The arithmetic is fine. The ocean is the wrong one.

F5 is the subtlest, and it cuts both ways. Verne explicitly states the Nautilus has no CO₂ scrubbing — absorbing it would need caustic-potash canisters he says aren't carried — so air is refreshed only by surfacing. Where he lets that constraint bite, he is right: the roughly 48-hour crisis under the Antarctic ice sits squarely inside the plausible band. The failure is everywhere else, where the narrative implies indefinite submersion that the stated design forbids.

And the single most operationally important number in the book — how long the boat can stay under — ranges across 6 to 211 hours depending on parameters Verne never gives. That 35× spread is the finding.

The genetic algorithm cheats, correctly

With the physics quiescent, the question becomes: can a physically honest Nautilus be designed to reach Verne's? A GA got 29 design genes and a push-to-limit mission that dives until hull stress hits its safe limit and cruises until the battery is flat.

The best evolved champion reaches 562 metres. Verne claims 16,000. The gap is about 28×, after pushing every parameter to its most favourable plausible value — the strongest steel, the thickest plating, the biggest battery, the sleekest hull.

The interesting part is what the optimiser did first. Its opening move to extend endurance was to add CO₂ scrubbing at 6.5 L/s — the exact system Verne states the Nautilus lacks, and the omission behind F5's surface tether. With no knowledge of the novel and no term in its objective mentioning air, the GA independently identified and fixed the book's single most consequential design gap. Range roughly doubled once the CO₂ clock came off.

What the fuzzer found

Nothing, which took some doing to trust.

A 24-hour random-command soak ran 42 lives and 879,643 ticks, killing the boat 29 times by crushing it, and produced zero invariant violations. Five consecutive fuzz campaigns, sixty more lives, also zero. A harness that finds nothing is indistinguishable from a harness that is broken, so it was validated against a deliberately poisoned invariant — which it caught, delta-debugged from seven commands down to two, and packaged as a reproducible exploit.

The only finding in the whole loop is filed under exploits/001-objective-pinning, and it is a flaw in the experiment, not the physics: the GA had been optimising survival rather than performance, because the original controller pinned depth and distance to fixed targets. No core patch. A design-of-experiment fix, and the Pareto front went from a collapsed cluster to a real trade-off surface.

Then the frozen Ada core was re-implemented in TypeScript and both were replayed against a golden corpus, with any divergence defined in advance as a TypeScript bug. Event sequences matched exactly. The worst numeric delta was about 5×10⁻⁷ — and all of it traces to the Ada core rounding its output to six decimals at the wire. The two integrators actually agree to about 10⁻⁷, tighter than the reference bothers to report. The only genuine computational difference is the actuator-disk cube root, where Ada's X ** (1.0/3.0) and JavaScript's Math.pow disagree in the last bits and surface as 10⁻¹⁰ watts.

No oracle means you build one

HAL/S could check itself against an independent interpreter written by someone else. There is no independent implementation of Jules Verne. The oracle here had to be constructed, which means the credibility of every number above rests entirely on the scaffolding — the two-pass claim extraction, the traceability gate, the frozen protocol, the human-signed invariants, the determinism enforced before any physics existed, the harness validated against a poisoned check.

An AI will produce a confident simulation of a submarine with far less than that. It will also produce confident numbers that mean nothing, and from the outside the two look identical.

Next: no spec and no oracle again, but this time the thing being rebuilt is a product people remember using, the substrate underneath it has been replaced, and every defect that mattered turned out to be invisible to the tests.

Code: https://github.com/singular-state/nautilus-sim. Every figure regenerates from the committed repo via scripts/check.sh; each finding re-derives from the hash-pinned canonical spec, and each GA champion replays from its (genome, spec_hash, seed, scenario) triple. Pinned GNAT, fixed timestep, seeded PCG — byte-identical replay.

DEV Community: Jean-Luc Martel

PHP: 30 Years of Making Developers Say "Wait, What?" (Part 2: It Gets Worse)

4. Filtering an Array Can Turn It Into a JSON Object

5. A Variable Can Exist and Still Not Be Set

6. Two Different Strings Can Be Equal Because They're Both Zero

7. switch and match Disagree About Reality

PHP Is Consistent. That's Not Always Comforting.

PHP: 30 Years of Making Developers Say "Wait, What?"

1. The String "0" Is False. The String "00" Is True.

2. PHP Arrays Can Silently Overwrite Your Data

3. A foreach Loop Can Quietly Corrupt Your Array

So, Is PHP Broken?

Migrating AKS workloads from AAD Pod Identity to Entra Workload ID

Inventory the plumbing

Validate the exact strings

Stage the migration

Prove it works

The Green Padlock That Almost Cost Me a Contract

The setup

The problem I wanted to show

The demonstration

The reaction

Why the reaction matters more than the bug

What should have happened

The takeaway

BigQuery BI Engine deserves a benchmark before you dismiss Looker Studio as slow

Azure subscription Owner is an access-delegation privilege

The data engineering reality of GNNs in production

A zero-disruption PodDisruptionBudget can block an AKS upgrade

I rebuilt four dead systems with AI. The failures all landed in the same place.

What was actually being varied

Determined beats hard

A strong oracle is not the protection you think

What the human actually contributed

Build the oracle before the artifact

Three results that went the wrong way

Where this doesn't reach

What I'd tell someone starting

The Decoder Scored 100%. The Encoder Scored 8.3%.

Choosing something the model hasn't memorised

The controls

The decoder: 100%, and it doesn't count for much

The encoder: 8.3%, and it counts for everything

The hypothesis was wrong, in a useful direction

The confabulation catalogue

The second hypothesis was also wrong, and that's the finding

What it cost to know any of this

Rebuilding Encarta showed me exactly where AI-written code breaks

The setup

The modern substrate is never the shape its docs claim

The decision that paid for everything

Testability turned into architecture, which turned out to be good

Where the defects actually were

Checking a model with code beats checking it with a model

Modernisation re-opens every surface the original had sealed

The last mile is where the sandbox stops being able to prove anything

The constraint was never throughput

AI proved that the Nautilus crushes at 340 metres. Verne dives it to 16,000.

Extracting a spec from fiction

The core is a pure function

Five findings

The genetic algorithm cheats, correctly

What the fuzzer found

No oracle means you build one

7. `switch` and `match` Disagree About Reality

3. A `foreach` Loop Can Quietly Corrupt Your Array