There is a difference between installing a system and proving that it is operating.
We learned that the hard way.
The project started with an attempt to equip Codex with a large Claude-derived harness: hooks, memory systems, ledgers, safety gates, self-tests, and supporting organs. The first question was simple:
Can the harness actually run?
The first answer was encouraging but incomplete.
A read-only copy was preserved. A separate writable runtime was created for state and ledgers. The configured hook registry replayed successfully. The organ battery produced 50 passes and 2 failures. A behavior probe produced 8 passes and 2 failures.
That was useful evidence—but it was not proof that Codex had become a native host for the Claude lifecycle.
The harness could run as an external exoskeleton. That was different from proving that Codex itself executed the same lifecycle natively.
There were also concrete gaps.
The action gate blocked a recursive Unix-style delete but missed the equivalent PowerShell shape. Another conscience check missed two strong-claim cases. A memory test contained a stale fixture expectation. A security test passed its assertions but encountered a Windows teardown problem afterward.
So I asked for another test.
The fresh proof pass caught a problem in my own test: Sentinel had moved from RED to YELLOW, but my probe still expected it to remain RED. That meant the probe was testing a frozen assumption instead of testing the live state machine.
The correction was simple but important:
A valid test should verify that the system reports its current state, not demand that the state remain unchanged.
Then came the more important request: build a Codex-native version.
The Claude clone stayed preserved as the specimen. The Codex adaptation lived in its own workspace. It used the same principles, but with Codex- and PowerShell-aware gates, explicit preflight and final-check entry points, and tests aimed at the holes the first pass had exposed.
The first run scored 9 out of 10.
The failure was not a safety bypass. The final-check expression was too rigid. It accepted a verification block only when it appeared in one exact format. The repair broadened the accepted evidence shape without weakening the requirement that evidence be present.
The second run reached 10 out of 10.
The adapted layer now blocked recursive deletion of the protected harness, remote fetch-and-execute patterns, force-pushing a Git branch, and strong completion claims without an explicit verification or receipt block.
That result was meaningful, but it still did not mean the entire Claude organism had been reproduced. The original transplant still had two meaningful behavior failures. The Codex-native layer was a smaller, explicit enforcement surface for Codex’s own runtime.
That distinction matters.
A passing test can still be misleading if the test is stale. A configured hook can still be inactive. A visible memory file can still be unobserved. A wrapper can still claim failure after the underlying process succeeded.
The standard became:
- configured is not loaded
- loaded is not trusted
- trusted is not executed
- executed is not automatically correct
- correct is not complete until the real receipt is inspected
The second-pass report was therefore more honest than the first one. The initial verification was mostly right, but underpowered. It proved that the transplanted harness could run and exposed real gaps. The new self-harness proved that Codex had its own tested enforcement layer.
That is the lesson I would carry into every AI system:
A feature is not real because its files exist. It is real when the runtime demonstrates the behavior, the negative case behaves correctly, and the final claim is backed by evidence.
The goal was never to make the system sound more alive. The goal was to make it more accountable.
And accountability begins with a simple rule:
Don’t tell me it is installed. Show me that it ran.
Top comments (15)
Thank you for taking the time to write this. Your distinction between evidence shape and semantic referent was exactly the right pressure test.
We took your triplet literally and hardened the gate around it: evidence against the current real object passes; the same shape against a stale, wrong, missing, or path-substituted object fails; and no evidence fails. The verifier now resolves the object independently and freshly rehashes its current bytes, so a self-report in the accepted format cannot close the gate by itself.
I genuinely appreciate the specificity of your feedback. You didn't just point at a risk—you gave us a reproducible negative test, and it made the system better. Thank you.
This is exactly how I hoped that comment would land — thanks for taking the triplet into the codebase rather than just agreeing with it.
One thing I would keep an eye on, now that "independently" is the load-bearing word: independence has to live in the resolution path, not just in the hash. If the verifier finds "the object" through a channel the verified thing also controls — an index it wrote, a path it named, a status it maintains — a fresh hash of what it found is still a receipt for a subject the gate did not choose. The strongest form is resolution through a channel the verified thing cannot write, or at minimum a different channel than the one that produced the self-report; an audit trail held by the thing being audited proves nothing about the audit. And Vinh's frozen-reporter point is the same lesson one level up: once the gate is honest, the probe that tests it has to drive a transition, because a stuck value and a correct value are indistinguishable at rest.
You’re right—the hash was independent, but the authority choosing the object still lived too close to the thing being audited. We split that path: the harness can produce a receipt, but a separate verifier-owned frozen scope now chooses the exact objects, rejects missing, extra, or substituted labels, and rehashes them itself. It immediately caught a stale receipt after a final README edit, which was better proof than any claim we could make. Thank you for pushing on the load-bearing word.
The README-edit catch is worth more than any positive claim — it's the transition case actually firing: the world moved, and the mechanism noticed without anyone re-reading the receipt. That's the "probe has to drive a transition" problem solved on the verification side, and you got it for free.
One seam I'd keep poking now that the scope is the authority: that catch happened at re-verify time. Between the edit and the next verification run, a consumer holding the old receipt still sees green. If the receipt carries the scope's fingerprint (or a scope epoch), staleness becomes a read-time failure — anyone can check "is this receipt's scope still current" without re-running the verifier at all.
And the same logic points one hop further: the frozen scope is itself a mutable object, so its update path deserves the triplet you just built. If the scope can be changed through a channel the harness also writes — a manifest in the same repo, say — the authority question just moved up one level instead of going away. Version the scope, make scope changes as observable as object changes, and the boundary holds at every hop.
That extends the lesson one layer further. We hardened it in two layers: every receipt now carries a versioned scope epoch and canonical scope fingerprint, and the verifier independently recomputes both. We also added a read-time check, so an old green receipt fails immediately if the scope, verifier, or scope bytes change—without waiting for another full run. Legacy unbound receipts fail closed too. Thanks for pushing the authority question all the way up the chain
That is the resolution-path independence idea applied to the scope itself -- the verifier no longer trusts the receipt's authority claim, it re-derives it (epoch + fingerprint) from the scope. Clean.
Two seams from where I am sitting:
The read-time check fails fast, but it fails on two different failures that need different responses. Epoch mismatch means a planned transition: re-run under v2, the old receipt was fine for its epoch. Fingerprint mismatch without an epoch bump means the scope changed without being versioned -- a discipline break, and now it is detectable, which is the "the frozen scope is itself mutable" point from my earlier comment turned into a checkable invariant. If the failure surface reports just "stale", operators cannot tell an upgrade from a scope-discipline break. Fail with a reason: stale-with-reason.
The transition window is where the model can still lie by omission. Between "v2 ships" and "the re-run lands", every in-flight v1 receipt is instantly invalid -- correct, but the consumer needs to read the verdict as epoch-bound, not just stale: "valid under v1 at observation T, invalid since v2". That turns a bare red into a re-run request with the reason attached, which is the difference between "re-run me" and "trust me, it failed".
Neither changes what you built -- both are about making the read-time check speak the operator's language.
Solid advice that I am going to test and implement when I get home. And this is precisely what I've been looking for for months real feedback, external feedback thank you so much!!
Looking forward to hearing how the two failure classes hold up in practice -- the epoch-vs-fingerprint split only shows its teeth in a real transition. Glad the write-up was useful.
Yes I will update you on that tonight when I get home
You were right that the split only shows its value when a transition is actually driven, so we ran the transition matrix.
That transition matrix is exactly what the epoch-vs-fingerprint split was designed to enable — the distinction only earns its keep when a real transition fires, and you drove all six rows. The row I would most want pinned as a permanent test is epoch_superseded_history_unverified: removing the historical archive flipping a "historically valid" receipt into fail-closed is the provenance-loss case most systems quietly treat as acceptable.
My suggestion: encode the matrix as the executable contract. Six rows, six tests — matching epoch returns valid/current; label change without bump returns scope_drift_without_epoch_bump; scope-only change returns scope_bytes_changed_without_epoch_bump; archived replay returns epoch_superseded plus rerun_under_current_epoch; missing archive returns history_unverified; legacy and future epochs fail closed. The hostile suite (self-selected subjects, receipt-controlled labels, frozen reporters) deserves the same home as regression tests.
One addition to consider: assert the outcome codes are mutually exclusive — every transition produces exactly one verdict. That is the difference between a checked gate and a gate that can emit two answers at once.
Thank you, I will work on this when I return home tonight.
The stale probe has a sharper form than RED versus YELLOW. A probe that asserts the system reports its current state also passes when the reporter itself is frozen, because a stuck value and a correct value have the same shape at the assertion; what separates them is driving a transition you control and asserting the report follows, so the test covers the state machine rather than one of its outputs. I hit the same shape in an API check last week: the endpoint I was querying does not carry the
publishedfield at all, so reading it returnedNoneand my is-not-published assertion passed for every article, public ones included. It only became discriminating when I switched to a request that returns404for the known negative.Thank you—this was the sharper test. We replaced the at-rest assertion with a controlled RED → GREEN → RED probe and now require both
STATUS.jsonand the scream marker to follow every transition; a frozen reporter fails. Your API example made the difference between checking an output and testing the state machine crystal clear.The 9/10 → 10/10 repair is the most instructive part of the writeup, because it names a fix but not the trap sitting right next to it: broadening the accepted evidence shape is exactly how a gate quietly stops gating.
The repair — "the final-check expression was too rigid; it accepted a verification block only in one exact format; the fix broadened the accepted evidence shape without weakening the requirement that evidence be present" — is the right instinct. But broadening's failure mode is not that the requirement weakens; it's that a shape-match stops implying a semantic match. A block in the new accepted format can reference nothing real: a verification section naming a check that never ran, a receipt pointing at state that was never touched. Your own first pass shows the same drift in gates: the action gate blocked the recursive Unix delete but missed the PowerShell shape — the gate wasn't missing, it was parsing a different representation of the same destructive action. Shape and semantics had drifted apart, and only a differently-shaped probe exposed it.
So the rule that falls out of your own chain: when you broaden the accepted evidence shape, re-run the negative cases under the new shape. After a broadening the fixtures should be a triplet:
That's "the negative case behaves correctly" applied to the evidence shape itself, not just the behavior. Your probe bug is the same lesson one level up: the probe expected RED when Sentinel had moved to YELLOW — the test was reading a frozen model of the system instead of the live state machine. The stale fixture wasn't in the memory test; it was the test itself.
And "correct is not complete until the real receipt is inspected" — the receipt has to be real in the same sense: produced by the thing it claims to verify, or by a channel outside it. A self-report in the accepted format is a receipt in shape only. The 10/10 is trustworthy because the re-probe existed, not because the shape was broadened.