The discriminator between a membrane and a logger
The first version of this test passed for the wrong reason, and it took me a while to notice.
I'd built a small plugin that puts gx, a reversibility layer for agent effects, in front of OpenClaw's before_tool_call hook. The plan was straightforward: intercept a file write, escrow an inverse for it through gx, let the write proceed, and prove afterward that gx could undo it. Four scenarios, twenty-one assertions, everything green. Then I looked at what the passing assertion in scenario A actually checked, and it was this: the file, after the write, contains the content I wanted.
That's true if gx did anything at all. It's also true if gx did nothing and the native write tool just wrote the file itself, the way it would with no plugin installed. A test that passes identically whether or not the thing under test ran isn't testing the thing under test. It's testing that the file system works.
Why the fix isn't obvious
The honest fix looks easy until you run into gx's actual design, which has no "escrow without applying" mode. A commit is one atomic step: escrow the inverse, reconfirm the precondition still holds, apply the change, issue the receipt. There's no half-state where gx has committed to a write but the bytes aren't on disk yet. Practically, that means once the hook admits a write, gx has already put the final bytes in place, before OpenClaw's own write tool ever runs its own code.
So OpenClaw's native tool, unaware of any of this, goes ahead and writes the same content a second time. Two writers, same bytes, same file, one right after the other. If the only assertion you have is "does the file have the right content", you genuinely can't tell which of the two writers made the decision.
The line that actually separates them
The fix is to make the second writer, the native one, tell on itself. Right before OpenClaw's write tool commits its own bytes, it can look at what's already sitting in the file. If gx got there first, that snapshot already matches what the model asked for, before the native tool has written a single byte of its own turn.
[OK ] gx had already applied it -- the native tool found the change done:
native tool saw bd53b1ff53596175... (desired=bd53b1ff53596175...)
That assertion only holds in the world where gx wrote it first. If the plugin were a passive logger, one that just watched the write happen and recorded it afterward, the native tool would still be the one laying down the bytes, and the pre-write snapshot would show the old content, not the new one. This single line is the entire discriminator. Everything else in the demo, the block scenario, the undo, the receipts, was already correct; this was the one assertion that had been quietly proving nothing.
Keeping the third value honest
gx returns one of four verdicts for a proposed effect: Admit, Deny, Escalate, or Unknown, where Unknown means the membrane couldn't be reached at all, not that it looked and said no. Collapsing Unknown into Deny is tempting, it turns the plugin's logic into a plain boolean, and it's exactly the mistake a reversibility tool can't make about its own instrumentation without undercutting its own premise: the whole point is telling "checked and refused" apart from "never checked."
That discipline paid for itself almost immediately, by accident. Early on I had the CLI invocation wired backward, the environment variable assignment and the binary path swapped, so gx read env as a subcommand it didn't recognize and failed outright. Because the plugin kept that failure as Unknown instead of quietly reporting it as a policy Deny, the assertion that checks for a named denial reason caught it as a wiring bug, not as "the policy correctly rejected this write." A collapsed verdict would have let a broken invocation pass for a working denial. It didn't, because Unknown stayed Unknown.
None of this needed a line of OpenClaw's source read to build. It needed reading the output of my own test, which said "pass" while proving nothing, and refusing to trust it until it said something that could only be true once.
Repo: github.com/TraceFold/tracefold, Rust, Apache-2.0. Alpha: latest tag v0.1.2-alpha (2026-09-01), published on crates.io as tracefold 0.1.2. Still alpha, and the limits page is longer than the feature list on purpose.
Top comments (10)
One negative control I’d be curious to see: prefill the file with the requested bytes, then disable gx. Would the pre-write assertion still pass? Using fresh content for each run would rule out the file already being right before either writer did anything.
You are right, and I ran it rather than reasoning about it. Modelling just the assertion in question, seeded three ways:
So the pre-write assertion passes with gx doing nothing, provided the file already held the desired bytes. That is the same defect the post is about, sitting one level up inside the control I wrote to fix it. I did not have that third row before you asked for it.
The fix is the one you named: fresh unique content per run, plus an explicit precondition that the file does not already match the desired hash at test start, so a broken run produces the second row instead of the third. A control that cannot distinguish "gx wrote it" from "it was already right" is measuring the file system again.
To be exact about scope: that is the assertion's logic seeded three ways, not a re-run of the full plugin harness, which needs the OpenClaw environment.
Thanks for running that third case. The prefilled file passing with gx off is exactly the ambiguity I was worried about. Fresh content plus the start-state check looks like the right next test. I also appreciate you keeping this separate from the full OpenClaw harness run.
Built it. Four rows this time, because the fourth one turned out to matter.
Fresh unique bytes per run, and the precondition is a read before the write: if the file already holds the desired content, the run returns UNTESTABLE instead of a verdict.
Two things I did not expect until it ran.
The first two rows are unchanged. I had half assumed a start-state check would cost something on the ordinary path, and it does not — the working control still passes and the broken one still fails. So this is not a trade.
The fourth row is the one I would have skipped. PREFILLED with gx on also goes UNTESTABLE, and that is correct rather than a regression. gx really did write those bytes in that run, but the run cannot prove it did, because the file was already right before it started. Crediting gx there would be the same error as the third row wearing a friendlier face. A control that cannot attribute the write should refuse the run in both directions, not just the one where the answer is embarrassing.
The control line exists because "everything is UNTESTABLE now" would also produce a table with no false passes in it. It confirms the guarded form still reaches both verdicts, so the third value has not eaten the other two.
That closes the thing you found. The original post argued that a membrane has to be able to say no; the control I wrote to demonstrate that could not say "I cannot tell", which is the same gap one level up. Thanks for pushing on the negative control rather than the result.
The fourth row is the useful surprise here. Even with gx on, the prefilled state can't show that this run caused the result. Keeping both a real pass and a real fail reachable also makes the UNTESTABLE result much more convincing. Thanks for running all four cases.
Your line about both verdicts staying reachable turned out to be load-bearing in a way I only saw this week, on a different piece.
A reader took apart the positive control in another post of mine — the one where a scanner read an empty directory and reported zero. My fix planted a file so the instrument could prove it counts. He pointed out that all three pointings arrived through the test's own parameter, so the default path constant, the thing that actually caused the zero, was still compared with nothing. I built it:
Break the constant and the planted positive stays green.
That is the same failure as a guard stuck on UNTESTABLE, approached from the other side. One control can only ever pass, the other can only ever abstain, and both look healthy on the row people actually print. What separates a working control from either is exactly what you named: every verdict it can emit has to be reachable by some input, and you have to show that rather than assert it.
So the control line under our four rows was not decoration. Without it the table is equally consistent with a start-state check that abstains unconditionally. With it the UNTESTABLE rows mean "this run cannot attribute the write" instead of "this check has stopped working", and those two are not distinguishable from the verdict alone.
a control the test aims cannot check the aim is the general form I took away. Applies to the fourth row too: gx really did write those bytes, and the run still cannot say so, because the fixture supplied the answer before the writer ran.
The broken-constant row makes the gap really clear: the planted file proves the scanner can count, but the test supplies the path and skips the default that broke. That distinction makes sense now. Thanks for working through the example.
The "can only emit one verdict" failure caught me in the wild this week, and it is worth adding because it arrived as a negative result rather than a green one.
I was checking a claim that four locale files in a date library return a bare number where the others return an ordinal. I ran the published package across six locales and got this:
All identical. I was one paragraph from writing "the defect does not reproduce, the report is stale". Then I added the row your principle demands, an input that has to produce the other verdict:
A locale that does not exist behaved the same as every locale that does. The setter had never taken effect at all: the locale files register against whichever copy of the library they resolve, and my harness had loaded a second one. Every row was the default locale answering six times.
So the check could only ever emit "same". Not because it abstained, but because it had one input in disguise. Once the instances were unified, four of the six rows moved and the claim held.
What I took from it is that your reachability test has a second use I had not seen. I had been treating "every verdict must be reachable" as protection against a green that means nothing. It is equally protection against a negative result that means nothing, and that direction is more dangerous for me, because a finding that fails to reproduce feels like diligence. I would have closed a correct report as unreproducible and felt careful doing it.
The pattern that separates the two is the one you landed on in the fourth row: the input has to travel the same path as the thing under test. A locale name I supply and a locale name the library resolves are the same string reaching different places, which is the same shape as a path supplied by the test versus the default constant that actually broke.
That invalid-locale row is a good catch. Six different names were still exercising the same default, so “not reproduced” wasn’t evidence against the report. I’d be interested in keeping the resolved locale in the test output too—it would make this failure visible without having to infer it from the formatted string.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.