I thought I had a fairly clear experiment.
An agent platform stages an external side effect for approval.
Something upstream decides that the action is allowed.
The platform records what happened.
So the question seemed straightforward:
Can a third party later prove, from the retained records alone, which decision authorized which action?
I started building a falsification study around that question.
Five rounds of cross-vendor adversarial review later, the study still has no registered result.
Every round told me not to freeze it.
That sounds like a failed experiment.
I think the opposite happened.
The reviews kept finding that some of the things I thought I was measuring either did not exist in the upstream system, could not be proven from the records I had chosen, or were weaker than the claims I had written around them.
The study became more useful before it ever ran.
That distinction became Study 015 - Cloudflare OS Boundary.
This post is about what the review process exposed.
The setup
The platform side is Cloudflare OS, pinned in the study at commit b2a51b5.
Cloudflare describes the August 2026 release as early access with rough edges. Everything here refers to that specific pinned source tree, not to some permanent definition of what Cloudflare OS is or will become.
Its governance architecture is interesting.
Agents and generated applications do not simply receive credentials to external services. A Gatekeeper mediates access. External side effects can be staged for approval rather than executed immediately.
Auto-approval requires two independent signals:
- the connector classifies the action as eligible
- the workspace user has enabled a rule allowing that action kind
The decision side of my study is different.
A portable judgment artifact receives facts and produces a structured disposition:
outcome
or
not-applicable
or
unresolved:
- unknown
- conflict
- missing evidence
The important property is that the judgment layer has a first-class way to say:
I cannot decide.
The study tries to join those two worlds:
facts + policy
|
v
judgment
|
v
proposed action
|
v
platform governance
|
v
approval / rejection / execution
|
v
retained records
And then asks:
What can someone prove afterward?
That last question turned out to be much harder than I expected.
First lesson: read the implementation contract, not the architecture you imagine
Before measuring anything, I had to establish what the pinned platform actually records and guarantees.
Several assumptions I had made were simply wrong.
The generic MCP Gatekeeper cannot become vetted through normal configuration
The generic MCP Gatekeeper hardcodes its trust tier:
const TRUST: ServerTrust = "byo";
Auto-approval requires vetted.
So the original scenario I had designed around a "vetted generic MCP endpoint" described a configuration that does not exist in the pinned implementation.
The MCP Portal path has a different trust mechanism.
My study had quietly invented an upstream capability.
Review round 2 caught it.
That is a useful kind of failure.
Before asking whether a system preserves some property, first verify that the state you are testing is actually reachable.
A read-only classification can bypass the action queue
The tool classifier honors an upstream readOnlyHint.
That means a tool classified as read-only can execute without going through the side-effect approval path.
Cloudflare's own source comments acknowledge the tradeoff.
This is not the same as saying:
The approval system is bypassable.
The condition is narrower:
If an upstream server labels a mutating operation as read-only, the platform may treat it as a read rather than as a staged action.
That changed the claim I was allowed to make.
"All external effects go through the approval queue" was too broad.
The source did not support it.
The outer action log cannot represent every execution outcome
The workspace-facing action state has three values:
export type ActionState =
| "pending"
| "approved"
| "rejected";
But the MCP connector's internal store has additional states, including failure.
That creates an interesting observability gap.
If a staged call is dispatched but the call fails before the outer layer receives a normal completion, the private connector record can know the call failed while the outer record remains:
pending
The outer log does not have a state capable of expressing what actually happened.
That matters if you are trying to reconstruct history later.
A missing state is not merely a UI inconvenience.
It limits what can be proven from that record.
approved is not proof that the external effect happened
The pinned MCP implementation describes its guarantee as:
at most once, not exactly once
That distinction matters.
An approved record tells you something about the platform's control flow.
It does not give you an independently verifiable receipt from the external system saying:
Yes, this effect happened.
There is no universal external transaction identifier or effect receipt attached to the action record.
So I had to separate:
platform approved execution
from:
external world changed
Those are not the same statement.
appliedAt is not necessarily "when the action was applied"
Another assumption cost me an actual study bug.
The field name suggested one interpretation.
The implementation had another.
appliedAt is populated when an action is approved and when it is rejected.
So it is closer to:
resolvedAt
than:
effectOccurredAt
A verifier reconstructing a timeline from the name alone could silently produce the wrong history.
Field names are not protocol semantics.
The useful join key is intentionally not public
The Gatekeeper has an opaque internal action identifier.
The client-facing action log does not expose it.
The source explicitly says that identifier should not be provided to the client.
That is a reasonable encapsulation decision.
But it has a consequence for my study.
A holder of the published action log cannot independently join that log to the connector's private execution records using the actual internal identity.
Retention makes the problem stronger.
The outer log is durable.
The private store retains only a bounded number of settled records.
Eventually, historical records can exist on one side after their corresponding private records have disappeared on the other.
Again, this is not necessarily a platform defect.
It means:
The platform's runtime records were not designed as a permanent offline evidence bundle.
My study initially treated them as though they were.
Approval rules do not create historical policy evidence
Auto-approval rules can be removed.
The retained records do not necessarily preserve a tombstone proving:
This rule existed at time T and authorized this action then.
So an action can have been validly auto-approved under the platform's state at execution time without leaving enough retained evidence to prove that fact later.
This connects directly to a problem I had already encountered in another study:
current state and historical authority are different things.
This does not mean Cloudflare OS is doing the wrong job
This distinction is important.
The pinned platform is primarily governing execution in the moment:
- capability isolation
- connector mediation
- human approval
- auto-approval constraints
- information-flow controls
My experiment was asking it to support something different:
Produce enough durable evidence that an independent third party can later reconstruct why a particular action was permitted and prove the causal relationship offline.
Those are different responsibilities.
The mistake would be to infer the second from the existence of the first.
And Cloudflare OS is an early open-source project.
Its architecture and retained evidence model can evolve.
This study is about the contract exposed by the pinned version, not a prediction about where the project will end up.
Then the reviewers started attacking the study itself
The upstream findings were only half of the story.
I also ran cross-vendor adversarial review against the study.
The reviewer was a different vendor's model, running read-only with the study tree and pinned upstream available.
The instruction was essentially:
Find every way this experiment could produce a result stronger than what the machinery actually proves.
Five rounds.
Every round said:
DO NOT FREEZE.
The sequence is probably the most useful artifact the study has produced.
Round 1 - seven blockers
The first review found something embarrassingly fundamental.
My bridge wrote a commitment describing the action it believed it was binding.
Later, the verifier checked downstream records against that same commitment.
So if the bridge committed the wrong action in the first place, every downstream check could still agree.
The system would report:
everything matches
But all it had shown was:
everything matches the bridge's own mistake
That is self-consistency.
Not independent verification.
The distinction sounds obvious after someone points it out.
It was not obvious enough when I built the first version.
Round 2 - five blockers, including fixes that were themselves wrong
Round 2 did something even more useful.
It rejected several of my Round 1 repairs.
One was the impossible vetted generic-MCP scenario.
Another involved what I had called "evidence backing."
The study checked whether an evidence digest looked like a valid digest.
But there was no retained evidence artifact whose bytes could actually be hashed and compared.
I had built:
digest-shaped string exists
and described it as:
evidence is cryptographically backed
Those are not remotely the same claim.
Round 3 - the historical witness problem
I had built a replay mechanism intended to establish that an approval had been lawful at the time it occurred.
The witness proving historical state came from the very store being examined.
That creates an obvious laundering path:
store claims historical state X
|
v
verifier asks same store for witness
|
v
store supplies evidence for X
The final check may be perfectly deterministic.
The independence claim is still false.
So the study claim got weaker.
Instead of:
historical lawfulness proved
it became:
consistency with a self-asserted historical witness
Much less impressive.
Much more accurate.
Round 4 - correct counts can hide the wrong cause
This was the finding I would defend hardest.
Imagine the records show:
1 approved action
1 external effect
The counts match.
But the effect was actually produced by a different invocation with the same arguments.
The original call never caused it.
A cardinality check says:
1 == 1
PASS
The causal claim is false.
The reviewer called this fatal.
It was right.
So instead of documenting the limitation, I changed the artifact.
Effect attestations now need to carry the identity of the staged action that produced them.
The check became an identity join rather than a count.
That is a broader lesson:
When causation matters, count matching is not identity matching.
Round 5 - one predicate left
By Round 5, the reviewer found one blocker.
One governed call had no commitment digest.
The study checked:
number of governed calls == number of committed calls
and the counts matched.
But the unbound call sat outside the set the later verification actually examined.
Once again:
count equality
was pretending to establish:
set identity
Round 1 asked:
Does this experiment verify anything independent?
Round 5 asked:
Should this exact predicate compare identities instead of counts?
That narrowing is the best evidence I have that the study is converging.
Not that the reviewer started agreeing with me.
That the disagreements became smaller and more mechanical.
Four things I personally wrote down that were false
The reviewers were not the only source of correction.
My own source-verification passes refuted several statements I had already committed.
I think these are worth publishing because they show how easy it is for technically plausible prose to outrun the implementation.
1. "This component cannot be typechecked from the committed tree."
False.
It typechecks after one committed code-generation step.
My own verification refuted the claim before a reviewer reached it.
2. "The adapter reproduces upstream behavior."
Not really.
Six documents cited a check that supposedly compared my adapter against upstream.
The implementation effectively did:
f(x) == f(x)
An automated edit that was supposed to insert the real comparison had silently failed because the anchor string had been renamed.
The commit message claimed the fix had landed.
The file disagreed.
3. "The tool name never appears in the log."
False.
It appears in prose and structurally inside the action-kind tag.
A grep disproved the claim.
4. "There is no action-log export anywhere."
Also false.
There is a listActions() API.
The accurate claim was narrower:
There is no signed, complete, offline-verifiable action-history export supporting the evidence properties this study requires.
The weaker sentence is the true one.
What I would carry into any similar experiment
A few practices survived all five review rounds.
Verify the reviewer
Do not immediately accept an adversarial review because it sounds sophisticated.
For every load-bearing finding, I went back to the source and reproduced the claim.
Most reviewer findings held.
A few needed narrower wording.
Sometimes the verification produced a better repair than the reviewer proposed.
Adversarial review still needs verification.
Fix the missing mechanism when you own the artifact
Some findings really are limitations.
Others identify missing machinery you can actually add.
The substituted-causation problem was not something I wanted to solve with:
This study does not claim causality.
Causality was the point of that artifact.
So the artifact gained the missing identity.
A disclaimer should not substitute for a mechanism you can reasonably build.
Narrow claims aggressively
Several claims became weaker during review:
proved historical lawfulness
became:
consistent with a self-asserted witness
and:
evidence-backed
became:
retained-preimage consistency
That can feel like losing ground.
It is actually progress.
A narrow falsifiable claim is more useful than a broad impressive sentence whose evidence does not support it.
Delete checks that cannot fire
A verdict branch with no reachable input is not defense in depth.
It is dead prose written as code.
The same goes for schema fields with only one possible reachable value.
If a mechanism cannot distinguish two states, do not describe it as though it does.
Verify scripted edits after they run
This one is painfully mundane.
An automated source edit silently failed.
The script exited.
The commit succeeded.
The message described the intended change.
The implementation never changed.
Now every scripted edit in this study gets verified against the resulting file.
"Automation ran" is not evidence that the intended change happened.
A study can be honest and still not be ready
Five consecutive DO-NOT-FREEZE verdicts are not something I need to hide.
They are the current result.
Study 015 - Cloudflare OS Boundary remains merged as a draft.
The study index says that explicitly.
There is no registered result.
Nothing in the current study should be cited as though the experiment has already established the decision-to-action binding it is trying to test.
That is the correct state.
The boundary that remains open
Even if every current blocker is resolved, one problem remains outside both systems.
Staleness and rollback.
Suppose a decision was correctly produced under policy version 3.
Later, version 4 becomes authoritative.
Someone presents an old-but-valid decision under version 3.
Or they coherently reconstruct a history around version 3.
Every digest may still match.
Every signature may still verify.
The chain can be perfectly self-consistent.
What it cannot say about itself is:
Was this policy still authoritative at the relevant point in time?
That requires an anchor outside the chain.
Something needs to establish the history of authoritative policy state:
policy identity
|
v
revision
|
v
artifact digest
|
v
effective time
|
v
next revision
This is similar to a boundary that surfaced in our earlier Study 014 - OpenWorkProof binding experiment.
A self-consistent proof can establish lineage inside its trust boundary.
It cannot establish that the state it contains is still the externally authoritative state.
That is not a Cloudflare OS problem.
It is not an OpenWorkProof problem.
It is not something a Judgment Pack should silently decide for itself.
It is a separate trust problem.
And it is probably the next one worth defining carefully.
The current study is public here:
Study 015 - Cloudflare OS Boundary
The repository includes:
- the draft study protocol
- the five verbatim cross-vendor review rounds
- the disposition of every finding
- the withdrawn and narrowed claims
- the deviations record
- the pinned upstream identity
It is evidence of a review process and of several architectural boundaries.
It is not a frozen experimental result.
And at this point, that distinction is the entire point.
Top comments (1)
Adversarial review is useful because it attacks the measurement, not just the conclusion. With AI systems especially, it is easy to measure a proxy that feels rigorous while the actual operational question remains untouched.