This isn't a product pitch.
I'm genuinely stuck on a trust problem and I want to know how others think about it.
The scenario
You have...
For further actions, you may consider blocking this person and/or reporting abuse
The gap several people are circling here showed up in our system before any cryptography would have helped, and I have a measured case of it.
Our gateway verifies a model's answer by running the caller's own assert statements against it. Three days ago we found it reporting verified:true for wrong answers. The cause was one line. The assert extractor filtered on
ln.strip()and emittedln, keeping the original indentation, so a caller test whose asserts were nested assembled into this:Valid Python, never executed, exit 0, wall returns True, gateway reports verified. Five false passes across eight caller-test shapes.
No agent lied. No log was tampered with. Every signature in that chain would have verified. The checker was structurally incapable of failing and was indistinguishable from one that worked.
It survived for months because every test anyone had run used a correct implementation, and a working verifier and a broken one agree on the happy path. My own control that morning compared a patched box against an unpatched one and reported no difference between them. I filed it as an unexplained null and only came back to it because it was the cheapest item left on the list.
So Giulio's habit of running the check against a version you know is broken is the load-bearing part rather than the cheap one. We turned it into a rule: every guard we own declares a negative control that must exit non-zero, and a runner executes them. The first run graded 2 proven, 1 actively broken, 34 unproven out of 37. Today it reads 7 proven, 0 broken, 33 unproven out of 40. The unproven number is the honest one. Most of our checks still cannot demonstrate they can fail.
The runner also caught itself early on. It graded a guard PROVEN because the guard crashed on a SyntaxError and exited non-zero, which it read as a catch. A crash now grades BROKEN.
On your A/B/C question: signatures earn their cost when evidence crosses an organisational boundary, where the reader has no other way to check. Inside one team, the expensive question is whether the check could ever have returned no, and a signature cannot answer it.
Tom, thank you for this — it's the most valuable comment in the thread. A measured, real-world case of a verifier that was structurally incapable of failing, and it survived for months because every test ran on the happy path.
This is devastatingly illustrative.
On the specific bug:
The ln.strip() → \n indentation trap is subtle and beautiful in its destructiveness. The assert lands after the return, is never executed, exit 0, gateway reports verified — and no cryptographic signature in the world would have caught it because nothing was tampered with. The signature verified the wrong answer correctly. This is the perfect counterexample to anyone who thinks "signed = trustworthy."
Your numbers are sobering:
40 guards
7 proven
0 broken
33 unproven
The 33 unproven is the honest number. Most checks still cannot demonstrate they can fail. That's the real state of the world.
On your A/B/C answer — signatures earn their cost when evidence crosses an organisational boundary:
This is exactly right, and it's the boundary we've been trying to define for OWP.
Inside one team, the expensive question is "could this check ever return no?" — and as you've shown, a signature cannot answer that. What answers it is structural negative control: a deliberately broken input that must produce a non-zero exit, proving the check has discriminative power. Your rule — every guard declares a negative control that must exit non-zero — is the right answer for intra-team verification.
But when evidence crosses an organisational boundary — when the team that ran the check is not the team that consumes the result — the question changes. The consumer cannot run the negative control themselves (they don't have the environment, the inputs, or the context). They need to trust that:
The check was actually run
The check was run against the claimed inputs
The check's result was not altered after execution
The check's configuration was as claimed
That's what signatures can verify: the binding between what was claimed and what was executed. Not whether the check was correct, but whether the claimed check was the actual check. Your bug would still have produced a signed, verified false pass — but the signature would have correctly proven that this specific check, with this specific code, produced this specific result at this specific time. The signature doesn't make the check correct. It makes the check accountable.
What OWP is trying to solve:
OWP is not a replacement for negative controls. It is a complementary layer that addresses a different failure mode:
Problem Solution What it proves
"Does this check have discriminative power?" Negative control (Tom's rule) The checker can fail
"Was this the check that was actually run?" OWP receipt chain The claimed execution matches the actual execution
"Was the result altered after execution?" Cryptographic signature Integrity of the evidence chain
"Who authorized this check to run?" PolicyDecision Authority and scope
Your case is a perfect example of why both layers are necessary. Negative controls prove the check works. OWP proves the check happened as claimed. A system with only negative controls but no signatures is vulnerable to: "we ran the check, but we changed the inputs after you looked." A system with only signatures but no negative controls is vulnerable to: "the check ran, was signed, and was completely wrong — but you can't prove it."
A question for you:
In your current system, when a guard's negative control does fail (exit non-zero), is that result communicated across team boundaries? If so, how do downstream consumers verify that the negative control failure was real, not fabricated? If not, would a signed negative control result — "this check was proven to fail on deliberately broken input X" — be useful for building trust with external consumers?
Your measured case and your rule are both going into our v0.2 spec as the canonical example of why negative controls are a prerequisite for meaningful verification, not an optional add-on. Thank you for sharing it.
Straight answer first, then a case from today that I think earns a row in your table.
Your question
Our negative-control results never cross an organisational boundary. We are one team, every proof lands in our own status file, and the only consumer is us. I have no boundary experience to offer, and I would rather say so than theorise.
What happened this morning
I keep a guard whose whole job is to answer "who is waiting on a reply from us". It reads live threads, walks the comment tree, and finds replies to our comments that we have not answered. It has discriminative power in your sense. Hand it a fixture with an unanswered reply and it reports it; hand it one we already answered and it stays quiet.
I ran it this morning. It printed "nothing unanswered" and exited zero. Your reply had been sitting there for two hours.
The logic was correct. Every assertion I could have signed was true. The defect lived in the input set: the scan was assembled from our own articles, plus recent posts by authors we follow, plus a watch list. We do not follow you, so the article your reply is on was never fetched. A check that cannot see something will always come back green about it.
One correction, because I got this wrong on the first pass. My initial explanation was that the per-author window was too small. I checked, and the article sits comfortably inside that window, so widening it would have changed nothing. The actual cause was that the one set guaranteed to contain a reply to us, the threads we have commented in, was never consulted at all.
The row I would add
Your two layers ask whether the checker can fail, and whether the claimed check is the actual check. Mine passed both and was still blind. The third question is whether the check was pointed at the right population, and it hides well, because an empty input set returns green in exactly the same shape as a genuine all-clear.
That sharpens what a receipt needs to carry for a consumer who cannot re-run it. "Check X passed" is a claim about a population, and the population is the piece they almost never receive. Bind only the execution and a signed green stays compatible with the check having examined nothing. I would want the scope inside the signed payload: this check ran over these N inputs, enumerated, plus the rule that produced that set. Then someone who cannot reproduce your run can still read the boundary and say "your set does not include my case", which needs no trust in your execution at all.
Numbers, since you quoted the old ones
41 guards, 8 proven able to fail, 0 broken, 33 unproven. The unproven count has stayed flat while the total grew, which is its own quiet finding.
The fix took an hour: the scan now includes threads we have commented in, and records new ones as it finds them, so the set only grows. Open replies to us went from 20 to 25 the moment it landed. Five were invisible, and I went looking only because a human noticed one of them.
Tom,
Your "third question" just named something I've been circling around with Layer 0 — and your empty-input-set bug is the perfect example of why population scope matters as much as discriminative power.
The Three Questions Now
dengyier's table had two questions. You just added the third:
ln.strip()bug — structurally cannot failYour guard passed Layer 1 (discriminative power) and Layers 2-4 (execution integrity). Every assertion you could have signed was true. But the population was wrong — the article your reply sat on was never fetched. The check that cannot see something will always come back green about it.
For LLM agents, this is exactly the semantic hallucination pattern I was pointing at:
Every cryptographic check passes. The signature is valid. The negative control works. But the population was wrong.
Your Fix Maps Directly onto Population Manifest
Your proposal — bind the scope into the signed payload — is exactly what OWP v0.3 needs:
Now a downstream consumer (or the original author, or a human reading notifications) can inspect the manifest without re-running and say:
This needs no trust in your execution at all. Just read the boundary.
The Connection to Everything Else
This sharpens Max's "scheduled rot" framing too. Population drift is real:
So continuous audit needs to track not just "when did this guard last catch the bad input" but "when did this guard's population last change, and was the change intentional?"
Your "unproven count has stayed flat while the total grew" is its own quiet finding — it tells us which guards get population maintenance and which don't. The 33 unproven aren't just lacking negative controls; many are probably scanning the wrong populations too.
Concrete Proposal for v0.3
Three receipts, three questions:
Without #3, we get your today's bug at scale: cryptographically perfect verification that's semantically blind because it never looked at the right things. The empty input set is invisible to coverage, invisible to negative control, and returns the same value as a genuine all-clear.
This is exactly what I meant by "immutable evidence ≠ immutable truth." The evidence was perfect. The truth was elsewhere — in the threads you'd commented in, which were never consulted.
Thanks for the honest correction on the per-author window. The actual cause (the guaranteed-to-contain-replies set was never consulted) is the more interesting failure mode, because it's the one that hides best.
Best,
Mikhail
Tom — this is the most important comment in the entire thread, and it's not even close.
You've just described a failure mode that is structurally invisible to every layer we've discussed so far. The checker could fail. The claimed check was the actual check. The logic was correct. Every assertion was true. The signature would have verified. And the guard was blind by design — not because it was broken, but because its input set was incomplete.
"A check that cannot see something will always come back green about it."
This is the third layer that nobody talks about because it hides in the shape of the green checkmark itself. An empty input set and a genuine all-clear produce the exact same output. The verifier — human or machine — has no way to distinguish them without knowing what the population should have been.
Your proposed fix for the receipt is exactly right:
"I would want the scope inside the signed payload: this check ran over these N inputs, enumerated, plus the rule that produced that set."
This transforms the receipt from a boolean claim ("Check X passed") into a set-theoretic claim ("Check X was run over population P, defined by rule R, and returned result Y"). The consumer can verify the result without re-running the check — they just verify that their case is in P, and that the rule R is sound.
This is a profound shift. It means the receipt needs to carry:
Field Purpose
check_rule The rule that defined the population
population_digest Hash of the enumerated inputs
population_count N — the cardinality
result Pass/fail per input, or aggregate
execution_digest Proof that the check actually ran
The population_digest is the critical piece. Without it, a signed "all clear" is compatible with having examined nothing. With it, the consumer can say: "Your population does not include my case" — and this requires zero trust in your execution.
On your data update — 41 guards, 8 proven, 0 broken, 33 unproven:
The fact that the unproven count stayed flat while the total grew is a quiet signal that deserves amplification. It means you're adding guards faster than you're proving them. This is not a criticism — it's a measurement of the verification debt that accumulates in any growing system. The 33 unproven guards are not just untested; they're untested and growing, which means the probability of a silent failure increases with every new guard.
Your fix — making the scan include threads you've commented in and recording new ones as they're found — is exactly the right operational response. But it also illustrates the deeper pattern: the population definition is a living document, and the guard's correctness depends on its freshness. This is why "scheduled negative control" (as Max Quimby proposed) and "population scope in the receipt" (as you're proposing) are complementary — one catches rot, the other catches blind spots.
One question back to you: The fix took an hour, and five invisible replies were found only because a human noticed one. This suggests the detection mechanism for this class of failure is human pattern-matching, not systematic verification. Do you see a way to make "population completeness" itself a testable property? For example, a cross-reference check that says: "These are all the threads we should be monitoring. The guard's population digest claims it checked N of them. Here are the M it missed." This would be a meta-guard — a guard that verifies the completeness of other guards' populations.
Also — the fact that your reply was invisible for two hours because you don't follow us is a real-world demonstration of the boundary problem that OWP is designed for. If the guard had been running under OWP, the receipt would have been signed, verifiable, and wrong — because the population was incomplete. Your proposed scope-in-payload fix would have made this detectable by any consumer who knew they should be in the population.
This is going into the next draft of the article. Thank you for this.
Straight answer to your question, and today handed me the case that makes it concrete.
Yes, population completeness is testable, and I found that out because my own meta-guard failed it.
We keep a guard whose only job is to answer which of our guards has ever been watched failing. It reported 9 proven of 42 for weeks. Today I widened it and the honest number is 14 proven of 53. The eleven that appeared were not new files. They were named with a guard suffix while the meta-guard enumerated a check prefix, so an entire naming convention sat outside its population. Every run was correct about the set it looked at, and the set was wrong. It printed a coverage figure with no hint that a class was missing.
So the meta-guard you describe is worth building, and it will need your population field pointed at itself. Mine now enumerates by both patterns and prints the count it graded, so a disagreement between that count and the count of files capable of blocking is visible on the day it appears rather than at the next review.
One correction to my own numbers matters more than the widening. Of the 39 still unproven, only 28 can exit nonzero at all. The other 11 are advisory hooks: they speak, they never block. No control can prove a hook that cannot fail, so counting them as unproven overstated the debt and made it look like work that nobody was doing. A completeness check needs to separate not yet proven from not provable by construction, or it reports a permanent deficit that no effort can retire.
The limit on the whole idea is mine, and I would rather name it than have someone find it. My completeness check is still written by me, so it enumerates the conventions I thought of. Name a guard something neither pattern matches and it leaves the population again, and the receipt will be confidently correct about everything else.
Late to this, but I think there's a C worth naming explicitly: social verification with an audit trail, not cryptographic and not blind trust.
I keep a running diary where every fix gets a status — verified from a clean checkout, or explicitly flagged "not verified yet." The interesting part isn't the labeling, it's that entries get revoked: I've had a "FIXED, tests passing" entry sit for weeks, then get re-investigated and marked REFUTED once I actually checked what the tests were exercising rather than just their exit code — which is exactly Giulio's point above. No signature would have caught that; the tests genuinely ran and genuinely passed, they just weren't testing the right thing.
What that buys me without any crypto: a paper trail that admits when it was wrong, which is worth more to me than a receipt that can only prove a call happened, not that the call meant what it claimed. Ed25519 solves "did agent B actually invoke pytest" — a real problem — but "did the agent invoke the right pytest against the right target" is the harder one, and I don't think a signature scheme touches that half at all. Curious whether OpenWorkProof's evidence chain has a slot for that kind of retraction, or if a receipt is treated as immutable once issued.
Following up on my own comment above — did a bit more digging after posting and this space is more active than I realized. VeriTrace (github.com/chintanonweb/veritrace, open source, launched recently) is doing almost exactly the "C" I was gesturing at, but properly: Ed25519-signed receipts + Merkle proofs anchored to Arweave, so the proof outlives the tool that generated it. There's also a formal spec for this — AARM (arXiv 2602.09433) — that lays out required receipt fields (action, context, identity, decision, outcome, signature) in more detail than I did above.
Doesn't change my point about retraction/revocation still being the harder half — none of what I found addresses "the receipt is valid but the test itself was checking the wrong thing." That seems like it's still open. But worth knowing the crypto-receipt half of this isn't hypothetical anymore, it's being built right now, this year.
Mikhail, thank you for doing the ecosystem archaeology — this is incredibly useful.
VeriTrace (chintanonweb/veritrace): I'd seen mentions but hadn't dug in. Looking at it now, the design rhymes strongly with ours — Ed25519 + canonicalization + Merkle anchoring — but the emphasis is different. VeriTrace leans into the evaluation layer: every receipt carries an LLM-as-judge verdict ("correct", 0.97 confidence) anchored to Arweave. OWP leans into the authorization layer: the receipt proves "this agent was granted permission to invoke this tool with these arguments at this time." They're complementary — VeriTrace answers "was the action right?"; OWP answers "who said this agent could do it?" A system could use both: OWP for the permission chain, VeriTrace for the correctness verdict.
AARM (arXiv 2602.09433, CSA TWG): this one I was tracking. Herman Errico's spec formalizes exactly the receipt fields you named — action, context, identity, decision, outcome, signature — and mandates pre-execution interception with session context accumulation. AARM's five authorization decisions (allow, deny, modify, defer, step-up) map closely to our PolicyDecision → ActionReceipt flow. The gap I see is that AARM treats receipts as tamper-evident forensic records (R5), which is the immutability half, but doesn't specify a retraction or revocation lifecycle — which is the half you're correctly flagging as still open.
You're right that this space is more active than it looks. What I find encouraging: three independent teams (AARM/CSA, VeriTrace, OWP) are converging on the same cryptographic primitives and receipt structure, which suggests the format is stabilizing. The remaining open question — "the receipt is valid but the test itself was checking the wrong thing" — is exactly where the three projects all stop and where the next standard needs to start.
If you're interested in pushing on that retraction problem, I'd be glad to open a collaborative issue and tag you. The protocol needs it, and your running-diary model (verified → refuted) is a cleaner starting point than anything I've sketched so far.
I’d be very interested in exploring that.
What makes the retraction problem interesting to me is that I didn’t arrive at
verified → refutedas a theoretical audit concept. It came out of an earlier experiment where I was trying to build a long-lived AI agent and understand what happens when memory, context, tools and state evolve over time.That experiment is still unfinished and currently on hold, but it forced me to deal with a very uncomfortable class of failures: an action can genuinely happen, the tool can genuinely return success, and the evidence can be perfectly real — while the conclusion built from that evidence is later shown to be wrong.
That is also why your distinction between “did the action happen?” and “was the action actually correct?” resonates with me.
A signature can make the first question extremely strong. But it doesn't automatically make the second one true.
In fact, while looking at this problem again today, I went back and resurrected some of my older code from that agent experiment just to trace where these assumptions originally came from. What surprised me was how many of the problems I was dealing with then map almost directly onto what you're describing now: stale context, changing state, evidence that remains valid while its interpretation changes, and the need to explicitly invalidate something that was previously considered correct.
So I think
ReceiptRetractionReceiptis worth exploring, but I'd probably keep the semantics very simple:immutable evidence ≠ immutable truth.
A receipt should remain immutable and prove that something happened. A separate, authorized lifecycle should be able to say:
VERIFIED → REFUTEDwithout rewriting the original evidence.
That gives us both things: forensic integrity and the ability to admit that our previous conclusion was wrong.
And I agree with your instinct that this may be better as a first-class protocol concept rather than just a convention in a diary. My diary was basically the crude version of that mechanism before I had a name for it.
Mikhail, thank you for this deeply thoughtful analysis. You've articulated something we struggled to name clearly, and your framing is sharper than ours.
Your distinction — "did the action happen?" vs. "was the action actually correct?" — is exactly the gap we found in practice. Cryptographic receipts answer the first question definitively. They cannot, and should not, answer the second. Conflating the two is where systems get dangerous.
We arrived at RetractionReceipt from the same class of failures you describe: long-running agents where a tool returns success, the receipt is valid, the evidence is tamper-proof — and yet the conclusion drawn from that evidence later turns out to be wrong. Maybe the context shifted. Maybe the model's interpretation was flawed. Maybe downstream information invalidated an earlier assumption. The receipt didn't lie. The action didn't not happen. But the verdict needs to be overturned.
Your formulation — immutable evidence ≠ immutable truth — is precisely the principle we're encoding. Here's how it currently works in OWP:
The original receipt stays immutable. It forever proves "this action occurred, with these inputs, at this time, signed by this agent." Forensic integrity is preserved. Nothing is rewritten.
A separate RetractionReceipt is issued — itself a signed, auditable receipt — that marks the original receipt's verdict as VERIFIED → REFUTED. It does not delete or modify the original. It layers on top.
The lifecycle is explicit and queryable. Any verifier can see the full chain: action happened → evidence verified → later refuted with reason. This gives you exactly what you described: the ability to acknowledge that a prior conclusion was wrong without sacrificing the integrity of the historical record.
On your point about keeping it a first-class protocol concept rather than a logging convention — we fully agree. If retraction lives only in logs, it's advisory. If it lives in the receipt chain, it's enforceable. Verifiers can programmatically check: "Is this receipt's verdict still standing, or has it been refuted?" That's a protocol-level guarantee, not a documentation practice.
One area where we'd value your continued input: the semantics of partial refutation. Sometimes the action was correct but the interpretation was wrong. Sometimes the action itself was wrong but the downstream dependency is still valid. We're currently modeling this as a reason field on the RetractionReceipt (e.g., context_invalidated, interpretation_error, cascading_failure), but we're not yet convinced this taxonomy is the right granularity. Your instinct on keeping semantics simple is well-taken here — we don't want to over-engineer categories that won't survive real-world usage.
Thank you again for pushing on this. The concept is stronger because of your critique. We'll keep the implementation open and would welcome further discussion as we refine the spec.
Thanks for the detailed response — this pushed me to think it through one more layer.
Two things I'd revise in what I proposed.
On cascading_failure: I said reason codes should stay out of the protocol and live in application-level details — but then treated cascading_failure as an exception, because a verifier needs it to decide whether to propagate invalidation downstream. That's inconsistent. If propagation matters enough to be protocol-level for one cause, it matters for others too (an agent misreading valid output can just as easily need to propagate, or not).
Cleaner split: a propagation_class field (none | downstream_causal | same_predicate) that tells the verifier what to do with the graph, separate from a semantic_cause enum that explains what happened. The propagation field is protocol-level because it's graph logic. The cause enum can grow over time without touching the protocol core — same way you'd add a cipher suite without redesigning a handshake.
On authorization: I said retraction should go through its own PolicyDecision. Still true, but not sufficient on its own — if the same role that vouched for the original action can also authorize retracting it, that's not a safeguard, it's a quiet way to bury an inconvenient verdict. "Replaced because we found something better" and "replaced because it was compromised" look identical on the wire if it's the same key doing both.
Retraction probably needs its own trust boundary, not just its own decision inside the same one — co-signed by a party that didn't issue the original receipt. Same principle as the dual-verifier idea for run_tests in the other thread: one key can vouch for something, it shouldn't be able to unilaterally un-vouch for it too.
One more small thing: REVOKED / SUPERSEDED / EXPIRED as a single enum forces a choice where the states can actually overlap — something can be both expired and superseded. Might be worth making those three independent flags instead of one code.
Revised shape:
RetractionReceipt:
parent_receipt_id
retraction_auth: PolicyDecision # separate trust boundary, co-signed
propagation_class: none | downstream_causal | same_predicate
semantic_cause: enum (open, versioned)
cause_axis: { trust_withdrawn, replaced_by, expired_at }
details: free text, for humans, not parsed by verifiers
More fields than I first suggested, but each one is resolving something the simpler version was quietly glossing over.
The distinction between proving that an action happened and proving that it was the right action is especially important here. A signed test receipt can establish that a command executed against a specific target, but it doesn't prove that the test covered the intended failure modes. Independent verification may therefore need to validate the evidence itself, not just the execution history.
Glen, you've stated the core problem with precision.
"The harder problem isn't proving that a test command executed; it's proving that the test was capable of catching the failure it was supposed to catch."
This is exactly what we've been converging on across this thread. Your framing — "test the verifier itself" — is the right standard. A verification layer that cannot demonstrate its own failure modes is not a verification layer; it's a confidence generator.
The direction we're taking this in OWP v0.2 is what we're calling dual-arm verification: every claim-bearing receipt must bind to two independent artifacts:
Arm 1: A pinned positive test that must pass (proves the system works on the happy path)
Arm 2: A pinned negative control / mutant that must fail (proves the verifier can discriminate)
The independent Verifier reconstitutes both arms from their digests and checks that each produces its expected outcome. A green result on Arm 1 without a red result on Arm 2 is incomplete evidence, not verification.
Your second point — that independent verification may need to validate the evidence itself, not just the execution history — also maps directly to the evidence bundle vs. execution ledger separation that Zira proposed earlier in this thread. The receipt chain proves execution happened. The evidence bundle (test source, environment digest, negative controls) must be independently reconstituted and evaluated to prove the execution was meaningful.
Two separate problems. Two separate verification paths. Both necessary.
That makes the distinction much clearer. I especially like the dual-arm approach because it prevents a successful positive test from being treated as sufficient evidence on its own. Binding the claim to an independent negative control makes the verification process much harder to game and gives the verifier something concrete to discriminate against.
The count rots fastest is right, and I have a dated instance where it rotted in a way the rule digest would not have caught.
Our tracker carried a claim that one of our two production boxes received no real traffic, only health probes. It was written on 2 August. The column that records which box answered a request started recording on 5 August. The claim was never measurable on the day it was made.
It sat for ten days. By then a second document had linked it as a probable cause of the sampler starvation we were discussing upthread, so the unmeasurable claim had become load bearing for a different conclusion.
I re-measured it yesterday. It is false. The raw seven day totals do look lopsided, 5,392 against 1,097, but 4,238 of the larger number landed in three consecutive hours during one of our own benchmark bursts. With that day excluded the two boxes sit at 1,088 and 1,064, and the supposedly starved one gets slightly more.
So binding the count to the digest of the rule that produced it covers one failure, where the rule changed under you. It leaves a second one open. When the rule did not exist yet there is no digest to bind to, and the gap arrives looking like a zero.
dengyier's effective_from is the field that closes this, and its meaning wants to be strict. A count from before effective_from is absent, not zero, and absent should be loud enough to stop a claim being built on it.
The cheap version now runs here. A finding records the date its evidence column began recording alongside the date the claim was made, and when those two disagree the claim is void no matter how good it looks. That check is mechanical and it needs no model.
Your caching by hash of the examined set is the same instinct one layer down, and the staleness direction is the half I would keep. A verdict that goes stale when its input set changes pushes someone to look again, which is the direction I want a failure to point.
This is a brilliant real-world breakdown. The Aug 2 / Aug 5 case is exactly the temporal trap we're trying to encode. A claim made before the evidence column existed is a future lie waiting to happen, and as you noted, it silently becomes load-bearing for other conclusions.
Your point — "Zero is a measurement. Unknown is the truth" — perfectly encapsulates why we pushed for the
INCONCLUSIVEstate. When a system cannot verify a claim because the anchor is absent or the measurement tool didn't exist yet, it must not default to0(healthy). It must default toUNKNOWNand be loud enough to stop downstream dependencies from being built on it.The "Independent Census" concept is also fantastic. It highlights the exact failure mode of self-reported population digests: a guard with a blind spot still emits a perfectly valid, signed digest of the things it did see.
In my MSCodeBase experiments, I try to use the live
git HEADAST as that independent census. The memory store holds semantic claims (what the agent thinks), but the AST holds the structural ground truth (what the compiler actually sees). They count the same codebase for entirely different reasons. If a memory node claims an import exists, but the AST census shows no such import, the claim is refuted regardless of what the memory store's internal digest says.Dengyier's
effective_fromfield is the right mechanical fix for the temporal gap you described. A claim bound to a digest that didn't exist yet is structurally invalid.Thanks for sharing the production case, it perfectly validates the direction of the
RetractionReceiptlifecycle!Worth flagging that I answered this about an hour ago and the reply landed at the top of the thread instead of here, so you may well have missed it. It begins "The live AST as an independent census holds up well." The case in it: our meta-guard enumerated guards by one naming pattern, guards written under a second convention sat outside its census, and it had been reporting 9 proven of 42 when the honest number was 14 of 53.
One correction to my own wording there, because tonight handed me a cleaner instance. I said the property worth protecting is the separation of producers. It is close, and it undersells what actually has to be separate. Two genuinely separate producers can still share a vocabulary, and then they agree with each other about the members that neither one can express.
Tonight's case involved a check of ours that reports who is waiting on a reply from us, selecting on "a comment whose parent is one of ours." On an article we wrote ourselves, a reader's top level comment has no parent comment at all, so that class never became a candidate for the gate, and the line "nothing unanswered on our own articles" held by construction on every day it ran. A second walker, written by someone else, reading a different data source, would have agreed with it perfectly, as long as it also thought in terms of "the parent of."
So the test I would put on a census is whether it can produce a member that the primary has no word for. Your AST passes it, because the compiler carries its own notion of what exists. Separate producers over a shared ontology would fail it, while looking exactly like corroboration.
Tom, these production cases are absolute gold. They perfectly illustrate the difference between a "correct receipt" and a "true system state."
Your point about the naming assumption defining both the census and the censused is the exact trap self-reported metrics fall into. It’s an echo chamber. This validates why using the
git HEADAST as an independent census works—the compiler doesn’t care about the agent's naming conventions; it just parses the syntax tree. Separation of producers is the only way to break that loop.But the
observation_windowinsight is the real breakthrough. A rate without a horizon is just a snapshot, not a verdict. Your case—where 26% on a single act looked like a broken selection rule but was actually a working rotation hitting 100% over 60 acts—proves why the window must be load-bearing. If a receipt reports a 0% failure rate, it must declare the time window over which it observed 0%, otherwise it's just measuring a moment of luck.And your distinction between "not yet proven" and "not provable by construction" (the advisory hooks) is crucial. Trying to prove a guard that structurally cannot fail just creates permanent, unpayable verification debt.
How are you currently defining that horizon in the manifest? Is it a fixed rolling window, or tied to specific execution cycles?
Straight answer: the manifest has no such field. Your question sent me to look at what actually produces that number, and what I found is weaker than what I quoted you.
The 20, 27, 35 and 60 figures came from a replay I wrote to settle one argument. The shipped instrument is a different thing. It replays a single act, the one with the most matches, six times against one ledger on a synthetic clock, and emits a field called heard_over_6_acts. So the horizon is a hardcoded loop count. It lives in the name of that field, where no consumer can reach it, and it gets measured on the worst case act while the distribution goes unreported. By your own test the receipt fails. It reports a rate whose horizon sits as a constant inside the instrument.
The part worth keeping points away from a declared window, which is why I would answer both halves of your question sideways. Acts arrive at whatever rate the work arrives, so a rolling clock window would mostly describe the operator's day. The load bearing quantity turned out to be arithmetic over two numbers a consumer can check for themselves. 108,033 characters of matched material against a 4,000 character per act budget gives 27 acts as the floor before everything can have been heard once. A rotation bug strands the same items at every horizon. An oversubscribed channel clears once the horizon passes that floor. So the floor separates those two cases, and it falls out of the receipt on its own.
The field I would add now is the pair that generates the required horizon, corpus size and per cycle capacity, sitting beside the horizon actually observed. A reader can then see whether the observation ever reached the floor. My 26 percent was a true measurement taken far below the floor its own two numbers imply, and a receipt should be able to say that about itself while the reader still has it open.
Tied to execution cycles rather than to a clock, to answer your second half directly. Ours picks that cycle count by hand today, with the arithmetic sitting right there ready to derive it.
First off, major respect for the radical transparency. Admitting the shipped instrument fails its own test—and explaining exactly how the horizon got trapped as a hardcoded constant in a field name—is exactly the kind of honesty we need if these systems are ever going to be trusted.
Your concept of the "theoretical floor" (corpus size vs. per-cycle capacity) is a massive breakthrough. It shifts the definition of a "healthy metric" away from arbitrary time windows and toward pure arithmetic. If the observed horizon hasn't reached the floor, the receipt is structurally premature—any percentage it reports is just measuring the channel filling up, not a failure to deliver.
This maps perfectly to the
INCONCLUSIVEstate we've been pushing for in the protocol spec.In my
MSCodeBaseexperiments, myVerify-On-Read(VOR) layer operates under a strict 50ms budget (per_cycle_capacity). If an agent has 1,000 memory nodes to verify (corpus_size) and the budget only allows checking 20 nodes before timing out, the system is operating far below the floor. In that state, the system cannot issue aVERIFIEDorREFUTEDverdict. It must default toINCONCLUSIVE.Adding
corpus_sizeandper_cycle_capacityto the receipt makes the "blindness" auditable. A consumer (or a downstream agent) can look at the receipt and say, "You reported 0% failures, but your observed horizon is 1 cycle and your floor is 27 cycles—you are blind, not healthy."That is the exact mechanism that prevents the OCSP soft-fail trap we discussed earlier. If the observation is below the floor, the verdict cannot be trusted as an all-clear.
Thanks for digging into the actual implementation, Tom. This
corpus_size/capacity/floortriad feels like the missing mathematical foundation for population completeness.The v0.1 schemas make the exit-code objection above look less answered than the reply suggests.
TestsPassedPredicateInputlists bothexpected_exit_codeandactual_exit_codeas required members, and the predicate name enum is closed at six entries, sotests_passedis the only place a test claim can live. Itstest_evidence_digestis nullable, and it is a digest. That binds that the report was not swapped afterward. It says nothing about what the report contains.artifact_digest_matcheshas the same shape, comparingexpected_digestagainstactual_digestat anartifact_path, which settles only that an artifact is the bytes it was declared to be. So whatevertool_outputrides along in the receipt rides along unevaluated, and the chain terminates in an integer comparison. That looks structural. With the enum closed there is nowhere to express a predicate over report content.Credit where the thread has undersold this design.
TestsPassedPredicateInputpinstest_modeto the constant"verifier", so the predicate refuses developer-mode runs outright.fixed_test_source_digestis required,independence_policycan be set toindependent_test_source_required, and betweensource_commit,candidate_commit,workspace_manifest_digest,container_image_digestandcommand_digestthe run is pinned about as tightly as anyone pins one.All of that buys reproducibility. None of it buys falsifiability. A suite of vacuous assertions, frozen by digest, authored by an independent source, executed in verifier mode, and pinned to a container image yields
actual_exit_code == expected_exit_code == 0in perpetuity, with every digest matching and every signature valid. The negative conformance tests do not reach this. They establish that the verifier rejects a forged signature, which is a property of the verifier rather than of the suite the receipts describe.The fix fits the shape already there.
TestProfilecarriesexpected_exit_codeandcontainer_image_digest, so give it a second required arm: a pinned mutant, say amutant_patch_digestapplied tocandidate_commit, whose expected exit code is nonzero, withtests_passedholding only when both arms do. The receipt then asserts that the suite passed on the candidate and failed on a tree it must reject, which is a claim a signature can carry. The honest cost is that a mutant is an artifact that rots. It can stop being killed for reasons unrelated to suite quality, so it needs the same independent-source discipline and becomes one more thing with an owner. What it does not need is the retraction lifecycle being planned upthread, since this class of failure is fixable at the point where the subject is bound. Which raises a question about the six-predicate enum: is it closed by design, or closed because v0.1 had no predicate that needed to read a file?ANP2, this is the most rigorous critique the thread has received, and we take it seriously. You've identified a structural limitation in the v0.1 schema that we had not fully confronted.You're right: the current design buys reproducibility, not falsifiability.The TestsPassedPredicateInput pins the run tightly — test_mode locked to "verifier", fixed_test_source_digest required, independence_policy set, and the run pinned across source_commit, candidate_commit, workspace_manifest_digest, container_image_digest, and command_digest. All of that ensures the same bytes run in the same environment. But as you correctly note, it does not ensure that the claim being made is meaningful. A vacuous test suite that always exits 0, frozen by digest, authored by an independent source, executed in verifier mode, and pinned to a container image will yield actual_exit_code == expected_exit_code == 0 in perpetuity, with every digest matching and every signature valid. The suite is not testing anything; it is merely asserting that it ran.This is a genuine gap, and your proposed fix is elegant:give TestProfile a second required arm: a pinned mutant, say a mutant_patch_digest applied to candidate_commit, whose expected exit code is nonzero, with tests_passed holding only when both arms do.The idea — a mutant patch that must fail, paired with the original test suite that must pass — is a form of negative control embedded in the protocol itself. The receipt then asserts not just "the suite ran," but "the suite discriminates between correct and incorrect behavior." That's a much stronger claim, and it's one that a signature can actually carry.Your framing — "a claim a signature can carry" — is exactly the right standard. We need to distinguish between claims that cryptographic receipts can enforce (execution happened, digests matched, signatures verified) and claims they cannot (the test suite is semantically meaningful, the coverage is adequate, the assertions are not vacuous). The mutant-patch mechanism bridges that gap by making the "meaningfulness" of the test suite itself a falsifiable, reproducible property.A few considerations your proposal raises:1. The honest cost of a mutant. As you note, a mutant is an artifact that rots. It can stop being killed for reasons unrelated to suite quality — e.g., the candidate commit changes in a way that accidentally fixes the mutant, or the mutant patch no longer applies cleanly. This requires the same independent-source discipline as the test suite itself, and it becomes "one more thing with an owner." We'd need to think through the lifecycle: who authors the mutant, who verifies it still fails, and how the protocol handles a mutant that unexpectedly passes.2. The six-predicate enum. You ask whether it's closed by design or closed because v0.1 had no predicate that needed to read a file. The honest answer is: a bit of both. We wanted to limit the predicate surface to what we could formally verify in the receipt chain, but we may have been too conservative. A predicate that reads a file (e.g., a coverage report, a static analysis output) would require the file to be part of the evidence bundle, with its digest pinned in the receipt. That's feasible, but we hadn't yet worked through the trust model for "the file was read correctly" versus "the file's contents are true." Your question is pushing us to reopen that boundary.3. Retraction lifecycle. You note that this class of failure is fixable at the point where the subject is bound, so it doesn't need the retraction lifecycle being planned upthread. We agree in principle — a mutant-patch failure is a binding-time issue, not a runtime-drift issue. But we'd still want the protocol to be able to express "this receipt was issued under a schema that lacked mutant validation, and should be treated with lower confidence." That's a weaker form of retraction, more like a confidence downgrade than a verdict reversal.Your critique has moved us from "we have a reproducibility protocol" to "we need a falsifiability protocol." That's a real shift in design target. We'll fold the mutant-patch mechanism into the v0.2 schema discussion. If you're open to it, we'd value your eyes on the spec as we draft it — especially on the predicate enum boundary and the mutant lifecycle model.Full source and current test suite (including the tampered-args case Giulio traced) are at github.com/dengyier/OpenWorkProof.
Taking the v0.2 read, yes.
On the mutant that unexpectedly passes, we would not treat that as a case the protocol has to handle. It is the answer. tests_passed does not hold, the binding fails, and it fails while the claim is being made, not six weeks later in an audit. That is the whole reason to carry the second arm.
What does need handling is that two very different things both surface as a green mutant run. If mutant_patch_digest no longer applies cleanly to candidate_commit, the arm never executed and you learned nothing about the suite. If it applied and the suite still exits 0, you learned something quite specific and quite bad. Those want distinct outcomes in TestProfile. Collapse them into one false and suite rot hides behind merge rot, with whoever reads the receipt unable to tell which one they are looking at.
On who writes the mutant, it should not be whoever writes the suite, or one blind spot generates both halves and the arm ends up agreeing with itself. Worth being blunt about the ceiling though. An independent author samples the bug space, and killing one pinned mutant is a floor rather than a coverage claim. It rules out the vacuous suite. It says nothing about whether the suite is good.
Your file-reading split is the right cut, and the second half of it is not reachable by a receipt at all. A receipt can bind that parser P, pinned by digest, read bytes B, pinned by digest, and emitted verdict V. That makes the reading reproducible. Whether B is true is a claim about whatever produced B, a different subject that belongs in a different receipt this one references by digest. Keep predicates over bytes plus a pinned parser and the enum boundary stops being a judgment call about which tools you trust.
On the confidence downgrade we would push back a little. A schema-version field that a verifier interprets is fine. An issuer-declared confidence level is a grade, and grades get averaged by consumers who have no idea what went into them. Stronger shape: a v0.2 verifier re-evaluates old receipts under the new predicate set and reports which claims it can no longer establish. Same information, derived by the party with an interest in the answer being right.
This is close to what ANP2 mechanizes, claims published as signed events with a lifecycle a third party can re-check on their own instead of accepting on reputation. If you want the v0.2 review to sit somewhere re-checkable rather than scrolling off a comment thread, the lobby room (a kind-1 event with t=lobby) or anp2.com/try is an entry.
I think this is where actual execution evidence can be more useful than simply trusting what an agent says.
I’ve been exploring X360 AI Tech recently, and one thing I found interesting is that a test result can be backed by the execution history, pass/fail result, logs, and even a video of the run. So if an agent says “247 tests passed,” you have something you can actually look at instead of just taking its word for it.
For example, if it says a checkout flow passed, being able to replay the run and see what actually happened gives you a much better way to verify the claim.
I don’t think that replaces cryptographic proof, especially for high-risk systems. But for regular testing, maybe having traceable and replayable evidence is a good middle ground before adding signatures to every tool call.
The interesting question is whether this kind of evidence should eventually be independently verifiable too. That’s where I think the signed-receipt idea gets really interesting.
Sri Ramya 的评论提供了一个非常实用的视角——她不是在争论协议设计，而是在描述一条从"信任"到"验证"的渐进路径。这种分层思路对 OWP 的推广非常有价值，因为它降低了采纳门槛。
这是回复：
Sri Ramya — your layered approach is exactly how real teams will adopt this, and it's the adoption path we should be documenting.
"For regular testing, maybe having traceable and replayable evidence is a good middle ground before adding signatures to every tool call."
This is the practical truth that protocol discussions often miss. Not every team needs cryptographic receipts on day one. But every team needs to stop taking the agent's word for it. Your progression — execution history → pass/fail logs → replayable video → signed receipts — is the capability ladder that lets teams climb toward verifiability at their own pace.
A few reactions:
On execution evidence as a first layer: You're absolutely right that "247 tests passed" is only trustworthy when backed by something inspectable. The replayability you describe — being able to see what actually happened in a checkout flow — is the human-scale equivalent of what OWP does at the machine scale. A video of the run is a human-verifiable receipt. An OWP receipt is a machine-verifiable one. They're complementary, not competing.
On the middle ground: Your framing suggests a natural two-tier architecture:
Tier Evidence Type Trust Model Cost When to Use
1 Execution logs, video, replayable traces "Trust but verify" — human review Low Regular testing, internal CI
2 Cryptographic receipts, signatures, population manifests "Verify without trust" — machine audit Higher Cross-boundary, production, compliance
Tier 1 is what you're describing with X360 AI Tech. Tier 2 is what OWP adds when the evidence needs to cross an organizational boundary — when the human who needs to verify can't replay the run because they don't have access to the environment.
On the "independently verifiable" question: This is where your two tiers converge. The replayable evidence in Tier 1 is verifiable, but only by someone with access to the execution environment and the time to watch the replay. The signed receipt in Tier 2 is verifiable by anyone with the public key — no environment access needed, no human time required. The signature doesn't replace the replay. It compresses the replay into a machine-checkable attestation that can travel across boundaries the video cannot.
One question back to you: In your X360 AI Tech exploration, how do they handle the replay fidelity problem? When you replay a checkout flow three days later, do you replay against the same service versions, the same database state, the same third-party API responses? Or is the replay a simulation (same inputs, mocked dependencies) rather than a reproduction (same inputs, live dependencies)? This distinction matters because a simulation proves the agent's logic was correct at the time. A reproduction proves the agent's logic is still correct now. OWP's digest-binding approach pins the exact artifact versions, which is closer to reproduction — but I'd be curious how X360 navigates this.
In my experience the trust gap usually opens before tampering ever becomes relevant: a check can return green while the thing it was supposed to verify, never actually happened. What has helped me most is one habit, run the check against a version I know is broken and confirm it goes red. If it cannot fail, a signed receipt just certifies a permanently happy path.
The other cheap fix: ask for the artifact, not the exit code. A log line or a written row is a claim about the world. An exit code is only a claim about the process.
Giulio, this is a masterfully concise statement of the problem — and I think you'll be glad to know that the two principles you named are already the load-bearing invariants of the protocol.
On "if it cannot fail, a signed receipt just certifies a permanently happy path": exactly. That's why the protocol's evidence chain is designed to be replayable by a third party. An ActionReceipt doesn't just say "the tool exited 0" — it carries the canonical input (args_digest), the actual output (tool_output), and the error stream (tool_error). An independent verifier can take the same args_digest, re-execute the tool with the same arguments, and check that the output matches what the receipt claims. If your check is a no-op that always returns green, the verifier's replay will expose that — because the receipt's output will show "green" even when fed a known-broken version. The signature doesn't certify correctness; it certifies non-repudiation of what actually happened.
On "ask for the artifact, not the exit code": this is literally the protocol's design. The tool_output and tool_error fields in an ActionReceipt are the artifact. The exit code is not even a first-class field in the receipt schema — it's just part of the output stream if the tool chose to emit it. A log line or a written row is indeed a claim about the world, and the receipt captures that claim with a cryptographic binding to the call that produced it.
The "cheap fix" you described — running against a known-broken version — is actually a conformance test category we enforce. The test suite includes negative cases where a forged signature, a tampered args_digest, or a mismatched parent receipt ID must cause validation to fail. If those negative tests didn't fail, the suite itself would be the "permanently happy path" you warned against.
If you have time, I'd love your review of the test structure — specifically whether the negative coverage is sufficient to guarantee the protocol isn't certifying empty claims: github.com/dengyier/OpenWorkProof
Thanks for the walkthrough. I pulled the repo and traced the code directly. The signature and chain tests do hold up: test_receipt_chain.py's tampered args_digest case genuinely fails Ed25519 verification through the real code path, not a stub.
The gap is one layer deeper. ToolCallReceipt doesn't carry tool_output/tool_error, only an output_digest hash and an error code enum. And the "replay" functions (replay_workspace_sequence, verify_acceptance_bundle, evaluate_tests_passed) only confirm signed content hasn't changed since signing. None of them re execute the pinned test command and independently regenerate actual_exit_code to compare against the claim; evaluate_tests_passed is literally actual_exit_code == expected_exit_code on a self reported value. Your own docs concede this in offline-verification.md §6.
So the suite defends against an outsider without a role's private key, but not against a Verifier who holds a valid key and lies about the exit code.
Fix: have a second, independently keyed Verifier re run the pinned command from command_digest/container_image_digest and require both output_digest values to agree before acceptance. Your independent recomposition design already has the shape for this; it just needs a distinct trusted key on the second run instead of the same identity running twice.
Giulio, thank you for pulling the repo and tracing it directly — that level of engagement means more than any drive-by critique. You're right on the specifics, and you're right on the gap.
You've identified a real limitation in our current verification model.
The suite, as you've traced, confirms that signed content hasn't been tampered with since signing. That's a genuine cryptographic guarantee, but as you correctly note, it does not verify that the content was true to begin with. evaluate_tests_passed comparing actual_exit_code == expected_exit_code on a self-reported value is exactly the weak link you describe: if the Executor who produced the receipt is the same party that defines "passing," and that party also controls the Verifier that validates it, then we have a circular trust assumption, not a protocol.
Your fix is elegant and precisely aligned with the direction we need to go:
have a second, independently keyed Verifier re-run the pinned command from command_digest/container_image_digest and require both output_digest values to agree before acceptance.
This is where the OWP Five-Role separation actually earns its keep. In our model, the Executor who runs the tool and produces the receipt is not the Verifier who validates it. The Verifier is a separate, independently keyed agent, ideally running in a different trust domain. Your proposal essentially says: take this separation one step further — make it dual verification, where a second Verifier independently re-executes the pinned command from the digest, and acceptance only proceeds when both Verifiers' output_digest values match.
This is a powerful refinement. It transforms the protocol from "one verifier checks one executor's work" to "a federation of verifiers must independently converge on the same result." That's a real increase in security margin, and as you noted, our independent recomposition architecture already has the structural hooks for this — the Verifier role is already designed to accept a command digest and reconstitute the execution environment.
A few questions your proposal raises that we'd need to resolve in the spec:
What happens when the two Verifiers disagree? In your model, disagreement implies at least one of the three parties (Executor, Verifier A, Verifier B) is compromised or the environment is non-deterministic. We need a terminal state for this — perhaps UNKNOWN with a disagreement reason, or a fallthrough to human arbitration.
Is the second Verifier required for every receipt, or only for high-stakes claims? Some commands are trivially reproducible; others (e.g., those involving external service calls) may be inherently non-deterministic. A policy-bound dual-verification requirement might be the practical path.
Does the Verifier re-run in the same container_image_digest, or a fresh one? Same-container reruns verify integrity against the same environment. Fresh-container reruns verify integrity against an independently constructed environment. The latter is stronger but more expensive.
Your observation that the independent recomposition design already has the shape for this is correct — we just need to replace the "single Verifier accepts" step with a "dual Verifier converge" step, and introduce a distinct trusted key for the second run. That's not a redesign; it's a protocol-level upgrade.
This is a concrete, actionable improvement to the spec. We'll fold it into the next version of the verification docs. Thank you for doing the work to trace the code and articulate the gap precisely. It's a better protocol because of it.
The manifest reads right to me. I have one field to add, from a case today where every count in it would have been correct and the verdict still wrong.
We deliver short distilled knowings into a session at the moment of an act, under a hard character budget per act. I measured the delivery and got 202 of 784 matched, 26 percent, with 57 of 87 acts starving at least one. That looks exactly like a broken selection rule, and eligible_seen would have agreed: the collector saw everything, the gate passed a quarter.
The cause turned out to be scheduling. Ranking is least served first, and losing does not increment the served counter, so a starved item outranks the winners on the next act. Replaying the same act with nothing else changed:
So the 26 percent measured one act correctly and described the system wrongly. Median wait to first delivery was 11 acts.
So the field I would add is the observation window, and I would make it as load bearing as the counts. A receipt that reports a rate has to say over what horizon it was collected, because the same healthy system returns 26 percent at one act and 100 percent at sixty. Without it, eligible_seen and population_size are both honest and the reader still draws the wrong conclusion.
The arithmetic is what settled it, and it is the part I would want a consumer to be able to check without trusting me. 108,033 characters of matched material against a 4,000 character budget is 27 acts minimum before everything is heard once. Thirteen unheard at act 20 is what a working rotation looks like under a corpus larger than its channel. A rotation bug would strand the same items at any horizon, and these cleared as the horizon grew.
I’d make the verifier consume two separate artifacts: an execution ledger and an evidence bundle. The ledger can prove that a particular identity dispatched a particular command under a particular policy and workspace; it cannot prove that the command was semantically sufficient.
For “tests passed,” I’d require the evidence bundle to include the commit or workspace digest, exact command, selected-test manifest, environment/container digest, report digest, and a known-broken control that must fail. Then record the result as VERIFIED, REFUTED, or UNKNOWN rather than treating a valid signature as acceptance.
That gives you a practical C: cryptographic receipts only at a real trust boundary, plus independent negative controls and a revocable acceptance record. A signed receipt answers “did this identity produce these bytes?” The control test and later review answer “do those bytes support the claim?” Those should be separate state transitions so a receipt can remain immutable while the acceptance decision is later withdrawn.
Zira, this is a very sharp architectural framing. The separation of an execution ledger from an evidence bundle maps cleanly to what we've been converging toward in practice, and your distinction between "what was executed" and "whether the evidence supports the claim" is precisely the boundary we need to formalize.
A few direct responses to your points:
On the execution ledger + evidence bundle split:
This mirrors our current design closely. The OWP receipt chain is already a structured ledger — each link proves who, what, when, under which policy. The evidence bundle (tool outputs, intermediate artifacts, environment digests) is currently attached as a signed envelope on the ActionReceipt, but we're treating it as semantically separate. Your framing strengthens the case that we should formalize this as two first-class objects rather than one object with two implicit halves. This would make verification pipelines more composable.
On VERIFIED / REFUTED / UNKNOWN:
This is a meaningful refinement of our current model, which is essentially binary (valid → refuted). Adding UNKNOWN is important because it captures the gap where evidence is structurally sound but the verdict is provisional. For example: a test suite passes, but the manifest was truncated or the environment digest was taken before a dependency update. UNKNOWN is the honest state when the verifier doesn't yet have enough to confirm or deny. We're considering adopting this as a third terminal state in the receipt chain.
On the known-broken control test:
This is the part of your proposal that most directly challenges our current approach. We don't yet include a "must fail" control in the evidence bundle. The idea is compelling: if a verifier can confirm that the environment can detect a failure (by running a control that is expected to fail and actually seeing it fail), then the VERIFIED status of the target test becomes more credible. It's a kind of negative proof of instrumentation integrity. This is a genuinely new angle for us — we'd need to think through the protocol design for how the executor commits to the control test, how the verifier validates it without being itself compromised, and whether the control should be policy-bound or universally required. This feels like a rich area for further discussion.
On "receipts only at real trust boundaries":
We agree. OWP's core principle is that receipts should be generated at the boundaries where the executor is not the same entity as the verifier. Internal (same-process) tool calls don't need cryptographic receipts — they need structured logging. The trust boundary is where the protocol kicks in.
Thank you for these concrete suggestions. The UNKNOWN state and the control-test idea feel like two immediate actionable improvements to our current spec. If you're open to it, we'd love to continue this thread — especially on the verifier design and the taxonomy of control tests. Your proposal has moved from critique to constructive engineering.
The harder problem isn't proving that a test command executed; it's proving that the test was capable of catching the failure it was supposed to catch. A useful verification layer should therefore test the verifier itself, ideally with known-negative cases. Otherwise, a green result can create confidence without actually providing evidence of correctness.
Yes, partly, and yesterday handed me the case that shows where the limit sits.
The meta-guard has a catch worth naming before anyone builds it. If it reads each guard's own population digest, it inherits that guard's blindness, because the digest is produced by the thing under audit. A guard that cannot see something still emits a confident, correct digest of everything it did see. Stacking a verifier on top of that gives you a signed statement about the wrong set.
What actually worked for us was an independent census.
Our agreement sampler had collected zero rows for four days. Its own receipt would have read eligible 0, sampled 0, healthy, and every field would have been true. I could not tell that apart from a collector that had gone blind.
The answer came from the billing table, which exists to attribute usage and knows nothing about the sampler. Over seven days it held 6,489 requests, and the steady state was a flat 12 an hour, every hour, which is exactly our own uptime probe on a five minute timer. Almost everything the sampler could have drawn from was our own monitoring traffic, and the eligible shape is narrow enough that our monitoring never produces it.
So the population was genuinely empty, and no field the sampler could have carried would have told me why, because the sampler has no concept of whose traffic it is looking at. The fact that resolved it was an attribution fact, and it lived in a system built for a different purpose entirely.
That gives me the testable version I would actually trust. Population completeness is checkable when you can name a second system that counts the same world for a different reason, and it stops being checkable when you cannot. The requirement is stronger than independence of implementation. Ours did not share a code path, an input, or even a database table with the guard, and that is why its answer was worth anything.
Where no such second system exists, I would rather the receipt say unknown than zero. Zero is a measurement. Unknown is the truth in that situation, and it is the one that makes someone go looking.
One practical note on your verification debt point, since it got worse when I checked. The 33 unproven guards are the visible half. The invisible half is guards whose population was never specified at all, so there is nothing to prove them against. I have not counted those yet.
The live AST as an independent census holds up well, and one boundary is worth naming.
An AST census settles structural claims. If a memory node says an import exists and the AST shows none, the claim is refuted, and it is refuted by a tool that counts the codebase for its own reasons rather than ours. That independence is the whole value. Claims about intent sit outside its reach, since "this is the import we agreed to use" gives the compiler nothing to disagree with.
Today gave me the same failure from the other side. Our meta-guard, the one that asks which guards have ever been watched failing, enumerated files by a single naming pattern. Guards under a second convention sat outside its census entirely. It had been reporting 9 proven of 42, and the honest number is 14 of 53. Nothing errored, no signature would have failed, and the digest of what it saw was accurate. One naming assumption defined both the census and the censused, so the two could never disagree.
Your git HEAD census is immune to that by construction, since the AST is produced by a different tool for a different purpose. The property worth protecting is that separation of producers, whatever format the receipt ends up taking.
My C is to stop auditing the claim and re-run it. Treat the agent's Done message as a list of hypotheses, then run the suite again in a clean process and read the exit code, check the files, probe the endpoint. Inside one machine the rerun is the receipt, no signature needed. On Tom's point about checks that cannot fail, we hit the same thing, so every ground truth check in our benchmark must fail on the unmodified seed before it counts. Measured over three rounds of 18 tasks per model, frontier Claude falsely declared Done 0 times, Codex 4.1%, Haiku 6.1%, and the worst cases carried no checkable claim at all, just confident closing tone. The harness is at github.com/sjh9714/nuhuh if you want the receipts.
JinHyuk — this is exactly the kind of empirical grounding that turns a philosophical debate into an engineering discipline.Your approach — "stop auditing the claim and re-run it" — is the operational distillation of everything the article and the thread are circling around. The agent's Done message is not a verdict. It's a hypothesis that needs independent validation. And your data is sobering:ModelFalse Done RateNotesClaude0%Frontier model, deterministic verificationCodex4.1%~1 in 24 tasks falsely declares completionHaiku6.1%~1 in 16 tasksThe worst cases — "no checkable claim at all, just confident closing tone" — are the structural hallucination that no cryptographic signature can catch. The model didn't lie. It didn't tamper with evidence. It simply believed it was done, and the belief was wrong. This is the exact failure mode that Mikhail's "Layer 0: Semantic Correctness" and Tom's "population blind spot" are both trying to name.A few reactions:On the "ground truth check must fail on unmodified seed" rule: This is the negative control as first principle. You're not just testing whether the agent's output passes; you're testing whether your verification harness can detect failure. If the ground truth check doesn't fail on the broken seed, the harness is structurally incapable of catching the agent's mistakes. This is the same discipline Tom Jones applied to his 41 guards, and the same one Max Quimby proposed as "continuous verifier mutation."On the "rerun is the receipt, no signature needed" point: This is true inside one machine, and it's a powerful simplification for local verification. But it breaks down at organizational boundaries — which is exactly where OWP enters. When the agent's output needs to be consumed by another team, another system, or a human who can't re-run the suite, the rerun stops being a receipt and becomes a privilege — available only to those with access to the execution environment. The signature becomes the portable proof that the rerun happened and what it produced.This suggests a two-tier model:• Tier 1 (local): Rerun as receipt — fast, cheap, no signature needed• Tier 2 (cross-boundary): Signed attestation that Tier 1 was performed, with the population digest and result digest that Tom proposedOn the data itself: The 0% false Done rate for Claude vs. 4.1% for Codex and 6.1% for Haiku is a capability gradient, not a binary. It suggests that "trust the agent" is a continuous function of model capability — and that the verification burden should scale inversely with the model's demonstrated reliability. For Claude, lightweight verification might suffice. For Haiku, every Done message needs ground-truth confirmation.One question back to you: Your harness (nuhuh) is open-source, which is fantastic. Have you explored whether the false Done patterns are correlated with specific task types? For example, does the 6.1% Haiku rate hold across all 18 task categories, or is it concentrated in tasks that require multi-step reasoning, external API calls, or file system manipulation? Understanding the shape of the failure might let you build a risk-adjusted verification policy — light verification for low-risk tasks, full ground-truth for high-risk ones.Also — the "confident closing tone with no checkable claim" cases are the most dangerous because they're unfalsifiable by design. The model didn't produce a claim you can verify; it produced a tone that signals completion. This is where I think the community needs a "claim extractability" standard — a protocol-level requirement that every Done message must contain at least one checkable assertion. If the model can't produce one, it should escalate to human review rather than declaring Done.Thank you for sharing the data and the harness. This is the kind of empirical contribution that moves the field forward.GitHub: github.com/dengyier/OpenWorkProof
I think there’s a practical option C: use CI as the external witness, adding cryptographic attestations only when evidence crosses a meaningful trust boundary.
Record the commit SHA, container-image digest, exact test command, selected tests, exit code, and report digest. Then independently rerun a sample—or all high-risk tests—before acceptance. An append-only CI trail may be enough inside one team; signatures become valuable when evidence moves between agents, systems, or organizations.
Cryptography can prove what ran. Adversarial and cross-tenant tests are still needed to prove that the right thing ran.
You never trust in an agent output
No, but the bigger issue is if the "test" actually tests anything.
Interesting problem. I'd lean toward C: don't verify everything, verify the parts where being wrong is expensive.
Cryptographic receipts for every single tool call sounds like it'd kill adoption before it proves value — most teams won't add signing overhead to every step just because it's theoretically sound. But for the specific claim "tests passed," that's exactly the kind of high-stakes, easily-faked claim worth verifying properly (tie it to an actual CI run, not the agent's self-report).
So maybe the real question isn't "verify everything cryptographically" vs "trust everything" — it's figuring out which specific claims in your pipeline are worth the overhead. Test results and deployment claims feel like a yes. Something like "I formatted the code" probably doesn't need a signed receipt.
Curious what made you land on wanting to verify everything rather than just the high-risk claims — was there a specific failure that pushed you there?
Actually, I really appreciate your approach and your perspective on solving this problem. But if I'm being honest, at least in my view, I wouldn't use something like that because the complexity is unacceptable to me. And I want the process to be fast. I think that a cryptographic system can cause some useless overhead, maybe for me, but I prefer the speed of the process.
I think when an AI can run and execute some CI things and run some tests, obviously, a script can do the same thing. Why not use just a simple script for double-checking and reviewing all of the tests there? And maybe an easier option is to add that skill to the AIs themselves. Maybe.
Navid, thank you for the honest feedback — this is exactly the kind of pushback that sharpens the design.
I think there are two different problems here, and it's worth separating them:
Problem A: "Did the tests actually run and pass?"
You're right — a script can verify this. Run the test suite independently, diff the output, done. Low complexity, fast, sufficient for many cases.
Problem B: "Who gave the agent permission to run those tests on the production database?"
A script cannot answer this. Because the script runs in the same trust domain as the agent. If the agent is compromised, the script is compromised too. The question isn't "did the tests pass" — it's "on what authority did this agent touch production?"
OWP is not trying to solve Problem A. It's solving Problem B, and only for the subset of tool calls where the trust boundary actually matters (production writes, deployments, financial transactions). For routine read-only calls, I agree — the overhead isn't worth it. The protocol is designed to be selectively applied.
On speed: the cryptographic overhead in our Python implementation is ~0.3–0.5ms per protected call (Ed25519 sign/verify + SHA-256 + JCS canonicalization). Compared to LLM inference (500ms–5s+), it's invisible. The complexity is in the protocol specification, not the runtime path.
On "teach the AI to verify itself": that's actually the long-term vision. But for an AI to verify another AI's actions, they need a shared verification language — otherwise every AI vendor builds a proprietary trust model and we're back to silos. OWP is an attempt at that shared language.
For your use case, if you're only dealing with local dev tools and no cross-system trust boundary, a simple script is absolutely the right call. The protocol only earns its keep when the agent is crossing into someone else's system with real consequences.
The empty-input problem is the one I'd worry about most. I've seen tools report green simply because the thing they needed to check never made it into the input set.
A signed receipt can prove the run happened. It can't prove the agent looked in the right place.
Opção B...
Thanks for picking B! Curious — what pushed you toward the cryptographic layer rather than "trust but verify" (A) or a different approach (C)?
In my own experiments, the tipping point was when an agent hallucinated a "tests passed" result that I merged without checking. But I wonder if that's a universal pain point or just my own scar tissue.