Here is the attack that motivated our newest experiment.
A policy evaluator receives an expense request for 250.00 and produces a disposition that supports that action.
That judgment is committed into a signed work order. A capability grant is issued. The action executes. Receipts are produced. Evidence is attached. An acceptance is signed.
An offline verifier checks the bundle:
- signatures valid
- causal parents correct
- evidence digests correct
- authorization chain valid
Everything passes.
Except the action that now appears in the coherently re-signed chain is:
2500.00
An insider changed the amount after the judgment and re-signed the affected artifacts with valid keys.
From the execution protocol's point of view, the new chain can still be internally consistent.
But it is no longer the action supported by the recorded judgment.
That distinction became Study 014.
Where this study came from
Study 014 follows directly from Study 013 - Agent Eval Forge integration.
Study 013 asked the runtime behavior question:
If the judgment is correct, does the surrounding agent actually respect it?
We integrated JPS with an independently developed agent regression harness, deliberately injected decision-layer and integration-layer failures, and measured which layer caught what.
That study showed an important separation:
- the judgment layer can detect errors inside the decision semantics
- an external trajectory harness can detect cases where the application misuses an otherwise correct decision
- neither layer automatically proves the historical relationship between the judgment and the action afterward
Study 014 starts exactly at that last boundary.
It asks the provenance question:
After execution, can an independent third party prove which recorded judgment corresponds to which recorded action?
So the progression is:
Study 013
Did the application behave consistently with the judgment?
↓
Study 014
Can a third party later verify the binding between
the judgment and the executed action?
Why EvalForge and OpenWorkProof are different
The external systems in Studies 013 and 014 solve different problems today.
Agent Eval Forge, as it currently exists, is primarily a regression-testing harness for agents. You define scenarios and expected behavior, run an agent, and inspect the resulting trajectory - which tools it called, which arguments it passed, whether it crossed a safety boundary, and whether a change introduced a regression.
Its question is roughly:
Did this version of the agent behave the way we expected?
OpenWorkProof, in its current form, operates further downstream. It creates signed work orders, capability grants, execution receipts, evidence relationships, and an offline verification bundle.
Its question is closer to:
Was this recorded action authorized, and can another party later verify the signed execution and evidence chain?
That is why they were useful for two different studies.
JPS
What should happen under these facts and rules?
|
+-------------------------+
| |
v v
Agent Eval Forge OpenWorkProof
Study 013 Study 014
Did the system Can the recorded
behave correctly? authorization and
execution lineage
be verified afterward?
These are descriptions of the projects as they exist today, not permanent category boundaries.
Both are early open-source projects, and their capabilities could expand or converge over time.
EvalForge could add more runtime verification or provenance features. OpenWorkProof could add richer policy evaluation, testing, or regression capabilities. JPS itself may also expand its integration surface.
The useful distinction for these studies is therefore not:
These projects can never overlap.
It is:
Which responsibility did each independently developed implementation actually own at the pinned version we tested?
Study 013 used EvalForge as an external observer of agent behavior.
Study 014 uses OpenWorkProof as an external protocol for authorization and receipted execution lineage.
That keeps the experimental claims tied to real implementations rather than drawing permanent product-category boxes around three projects that are still evolving.
Two independently designed layers
Study 014 composes two systems that were designed independently.
Judgment Pack / JPS handles judgment:
Given this exact pack and these exact facts, what disposition follows?
OpenWorkProof handles authorization and execution lineage:
Was this actor allowed to perform this action, what happened afterward, and does the signed receipt chain verify?
The distinction matters.
JPS does not issue capability grants.
A JPS disposition does not authorize execution.
OpenWorkProof does not decide whether a business judgment is substantively correct.
Each system owns a different part of the chain.
Study 014 asks whether those parts can be bound strongly enough to survive adversarial substitution.
Why signatures alone are not enough
A digital signature tells you something important:
These bytes were signed by this key.
What it does not automatically tell you is:
These bytes still represent the same business judgment that originally justified this action.
That gap becomes especially interesting under an insider threat model.
Simple tampering is easy to detect.
Change an amount without re-signing the receipt and signature verification fails.
But suppose the attacker controls valid signing keys.
They can change the action and construct a new, cryptographically valid chain.
Now the question is no longer:
Was something tampered with?
It becomes:
Is the newly valid chain still bound to the exact judgment, facts, and action that belong together?
That was the attack surface we wanted to test.
We did not modify OpenWorkProof
This was important methodologically.
We pinned OpenWorkProof at commit 8eeca6f and used its verify_acceptance_bundle verifier unchanged.
For the pinned version we tested, the cleanest interoperability boundary was to let OpenWorkProof remain unaware of JPS.
OpenWorkProof can carry external data inside fields it signs without needing to understand what that data means.
So we created a small adapter between the two systems.
The adapter builds a judgment commitment over:
- exact Judgment Pack bytes
- exact normalized input bytes
- canonical JPS disposition
- evaluator replay tuple
- exact authorized action
That commitment is then carried at signed points in the OpenWorkProof chain.
The verification ceremony has three separate layers.
Layer 1 - OpenWorkProof verification
Are the signatures, grants, causal relationships, authorization windows, receipts, and evidence sets internally valid?
Layer 2 - judgment binding
Do the retained pack, facts, disposition, and action recompute to the commitment carried by the signed chain?
Layer 3 - deterministic replay
If we evaluate the retained pack again with the retained inputs and pinned evaluator, do we reproduce the recorded disposition?
No layer is asked to pretend it knows something it does not.
And importantly, we do not claim that both signed commitment points are independently necessary. In this study they are defense in depth, not a proven minimal construction.
Then we tried to break it
The locked study stratum contained 39 registered cells.
That included 33 one-at-a-time mutation constructions, five validity controls, and one demonstration. Of those, 32 were endpoint cells used for the primary registered result.
The attacks covered six broad areas:
- judgment artifact substitution
- fact substitution
- disposition forgery
- action substitution
- replay and drift
- causal-chain corruption
Where it mattered, we tested two attacker capabilities.
The easy version was ordinary tampering:
Change something and leave the old signature behind.
The harder version was coherent reminting:
Change something and re-sign the resulting artifacts with valid study keys.
That second category is where the architectural boundaries became much more interesting.
Different layers caught different attacks
The detection ownership ended up being one of the most useful outputs of the experiment.
OpenWorkProof caught chain failures
The unchanged OpenWorkProof verifier caught the things its protocol is designed to own:
- post-signature tampering
- broken causal parents
- authorization-window violations
- evidence-set inconsistencies
- malformed execution chains
In those cases, JPS did not need to know anything was wrong.
The execution proof itself was invalid.
The binding layer caught judgment-side substitutions
Now consider the opening attack.
The amount changes from:
250.00
to:
2500.00
and the attacker coherently re-signs the affected chain.
The OpenWorkProof verifier can still pass because the newly produced artifacts are cryptographically self-consistent.
But the adapter recomputes the action commitment and sees:
This is not the action bound to the recorded judgment.
The same pattern applied to substitutions such as:
- different pack bytes
- different pack version
- changed facts
- different action
- altered action arguments
This is not a weakness in signature verification.
The signatures are doing exactly what signatures do.
The missing property is semantic binding across independently owned layers.
Replay caught another class
There was a third failure class.
Suppose an attacker creates retained artifacts that are internally consistent with one another.
The pack digest matches.
The facts digest matches.
The recorded disposition matches the commitment.
The chain verifies.
But the disposition itself is forged.
Now neither cryptographic integrity nor digest agreement necessarily tells you that the judgment is the one the evaluator would actually produce.
So we replay it.
Same pack.
Same inputs.
Same pinned evaluator.
Recompute the disposition.
Several mutations were caught only at this layer.
That gave us a useful separation:
Internal consistency is not the same as semantic reproducibility.
The sharpest holdout case
The locked matrix was useful, but it was still developed while we were building the study.
So the reviewer authored a separate holdout set.
Those eight cases were committed before freeze and were not executed until the registered attempt.
One of them was particularly useful.
The attacker constructed a self-consistent wrong action.
The commitment agreed with the receipt.
The receipt chain was valid.
OpenWorkProof verification passed.
But the action was not compatible with the recorded JPS disposition under the study's registered disposition-to-action map.
Only that final binding check rejected it.
That is probably the cleanest demonstration of the boundary we were trying to study:
Agreement between attacker-controlled records is not enough to establish that the action follows from the judgment.
The holdout finished:
8 / 8 adjudicated, 0 divergent, 0 pipeline-invalid.
The locked stratum also finished with:
39 / 39 adjudicated, 0 endpoint divergences, 0 pipeline-invalid.
Two things nothing inside the chain could detect
I think these results matter as much as the successful detections.
Two registered boundaries remained outside the composition.
Decision currency
Suppose the chain contains a perfectly valid judgment produced from a perfectly valid pack.
But that pack is no longer the organizational policy that should be used today.
Nothing inside the old chain can discover that by itself.
The chain can prove:
This action corresponds to this recorded judgment.
It cannot prove:
This is still the judgment artifact the organization currently wants applied.
That requires some source of current authority outside the chain.
Alternative valid WorkOrder rollback
A stronger insider can also coherently construct an alternative valid WorkOrder and remint the associated chain.
All signatures can be valid.
All internal references can agree.
All three verification layers can pass.
From inside that self-contained universe, there is nothing distinguishing:
the currently authoritative chain
from:
another coherently signed chain.
Detecting that requires an external anchor.
A transparency service, monotonic registry, trusted current-version pointer, or another external authority could potentially provide such an anchor.
Study 014 does not.
We registered that boundary instead of patching around it after seeing the result.
One practical integration trap: unsigned metadata
The study also found a much simpler lesson.
OpenWorkProof has a generic evidence-bundle metadata area.
It is useful metadata.
It is not a judgment-binding location.
We created a demonstration where the JPS reference lived only there.
Substitute that reference and OpenWorkProof verification remains green.
That is expected because the verifier was never given a signed semantic commitment to check.
For our integration, moving the commitment into signed carriage through WorkOrder.objective and AgentRequest.context_source_digest closed that gap.
The practical lesson is simple:
Do not confuse "included in the bundle" with "cryptographically bound to the action."
That distinction may be worth making explicit in integration documentation.
The study itself failed review several times before the system did
The technical result matters.
The review process may matter more.
The review record spans seven cross-vendor rounds.
The reviewer was a different vendor's model running read-only, with every prompt and response retained verbatim.
Round 1 opened with 14 findings, including four blockers.
Those blockers were not cosmetic.
One found that our fixtures bypassed OpenWorkProof's real patch execution path.
That would have made the verifier look stronger by testing chains that a real deployment might never produce.
Another found that our own execution-receipt selection logic could be bypassed simply by omitting a marker.
That was almost perfectly embarrassing:
We built a study to detect integration bypasses, and the study's own integration ceremony had one.
Another found that what we initially wanted to call prospective evidence was actually postdictive.
That forced the split between the locked replication and the reviewer-authored holdout.
Another found that our freeze-integrity mechanism was not actually sufficient to freeze the study.
Later rounds kept finding problems.
One draft expectation was falsified during pilot work and changed before freeze.
The holdout machinery itself was repeatedly tightened so the reviewer cases could not accidentally be executed early or silently rebuilt after seeing the result.
Even the seventh review round still found a package-metadata shadowing path that needed to be closed before the study was ready.
That is why I do not describe the review as validation that our initial design was good.
It did the opposite.
It repeatedly showed where it was not good enough.
Why the holdout matters
The 39-cell locked stratum is useful evidence.
But it is still a replication of behavior we had observed while constructing and hardening the study.
The eight holdout cells are different.
They belonged to the reviewer.
They were registered before execution.
Their first execution happened only after freeze.
One was deliberately designed so that a specific review fix had to be correct or the study would diverge.
All eight landed on their registered outcomes.
That does not turn eight cases into a security proof.
But it gives the study at least one genuinely prospective stratum rather than letting the entire result be a polished description of behavior already seen during development.
What a green result actually means
After all of that, the final claim is narrower than the title might suggest.
A green Study 014 verification means:
The retained artifacts and receipted lineage are internally consistent, and the executed call represented by the chain is the one associated with the recorded judgment under the registered binding ceremony.
It does not mean:
- the policy is correct
- the supplied facts are true
- the JPS disposition authorizes the action
- the physical action necessarily happened
- OpenWorkProof is secure against arbitrary attackers
- JPS is conformant because this experiment passed
- every possible binding attack has been covered
The trust roots still exist.
The study relies on the work order's study-minted keys, the pinned JPS evaluator, the adapter implementation, and the retained artifact store.
This is a binding and lineage result.
Not a truth result.
Why I think the separation matters
Agent systems increasingly have several distinct control problems.
Judgment
Given these facts and organizational rules, what should happen?
Authority
Is this actor permitted to perform that action?
Execution
What action actually ran?
Evidence
Can another party later verify the relationship among those three?
Trying to make one component answer all four questions tends to blur the trust boundaries.
Study 014 instead let each component remain narrow.
JPS did not become an authorization protocol.
OpenWorkProof did not become a business judgment engine.
And the adapter did not attempt to decide either question.
It only committed the relationship between them strongly enough that the relationship could be independently recomputed.
That is the result I find most interesting.
Not that every attack was caught.
Some deliberately were not.
The interesting part is that we can now say which layer should see which failure, and where the evidence simply does not exist to see it at all.
Receipts
Everything is public.
Study 014 - decision-to-execution binding
This is the study described in this post - offline provenance and binding between a recorded JPS judgment and the action represented in an OpenWorkProof receipt chain.
Study 013 - Agent Eval Forge integration
Study 013 is the runtime sibling of this work. It asks whether the surrounding agent/application actually behaves consistently with the judgment and uses an independently developed regression harness to distinguish judgment failures from integration failures.
The two studies answer different questions:
013 - Did the system respect the judgment at runtime?
014 - Can we prove the judgment-to-action binding afterward?
Study 014 analysis
Study 014 detection matrix
Cross-vendor review record
OpenWorkProof
https://github.com/dengyier/OpenWorkProof
Study 014 is not a security audit or endorsement of OpenWorkProof. It is one interoperability falsification study against one pinned commit, one Judgment Pack, one action encoding, and one adapter written by us.
The most useful next question is probably the one the expected-undetected boundaries leave behind:
Where should the external anchor live when a perfectly valid receipt chain is no longer the authoritative one?
Top comments (1)
This is exactly why receipts need semantic context, not just cryptographic validity. A signed record can prove something happened while still failing to prove that it was the right action for the approved judgment.