An executable synthetic test for a subtle failure in AI-agent review evidence.
Two AI agents can approve the same pull request. They can even attach two different names to their reports. That still leaves a question: who actually had the authority to produce the review evidence?
In discussions following my posts on what an AI agent means by “done” and whether two AI reviewers are truly independent, TokenLat made a useful distinction: establish that the required evidence exists and is bound to the right candidate first; then test whether an unauthorized producer could make that evidence look independent.
I turned that question into a small, fully synthetic Node.js fixture. It uses no AI model, real repository, or production credentials.
The one-field problem
Imagine an evaluation event:
{
"id": "e-1",
"type": "evaluation",
"candidate": "sha256:synthetic-pr-head-a",
"claimedProducer": "independent-reviewer"
}
A naive checker might look for an evaluation event, compare the candidate, see independent-reviewer, and return PASS.
But what if the implementer emitted that exact JSON?
Changing a producer label is not a change of authority. If the same component can write both the alleged independent review and the receipt naming its author, the evidence is self-attestation.
Four cases, not just one happy path
I used one fixed candidate identity and three event types: declaration, evaluation, and acceptance. The fixture tests:
| Case | What changes? | Required verdict |
|---|---|---|
| 1. Complete | All three events exist and have matching, authorized provenance | PASS |
| 2. Missing evaluation | Remove the evaluation event | PENDING |
| 3. Wrong candidate | Bind the evaluation to a different candidate | DENY |
| 4. Spoofed reviewer | The executor emits an evaluation while claiming to be the independent reviewer | DENY |
The fourth test only makes sense after the first three: otherwise a rejection could simply be caused by a missing or mismatched record, not a failure to verify producer independence.
A claimed name is not an authenticated identity
The essential check is not:
receipt.claimedProducer === 'independent-reviewer'
It is closer to:
hostRecord.authenticatedRole === 'independent-reviewer' &&
hostRecord.candidate === receipt.candidate &&
hostRecord.eventDigest === digest(receipt)
The crucial distinction is where hostRecord comes from. It must come through a separately controlled, authenticated evidence channel. A producer-supplied field named authenticatedRole would accomplish nothing.
In my toy fixture, a mock host journal supplies that trusted provenance. The check is deliberately narrow: it demonstrates how a string-only verifier can wrongly accept a forged reviewer label, while a verifier using independent provenance rejects it. The mock does not provide real authentication, signature verification, filesystem isolation, or runtime enforcement.
I ran the four cases and one additional regression test against the naive checker. All five Node.js tests passed. In the spoof case, the string-only verifier returned PASS and the provenance-aware toy verifier returned DENY.
What this does—and does not—say about ARC
I maintain Agent Role Contracts (ARC), which checks the consistency of declared agent roles, task scopes, and review relationships. That is a declaration-level check.
An ARC declaration by itself does not authenticate the reviewer who produced an event, prevent an executor from writing to a review channel, or prove that the reviewed revision was the one merged. Those responsibilities belong to the host, authorization system, CI, and final acceptance process.
The same discipline applies to exemptions. “Independent review not required for this task” should be an explicit, authorized scope decision. Silence or a missing event is not an exemption.
The question I would ask your agent system
If the implementer changed just the producer field in its own evaluation receipt to the expected reviewer name, what independent control would prevent that receipt from authorizing the next step?
I am particularly interested in minimal examples that prove the boundary without adding an entire second orchestration platform.
Disclosure: This is a synthetic fixture and a proposed evidence boundary, not a claim that ARC currently enforces runtime producer authentication. The runnable .mjs file accompanies the draft and should be published at a stable source URL before referring to it as publicly reproducible.
Top comments (1)
The one-field problem is real, and the field to distrust is claimedProducer itself: while the checker reads it, an implementer can write it. The fix is not a stricter comparison, it is evidence the implementer cannot produce - an HMAC over (candidate, verdict) with a key scoped to the reviewer identity, so forging requires a key rather than a string.
There is also a cheap test for whether that field ever mattered: delete claimedProducer from a legitimate event and re-run. If the verdict stays PASS, the field was decorative and can go. If it flips to FAIL, the field was load-bearing, and so is the forgery.
That pair - absent field, present forgery - is what makes the fixture about authority instead of labels.