DEV Community

Cover image for ARC Failure Lab #2: A Signed PASS for the Wrong Commit
asagiman for Netsujo

Posted on Edited on

ARC Failure Lab #2: A Signed PASS for the Wrong Commit

A pull request passes independent review. The reviewer signs a PASS receipt. The implementation branch advances by one commit, but the old receipt is still available.

What, exactly, does that signature approve?

We build Agent Role Contracts (ARC) at Netsujo. ARC validates the consistency of declared roles, scopes, review relationships, and handoffs. Its declaration-level PASS does not authenticate reviewer keys or authorize a merge. Those are responsibilities of the surrounding runtime and integration gate.

This second Failure Lab is a host-gate companion experiment, not a claim that ARC implements cryptographic review receipts. It is small enough to run with Node.js alone.

One receipt, five decisions

Our example gives an authorized reviewer a temporary Ed25519 key pair and pins its public key in a tiny verifier. The receipt includes a run identity, exact candidate SHA, policy ID, and verdict. Its signature covers all four fields using an explicitly ordered serialization.

We test the following behavior:

Case Change Expected decision
Matching reviewer receipt Candidate A, policy v1, trusted key EVIDENCE_MATCHED_ONLY
Stale candidate Present A's valid receipt while checking candidate B DENY_STALE_CANDIDATE
Changed policy Present v1's valid receipt under policy v2 DENY_STALE_POLICY
Different signer Implementer signs the same fields with another key DENY_UNTRUSTED_SIGNER
Missing receipt No verifiable input PENDING_MISSING_RECEIPT

The success condition is intentionally called EVIDENCE_MATCHED_ONLY. A valid receipt is evidence for a particular subject. It is not by itself permission to merge, deploy, or perform external actions.

Run the fixture

Save the following as review-receipt-fixture.mjs and run it with Node.js 22 or later:

node review-receipt-fixture.mjs
Enter fullscreen mode Exit fullscreen mode
import assert from 'node:assert/strict';
import { generateKeyPairSync, sign, verify } from 'node:crypto';

// Synthetic identifiers; no real PR, credentials, agent or network request.
const shaA = 'a'.repeat(40);
const shaB = 'b'.repeat(40);
const runId = 'lab-run-001';
const policyV1 = 'review-policy-v1';
const policyV2 = 'review-policy-v2';

const reviewer = generateKeyPairSync('ed25519');
const implementer = generateKeyPairSync('ed25519');

// Deliberately fixed serialization for this small educational fixture.
// A production protocol needs a specified canonical encoding and key lifecycle.
function signedBytes(r) {
  return Buffer.from(JSON.stringify([
    1, r.runId, r.candidateSha, r.policyId, r.verdict
  ]), 'utf8');
}
function issue(receipt, privateKey) {
  return {
    receipt,
    signature: sign(null, signedBytes(receipt), privateKey),
  };
}
function inspect({ signedReceipt, currentSha, currentPolicy, pinnedReviewerKey }) {
  if (!signedReceipt?.receipt || !Buffer.isBuffer(signedReceipt.signature)) {
    return 'PENDING_MISSING_RECEIPT';
  }
  const { receipt, signature } = signedReceipt;
  if (!verify(null, signedBytes(receipt), pinnedReviewerKey, signature)) {
    return 'DENY_UNTRUSTED_SIGNER';
  }
  if (receipt.candidateSha !== currentSha) return 'DENY_STALE_CANDIDATE';
  if (receipt.policyId !== currentPolicy) return 'DENY_STALE_POLICY';
  if (receipt.verdict !== 'PASS') return 'DENY_REVIEW_NOT_PASS';
  return 'EVIDENCE_MATCHED_ONLY';
}
const receipt = {
  runId,
  candidateSha: shaA,
  policyId: policyV1,
  verdict: 'PASS',
};
const signedByReviewer = issue(receipt, reviewer.privateKey);
const signedByImplementer = issue(receipt, implementer.privateKey);
const cases = [
  ['matching reviewer receipt', signedByReviewer, shaA, policyV1, 'EVIDENCE_MATCHED_ONLY'],
  ['valid receipt replayed on newer commit', signedByReviewer, shaB, policyV1, 'DENY_STALE_CANDIDATE'],
  ['valid receipt after policy change', signedByReviewer, shaA, policyV2, 'DENY_STALE_POLICY'],
  ['implementer uses its own signing key', signedByImplementer, shaA, policyV1, 'DENY_UNTRUSTED_SIGNER'],
  ['receipt missing', null, shaA, policyV1, 'PENDING_MISSING_RECEIPT'],
];
for (const [label, input, currentSha, currentPolicy, expected] of cases) {
  const actual = inspect({
    signedReceipt: input,
    currentSha,
    currentPolicy,
    pinnedReviewerKey: reviewer.publicKey,
  });
  assert.equal(actual, expected, label);
  console.log(`PASS: ${label} -> ${actual}`);
}
console.log('5/5 fixture assertions passed. No merge or runtime execution was authorized.');
Enter fullscreen mode Exit fullscreen mode

We executed this fixture on November 17, 2026, using Node.js v22.16.0. The five local assertions passed:

PASS: matching reviewer receipt -> EVIDENCE_MATCHED_ONLY
PASS: valid receipt replayed on newer commit -> DENY_STALE_CANDIDATE
PASS: valid receipt after policy change -> DENY_STALE_POLICY
PASS: implementer uses its own signing key -> DENY_UNTRUSTED_SIGNER
PASS: receipt missing -> PENDING_MISSING_RECEIPT
5/5 fixture assertions passed. No merge or runtime execution was authorized.
Enter fullscreen mode Exit fullscreen mode

The negative cases are successful tests: the simulated gate rejects exactly what it should reject.

Why the key boundary matters

The important control here is not the claimedProducer text inside a JSON receipt. The verifier compares the signature against a pinned reviewer public key. A receipt signed with the implementer's separate key does not satisfy that trust condition.

In this lab both key pairs are created by one local process. That makes the exercise reproducible, but it does not demonstrate operational independence. A production system would need to establish who can use the reviewer's private key, how the key is registered or revoked, and whether the review actually occurred. Signature verification alone cannot prove those facts.

The serialized byte format in this toy is also fixed only for the demonstration. A real protocol must specify canonical encoding, key IDs, algorithm choices, policy transitions, and how the verifying authority obtains the trusted inputs.

Where an actual merge gate goes further

To rely on a review receipt, an integration gate should establish at least:

  1. Subject: the receipt covers the exact candidate revision and relevant base or merge result.
  2. Authority: the reviewer's identity and key were authorized for that task when the receipt was issued and remain acceptable under the current policy.
  3. Coverage: all required review and CI obligations are satisfied; missing evidence remains missing.
  4. Currentness: a new commit, policy revision, or invalidated assumption triggers the applicable re-verification rather than inheriting an old PASS.
  5. Acceptance: the protected merge and any production verification remain their own gates.

A reused signature is not inherently malicious. Reusing it for a different subject or authority state is the problem. The host must detect that difference.

One more failure to propose

Our five cases do not yet test concurrent admission of the same receipt or a lost response after a side-effecting action. Those require durable identity, deduplication, and external-effect reconciliation beyond a local signature check. We would treat them as separate follow-on tests.

What is the smallest fixture that would falsify your own review gate? A stale candidate, a forged producer, a changed policy, or something else?

We welcome public, synthetic counterexamples in ARC's GitHub repository. Please omit credentials, customer data, and private system paths.

Scope disclosure: the results above come from a self-contained local fixture authored by Netsujo. They are not third-party validation, a test of live reviewer key custody, or proof of ARC runtime enforcement.

Top comments (0)