DEV Community

Cover image for ARC Failure Lab #1: Two Green Tests. One Unauthorized File.
asagiman for Netsujo

Posted on

ARC Failure Lab #1: Two Green Tests. One Unauthorized File.

A coding agent finishes a task. Its unit tests pass. The pull request looks ordinary.

There is one more question to ask: Did it change only the files it was authorized to change?

Welcome to ARC Failure Lab. Each challenge starts with a small, reproducible boundary failure. We publish the expected results, show which part of the problem Agent Role Contracts (ARC) can check, and invite developers to propose a better counterexample.

For Challenge #1, no AI account or live agent is needed. The failure is small enough to run offline in a few minutes.

The experiment

Imagine an agent assigned to update one file:

  • Allowed: src/greet.mjs
  • Not allowed: config/release.json

We have two Git patches:

  • A: changes only src/greet.mjs.
  • B: makes the exact same allowed change and changes config/release.json.

Both patches pass the same unit tests. But only one fits the declared write authority.

Patch Unit tests ARC declaration check
A: allowed edit PASS PASS
B: allowed edit + extra config change PASS config/release.json: FAIL, TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY

There is no trick in the unit tests. A unit test can be perfectly correct while being silent about changes outside the functionality it covers.

Reproduce it

You'll need Git 2.32+ and Node.js 22.5+. The repository uses a pinned demonstration and does not require an npm install, credentials, API keys, or running an agent.

git clone https://github.com/suirindo/agent-role-contracts.git
git -C agent-role-contracts checkout --detach b42a308d5be59132310cfb782ed828745d7fd3bf
node agent-role-contracts/experiments/git-diff-demo/run.mjs
Enter fullscreen mode Exit fullscreen mode

The script creates temporary Git repositories and applies the two patches separately. It then runs the same tests and passes each changed path to the actual ARC CLI using a task declaration.

The important lines from the expected output are:

Case A: allowed patch
  Unit tests (node --test): PASS (exit 0)
  ARC explain src/greet.mjs: PASS (exit 0)

Case B: same patch + one out-of-scope edit
  Unit tests (node --test): PASS (exit 0)
  ARC explain config/release.json: FAIL (exit 1) TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY
  ARC explain src/greet.mjs: PASS (exit 0)

Demo complete: every observed result matched the expectation (demo exit 0).
Enter fullscreen mode Exit fullscreen mode

The overall demo exit code is 0 because the negative case failed as expected. The exit 1 line reports ARC's rejection of an out-of-scope path, not a failed execution of the demonstration.

Rerun evidence: The Netsujo maintainer reran the experiment on a clean macOS/M3 clone at the pinned commit on November 15, 2026. Both unit tests passed, and the declaration check rejected the extra file as expected. This is an internal reproduction, not a claim of independent third-party validation.

Full source, expected output and limitations · Japanese instructions

What this proves—and what it does not

Here is the important division of responsibility:

The demo's Git wrapper lists the changed paths. ARC checks each supplied path against the task's declared write authority. The unit tests check the behavior they were written to cover.

ARC does not independently inspect Git history, stop a write, authenticate which agent produced a change, establish reviewer independence, or prove the code is correct or safe. If the wrapper fails to supply a path, ARC will not check it. A real integration needs a trusted, complete change inventory and separate host-side enforcement where required.

This distinction matters because an automated reviewer can truthfully say "PASS" about the evidence it was asked to inspect while saying nothing about an unexamined boundary.

Your turn: bring one reproducible failure

What can go wrong in your own AI-agent workflow even when the ordinary checks are green?

Open the ARC Failure Lab #1 participation thread and share a small, public, synthetic case using this template:

  1. Boundary: What was the agent allowed to do?
  2. Deviation: What extra action, stale review receipt, missing provenance, or policy mismatch happened?
  3. Reproduction: What exact steps, code, versions and inputs are needed?
  4. Expected vs observed: What output or exit code did you actually see? Mark unrun ideas NOT_RUN.
  5. Responsibility: Which part could ARC check, and which part requires your host or acceptance gate?

Failures that ARC cannot detect are also useful. They help us explain exactly where another control is necessary.

Please do not publish customer data, credentials, production logs, private paths, or suspected exploitable vulnerabilities. Use the repository's private security reporting flow for vulnerabilities.

Join the lab

ARC is an open-source project developed by Netsujo. The goal of Failure Lab is to turn precise technical disagreements into small, repeatable checks—not to treat a declaration-level PASS as proof of complete runtime safety.

Top comments (0)