<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Netsujo</title>
    <description>The latest articles on DEV Community by Netsujo (netsujo).</description>
    <link>https://dev.to/netsujo</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Forganization%2Fprofile_image%2F16125%2F5e53f1ed-d657-437c-ad37-bb4a65dd4c33.jpg</url>
      <title>DEV Community: Netsujo</title>
      <link>https://dev.to/netsujo</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/netsujo"/>
    <language>en</language>
    <item>
      <title>ARC Failure Lab #1: Two Green Tests. One Unauthorized File.</title>
      <dc:creator>asagiman</dc:creator>
      <pubDate>Mon, 16 Nov 2026 10:19:12 +0000</pubDate>
      <link>https://dev.to/netsujo/arc-failure-lab-1-two-green-tests-one-unauthorized-file-50h5</link>
      <guid>https://dev.to/netsujo/arc-failure-lab-1-two-green-tests-one-unauthorized-file-50h5</guid>
      <description>&lt;p&gt;A coding agent finishes a task. Its unit tests pass. The pull request looks ordinary.&lt;/p&gt;

&lt;p&gt;There is one more question to ask: &lt;strong&gt;Did it change only the files it was authorized to change?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Welcome to &lt;strong&gt;ARC Failure Lab&lt;/strong&gt;. Each challenge starts with a small, reproducible boundary failure. We publish the expected results, show which part of the problem &lt;a href="https://github.com/suirindo/agent-role-contracts" rel="noopener noreferrer"&gt;Agent Role Contracts (ARC)&lt;/a&gt; can check, and invite developers to propose a better counterexample.&lt;/p&gt;

&lt;p&gt;For Challenge #1, no AI account or live agent is needed. The failure is small enough to run offline in a few minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The experiment
&lt;/h2&gt;

&lt;p&gt;Imagine an agent assigned to update one file:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Allowed:&lt;/strong&gt; &lt;code&gt;src/greet.mjs&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not allowed:&lt;/strong&gt; &lt;code&gt;config/release.json&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We have two Git patches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A:&lt;/strong&gt; changes only &lt;code&gt;src/greet.mjs&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;B:&lt;/strong&gt; makes the exact same allowed change &lt;strong&gt;and&lt;/strong&gt; changes &lt;code&gt;config/release.json&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both patches pass the same unit tests. But only one fits the declared write authority.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Patch&lt;/th&gt;
&lt;th&gt;Unit tests&lt;/th&gt;
&lt;th&gt;ARC declaration check&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A: allowed edit&lt;/td&gt;
&lt;td&gt;PASS&lt;/td&gt;
&lt;td&gt;PASS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B: allowed edit + extra config change&lt;/td&gt;
&lt;td&gt;PASS&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;config/release.json&lt;/code&gt;: FAIL, &lt;code&gt;TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There is no trick in the unit tests. A unit test can be perfectly correct while being &lt;strong&gt;silent&lt;/strong&gt; about changes outside the functionality it covers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproduce it
&lt;/h2&gt;

&lt;p&gt;You'll need Git 2.32+ and Node.js 22.5+. The repository uses a pinned demonstration and does not require an npm install, credentials, API keys, or running an agent.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/suirindo/agent-role-contracts.git
git -C agent-role-contracts checkout --detach b42a308d5be59132310cfb782ed828745d7fd3bf
node agent-role-contracts/experiments/git-diff-demo/run.mjs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script creates temporary Git repositories and applies the two patches separately. It then runs the same tests and passes each changed path to the actual ARC CLI using a task declaration.&lt;/p&gt;

&lt;p&gt;The important lines from the expected output are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Case A: allowed patch
  Unit tests (node --test): PASS (exit 0)
  ARC explain src/greet.mjs: PASS (exit 0)

Case B: same patch + one out-of-scope edit
  Unit tests (node --test): PASS (exit 0)
  ARC explain config/release.json: FAIL (exit 1) TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY
  ARC explain src/greet.mjs: PASS (exit 0)

Demo complete: every observed result matched the expectation (demo exit 0).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;strong&gt;overall demo exit code is 0&lt;/strong&gt; because the negative case failed &lt;em&gt;as expected&lt;/em&gt;. The &lt;code&gt;exit 1&lt;/code&gt; line reports ARC's rejection of an out-of-scope path, not a failed execution of the demonstration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rerun evidence:&lt;/strong&gt; The Netsujo maintainer reran the experiment on a clean macOS/M3 clone at the pinned commit on November 15, 2026. Both unit tests passed, and the declaration check rejected the extra file as expected. This is an internal reproduction, not a claim of independent third-party validation.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/suirindo/agent-role-contracts/blob/b42a308d5be59132310cfb782ed828745d7fd3bf/experiments/git-diff-demo/README.md" rel="noopener noreferrer"&gt;Full source, expected output and limitations&lt;/a&gt; · &lt;a href="https://github.com/suirindo/agent-role-contracts/blob/b42a308d5be59132310cfb782ed828745d7fd3bf/experiments/git-diff-demo/README.ja.md" rel="noopener noreferrer"&gt;Japanese instructions&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What this proves—and what it does not
&lt;/h2&gt;

&lt;p&gt;Here is the important division of responsibility:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The demo's Git wrapper&lt;/strong&gt; lists the changed paths. &lt;strong&gt;ARC&lt;/strong&gt; checks each supplied path against the task's declared write authority. &lt;strong&gt;The unit tests&lt;/strong&gt; check the behavior they were written to cover.&lt;/p&gt;

&lt;p&gt;ARC does &lt;strong&gt;not&lt;/strong&gt; independently inspect Git history, stop a write, authenticate which agent produced a change, establish reviewer independence, or prove the code is correct or safe. If the wrapper fails to supply a path, ARC will not check it. A real integration needs a trusted, complete change inventory and separate host-side enforcement where required.&lt;/p&gt;

&lt;p&gt;This distinction matters because an automated reviewer can truthfully say "PASS" about the evidence it was asked to inspect while saying nothing about an &lt;strong&gt;unexamined boundary&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your turn: bring one reproducible failure
&lt;/h2&gt;

&lt;p&gt;What can go wrong in your own AI-agent workflow even when the ordinary checks are green?&lt;/p&gt;

&lt;p&gt;Open the &lt;strong&gt;&lt;a href="https://github.com/suirindo/agent-role-contracts/issues/134" rel="noopener noreferrer"&gt;ARC Failure Lab #1 participation thread&lt;/a&gt;&lt;/strong&gt; and share a &lt;em&gt;small, public, synthetic&lt;/em&gt; case using this template:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Boundary:&lt;/strong&gt; What was the agent allowed to do?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deviation:&lt;/strong&gt; What extra action, stale review receipt, missing provenance, or policy mismatch happened?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reproduction:&lt;/strong&gt; What exact steps, code, versions and inputs are needed?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expected vs observed:&lt;/strong&gt; What output or exit code did you actually see? Mark unrun ideas &lt;code&gt;NOT_RUN&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Responsibility:&lt;/strong&gt; Which part could ARC check, and which part requires your host or acceptance gate?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Failures that ARC &lt;strong&gt;cannot&lt;/strong&gt; detect are also useful. They help us explain exactly where another control is necessary.&lt;/p&gt;

&lt;p&gt;Please do not publish customer data, credentials, production logs, private paths, or suspected exploitable vulnerabilities. Use the repository's &lt;a href="https://github.com/suirindo/agent-role-contracts/security/advisories/new" rel="noopener noreferrer"&gt;private security reporting flow&lt;/a&gt; for vulnerabilities.&lt;/p&gt;

&lt;h2&gt;
  
  
  Join the lab
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Run the fixture:&lt;/strong&gt; &lt;a href="https://github.com/suirindo/agent-role-contracts/tree/b42a308d5be59132310cfb782ed828745d7fd3bf/experiments/git-diff-demo" rel="noopener noreferrer"&gt;ARC Failure Lab #1 demo&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Submit a failure case or challenge the test:&lt;/strong&gt; &lt;a href="https://github.com/suirindo/agent-role-contracts/issues/134" rel="noopener noreferrer"&gt;GitHub Issue #134&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read or contribute to ARC:&lt;/strong&gt; &lt;a href="https://github.com/suirindo/agent-role-contracts" rel="noopener noreferrer"&gt;GitHub repository&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ARC is an open-source project developed by &lt;a href="https://netsujo.jp/" rel="noopener noreferrer"&gt;Netsujo&lt;/a&gt;. The goal of Failure Lab is to turn precise technical disagreements into small, repeatable checks—not to treat a declaration-level PASS as proof of complete runtime safety.&lt;/p&gt;

</description>
      <category>agents</category>
      <category>opensource</category>
      <category>testing</category>
      <category>javascript</category>
    </item>
  </channel>
</rss>
