DEV Community

asagiman
asagiman

Posted on Edited on

How to catch out-of-scope coding-agent tasks before execution

#agents #opensource #javascript #testing

Coding agents work across files, roles, and review stages.
Here's a small failure case worth checking before handing a task to an agent:

  • An implementer is allowed to change files under src/**.
  • A separate read-only reviewer checks the proposed work.
  • The next task requests secrets/production.txt.

The task conflicts with the declared write scope. The useful response is a deterministic preflight failure, before an agent or tool attempts the write.

I maintain Agent Role Contracts (ARC), a small MIT-licensed Node.js library and CLI for checking those declarations. I reproduced the example below on macOS with Node.js 24.11.1 and the pinned v1.0.0 source on October 19, 2026.

Run the three-minute check

Requirements: Git and Node.js 22.5 or newer.

git clone --depth 1 --branch v1.0.0 https://github.com/suirindo/agent-role-contracts.git
cd agent-role-contracts
npm run demo --silent
Enter fullscreen mode Exit fullscreen mode

The demo reads the packaged example declarations and exercises the actual checker. Its relevant output is:

1. Normal task
   Scope: src/example.mjs; allowed: src/**
   PASS: declarations are consistent.
   Implementer: implementer (write_scoped)
   Reviewer: reviewer (read_only; separate declared role)

2. Request an out-of-scope change
   Scope: secrets/production.txt; allowed: src/**
   FAIL (expected): TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY

3. Repair the task scope
   Scope: secrets/production.txt -> src/example.mjs
   PASS: declarations are consistent again.

Demo complete: PASS -> FAIL (expected) -> PASS.
Checks declarations only; execution is NOT authorized.
Enter fullscreen mode Exit fullscreen mode

The example uses fictional paths. It opens no target task file, invokes no model, and performs no agent action. The demo returns exit 0 when it observes the expected PASS → FAIL → PASS sequence.

See the actual CLI rejection

Run the invalid task directly:

node bin/agent-role-contracts.mjs explain \
  --bundle examples/starter-bundle.json \
  --task examples/starter-task-outside-scope.json \
  --format text
Enter fullscreen mode Exit fullscreen mode

This is an expected failure. The CLI reports:

FAIL: explain (declarations only; execution NOT authorized)
Accountable: implementer
Implementers: implementer
Reviewers: reviewer
Human approval required: true
TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY task/inputs/scope: Task scope is outside declared write scopes for role implementer: secrets/production.txt
Enter fullscreen mode Exit fullscreen mode

Exit status: 1. The shortened output above omits the role-capability detail lines that the CLI also prints.

Try the allowed task:

node bin/agent-role-contracts.mjs explain \
  --bundle examples/starter-bundle.json \
  --task examples/starter-task.json \
  --format text
Enter fullscreen mode Exit fullscreen mode

The checker reports PASS: explain and exits 0. A CLI/input error exits 2. When integrating this preflight into CI or task dispatch, treat exit 1 or 2 as a stop condition and use your runtime's existing permissions and human-approval gates after a valid declaration check.

What ARC checks

The shipped declaration checks cover several common workflow inconsistencies:

Supplied declarations Example of a detectable mismatch
Implementer write authority + task scope Requested path exceeds src/**
Implementer + reviewer roles Same declared role assigned to both
Reviewed task/action + binding Task or action changed after its declared review binding
Handoff and lifecycle declarations Declared task/subject relationships become stale

ARC works offline with JSON inputs. The example is model-agnostic: it can be evaluated alongside a Codex, Claude Code, or other coding-agent workflow, because the check itself invokes no LLM. Integration into an agent runtime remains the responsibility of that workflow's maintainer.

What a PASS establishes

A PASS establishes internal consistency of the supplied declarations under the checks executed. Actual file permissions, actor identity, reviewer independence at runtime, approval authenticity, side effects, and resulting code quality require checks from IAM, sandbox, CI, and runtime/evidence systems.

For instance, an independent reviewer role ID in JSON expresses the intended separation; the execution environment must authenticate the real reviewer. A write-scope declaration needs an actual filesystem/tool boundary to enforce it.

Test one of your workflows

The recently merged external evaluation kit adds a source-only helper pinned to the immutable v1.0.0 revision. It supports the expected scope contradiction, its repair, and a second distinct declaration check with the same bundle. The last step is useful for examining reusability; proof of a second real task still requires an actual second operation or independently linkable CI/commit evidence.

If you already use CI, a sandbox, a reviewer, or another preflight, I'd especially value a comparison on the same deliberate contradiction. A negative result — your current controls catch everything ARC catches — is useful feedback too.

The first external workflow feedback report, Burs-IA WORKNET, described a PASS → rejected scope mismatch → repaired PASS using ARC 0.5.0. That report is operator-provided; separate actual second-task reuse and independent reproduction remain open evaluation goals.

From the demo to your own task — November 7, 2026 update

Try the task you actually intend to hand off

Once the demo works, choose one small, non-sensitive code change you are already allowed to make. The useful question is whether checking its declaration adds anything to your existing workflow.

Start with the standalone consumer setup. This experimental kit pins ARC core 1.0.0. Record the kit's source commit separately; copying a newer kit does not upgrade that pinned dependency.

Then use the first-task walkthrough to create a new .arc/task-first.json. Keep the supplied bundle unchanged. Replace the example ID, objective, scope, and acceptance criteria with your intended task. The starter allows src/**; if your actual task needs another scope, use your repository's existing authority-review process.

From the consumer root, check that exact file:

npx --no-install agent-role-contracts explain --bundle .arc/bundle.json --task .arc/task-first.json --format text
Enter fullscreen mode Exit fullscreen mode

In PowerShell, use npx.cmd and inspect $LASTEXITCODE immediately afterward. In a POSIX shell, inspect echo $?.

Use one deliberate contradiction to see what the check covers:

The expected results below assume that all other declarations are valid and remain unchanged.

Declaration Expected result
Your accurate task scope under the starter's src/** authority PASS, exit 0
Temporarily change only the task scope to secrets/private.json TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY, exit 1
Restore the accurate task scope PASS, exit 0

On November 7, 2026, we reproduced this exercise on macOS with Node v24.22.1, ARC core 1.0.0, and kit commit 873b4104a95e443ba22367cf8378deb7dcbe6cfd. The synthetic src/greeting.mjs declaration produced exit 0, secrets/private.json produced exit 1, and the restored scope produced exit 0. The authority bundle's SHA-256 stayed unchanged.

The temporary scope edit is a declaration test, not an instruction to access that path. Do not widen the bundle to make the deliberate failure pass.

Creating task-first.json does not change what the existing npm run arc:check, hook, or CI example reads: those examples still use .arc/task.json. Keep the filename explicit while trying your own task.

A PASS concerns the declarations supplied to the checker. It does not show that a file exists, an agent stayed within scope, tests passed, a reviewer acted independently, or the work was completed. Perform the real work through your existing approval, runtime, testing, and review controls.

Report your first attempt, including a command that could not start. State whether you checked the supplied example or your own task, and include the command, observed result, and versions when known. Remove secrets and private project details before submitting.

If you later use ARC on a different real task, add that evidence to workflow feedback. These three declaration checks are one exercise, not three completed tasks. If your existing controls already catch the same mismatch, that is useful feedback too.

If the project is useful to you, you can star the repository. A star or a positive review is not required to try it or report a problem.

Disclosure for the update: ARC is maintained by Netsujo. AI assistance was used to prepare this text. The exercise uses synthetic declarations; it is not evidence of a production incident or external adoption.

日本語：次にエージェントへ渡す、自分のタスクで試す

デモが動いたら、すでに実施権限のある、小さなコード変更を1件選んでください。確認したいのは、そのタスク宣言をチェックすることで、いまの開発手順に役立つ情報が増えるかです。機密情報を含む作業は、この試用へ持ち込まないでください。

まずstandalone consumerの準備を行います。この実験用キットが使うARC coreは 1.0.0固定です。キットの取得元commitは、coreのバージョンとは別に記録します。新しいキットをコピーしても、この固定された依存は更新されません。

次に最初のタスクを書く手順で、新しい .arc/task-first.json を作ります。同梱のbundleは変えず、例のID・目的・scope・受入条件を、予定している作業に合わせます。このstarterの許可範囲は src/** です。実際のタスクに別の範囲が必要なら、リポジトリで決めている権限レビューを使ってください。

consumerのルートで、作ったファイルを明示して実行します。

npx --no-install agent-role-contracts explain --bundle .arc/bundle.json --task .arc/task-first.json --format text
Enter fullscreen mode Exit fullscreen mode

PowerShellでは npx.cmd を使い、直後に $LASTEXITCODE を確認します。POSIX shellでは直後の echo $? で確認できます。

scopeを一度だけ意図的に外し、チェックの範囲を確かめます。

以下の期待結果は、ほかの宣言が妥当で、変更されていない場合のものです。

宣言 期待する結果
starterの src/** 内にある、正確なtask scope PASS、終了コード0
taskのscopeだけを一時的に secrets/private.json へ変更 TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY、終了コード1
正確なtask scopeへ戻す PASS、終了コード0

2026年11月7日、macOS・Node v24.22.1・ARC core 1.0.0・kit commit 873b4104a95e443ba22367cf8378deb7dcbe6cfd で、この練習を再現しました。合成した src/greeting.mjs の宣言は終了コード0、secrets/private.json は1、元のscopeへ戻すと0でした。権限bundleのSHA-256は変わっていません。

一時的なscope変更は宣言の検査であり、そのパスへアクセスする指示ではありません。意図的な失敗を通すために、bundleの権限を広げないでください。

task-first.json を作っても、既存の npm run arc:check・hook・CI例が読むファイルは変わりません。それらは .arc/task.json を使います。自分のタスクを試す間は、上のように対象ファイルを明示します。

PASSが示すのは、チェックへ渡した宣言についての結果です。ファイルの存在、実際の変更範囲、テスト成功、独立したレビュー、作業完了を証明しません。実作業は、既存の承認・実行環境・試験・レビューの手順で進めます。

実行できなかった場合も、初回の試用結果を報告できます。同梱サンプルと自分のタスクのどちらを確認したか、実行したコマンド、観測した結果、分かる範囲のバージョンを記してください。秘密情報や非公開プロジェクトの詳細は除いてください。

後日、別の実タスクで使った場合は、workflow feedbackへ証拠を追加できます。今回の3回の宣言チェックは1つの練習であり、3件の仕事を完了した実績には数えません。既存の制御で同じ矛盾を検出できる場合も、有用なフィードバックです。

役立つと感じたら、GitHubリポジトリへのスターで応援いただけます。試用や問題報告に、スターや肯定的な評価は必要ありません。

更新時の開示：ARCはNetsujoが開発しています。この原稿の準備にはAIを使用しました。例は合成した宣言であり、本番事故や外部導入の実績を示すものではありません。

Here are the relevant links:

One concrete question: What mismatch between a coding agent's task, its declared authority, and its review handoff would you most want checked before dispatch? I can help map one small, non-sensitive workflow and compare the result against your existing controls.

Disclosure: I maintain ARC at Netsujo Inc. The example above is a synthetic preflight demonstration; external production adoption and second-task reuse are tracked separately.

Top comments (2)

Collapse
 
abhinavs1920 profile image
Abhinav Sharma •

A deterministic preflight failure beats cleaning up after the fact, catching the scope violation before any write attempt keeps the audit trail clean. Declaring the writable paths upfront also makes the reviewer step much simpler.

Collapse
 
suirindo profile image
asagiman •

Thanks, Abhinav. That's exactly the boundary we're trying to make explicit: reject an inconsistent task before dispatch, with a clear reason the reviewer can inspect.

ARC checks the declared scope and review bindings, while runtime permissions enforce the actual boundary.

Have you encountered cases where a coding agent's task scope changed after review or during handoff? I'd be interested in comparing a concrete example with ARC's current checks.