DEV Community

asagiman
asagiman

Posted on

Your AI Agent Passed Review. Then the Task Changed.

#ai #agents #testing #opensource

Here is a failure mode that is easy to overlook in an otherwise well-organized AI coding workflow.

An agent proposes a change. A reviewer approves it. Then someone edits the task or action before execution. The review record still says approved.

What, exactly, was approved?

A checkbox alone cannot answer that question. The useful link is between the review decision and the exact subject it covered.

A concrete before-and-after

Consider this fictional action declaration:

{
  "kind": "software-change",
  "parameters": {
    "output": "src/example.mjs",
    "change": "Handle empty input",
    "test_plan": "Check empty and populated inputs"
  }
}
Enter fullscreen mode Exit fullscreen mode

The reviewer sees the task and approves the declared action.

Afterward, the action changes in a meaningful way:

- "change": "Handle empty input"
+ "change": "Handle malformed input"
Enter fullscreen mode Exit fullscreen mode

The new requirement may be reasonable. It still deserves a new review decision, because the old approval describes a different subject.

Reproduce the mismatch

The public Agent Role Contracts repository includes a G1 task/action binding demo. It is an offline declaration check: no model call, tool execution, or real approval occurs.

With Git, Node.js 22.5+, and npm:

git clone https://github.com/suirindo/agent-role-contracts.git
cd agent-role-contracts
npm run demo:binding
Enter fullscreen mode Exit fullscreen mode

The checked-in demo source runs three synthetic scenarios. Each starts with a review and approval declaration bound to the current task/action digest. It then changes one meaningful input while retaining the old binding.

The significant outcomes reported by that script are:

Software-change example:
  PASS: routed review and accountable approval match the subject.
  FAIL (expected): G1_SUBJECT_MISMATCH after changing action parameter change.

Data-cleaning example:
  PASS: routed review and accountable approval match the subject.
  FAIL (expected): G1_SUBJECT_MISMATCH after changing action parameter deduplicate.

Support-drafting example:
  PASS: routed review and accountable approval match the subject.
  FAIL (expected): G1_SUBJECT_MISMATCH after changing task input test_plan.
Enter fullscreen mode Exit fullscreen mode

The labels above summarize the three cases; the PASS/FAIL messages and diagnostic are taken from the published script. This article does not claim a new independent execution of the demo.

The demo treats all three mismatches as expected failures. When all six expected binding outcomes are observed, the overall demo succeeds. For a production dispatcher, the individual stale-binding result is the important stop signal.

What is being bound?

ARC's G1 checker derives a deterministic subject digest from the supplied bundle, task, and action declarations. The binding contains that subject digest alongside declared review and approval decisions.

A meaningful change to those inputs changes the current digest. Presenting an older binding to the new subject produces G1_SUBJECT_MISMATCH.

This gives a workflow a precise question to ask before dispatch:

Does this review record refer to the task, policy, and action we are about to use?

The digest is a comparison mechanism for supplied declarations. It is not a cryptographic signature from the reviewer, proof of the reviewed source-code bytes, evidence of real independence, or permission to execute. A deployment system still needs authenticated approval, actual access control, exact artifact verification, and current policy checks.

Where to put the check

A narrow integration can follow this sequence:

  1. Choose the canonical policy and task definitions from a trusted source.
  2. Capture the subject being reviewed and record the review/approval decision against it.
  3. Immediately before the consequential action, reconstruct the current subject and reject any stale binding.
  4. Authenticate the actor, enforce real permissions, verify the exact artifact and approval, and then use your normal execution controls.

The third step is especially valuable when another agent takes over, the task description evolves, or an approval sits in a queue while code keeps changing.

What I want to learn from other teams

I maintain ARC at Netsujo, and I am trying to measure its incremental value against existing controls rather than assume every team needs another layer.

How does your team prevent an AI agent from acting on a review that belongs to an older task or revision?

Do you bind approvals to a commit SHA, a pull-request head, a structured task digest, a deployment manifest, or something else? What changes can happen after approval in your current workflow?

If your CI system already solves this cleanly, I would particularly like to hear how.

Disclosure: I maintain ARC. This example uses its published fictional fixtures. A PASS confirms declaration consistency under the invoked check; execution authorization and real review identity stay with the integrating runtime. G1 source · Integration guide.

Top comments (0)