DEV Community

imokokok
imokokok

Posted on

How to Test Whether an Agent’s Risk Check Actually Gates Signing

#ai #web3

A signed risk result can be valid and still be ignored by an agent's signing code.

That is the question behind this test: can we show that risk evidence and user authorization actually constrain the call that reaches the signer?

A concrete swap example

Our synthetic Base ETH → USDC fixture connects four separate steps:

  1. Insight supplies signed, advisory risk evidence.
  2. The agent's governor makes its policy decision.
  3. The principal signs an authorization for one exact EVM call.
  4. PriorSeal checks the authorization against the exported execution evidence.

A favorable risk verdict is not permission to sign. The final transaction must also match the principal's authorization, including its chain, target, calldata hash, value, nonce, and validity window.

Reproduce the test

From a PriorSeal checkout with Node.js 22 or later:

npm ci
npm run example:web3-agent-kit-base-swap-v2
Enter fullscreen mode Exit fullscreen mode

The verifier reports status: PASS when the baseline and all expected rejection cases behave as specified. The baseline returns OK.

Three useful failure cases are:

  • Changed governor decision → GOVERNOR_DECISION_MISMATCH
  • Policy branch that skips principal authorization → GOVERNOR_AUTHORIZATION_GATE_BYPASS
  • Changed transaction calldata → DRAFT_CALLDATA_HASH_MISMATCH

The fixture also rejects a revoked historical Insight key and tampered observed-execution evidence.

You can inspect the reproduction guide and the test record.

What this does not prove

This is a deterministic, no-broadcast fixture. No trade occurred and no funds moved. Its results establish how the exported evidence and verifier behave; they do not prove that a deployed agent reads every risk field or that every route to its signer is protected.

That runtime question needs source-path review and negative tests at the signing boundary. In a review of one pinned Web3 Agent Kit commit, we found direct signing paths outside its wallet wrapper. A check placed only in that wrapper would miss those paths. The call-site audit identifies the reviewed revision; it makes no claim about every later version.

To establish enforcement in an integration, I would want to see one required signing entry point, every direct signer route accounted for, and a test that fails whenever an unauthorized call reaches signing.

A signed field can be present in evidence without being used by the application. What negative test would you use to prove that your agent's risk decision actually changes whether a transaction can be signed?

Disclosure: This post was drafted with AI assistance. Its technical claims were checked against the linked fixture, and the verifier was rerun on September 25, 2026.

Top comments (0)