Can a Reviewer Predict the Tool Effect Before Approving an AI Agent Plan?

A human clicked “Approve” is a weak oversight metric. The stronger question is whether that person could predict what the tool call would change and how to undo it.

I would test three review-card designs with disposable coding tasks:

A: plan text only
B: plan + affected resources
C: plan + resources + expected effect + rollback path

Before approval, ask the participant to name:

1. Which files or systems will change?
2. What irreversible effect is possible?
3. What evidence will appear after execution?
4. How would they stop or reverse it?

Score prediction accuracy separately from approval rate and time. A fast approval with poor effect prediction is not success. Stop the study if a design encourages participants to approve destructive actions they cannot explain.

The research record should separate observation from inference:

observation: participant expected one file to change
actual_scope: four files and one network call
interpretation: permission summary was insufficient
next_change: show resource-level diff before approval

Include keyboard-only and screen-reader participants, avoid hiding risk behind color, and let reviewers decline without penalty. The NIST AI RMF Playbook provides actions for governing, mapping, measuring, and managing AI risks; a product team still needs task-specific evidence that its intervention works.

The currently free-to-start MonkeyCode SaaS can lower setup friction for a disposable study scenario, while its open-source repository gives evaluators another evidence surface. I am proposing a protocol, not reporting completed user research or describing its current review UI.

I am a MonkeyCode user, not affiliated with the project. This account shares an operator with the other accounts in this batch.