A human-in-the-loop step only works if the reviewer has a real decision to make and can see the evidence needed to make it. It stops working the moment the queue can be cleared by approving everything. Most approval steps I review fail on both counts, and the audit log then faithfully records humans "approving" things nobody read. If you can't prove your reviewer would catch a bad item, the step is decoration.
Why does approval fatigue set in so fast?
Volume and sameness. An agent drafts two hundred items a day and ninety-five percent of them are fine, so the reviewer learns within a week that approving is the fast path. Four seconds per item. Batching before month end. The step still exists on the architecture diagram, and it still fails.
We learned this on ourselves. When we tried running parts of our own operations through agent "departments", one internal agent prepared vendor follow-up emails for a person to approve. After a week the approval rate was 100 percent. So we seeded five deliberately wrong drafts (a wrong amount, a wrong recipient, a duplicate of one already sent, and one referencing a contract that didn't exist). All five went out approved. The reviewer was diligent by nature; the interface had trained them that nothing needed reading.
Seeded failures are now the first thing I ask any team to run. If the catch rate on planted errors is low, the human is near the loop rather than in it.
| Queue design | What the reviewer faces | What we saw happen |
|---|---|---|
| Everything routed to a person | 200 items daily, most of them fine | Four seconds each, 100 percent approved |
| Low-risk auto-approved and sampled | Roughly 20 hard items with diffs | Items get read properly |
| Seeded failures mixed in | Same queue, planted errors inside | Catch rate becomes a real number |
What should the approver actually see?
The approver should see three things: the diff of what changed, the raw evidence the agent used, and one line explaining why this item needed a human at all. Anything else on the screen is cost. A reviewer who can check a claim in twenty seconds will check it.
- The diff instead of the document. If the agent changed three fields on a record, show those fields with old and new values.
- The evidence: the retrieved records, the tool results, the policy snippet it matched, the customer's original message. The evidence itself, never a summary of it.
- A risk grade, so the reviewer spends attention where it matters.
Then stop showing them the easy stuff. Auto-approve the low-risk slice with random sampling for QA, and route only the top-risk slice to a person. A reviewer looking at twenty hard items reads them. A reviewer looking at two hundred mostly-fine items reads none.
How do you make approval a real decision?
Approval becomes a real decision when approving costs the reviewer something small: a reason code, a second button, a cap on volume, a randomised order. Each mechanic breaks the reflex that turns a queue into a rhythm. The number that tells you whether it worked is the disagreement rate.
- Require a reason on approval for high-risk items. A dropdown with four options is enough; the act of choosing breaks the reflex.
- Separate "looks fine" from "verified." Two buttons, with different downstream permissions.
- Cap approvals per hour, and randomise order so the reviewer can't pattern-match on batches.
- Measure the disagreement rate. If your reviewer rejects or edits zero percent of items over a month, the step is either unnecessary or broken. Either way, act on it.
- Keep running seeded failures, quietly, and put the catch rate on the dashboard next to the agent's own accuracy.
When should you take the human out?
Take the human out when sampling shows the low-risk slice has been clean for a sustained period, and only for reversible actions. Keep a person on anything irreversible: payments, deletions, outbound messages to customers, anything with legal weight. The reviewer then moves up to the harder slice.
That split (automate the reversible, gate the irreversible) is how we design agent workflows at Shanti Infosoft, and it's the first thing we redraw during an AI consulting engagement on a stalled agent rollout. It keeps the approval queue short enough that people read it, which is the only property that matters. If you want to pressure-test your own queue design, a call and a whiteboard usually gets there.
If you planted five wrong items in your approval queue tomorrow, how many would get caught?
Sagar Jain is the technical co-founder of Shanti Infosoft, which has built software for 700+ companies and now designs the approval layers that sit around their agents.
Top comments (0)