Part 1 ended with a confession: you'll never test your way to 100% correctness. For critical workflows the final safety net is a human. But "human in the loop" has a dirty secret. Badly designed, it's theater.
The rubber-stamp problem
Route every agent action to a human for approval and watch what happens. Week one, careful reviews. By week three approval fatigue sets in and people click approve at the speed of thought. You've paid a human salary to become an Enter key, and the loop provides zero actual oversight.
The goal isn't humans reviewing everything. It's humans reviewing exactly the things that need judgment, at a volume they can sustain.
Designing oversight people can actually do
Filter before you route. Confidence thresholds decide who sees what: high-confidence routine actions execute directly with logging, low-stakes medium-confidence ones get sampled, and genuinely uncertain or high-impact ones go to a human. Careful UX design here keeps human load limited. That's the whole game.
Show diffs, not documents. Nobody re-reads a generated contract. Show what changed from the template, highlight which fields the model filled, and make deviations impossible to miss. Reviewing a diff takes seconds; reviewing a page takes minutes, and minutes don't survive week three.
Sample with LLMs, confirm with humans. An LLM auditor grades a random slice of automated outputs daily. Scores trending down? Raise the human-review rate automatically. This is how you catch silent drift (Part 4's scariest failure mode) without hiring a review department.
┌─────────────────────────┐
│ Agent output │
└────────────┬────────────┘
▼
confidence + impact triage
┌──────────────────┼──────────────────────┐
▼ ▼ ▼
high conf, medium stakes low conf or high stakes
low stakes LLM-sampled audit human review, diff-based UI
auto-execute trend monitoring │ like / unlike
(logged) │ trend drops ▼
└──────────────► feedback loop back into evals
Close the loop. Every like, unlike, and human correction is labeled data. Feed it back into your eval suite from Part 2. Your reviewers aren't just gatekeeping; they're continuously building your regression tests.
One thing to stop doing: 100% human review of everything. It feels safe, produces rubber stamps, and hides real failures inside noise. Oversight is a budget. Spend it where judgment matters.
This closes the series arc: constrain the workflow, buy only the intelligence you need, secure the credentials, engineer for failure, and put humans where humans add value. None of it requires a smarter model. All of it requires deliberate design.
Your action
Pick one approval queue in your system. Measure what fraction gets approved, how fast, by whom. If approvals take under five seconds each, you don't have oversight; you have rubber stamping. Redesign it with filtering and diffs this month.
Top comments (1)
The approval fatigue framing is exactly right - I've seen a team rubber-stamping within a week of going live, not three. The diff-based UI point is underappreciated. We switched to showing changed fields only and review quality jumped noticeably. What's the recalibration cadence for the confidence thresholds - ad hoc when the human-review rate creeps up, or something scheduled?