The diagram vs. the Tuesday afternoon
On the architecture diagram, human-in-the-loop looks great. The model makes a decision, a box labelled "human review" sits between the model and production, and everyone signs off on the design doc feeling safer.
Then the system ships. The reviewer is a real person with a real queue, and on a normal Tuesday afternoon that queue holds 340 items. Most of them are obviously fine. By item 40, they are reading the first line. By item 120, they are clicking "approve" on rhythm.
The box on the diagram is still there. The review isn't.
This is the gap: we treat the human as a component with infinite attention, when attention is the scarcest resource in the whole pipeline. Below are the failure modes I see most often, with illustrative examples (composites, not specific companies).
Failure mode 1: Everything goes to the human
The simplest design routes every model output to review. It feels conservative. In practice it is the least safe option, because it trains the reviewer that approval is the expected answer.
Illustrative example: an invoice-classification model is right 97% of the time. The reviewer sees 300 invoices a day, so roughly 9 are wrong. Those 9 are buried among 291 correct ones that look identical at a glance. After two weeks, the reviewer's override rate drops to near zero. Not because the model improved, but because the human stopped looking.
The fix is routing. Only escalate when the model is uncertain, or when the cost of being wrong is high:
def route(decision, confidence, impact):
# High-impact actions always get a human, whatever the confidence
if impact == "high":
return "human_review"
# Low confidence gets a human
if confidence < CONFIDENCE_THRESHOLD:
return "human_review"
# Everything else is auto-approved, but sampled for audit
if random.random() < AUDIT_SAMPLE_RATE:
return "audit_queue"
return "auto_approve"
Two details matter here. First, confidence alone is not enough: a model can be confidently wrong, and raw scores from many models are poorly calibrated. Check calibration on held-out data before you trust a threshold, and pair confidence with an impact rule. Second, the audit sample on auto-approved items is how you find out whether your threshold is still right.
Failure mode 2: The reviewer has to reconstruct the reasoning
The second failure is a review screen that shows the model's answer and nothing else. The reviewer now has two jobs: redo the task from scratch, or trust the answer. Under time pressure, they trust the answer.
Illustrative example: a loan-document extraction tool flags "annual income: 84,000". To check it, the reviewer opens a 22-page PDF and searches manually. They do it carefully twice, then stop.
A good review screen shows the exact inputs the model relied on: the highlighted span in the source document, the fields that drove the score, the rule that triggered the escalation. Verification becomes a glance instead of an investigation. If you can't surface what the model used, that is worth fixing before you add more reviewers.
Failure mode 3: Overriding is harder than approving
If "approve" is one click and "reject" opens a form with six required fields, you have designed a system that approves things.
The override path should be as cheap as the approve path: one action to correct, an optional short reason, and the corrected value captured automatically. Every override should be logged with the original output, the correction, who made it and when.
That log is not paperwork. It is the most useful dataset you have, because it tells you exactly where the model fails and whether your threshold is set in the right place.
Failure mode 4: Nobody measures the reviewers
If you don't track review behavior, you won't notice when the loop goes quiet. A few signals are worth watching:
- Time spent per review (a sudden drop is a warning sign)
- Override rate over time (near zero is suspicious, not reassuring)
- Errors found in the audit sample that a reviewer had approved
These tell you whether the human in the loop is still doing anything.
The reframe
Human-in-the-loop is really an attention-allocation problem with a model in front of it. The model's job is to decide what is safe to skip. The interface's job is to make the remaining decisions fast and easy to get right. The log's job is to show you when either of those stops working.
We think about this a lot at Cyclopt, because every AI action that touches a codebase raises the same question: which ones need a person, and how do you make that person's review count?
So here's my question for you: what is the override rate in your human review queue, and do you actually know what it is?
Top comments (2)
Really sharp analysis. The Tuesday-afternoon queue (reading the first line by item 40, approving on rhythm by item 120) is exactly how review dies in practice, and framing it as an attention-allocation problem is the right call.
The line I'd push a bit further is "near zero is suspicious", because once you add the routing from failure mode 1 it becomes ambiguous. The queue now only holds uncertain or high-impact items, so a healthy override rate should be well above zero. When it sits near zero, either the threshold is sending obvious cases to humans, or the humans stopped looking. The rate alone can't tell you which.
Airport security has dealt with this for years: X-ray software projects fake threats onto real bags (threat image projection), and screeners are scored on how many of those they catch. The same trick works in a review queue. Slip in a few items you know are wrong and measure the catch rate. That's the number that tells you the review still happens.
Failure mode 3 has a mirror image: when the action can't be undone, the approve path is the one that should cost something. On an internal tool that writes to one of our ad accounts, archiving is a dry run by default, only accepts items that are already paused, and makes the operator retype how many entities the plan touches. If you expected 3 and the plan says 40, the mismatch does the review for you.
Have you seen teams seed errors on purpose, or does it feel too much like testing the reviewers?
That's a really good distinction. The override rate on its own doesn't tell you much if the queue has already filtered out the obvious cases. Seeding known errors and measuring whether reviewers catch them sounds like a much better signal, provided it's done transparently and without turning it into a performance trap for the people reviewing.
I also like the idea of making the approval path more deliberate when an action is hard to undo. Dry runs and explicit confirmation of the expected impact are simple, but they force people to actually look at what they're approving. I haven't seen teams deliberately seed errors in practice, but I think it's worth exploring. The key is testing whether the process works, not trying to catch people out.