Goutham Nishkal Deepu EDA

Posted on Jul 10

Human-in-the-Loop Is Not a Governance Strategy

#ai #agents #architecture #machinelearning

Picture the human at the end of an agent pipeline.

The agent drafts an action. A modal pops up. Approve or reject. The human has seen forty of these today, they all looked fine, and the queue is backing up. They click approve. They will click approve on the next one too.

That is "human-in-the-loop." It is on almost every agentic-AI architecture diagram, usually drawn as a small person icon standing between the model and the irreversible action, labelled governance.

It is not governance. It is a rubber stamp with a person's name on it, and its main function is to move liability from the system to the human who clicked. I want to argue that most HITL, as shipped, makes systems look accountable while making them slightly less safe, and then describe what actually works instead, with code you can adapt.

No prior background assumed beyond having built or used an agent that takes actions.

The wrong question

The phrase "human-in-the-loop" quietly smuggles in a claim: that the presence of a human constitutes oversight. So teams optimise for presence. Is there a human in the loop? Yes? Ship it.

But presence is not the variable that matters. A human who is present but cannot realistically change the outcome is not a safeguard. They are decoration, and worse, they are load-bearing decoration, because everyone downstream now assumes the action was reviewed. The audit log says "approved by a human." The incident review will say it too.

The right question is not "is there a human in the loop?" It is:

Is the human positioned to actually change the outcome?

That reframe turns governance from a checkbox into a design problem. And design problems have structure. Three parts of it matter, and then a fourth thing that sits above all three.

1. Surface the decision, not the data

The rubber-stamp modal usually shows the human the action: here is the SQL we're about to run, here is the email we're about to send. Raw output. To approve it meaningfully, the human has to reconstruct the entire context the agent already had, in the two seconds before the queue guilt-trips them into clicking.

Real review surfaces the decision, not the payload. What is the agent trying to accomplish, what did it consider, what is it unsure about, and what happens if this is wrong?

Concretely, that is the difference between handing the human this:

DELETE FROM users WHERE last_login < '2024-01-01' AND status = 'trial';

and handing them this:

{
    "doing": "delete duplicate trial user",
    "affects": "1 record",
    "risk": "irreversible",
    "agent_confidence": "60%",
    "if_wrong": "restore from nightly backup",
}

The first asks the human to be a database reviewer at queue speed. The second asks them to make a judgment they are actually equipped to make: is a 60%-confident irreversible delete of one record worth a human's attention right now? That is a question a tired person can answer well. Parsing a DELETE filter for a subtle bug is not.

If your HITL layer shows the human the same thing your logs show, you have built an audit trail, not an oversight step. The payload is for the log. The decision summary is for the human.

2. Interrupt on what matters, not on everything

Here is the naive policy almost everyone ships. Anything that writes gets a human; anything that only reads runs free.

def naive_needs_human(action) -> bool:
    return action.kind != "read_file"

It feels safe. It is actually the source of the rubber stamp, because it interrupts constantly, and constant interruption is how you train a human to approve reflexively. A gate that fires on every write is, functionally, a gate that fires on nothing, because the human behind it has stopped reading.

A better policy interrupts on the properties that actually make an action dangerous: is it reversible, how large is the blast radius, and how confident is the agent in its own decision? And it recognises a third outcome beyond allow and review: some actions should never reach a human at all, because no realistic human review makes them safe.

from dataclasses import dataclass
from enum import Enum

class Decision(Enum):
    ALLOW  = "allow"    # run without asking
    REVIEW = "review"   # surface to a human
    BLOCK  = "block"    # never automate; architectural "no"

@dataclass
class Action:
    kind: str
    reversible: bool
    blast_radius: int      # records or people affected
    confidence: float      # agent's own confidence, 0..1
    intent: str = ""       # what the agent is trying to do
    undo: str = ""         # how it gets reversed, if it can be

def govern(a, conf_floor=0.85, radius_cap=50, hard_radius=500) -> Decision:
    # Top tier: irreversible AND huge -> not an approval question, an architecture one.
    if not a.reversible and a.blast_radius > hard_radius:
        return Decision.BLOCK
    # Reversible -> let it run; you can undo.
    if a.reversible:
        return Decision.ALLOW
    # Irreversible but bounded -> review only if big or uncertain.
    if a.blast_radius > radius_cap or a.confidence < conf_floor:
        return Decision.REVIEW
    return Decision.ALLOW

Run a spread of actions through it:

read_file    -> allow
db_write     -> allow    (reversible, undo by id)
delete_row   -> review   (irreversible, only 60% confident)
send_email   -> block    (irreversible, 1,200 recipients)
wipe_table   -> block    (irreversible, 900 rows)

Notice what happened to the mass email. The naive policy would have shown it to a human for approval, which feels responsible. But "email 1,200 customers, irreversibly, right now" is not a decision you want made by whoever happens to be watching the queue. It is a decision that belongs in the architecture, made once, cold, with a rate limit and a second sign-off path, not in a modal at 4pm. Routing it to BLOCK is not ducking governance. It is putting the governance where it can be done well.

The middle tier, REVIEW, now fires rarely, on exactly the actions where a human's judgment is both needed and possible. Governance quality is not proportional to how often you stop. It is inversely proportional to how often you stop for no reason.

3. Give the human a real "no"

An approval step where "reject" has no defined path is not a decision, it is a formality. If rejecting the action just dead-ends the workflow, throws an error, or dumps the problem back on a user with no next step, everyone learns quickly that "reject" is the button that breaks things, and they stop pressing it.

A real "no" needs somewhere to go. At minimum:

A fallback the system runs instead (a safer, smaller version of the action).
A re-plan, where rejection feeds back to the agent as a signal to try a different approach rather than halting.
An escalation to someone with more context or authority, not a dead end.
Graceful degradation, where the workflow completes in a reduced but coherent state.

The human also needs time and the ability to be right: enough context to form a judgment, and a system that treats their rejection as information rather than an obstacle. If your reject path is slower, riskier, or more painful for the human than the approve path, you have not built a decision point. You have designed the human into compliance, and the approve button is the only one that works.

A quick test: pull up your agent's reject handler. If it is a raise or a return None, you do not have human-in-the-loop. You have human-in-the-way, and your operators already know it.

The counterintuitive core

Put the three together and you get a conclusion that surprises people: good human-in-the-loop design interrupts the human less, not more.

This runs against the instinct that more oversight is safer. But oversight has a capacity, and it is small. Every unnecessary approval spends down the human's attention and trains the reflex that makes the necessary approval worthless. A system that stops the human ten times a day for trivia and once a month for something that matters has all but guaranteed they will sleepwalk through the one that matters.

There is a body of human-factors research behind this, under names like automation complacency and alarm fatigue: when a signal fires constantly and is almost always benign, operators stop attending to it, and they miss the rare real one. HITL modals are alarms. The same failure mode applies. A system that cries wolf on every database write has trained its humans to ignore the wolf.

So concentrate the interruptions. Make each one rare, high-context, and consequential. That is not less governance. It is the only kind that survives contact with a tired human and a full queue.

A cleaner way to think about it: levels of autonomy

The allow / review / block split hints at a more useful mental model than "is there a human in the loop." Think of each action type your agent can take as sitting at one of a few autonomy levels:

Level 0, Blocked. The agent may propose it, but a human must initiate it through a separate, deliberate path. Reserved for irreversible, high-blast-radius actions. The governance is architectural.
Level 1, Review. The agent prepares the action and a decision summary; a human approves before execution, with a real reject path. Reserved for irreversible-but-bounded, or low-confidence actions.
Level 2, Notify. The agent acts autonomously but the action is reversible and logged loudly, so a human can catch and undo it after the fact. Most write actions live here.
Level 3, Autonomous. The agent acts freely. Reads, idempotent operations, anything with no meaningful blast radius.

The real governance work is not building the modal. It is deciding, per action type, which level it belongs to, and being honest that most teams default everything to Level 1 (approve everything) because it feels safe and requires no thought. It is the least safe option that looks the safest. Sorting your actions into these tiers, deliberately, is the work.

The honest limit

There is a case none of this solves, and pretending otherwise is how HITL became a liability blanket in the first place.

Sometimes the right answer is not "add a human." It is "do not automate this action." If an action is irreversible, high-blast-radius, and the agent's confidence is not trustworthy, no approval modal fixes that, because you have handed a human a decision they cannot make well under the conditions you have given them. That is what Level 0 is for. At the top of the risk spectrum, the governance decision is made long before any human sees a modal: you simply keep the action out of the agent's autonomous reach.

And even a well-designed review step has a ceiling. It assumes the human can evaluate the decision in the time available. For genuinely complex actions, where correctness depends on context the human cannot absorb at a glance, "surface the decision" is not enough, and the answer is either better tooling to make the decision legible or a demotion to Level 0. HITL is a control for the middle of the spectrum. It is not a universal solvent, and treating it as one is the original mistake.

The takeaway

"Is there a human in the loop?" is the wrong question, and it is dangerous precisely because it is so easy to answer yes.

The questions that matter are harder. Can the human see the decision, or just the data? Are they interrupted for reasons, or reflexively? Do they have a real "no"? Which autonomy level does this action actually belong at? And should it have been automated at all?

Human-in-the-loop is not a governance strategy. It is one control, useful in a specific band of risk, and only when it is designed so the human can actually govern. A person positioned to change nothing is not oversight. They are the place the accountability goes to disappear.

Top comments (1)

Tae Kim • Jul 10

The "surface the decision, not the data" framing is exactly right — in every agent we've shipped, the failure mode was humans approving actions they understood syntactically but couldn't evaluate semantically, because the modal showed them what the model would do rather than what assumptions it had made or what the intended outcome was. The fix we landed on was structuring the approval payload as a diff against the human's last known state: not "here is the SQL", but "here is what will change from what you last verified, and here is the model's stated reason that change is correct". This turns the modal from a read operation into an actual decision — the human assesses the delta and the justification, which they can evaluate in context, rather than the full raw action, which they usually cannot in thirty seconds under queue pressure. The remaining failure mode after that is volume: high-frequency approval fatigue is a signal the agent is deployed in a scope where meaningful review isn't feasible, and the answer is scope reduction rather than a faster modal.