The human-in-the-loop agent: an 8B model proposes, Python pauses the risky calls, a human decides

You don't let an 8B model email your customers or move money on its own. So this agent stops and asks. A safe lookup auto-runs; sending an email or issuing a refund pauses for a human to approve, deny, or edit — and every proposal, verdict, and decision is written to an audit trail. The design principle is a hard separation of powers: the model plans and rates its own confidence, but the risky decision is taken away from it and given to deterministic Python and a human.

Uncertainty detection: one verdict from three signals

For every proposed action, a pure-Python gate combines three independent signals — the tool's declared risk, a sensitive-capability check, and the model's own confidence. Any one of them demands a human.

SENSITIVE_CAPABILITIES = {"send_email", "spend_money", "delete", "external_write"}

class ApprovalGate:
    def __init__(self, min_confidence=0.60): self.min_confidence = min_confidence
    def assess(self, tool, args, confidence):
        signals = []
        if tool.risk == "high":
            signals.append(f"tool '{tool.name}' declares risk=high")
        if tool.capabilities & SENSITIVE_CAPABILITIES:          # defence in depth
            signals.append("touches sensitive capability")
        if confidence < self.min_confidence:                    # low conf == risk
            signals.append(f"model confidence {confidence:.2f} < {self.min_confidence:.2f}")
        return RiskVerdict(requires_approval=bool(signals),     # ANY signal → pause
                           risk=tool.risk, confidence=confidence, signals=signals)

Risk is declared by the tool author once, not argued by the model at runtime. The sensitive-capability check is a backstop — even an unmarked tool that sends email or spends money trips the gate. And a safe tool the model rated below 0.60 pauses too: when the model is guessing an amount or address, that uncertainty is itself a reason to ask. Because it's deterministic, the same proposal always earns the same verdict.

Pause, then resume with validated context

On a risky action the loop suspends and asks a decision source (scripted for a reproducible run, or a live CLI [a]pprove / [d]eny / [e]dit). A Decision carries a verdict, the approver, a reason, and — for an edit — a patch of argument overrides. On resume the human's patch is merged over the model's args, so the human's fields win; a deny simply skips the tool and the task carries on.

if decision.is_go:
    run_args = dict(action.args)
    if decision.verdict == "edit" and decision.edited_args:
        run_args.update(decision.edited_args)        # <- human patch wins
    res = tool.run(run_args)                          # RESUME → execute
    self.audit.executed(i, action.tool, run_args, res)
else:                                                 # DENY
    self.audit.refused(i, action.tool, decision.reason)  # carry on gracefully

And if nobody answers at all, the source returns None and the agent fails closed — it synthesises a safe-default denial by auto-escalation. Unsupervised never means unleashed.

Every decision is audited

The audit trail is deliberately boring and durable: every event is one JSON line appended to a .jsonl file, stamped with a monotonic seq — a step counter, not a wall clock — so two identical runs produce byte-identical logs and nothing leaks about when it ran.

class AuditLog:
    def record(self, event, step, **fields):
        self._seq += 1
        entry = {"seq": self._seq, "step": step, "event": event, **fields}
        with open(self.path, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
        return entry

A real run

The captured transcript is a live run against NVIDIA NIM (meta/llama-3.1-8b-instruct). The task: a customer's order arrived damaged — look it up, email an apology, refund the $60 shipping. The model planned three actions and rated each: [('lookup_order', 1.0), ('send_email', 0.9), ('issue_refund', 0.8)].

The gate auto-ran the safe lookup, then suspended on both send_email and issue_refund despite high confidence — because risk overrides. The email was approved with a human edit to the subject and resumed; the refund was denied, so no money moved. Fifteen events landed in audit-log.jsonl, the refusal was carried into the final reply (which never claims the refund happened), and the refund's absence from the money ledger is provable, not merely asserted.

Step through the real assess → pause → decide → resume → audit pipeline, read the full audit trail, and browse the code at: https://dev48v.infy.uk/agentic/project6-hitl.html (repo: github.com/dev48v/agentic-ai-from-zero)

Next up, Project 7: a cost-aware agent router.