How I Built an Override Detector That Actually Teaches the Agent Something

The agent recommended approval. The human said no. That's an override — and in most systems, it's a dead end. The recommendation is discarded, the human's decision is recorded, and next time the same exception comes in, the agent makes the same wrong recommendation again.

I wanted the override to be a lesson. In MemoryOps, when a human disagrees with the agent, that disagreement is retained in Hindsight as a precedent with an explicit override flag. The next time the agent faces a similar case, it can see that its previous recommendation was wrong and adjust.

Detecting the Override

An override happens when the agent recommends a specific action and the human records a different one. Not an escalation — escalations are "I don't know, you decide," so there's nothing to override. But when the agent says approve and the human rejects, or the agent says adjust and the human approves, that's a genuine disagreement.

# services.py — override detection at decision time
d = Decision(
    exception_id=e.id,
    recommendation_id=reco.id if reco else None,
    decision=decision,
    approver_name=approver_name.strip(),
    approver_role=approver_role.strip(),
    reason=reason.strip(),
    is_override=bool(
        reco
        and reco.action != "escalate"      # escalations don't count
        and reco.action != action           # human chose differently
    ),
)

reco.action is what the agent said. action is what the human decided. If they differ and the agent wasn't escalating, is_override = True.

This happens at decision time, before the record is retained into memory. The override flag travels with the decision into Hindsight.

What Gets Stored

The decision record stored in Hindsight includes the override flag in its text:

# services.py — the text written to Hindsight for each decision
override = " This overrode the agent's recommendation." if d.is_override else ""
text = (
    f"On {d.decided_on}, {d.approver_name} ({d.approver_role}) {d.decision} "
    f"invoice {inv.invoice_number} from {v.name} ({v.code}): "
    f"{e.type.replace('_', ' ')}{size}. Reason: {d.reason}.{override}"
)

The phrase "This overrode the agent's recommendation" is embedded in the text of the memory record. When the agent recalls this record in a future case, it sees not just the decision but the fact that a human corrected the agent on this vendor and exception type.

It also appears in the vendor pattern fact (VF record), which is rebuilt after every decision:

# services.py — vendor fact includes full decision history
parts = []
for dec, ds in sorted(by_dec.items()):
    parts.append(f"{dec} {len(ds)}x ({ds[0].decided_on} to {ds[-1].decided_on})")
last = rows[-1]
text = (
    f"Vendor pattern for {v.name} ({v.code}), {label}: {len(rows)} past decisions: "
    f"{'; '.join(parts)}. Most recent: {last.decision} on {last.decided_on} "
    f"by {last.approver_name} - \"{last.reason}\"."
)

If the most recent decisions are rejections (including an override), the vendor pattern fact will say so. The LLM reads this and understands that the recent trend is different from the older history.

The Effect in Practice

In our second batch, one exception was overridden: the agent recommended approval on ARB-1112, but the human rejected it with the reason "Escalator clause lapsed 30 June."

That override was retained into Hindsight with is_override=True. The vendor pattern fact for that vendor was rebuilt to show the rejection as the most recent decision.

In the third batch, ARB-1120 came in — same vendor, same exception type, similar size. The agent's recommendation was rejection, citing the overridden decision and the pattern shift:

"Pattern changed: the two most recent cases were rejected, while earlier ones were approved. Following the recent decisions."

The cited IDs included the override from batch 2. The agent had learned from the correction without any prompt engineering, without any retraining, without any code change. It saw the override in memory and weighted it appropriately.

What the UI Shows

When a row in the exceptions table carries an override, it's visually flagged in the decision column with a rose-coloured badge. This makes it easy for reviewers to spot: "this is a case where we disagreed with the agent." That visibility matters for auditing and for building trust in the system over time.

The audit log captures every override as a timestamped event, with the agent's recommendation and the human's decision both recorded. Nothing is overwritten — the agent's original recommendation is preserved alongside the human's correction.

The Limitation I Haven't Solved

Override detection works well when the agent is clearly wrong. It works less well on genuinely ambiguous cases.

If the agent recommended escalation (correctly, because the case was ambiguous) and the human approved it, that's not detected as an override — because escalations are excluded from the override logic. The human's approval is retained as a precedent, but the "the agent was too cautious here" signal isn't explicitly captured.

This means the agent can remain overly cautious on a type of exception it's been escalating unnecessarily. The memory grows with approvals, and eventually the agent starts recommending approval on its own. But it happens slowly, through accumulated positive examples, rather than through explicit "you were too cautious" feedback.

A more complete system would track "the agent escalated and the human resolved it" as a distinct signal. That's on the list.

Why Memory Makes This Work

Without persistent agent memory, override detection is a logging feature. You can record that an override happened. But the agent won't know about it next time.

Hindsight is what makes the override useful. The decision with the override flag goes into the memory bank. The vendor pattern fact is rebuilt. The next recall query for that vendor returns a picture of the history that includes the correction. The agent reasons from that picture.

The mechanism is simple. The retention schema is what makes it work — specifically, embedding the override signal in the text of the memory record so it's visible to the LLM during recall, and updating the vendor pattern fact so the trend is reflected in a single synthesised record rather than scattered across individual precedents.

If you're building an agent that takes actions based on recalled memory, add override detection early. The cases where humans correct the agent are the most valuable training signal you have. Don't let them disappear into a log file.