A payment landed for exactly half an invoice's value. The payer's email matched the customer on file. The reference generated by Paystack — the Str...
For further actions, you may consider blocking this person and/or reporting abuse
This is the part that matters: “the model proposes, deterministic code disposes.”
A lot of agent safety writing focuses on catching confident wrong answers. Your example shows the more useful design pattern: treat uncertainty as a first-class output and route the workflow around it.
For financial reconciliation, I’d want the policy layer to make the thresholds boring and explicit:
That last one is important. If the agent keeps finding the same uncertainty shape, the product needs a better deterministic rule or data capture step.
Related angle on approval gates for AI/database workflows: https://conexor.io/blog/approval-gates-for-ai-database-actions?utm_source=devto&utm_medium=comment&utm_campaign=engagement
The best demo here is not “AI got it right.” It is “the system knew when correctness was not enough to act.”
That last row is the one I don't have an answer for yet. Every escalation logs full reasoning and a flags array (partial_payment, no_match, currency_mismatch) into agent_decisions, but there's no step that looks across escalations for a repeating shape and proposes a rule change. Right now that's still a human reading logs.
Curious how you'd actually detect "same uncertainty shape" in practice — clustering on the flags combination, or embedding the reasoning text and looking for repeats? I'd bucket on flags first since it's cheap and explainable, but that only catches shapes I already named.
Bucket on flags first is the right start, but watch what happens after you find a repeating shape. The step you're missing (nobody reading agent_decisions for patterns) isn't just a missing cron job, it's a maker/checker split one level up. If the same agent that produces the escalations also proposes which flag-cluster becomes a new deterministic rule, you've moved the guessing problem from "which invoice" to "which pattern is real" without changing who's grading.
The boring fix: let the agent nominate clusters (cheap, it's good at noticing repetition), but the promotion from "flagged pattern" to "new rule in code" goes through something the agent doesn't write to. Could be as dumb as a weekly human pass over the top 3 flag-clusters by frequency. The auto-close logic you already built (exact amount + currency + confidence, checked in code, never trusted from the prompt) is the same shape. You just haven't extended it to the meta-level yet: rules about rules need the same "model proposes, deterministic code disposes" discipline.
good point, thanks Mike for noticing. The meta-level reframe is the part I hadn't named. "rules about rules" is exactly what's missing not a missing cron job.
Recona doesn't have the clustering step at all yet, so even the maker half doesn't exist. Building it your way — agent nominates, something that doesn't write to prod promotes means the discipline has to survive one level up: does the weekly human pass start rubber-stamping the agent's clusters after enough uneventful weeks, the same way confidence-blindness happens at the transaction level??
Rules about rules is the right name for the gap, and your own question about the weekly pass answers itself the moment you write it down: yes, it starts rubber-stamping after enough uneventful weeks, for the same reason the underlying transaction check does. A human reviewing the same low-stakes decision every week for a month develops exactly the confidence-blindness you're trying to catch at the layer below, just with a longer period.
Which means the meta-level needs the same kind of forcing function the base level does, not a human promise to stay sharp. If the base check works because it's cheap enough to run every time and specific enough that skipping it is visible, the weekly pass needs an equivalent, something that makes a rubber-stamped week look different from an actually-reviewed one in the record, not just trust that the human noticed the difference internally. Otherwise you've built a maker/promoter split at the transaction level and left the promoter's own attention ungated, which is the same failure moved up one floor instead of removed.
The forcing function has to look like the model's own reasoning field, not a checkbox. Require the reviewer to write an actual sentence justifying each promotion, log it next to the decision, and let staleness show up in the log itself . the same six-word justification recurring for a month is visible in a way "approved" isn't. Have you seen a good way to catch that decay from the justification text itself, or does it always end up needing a second human spot-checking the first??
The part that stuck with me is Qwen naming the right invoice in its reasoning text but still returning 30% confidence and refusing to commit. I have seen that gap too, where the model's chain-of-thought is more certain than the field it actually emits, which makes reading confidence off the wrong signal risky. Did you ever try scoring the reasoning trace separately from the committed answer, or did the deterministic guard make that unnecessary?
No, the deterministic guard made it unnecessary for auto-close specifically, since exact amount/currency still blocks a bad match even if the committed confidence were inflated by whatever's in the reasoning. What I haven't tried is using the reasoning trace as a second signal for the escalation cases, where the model names an invoice internally but won't commit .
That gap between "confident enough to reason about" and "confident enough to commit" is basically the same problem your observation is pointing at. Have you found a reliable way to score trace certainty separately or does it always come back to needing the committed field anyway??
Our own evidence system uses three tiers — deterministic, independent, agent claim — and the gap you're pointing at is the one we haven't closed: the agent's own uncertainty is not a field the system can read. An absent evidence entry defaults to 'uncovered' in our setup, which conflates 'the agent didn't check' with 'the agent checked and found nothing.'
The Qwen case — reasoning named the right invoice, confidence field said 30% — is exactly the distinction we need to surface. The meta-level point from the thread — rules about rules need the same maker/checker split — also applies to our rule lifecycle. The agent nominates patterns; the promotion to a new rule has to go through a path the agent doesn't write to. We don't have this yet.
This hits close to home. Half the AI systems in my Stratagems series were confidently wrong — a benchmark scoring 96.8% on fake data, a scanner missing everything important at 99.97% accuracy. Your rule — treat uncertainty as a first output, not a fallback — would've saved a few fictional careers 🤣
96.8% on fake data and 99.97% on a miss-everything scanner are the kind of numbers that read as more suspicious the rounder and higher they get — Qwen's 30% is uglier and far less quotable, which is probably why it's trustworthy.
Does Stratagems ever show what the fix looked like, or does it stay in the failure mode? I'd read that half.
You're asking the exact question the series is built around.
Honestly, reading it will give you a better answer than me spoiling half of it here. The fix isn't always a patch — sometimes it's a choice, sometimes it's a conversation in a parking lot, sometimes it's a line of config nobody noticed. That's the part I'd rather you discover yourself.
If you do read a few and want to argue about whether they actually fixed anything — I'm all in for that conversation.
Parking lot conversation as a fix is the one I want to read first . that's not a fix pattern software people usually admit to, but it's probably the most common one in practice. I'll read a few and come back for the argument you're offering.
The 30% confidence plus explicit defer is a design choice worth emphasizing: most production agents fail not because the model is wrong when it decides, but because it was never given a clear path to say it doesn't know. Setting the threshold before seeing production data is the hard part — starting low and tightening based on how often defers are actually correct in retrospect is a pattern that works well. One edge case worth watching as you scale is confidence sitting just above threshold on a structurally wrong match, like a close fuzzy string on the wrong invoice date range, so pairing the confidence gate with a hard constraint on one or two key fields tends to catch that. The audit log of every defer decision is also underrated as a training signal for teaching future versions of the agent where the boundary cases actually live.
The amount-and-currency guard is already doing a version of that hard constraint — auto-close requires exact match on both before confidence ever gets checked, precisely to catch a near-threshold score on a structurally wrong invoice. What I don't have yet is the training-signal loop you're describing. The audit log exists (every defer, with reasoning), but nothing's reading it back to tighten thresholds or flag repeat shapes. Right now that's a human occasionally scrolling agent_decisions.
Have you actually closed that loop somewhere — automated threshold-tightening from retrospective defer accuracy or is it still a manual review cadence in what you've built??
Knowing when not to guess is one of the most valuable agent behaviors. The trick is making uncertainty operational: the agent should know what evidence is missing, what tool would reduce uncertainty, and when the right answer is to stop instead of producing a confident-looking filler.
I think confidence awareness is an underrated capability in AI agents. The ability to pause, gather additional context, or invoke the right tool instead of guessing can significantly improve reliability. In production systems, reducing confident mistakes is often more valuable than maximizing the number of answers an agent can produce.
"Knows when not to guess" is the feature most payment/ops agents are missing.
Half-paid invoices + fuzzy reference matching is exactly where confident wrong matches get expensive. Forcing an explicit uncertain state (and a human handoff) is more valuable than a higher match score. Curious how you surface those "I won't guess" cases in the UI — queue vs chat vs both.
What I like most here is that the model didn’t just try to be “right,” it knew when not to commit. That’s way more useful in a real system than a model that guesses confidently every time.
Treating confidence as part of the output, not just a side detail, feels like the key insight. Once you trust that signal, you can actually build clean workflows around it instead of trying to catch mistakes after the fact.
It’s less about forcing automation everywhere and more about knowing exactly when not to automate, and that’s what makes something like this production-ready.
Knowing when not to automate turned out to be the harder engineering problem than the matching itself.
The part worth pushing on: where you draw the deferral line is an economics question, not a model-tuning one. Every defer costs human minutes, every wrong auto-close costs a downstream mess to unwind, and the right cutoff sits where those two meet. Deferring the ugly 30% cases is the easy call. The harder one is what happens to that line when volume triples and the humans catching escalations become the new bottleneck.
good thinking valentin. The threshold is one global number (0.85),picked because it sounded plausable not because I measured anything.
Your framing exposes that it probably shouldn't be a single number at all . a wrong auto-close on a small invoice costs an awkward email, a wrong auto-close on a large one costs a real dispute, so the cutoff should scale with what's actually on the line, not the confidence score alone. What I don't have is a way to find that number before there's enough volume to have failures to learn from . how would you actually measure cost-per-escalation with the small sample a project like this starts with??
the 30% confidence plus "no invoice named" return is the right design and rarer than it should be. most agent tools fail by guessing confidently or blocking with an error. surfacing uncertainty as an explicit output is what earns trust for automation.
we built something adjacent: an invoice categorization step in an AP automation pipeline. biggest lesson: the escalation threshold is not a fixed number. it drifted 8 points over six months as invoice layouts changed. logged every escalation with actual vs predicted confidence so we could retrain on real drift.
how are you handling the confidence cutoff in Recona — fixed or learned over payment history?
Excellent insight AI agents become far more reliable when they know when to defer instead of guessing.
“The model proposes, deterministic code disposes” is absolute gold. Treating uncertainty as a first-class feature instead of a failure mode is how agents actually survive prod.