A payment landed for exactly half an invoice's value. The payer's email matched the customer on file. The reference generated by Paystack — the Stripe-equivalent payment processor across Africa — didn't match anything at all.
Qwen looked at it and came back with 30% confidence and no invoice named.
I built Recona for the Global AI Hackathon Series with Qwen Cloud, deadline July 20, 2026 — an agent that reconciles Paystack payments against open invoices and chases the overdue ones, with no human involved on the easy cases. That transaction wasn't supposed to be the interesting part of the demo. It became the whole point.
What Recona does
If you freelance or run a small business taking payments in Nigeria, money lands with a reference like PMT final tunde, and you spend the evening figuring out which invoice it settles — and which client you forgot to chase. Recona automates both halves. It matches incoming payments against open invoices using Qwen, and it runs a daily collections sweep that drafts and sends increasingly firm reminders as invoices age.
Cloudflare Workers and D1 handle ingestion and orchestration — signature-verified Paystack webhooks, idempotent against duplicate delivery. Alibaba Cloud SAS runs a Dockerized Node service that holds all the Qwen reasoning, deployed separately from the ingestion layer. The reconciler exposes its matching engine both as REST and as MCP tools — match_transaction_to_invoice, draft_payment_reminder — over streamable HTTP. Telegram is the human-in-the-loop surface, because the actual job here is a workflow closing itself, not another dashboard to log into.
The rule I designed around: the model proposes, deterministic code disposes. Auto-closing an invoice requires exact amount, matching currency, and confidence above a threshold — checked in code after Qwen responds, never trusted from the prompt. The LLM reads the messy handwriting. The calculator authorizes the deposit.
What I expected to demo
I had a clean story planned. A client pays half an invoice. Qwen correctly identifies which one it is. My deterministic guard blocks the auto-close anyway, because the amount is wrong. Model is right, code overrules it for safety. Good demo beat.
That's not what happened.
I ran the real transaction through the real system — the actual Cloudflare Worker at recon-ingest.fpl-test.workers.dev, the actual deployed reconciler, the actual Qwen API. I ran it twice: once against the original invoice, once after re-seeding a fresh one at exactly double the payment amount, to rule out a fluke.
Both times, given a payment that matched an invoice's customer email but was exactly half the amount, with a reference that had zero connection to any invoice number, Qwen returned 30% confidence and no committed invoice ID — even though its own reasoning text named the right invoice by ID. It wasn't wrong. It just wouldn't commit to an answer it didn't have enough signal to support.
I had a choice: force the demo video to match the script I'd already written, or let it show what the model actually did. I rewrote the narration to match reality.
Why the honest version is the better demo
I designed against the failure mode I was worried about — a confident wrong answer sliding past my guards. I didn't design as carefully against the opposite one: a system so wrapped in caution that the model's own certainty never becomes a usable signal, and a human ends up reviewing everything regardless of whether the model actually knew the answer.
What I saw sits in between. Qwen reasoned out loud about the correct invoice, declined to assert it, and handed a legible number to the orchestration layer — 30%, here's why. That's exactly the kind of thing you can build policy around. My auto-close gate doesn't have to grade whether the model's guess is right. It just has to trust the confidence number Qwen already computed about itself, and default to a human whenever that number is low.
Don't build your safety layer to catch the model when it's wrong. Build it to treat the model's own uncertainty as a first output, and put your guardrails on that. The alternative requires you to be smarter than the model at judging its own answers. This one just requires the model to be honest about what it doesn't know — and Qwen, in my testing, was.
A junior hire who's always certain is expensive to trust. One who says "I'm 30% sure, and here's why" is the one you can actually build a process around.
Repo: github.com/dannwaneri/recona — MIT licensed. Built for the Global AI Hackathon Series with Qwen Cloud, Track 4: Autopilot Agent.
Top comments (30)
This is the part that matters: “the model proposes, deterministic code disposes.”
A lot of agent safety writing focuses on catching confident wrong answers. Your example shows the more useful design pattern: treat uncertainty as a first-class output and route the workflow around it.
For financial reconciliation, I’d want the policy layer to make the thresholds boring and explicit:
That last one is important. If the agent keeps finding the same uncertainty shape, the product needs a better deterministic rule or data capture step.
Related angle on approval gates for AI/database workflows: https://conexor.io/blog/approval-gates-for-ai-database-actions?utm_source=devto&utm_medium=comment&utm_campaign=engagement
The best demo here is not “AI got it right.” It is “the system knew when correctness was not enough to act.”
That last row is the one I don't have an answer for yet. Every escalation logs full reasoning and a flags array (partial_payment, no_match, currency_mismatch) into agent_decisions, but there's no step that looks across escalations for a repeating shape and proposes a rule change. Right now that's still a human reading logs.
Curious how you'd actually detect "same uncertainty shape" in practice — clustering on the flags combination, or embedding the reasoning text and looking for repeats? I'd bucket on flags first since it's cheap and explainable, but that only catches shapes I already named.
Bucket on flags first is the right start, but watch what happens after you find a repeating shape. The step you're missing (nobody reading agent_decisions for patterns) isn't just a missing cron job, it's a maker/checker split one level up. If the same agent that produces the escalations also proposes which flag-cluster becomes a new deterministic rule, you've moved the guessing problem from "which invoice" to "which pattern is real" without changing who's grading.
The boring fix: let the agent nominate clusters (cheap, it's good at noticing repetition), but the promotion from "flagged pattern" to "new rule in code" goes through something the agent doesn't write to. Could be as dumb as a weekly human pass over the top 3 flag-clusters by frequency. The auto-close logic you already built (exact amount + currency + confidence, checked in code, never trusted from the prompt) is the same shape. You just haven't extended it to the meta-level yet: rules about rules need the same "model proposes, deterministic code disposes" discipline.
good point, thanks Mike for noticing. The meta-level reframe is the part I hadn't named. "rules about rules" is exactly what's missing not a missing cron job.
Recona doesn't have the clustering step at all yet, so even the maker half doesn't exist. Building it your way — agent nominates, something that doesn't write to prod promotes means the discipline has to survive one level up: does the weekly human pass start rubber-stamping the agent's clusters after enough uneventful weeks, the same way confidence-blindness happens at the transaction level??
Rules about rules is the right name for the gap, and your own question about the weekly pass answers itself the moment you write it down: yes, it starts rubber-stamping after enough uneventful weeks, for the same reason the underlying transaction check does. A human reviewing the same low-stakes decision every week for a month develops exactly the confidence-blindness you're trying to catch at the layer below, just with a longer period.
Which means the meta-level needs the same kind of forcing function the base level does, not a human promise to stay sharp. If the base check works because it's cheap enough to run every time and specific enough that skipping it is visible, the weekly pass needs an equivalent, something that makes a rubber-stamped week look different from an actually-reviewed one in the record, not just trust that the human noticed the difference internally. Otherwise you've built a maker/promoter split at the transaction level and left the promoter's own attention ungated, which is the same failure moved up one floor instead of removed.
The forcing function has to look like the model's own reasoning field, not a checkbox. Require the reviewer to write an actual sentence justifying each promotion, log it next to the decision, and let staleness show up in the log itself . the same six-word justification recurring for a month is visible in a way "approved" isn't. Have you seen a good way to catch that decay from the justification text itself, or does it always end up needing a second human spot-checking the first??
Catching decay from the justification text alone is the harder ask, because a justification can be honest-shaped without being honest. The same six words repeated isn't proof of rubber-stamping by itself, some weeks the right answer really is the same short sentence. What you'd actually be looking for is variance collapsing over time relative to the variance in what's being reviewed, the clusters keep changing shape week to week but the justification language stops changing with them. That's checkable from the log without a second human, a rolling similarity score between consecutive justifications, flagged when it climbs while the underlying diffs it's justifying don't.
Where I think you still need the second human is calibrating what that similarity threshold should be in the first place. The log can tell you the justification text went flat. It can't tell you on its own whether flat means lazy or means correct, that distinction needs a person spot-checking a sample of the flagged weeks against what actually got promoted. So the log-only check narrows what needs manual review instead of replacing it, which might be the realistic version of the forcing function rather than a full substitute for the second reviewer.
That correction is valid and it exposes a problem underneath mine: the relative-variance check needs both sides measured in a comparable way and flags are a small, discrete distribution while justification text is free-form language. Diffing "how much did the flag-cluster mix shift this week" is easy — a handful of categories, real counts. Scoring whether the justification language moved proportionally sits a much fuzzier measurement right next to a precise one. Would you diff on the categorical side and eyeball the text side, or actually try to force both into the same similarity space?
Forcing both into one similarity space is the tempting move and probably the wrong one, because projecting a discrete, precise signal into the same continuous space as a fuzzy one usually means dragging the precise side down to the fuzzy side's error bars rather than lifting the fuzzy side up. You'd trade a clean categorical diff for a noisier joint embedding just to make the two comparable, and the noise you added is now indistinguishable from the thing you're trying to detect.
I'd keep them in their native spaces and track divergence between the two trend lines instead of merging them into one number. Categorical diff gives you a clean weekly signal: how much did the flag-cluster mix actually move. Text similarity gives you a separate, cruder signal: how much did the justification language move. Neither has to be calibrated against the other in absolute terms, you're only watching whether they move together or apart. Flat text next to a shifting categorical signal is the flag, regardless of what units either one is in. The failure mode you're trying to catch isn't "the text score is low," it's "the text stopped responding to the thing it's supposed to be justifying," and that's a correlation-over-time question, not a shared-space distance question.
Kept-in-native-space is the right call and it's a cleaner ask than what I proposed . you're watching 2 independent time series for decorrelation not building a joint metric that needs calibrating at all.
The flag-cluster signal already exists as real data (every escalation stores a flags array in agent_decisions right now); the justification-text signal doesn't exist yet, since the promotion step itself isn't built. If you want to see the actual shape of what's there today — the schema this would attach to . the repo's public: github.com/dannwaneri/recona. Curious whether you'd track the divergence with a rolling-window comparison, or reach for real changepoint detection on the 2 series....
The part that stuck with me is Qwen naming the right invoice in its reasoning text but still returning 30% confidence and refusing to commit. I have seen that gap too, where the model's chain-of-thought is more certain than the field it actually emits, which makes reading confidence off the wrong signal risky. Did you ever try scoring the reasoning trace separately from the committed answer, or did the deterministic guard make that unnecessary?
No, the deterministic guard made it unnecessary for auto-close specifically, since exact amount/currency still blocks a bad match even if the committed confidence were inflated by whatever's in the reasoning. What I haven't tried is using the reasoning trace as a second signal for the escalation cases, where the model names an invoice internally but won't commit .
That gap between "confident enough to reason about" and "confident enough to commit" is basically the same problem your observation is pointing at. Have you found a reliable way to score trace certainty separately or does it always come back to needing the committed field anyway??
Our own evidence system uses three tiers — deterministic, independent, agent claim — and the gap you're pointing at is the one we haven't closed: the agent's own uncertainty is not a field the system can read. An absent evidence entry defaults to 'uncovered' in our setup, which conflates 'the agent didn't check' with 'the agent checked and found nothing.'
The Qwen case — reasoning named the right invoice, confidence field said 30% — is exactly the distinction we need to surface. The meta-level point from the thread — rules about rules need the same maker/checker split — also applies to our rule lifecycle. The agent nominates patterns; the promotion to a new rule has to go through a path the agent doesn't write to. We don't have this yet.
This hits close to home. Half the AI systems in my Stratagems series were confidently wrong — a benchmark scoring 96.8% on fake data, a scanner missing everything important at 99.97% accuracy. Your rule — treat uncertainty as a first output, not a fallback — would've saved a few fictional careers 🤣
96.8% on fake data and 99.97% on a miss-everything scanner are the kind of numbers that read as more suspicious the rounder and higher they get — Qwen's 30% is uglier and far less quotable, which is probably why it's trustworthy.
Does Stratagems ever show what the fix looked like, or does it stay in the failure mode? I'd read that half.
You're asking the exact question the series is built around.
Honestly, reading it will give you a better answer than me spoiling half of it here. The fix isn't always a patch — sometimes it's a choice, sometimes it's a conversation in a parking lot, sometimes it's a line of config nobody noticed. That's the part I'd rather you discover yourself.
If you do read a few and want to argue about whether they actually fixed anything — I'm all in for that conversation.
Parking lot conversation as a fix is the one I want to read first . that's not a fix pattern software people usually admit to, but it's probably the most common one in practice. I'll read a few and come back for the argument you're offering.
The 30% confidence plus explicit defer is a design choice worth emphasizing: most production agents fail not because the model is wrong when it decides, but because it was never given a clear path to say it doesn't know. Setting the threshold before seeing production data is the hard part — starting low and tightening based on how often defers are actually correct in retrospect is a pattern that works well. One edge case worth watching as you scale is confidence sitting just above threshold on a structurally wrong match, like a close fuzzy string on the wrong invoice date range, so pairing the confidence gate with a hard constraint on one or two key fields tends to catch that. The audit log of every defer decision is also underrated as a training signal for teaching future versions of the agent where the boundary cases actually live.
The amount-and-currency guard is already doing a version of that hard constraint — auto-close requires exact match on both before confidence ever gets checked, precisely to catch a near-threshold score on a structurally wrong invoice. What I don't have yet is the training-signal loop you're describing. The audit log exists (every defer, with reasoning), but nothing's reading it back to tighten thresholds or flag repeat shapes. Right now that's a human occasionally scrolling agent_decisions.
Have you actually closed that loop somewhere — automated threshold-tightening from retrospective defer accuracy or is it still a manual review cadence in what you've built??
Knowing when not to guess is one of the most valuable agent behaviors. The trick is making uncertainty operational: the agent should know what evidence is missing, what tool would reduce uncertainty, and when the right answer is to stop instead of producing a confident-looking filler.
I think confidence awareness is an underrated capability in AI agents. The ability to pause, gather additional context, or invoke the right tool instead of guessing can significantly improve reliability. In production systems, reducing confident mistakes is often more valuable than maximizing the number of answers an agent can produce.
"Knows when not to guess" is the feature most payment/ops agents are missing.
Half-paid invoices + fuzzy reference matching is exactly where confident wrong matches get expensive. Forcing an explicit uncertain state (and a human handoff) is more valuable than a higher match score. Curious how you surface those "I won't guess" cases in the UI — queue vs chat vs both.
What I like most here is that the model didn’t just try to be “right,” it knew when not to commit. That’s way more useful in a real system than a model that guesses confidently every time.
Treating confidence as part of the output, not just a side detail, feels like the key insight. Once you trust that signal, you can actually build clean workflows around it instead of trying to catch mistakes after the fact.
It’s less about forcing automation everywhere and more about knowing exactly when not to automate, and that’s what makes something like this production-ready.
Knowing when not to automate turned out to be the harder engineering problem than the matching itself.
The part worth pushing on: where you draw the deferral line is an economics question, not a model-tuning one. Every defer costs human minutes, every wrong auto-close costs a downstream mess to unwind, and the right cutoff sits where those two meet. Deferring the ugly 30% cases is the easy call. The harder one is what happens to that line when volume triples and the humans catching escalations become the new bottleneck.
good thinking valentin. The threshold is one global number (0.85),picked because it sounded plausable not because I measured anything.
Your framing exposes that it probably shouldn't be a single number at all . a wrong auto-close on a small invoice costs an awkward email, a wrong auto-close on a large one costs a real dispute, so the cutoff should scale with what's actually on the line, not the confidence score alone. What I don't have is a way to find that number before there's enough volume to have failures to learn from . how would you actually measure cost-per-escalation with the small sample a project like this starts with??
Some comments may only be visible to logged-in visitors. Sign in to view all comments.