A payment landed for exactly half an invoice's value. The payer's email matched the customer on file. The reference generated by Paystack — the Stripe-equivalent payment processor across Africa — didn't match anything at all.
Qwen looked at it and came back with 30% confidence and no invoice named.
I built Recona for the Global AI Hackathon Series with Qwen Cloud, deadline July 20, 2026 — an agent that reconciles Paystack payments against open invoices and chases the overdue ones, with no human involved on the easy cases. That transaction wasn't supposed to be the interesting part of the demo. It became the whole point.
What Recona does
If you freelance or run a small business taking payments in Nigeria, money lands with a reference like PMT final tunde, and you spend the evening figuring out which invoice it settles — and which client you forgot to chase. Recona automates both halves. It matches incoming payments against open invoices using Qwen, and it runs a daily collections sweep that drafts and sends increasingly firm reminders as invoices age.
Cloudflare Workers and D1 handle ingestion and orchestration — signature-verified Paystack webhooks, idempotent against duplicate delivery. Alibaba Cloud SAS runs a Dockerized Node service that holds all the Qwen reasoning, deployed separately from the ingestion layer. The reconciler exposes its matching engine both as REST and as MCP tools — match_transaction_to_invoice, draft_payment_reminder — over streamable HTTP. Telegram is the human-in-the-loop surface, because the actual job here is a workflow closing itself, not another dashboard to log into.
The rule I designed around: the model proposes, deterministic code disposes. Auto-closing an invoice requires exact amount, matching currency, and confidence above a threshold — checked in code after Qwen responds, never trusted from the prompt. The LLM reads the messy handwriting. The calculator authorizes the deposit.
What I expected to demo
I had a clean story planned. A client pays half an invoice. Qwen correctly identifies which one it is. My deterministic guard blocks the auto-close anyway, because the amount is wrong. Model is right, code overrules it for safety. Good demo beat.
That's not what happened.
I ran the real transaction through the real system — the actual Cloudflare Worker at recon-ingest.fpl-test.workers.dev, the actual deployed reconciler, the actual Qwen API. I ran it twice: once against the original invoice, once after re-seeding a fresh one at exactly double the payment amount, to rule out a fluke.
Both times, given a payment that matched an invoice's customer email but was exactly half the amount, with a reference that had zero connection to any invoice number, Qwen returned 30% confidence and no committed invoice ID — even though its own reasoning text named the right invoice by ID. It wasn't wrong. It just wouldn't commit to an answer it didn't have enough signal to support.
I had a choice: force the demo video to match the script I'd already written, or let it show what the model actually did. I rewrote the narration to match reality.
Why the honest version is the better demo
I designed against the failure mode I was worried about — a confident wrong answer sliding past my guards. I didn't design as carefully against the opposite one: a system so wrapped in caution that the model's own certainty never becomes a usable signal, and a human ends up reviewing everything regardless of whether the model actually knew the answer.
What I saw sits in between. Qwen reasoned out loud about the correct invoice, declined to assert it, and handed a legible number to the orchestration layer — 30%, here's why. That's exactly the kind of thing you can build policy around. My auto-close gate doesn't have to grade whether the model's guess is right. It just has to trust the confidence number Qwen already computed about itself, and default to a human whenever that number is low.
Don't build your safety layer to catch the model when it's wrong. Build it to treat the model's own uncertainty as a first output, and put your guardrails on that. The alternative requires you to be smarter than the model at judging its own answers. This one just requires the model to be honest about what it doesn't know — and Qwen, in my testing, was.
A junior hire who's always certain is expensive to trust. One who says "I'm 30% sure, and here's why" is the one you can actually build a process around.
Repo: github.com/dannwaneri/recona — MIT licensed. Built for the Global AI Hackathon Series with Qwen Cloud, Track 4: Autopilot Agent.
Top comments (0)