Asking a bank statement questions in plain English
A bank statement is a PDF full of numbers that a human has to squint at and a spreadsheet has to be coaxed into. The Financial Document Intelligence Platform turns that PDF into something you can interrogate: upload a statement, then ask "what's my average monthly burn rate?" in plain English and get an answer computed from your actual transactions — not guessed.
The hard part of building it wasn't the happy path. It was three specific problems that every document-plus-LLM system runs into, and the choices I made to get around them. That's what this post is about.
The architecture, in one breath
The system is split down the middle on purpose: a FastAPI backend handles the heavy lifting (async, high-throughput), a Flask frontend serves the dashboard, Supabase (Postgres) stores the structured data, OCR runs through Azure or EasyOCR, and documents live in Cloudinary. The natural-language understanding is handled by an LLM integration. It's Docker-ready so the whole thing comes up as one stack.
Now the three problems.
Problem 1: heavy extraction was freezing the UI
Parsing a large PDF or Excel statement means chewing through thousands of transactions. In a monolithic design, that heavy extraction work sat on the same path as the user's clicks — so uploading a big statement would either time out or freeze the interface while it ground away.
The fix was architectural, not a faster loop. I separated the heavy extraction logic from the user-facing query engine into a microservices-oriented split. Extraction runs as its own concern; the dashboard stays responsive because it's no longer waiting behind a document parser. The lesson generalizes: when a slow operation degrades an unrelated part of your app, the answer is usually a boundary, not an optimization.
Problem 2: the LLM kept inventing numbers
This is the problem that sinks most "AI for finance" demos. Ask a generic LLM a specific question about your money and it will happily hallucinate — invent a transaction, round a figure into fiction, confidently state a number that appears nowhere in your data. In a finance product, that isn't a quirky failure; it's a disqualifying one.
The fix is the design decision I'm most deliberate about: the LLM never answers from its own knowledge. The system uses a Retrieval-Augmented Generation (RAG) pipeline where the model's only job is to generate a SQL query or a set of filters against a strict schema. The actual numbers come back from the database, computed over the user's real transactions. The model decides what to ask the data; it never decides what the answer is.
That single constraint — LLM produces queries, database produces facts — is the whole trick to making an LLM trustworthy over financial data. It can't invent a transaction because it's never in a position to state one.
Problem 3: every bank uses a different layout
There is no standard bank statement. Every institution has its own columns, date formats, and structure, which makes pure rule-based parsing brittle — write a parser for one bank and it shatters on the next.
The answer was a hybrid parser. For common, well-structured formats it uses fast regex heuristics. When a document doesn't match — an unusual layout, a scanned page — it falls back to an LLM-vision approach that can read structure the regex can't. Rules where rules work, a model where they don't. You get the speed and determinism of heuristics on the 80% case and the flexibility of a model on the messy tail, instead of paying the cost of the model on every document.
Security wasn't an afterthought
Because this touches financial data, the query interface is built defensively:
- Input sanitization — regex filters against SQL injection and XSS on the way in.
- Rate limiting — so the endpoints can't be hammered into a resource-exhaustion problem.
- Opaque error responses — standardized errors that don't leak internal detail, so a failure never becomes a reconnaissance tool for an attacker.
And once the data is structured, the analytics go past a transaction list: monthly burn rate, an income-stability score, category-wise spending breakdowns, and recurring-subscription detection.
What I'd take to the next project
- RAG-as-a-guardrail, not just retrieval. Constraining the LLM to query generation is the most reliable way I've found to use one over data that has to be correct.
- Boundaries fix responsiveness. Splitting extraction from querying did more for the user experience than any single performance tweak.
- Hybrid beats pure. Regex-or-model, chosen per document, beats committing entirely to either one.
The full architecture, the problems-faced write-up, the security model, and the deployment setup are in the repository.
Top comments (1)
Nice write-up — separating extraction from the query path is the right call, and your point about LLMs never inventing a number they can't ground is the single most under-discussed reliability problem in this space. Agree on all three.
One thing I'd push on for v2: the biggest accuracy lever isn't the grounding layer, it's the source. Every problem you list (OCR errors, category heuristics, 90-day history limits) is downstream of starting from a PDF. A statement is a rendering of data the bank already holds as structured rows — if you can read those rows directly instead of re-parsing the rendering, hallucination risk collapses to ~zero because there's no extraction step to fail.
In the EU/UK that's available without anything exotic: PSD2 mandates machine-readable transaction APIs, and the newer eIDAS QWAC-free profiles (Berlin Group NextGen PSD2 / OBIE VRS-style) mean you don't even need an eIDAS certificate to read a live account's transactions as JSON — just an OAuth consent. For an MVP you still want the PDF path (works everywhere, no bank-by-bank onboarding), but if a power user offers to connect their account, pulling structured TXNs cuts your category-heuristic and reconciliation work dramatically.
Quick correctness nit on the grounding rule itself: "answer must be backed by ≥N transactions" can still pass for a wrong answer if the underlying OCR misread several amounts in the same direction (correlated errors aren't caught by count thresholds). Worth adding a checksum against the statement's stated opening/closing balance — if Σ(parsed amounts) doesn't reconcile to closing − opening, fail loud before the LLM ever sees the data. That one invariant catches the failure mode that count-based grounding can't.
Curious whether you've benchmarked Azure OCR vs EasyOCR on statement tables specifically — the digit-recall difference on dense transaction grids is usually where the PDF path actually breaks.
(John Frandsen — open-banking.io)