This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
If you've ever paid a tuition fee, a society maintenance bill or a home baker in India, you know how it goes. You pay by UPI, take a screenshot, send it on WhatsApp. Done.
Now think about the person on the other end. A tuition teacher with 40 students gets 40 screenshots a month, at random times, buried in the rest of her WhatsApp. At some point she sits down with her bank statement and ticks them off one by one, reading 12-digit reference numbers off a phone screen.
And she can't fully trust any of them. A screenshot is just a picture. It can be edited. It can be of a payment that failed. It can be last month's screenshot sent again. Fake payment screenshots are a common enough scam that banks and payment companies keep publishing guides telling shopkeepers to check their account, not the screenshot.
That check is the whole tool. ProofCheck takes a folder of payment screenshots and a bank statement export, reads the UPI reference number (the UTR) off each screenshot, and looks for it in the statement. Every screenshot ends up as one of these:
| Result | What it means |
|---|---|
| Verified | The UTR is in the statement and the amount matches. The money came in. |
| Amount differs | The UTR is there, but the screenshot shows a different amount. |
| Not in statement | No such UTR. Could be fake, could be a payment that hasn't settled yet. |
| Duplicate | Another screenshot already used this UTR. |
| Outside dates | The screenshot is dated after the statement ends. Check a newer statement. |
| Unreadable | Couldn't find a UTR. You'll have to look at this one yourself. |
You'll notice it never says "fake". I went back and forth on that. A missing UTR is suspicious, but a payment made an hour ago is missing too, and accusing a parent of fraud over a settlement delay is a good way to lose a student.
Give it a small roster as well (who owes how much, and who usually pays for them) and you get the list a teacher actually cares about on the 5th of the month: who has paid, who paid part, who only sent a screenshot, and who hasn't paid at all. Money that arrived without any screenshot still counts, because the statement is the source of truth, not WhatsApp.
So who would pay for this? Anyone collecting the same payments every month into a personal account and matching them by hand. Tuition classes are the obvious case. Housing society treasurers are another: a 120-flat society has 120 maintenance payments to match every month, often in a spreadsheet. The amounts are small, so the tool has to cost almost nothing to run, and that's a big part of why it's built on open models.
Demo
The demo data is a made-up tuition class collecting October fees. There are eight screenshots, and between them they cover every case: payments that went through, one that never arrived, a failed payment sent as proof, a screenshot where ₹200 was edited to ₹2,000, and the same screenshot sent twice for two kids.
Results show up row by row while it's still reading:
On my laptop (a GTX 1650 with 4 GB of VRAM, nothing fancy) it got all 8 screenshots and all 8 roster lines right. With the model already loaded, the run took about 94 seconds, so roughly 12 seconds a screenshot.
What that does and doesn't prove: the demo data is synthetic, so it shows the pipeline works end to end. It says nothing yet about how well it reads real screenshots from real phones.
About those demo screenshots. They come from a fake app I called "Demo Pay", with SAMPLE - NOT A REAL PAYMENT stamped across them. It felt wrong to build a tool that catches fake payment screenshots and then ship a folder of convincing fake payment screenshots with it.
If you want to run it yourself:
ollama pull gemma3:4b
cd frontend && npm ci && npm run build && cd ..
cd backend && uv sync && uv run proofcheck # then open http://127.0.0.1:8000
Code
Surge77
/
proofcheck
Check UPI payment screenshots against your bank statement with local open-weight models (Gemma + RapidOCR). Your statement never leaves your laptop.
ProofCheck
Check UPI payment screenshots against your bank statement, on your own laptop.
Tuition teachers, housing-society treasurers, home bakers and tiffin services collect payments by UPI into a personal account. The "proof" arrives as a screenshot on WhatsApp, and every month someone matches dozens of them against the bank statement by hand. Fake payment screenshots are a common scam, and they look exactly like real ones.
ProofCheck reads each screenshot with local open-weight models, finds its 12-digit UPI reference (UTR), and looks for that reference in your bank statement. It then tells you, in plain words who really paid.
Nothing is uploaded anywhere. The models run on your machine, uploads stay in memory, and your bank statement never leaves your laptop.
What it tells you
| Result | Meaning |
|---|---|
| Verified | The UTR is in your statement with the same amount. The money arrived. |
| Amount differs | The UTR is there, but the screenshot |
How I Built It
screenshot ──► RapidOCR ─────► exact characters ─┐
│ ├─► cross-check ─► receipt (high / low confidence)
└───────► Gemma 3 4B ───► what they mean ───┘ │
bank statement (CSV / XLS / XLSX) ──► credits with UTR, payer, UPI id ──► reconcile ─► report
The hard part is reading the screenshot. A typical payment screen has a phone number, a transaction ID, a date and the UTR all in one place, and a phone number with the 91 prefix is also 12 digits.
I ended up using two open models and making them check each other.
RapidOCR, which runs PaddleOCR's models on onnxruntime, reads the text exactly, character by character. It has no idea which number is the UTR, though.
Gemma 3 4B, running locally in Ollama, looks at the same screenshot and understands it. It can tell the UTR from the phone number, and "Payment Successful" from "Payment Failed". What it can't always do is read twelve small digits perfectly.
So neither of them gets the final say. When Gemma says "this is the UTR", ProofCheck only accepts it if OCR saw exactly the same digits:
def _fuse_utr(lines, vision_utr, issues):
seen_by_ocr = {c.value for c in utr_candidates(lines)}
claimed = re.sub(r"\s", "", vision_utr or "")
if claimed in seen_by_ocr:
return claimed, True # both models agree
fallback = _ocr_utr(lines) # a single labelled OCR candidate, or nothing
if claimed:
issues.append("The vision model's UTR was not confirmed by OCR; please check it.")
if fallback is None:
issues.append("Could not tell which number is the UPI reference.")
return fallback, False
The amount, the payment status and the payer's name go through the same kind of check. A screenshot only gets "high confidence" when both models agree and the screen itself says the payment went through. Anything else gets flagged for a person to look at, with the reason written out.
The models only read. Deciding whether something is verified, a duplicate or missing is ordinary Python with tests around it. I didn't want an LLM making the call on whether someone paid.
Bank statements were their own small adventure. There's no UTR column. Each bank stuffs it into a free-text description, and each one does it differently: SBI writes UPI/CR/<utr>/<name>/..., HDFC writes UPI-<name>-<upi id>-<ifsc>-<utr>. ProofCheck works out the bank from the header row, skips the address and account-number lines at the top, and reads CSV, XLSX and the old XLS format that some banks still hand out.
Around that sits a FastAPI backend that streams results back as they're ready, a React and Tailwind frontend, and Decimal everywhere money shows up.
Things that bit me
I'd written "nothing is written to disk" in the README before checking it, and it wasn't true. I ran a security review on the upload code, and the first finding was that FastAPI's file uploads let Starlette save anything over 1 MB to a temp file. Most screenshots and bank statements are over 1 MB. I swapped in a parser that keeps everything in memory and added a test that fails if Python's tempfile is ever touched. The review found 10 problems in total, including a DNS rebinding hole against localhost. They're all fixed now, but that first one stung.
Screenshots can carry prompt injection. Text inside an image can nudge a vision model into calling a failed payment a success. That's why the status and the payer name also have to match what OCR read, not just the UTR.
Then there was capital I versus lowercase l. Gemma read "Meera Iyer" correctly, OCR read "Meera lyer", and my strict name check threw out the right answer. I only caught it by running the whole thing on realistic data, which is a decent argument for doing that early.
Why Does Open Innovation Matter?
Without open models, this project doesn't really work.
The input is someone's bank statement plus other people's names, UPI IDs and payment history. I wouldn't upload that to a third-party API, and I don't think a tuition teacher should either. With open weights everything stays on the laptop. The OCR models come bundled inside the Python package, Gemma runs in Ollama on localhost, and ProofCheck refuses to start if the Ollama URL points at another machine.
Then there's the cost. Every screenshot goes through two models. With a paid vision API that's a charge per screenshot, every month, for a customer who's paying very little. Locally, once the models are downloaded, the next screenshot costs nothing.
The cross-check depends on it too. I can run a dedicated OCR model and a vision-language model side by side and compare their answers because both run on my machine. A single hosted endpoint would give me one answer and nothing to check it against.
And once it's set up it works offline. No API key, no rate limits, no pricing page that can change under me.
Prize Categories
- Best Use of Gemma: Gemma 3 4B runs locally in Ollama and reads every payment screenshot. ProofCheck checks its answers against OCR before trusting them.
A few DEV posts I read along the way: I ran OCR in the browser so receipt images never leave the device has the same privacy instinct, How to Automate Invoice Reconciliation covers matching from the business side, and What Really Happens When You Make an Online Payment? is a good explainer on where payment references come from.
Top comments (0)