ReceiptRecall: My Groceries Got Expensive, So I Gave My Receipts a Memory

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

Started as: "what did I pay for milk last time?" Ended as: a local-AI pipeline, one production-only crash, a format war with my own iPhone, and a language model confidently inventing a Big Mac that was never on the receipt.

What I Built

Somewhere between the cashier and my front door, my brain deletes the price of milk.

It keeps song lyrics from 2020. That, apparently, is essential infrastructure. The price of soap from last week? Evicted without notice.

That gap is ReceiptRecall: a receipt scanner that turns the slip of paper from the till into something useful after you leave the shop.

The loop is deliberately small: photograph a receipt → check the extracted draft → confirm → keep a purchase history you can actually use. No barcode-by-barcode scanning. No spreadsheet guilt.

It exists to answer ordinary questions without an archaeological dig through my handbag:

• What did I pay for this last time?
• Did the brand get pricier, or did the packet quietly get smaller?
• Is the cheaper-looking option actually cheaper per litre?
• What have I spent this month?

The review step is the whole point. OCR misreads faded ink; a small model will cheerfully confuse a row total with a unit price. So you get an editable draft before anything is saved. A confident wrong total is still a wrong total — just better dressed.

What's shipped: photo upload (including iPhone HEIC), OCR + AI draft, review-and-correct, atomic confirm, and a spending history. Pantry stock and the shopping list are designed and in the schema, but I deferred them to keep the first version honest about what works today.

Who would pay for it?

My hypothesis: a household that shops regularly and wants a private receipt archive with real price history — paid value in backup, shared access, or a supported self-hosted install. That's a hypothesis to test, not proof of demand. A subscription button can't tell me whether reviewing a scan actually beats typing.

Demo

Run it locally (the flow I verified end to end):

git clone https://github.com/CynthiaWahome/receipt-recall && cd receipt-recall
npm install
ollama pull gemma3:1b          # local, open-weight, no API key
npm run dev                    # http://localhost:3000

Then: Scan Receipt → upload a photo → Extract with OCR + AI → correct the rows → Confirm → check History.

Code

ReceiptRecall on GitHub

Stack: Next.js 16 + TypeScript, SQLite (better-sqlite3), Tesseract CLI OCR, Gemma 3 1B via Ollama, Zod, sharp. Receipt images live in private storage, never a public folder; money is stored in integer minor units so no float ever rounds my milk into oblivion.

How I Built It

The architecture separates reading a receipt from trusting it:

Photo → OCR text → structured AI draft → human review → confirmed history.

Tesseract turns the picture into text. Gemma 3 1B (open weights, run locally through Ollama) organises that text into fields. Gemma is text-only here — nobody is asking a text model to squint at a JPEG and pray. Output is constrained with Zod + Ollama's JSON Schema format, which enforces shape, not truth. Hence the review step.

Then reality showed up, in four acts.

1. The scanner that couldn't scan. The OCR and extraction code existed, got merged, then got reverted — because the one thing it depended on, actually uploading a photo, was never built. So the model had nothing to read and looked broken. Lesson I keep re-learning: build the boring plumbing before the clever part.

2. My iPhone declared format war. Every photo is HEIC, and HEIC hides its pixels behind HEVC compression that sharp's bundled libheif can't decode. It read the metadata fine and then failed on the actual image — the most polite kind of failure. Fix: a pure-JS HEIC→JPEG step before sharp, so it works on my laptop and in Docker without system libraries. Bonus trap: newer iPhones shoot 48MP, which blew past my pixel cap. The cap was guarding against decompression bombs, not punishing people for owning a good camera.

3. The crash that only existed in production. Locally, flawless. Built for deploy, every write died with no such table: receipts. The database schema was loaded from a file via __dirname, which the bundler rewrites to a path that doesn't exist in the server output. Inlined the schema; crash gone. A green local run proves exactly nothing about the bundle.

4. Gemma invented a Big Mac. On a real Pandamart receipt — rope, sponge cloth, towels — the model returned four line items all named "Big Mac," mangled the quantities, and filed the discount as a product. This is not a bug to patch away; it's a 1B model being a 1B model. It's also the entire argument for the review screen. The AI drafts; you confirm. I made the draft schema permissive (so one weird field doesn't nuke the whole extraction) and kept the strictness at confirm, where it belongs.

The arithmetic stays in plain code, where it's boring and testable: two 1L packets at 260 total = 130/litre; one 500ml packet at 70 = 140/litre. The smaller sticker loses. Money in integer minor units, row totals kept distinct from unit prices, units and currencies never mixed, confirm wrapped in one idempotent transaction so a double-tap can't buy your milk twice. 39 unit tests guard the maths; I verified the upload→OCR→draft path end to end on a synthetic receipt and a 48MP HEIC, plus duplicate-upload and cross-origin rejection.

Why Does Open Innovation Matter?

A receipt quietly reveals where you shop, when, and what you buy. I wanted to control where that gets processed.

With OCR and inference running locally, extraction can stay on my machine after the initial downloads. Host it on a server instead and the photos travel there — so I say it plainly: self-hosted is not the same as "never leaves your phone." The privacy claim has to match the deployment.

Open made three things real:

1. Control over processing — local inference instead of a third-party API per receipt.
2. Room to fix the weak link — I can swap OCR engines or models against the same test receipts without rebuilding the product around one vendor's response format.
3. A different cost curve — no per-token billing. Hardware, power and hosting still cost money, so I won't call it free to run.

Credit where due: Tesseract is open-source software; Gemma ships open weights under its own model terms. Different licences, both worth reading.

Prize Categories

• Best Use of Gemma: Gemma 3 1B (open weights, local via Ollama) is the extraction brain — it turns numbered OCR lines into a structured, schema-validated draft, and its honest roughness is exactly why the human-in-the-loop review is the product, not an afterthought.