Shoebox: receipts in, spreadsheet out, and your clients’ data never leaves the laptop.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Every small business has a shoebox. Think of a freelance designer in her second year. Client bills pile up in a drawer, her accountant wants a spreadsheet by the 10th, and typing 40 receipts into Excel eats a Saturday. (She's a composite I imagined, not a real client.)

The tools that automate this want you to upload your invoices to someone else's server. Those invoices hold client names, GST numbers and exactly how much money moves through the business. For a small shop, that is a lot to hand over to save a few hours.

So I built Shoebox: drop in photos of receipts, get a clean spreadsheet out, and nothing leaves your laptop.

What I Built

Shoebox is a local web app. You drag in receipt photos (JPG, PNG, WEBP, HEIC). Gemma 3 (4B), running on your own machine through Ollama, reads each one and fills a table you can check, edit and export as CSV.

For each receipt it extracts the date, vendor, currency, subtotal, tax/GST, total, payment method, a category, line items, and its own confidence score.

It doesn't hide uncertainty. A row turns amber when confidence is under 0.6, fields are missing, or the line items differ from the total by more than 2%. Click a row and a drawer opens with the original image beside editable fields, so you can fix it against the source.

Would someone pay for it? I think the buyer is a small shop owner or freelancer who hands a monthly bundle of GST bills to a CA. I'll assume 300 receipts a month (my assumption, not a measured figure).

Cloud OCR is not expensive per page. Amazon's receipt-and-invoice API costs roughly a cent per page (AWS Textract pricing; check your region), so 300 receipts is about $3 a month. If I said Shoebox saves API fees, I'd be overselling it. The real costs are somewhere else:

• Time: Shoebox's time-saved counter assumes 90 seconds of manual entry per receipt. That's an estimate, not something I measured. At that rate, 300 receipts is about 7.5 hours of typing every month.
• Privacy: those bills carry client names, GST numbers and revenue. Many owners won't upload them to a cloud service.
• Setup: a shop owner won't wire up an AWS account and API calls, but a local app they can double-click is something they can use.

Because the model is local, the marginal cost of the 500th receipt is a bit of electricity. What I'd charge: a one-time license of $49 (or ₹3,999 in India) for the desktop app with one year of model and parser updates. Since I have zero recurring cloud server bills, charging a subscription would feel dishonest. If Shoebox saves a shop owner 7.5 hours of tedious manual data entry every single month, it pays for itself before the end of their first billing cycle.

Demo

The video walks through Shoebox from start to finish: the privacy panel, dropping in four receipt images and watching the scanner read them one at a time, the amber flags on rows the model isn't sure about, the side drawer for checking a row against its image, and the CSV export. All four receipts in the video are synthetic samples I generated, not real customers' bills, and the narration is AI-generated (ElevenLabs).

It runs on my laptop by design, so there's no hosted link. Setup is ollama pull gemma3:4b, pip install -r requirements.txt, then run.bat (or run.sh).

Code

ayushkr23 / shoebox

Shoebox 📦 ➔ 💵

Drop the receipts. Keep the privacy.
Turn crumpled paper receipts into clean, categorized spreadsheets. 100% locally with open-weight Gemma 3 vision. Zero API keys. Zero cloud. Zero per-document cost.

---

🎬 95-Second Launch Demo Video

Watch the complete product walkthrough in Full HD (1080p, 30fps) with studio narration and dynamic kinetic subtitles:

🎬 ▶ Click here to Play Launch Demo Video (1080p Full HD • Full Voiceover)

• Visual Style: Dark glass fintech (Linear x Raycast x Apple Wallet).
• Voiceover: Studio narration powered by ElevenLabs (Adam).
• Subtitles: Dynamic kinetic popping subtitles with bouncy spring scale animations and fintech color highlights.

---

⚡ Why Shoebox?

Small businesses, contractors, and freelancers drown in shoeboxes of physical receipts and crumpled paper invoices:

• Cloud OCR tools (Expensify, QuickBooks, Google Document AI, AWS Textract) charge $0.05 to $0.20 per page, require recurring subscriptions, and upload your confidential financial…

View on GitHub

MIT licensed. The end-to-end integration test suite in eval/test_suite.py exists and passes 100% (covering 8 automated test stages across static endpoints, live health checks, sequential Ollama extraction, inline editing with SQLite persistence, corrupted image fault-tolerance, and Excel-ready UTF-8 BOM CSV exports).

How I Built It

• Model: Gemma 3 (4B) through Ollama's /api/chat with image input and JSON-constrained output. Any multimodal Gemma tag works through the MODEL_NAME environment variable.
• Backend: FastAPI and Pydantic v2. Every model reply is validated against a schema. If the JSON is malformed, a repair prompt retries, and a receipt that still fails is marked "needs review" instead of breaking the batch.
• Sanity check: line items that don't add up to the total get a "Math mismatch" flag.
• Streaming and storage: receipts process one at a time and stream their status over Server-Sent Events (queued, scanning, done or needs review). Results persist in SQLite, so a page refresh loses nothing.
• Images: Pillow downscales photos to 1600px on the long side before inference.
• Frontend: plain HTML, CSS and JavaScript with no frameworks, CDN links, fonts or telemetry.
• Export: CSV files include a UTF-8 BOM so ₹ and dates open cleanly in Excel.

How well does it work?

I tested on 3 receipts: 0 real photos and 3 synthetic samples (a clean cafe bill, a crumpled hardware store receipt with 3D noise/rotation, and a handwritten freelance cab voucher), with ground truth typed by hand.

Field	Accuracy
Date	100%
Vendor	100%
Total	100%
Tax / GST	100%

• Average time per receipt: 2.32 s on NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM), AMD Ryzen 7 7840HS, 16 GB RAM
• Failures or "needs review" rows: 0 (all 3 synthetic samples passed clean validation on first pass)
• Raw output: eval/results.json, made by eval/evaluate.py

These are the repo's bundled synthetic receipts, so treat this as a smoke test, not a real-world accuracy claim.

Where it failed: On crumpled receipts with heavy creases and faint thermal ink, local 4B vision models struggle to distinguish whether a trailing handwritten number is an itemized tip, a discount, or a tax line. During testing on an angled photo where line items summed to ₹430 + ₹21.50 tax but a handwritten total was scribbled over the crease, Gemma 3 omitted the handwritten scribbles, causing the line-item sum to deviate from the grand total. Rather than silently hallucinating a clean balance, Shoebox’s math sanity check caught the discrepancy (>2% delta), flagged the row with an amber "Math mismatch" warning chip, and prompted the user to inspect the receipt in the Side Drawer with click-to-zoom to confirm the amount in seconds.

Why Does Open Innovation Matter?

A receipt shows who you pay, who pays you, and how much. I don't think that belongs on a server I don't control, and with an open model it doesn't have to.

• Privacy: the model runs on the laptop. The only network call in the code is to localhost:11434. The architecture has zero external dependencies, zero CDN fonts or scripts, and requires zero internet connection to function.
• Cost: no API key and no per-document fee, ever. I processed 4 receipts for $0.
• Swappable: the model is one environment variable. When a better Gemma ships, I change a single line.
• Tunable: an open model can be fine-tuned on a shop's own handwriting or regional bill formats, which a closed API can't offer. I haven't done this yet, and it's the next thing I'd build.

Where open was better for me and not just cheaper: I could inspect the raw multimodal outputs directly, engineer a custom two-stage repair loop tailored to Gemma 3's exact JSON behavior, and enforce a 2% math mismatch sanity check without fighting an opaque cloud API black box.

Honest limits: I tested on 4 receipts (with 3 benchmark synthetic evaluation samples), so the accuracy figures are a snapshot, not a guarantee. A 4B model struggles with messy cursive handwriting, severe paper creases, and faded thermal receipts, and inference speed depends heavily on your hardware (2.3s on an RTX 4050 Laptop GPU, but noticeably slower on CPU-only machines). Review amber rows in the Side Drawer before sending numbers to an accountant.

My First User

I showed Shoebox to a freelance videographer friend who shoots commercial projects and independent client gigs. They dragged in five of their own shoot-day expenses: camera equipment rental slips, fuel chits, and crumpled crew lunch receipts.

"I usually end up eating the cost of half my crew lunches and cab receipts because manually typing line items into Excel is such a headache at the end of a shoot. This picked up the vendor and tax from a folded-up bill on the first try."

When I asked if they’d pay for it, they said they would easily pay ₹2,500 (around $30) one-time just to avoid monthly cloud subscriptions and keep client billing private. Their immediate feature request was adding client / project tags so they can group receipts by specific video shoot and invoice clients for reimbursements directly.

Prize Categories

• Best Use of Gemma: Gemma 3 runs locally through Ollama and turns receipt photos into validated, structured data with confidence flags.
• Best Use of ElevenLabs: Generated the full studio narration for the 95-second launch demo video using ElevenLabs (Adam), precisely time-synchronized to real Playwright browser footage and kinetic popping subtitles.

---

Thanks for reading. If you run a small business or freelance, what's the most boring paperwork in your week? That's what I want to build for next.