Receipt Shoebox: The Laziest Way to Do Your Expenses, Powered by a Local Gemma 4

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

My receipts have always lived in three places: crumpled in a jacket pocket, fading in my wallet, or buried in my camera roll between photos of my lunch. Then tax season comes, and I sit down with a spreadsheet and type them in one by one.

Receipt Shoebox fixes that the sloth way: do as little as possible, whenever you get around to it, and let an open model do the rest.

On the human side, the whole workflow is two steps:

1. Snap a receipt and save it to one Google Drive folder, with the Drive app you already have on your phone. There's no new app to install.
2. Whenever you feel like it (the next day, next month, or the night before taxes are due), open Receipt Shoebox on your laptop and click Refresh.

That's it. Behind that button, Gemma 4 runs locally through Ollama and reads every new photo. It pulls out the merchant, date, subtotal, tax, tip, total, currency and category. Then the app:

• Renames the photo so you can find it again. IMG_2231.HEIC becomes 2026-10-14_Trader-Joes_42.17.jpg.
• Files it in Drive under Sorted/2026/10/, so your Drive folder turns into an organized archive without any effort from you.
• Adds a row to a local database that exports to .xlsx. You get a By Category sheet for tax season, and every row links back to the original photo.
• Flags anything it isn't sure about and moves it to a Review queue, where fixing a receipt takes one keystroke.
• Reads you a summary every Sunday evening: a three-sentence spending digest, written by Gemma and voiced by ElevenLabs.

It also gets lazier the more you use it. Correct "TRADER JOE'S #552" to "Trader Joe's" once and it remembers the alias. Recategorize a merchant once and every future receipt from that merchant follows. The more you use it, the less you have to touch it.

Who gets to be lazier? Freelancers and side-hustlers who need their expenses sorted for taxes, anyone trying to keep a budget without becoming their own data-entry clerk, and anyone who would rather not upload a year of their life to an AI company to get there.

Demo

Code

Farhan-Bytes / Receipt-Shoebox

Receipt Shoebox

A local web app that turns a Google Drive folder of receipt photos into a categorized expense spreadsheet. Gemma 4 runs locally through Ollama and reads every receipt; no receipt image is ever sent to a hosted AI model. Once a week, a short spending digest is written locally and read aloud by ElevenLabs.

Setup

ollama pull gemma4:e4b
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env               # optional: add ELEVENLABS_API_KEY
# put the Google OAuth credentials.json in config/
uvicorn app.main:app --host 127.0.0.1 --port 8000

Then open http://localhost:8000 and follow the first-run checklist.

Google Drive access modes

Full	Read-only
Scope	drive	drive.readonly
Reads and categorizes receipts	✓	✓
Renames files, files them into Sorted/ and Needs Review/	✓	— Drive is never changed
Delete in the app	Moves the file to Drive trash	Hides it in the app; the file stays

…

View on GitHub

How I Built It

The core: Gemma 4 on Ollama

Every receipt is read by Gemma 4 (gemma4:e4b by default), an open-weight model running on my laptop through Ollama. Gemma 4 reads images natively, so there's no separate OCR step. A photo goes in and structured data comes out.

The trick that makes this reliable is schema-constrained output. You pass Ollama a JSON schema through its format parameter, and the model's output is forced to match it, so the response always parses. The category field is an enum built from the user's own category list in Settings, so Gemma can only choose a category you actually use.

class Extraction(BaseModel):
    is_receipt: bool
    multiple_receipts: bool   # more than one receipt in the photo
    is_refund: bool           # refund or return; total stored as negative
    merchant: str | None
    date: str | None          # YYYY-MM-DD
    year_missing: bool        # the printed date has no year
    subtotal: float | None
    tax: float | None
    tip: float | None
    total: float | None
    currency: str             # ISO 4217, e.g. USD
    category: str             # enum injected from the category list in Settings
    confidence: float         # 0 to 1

resp = client().chat(
    model=model,
    messages=[{"role": "user", "content": prompt, "images": [str(image)]}],
    format=extraction_schema(categories),
    options={"temperature": 0},
    think=False,
    keep_alive="10m",  # stay loaded for the whole batch
)
ext = Extraction.model_validate_json(resp.message.content)

The prompt covers the mistakes receipt readers make most often: reading the subtotal as the total, missing a handwritten tip, guessing the wrong currency, and confidently "reading" a photo that isn't a receipt at all.

Before the model sees anything, Pillow cleans up the photo. It fixes the EXIF rotation (phones love storing photos sideways), converts HEIC to JPEG, resizes the image so the long edge is at most 1600 px, and applies autocontrast to rescue faded thermal paper. PDFs, including scans from the Drive app, are read from their first page.

Escalation instead of overspending

Gemma 4 comes in several sizes, and the app takes advantage of that. Most receipts are easy, and the small model handles them quickly. If its confidence drops below 0.7 and a larger model (gemma4:12b or gemma4:26b) is installed, the app retries once with the larger model and keeps whichever answer is more confident. Only the hard receipts, like crumpled, blurry or faded ones, pay for the slow model.

first = _call(base, image, categories, year_hint, currency)
if first.extraction.confidence >= threshold or not have_big:
    return first
second = _call(big, image, categories, year_hint, currency)
return second if second.extraction.confidence > first.extraction.confidence else first

Trust, but verify

A model can't tell when it's wrong, so plain Python double-checks every extraction:

Check	What it catches
is_receipt is true	The photo of your cat
Date parses, isn't in the future, is within 2 years	Misread dates
Total is above 0 and below a set maximum	A misplaced decimal point
Subtotal + tax + tip equals the total, within 2 cents	The subtotal read as the total
Confidence is at or above the threshold	Blurry or cropped photos
No other receipt has the same merchant, date and total	The same receipt photographed twice
No other file has the same Drive checksum	The same photo uploaded twice

If every check passes, the receipt is marked ok and filed under Sorted/YYYY/MM/. If any check fails, it goes to Needs Review/, and the reasons appear next to the image. Review is keyboard-driven: Enter approves, J and K move between receipts, and Tab moves into the fields.

The rest of the design is built to be boring in a good way:

• Money is stored as integer cents, so totals never pick up rounding errors.
• The Drive Inbox is the queue. If the app crashes mid-run, the next Refresh picks up whatever is still there.
• The database row is saved before Drive is touched, so a failed rename can be retried without running the model again.

The voice: Gemma writes, Python checks, ElevenLabs speaks

The weekly digest is where the two partner technologies meet, and I didn't want a voice confidently reading out a wrong number. So the pipeline splits the work:

1. Python computes every number. It totals the week's receipts, the spend per currency, the top categories and the largest receipt, plus how this week compares with the average of the previous four and how many receipts are waiting for review.
2. Gemma 4 writes the words. It gets only that summary JSON (no images, no line items) and writes at most three short, warm sentences meant to be heard, not read.
3. Python checks Gemma's work. It pulls every number out of the text, both digits and spelled-out ones like "eighty-five", and confirms each one appears in the summary. If Gemma made up a figure, the app throws the text away and uses a fixed template instead.
4. ElevenLabs speaks it. The text goes to ElevenLabs text-to-speech, and the MP3 plays on the Dashboard. You pick the voice from a dropdown (with a Preview button) in Settings.

A digest sounds something like this:

This week you saved 9 receipts, totaling 312 dollars. Meals took the biggest bite at 140 dollars, about a third more than a usual week. Two receipts are waiting for your review.

If there's no API key, the network is down, or the quota has run out, the digest is still saved and shown as text. The voice is a bonus and never blocks anything.

Stack

Layer	Choice
Vision and writing model	Gemma 4 via Ollama (e4b by default, 12b or 26b for escalation)
Backend	FastAPI, one Python process bound to 127.0.0.1
Frontend	Jinja templates + HTMX + Tailwind, with live progress and no build step
Database	SQLite via SQLModel
Storage	Google Drive API v3
Image handling	Pillow, pillow-heif, pypdfium2
Spreadsheet	openpyxl
Voice	ElevenLabs Python SDK
Scheduling	APScheduler (the Sunday digest, caught up on next launch if the laptop was off)

The default model runs on a laptop with 8 GB of RAM.

Why Does Open Innovation Matter?

Receipts are a diary you don't realize you're keeping. A year of them shows where you live, where you eat, which pharmacy you go to, when you travel, and how much you spend. That's financial information I don't want sitting on a server I don't control. With a hosted vision API, every single receipt image would be uploaded to one.

Running an open model locally changed what I could build:

• Receipt images never go to a hosted AI model. They stay in my own Drive (where they already were) and in a local cache. The extracted fields live in a SQLite file on my laptop.
• It works offline. Turn off Wi-Fi, click Reprocess on a cached receipt, and extraction still works. Only the Drive sync and the voice need a connection.
• It costs nothing per receipt, so I can afford to be thorough. Escalation means calling a model twice on hard receipts. Reprocessing my whole history after I change my categories is free. With a pay-per-image API, I'd be rationing calls instead of making the pipeline more accurate.
• Swapping models is a settings change. I use e4b for speed and 12b or 26b for accuracy, with the same code and the same schema. No vendor deprecation notice can break my tax records.
• I control the model's behavior. Schema-constrained decoding, temperature 0, and keeping the model loaded for a whole batch are all settings I choose. Gemma 4's weights are Apache 2.0.

To be upfront about the one hosted piece: ElevenLabs is a cloud service, and I designed around that. It only receives the three-sentence summary that Gemma has already written from totals. It never sees an image or a line item. If you turn on Hide merchant names, it doesn't see store names either. The open model does all the reading and reasoning; the hosted service just reads the summary aloud.

If you want even less exposure, there's a read-only Drive mode (drive.readonly). The app reads and categorizes everything but never changes a file in your Drive.

The Lazy Payoff

Before: dig the receipts out of pockets and the camera roll, type each one into a spreadsheet (merchant, date, amount, category), rename and sort the photos if you ever want to find them again, then total up each category at tax time.

After: take a photo, click Refresh, and press Enter on the occasional flagged receipt.

The budget builds itself while I do nothing, and on Sunday evening it tells me how my week went without my opening a single file. Saving a receipt now takes as much effort as taking a photo, and tax season takes one click on Export.

(Disclaimer - Before you trust it with your taxes: Gemma is good at reading receipts, but it still makes mistakes. The checks above catch amounts that don't add up. A receipt that only prints a total has nothing to cross-check, though, so a misread total can still be marked ok. Before you budget or file from the export, compare it with your receipts. Every row links to its photo, so it only takes a few minutes. Receipt Shoebox organizes your receipts; it isn't tax or accounting advice.)

Prize Categories

• Best Use of Gemma: Gemma 4 is the core of the project. Running locally through Ollama, it reads every receipt image with schema-constrained output, escalates across model sizes based on its own confidence, and writes the weekly digest. Without Gemma, there's no project.
• Best Use of ElevenLabs: ElevenLabs gives the open-model pipeline a voice. It narrates the weekly digest that Gemma writes. Python checks the text so the voice never says a number Gemma made up, it only ever receives a short summary, and the app falls back to text if it's offline.

Happy (belated) International Sloth Day. 🦥