receipt2bean: turning my partner's receipt photos into Beancount with a local vision model

MinchinWeb — Mon, 05 Oct 2026 06:09:48 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My business partner keeps the receipt for everything she spends for the business. She photographs them on her phone, and at the end of the year I have to sit down and type them into my books by hand. I keep the books in Beancount, a plain-text accounting system. She has been clear that she will not learn plain-text accounting.

So I built receipt2bean. She sends me the photos. A script reads each photo with a local vision model and writes a ready-to-review Beancount transaction, with the GST split out. I check the entries; she never has to see Beancount.

Demo

Here is a screen recording of the project in action locally:

Sample output. GST doesn't match 5% of the total due to a tip.

2026-10-03 ! "CORNER CAFE" ""
  receipt: "03_corner_cafe_tip_photo.jpg"
  receipt-sha: "73f3864916bb"
  scanned-by: "qwen3-vl:8b"
  Expenses:Meals                                 20.50 CAD
  Assets:GST-Receivable                           0.93 CAD
  Liabilities:CreditCard                        -21.43 CAD

Code

minchinweb / hacktoberfest-2026-0

Weekend Challenge: Build for a Friend

receipt2bean

Photograph a receipt, get a Beancount transaction.

Built for the Hacktoberfest 2026 Weekend Challenge ("Build for a Friend") for my business partner, who photographs receipts but doesn't want to learn plain-text accounting. A local open-weight vision model reads each photo. Plain Python then checks the arithmetic and writes the entry. Nothing leaves your machine.

State: Proof of concept; works with provided sample datea. Much more customization is likely needed, and I'd like to eventually have a decent test suite too.

receipt photo --> open-weight vision model (Ollama) --> JSON
              --> Python: validate totals + GST, map accounts
              --> staging.bean  (pending "!" entries, reviewed before they join the ledger)

Design choices

The model reads, code checks. The model returns JSON constrained to a schema. Then Python verifies subtotal + GST = total, that line items add up, that GST is plausible (about 5%), and that the date is…

View on GitHub

How I Built It

The model is Qwen3-VL 8B, an open-weight vision-language model running locally through Ollama.

The pipeline has four steps:

Read. The photo goes to the model with a JSON schema (merchant, date, line items, subtotal, GST, total, payment method). Ollama's structured output keeps the reply in that shape.
Check. The model reads; plain Python checks. It verifies that subtotal + GST = total, that the line items add up, that the GST is plausible for a 5% tax, and that the date makes sense. If something is off, the entry gets a ; REVIEW: comment instead of a silent "fix".
Categorize. A small rules.toml maps merchant names to my accounts. I deliberately don't let the model invent accounts in my ledger.
Stage. Entries are written as pending (!) to a staging file. Each photo's hash is stored, so running the script twice never duplicates anything. The expense posting is computed as total minus GST, so every entry balances.

I tested it on 1 real receipt and 3 synthetic receipts (hard to share real data without giving up the privacy that this whole project is about...). Results: Two were fully correct; two failed due to OCR issues: the real one, from Staples with four individual "sub-invoices" and the "Maple Print" synthetic one that inherited the format (and I ran out of time to fix them).

Why Does Open Innovation Matter?

Receipts are a surprisingly complete record of someone's life: where we shop, what we buy, when we're away, what we pay. For something like that, I didn't want a cloud API in the middle if it can be helped.

Privacy. Every photo is processed on my own computer. Nothing is uploaded, and the model sees receipts only while it's reading them.
No per-receipt cost or account. A year of receipts is hundreds of photos. Running a model locally means no meter running and no API keys for my business partner to worry about.
I can see and control the pipeline. Because the model is swappable (--model), I can try a different open-weight one without rewriting anything; hopefully newer, better ones will keep coming out! Because the checks around the model output are ordinary Python, I can test them.
Failures are fixable. Some of the current receipts failed. I can either fix the code once and every future receipt from that vendor should work, or still input them by hand, which is no worse than what I started with.

What's next

A synced phone folder so receipts are scanned as she takes them
Matching staged entries to my bank-import transactions
Plugging into my GST tracking so the GST return worksheet builds itself
Splitting multi-category receipts by line item

Thanks for reading. Next step: see what my business partner thinks of this in the morning!

My Agent Session

I'm not sure how to export the full transcript, but these are the three prompts I used to have it built for/with me.

Could we build a receipt scanner that would create beancount transactions?
What open Source [sic] model could we use at the core?

Let me [sic] friend be my [business partner], who otherwise I have to enter these myself at the end of the year; she doesn't want to deal with beancount.

Can you create a first pass version of the code, a readme, instructions on how to put together the screen recording, and draft the post about it. Please ask any questions you need addressed to move forward

Can you generate a sample receipt? You can base it on this if it helps [real sample Staples receipt attached]

Author's Notes

N.B. Yes, this is very heavily written (code and documentation and the first draft of this post) using AI, but I wanted to test how well I could (nearly) one-shot this. It helps that the problem is small and well understood by me. The open question is what happens when it breaks (which it already has...): Can the AI fix it? Can I fix it "manually" if the AI fails? But as an experiment, I say it passes.

DEV Community: MinchinWeb