receipt2bean: turning my partner's receipt photos into Beancount with a local vision model

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My business partner keeps the receipt for everything she spends for the business. She photographs them on her phone, and at the end of the year I have to sit down and type them into my books by hand. I keep the books in Beancount, a plain-text accounting system. She has been clear that she will not learn plain-text accounting.

So I built receipt2bean. She sends me the photos. A script reads each photo with a local vision model and writes a ready-to-review Beancount transaction, with the GST split out. I check the entries; she never has to see Beancount.

Demo

Here is a screen recording of the project in action locally:

Sample output. GST doesn't match 5% of the total due to a tip.

2026-10-03 ! "CORNER CAFE" ""
  receipt: "03_corner_cafe_tip_photo.jpg"
  receipt-sha: "73f3864916bb"
  scanned-by: "qwen3-vl:8b"
  Expenses:Meals                                 20.50 CAD
  Assets:GST-Receivable                           0.93 CAD
  Liabilities:CreditCard                        -21.43 CAD

Code

minchinweb / hacktoberfest-2026-0

Weekend Challenge: Build for a Friend

receipt2bean

Photograph a receipt, get a Beancount transaction.

Built for the Hacktoberfest 2026 Weekend Challenge ("Build for a Friend") for my business partner, who photographs receipts but doesn't want to learn plain-text accounting. A local open-weight vision model reads each photo. Plain Python then checks the arithmetic and writes the entry. Nothing leaves your machine.

State: Proof of concept; works with provided sample datea. Much more customization is likely needed, and I'd like to eventually have a decent test suite too.

receipt photo --> open-weight vision model (Ollama) --> JSON
              --> Python: validate totals + GST, map accounts
              --> staging.bean  (pending "!" entries, reviewed before they join the ledger)

Design choices

• The model reads, code checks. The model returns JSON constrained to a schema. Then Python verifies subtotal + GST = total, that line items add up, that GST is plausible (about 5%), and that the date is…

View on GitHub

How I Built It

The model is Qwen3-VL 8B, an open-weight vision-language model running locally through Ollama.

The pipeline has four steps:

1. Read. The photo goes to the model with a JSON schema (merchant, date, line items, subtotal, GST, total, payment method). Ollama's structured output keeps the reply in that shape.
2. Check. The model reads; plain Python checks. It verifies that subtotal + GST = total, that the line items add up, that the GST is plausible for a 5% tax, and that the date makes sense. If something is off, the entry gets a ; REVIEW: comment instead of a silent "fix".
3. Categorize. A small rules.toml maps merchant names to my accounts. I deliberately don't let the model invent accounts in my ledger.
4. Stage. Entries are written as pending (!) to a staging file. Each photo's hash is stored, so running the script twice never duplicates anything. The expense posting is computed as total minus GST, so every entry balances.

I tested it on 1 real receipt and 3 synthetic receipts (hard to share real data without giving up the privacy that this whole project is about...). Results: Two were fully correct; two failed due to OCR issues: the real one, from Staples with four individual "sub-invoices" and the "Maple Print" synthetic one that inherited the format (and I ran out of time to fix them).

Why Does Open Innovation Matter?

Receipts are a surprisingly complete record of someone's life: where we shop, what we buy, when we're away, what we pay. For something like that, I didn't want a cloud API in the middle if it can be helped.

• Privacy. Every photo is processed on my own computer. Nothing is uploaded, and the model sees receipts only while it's reading them.
• No per-receipt cost or account. A year of receipts is hundreds of photos. Running a model locally means no meter running and no API keys for my business partner to worry about.
• I can see and control the pipeline. Because the model is swappable (--model), I can try a different open-weight one without rewriting anything; hopefully newer, better ones will keep coming out! Because the checks around the model output are ordinary Python, I can test them.
• Failures are fixable. Some of the current receipts failed. I can either fix the code once and every future receipt from that vendor should work, or still input them by hand, which is no worse than what I started with.

What's next

• A synced phone folder so receipts are scanned as she takes them
• Matching staged entries to my bank-import transactions
• Plugging into my GST tracking so the GST return worksheet builds itself
• Splitting multi-category receipts by line item

Thanks for reading. Next step: see what my business partner thinks of this in the morning!

My Agent Session

I'm not sure how to export the full transcript, but these are the three prompts I used to have it built for/with me.

Could we build a receipt scanner that would create beancount transactions?
What open Source [sic] model could we use at the core?

Let me [sic] friend be my [business partner], who otherwise I have to enter these myself at the end of the year; she doesn't want to deal with beancount.

Can you create a first pass version of the code, a readme, instructions on how to put together the screen recording, and draft the post about it. Please ask any questions you need addressed to move forward

Can you generate a sample receipt? You can base it on this if it helps [real sample Staples receipt attached]

Author's Notes

N.B. Yes, this is very heavily written (code and documentation and the first draft of this post) using AI, but I wanted to test how well I could (nearly) one-shot this. It helps that the problem is small and well understood by me. The open question is what happens when it breaks (which it already has...): Can the AI fix it? Can I fix it "manually" if the AI fails? But as an experiment, I say it passes.

Comments

[Talha Tahir]

Go to Agent Sessions and upload your session (Claude Code, Open Code or whichever one you are using) and export that. On OpenCode, you use "/export", and it exports the entire session in JSON. Upload that JSON

[MinchinWeb]

I'm not sure where to find them from Claude desktop...? (And it's not on the folder listed on the upload page; that folder is empty)