DEV Community: MinchinWeb

Only two lines on an Alberta power bill matter. A local model finds them.

MinchinWeb — Thu, 15 Oct 2026 22:32:08 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

What I Built

I run a small utility retailer in Alberta. Many new customer starts the same way: someone looks at their current bill, works out whether switching would save them money, and then types the site IDs, name and address into an enrolment form. A referral partner can bring in hundreds of homes at a time, and many of those homes have two sites, one for power and one for gas. That is a lot of squinting at photographs of bills.

bill2quote turns a photo or PDF of a utility bill into two things:

A one-page savings comparison the customer can read: what this bill cost on their current plan, what the same usage would cost on our offer, and the difference. If their current plan is cheaper, the page says that, plainly.
An enrolment-ready row with site ID(s), service address, name and current retailer, plus the partner's referral code.

A vision model running on my own machine reads the bill. Plain Python checks what it read, does every calculation, and writes the outputs. A person confirms the numbers on a review screen before any customer-facing quote exists.

The insight that makes this tractable: of the dozens of lines on an Alberta bill, only two belong to the retailer, the energy charge and the admin fee. Delivery, riders and the local access fee are set by regulators and municipalities and are the same whichever retailer you pick (Utilities Consumer Advocate, "How to read your bill"). So the comparison is two lines, plus GST on the difference. The arithmetic is easy. Reading the bill is the hard part.

Demo

The clip is the real flow on my Pixel 7, with the slow parts sped up. I photograph a printed bill, a local model reads it, and the review screen shows what doesn't add up. Two red checks block the quote until I fix the numbers, then the comparison appears, and "Edit numbers and re-run" fixes a last warning. The final screen is the enrolment list.

N.B.: Every bill in this post and in the repo is synthetic. The retailer, people, addresses, account numbers and site IDs are all invented.

Code

minchinweb / hacktoberfest-2026-2

Hacktoberfest 2026 Open-Source AI Challenge, Week 2: Side Hustle

bill2quote

Hacktoberfest 2026 Open-Source AI Challenge, Week 2: Side Hustle.

A photo or PDF of an Alberta utility bill in; an honest one-page savings comparison and an enrolment-ready record out. A local open-weight vision model (via Ollama) reads the bill Plain Python checks the arithmetic, does the money math and writes the outputs. A person confirms the numbers on a review screen before any customer-facing quote exists.

Only two lines on the bill belong to the retailer: the energy charge and the admin fee. Delivery riders and the local access fee are the same whichever retailer the customer picks, so savings come only from those two lines (plus GST on the difference). If the customer's current plan is cheaper, the page says so.

Data note

Every bill in this repo is synthetic. The retailer, people, addresses, account numbers and site IDs (0010099… electricity, 0000099… gas) are fictional, and the…

View on GitHub

Python 3.11+, FastAPI, Jinja2, pypdfium2, Pillow, Ollama. No build step, no database, no hosted API. bill2quote batch DIR for the office, bill2quote serve for the phone.

How I Built It

The model reads. The code checks.

The rule I wrote at the top of the spec and kept to: the model transcribes what is printed and never calculates, converts or corrects anything. A suspect number is flagged and shown to a person, never quietly fixed.

The model gets one request per bill with every page attached, a JSON schema, temperature 0 and thinking off (to encourage deterministic processing). Every field in the schema is required but nullable, so the output is either complete or rejected, and "I couldn't read it" is an answer the model is allowed to give. That is a lesson from Week 1, where every field was required too, and where the most useful failure was a garbled odometer reading that the pipeline happily booked as a plausible 642 km trip.

Then 20 deterministic checks run, each returning a flag with a severity. A few of them:

C01: does the energy charge equal usage × rate? A small gap is a warning, since a tiered rate can explain it. More than three times apart blocks the quote, since nothing explains that.
C04: do the meter readings add up to the usage?
C06: is the rate plausible for its unit? (This one catches ¢ vs $ mix-ups and blocks the quote.)
C07 / C17: is the site ID 13 digits starting with 00 (electricity, never 000) or 000 (gas), and is it not the account number? Some account numbers are also 13 digits, so a model can easily put the wrong one in the wrong box.
C12: is anything quote-critical missing?
C20: do the lines add up? Energy charge + admin fee + delivery should equal the printed subtotal. A value read from the wrong row breaks that straight away.
C19: does the comparison itself make sense? If the two retailer lines on the offer come out three times higher or lower than on the current plan, a number was misread, and the quote is not shown.

Blocking flags stop a customer quote. Warnings must be acknowledged on the review screen with a tick box. Every edit a reviewer makes is logged with the model's value, the new value and the time.

How it did

I generated eight synthetic bills across three layouts (a one-page table, a summary plus detail page, and a two-column dual-fuel statement) with varied wording, plus a Pillow "phone photo" of one of them: rotated, in perspective, shadowed, JPEG at quality 60. Ground truth comes from the same data that rendered each bill. Two models, same prompt, same bills:

Model	Bills with every quote-critical field right	Valid output	Processing time per bill
Gemma 4 e4b	5 of 8	8 of 8	about 9s
Qwen3-VL 8b	7 of 8	7 of 8	about 73s

"Quote-critical" means usage, days, energy charge or rate, and admin fee. Field-level accuracy was 95% for Gemma and 97% for Qwen, which sounds great and is exactly the wrong number to look at: one wrong digit in the one field that matters is a wrong quote.

One failure, told straight. On an earlier version of my synthetic gas bill, Gemma put the usage (37.49 GJ) into the energy charge field. (That version had a bug of mine: the generator printed "Electricity Supply" on the energy line of a gas bill. With the label fixed, Gemma reads that PDF correctly, so a wrong label alone was enough to derail it. The misread still shows up on the photographed print, which predates the fix.) The correct charge was $183.33. A quote built from that would have told a customer they'd lose $117.70 by switching, when they'd have saved money. Nothing about it looks wrong at a glance. What caught it was check C01: charge 37.49 ≠ usage 37.49 × rate 4.89 = 183.33. (At that point C01 was only a warning; it now blocks the quote, see below.) Two more, from the current run: on the phone photo it read usage as 30.937 kWh, which looks like the previous meter reading (30,937) with the comma taken for a decimal point, and C04 flagged it; on another bill it labelled a 12.05 ¢/kWh rate as dollars, and C06 blocked the quote. Another bill came back with both site IDs one digit short: Gemma read 0010099000016 as 001009900016, and the gas ID on the same bill lost a zero the same way, apparently dropping one from a run of zeros. C07 blocked enrolment on both, with site ID is not 13 digits starting with 00. A truncated ID would be quietly accepted by any tool that only checks the field is filled in. Three of eight bills needed a human fix. That is why the review screen is the product, not an afterthought.

Real phone photos

I then printed the bills and photographed them with a Pixel 7 on a table: some tilted, one with my finger over part of the page, bill 03 as two photos. Seven bills, scored separately from the table above, which still includes the simulated photo. Same prompt, same models:

Model	Bills with every quote-critical field right	Valid output	Processing time per bill
Gemma 4 e4b	3 of 7	7 of 7	about 10
Qwen3-VL 8b	5 of 7	5 of 7	about 72

This is the hardest test so far, and Gemma lost ground: 3 of 7, against 5 of 8 on the clean PDFs. On the bills with a different label wording it often put values against the wrong row. My guess is that the tilt of the page shifts the right-hand values against the left-hand labels, but I haven't tested that. Its four failures were bills 02, 05, 06 and 08. On a second read of those four, a check fired on every one: C06 blocked the gas bill, where a usage of 37.49 had landed in the rate field, C12 blocked the one where usage came back empty, and the others raised C01, C03 or C04 warnings (for example meter reads give 0.700 kWh but usage says 700.0). Qwen got both of its misses by returning nothing usable: on bills 05 and 08 the JSON was cut off.

A warning was not enough. While recording the demo on bill 02, I fixed the fields the checks had flagged and left two that weren't, usage (the m³ figure had landed in the GJ field) and the energy charge, then ticked the box and confirmed. The page cheerfully said my current plan was cheaper by $4,072.21, or $47,947 over a year. C01 had only been a warning, and I had acknowledged it. So I changed two things. A C01 gap of more than 3× now blocks the quote, and a new check (C19) refuses any comparison whose two sides are more than 3× apart. The same data now gets a red field on the review screen and no quote. And the quote page has an Edit numbers and re-run link that reopens the form with the saved values and the offer choices, so fixing a number and re-running the comparison takes seconds. The new blocks can be overridden, since a customer on a very low promotional rate really could be more than 3× away, but only with a written reason that is saved with the bill, and the override stops applying the moment a number or the offer changes.

Qwen is ahead on both sets, but it always thinks (the think: false flag has no effect on it), it is about eight times slower, and one of its eight PDF outputs failed to validate (two did in the previous run, on different bills). I suspect the thinking ate the whole 8,192-token budget, but I haven't proved it. I kept Gemma as the default, but less comfortably than before: Qwen is two bills ahead on both sets (7 of 8 against 5 of 8 on the PDFs and the simulated photo, 5 of 7 against 3 of 7 on the real photos), though that is still a small sample and it is 8 times slower. Gemma is fast enough to use at a kitchen table, and when it is wrong the checks have caught it every time I looked. That is the trade I'm making, and more bills could change it. The reader is one config value (models.extract) and bill2quote eval --model X scores any Ollama vision model on the same bills, so swapping in a better model when one is released, or a locally fine-tuned one, means changing a setting and re-running the eval, not touching the code.

Honest limits

Every bill here is synthetic, printed and photographed. None is a real customer's bill. Real retailer layouts are the next test -- all different layouts, not nearly as clear (colour and contrast), extra text (up ot 7 pages each) -- and I expect them to be harder. Seven photos is a small sample. The photos are in samples/photos/, and the results are in docs/eval/photos/.
The printed bills I photographed were made before I fixed the generator's "Electricity Supply" label on gas bills, so photos 02 and 08 carry the old label. The PDF eval uses the fixed bills.
A frontier hosted model would probably read a blurry phone photo better. That is a real cost of staying local.
The 12-month estimate for gas is a rough days-scaling, labelled as such on the page, until I add heating-degree-day data. Per-bill numbers get top billing for that reason.
The offers in the demo are made-up round numbers. This is not a statement about any real price.

Why Does Open Innovation Matter?

The data is sensitive in a specific way. A bill carries a name, a service address, an account number, and a site ID. A site ID is the "address", the identity in the market and most of what's needed to enrol a site. Sending a stack of those to a hosted API is the wrong default for a business that handles thousands. Here the bills never leave the machine.
No meter running. Reading a thousand multi-page bills costs nothing per page, and re-reading them after a prompt change is free. The eval above would have been a line item on an API invoice.
The model is a config value, and I can measure it. Swapping Gemma for Qwen was one flag, and the table above is the result. The choice of model rests on numbers, not on a vendor's say-so.
Room to specialise. Alberta bills are a narrow layout family. That is the kind of place a small model could later be fine-tuned. Not this week, but it's possible, which is the point.

My Agent Session

I built this with Claude Code from a written spec, in two sessions. The first took the spec to a working pipeline: scaffold, checks, ingest, extraction, quote math, the web review flow and tests.

bill2quote, part 1: from a written spec to a working pipeline in one session

You

Build the @spec.md

Agent

I'll start with M0 (scaffold), then M1, following the kickoff prompt. First I'll check the repo and toolchain.

▸ PowerShell ls -Force; python --version; pip list

Input

ls -Force; python --version; pip list

Output

Python 3.15.0; repo has .git, README.md, SPEC.md

Writing the first-commit files: .gitignore (per rule 3), licence, and pyproject.toml.

▸ Bash mkdir -p src/bill2quote/web/templates prompts samples tests docs; cat > .giti...

Input

$ mkdir -p src/bill2quote/web/templates prompts samples tests docs; cat > .gitignore

Output

(no output)

Dependencies installed. Now the core: schema and config.

▸ Bash cat > src/bill2quote/__init__.py, schema.py, config.py

Input

$ cat > src/bill2quote/__init__.py, schema.py, config.py

Output

(no output)

Now the deterministic checks (§9).

▸ Bash cat > checks.py <<'EOF'

Input

$ cat > checks.py <<'EOF'

Output

Exit code 2: unexpected EOF while looking for matching quote

The shell choked on that heredoc, so I'll use the Write tool for the files instead.

▸ Write src/bill2quote/checks.py

Input

src/bill2quote/checks.py

Output

File created successfully

Now tests for the checks (pass, fail, and missing-value cases).

▸ Write tests/test_checks.py

Input

tests/test_checks.py

Output

File created successfully

▸ Bash pytest -q

Input

$ pytest -q

Output

33 passed in 0.29s

33 checks tests pass. Now ingest, extract, the prompt, and the CLI with doctor.

▸ Write ingest.py, prompts/extract_v1.md, extract.py, doctor.py, cli.py

Input

ingest.py, prompts/extract_v1.md, extract.py, doctor.py, cli.py

Output

Files created

Now the synthetic sample generator: 8 bills across 3 layouts with varied wording, truth files built from the same data, plus a Pillow "phone photo" for #4.

▸ Write samples/make_samples.py

Input

samples/make_samples.py

Output

File created successfully

▸ Bash python samples/make_samples.py

Input

$ python samples/make_samples.py

Output

wrote 01_elec_clean.pdf, 02_gas_clean.pdf, 03_dual …

The generator works (it needs numpy, so I'll add that to the dev deps). Next I'll check that every truth file passes the checks as expected, then look at the output.

▸ Bash pytest -q

Input

$ pytest -q

Output

42 passed

All eight truths behave as designed. I'll look at what the bills actually look like.

▸ Read samples/04_phone_photo_dual_fuel.jpg

Input

samples/04_phone_photo_dual_fuel.jpg

Output

[image omitted]

The page content is legible, but the shadow is wrong: a hard diagonal band from the rotated gradient, not a soft shadow. I'll replace it with a smooth numpy gradient.

▸ Edit samples/make_samples.py

Input

samples/make_samples.py

Output

updated

The photo now reads like a phone shot. Next I'll clean up the lint, run the tests, and try doctor against Ollama.

▸ Bash ruff check --fix; pytest -q

Input

$ ruff check --fix; pytest -q

Output

Found 1 error (1 fixed)

Ollama isn't running here, so I can't test model reading yet. I'll commit what's done.

▸ Bash git add -A && git commit -m "M0 scaffold + M1 (partial)"

Input

$ git add -A && git commit -m "M0 scaffold + M1 (partial)"

Output

committed 2da9267

I've built M0 and most of M1, and committed it as 2da9267, but I haven't tested reading a bill with a model yet. Ollama isn't running on this machine, and bill2quote doctor fails with connection refused.

What works
- Scaffold: .gitignore, MIT licence, pyproject.toml, and a .venv. The CLI has init, doctor, read and check.
- Reading code: the schema, config loader, image/PDF ingest, and the Ollama call (schema-constrained, temperature 0, thinking off). Written but unrun against a model.
- Checks: C01–C18, each with tests for the pass, fail and missing-value cases.
- Samples: 8 bills with truth files, in three layouts with varied wording. Bill #4 is a Pillow-made phone photo of #3. Every truth file raises exactly the flags it should.

You

Ollama should now be up. Carry on

Agent

▸ Bash bill2quote doctor; ollama list

Input

$ bill2quote doctor; ollama list

Output

Ollama reachable; only gemma4:e4b is pulled

qwen3-vl:8b isn't pulled, although the spec says it is. Eval needs a second model, so I'll pull it in the background and start reading bills with Gemma meanwhile.

▸ Bash ollama pull qwen3-vl:8b (background); bill2quote read samples/01_elec_clean.pdf

Input

$ ollama pull qwen3-vl:8b (background); bill2quote read samples/01_elec_clean.pdf

Output

real 0m8.396s

Gemma read bill #1 in 8 seconds. Now eval, which scores each model field by field against the truth files, and also gives the M1 check for all 8 bills.

▸ Write src/bill2quote/evaluate.py

Input

src/bill2quote/evaluate.py

Output

File created

All 8 bills come back as valid BillFacts, which meets the M1 gate. Quote-critical accuracy is 5/8, so I'll look at what went wrong on the three misses.

Qwen failed all 8, with 0 valid reads. I'll read the actual error before treating that as a result.

▸ Bash print the error for each Qwen result

Input

$ print the error for each Qwen result

Output

model output did not match the schema: 1 validation error

Qwen returns empty content. I'll look at the raw response to see where its output is going.

▸ Bash raw request to Ollama /api/chat with Qwen

Input

$ raw request to Ollama /api/chat with Qwen

Output

content: '' | thinking: 1815 chars | done_reason: length at 637 tokens

Qwen always emits thinking (the think:false flag has no effect on it) and is cut off with done_reason: length at about 637 tokens. I'll try a bigger context and output budget.

▸ Bash same request with num_ctx and num_predict raised

Input

$ same request with num_ctx and num_predict raised

Output

51 s, done_reason: stop, content: valid JSON

That works with the bigger budget, at 51 s per bill because of the thinking. I'll make num_ctx and num_predict config values, applied to every model.

▸ Write annualize.py, quote.py, tests/test_quote.py

Input

annualize.py, quote.py, tests/test_quote.py

Output

Files created

▸ Bash pytest -q

Input

$ pytest -q

Output

49 passed

The 49 tests pass, and the hand calculation (79.32 − 64.0188 = 15.30) matches. Bill #8 is negative with the demo offers. Now the quote page renderer, enrolment CSV and batch.

▸ Write quote.html, render.py, enrol.py, store.py

Input

quote.html, render.py, enrol.py, store.py

Output

Files created

The background eval is still running. In the meantime I'll write the web review flow (M3): upload, polling, review with flags, edit log, quote and enrol.

▸ Write src/bill2quote/web/app.py

Input

src/bill2quote/web/app.py

Output

File created

flags_for is more convoluted than it needs to be. I'll make duplicate detection simply exclude the bill's own folder.

Now the templates: base styles, upload, polling wait page, review, quote bar, enrolments and login.

The shell choked on a heredoc again, so I'll write these with the Write tool. Writing each template directly.

▸ Write base.html, login.html, index.html, wait.html, review.html, quote_bar.html, en...

Input

base.html, login.html, index.html, wait.html, review.html, quote_bar.html, enrolments.html

Output

Files created

Next, a test for the whole web flow without a model: seed a bill from truth #7 (the planted error), then go through review, the edit log, quote and enrol.

▸ Write tests/test_web.py

Input

tests/test_web.py

Output

6 passed

All six pass, but two of my assertions are loose (they have or clauses that can't fail). I'll tighten them so they test what they claim.

Eval finished. Gemma is 5/8 quote-critical in 6.7 s per bill. Qwen is 6/8 but takes 78 s per bill, and 2 of its 8 outputs weren't valid. I'll see why those two failed before drawing conclusions.

The checks caught each real misread: a meter reading taken as usage, a ¢/$ unit mix-up, and a missing usage value. I'll commit, then smoke-test the live server with a real upload.

The review page renders, but the page image returns 404. I'll check on disk.

That's a real bug. This bill had already been read by batch, so upload skipped the background job, and page images are only written by that job. I'll make upload always ensure the pages exist.

I'll add a regression test for that, then restart the server and run the whole flow (review → quote → enrol) through real HTTP.

▸ Bash pytest -q; run the flow against the live server

Input

$ pytest -q; run the flow against the live server

Output

56 passed; review 200, quote 200, enrolment row written

M0 to M3 are built and committed (latest 76f2c31). All 56 tests pass. I also ran the whole flow against the real server and Gemma. The phone run on your Pixel 7 is still to do, so M3 isn't fully done.

What now works
- read, check, batch, eval and serve, along with doctor.
- Quote math: checked by hand. For bill #1 the retailer lines come to $79.32 now against $64.0188 on the offer, a difference of $15.30, which is $16.07 with GST.
- Customer view: masks site IDs and hides flags.
- Negative savings: bill #8 shows "your current plan is cheaper" with the demo offers.
- Web flow over real HTTP: upload, review, confirm, the customer quote, and an enrolment row with no account number.
- Bug found and fixed: a bill that had already been read skipped creating its page images. There's a regression test.

Model eval (8 synthetic bills)
- Gemma 4 e4b: 5 of 8 quote-critical correct, all 8 valid, about 7 s a bill.
- Qwen3-VL 8b: 6 of 8, 2 outputs invalid, about 78 s a bill.

You

Re "Batch duplicate flag: batch doesn't run the duplicate-bill check (C16). The web app does." Fix this. Also draft the Dev post

Agent

Fixing C16 in batch first. I'll also show flag IDs rather than counts in the flags column, so the duplicate flag is visible.

▸ Bash edit cli.py; pytest -q

Input

$ edit cli.py; pytest -q

Output

58 passed

▸ Write POST.md

Input

POST.md

Output

File created

▸ Bash git commit -m "POST: correct claims about the phone photo and hardware"

Input

$ git commit -m "POST: correct claims about the phone photo and hardware"

Output

committed 1ccf427

batch now runs the duplicate check, and the post is drafted in POST.md. 58 tests pass and both changes are committed.

C16 in batch
- It flags a bill whose site and period match a different bill already processed.
- On the samples it now flags #3 and #4, which are the same bill as a PDF and as a phone photo.
- A bill never flags against its own folder.
- The flags column now lists flag IDs (like C01 C16) instead of counts. There are two new tests, with no model needed.

The post
- It follows the DEV template headings in order, with the intro line and the four tags.
- The hook, the "two lines" insight, how the checks work, the eval table, one failure story told straight, limits, why open, and the Author's Note are all written from what's in the repo.
- The failure story is the Gemma gas bill. It put the usage into the charge field, which would have told the customer they'd lose $117.70 when they'd actually have saved money. Check C01 caught it.
- Marked [TODO] in the post: the demo, the first-user timings, and the real phone photos.

The second is the last two days: real phone photos, the plausibility checks, the override, and the demo.

bill2quote, part 2: real phone photos, plausibility checks and the demo

You

Fix the site ID coding to match the SPEC. An electricity Site ID will not start 000. Regenerate the synthetic bills and related testing.

Agent

Electricity site IDs must now be 13 digits starting 00 (never 000), gas IDs 13 digits starting 000. I changed the regexes in config.py and config.example.toml, made the C07 message name the right prefix per commodity, regenerated the synthetic bills, and added a test for both prefixes. 60 tests pass.

▸ Edit/Bash config.py, checks.py, make_samples.py, tests

Input

config.py, checks.py, make_samples.py, tests

Output

60 passed

You

Re-run the eval on the regenerated bills and re-read the worked examples in the post.

Agent

Eval on the regenerated PDFs: Gemma 4 e4b 5 of 8 quote-critical, 8 of 8 valid, about 9 s a bill. Qwen3-VL 8b ahead, about 73 s a bill. I re-read all three worked examples against fresh runs and they still reproduce. Added a C07 example (Gemma dropped a digit from the site IDs and C07 blocked enrolment).

You

I printed the bills and photographed them with my phone. Please remove the location from EXIF if we commit them. Crop the slippers out of two of them; the hand is fine.

Agent

All eight photos had GPS coordinates. I re-saved each from pixels only, so no EXIF or GPS survives, cropped the slippers out of photos 02 and 08, and extended the eval to accept multi-photo bills (bill 03 is two photos). My first patch to evaluate.py applied only halfway because the file uses Windows line endings, so the first eval run crashed; I fixed the loop and re-ran.

Result on 7 real photos: Gemma 3 of 7 (all valid), Qwen 5 of 7 (only 5 valid). A check fired on every Gemma miss.

You

Stick with Gemma for now, but note the project is designed so models can be swapped as better ones are released or fine tuned locally.

Agent

Added a sentence to the post: the reader is one config value (models.extract), and bill2quote eval --model X scores any Ollama vision model on the same bills.

You

Next is the GIF demo. Review my screen recording, trim it, add subtitles, make an 8 FPS GIF.

Agent

I dumped frames at a constant rate (the recording is variable frame rate, which made seeking unreliable), cut segments with speed-ups for the slow waits, covered the browser address bar, burned in captions, dropped partial and dimmed frames, and encoded at 8 fps. A first pass leaked the LAN address in a dimmed file-chooser frame and an automated scan caught it, so those frames are skipped.

You

In the recording I confirmed bill 02 and it said my current plan was cheaper by $4,072. Fix that, and allow returning to the field entry form to fix numbers and re-run the comparison.

Agent

Root cause: I had fixed the flagged fields but left a wrong usage (the m3 figure in the GJ field) and a wrong energy charge, and C01 was only a warning I had acknowledged. Changes: a C01 gap over 3x now blocks the quote; a new check C19 refuses any comparison whose two retailer-line sides are more than 3x apart; the quote page gets an 'Edit numbers and re-run' link that reopens the form with the saved values and offer choices. On the real session data the $4,072 quote is now refused and the energy charge shows a red C01.

You

Allow the review override option. Add the check on addition.

Agent

Added assess.py, shared by the web app and the CLI. Only C01-block and C19 can be overridden, a written reason is required and logged, and the override stops applying as soon as a number or the offer changes. Added C20: energy charge + admin fee + delivery must equal the printed subtotal. A stray newline from my patch script broke a string literal in app.py and I fixed it directly. 72 tests pass.

You

On Bill 2 the line reads 'Electricity Supply' but this is a gas bill.

Agent

That was a bug in my sample generator: the alternate wording reused the electricity label for the gas energy line. I gave gas its own label ('Natural Gas Supply'), regenerated, and re-ran the eval. The post now says the printed photos predate the fix. Then: fresh-clone test (install, 72 tests, doctor, serve), and found git's autocrlf had altered the PDFs, so I added .gitattributes marking them binary and verified a clean GitHub clone byte-exact.

You

Use the video times for the tool time. My manual time was 17:44 for 7 bills. I caught a $25 promotional discount and confirmed no bill carried a balance forward. Value to me is as part of a bigger CRM package.

Agent

Wrote up the first-user session in docs/first-user.md and the post: manual about 2:32 a bill; tool about 1:02 on a clean read and 3:09 on a bad one, measured to the first correct quote so enrolment is excluded on both sides. The post says plainly that the tool never reads the previous balance, and that the CRM link is not built.

You

Add a Markdown link to our Week 1 post, add the link to the GIF demo, add the agent sessions, remove the top comment.

Agent

Done in POST.md. I also corrected one sentence to match what the Week 1 post actually says (every field required; the most useful failure was a garbled odometer booked as a plausible 642 km trip).

Two moments worth pointing at. In the first session, Qwen came back with empty output; reading the raw response showed its thinking was eating the whole token budget, so the fix was a bigger budget, not a prompt change. In the second, I confirmed a bill that still had a wrong usage and energy charge and got a quote saying my current plan was cheaper by $4,072. That is where C19 and the override came from.

My First User

My first user is me, as the retailer: internal use by the buyer, which is a weaker test than an outside customer, but the retailer is the one who pays. The full notes are in docs/first-user.md.

I reviewed the seven printed bills by hand and quoted the comparison, then ran the tool on two of them. Enrolment is extra time in both cases and isn't counted in either case, since I have no way to automate it yet. All seven bills are in one basic layout with no extra text, so reading them by eye is easy and fast. That flatters the manual side.

Manual: 17 minutes 44 seconds for seven bills, about 2 min 32 s each.
With bill2quote: about 1 min 02 s for a bill that read cleanly, and about 3 min 09 s for one the model misread badly and I had to correct field by field. The mean of those two is about 2 min 06 s. Those were demo recordings, not a stopwatch session, so I'm not making a speed claim beyond this: a little faster on average, much faster when the read is clean, slower when it isn't.
What I caught that the tool didn't: a $25 promotional discount on one bill, which I need to discuss with the client, and that none of the bills carried a balance forward. A balance carried forward is a credit-risk signal, and we might not want that customer. The tool reads credits and lists them as an informational note, but it never reads the previous balance at all.
What the tool flags that I missed: I did not see any of the issues it would have flagged. On these bills that is the planted $20 error in the energy charge on bill 07 (C01), the estimated meter read on bill 05 and the credit on bill 06. I did not run bill 07 through the tool live, so that catch comes from the tests and the eval, not from the session.
Verdict as the buyer: as a stand-alone tool its value is hard to pin down. For me it would be part of a bigger package: launching a CRM. A CRM that does this wins over one that doesn't, and no CRM does it out of the box, so the value is as a differentiator inside a package deal, not a price per bill. What I expect from it: it can deal with different bill formats once properly configured, whereas this test was one format I could read by eye; a properly formatted quote for the client instead of my scribbles on their bill; and a record of each enrolment (partner code, bill, offer) that can go into a CRM for follow-up and to confirm the enrolment went through. The quote and the record exist today. The CRM connection and the success tracking do not, and neither does reading the previous balance or treating a promotional credit as a conversation with the client.

Prize Categories

Best Use of Gemma. Gemma 4 e4b is the default reader. It reads every bill in the eval to a valid result and takes about nine seconds a bill on my Windows laptop. I report where it fails and how the checks catch it, and I report that Qwen3-VL scored two bills higher on both the PDFs and the real photos. I'm not claiming any other category.

Author's Notes

I drafted this post with AI help, as I did the earlier entries, and edited it myself. The software was built with Claude Code from a spec I wrote with it. All numbers above come from runs saved in the repo (docs/eval/).

roadbean: say it in the car, get it in the ledger

MinchinWeb — Sat, 10 Oct 2026 20:19:13 +0000

This is my submission for the Hacktoberfest Open-Source AI Challenge: Week 1 (Touch Grass). It is also entered for **Best Use of Gemma.

What I built

roadbean turns voice memos recorded on my phone, while I'm in the car, into entries in my Beancount ledger. I say "trip start, odometer 1234, heading to the library", and later "trip end, 1254", and say "Safeway, 42 dollars 28, GST 7 cents" when I buy groceries. I get a mileage log I can hand to the CRA (the Canada Revenue Agency, i.e. the "tax man") and receipts waiting for review.

phone: voice memo / receipt photo
  -> git inbox (private repo, synced to the laptop)
  -> local model: speech to text, then text to JSON
  -> plain Python: pairs trips, checks the arithmetic, writes entries
  -> staging.bean: pending (!) entries you review and promote

The design rule is the model reads, the code checks. The model is the only non-deterministic step. Everything after it (pairing a start with an end, comparing odometer difference to the stated distance, splitting GST, applying the allowance tiers) is ordinary Python with tests. Nothing is silently corrected: if something doesn't add up, the entry is written with a ; REVIEW: line and stays pending (!) until a human looks.

Demo

This is the laptop side. On the phone, three Termux:Widget buttons run a script that records a memo, takes a photo (e.g. a receipt) or pushes the inbox to a private git repo that syncs to the laptop. The recording below is roadbean process run against real memos from my phone.

Look at what the demo does not do. My first real memo was transcribed as "uh memo". The model couldn't make a trip or receipt of it, so the entry says not a trip or receipt; file by hand instead of inventing something. The receipt memo came out as "purchase staple safeway for 42 28 and gst of 7 7", and the entry was written with no rule matched payee; left uncategorized and a review flag. (It also got the GST wrong; more on that below.) The run also flagged a 305,023 km "trip" as exceeding max_trip_km, because a stale open trip from my earlier testing got paired with the new start. It was caught by a validation rule, not by me. Real input is messy; the point is that every miss lands as a visible review line rather than a wrong number in the books.

How it works

Speech to text. Gemma's native audio input through Ollama (gemma4:e4b), after ffmpeg converts the phone's .m4a to 16 kHz mono WAV. Whisper.cpp and Cactus's Whistle are drop-in alternatives ([models].stt).
Extraction. The transcript goes to Gemma with a JSON schema (format=<schema>, think: false). Every field is required, so the output is either complete or rejected. Seven few-shot examples live in prompts/extract_v1.md.
Pairing and validation (pure Python, no model):
- start and end memos for the same vehicle become one trip; odometer difference vs. stated distance is checked;
- a balance assertion on the odometer is dated the day after the trip, and is dropped (with a review note) if the model's confidence was low;
- GST is split from the total, and the CRA allowance tiers are applied to business km;
- a start with no end memo is written as a commented-out ; OPEN TRIP: block and replaced in place when the end memo arrives. If I close it by hand, roadbean notices and doesn't recreate it.
Idempotent. Each memo's SHA is stored in its entry, so re-running process never duplicates anything.
Logbook. roadbean logbook reads the real ledger through the Beancount loader and writes a CRA-style CSV with totals.

Every entry carries source: "roadbean 0.1.0", so I can tell later which version wrote it.

What a pending entry looks like

2026-10-10 ! "Safeway" "Null"
  source: "roadbean 0.1.0"
  memo-sha: "cc60a49a8f7b"
  receipt: "2026-10-10T121405.jpg"
  gst: "g"
  transcript: "purchase staple safeway for 42 28 and gst of 7 7"
  ; REVIEW: no rule matched payee; left uncategorized
  Expenses:Uncategorized      40.27 CAD
  Assets:GST-Receivable        2.01 CAD
  Liabilities:Visa           -42.28 CAD

Look closely at the GST line: 2.01 is wrong. I said "7 cents". The extractor only records a GST code (g = taxable, z = zero-rated, e = exempt), not an amount, so it heard "GST applies" and the Python code split the total at a flat 5%. That is a bad assumption for groceries, where a basket usually mixes zero-rated items (milk, vegetables) with taxable ones (snacks, paper goods). It is exactly the kind of error the review step exists for, and it is still a limitation: roadbean can't yet take a spoken GST amount. I'm leaving the real output here instead of a tidy example. (The odd "Null" narration is also real output, and also still a bug.)

Why Gemma

I wanted everything to run on hardware I own. The same family covers both ends:

On the laptop: gemma4:e4b via Ollama does speech-to-text and extraction. One model, no separate STT stack.
On the phone: the plan was a smaller Gemma (E2B) through llama.cpp in Termux, so a memo could be processed without the laptop. I didn't get speech recognition working on the phone in time, so the phone only records, photographs and syncs; all the model work happens on the laptop.

Native audio vs. a dedicated STT model

I tried Whistle (Cactus's tiny STT model) against Gemma's native audio on five synthetic memos (clean but robotic Windows SAPI voices, so an easy case for both):

STT	Odometer read correctly (3 trips)	Failure mode
Gemma native audio	3 / 3	none on odometer readings (one free-text word misheard)
Whistle	0 / 3	produced a plausible but wrong 642 km trip

Whistle's wrong trip was only caught by the low-confidence flag. That result is why native audio is the default. Five synthetic recordings is a small, friendly sample, not a benchmark. Real voice samples proved harder to process, but I don't have enough of them to give meaningful numbers like the table above.

I also considered EmbeddingGemma 2 for matching payees to rules. I decided against it: payees are a few dozen strings, normalized matching plus explicit rules is predictable, and an embedding match would add a non-deterministic step exactly where I want determinism.

Taking it outside (Touch Grass)

The idea is to keep phone time to a minimum while still keeping the records the CRA requires.

Set up the phone side at home: install the Termux shortcuts and add them to the home screen.
Get in the car at the start of the trip, tap the "memo" shortcut and say "Trip start. Odometer xxx km."
Drive around town and run errands. Not a lot of grass, as it's snowing, but we still get fresh air.
After the purchase, tap the "memo" shortcut again and say "Safeway purchase. 42.28 total, 7 cents GST."
Tap the "photo" shortcut to photograph the receipt. There is no photo preview, so it's a best guess, but the idea is to keep phone time as low as possible.
Back home, tap the "memo" shortcut and say "Trip end. Odometer xxx km."
Tap the "sync" shortcut.
The laptop processes the data later, at home.

The voice memos landed in the inbox as they were supposed to, and git made sure they ended up on my laptop for review; so that part worked.

Gemma's transcription of the car audio was a coin flip. It's hard to know whether the cause is the low-bandwidth recording from the phone, the phone's microphone placement, or the model itself. More investigation is needed. But in a pinch, the original recordings are available for human review.

For changes, I would provide more feedback when using the shortcuts; it's hard to be confident that everything is working as expected. Also, installing a bunch of scripts in Termux is cool, but not really user friendly; a full Android app would be a (much more involved) option. Finally, I would love to figure out how to do the audio processing on my phone; the model seems small enough to support this, in theory.

Honest limitations

Tested end to end on the laptop with synthetic and a few real memos. The Termux shortcuts work on the phone, but I didn't get on-phone voice recognition working.
Speech recognition errors become review lines, not wrong data, but they still need a human. Numbers read aloud ("1 2 3 4") are the weak spot.
GST is a flat 5% split when the receipt is marked taxable. Mixed baskets, such as groceries, come out wrong, and a spoken GST amount is ignored.
Single user, single currency (CAD default), personal vehicle with the CRA allowance enabled; the account names are the Beancount ones from my own setup.
There is almost no feedback on the phone. That was a feature of the design brief, but it makes it hard to feel confident the process hasn't silently broken.
It still requires a laptop to process.
It syncs through git. GitHub private repos are private to other users, but GitHub still holds the data, and trip and financial data are sensitive. Self-hosting is an option, but it's a lot of setup for one project.

Why open matters here

Trip logs and receipts say where I go, when I'm away and what I spend. I didn't want a cloud API in the middle of that.

Privacy. Audio, photos and transcripts are processed on my laptop with Gemma through Ollama. Nothing is uploaded except to my own git remote, which I can control.
Swappable models. The speech engine and the extractor are config settings (native, Whistle, whisper.cpp; Ollama or llama.cpp). That is how I found that Gemma's native audio read odometers better than Whistle, with a one-line change. Hopefully better models keep coming out.
No meter running. A year of memos and receipts costs nothing per call. Re-runs are also "free".
Where it was worse. A closed cloud speech API would probably have handled my noisy car audio better. On the phone, the open pieces didn't get there in time.

Try it

On the laptop (needs Python 3.11+, Ollama with a Gemma model, and ffmpeg):

git clone https://github.com/minchinweb/hacktoberfest-2026-1
cd hacktoberfest-2026-1
pip install -e .[ledger]
ollama pull gemma4:e4b
roadbean init ~/roadbean        # creates config, rules, inbox
roadbean doctor ~/roadbean      # checks Ollama, ffmpeg, the model
roadbean process ~/roadbean     # inbox -> staging.bean
roadbean logbook ~/roadbean     # CRA-style CSV

You can try it without a phone: drop .m4a or .wav memos named by time (2026-10-10T102541.m4a) into ~/roadbean/inbox/, or use the synthetic samples in samples/.

Setting up the phone (Android + Termux)

The phone only records, photographs and syncs. Memos reach the laptop through a private git repo, because they contain your voice and optionally your location.

Make a private inbox repo, e.g. gh repo create roadbean-inbox --private.
Install Termux, Termux:API and Termux:Widget from F-Droid (all three from the same source), then in Termux:

   pkg install git termux-api ffmpeg openssh
   git clone https://github.com/minchinweb/hacktoberfest-2026-1 ~/roadbean-src
   cd ~/roadbean-src && bash termux/install.sh

This puts rb-memo, rb-sync and rb-photo in ~/.shortcuts/tasks/, so a widget tap runs in the background instead of opening a terminal. Run termux-microphone-record -d once by hand and allow the microphone prompt.

Let the phone push. Make an SSH key (ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519), add the public key as a deploy key with write access on the inbox repo, then run bash termux/init-repo.sh git@github.com:YOU/roadbean-inbox.git.
Add the widgets. Long-press the home screen, add a Termux:Widget, and pick rb-memo, rb-photo and rb-sync. Tap rb-memo to start recording and again to stop (60 second cap).
On the laptop, clone the inbox repo as your roadbean folder: git clone git@github.com:YOU/roadbean-inbox.git ~/roadbean, then roadbean init ~/roadbean. After each trip, git -C ~/roadbean pull && roadbean process ~/roadbean.

The full instructions, including removing everything with termux/uninstall.sh, are in the repo README.

Code

minchinweb / hacktoberfest-2026-1

roadbean: voice memos to Beancount mileage + receipt entries, local open-weight models

roadbean

Say it in the car, get it in the ledger.

roadbean turns short voice memos recorded on a phone into Beancount entries. A memo can mark the start or end of a business drive (with the odometer reading) or describe a receipt (fuel, parking, a coffee with a client). Local open-weight models turn each memo into structured JSON. Plain Python pairs trips, checks the numbers, and writes pending (!) entries to a staging file that your main ledger includes.

The phone is barely touched: one tap to record, one tap to stop, a vibration to confirm. All interpretation happens later, offline, on hardware you own. Nothing is sent to a hosted API.

Built for Hacktoberfest 2026 (Open-Source AI Challenge, week 1). Design: SPEC.md.

PHONE (Termux)                                  PHONE or LAPTOP
Termux:Widget "rb-memo" -> inbox/*.m4a          1. transcribe   Whistle | whisper.cpp | Gemma audio
  (+ optional location, receipt photo)          2.

…

View on GitHub

MIT licensed, and built this week for the challenge.

How it was built

The whole thing was built in a Claude Code session, with me steering from the terminal. The curated transcript:

Building roadbean: voice memos to Beancount entries claude-sonnet-5-5

You

1) Init a git repo here, and create this as a new repo on GitHub. 2) Read @spec.md , ask me any questions you need answered, and then go build it (while I'm away).

Agent

Authenticated. Creating the repo now.

Agent

Status: the repo is live on GitHub, and the config, memo discovery, STT (including Whistle), backends and emit modules are written. I'm now writing the pairing, validation and receipt core.

Agent

Nothing was written because the heredoc broke. I'll use the Write tool for these two files.

Agent

Status: 20 tests pass. I'm fixing one bug where a photo's .extract.json sidecar gets picked up as a memo. After that come the Termux shortcuts, the README, and a commit and push.

Agent

No GGUF or Ollama is installed, so extraction can't be tested against a real model here. I'll cover the backend request and response handling with a fake local HTTP server instead.

Agent

roadbean is built and pushed to https://github.com/minchinweb/hacktoberfest-2026-1 (public). All 24 tests pass, but nothing has run against a real Gemma model yet. Ollama, Gemma weights and the Pixel weren't available here.

What works
- Core pipeline: pairing, odometer checks, GST split, the allowance tier split at 5,000 km, the Odometer opening entry for a vehicle's first trip, and re-runs that add nothing.
- Commands: doctor, process (--dry-run, --from-json), and logbook.
- Backends and shortcuts: Ollama and llama.cpp backends, plus the Termux rb-memo, rb-sync and rb-photo shortcuts.
- Whistle: added as stt = "whistle".

You

1) Should we use this model? https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/ 2) draft the DEV post

Agent

1) EmbeddingGemma 2: no, not for this week. roadbean has no retrieval or similarity step. Extraction is Gemma plus a schema, and categories come from your rules.toml on purpose, so an embedder would add a dependency to a deterministic pipeline. It doesn't transcribe, so it can't replace Whistle. Two real uses might come later: fuzzy matching of payees/destinations, and searching old memos by meaning. Both are post-challenge ideas.

2) DEV draft is saved (unpublished).

You

Ollama and the model installed here locally

Agent

Gemma 4 is running locally and the full pipeline now works end to end on laptop-synthesized audio. I ran the five sample memos through Ollama with gemma4:e4b:

Speech engine	Odometer readings right	Whole run (5 memos)	Output
Gemma native audio	3 of 3	about 26 s	Matches expected except one free-text word
Whistle	0 of 3	about 20 s	A wrong 642 km trip

What the real model run turned up, all now fixed:
- Thinking mode: Gemma 4 needs think: false. With thinking on, the schema-constrained output came back empty or wrong and took 7-17 s per memo. With it off, a memo takes about 2.7 s.
- Plausible wrong trip: garbled Whistle text produced a plausible wrong trip; low-confidence extractions now get a ; REVIEW: line and lose their balance assertion.

You

Add a termux uninstall script

Agent

I added termux/uninstall.sh and pushed it. It stops any recording left running, removes the three shortcuts, and clears the toggle file. --purge deletes $RB_HOME after a y/N prompt.

You

pip install -e .[dev] failed on Termux [Image #1]

Agent

The cause is the beancount dependency. It has no wheel for Android/aarch64, so pip builds it from source with Meson, and that build needs bison, which Termux doesn't have. On the phone you don't need Beancount at all, so I made it an optional extra. On Termux, run pip install -e . without [dev].

You

The build directory needs to be in the .gitignore

Agent

build/ was already committed (it came from my clean-venv check), so I added it to .gitignore and untracked it too.

You

The termux scripts drop into the Termux screen, and don't exit from there. Also, the (phone) install instrcutions need to include setting up the git repo to push these things to your laptop

Agent

Two real problems. Termux:Widget keeps a terminal open for scripts in ~/.shortcuts/; scripts in ~/.shortcuts/tasks/ run in the background, so install.sh now targets that. I also added termux/init-repo.sh <remote> and README steps for phone-to-laptop git sync. The rb-memo toggle now uses a timestamp instead of a background sleep.

You

Fix the file permissions (+x) on the init-repo.sh script

Agent

Git on Windows recorded every script as 100644, and chmod doesn't change that here. I set the +x mode with git update-index --chmod=+x on all the scripts and pushed.

You

warning: 2026-10-10T102519: whisper-cli not found on PATH; (on my laptop) how do I fix this?

Agent

The built-in default speech engine was whisper.cpp:base.en, which needs a whisper-cli binary. The default is now native (Gemma audio through Ollama), and the whisper error message says how to fix it.

You

Rather than asking it me to provide a bunch of paths, let's just point at a single folder, that is assumed to have the rules, configuration, and inbox in it

Agent

Done: roadbean init|doctor|process|logbook [folder]. The folder holds roadbean.toml, rules.toml, inbox/ and staging.bean, and defaults to $RB_HOME then the current directory.

You

My run threw this: roadbean process ..\hacktoberfest-2026-1-sync\ -- warning: 2026-10-10T102519: http://localhost:11434/v1/chat/completions: HTTP Error 400: Bad Request; skipped (will retry next run) -- 0 new entries, 0 memos already staged

Agent

roadbean sent your phone's raw .m4a to Ollama's native-audio endpoint, which only accepts WAV. My earlier tests used .wav, so these were the first real phone recordings on that path. It now converts every memo to 16 kHz mono WAV first. I added a regression test, and backend errors now include the response body, which is how I found the real cause.

You

Note somewhere on processing when we have an open trip (i.e. no finishing memo)

Agent

roadbean process now reports any open trip on every run, with its start time, odometer and destination, and when it will be flagged as stale.

You

Use the [GOOD] [WARN] [ERRO] logging on lines that we were using for battlestation

Agent

Found the convention in strathcona-power-agents-1. Every status line roadbean prints now uses it: only the four-letter code is coloured, [GOOD] green, [WARN] yellow, [ERRO] red, and [INFO] cyan.

You

add the open trips as commented out entries in the ledger, and then update them when the trip completes

Agent

While a trip is open, staging.bean gets a commented-out placeholder block. When the end memo arrives, or the start goes stale and gets flagged, that same block is replaced in place by the real entries. Beancount ignores the placeholder, and re-runs leave the file byte-for-byte unchanged.

You

Don't recreate the open trip if I manually edit it to close it (i.e. manually provide a closing mileage)

Agent

roadbean now recognises a hand-closed trip by vehicle and odometer-start/end, skips the start memo and any late end memo, and prints an [INFO] line saying why.

You

All ledger entries should have a "source" metadata key (or is there a difference beancount conversion), including the script name and version

Agent

Beancount has no formal convention; metadata keys are free-form lowercase names and the loader reserves only filename, lineno and __-prefixed keys. Importers conventionally use plain source, so every entry now carries source: "roadbean 0.1.0".

You

Pyproject should read the version from the python source (single source of truth)

Agent

pyproject.toml now uses dynamic = ["version"] and [tool.setuptools.dynamic] version = { attr = "roadbean.__version__" }. I tested it by bumping the source to 9.9.9, confirming pip reported it, and reverting.

You

GiF demo in docs folder. Draft the Dev post as a Markdown file I can edit

Agent

I've written the draft at docs/devto-post.md, unpublished. It uses the demo GIF's real, messy results as the honest story: a garbled memo became a review line, and a 305,023 km trip was caught by max_trip_km instead of being booked.

You

Re 2: I wasn't able to get the voice recognition working on my phone. Re 3: "so so", not measured. But the raw samples are there and can be reviewed manually if needed. Re 4: yes, add a session embed. 5) Rename the GIF

Author's Notes

N.B. Yes, this is very heavily written (code and documentation and the first draft of this post) using AI, and this required a fair bit of tinkering, so no "one shotting" here. As an experiment, I'm going to continue.

receipt2bean: turning my partner's receipt photos into Beancount with a local vision model

MinchinWeb — Mon, 05 Oct 2026 06:09:48 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My business partner keeps the receipt for everything she spends for the business. She photographs them on her phone, and at the end of the year I have to sit down and type them into my books by hand. I keep the books in Beancount, a plain-text accounting system. She has been clear that she will not learn plain-text accounting.

So I built receipt2bean. She sends me the photos. A script reads each photo with a local vision model and writes a ready-to-review Beancount transaction, with the GST split out. I check the entries; she never has to see Beancount.

Demo

Here is a screen recording of the project in action locally:

Sample output. GST doesn't match 5% of the total due to a tip.

2026-10-03 ! "CORNER CAFE" ""
  receipt: "03_corner_cafe_tip_photo.jpg"
  receipt-sha: "73f3864916bb"
  scanned-by: "qwen3-vl:8b"
  Expenses:Meals                                 20.50 CAD
  Assets:GST-Receivable                           0.93 CAD
  Liabilities:CreditCard                        -21.43 CAD

Code

minchinweb / hacktoberfest-2026-0

Weekend Challenge: Build for a Friend

receipt2bean

Photograph a receipt, get a Beancount transaction.

Built for the Hacktoberfest 2026 Weekend Challenge ("Build for a Friend") for my business partner, who photographs receipts but doesn't want to learn plain-text accounting. A local open-weight vision model reads each photo. Plain Python then checks the arithmetic and writes the entry. Nothing leaves your machine.

State: Proof of concept; works with provided sample datea. Much more customization is likely needed, and I'd like to eventually have a decent test suite too.

receipt photo --> open-weight vision model (Ollama) --> JSON
              --> Python: validate totals + GST, map accounts
              --> staging.bean  (pending "!" entries, reviewed before they join the ledger)

Design choices

The model reads, code checks. The model returns JSON constrained to a schema. Then Python verifies subtotal + GST = total, that line items add up, that GST is plausible (about 5%), and that the date is…

View on GitHub

How I Built It

The model is Qwen3-VL 8B, an open-weight vision-language model running locally through Ollama.

The pipeline has four steps:

Read. The photo goes to the model with a JSON schema (merchant, date, line items, subtotal, GST, total, payment method). Ollama's structured output keeps the reply in that shape.
Check. The model reads; plain Python checks. It verifies that subtotal + GST = total, that the line items add up, that the GST is plausible for a 5% tax, and that the date makes sense. If something is off, the entry gets a ; REVIEW: comment instead of a silent "fix".
Categorize. A small rules.toml maps merchant names to my accounts. I deliberately don't let the model invent accounts in my ledger.
Stage. Entries are written as pending (!) to a staging file. Each photo's hash is stored, so running the script twice never duplicates anything. The expense posting is computed as total minus GST, so every entry balances.

I tested it on 1 real receipt and 3 synthetic receipts (hard to share real data without giving up the privacy that this whole project is about...). Results: Two were fully correct; two failed due to OCR issues: the real one, from Staples with four individual "sub-invoices" and the "Maple Print" synthetic one that inherited the format (and I ran out of time to fix them).

Why Does Open Innovation Matter?

Receipts are a surprisingly complete record of someone's life: where we shop, what we buy, when we're away, what we pay. For something like that, I didn't want a cloud API in the middle if it can be helped.

Privacy. Every photo is processed on my own computer. Nothing is uploaded, and the model sees receipts only while it's reading them.
No per-receipt cost or account. A year of receipts is hundreds of photos. Running a model locally means no meter running and no API keys for my business partner to worry about.
I can see and control the pipeline. Because the model is swappable (--model), I can try a different open-weight one without rewriting anything; hopefully newer, better ones will keep coming out! Because the checks around the model output are ordinary Python, I can test them.
Failures are fixable. Some of the current receipts failed. I can either fix the code once and every future receipt from that vendor should work, or still input them by hand, which is no worse than what I started with.

What's next

A synced phone folder so receipts are scanned as she takes them
Matching staged entries to my bank-import transactions
Plugging into my GST tracking so the GST return worksheet builds itself
Splitting multi-category receipts by line item

Thanks for reading. Next step: see what my business partner thinks of this in the morning!

My Agent Session

I'm not sure how to export the full transcript, but these are the three prompts I used to have it built for/with me.

Could we build a receipt scanner that would create beancount transactions?
What open Source [sic] model could we use at the core?

Let me [sic] friend be my [business partner], who otherwise I have to enter these myself at the end of the year; she doesn't want to deal with beancount.

Can you create a first pass version of the code, a readme, instructions on how to put together the screen recording, and draft the post about it. Please ask any questions you need addressed to move forward

Can you generate a sample receipt? You can base it on this if it helps [real sample Staples receipt attached]

Author's Notes

N.B. Yes, this is very heavily written (code and documentation and the first draft of this post) using AI, but I wanted to test how well I could (nearly) one-shot this. It helps that the problem is small and well understood by me. The open question is what happens when it breaks (which it already has...): Can the AI fix it? Can I fix it "manually" if the AI fails? But as an experiment, I say it passes.