A receipt reader on open vision models: 4 models, 50 receipts, 25x price gap

This is a submission for the Hacktoberfest Challenge, Week 2: Side Hustle.

What I Built

Freelancers keep a pile of receipt photos and type each one into a spreadsheet: company, date, total. The receipts are private financial records, and many people would rather not upload them to a closed service.

The receipt sorter takes receipt photos and returns a CSV with the company, date and total of each. The side hustle idea is a small paid service that does this for customers.

One prompt holds the instructions and the photo, and a vision model reads it. I tested four models on 50 labelled receipts, which means each receipt has a known correct company, date and total. Gemma 4 31B, Qwen3-VL 30B and Qwen3-VL 8B have open weights. GPT-5.4 mini is closed, and I used it as the reference to compare against. I work on AcruxCore, the open-source LLM-ops platform (GitHub) that ran these tests. It stores prompts, sends model calls through one gateway, runs evaluations and records traces.

Demo

Drop photos in, press Extract, and download the CSV. Every row links to a trace, the record AcruxCore keeps of one model call with its tokens, cost and time, so you can inspect a wrong total.

Code

github.com/talhaanwarch/receipt-sorter (Apache-2.0) has the code and a one-command setup.

LLMops platform used to compare models and prompt is AcruxCore, an opensource platform.

The step-by-step build is in a separate tutorial: Build a receipt sorter with an open vision model and AcruxCore.

How I Built It

1. Register the models. Four vision models on OpenRouter are registered in the AcruxCore gateway, the service that sends each call to a model provider, with their prices: Gemma 4 31B, Qwen3-VL 30B, Qwen3-VL 8B, and GPT-5.4 mini as the closed reference.
2. Write one prompt with an image. The prompt receipt-extract has a system message with field rules and a user message with an image part, {{ receipt_url }}. Each edit is a new version. Version 1 has short instructions, and version 2 adds field rules, such as the day coming first in the date. An alias is a movable name that points at one version, and the alias production points at the version the app uses.
3. Call it through the gateway. The Streamlit app renders the prompt for the alias and sends it to the model you choose. The same code reads a public link or an uploaded photo sent as a data: URL.
4. Build a dataset. The 50 receipts are a sample of the public SROIE set, each with its labelled company, date and total. The dataset stores links to the images, not the images.
5. Run an experiment. An experiment crosses prompt versions with models. I ran two versions on four models, three times. I scored exact matches from the raw outputs. An LLM judge, a second model that reads the receipt, the answer and the correct values, also gave each answer a score from 0 to 100.

Every call in these steps wrote a trace like this one, so the cost and seconds in the table below come from the spans, not from a stopwatch.

1. Promote the winner. Moving the production alias from version 1 to version 2 changed the app's behaviour with no code change.

What the measurement shows

Each row is 50 receipts, three runs of the same prompt (version 2). "All three right" means company, date and total were all correct. Cost is the extraction call only.

Model	Open weights	All three right (of 50)	Median seconds	$ per 1,000 receipts
Gemma 4 31B	yes	45–46	5.0	0.05
Qwen3-VL 30B	yes	39–44	3.1	0.21
Qwen3-VL 8B	yes	41–42	2.2	0.34
GPT-5.4 mini	no	45	1.8	1.25

Gemma 4 31B and GPT-5.4 mini cannot be told apart on accuracy in this sample. Gemma costs 25 times less per receipt: $0.05 against $1.25 per 1,000 receipts. GPT-5.4 mini is faster, at 1.8 seconds against 5.0.

The Qwen models are faster than Gemma, but they got fewer receipts fully right. Qwen3-VL 8B costs more per 1,000 receipts than the 30B because its registered price per token is higher: $0.25 in and $0.75 out per million tokens, against $0.15 and $0.60.
The order of the models changes between prompt version 1 and version 2, so I do not name an accuracy winner. The table supports a smaller claim: an open model matched the closed one on this sample.

Why Does Open Innovation Matter?

Cost. At 40 receipts a month, Gemma 4 31B costs $0.002 and GPT-5.4 mini costs $0.05. Both amounts are small, but the closed model costs 25 times more, with accuracy this sample cannot tell apart, and that difference grows with every customer.

Control of financial records. Open weights can run where the data stays. I ran these models through OpenRouter, not on a laptop, so this article does not show a local run. The measurement shows that models you could host yourself read these receipts as well as the closed one. You need that evidence before you move them in-house.

What the judge cost

The judge is the expensive part of an image evaluation. One run covers eight combinations of prompt version and model, with 50 receipts each, so 400 extractions and 400 judge calls. The four extraction models cost $0.18 in total, and the judge cost $2.33.

The judge was billed about 37,000–41,000 input tokens per call. The extraction models were billed 340–1,500 input tokens. The most likely cause is that the judge also receives the receipt image, but I did not test a different judge to confirm it. Check the token count of one judge call before you start a full run.

Limits

• SROIE is public, so a model may have seen it in training. A private set of receipts could give different exact-match counts.
• 50 receipts is small. The ranges show how much three repeat runs differ. They do not show how much a different set of 50 receipts would change the result.
• The receipts are Malaysian English. Other languages and layouts are untested.
• Five receipts are missed by all four models. Two print two valid totals, one has a cut-off year, one has a low-resolution year, and one shows two company names. I kept the published labels. On the other 45 receipts, Gemma missed none, GPT-5.4 mini missed one, Qwen3-VL 8B missed four and Qwen3-VL 30B missed seven.
• Qwen3-VL 8B hit provider rate limits in one run. I ran it again and report only runs without provider errors.

Prize Categories

Best Use of Gemma. Gemma 4 31B, Google's open-weight model, reads the receipts through a provider (OpenRouter). In the measurement, its exact-match range overlapped the closed reference model's, and it cost 25 times less..

The prompt, gateway, dataset, evaluation and tracing layers run on AcruxCore, which is open source under Apache-2.0.

Comments

[Arhan Canli]

The 25x holds up: $1.25 / $0.05 is exactly 25. Two things I'd check before reading the accuracy column as a tie.

First, the three runs are the same 50 receipts, so the 45–46 vs 45 range measures sampling noise in the model, not uncertainty from which 50 receipts you picked. At 45 of 50 the 95% interval on the underlying rate is roughly 79–96%, so Gemma, GPT-5.4 mini and probably the 41–44 Qwen rows are not separable on this sample. Your wording ("matched on this sample") is right; I'd put the interval next to the table.

Second, Qwen3-VL 8B at $0.34 costs more per 1,000 receipts than the 30B at $0.21. That is probably provider pricing or image-token counts differing, but it is worth a line, because "smaller means cheaper" is what readers will assume.

A question: are the 4–5 receipts that every model misses the same ones? If they are, they are likely label problems (total printed twice, rounding line, date format) rather than model errors, and fixing the labels would move all four rows at once.

[Talha Anwar]

Thanks, you were right on all three points. I have updated the article.

1. The 45–46 range and the interval. The 45–46 range shows how much the same model changes when it reads the same 50 receipts again. It does not show how much a different set of 50 receipts would change the result. At 45 of 50, the 95% interval is about 79–96%. Gemma, GPT-5.4 mini and the Qwen models overlap inside it, so the article no longer ranks them on accuracy. The only differences it claims are cost and speed. I added the interval beside the table.

2. The 8B costs more than the 30B. The cause is the price per token. I checked OpenRouter's price list on 14 October 2026. The 8B is $0.25 in and $0.75 out per million tokens. The 30B is $0.15 and $0.60. I added a line, because readers will assume that smaller means cheaper.

3. Are the misses the same receipts? Yes. Five receipts are missed by all four models, and they explain the 45-of-50 ceiling. I looked at each photo, and they are not model errors:

4. Two receipts print two valid totals.

5. One receipt has a cut-off year.

6. One receipt has a year printed too small to read reliably.

7. One receipt shows two company names.