Build a receipt sorter with an open vision model and AcruxCore

You will build a receipt sorter: drop receipt photos into a web page and get a CSV with the company, date and total of each one. Then you will use AcruxCore (an LLMops platform) to choose the model with evidence instead of a guess. You finish with a versioned image prompt, a 50-receipt test set, a model comparison you can repeat, and a trace for every receipt the app reads.

Every AcruxCore step shows the dashboard and the Python SDK side by side. The full code is in github.com/talhaanwarch/receipt-sorter. The measured result of the comparison is in the challenge article; this page teaches the build.

You need: Python 3.11+, an OpenRouter API key with a few dollars of credit (the whole tutorial costs under $3, most of it the judge), and an AcruxCore account. The screenshots use a local server at http://localhost:5173; the steps are the same on a hosted account.

1. Set up the project

git clone https://github.com/talhaanwarch/receipt-sorter && cd receipt-sorter
python -m venv .venv && source .venv/bin/activate
pip install acruxcore streamlit python-dotenv pandas pyarrow pillow
cp .env.example .env

In the dashboard, open Account & keys, create an API key, and put it in .env as ACRUXCORE_API_KEY. Set ACRUXCORE_BASE_URL to your API (http://localhost:3001/api/v1 locally) and OPENROUTER_API_KEY to your OpenRouter key.

2. Connect OpenRouter and register the models

The AcruxCore gateway serves named models. A request says model="rs-gemma-4-31b", and the gateway resolves that name to an upstream model and a stored key. So the app code never holds a provider key, and switching a model is a name change.

Dashboard. Open Credentials, add an OpenAI-compatible credential with base URL https://openrouter.ai/api/v1 and your OpenRouter key. Then open Models and press New model once per model, choosing that credential and entering the price per million tokens.

Python. [scripts/setup_acruxcore.py](https://github.com/talhaanwarch/receipt-sorter/blob/main/scripts/setup_acruxcore.py) registers the same five models (four readers and the judge, which step 7 explains):

MODELS = {
    "rs-gemma-4-31b":  ("google/gemma-4-31b-it", 0.09, 0.34),
    "rs-qwen3-vl-30b": ("qwen/qwen3-vl-30b-a3b-instruct", 0.15, 0.60),
    "rs-qwen3-vl-8b":  ("qwen/qwen3-vl-8b-instruct", 0.25, 0.75),
    "rs-gpt-5.4-mini": ("openai/gpt-5.4-mini", 0.75, 4.50),
    "gpt-4o-mini":     ("openai/gpt-4o-mini", 0.15, 0.60),  # the judge
}

Prices matter later: AcruxCore multiplies them by the token counts of each call, which is where the dollar figures in traces come from. All reader models are open-weight except rs-gpt-5.4-mini, which is the closed reference. Gemini models are left out because the gateway cannot pass them an image link.

3. Get 50 test receipts

tools/build_sample.py downloads the public SROIE test split, picks 50 receipts with a fixed seed, shrinks them to 1,600 pixels on the long side, and writes data/labels.json with the correct company, date and total of each. The images are already committed to the repo, so you can skip the script. SROIE is licensed CC BY 4.0, and the NOTICE file carries the credit.

The images live on raw.githubusercontent.com, so each one has a public link. That is deliberate: a dataset example's input is capped at 8,192 bytes, which a photo does not fit in. The example stores the link, and the model fetches the image.

4. Write the prompt with an image

A prompt in AcruxCore is a list of messages. A user message can hold several parts, including an image part whose URL is a variable.

Dashboard. Open Prompts, create receipt-extract, and in the editor add a system message with the rules, then a user message with a text part ("Read this receipt.") and an Image link part set to {{ receipt_url }}. Press Commit new version.

Python.

USER_PARTS = [
    {"type": "text", "text": "Read this receipt."},
    {"type": "image_url", "image_url": {"url": "{{ receipt_url }}"}},
]
prompt = await hub.prompts.create("receipt-extract")
v1 = await hub.prompts.commit_version(prompt.id, messages(1))
await hub.prompts.promote_alias(prompt.id, "production", v1.version_number)

Two details. The image part is allowed only in a user message. And promote_alias creates the alias production: a movable name that points at one version. The app asks for production, never for a version number, so you can change what runs without touching the app.

5. Call it through the gateway

Rendering fills the variable and returns messages ready to send. The gateway call then runs them on a registered model and records a trace.

rendered = await hub.prompts.render("receipt-extract", "production", {"receipt_url": receipt_url})
result = await hub.gateway.chat(
    "rs-gemma-4-31b", rendered.messages, temperature=0, max_tokens=500,
    prompt_version_id=rendered.version_id, variables=rendered.variables,
    trace={"name": "extract-receipt-fields", "tags": ["receipt-sorter", "production"]},
)
fields = parse_fields(result.content)   # {"company": ..., "date": ..., "total": ...}

receipt_url is either a public link or an uploaded photo turned into a data:image/jpeg;base64,... value by to_data_url. Both go through the same call. Passing prompt_version_id links the trace to the exact prompt version that produced it, and temperature=0 keeps repeat runs comparable.

Models sometimes wrap the JSON in a code fence, so parse_fields finds the first {...} in the reply and parses that. The extraction code is in receipt_sorter/extract.py. For image parts in more detail, see the guide Send an image to a model.

6. Build the dataset

A dataset is a set of examples: an input, and a description of what a good answer looks like. Here each input is one receipt link, and the criteria spell out the expected company, date and total.

Dashboard. Open Evaluations, create a dataset, and use Add example for each one. For 50 examples, the SDK is quicker.

Python.

ds = await hub.datasets.create("receipt-sorter-sroie-50")
await hub.datasets.add_examples(ds.id, [
    {"input": {"receipt_url": f"{IMAGE_BASE_URL}/{label['id']}.jpg"}, "criteria": criteria(label)}
    for label in labels
])

criteria writes one sentence per receipt, for example: expected company "F&P PHARMACY", date "02/03/2018", total "31.90"; the company may differ in capitalisation or punctuation only; any other date or total means a low score. The judge in step 7 reads this text.

7. Compare prompts and models

An experiment crosses prompt versions with models over one dataset. A run is one execution of that grid, and each combination of version and model is a cell. So there are two things to compare, and you get both from one run.

First commit a second prompt version with stricter field rules (the day comes first in a Malaysian date, the total is the amount owed after tax, not the subtotal). Put it on the staging alias, and open the Diff tab to see exactly what changed from version 1.

Then create the experiment: dataset, both versions, the four reader models.

Dashboard. Open the dataset and press Run experiment, tick both versions and the four models, and start.

Python.

exp = await hub.experiments.create(
    dataset.id, [v1_id, v2_id], ["rs-gemma-4-31b", "rs-qwen3-vl-30b", "rs-qwen3-vl-8b", "rs-gpt-5.4-mini"],
    prompt_id=prompt.id, name="receipts: all 4 models, prompt v1,2",
)
run = await hub.experiments.start_run(exp.id)
while (await hub.runs.get(run.run_id)).status not in ("succeeded", "failed", "cancelled"):
    await asyncio.sleep(10)
report = await hub.runs.get_report(run.run_id)

A worker process runs the jobs in the background, so a run stays queued until a worker is running. The run takes a few minutes: 400 model calls plus 400 judge calls.

Name experiments after what they contain. The Runs list shows only the name, and "run 3" tells you nothing a month later.

Read the report

The report has a leaderboard and a matrix with one cell per version and model. Each cell shows a judge score and how many of the 50 examples were scored.

The judge is another model that reads the receipt, the model's answer and your criteria, and gives a 0–100 score with a reason. Open any example to read that reason. Three things to know:

• The judge model is set by the server (EVAL_JUDGE_MODEL, default gpt-4o-mini) and must be registered in Models, as in step 2. If it is missing, every example stays unscored.
• A run ends succeeded even when cells failed or went unscored. [scripts/run_eval.py](https://github.com/talhaanwarch/receipt-sorter/blob/main/scripts/run_eval.py) checks scoredCount for every cell and stops if any is zero.
• The judge is the expensive part. It receives the image too, and in this project the judge cost about 13 times the readers' cost per run. Read the token count of one judge trace before starting a full run, and consider a cheaper vision model as judge.

Score from the raw outputs too

A judge score is a good overview, but the question "were all three fields right?" has an exact answer. [scripts/score_run.py](https://github.com/talhaanwarch/receipt-sorter/blob/main/scripts/score_run.py) reads each cell's raw replies and compares company, date and total to the labels, then prints the counts. Provider errors, such as a rate limit, are counted in their own column and are not treated as wrong answers.

Repeat the run

The same prompt scores a little differently each time. Run the experiment three times (python scripts/run_eval.py --repeat 3) and compare ranges, not single numbers. If two models' ranges overlap, treat them as tied. For the guide to this step, see Evaluate a prompt that reads an image.

8. Promote the winner

When version 2 reads better, move the alias.

Dashboard. Open the prompt's Versions tab and move the production alias to version 2. The badges under the prompt name show which alias points where.

Python.

await hub.prompts.promote_alias(prompt.id, "production", 2)

The app asks for production, so its next extraction uses version 2 with no code change and no redeploy. If version 2 turns out worse, point the alias back at version 1.

9. Read the traces

Every gateway call wrote a trace. Open Traces and pick one.

The trace shows the model that ran, the tokens, the dollar cost, and the latency. The tags receipt-sorter and the alias name let you filter all receipts read under production. Because step 5 passed prompt_version_id, the trace also links back to the prompt version, so a wrong total can be traced to the exact wording that produced it. The median seconds and cost per 1,000 receipts in the challenge article come from these spans.

10. Add the app

app.py is about 40 lines of Streamlit. It lets you choose a model and an alias, takes uploaded photos, calls extract_fields for each, shows a table with a link to each trace, and offers the CSV download.

streamlit run app.py

The app holds no prompt text and no provider key. Everything it needs, the wording, the version and the model routing, lives in AcruxCore.

Questions you may have

Can I choose the judge model and the judge prompt?

Yes. You pick both when you set up an experiment. The judge model can be any model registered in Models, so you can use a cheaper vision model than gpt-4o-mini. The judge prompt holds the instructions the judge follows when it scores an answer, and you can edit it for your task. For receipts, you could tell the judge to ignore capitalisation in the company name.

Can I compare several prompts and several models in one run?

Yes. One experiment crosses any number of versions of one prompt with any number of models. Step 7 uses two versions and four models, so one run has eight cells. Each cell has its own score, and the matrix puts all of them on one page. The Evaluate a prompt guide covers the same flow for a text prompt.

Can I use a model that is not on OpenRouter?

Yes. The gateway serves any model that has an OpenAI-compatible API, whether a hosted provider or a server you run yourself. Register it in Models with its base URL and prices, and use its name in the experiment like any other model. A model must accept image input to read receipts.

Can I check the answers with my own code as well as the judge?

Yes. An experiment always scores with a judge, which gives each answer a 0–100 score and a reason. A run also keeps the raw reply of every example. If your task has one correct answer, as the receipt fields do, you can compare those replies to your labels in code, as scripts/score_run.py does. This exact count is made by your script, and the dashboard does not show it. The judge score is a quick overview, and the exact count is the number you can check.

How much does a run cost, and how do I lower it?

The judge is the largest cost, because it receives the receipt image as well as the answer. Run the experiment on 10 examples first, read the token count of one judge trace, and then start the full run. A cheaper judge model lowers the cost of every later run.

Where is my OpenRouter key stored?

In the AcruxCore credential you created in step 2. The app sends a model name to the gateway, and the gateway adds the key. The code in the repository holds only the AcruxCore API key.

Can I run AcruxCore on my own server?

Yes. AcruxCore is open source under Apache-2.0, and the screenshots in this tutorial come from a local server. The steps are the same on a hosted account. The code is at github.com/AcruxCore/AcruxCore.

Where to go next

• The measured comparison of the four models, with the table and the limits of the test: the challenge article.
• Replace the 50 SROIE receipts with 50 of your own, with labels you checked yourself. SROIE is public and some labels are wrong, so a private set tells you more.
• Add a fifth model: register it in step 2 and run the experiment again.