ReceiptPilot: a local-first AI receipt assistant for freelancers

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Every freelancer I know keeps the same folder: phone photos of taxi receipts, PDF invoices from software vendors, and hotel bills scanned at an angle. Once a month, or worse, once a year at tax time, someone types all of it into a spreadsheet. Merchant, date, currency, subtotal, tax, total. One receipt at a time.

ReceiptPilot does that typing. You upload JPG, PNG, or PDF files. An open-weight vision model can read them locally on your own computer. Ordinary code checks the numbers. Then you review every field next to the original document before it becomes a confirmed expense. Confirmed expenses go into a ledger you can filter and export as CSV for your accountant.

The customer is deliberately narrow: a freelancer or two-person business handling somewhere between 20 and 200 receipts a month, who enters them by hand today and doesn't want to send financial documents to another cloud service. The value is less repetitive data entry, fewer manual transcription errors, and more control over sensitive documents.

I can see two ways to charge for it: a paid desktop bundle with the model preconfigured, or a managed installation that an accountant rolls out to their freelance clients.

What it does:

• Reads documents with a local open-weight model: Gemma 3 4B through Ollama. Text-based PDFs are read from their embedded text; photos and scans are processed visually.
• Checks arithmetic in code, not in the prompt. Amounts are parsed as exact decimals, with both 1.234,56 and 1,234.56 supported, and subtotal plus tax is compared with the total.
• Refuses to guess. A date like 03/04/2026 or a bare $ is flagged for you to resolve instead of being silently interpreted.
• Catches potential duplicates. It checks identical files, matching invoice numbers, and combinations of total, date proximity, and similar merchant names. It gives you a score and a plain-English reason. You decide what to do; nothing is automatically deleted.
• Shows where every value came from: extracted from the document, suggested, or edited by you.
• Handles extraction failures gracefully. If extraction fails, the receipt can still be saved so you can retry or enter the details manually.

Demo

Try ReceiptPilot: Open the live application

Source code: GitHub — SHWETANK-SAURABH/ReceiptPilot

The live web application provides a way to explore the interface. The local setup below is the intended way to run the complete application with your own Ollama model.

The dashboard keeps confirmed totals separate by currency:

The ledger supports filtering and CSV export:

To run the application locally, install Ollama, ensure Docker is installed, and run:

ollama pull gemma3:4b
docker compose up --build

Then open http://localhost:3000.

The complete setup instructions and configuration are available in the GitHub repository.

Code

ReceiptPilot on GitHub

The project is MIT licensed and uses a Next.js 16 frontend, a FastAPI backend, and SQLite or PostgreSQL for storage. The repository includes backend and frontend tests, Playwright end-to-end tests, Docker Compose configuration, and a Vercel configuration.

How I Built It

flowchart LR
    U([User]) --> FE["Next.js web app"]
    FE --> API["FastAPI"]
    API --> P["PDF text extraction / page rendering<br/>Image preprocessing"]
    P --> M["Ollama<br/>Gemma 3 4B"]
    M --> V["Code validation<br/>Decimals, dates, arithmetic"]
    V --> D["Duplicate detection"]
    D --> DB[("Database")]
    DB --> FE

The open-weight models

The default model is gemma3:4b, served through Ollama. I developed and tested the application on a laptop with an NVIDIA RTX 4060 GPU with 8 GB of VRAM.

I also compared it with qwen2.5vl:7b, which uses the Apache 2.0 license. In my 40-document synthetic evaluation, Qwen was slightly more accurate, while Gemma was faster and required fewer resources.

The measured median processing times were 4.6 seconds per document for Gemma and 6.4 seconds for Qwen. These figures describe my test environment, not a guaranteed performance level on every machine.

Gemma is the default because the goal is to make the application practical on more modest hardware. Qwen remains an alternative. The model interface is separated from the business logic, so swapping models or inference runtimes does not require rewriting the rest of the application.

The design decision that mattered most: the model reads, the code decides

Ollama's structured output constrains the model to a JSON schema in which fields are allowed to be empty. The model does not have to invent a value just because a field exists.

After extraction, deterministic Python code validates the response. It parses amounts as decimals, checks totals, identifies ambiguous dates, and looks for potential duplicates.

Nothing is automatically changed to make the numbers add up. If a total does not match, the application shows a warning and lets the user decide what to do.

That decision mattered during development. In an early test, the model returned a line item as 2 × 9.98 = 9.98. The arithmetic validation caught the inconsistency.

Another model run showed Gemma copying line totals into unit-price fields when a receipt printed only one amount per line. This produced warnings on records that were actually correct, which could eventually train users to ignore warnings.

I revised the prompt and reran the evaluation for both models. The revised evaluation no longer showed that particular source of warning noise. This was a useful reminder that validation logic is only as useful as the quality of the data it validates and the warnings it produces.

Measuring it honestly

I created a labelled set of 40 synthetic documents, including:

• Clear receipts and invoices.
• Text-based and scanned PDFs, including multi-page documents.
• Blurred, skewed, rotated, dark, faded, and low-resolution images.
• Missing fields and ambiguous dates.
• Planted arithmetic errors.
• True duplicates and similar-looking documents that were not duplicates.

Every document went through the actual extraction and validation pipeline.

Evaluation metric	Gemma 3 4B	Qwen2.5-VL 7B
Documents with every field correct	38/40	39/40
Values invented where a field should be empty	0	0
Planted total errors caught	2/2	2/2
False alarms on planted total-error checks	0	0
Ambiguous dates correctly left for the user	4/4	4/4
Duplicates detected	5/5	5/5
Look-alikes incorrectly flagged as duplicates	0/4	0/4
Median processing time per document	4.6 s	6.4 s

The remaining extraction misses were fields left blank rather than incorrect values. The application marks these fields as missing instead of silently filling them with guesses.

There are important limitations. These are synthetic documents, the sample contains only 40 documents, and it does not represent the full variety of real-world receipts. Crumpled paper, unusual layouts, handwritten notes, unfamiliar currencies, and poor scans may produce worse results.

These numbers should be treated as an initial engineering evaluation, not a claim of production-grade accuracy. The dataset, evaluation runner, and raw results are available in the repository so that others can reproduce or extend the tests.

The evaluation also caught errors in my own ground-truth labels. Two labels needed correction because dates such as 01-10-2026 are ambiguous without an agreed date convention. The application had correctly refused to choose between 1 October and 10 January. I corrected the labels and documented the change rather than silently adjusting the results.

Why Does Open Innovation Matter?

Receipts can contain names, partial card details, addresses, and a detailed record of how someone spends their money.

A receipt-processing service built entirely around a closed API can require users to send financial documents to an external provider. With open weights and local inference, ReceiptPilot can run its extraction pipeline on a user's own machine without sending documents to an external AI API.

That makes local operation possible without a per-page AI API fee. It also gives users more control over where their documents are processed.

Open weights made experimentation practical, too. I could run two models on the same dataset, compare their results, and choose a model based on accuracy, speed, and hardware requirements instead of relying on a single provider.

Privacy still depends on configuration. If ReceiptPilot is pointed at a remote Ollama server, documents sent for inference may leave the user's device. A local model alone does not guarantee that every part of a web application is private.

That is why the distinction between a hosted interface and a fully local deployment matters. Users should understand where documents are stored, which server processes them, and whether data leaves their device.

For this project, open innovation is not just about avoiding a proprietary API. It makes model choice, reproducibility, and local-first deployment practical design options.

What's Next

The next step is validation with real freelancers: processing their own receipts with permission, measuring the time saved, and publishing the results alongside the synthetic evaluation.

After that, I want to work on a one-click desktop build, support tips, fees, and discounts as proper fields, and add export formats for accounting tools such as Xero and QuickBooks.

ReceiptPilot is an attempt to make a small but tedious task easier without forcing users to give up control of their financial documents. There is still work to do, especially around real-world accuracy and usability, but the core workflow is in place.