Building a 100% Local Receipt Sorter with Ollama Vision

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

A Local Receipt & Invoice Sorter — point it at a folder of receipt/invoice photos or PDFs, and it extracts vendor, date, amount, currency, and category into a clean, editable CSV. The whole extraction pipeline runs through a local vision-language model via Ollama, so no financial document ever leaves your machine.

Who pays for this: freelancers, consultants, and small shop owners doing their own bookkeeping. Right now they either manually type receipts into a spreadsheet (tedious) or use a cloud tool like Expensify (recurring cost, and your financial documents go through a third-party API). This tool kills both problems — it's free to run after setup, and privacy-sensitive users who'd never upload a receipt to ChatGPT have a real reason to use it.

Demo

Code

loveyadav1015 / ollama-tax-man

Receipt & Invoice Sorter

Local-first tool that turns a folder of receipt or invoice images and PDFs into a CSV (vendor, date, amount, currency, category, source_filename) using a vision-language model that runs in Ollama on your machine. Documents are never sent to a cloud API.

Prerequisites

1. Python 3.10+ with the project virtualenv (already created in venv/ for this repo).
2. Ollama installed and running locally.
3. A vision model pulled before you extract:

ollama pull qwen2.5vl:7b

If qwen2.5vl:7b is unavailable, pull the fallback:

ollama pull llama3.2-vision

The app picks qwen2.5vl:7b (or any installed qwen2.5vl tag) first, then llama3.2-vision. If Ollama is down or neither model is present, the UI shows those exact ollama pull … commands instead of a stack trace.

Qwen2.5-VL needs a recent Ollama (0.7+).

Setup

cd ollama-tax-man
source venv/bin/activate

Packages already in the environment: streamlit, pandas, pydantic…

View on GitHub

How I Built It

Built around Ollama running qwen2.5vl:7b locally for vision-based extraction — no OpenAI/Anthropic/Google vision API calls anywhere in the pipeline.

Stack:

• Ollama + qwen2.5vl:7b (local inference, vision-capable)
• PyMuPDF to rasterize PDF pages to images before sending to the model
• Pydantic to validate the model's JSON output against a strict schema (vendor, date, amount, currency, category, confidence, notes)
• Pandas for the tabular CSV output
• Streamlit for a drag-and-drop UI with an editable results table before export

The extraction prompt forces strict JSON output, and the app retries once with a stricter instruction if parsing fails, falling back to a flagged "needs review" row rather than crashing. Low-confidence rows are visually highlighted so the user knows what to double-check before trusting the CSV for taxes.

Architecture

flowchart TD
    A[Folder of receipts<br/>images / PDFs] --> B{File type?}
    B -->|PDF| C[PyMuPDF:<br/>rasterize pages to images]
    B -->|Image| D[Load directly]
    C --> E[Ollama: qwen2.5vl:7b<br/>vision extraction]
    D --> E
    E --> F{Valid JSON?}
    F -->|No| G[Retry with stricter<br/>JSON-only prompt]
    G --> F
    F -->|Still fails| H[Flag row:<br/>needs_review]
    F -->|Yes| I[Pydantic schema<br/>validation]
    I --> J[Pandas DataFrame]
    H --> J
    J --> K[Streamlit editable table<br/>low-confidence rows highlighted]
    K --> L[Download CSV]

Processing sequence

sequenceDiagram
    participant U as User
    participant S as Streamlit UI
    participant L as file_loader
    participant O as Ollama (local)
    participant P as pandas / CSV

    U->>S: Drop folder of receipts
    S->>L: Load each file
    L->>L: Convert PDFs to images
    L->>O: Send image + extraction prompt
    O-->>L: JSON (vendor, date, amount, category...)
    alt JSON valid
        L->>P: Append row
    else JSON invalid
        L->>O: Retry with stricter prompt
        O-->>L: JSON or failure
        L->>P: Append row (flagged if still failing)
    end
    P->>S: Render editable table
    U->>S: Review / fix rows
    S->>U: Download CSV

Why Does Open Innovation Matter?

The entire value proposition of this tool depends on the model being local. A closed API (even a cheap one) would reintroduce the exact objection this tool is designed to eliminate: your receipts — vendor names, amounts, dates, spending patterns — leaving your machine. That's not a hypothetical concern for freelancers; it's the actual reason some of them avoid expense-tracking SaaS tools in the first place.

Open-weight vision models made this possible: a few years ago, "good enough" document OCR+extraction meant a cloud API, full stop. Now a 7B vision model running on a laptop CPU/GPU can read a crumpled receipt photo and return structured JSON reliably enough to be useful. Open models didn't just make this cheaper — they made a privacy-first version of this product possible at all.

Prize Categories

• Best Use of a .tech Domain — Deployed the Streamlit demo at
  receiptsorter.tech, giving the project a
  memorable, purpose-built home instead of a generic subdomain — fitting
  for a dev-facing, open-source AI tool.

• Best Use of Sentry Agent Tracing — Used Sentry's agent tracing to
  monitor the Ollama extraction pipeline in real time: tracked each
  vision-model call's latency, caught JSON-parsing failures before they
  silently became bad CSV rows, and traced the retry logic (strict
  re-prompt on malformed output) so failure patterns were visible instead
  of hidden inside a black-box local inference call.

• Best Use of GitHub Copilot — Used Copilot throughout the build to
  scaffold the Pydantic schema validation, speed up the Streamlit
  data_editor wiring, and generate the synthetic placeholder receipts in
  sample_data/ — cutting a sizeable chunk of the boilerplate so more time
  went into getting the extraction prompt and retry logic right.