Sloth Receipts: I photograph a receipt and a local AI does my expense report

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Sloth Receipts is a small Python tool that turns a folder of receipt photos into an expense spreadsheet, using an open-weight vision model running on my own laptop.

The lazy workflow:

1. Snap a photo of the receipt and drop it in a folder.
2. Go to sleep.
3. Wake up to expenses.csv with the date, merchant, category and total filled in, plus a monthly summary by category.

Expense reports are a chore everyone puts off. Nobody enjoys typing numbers off crumpled paper, so I let a sloth do it.

It is lazy and honest about its mistakes:

• Two-pass self-check. Each receipt is read twice. If the two passes disagree on the total, the row is flagged needs_review instead of being silently trusted.
• Sanity checks. Missing totals, impossible or future dates, and unknown categories are flagged too.
• Never re-processes. Each photo is hashed, so a nightly run only handles new receipts.
• I only review the flagged rows, not every receipt.

Demo

Try it in two minutes. Install Ollama, then:

ollama pull qwen2.5vl:3b
python3 sloth_receipts.py ~/Receipts

The output is expenses.csv with these columns:

file,date,merchant,category,currency,total,needs_review,reason

Rows the model is unsure about are marked YES in needs_review, with the reason (for example, passes disagree on total), so a human only checks those.

Code

GitHub repo: sloth-receipts by SandipGhorai-max

sloth_receipts.py (Python standard library only):

#!/usr/bin/env python3
"""Sloth Receipts: receipt photos -> expense CSV, using a local open-weight vision model (Ollama)."""
import argparse, base64, csv, hashlib, json, re, urllib.request
from collections import defaultdict
from datetime import date, datetime
from pathlib import Path

OLLAMA = "http://localhost:11434/api/chat"
MODEL = "qwen2.5vl:3b"
CATEGORIES = ["Food", "Groceries", "Transport", "Travel", "Office",
              "Software", "Health", "Utilities", "Entertainment", "Other"]
IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp"}
FIELDS = ["file", "date", "merchant", "category", "currency", "total", "needs_review", "reason"]
LEDGER = ".sloth-receipts.json"


def read_receipt(path: Path, model: str) -> dict:
    prompt = (
        "Read this receipt photo. Reply ONLY with JSON: "
        '{"merchant": str, "date": "YYYY-MM-DD", "currency": "ISO code like INR or USD", '
        f'"total": number (the final amount paid), "category": one of {CATEGORIES}}}. '
        "Use null for anything you cannot read. Do not guess."
    )
    img = base64.b64encode(path.read_bytes()).decode()
    body = json.dumps({
        "model": model, "stream": False, "format": "json",
        "options": {"temperature": 0.2},   # slight variation makes the 2nd pass a real check
        "messages": [{"role": "user", "content": prompt, "images": [img]}],
    }).encode()
    req = urllib.request.Request(OLLAMA, body, {"Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=300) as r:
            return json.loads(json.loads(r.read())["message"]["content"])
    except Exception:
        return {}


def to_amount(value):
    if value is None:
        return None
    s = str(value).strip()
    if re.search(r",\d{2}$", s) and "." not in s:   # "12,50" -> "12.50"
        s = s.replace(",", ".")
    s = re.sub(r"[^0-9.\-]", "", s.replace(",", ""))
    try:
        return round(float(s), 2)
    except ValueError:
        return None


def to_date(value):
    try:
        d = datetime.strptime(str(value), "%Y-%m-%d").date()
        return d if d <= date.today() and d.year >= 2000 else None
    except ValueError:
        return None


def process(path: Path, model: str, passes: int) -> dict:
    reads = [read_receipt(path, model) for _ in range(passes)]
    first = reads[0] or {}
    totals = [to_amount(r.get("total")) for r in reads]
    reasons = []

    total = totals[0]
    if total is None or total <= 0:
        reasons.append("total unreadable")
    if len(set(totals)) > 1:
        reasons.append(f"passes disagree on total {totals}")

    d = to_date(first.get("date"))
    if d is None:
        reasons.append("date missing or invalid")

    cat = first.get("category")
    if cat not in CATEGORIES:
        reasons.append("unknown category")
        cat = "Other"

    return {
        "file": path.name,
        "date": d.isoformat() if d else "",
        "merchant": (first.get("merchant") or "").strip(),
        "category": cat,
        "currency": (first.get("currency") or "").upper(),
        "total": f"{total:.2f}" if total else "",
        "needs_review": "YES" if reasons else "",
        "reason": "; ".join(reasons),
    }


def summarize(csv_path: Path):
    sums = defaultdict(float)
    review = 0
    with csv_path.open(newline="") as f:
        for row in csv.DictReader(f):
            review += bool(row["needs_review"])
            if row["total"] and row["date"]:
                sums[(row["date"][:7], row["currency"] or "?", row["category"])] += float(row["total"])
    print("\nMonthly summary")
    for (month, cur, cat), amount in sorted(sums.items()):
        print(f"  {month}  {cat:<14} {amount:>10.2f} {cur}")
    print(f"\n{review} row(s) flagged needs_review in {csv_path.name}")


def main():
    p = argparse.ArgumentParser(description="Receipt photos -> expense CSV, locally.")
    p.add_argument("folder", type=Path)
    p.add_argument("--model", default=MODEL)
    p.add_argument("--passes", type=int, default=2, help="reads per receipt (2+ enables the self-check)")
    a = p.parse_args()

    ledger_path = a.folder / LEDGER
    ledger = json.loads(ledger_path.read_text()) if ledger_path.exists() else {}
    csv_path = a.folder / "expenses.csv"
    new_file = not csv_path.exists()

    with csv_path.open("a", newline="") as out:
        w = csv.DictWriter(out, fieldnames=FIELDS)
        if new_file:
            w.writeheader()
        for img in sorted(a.folder.iterdir()):
            if img.suffix.lower() not in IMAGE_EXT:
                continue
            digest = hashlib.sha256(img.read_bytes()).hexdigest()
            if digest in ledger:
                continue
            row = process(img, a.model, a.passes)
            w.writerow(row)
            out.flush()
            ledger[digest] = img.name
            ledger_path.write_text(json.dumps(ledger, indent=2))
            flag = "  <-- REVIEW" if row["needs_review"] else ""
            print(f"{img.name}: {row['merchant']} {row['total']} {row['currency']}{flag}")

    summarize(csv_path)


if __name__ == "__main__":
    main()

To run it nightly while I sleep (crontab -e, use absolute paths):

0 3 * * * /usr/bin/python3 /home/YOUR_USER/sloth-receipts/sloth_receipts.py /home/YOUR_USER/Receipts

How I Built It

• Model: an open-weight vision-language model (Qwen2.5-VL) served locally by Ollama. It can be swapped with --model.
• Inference: one HTTP call per read to Ollama's /api/chat, with the image sent as base64 and format: "json" to force parseable output.
• Trust, but verify: a small model can misread a blurry total, so I don't trust a single read. Each receipt is read twice at a slightly non-zero temperature, and any disagreement flags the row for human review. Because inference is local and free, the second pass costs only a few seconds of compute.
• Validation layer: the model only proposes values. My code parses amounts (including 12,50 style decimals), rejects future dates, whitelists categories and flags anything doubtful.
• Idempotent: SHA-256 hashes of the photos mean the nightly job only touches new receipts.
• Dependencies: just the Python standard library.

Why Does Open Innovation Matter?

A receipt is more personal than it looks. It shows where I shop, what I buy, when I travel, and often part of a card number. Sending a shoebox of those to a cloud vision API is a hard sell, and a business sending client receipts would face compliance questions on top of that.

With an open-weight model:

• Privacy: every photo stays on my machine. There is no upload step to audit because there is no upload.
• Self-consistency is free: reading each receipt twice would double the bill on a paid API. Locally it costs nothing extra, so the added reliability is effectively free.
• Zero running cost: a month of receipts costs nothing in tokens, so I never have to ration it.
• Offline: it works in airplane mode, which helps when receipts pile up while traveling.
• Swappable and tunable: if a better small vision model ships, I change one flag. A business could fine-tune on its own receipt formats and categories.
• Pinned behavior: the model sits on my disk, so a provider update can't quietly change how my books get categorized.

Limitations

A 3B vision model is not perfect. Crumpled, faded or very long receipts can trip it up, and handwritten receipts are a weak spot. That is exactly why the tool flags uncertain rows for review instead of pretending to be accurate. For better accuracy, run --model qwen2.5vl:7b if your hardware allows.