A local AI that types up a Brazilian accountant's receipts (and knows when it's wrong)

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 2 — Side Hustle.

What I Built

A self-employed accountant in Brazil gets ~100 documents a week from her small-business clients: NF-e invoices (PDF), NFC-e receipts photographed on a phone, Pix transfer screenshots, and the occasional handwritten recibo. Today she types every one of them into a spreadsheet by hand.

Leitor de notas ("receipt reader") turns that pile into a reviewed Excel file:

1. Drop the files into a Streamlit page and pick the accounting month (competência).
2. Each document is read by an open-weight model running locally in Ollama.
3. Python validates everything the model returned and flags the rows a human must check.
4. She fixes the flagged rows in an editable table and exports to Excel (one sheet of entries, one summary by category).

Who pays for it? Accountants bill small businesses a monthly fee, and data entry is the part of that work that doesn't scale. Saving a few hours a week per client is worth money. The catch is that these are clients' financial records, covered by Brazil's data protection law (LGPD). "Upload your clients' bank receipts to our cloud" is a hard sell to her, and to her clients.

Why open matters here

This project only works because the model is open-weight and runs on the machine:

• Privacy is the product. Nothing leaves the PC. The only network call is to Ollama on localhost:11434. I even turned off Streamlit's usage stats (gatherUsageStats = false), which are on by default, and bound the server to localhost.
• The cost per document is zero. At ~100 docs/week across many clients, a per-page API bill eats directly into an accountant's margin. A one-time GPU purchase doesn't.
• I can swap models freely. Model names live in one constant. When a better small vision model comes out, it's a one-line change and a re-run of the evaluation script, with no new contract and no new pricing.

Demo

Evaluation on 10 fictitious documents (text PDFs, Pix receipts, tilted phone photos, a handwritten receipt, and one deliberately unreadable blurry receipt):

field	date	amount	CNPJ	category	path	needs review
correct	9/10	9/10	9/10	10/10	10/10	10/10

95% of fields correct. All three misses come from the blurry receipt. The model hallucinated a date and a CNPJ there, and validation caught both, so the document landed in the review queue, which is exactly what should happen. Average time on an AMD RX 9060 XT (Ollama, Vulkan): 2.2 s per text PDF, 4.0 s per image. That's about 5 minutes for a week of documents.

Code

MathFe / leitor-notas-local

Leitor local de notas fiscais e comprovantes de Pix com IA open-source (Ollama) — DEV Hacktoberfest 2026 · Side Hustle

Leitor local de notas fiscais e comprovantes de Pix

Projeto para o DEV Hacktoberfest 2026 — desafio "Side Hustle": IA open-source rodando 100% local (Ollama + Qwen 2.5 / Qwen 2.5-VL) para ler notas fiscais, cupons NFC-e, comprovantes de Pix e recibos de uma contadora autônoma e transformar tudo numa planilha Excel, sem nenhum dado sair da máquina (LGPD).

• leitor_notas/ — o aplicativo (pipeline, validação, avaliação e interface Streamlit). Veja o README dele.
• amostras-mariana/ — 10 documentos fictícios + gabarito para avaliar a acurácia.

Resultado nas amostras: 95% dos campos corretos, com os 2 documentos problemáticos enviados para revisão.

Licença: MIT.

View on GitHub

How I Built It

Stack: Python, Ollama, qwen2.5:3b (text), qwen2.5vl:7b (vision), pdfplumber, Pillow, pydantic, pandas/openpyxl, Streamlit.

A hybrid pipeline: the cheapest model that can do the job

PDF with extractable text (> 50 chars) ──► pdfplumber ──► qwen2.5:3b
Scanned PDF ──► render page to image ───┐
JPG / PNG ──────────────────────────────┴──► qwen2.5vl:7b
                         │
                         ▼
   JSON forced by schema ─► Python validation ─► needs_review = True/False

Most NF-e invoices arrive as real PDFs with a text layer, so they never need a vision model. Output is constrained with Ollama's format parameter, which takes the JSON Schema of a pydantic model, so the response always parses.

Lesson 1: don't let a small model reason about roles

My first prompt asked the model for "the counterparty: the issuer if it was a purchase, the payer if it was income". The 3B model kept returning the recipient of the invoice, which is the client herself.

The fix was to ask only for what is literally on the page, "who received the money" and "who paid", and let Python decide the counterparty. Better still, a Brazilian invoice's 44-digit access key embeds the issuer's CNPJ (company tax ID), so Python can check the model's answer and undo a swap deterministically:

if chave and validar_chave_acesso(chave):
    emitente = cnpj_da_chave(chave)          # digits 7–20 of the access key
    if normalizar_cnpj(pagou[1]) == emitente != normalizar_cnpj(recebeu[1]):
        recebeu, pagou = pagou, recebeu      # model swapped issuer and buyer

The same idea applies to the access key itself. A 3B model copying 44 digits will drop one, so for text PDFs a regex finds the key and its mod-11 check digit confirms it.

Lesson 2: one small prompt per decision

Category accuracy was 70% when the extraction prompt also picked the category. "Limpeza e recarga gás split" (A/C cleaning) came out as cleaning supplies, and a received Pix came out as food.

Two changes took it to 100%:

1. Whether money came in or went out is decided by rules: the document type, plus whether the client's own CNPJ is the receiver. Income is always "Receita". The model never gets to guess it.
2. Expenses get a second, tiny call to the 3B model with only the supplier name, the document type and a short description, and an enum of expense categories with one-line hints. It costs ~0.3 s.

Small models are much better at one narrow question than at five questions in one prompt.

Lesson 3: validation is the feature

Why an LLM Alone Cannot Do Invoice Extraction Yet by Taranpreet Kaur makes the case that model output needs an independent check, and that matched what I saw. Every record goes through Python before anyone trusts it:

• CNPJ check digits, including the new alphanumeric CNPJ in force since July 2026
• NF-e access key: 44 digits plus mod-11 check digit, and the CNPJ must match the issuer inside the key
• amount > 0, and date inside the chosen competência
• an expense with no supplier CNPJ (e.g. a recibo from an individual) is flagged so she can check withholding taxes

Any failure sets revisar = True with a human-readable reason, such as "data 29/09/2026 fora da competência 10/2026" ("date outside the accounting month"). A document that crashes the pipeline becomes a flagged row instead of killing the batch. The accountant doesn't need the AI to be perfect. She needs to know which 2 of the 10 to look at.

This is the same layering described in Structured Output From LLMs: A Retry-Repair Loop Your Parser Never Sees Through by devshakib: constrained decoding, then validation you own. I chose to route failures to a human rather than retry, because a retry on a blurry photo just produces a different hallucination.

Measure, don't vibe

avaliar.py runs the whole set against a gabarito.json (answer key) and prints accuracy per field and per document, plus the mandatory check that documents 09 and 10 end up in review. Every change above was kept or reverted based on that table. The test documents are generated by a script (fake companies, valid fake CNPJs and access keys, simulated phone photos with rotation, blur and noise), so anyone can reproduce the numbers.

Her reaction

When I showed it to her, she loved it. Typing up these documents is a problem she has had for years, and she didn't know it was something that could be solved. She told me that if she had known, she would have done this a long time ago. She was really grateful, and she said she would pay R$ 2,000 for it.

What's next

• Run it on her own machine with her real documents and collect what she asks for next.
• Multi-page NF-e with many items, and reading the NFC-e QR code directly instead of relying on the vision model to read the key.
• An export in the layout her accounting software imports.

---

Built with an AI coding agent. The repo is MIT-licensed.