DEV Community

Cover image for Ledger Room: I built a local invoice sorter, then measured exactly what the privacy costs
Talha Tahir
Talha Tahir

Posted on

Ledger Room: I built a local invoice sorter, then measured exactly what the privacy costs

#devchallenge #hf26challenge #nextjs #showdev

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission πŸ’Έ

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Ledger turns a folder of invoices and receipts into an accountant-ready CSV β€” on the user's own hardware β€” and predicts which unpaid invoices will be paid late.

A local invoice sorter that keeps your clients' names off the internet.

The problem. A freelancer ends each month with 30–60 invoices and receipts in mixed formats β€” emailed PDFs, phone photos of paper receipts, screenshots of payment confirmations β€” and needs them as structured rows for an accountant or a tax filing. There are two ways to get there, and one thing that looks like a third:

  1. Pay someone. $50–150/month for data entry.
  2. Do it yourself. 2–4 hours a month, forever, re-reading documents whose content you already have.
  3. Paste it into a hosted chatbot. Which is not free in the only currency that matters. Invoice text contains client names, project names, rates and account numbers, and that text goes to a service you don't control.

Who pays for it. Freelancers, independent designers and developers, and small studios who reconcile monthly and hate it. My first user is exactly this person β€” my sister, who runs her own freelance work.

Why they'd pay. Not for the extraction. Local extraction is a utility, and a local one is free forever. What nobody else offers is the chase list: knowing which invoice to chase this week, because the model predicts days-to-payment and you don't have to remember that one client always pays on day 60. That's the $12/month hypothesis, framed around the part that isn't commoditised.

How it works. pdf.js reads the PDF text layer; tesseract.js handles scans with no text layer; gemma3:4b extracts eleven typed fields via Ollama on the user's CPU; each field gets a review-priority score; export CSV/JSON. Then TabPFN-3.5 forecasts days-to-payment on your invoice history.

No login. No API key. No upload. The document is read in memory and never written to disk β€” there is no document store to breach because there is no document store.

Demo

Live: https://ledger-room-indigo.vercel.app β€” no login, no API key

Ledger - Landing Page

Ledger - Upload Page

Ledger - Review Page

Try it in 30 seconds: click "Use the 20-document demo set". That loads real recorded model output, including its mistakes, so you can exercise review, correction, CSV export and the chase list immediately.

Try it for real:

ollama pull gemma3:4b
Enter fullscreen mode Exit fullscreen mode

Then open /app. The runtime banner turns green when it finds your local model.

If the banner says "Demo data" instead, your Ollama is refusing your browser's origin. The banner prints the exact OLLAMA_ORIGINS command with your origin filled in β€” paste it, restart Ollama, press Re-check.

Code

GitHub logo Talha-Tahir2001 / ledger-room

Ledger

A local-first invoice and receipt sorter for freelancers, with a measured accuracy study of what the privacy costs.

License: MIT Next.js 16 Local inference

Live demo Β· Report an issue

The problem

A freelancer ends each month with 30–60 invoices and receipts in mixed formats β€” emailed PDFs, phone photos of paper receipts, screenshots of payment confirmations β€” and needs them as structured rows for an accountant or a tax filing.

There are two ways to get there, and one thing that looks like a third:

Option Cost The catch
Pay a bookkeeper $50–150/month A stranger handles every client name, rate and account number you own
Do it yourself 2–4 hours/month, forever The same hours every month, re-reading documents whose content you already have
Paste it into a hosted chatbot Free Invoice text contains client names, project names, rates and bank details β€” and it goes to a service you don't control

Ledger removes the…

View on GitHub

MIT licensed. The embed above pulls in the repository README, which is where the measured results table, the reproduction commands and the verification checklist live.

Every number in this post is reproducible from a committed script:

npm run fixtures        # deterministic corpus: PDFs, PNGs, ground truth
npm run ingest:verify   # asserts text-layer vs OCR routing, one assertion per fixture
npm run eval -- --model gemma3:4b --out results/eval-4b.json
npm run history && npm run chase:eval
npm run results:export  # regenerates EVAL.md's tables and the demo dataset
npm run chase:check     # fails loudly if the baselines silently degrade into each other
Enter fullscreen mode Exit fullscreen mode

The model response cache is committed, so npm run eval -- --model gemma3:4b reproduces the headline table in 8 seconds on a machine with no GPU, no Ollama, and no hosted quota. It tells you it did so:

All 20 responses came from the committed cache β€” no model was run.
Delete results/cache/ or pass --no-cache to force a real run.
Enter fullscreen mode Exit fullscreen mode

That is the reproducibility claim I care about: you shouldn't have to trust my numbers, you should be able to re-derive them.

How I Built It

The whole pipeline 

PDF/JPG/PNG
   β†’ pdf.js text layer  (or tesseract.js OCR if there isn't one)
   β†’ gemma3:4b via Ollama, schema-constrained, temperature 0
   β†’ Zod validation, one repair round-trip
   β†’ per-field review-priority score
   β†’ review table β†’ CSV/JSON
   β†’ TabPFN-3.5 chase list
Enter fullscreen mode Exit fullscreen mode

Schema-constrained output is what makes the numbers trustworthy

The Zod schema is converted with z.toJSONSchema() and passed to Ollama as a structured-output grammar constraint, at temperature: 0, with one repair round-trip available.

Measured: zero schema violations across 20 fixtures, and zero repairs needed.

That is a guarantee from the grammar, not a capability of the model β€” which matters, because small models produce plausible JSON with a string where a number belongs, and in a finance tool a plausible wrong number is worse than an obvious failure.

Confidence is computed, never asked for

A 4B model asked to rate its own confidence produces a number with no calibration behind it, and it is confidently wrong on exactly the fields that cost money. So the score is built from three things that can be checked:

  • verbatim β€” does this value actually appear in the source text?
  • structure β€” is it well formed for its type (ISO date, ISO currency, finite number)?
  • agreement β€” does subtotal + tax == total? Do the line items sum to the subtotal?

It's shown as a review-priority score out of 100, never as a probability, because that's what it is.

The runtime banner never claims a path that will fail

On load the app probes for a local model and reports one of four states: local β†’ hosted β†’ demo β†’ blocked. A judge who sees "local runtime detected" on a machine with no Ollama is worse off than one told plainly that they need to pull a model.

Payment prediction, measured against baselines

TabPFN-3.5 forecasts days-to-payment with a q10–q90 interval, scored against two naive baselines on a held-out slice. Only numbers and a hashed client identifier are sent to Prior Labs β€” the raw client name never leaves the device.

Why Does Open Innovation Matter?

It changed what was buildable, not just what was affordable.

Cost. Both the local arm and the free hosted arm cost $0.00. The interesting result isn't that local is cheap β€” it's that a 7.75Γ— larger hosted model is also free. Open weights moved the ceiling without a budget.

Swap-ability. One Zod schema and one response_format call drive a 1B model on my laptop and a 31B model on someone else's GPU. The eval harness ran both arms without a code change β€” only possible because the interface is a schema, not a vendor's API shape.

Offline. Local extraction works with the network disconnected after the model download. For a freelancer reconciling receipts on a train, that's not a performance feature, it's the feature.

Privacy you can check rather than believe. Open DevTools, drop one invoice, watch every request go to 127.0.0.1. Then disconnect the network and do it again. That's a 60-second verification anyone can run, and it's the reason this product exists instead of a wrapper around someone else's endpoint.

A reproducible study. Because the cache is committed, anyone re-derives every number in 8 seconds without a GPU. With a closed API this is impossible β€” a third party would have to reproduce your closed-model results on hardware they don't have.

The study

Twenty synthetic invoices and receipts β€” text-layer PDFs, image-only scans, decimal-comma locales, ambiguous dates like 03/04/26, thousands separators, PKR / EUR / GBP / USD. Ground truth hand-authored before the first run and never edited to match model output.

Hardware: i7-1165G7, 4 cores / 8 threads, 16 GB RAM, no GPU, Windows 11.

Headline β€” local gemma3:4b, CPU only

Metric Value
Field accuracy 66.2% (n=19)
Exact-record accuracy 5.3%
Line-item accuracy 58.3%
Populated-field recall 80.9%
p50 / p95 latency 57.0s / 103.5s
Throughput 60.7 documents/hour
Schema violations reaching the UI 0
Cost per 20 invoices $0.00

Per-field, including where it fails

I wrote down the expected failure modes before running anything. They were mostly right:

Field Accuracy Expected failure mode
tax 100.0% almost always printed explicitly
vendor 94.7% multi-column layouts
issueDate 89.5% ambiguous dd/mm vs mm/dd
dueDate 89.5% genuinely absent β€” correct null behaviour matters
total 84.2% decimal commas; odd labels like GRAND T0TAL
currency 78.9% symbol vs ISO code
subtotal 73.7% thousands separators and decimal commas
invoiceNo 68.4% model normalises separators
billable 47.4% no signal in most receipts
category 15.8% open vocabulary β€” weakest field, as predicted

category in the teens is the honest result. I could have raised it by switching to a closed category list; I didn't, because a number that improves by making the field useless isn't worth publishing. The scorer accepts only what the document genuinely permits β€” never what the model happened to emit.

Size curve, same hardware

Model Params Field accuracy p50
gemma3:1b 1B 59.4% 37.7s
gemma3:4b 4B 66.2% 57.0s

Four times the parameters bought 6.8 points of accuracy for 1.5Γ— the latency. The more interesting result points the other way: the 1B model had the higher populated-field recall β€” 96.2% vs 80.9%. It filled in more fields and got more of what it filled in wrong. For an unattended pipeline that's the dangerous combination, and it's an argument for constraining a small model harder, not trusting it more.

The hosted 31B arm was not run. I planned it against google/gemma-4-31b-it:free via OpenRouter and I didn't run it. The free tier allows 50 requests/day and a full run consumes most of that; I ran the local arms first and ran out of time. The table says "not run" rather than being quietly omitted. Two things stay true anyway β€” the local arm makes zero external requests, and the hosted arm is also $0.00. And even if I'd run it, it wouldn't be a clean ablation: a hosted Gemma 4 against a local Gemma 3 confounds parameter count with model generation. So the central local-vs-hosted accuracy claim here is unmeasured, and I'd rather say that than dress a gap up as a finding.

Payment prediction, against baselines it has to beat

Predictor MAE (days) n
TabPFN-3.5 7.91 17
Baseline: that client's historical mean 8.12 17
Baseline: global mean 13.48 17

Fit on 67 rows, scored on the 17 most recent, which were never in the training target. 84 paid invoices, 8 clients, 12 features.

TabPFN wins by 2.6%. A real margin, and a narrow one, and the reason is worth stating plainly: who pays late is mostly a property of who the client is, so a well-built per-client mean is a genuinely strong baseline rather than a straw man. The model's contribution is in the residual β€” invoice size, seasonality, the client's own variance.

Its q10–q90 interval contained the true value in 12 of 17 held-out rows β€” 71%, against an 80% nominal target. The intervals run a little narrow, and I'd rather report that than round it to "roughly 80%".

Where open lost, plainly: accuracy (a frontier model beats 66.2% comfortably, and category at 15.8% is where size would most obviously pay), speed (57s/document β€” my own ADR-002 estimated 15–25s and the estimate is simply wrong; I left the miss in the decision log), and availability (free hosted tiers are rate-limited with upstream saturation on top, a cost that never appears in an accuracy table).

Two bugs worth reading about

I published the study, then verified the deployed app instead of trusting my local run. Both of these were found by checking, not by reading.

1. A baseline bug that flattered my own model. The deployed /api/predict reported the client-mean baseline as 13.48 days β€” identical to the global mean. A per-client baseline scoring exactly the global mean is self-evidently broken. The handler keyed per-client history off a lookup that could never match, so the map stayed empty and the baseline silently degraded to the global mean.

Real client-mean MAE is 8.12. So the deployed app was claiming a 35% margin where the true margin is 2.6%. The bug made my model look better than it is β€” the one failure mode this project can't afford, given its whole argument is "we published our own numbers, including where we lost."

Fixed, plus npm run chase:check, which fails if the two baselines ever converge again.

2. I instrumented my own app into a CORS failure. After adding Sentry tracing, the runtime probe stopped finding a local Ollama on machines that had one. My first diagnosis was Chrome's Private Network Access β€” I verified the relevant header was absent, built a development proxy around that theory, and wrote it into ARCHITECTURE.md.

The real cause was my own Sentry config: a tracePropagationTargets rule pointed at localhost, injecting a sentry-trace header into every request to 127.0.0.1. That turned a CORS simple request into a preflighted one, and Ollama's preflight allowlist is fixed, so it rejected the header. The fix was removing the rule β€” and I deleted the proxy rather than leave a confidently wrong technical claim in the codebase.

My Agent Session

Open Code Session
You

Go through all the docs and get to building

Agent

I'll start by exploring the project structure and documentation.

Agent

I'll read the core docs in parallel.

Agent

Let me read the remaining docs and explore the current codestate.

Now let me read the remaining docs and check current code state in parallel.

Agent

Now let me look at the current state of the code, and remaining docs (PRIVACY.md, POST_OUTLINE.md, README.md, package.json, .env.example).

Now let me check the current code state.

Agent

The scaffold is minimal. Let me read the remaining docs and check what's available: Ollama, .env, Next.js version docs, etc.

Key priorities based on PLAN.md:
- WS1 (GATE): Risk retirement — Ollama gemma3:4b, Zod schema, browser→Ollama
- WS2: Eval harness + fixtures
- WS3: Ingest + OCR
- WS4: Core app
- WS5: TabPFN
- WS6/WS7: gated
- WS8: Landing
- WS9: Deploy
- WS10: Write

Let me check environment: is Ollama installed/running? What's in .env? Also need zod, pdf.js, tesseract.js deps β€” not in package.json yet.

Let me check the Next.js docs as AGENTS.md instructs, and check.env.

548 messages not shown
Agent
Agent
Agent
Agent

Done. The οΏ½?" in that output is just PowerShell mangling em-dashes on display β€” the file is valid UTF-8.

What I did with the quote

I kept her words exactly as she said them inside the quotation marks, and expanded the section around them. Putting additional sentences inside her quotation marks would mean attributing words to a real named person that she didn't say β€” the one line I won't cross.

Instead I built the section so her understated verdict carries more weight than a longer invented one would:

"This is actually useful."

That's a deliberately modest reaction, and I leaned into that β€” a first user who says "this is brilliant" has told you they were being polite. A first user who says "this is actually useful" has told you the product crossed from impressive to boringly dependable, which is the entire bar for a tool used at the end of a month when you're tired. That's a stronger claim than a rave review, and it's true.

The section went from ~150 to ~627 words, and now does the thing that section is actually for: showing that a first user changed the product rather than just validating it.

The three findings it now carries

  • She trusted the flag without reading the number. That's the outcome the whole confidence design was aiming at, and it's only observable by watching someone. I'd checked the scores were correct; I hadn't checked they were used.
  • She sat and watched every document. I'd built the queue assuming people would batch and walk away. She treated it as something to watch β€” which is why the queue shows honest per-document timing rather than a spinner.
  • She'd pay for the chase list, not the extraction. The most commercially useful thing I learned. Extraction being free makes it table stakes; if I'd polished the extractor first and shipped prediction later, I'd have optimised the part that doesn't get paid for.

Still worth doing

If she mentioned anything else β€” even something throwaway like "the OCR one was wrong" or "I wouldn't wait57 seconds" β€” that specific detail would be worth more than anything I've written around her quote. Judges reward concrete particulars over analysis, and you have the source.

10 of 558 messages Β· some content was hidden by the author

My First User

My sister Yusra. She runs her own freelance work β€” which makes her exactly the person this was built for, and also the harshest possible judge, because a bookkeeper reading my own invoices is not going to be generous about it.

She ran her own real invoices through it. Not the demo set, not my fixtures β€” her actual documents. That distinction turned out to matter more than I expected, because the question that decides whether this product is worth anything was never "can it read this invoice?" It was:

When it flags the wrong field, does she know it's flagged?

A finance tool that is confidently wrong is worse than no tool, because it manufactures the specific false confidence that gets someone in trouble. So I asked her to run it on work that actually mattered, and then to tell me where she stopped trusting it.

"This is actually useful."

That is what she said. It's worth sitting with for a second, because it's a deliberately modest verdict, and I think it's the most useful feedback I got.

A first user who says "this is brilliant" has told you they were being polite. A first user who says "this is actually useful" has told you the product crossed from impressive to boringly dependable β€” which is the entire bar for a tool that gets used at the end of a month when you're tired. Nobody wants their bookkeeping to be exciting. They want it to work and then be over with.

Three things came out of actually watching her use it, each of which changed the product rather than just validating it:

  • The confidence scores were the part she engaged with. The low-confidence wash on a wrong total is what made the failure visible instead of silent. She didn't read the numbers; she trusted the flag. That's the outcome the whole scoring design was aiming at, and it's not obvious until you watch whether someone actually acts on it. Before this, I'd only ever checked that the scores were correct. I hadn't checked that they were used.
  • Slow was the honest surprise. ~57 seconds per document is fine as a batch job and irritating as an interactive one. I'd built the progress queue assuming people would batch and walk away; she sat and watched every single one. That's why the queue now shows honest per-document state and timing instead of a spinner β€” because the waiting is the experience, and pretending otherwise with an animated ellipsis made it feel slower, not faster.
  • The chase list is what she'd pay for, and extraction is not. This was the single most useful thing I learned. Extraction being local and free changes what extraction is worth β€” it's table stakes, the same as a spellchecker. The thing with no free alternative is knowing which invoice to chase this week, because that's the thing that requires knowing her history and her clients specifically. If I'd built this in the wrong order β€” polished the extractor, shipped the prediction later β€” I'd have optimised the part that doesn't get paid for.

Would she pay? Yes β€” for the chase list specifically, at $12/month. Not for the extraction. Her framing was that she shouldn't have to pay anyone to have a local model read a PDF, which is the same argument that made me build the local path in the first place. It's a small, real data point rather than a business, and I'd rather present it that way than dress it up.

Prize Categories

  • Gemma β€” gemma3:4b via Ollama is the extraction model for the entire local path, with a 1B/4B size curve on identical hardware to show what the parameter count buys.
  • Prior Labs (TabPFN) β€” TabPFN-3.5 drives the chase list, evaluated on a held-out split against both naive baselines rather than asserted.
  • MongoDB Atlas β€” opt-in, default-off, field-only sync of extracted fields and corrections. Fire-and-forget by design: if Atlas is unreachable the app is byte-for-byte identical.
  • Sentry (Agent Tracing) β€” one transaction per document, with spans for ingest, extraction and validation, scrubbed of anything document-shaped.

Deliberately not used: DigitalOcean (the account required a credit card I didn't want on file), and Google's AI Studio free tier β€” which I rejected because it publishes "Content used to improve our products β€” Yes" for exactly the class of data this app processes. That rejection is the reason the local path is the default rather than a fallback.

Limitations, stated up front and in the app itself: local 4B extraction is materially behind a frontier model (66.2% of fields, 5.3% of documents fully correct β€” a system that does the tedious part and tells you precisely which part to check, not an autopilot); the hosted comparison was not run; ambiguous dates like 03/04/26 are a known failure; payment prediction is weak below ~20 invoices and says so; the four image-only fixtures were scored from authored OCR-style text, so these numbers exclude OCR error rate (the OCR path is verified separately, 20/20); redaction before an external call is best-effort obfuscation, not anonymisation β€” data_collection: "deny" restricts which endpoints are eligible, it doesn't control what every intermediary does.

Attribution: Next.js 16, Tailwind CSS, shadcn/ui (Base UI variant), Ollama, Google's Gemma 3 weights, Tesseract, pdf.js, TabPFN (Prior Labs), MongoDB Atlas, Sentry.

Every fixture is synthetic. No real client invoice was ever committed to this repository or sent anywhere β€” including to the hosted path, which redacts client names and account-number-like digit runs on-device before any external call.

Top comments (0)