This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
My neighbours run a grocery shop. Like thousands of small shops, they sell a lot on credit and keep track in a khata: a paper ledger where every row is a customer, a date, an item, and either udhar (the customer took goods on credit) or jama (they paid something back).
It works until you need an answer. Who owes what? Who hasn't paid in over a month? The answer is somewhere in the pages, and finding it means turning them over by hand.
So I built Ledgerly. You photograph a page. A vision model reads every row. You get a review screen with your photo beside the extracted rows, fix whatever it got wrong, confirm, and the app keeps running balances per customer, with an aging report: who is over 30 days.
The part I care about most is the loop. Every page you confirm carries your corrections, and those corrections are the training data for the next fine-tune (python ft/train.py --from-db). Using the app produces training data.
It reads Hindi (Devanagari), English, and the mixed pages that real ledgers actually are.
Demo
The demo covers gemma running locally and extracting entries from a handwritten ledger page and other functionalities of Ledgerly web app.
What my neighbours said
I asked my neighbours to try it. They found the screens easy to understand almost immediately: the photo next to the rows, the big buttons, the "owes money" and "over 30 days" filters. Their own khata is laid out differently from the pages I trained on, so they couldn't run their book through it yet. They told me they would definitely use it if I rolled it out as an app.
That told me what to build next. The model has to learn their page, and that is what fine-tuning is for: photograph a few pages, label them, and train again for about forty cents.
The side hustle
Who would pay: small shops that sell on credit and still run their books on paper, like my neighbours. They don't need another accounting suite. They need to stop turning pages to find out who owes what.
What I'd sell: the model and the pipeline are open and stay free. I'd charge for the convenience: a hosted version where the shopkeeper photographs a page and the ledger is ready, plus the part that matters most, teaching the model that shop's own page layout. My neighbours' khata looks different from mine, so a one-time setup is exactly what they need: a few photographed and labelled pages, then a fine-tune. The compute for my 9-page run cost about forty cents.
Pricing:: I asked my neighbours what would feel fair for an app like this, and they said ₹100 to ₹150 a month. So I'd offer a free tier of about 20 pages a month, so a shop can try it on its own khata, then roughly ₹120-₹130 a month per shop. A one-time setup would teach the model that shop's own page layout. That's a few rupees a day, far less than the hours lost to the paper book, and in line with what small shops already pay for billing apps.
The price comes from one shop's answer and the idea hasn't been tested beyond that, but they told me they would use it.
Code
Ledgerly
Photograph a page of a paper credit ledger. A local Gemma model (via Ollama) reads it, you correct mistakes, and the app reports who owes what and for how long.
Layout
-
extraction/Step 1: model prompt, schema, single/batch runners, scoring -
backend/Step 2: Django API (SQLite, no login) -
frontend/Step 3: React + TypeScript app (Vite) -
finetune/Optional: fine-tune a vision model on Tinker and compare with the baseline (see its README) -
data/pages/test photos (gitignored: may contain customer data) -
data/ground_truth/your hand-written correct answers, one JSON per photo -
data/outputs/model outputs per run (gitignored)
Setup (Windows PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
ollama pull gemma4:e4b
Step 1 workflow
- Put photos in
data/pages/(page01.jpg, page02.jpg, ...). - For each, copy
data/ground_truth/_template.jsontopage01.jsonand fill in the true rows Set "page_type" to neat, messy or mixed. - Edit the layout line in
extraction/prompt.txtto match your ledger. - …
Django + SQLite backend, React + TypeScript frontend, and an extraction/ft/ folder that holds the whole fine-tuning pipeline: dataset.py, train.py, sampler.py, evaluate.py, compare.py.
How I Built It
photo ──▶ preprocess
──▶ vision model ──▶ JSON { rows: [ {date, name, item, amount, type, struck_out} ] }
──▶ pydantic schema check
──▶ review screen (you fix rows) ──▶ confirm ──▶ ledger + aging report
│
corrections ◀───────────────┘ (next training run, --from-db)
Each row has exactly six fields. type is credit (udhar) or payment (jama), and struck_out is true when the shopkeeper has crossed a row out, so it is stored but not counted:
{ "date": "15/10", "name": "Ramesh Kumar", "item": "Atta (5 kg)",
"amount": 240, "type": "credit", "struck_out": false }
Step 1: the baseline, with a local Gemma
The first version ran Gemma 4 e4b locally through Ollama. Nothing leaves the computer, which matters for a book full of customers' names and debts. It works, but it takes about 159 seconds per page on my CPU.
Step 2: the data (the boring part that decides everything)
I labelled 14 pages by hand (took me 2 hours), one JSON of ground truth per photo. Then I split them so the model is judged on pages it has never seen:
- Held out (5 pages, 50 rows): page02, page04, page06 (my original neat, messy and mixed pages) plus page09 and page10.
- Trained on (9 pages): the rest.
I picked page09 and page10 as test pages on purpose. Several of my new pages repeat each other's customers and amounts (page 8 and page 14 are nearly identical). If a test page has a twin in training, the score is inflated. Those two had no twins.
Nine pages is tiny, so dataset.py adds three augmented copies of each page (small rotation, brightness, contrast and crop). That makes 36 training examples, 79,764 tokens per epoch.
Step 3: fine-tuning on Tinker
Choosing the model
The input is a photograph, so I needed a vision model. My first choice, Qwen3-VL-30B-A3B, died on the first real run with 400: base_model ... is not supported. The public docs still listed it. So I asked the server what my key could use (ServiceClient().get_server_capabilities()), and the list had moved on to Qwen3.5 and Qwen3.6, plus Nemotron and others.
I picked Qwen3.6-35B-A3B: a mixture-of-experts model with about 3B active parameters, which keeps training and sampling cheap, in the same size class as the model I'd planned. A free --dry-run rendered every example with the real tokenizer and image processor before I spent anything, and the renderer it chose (qwen3_5) matched what tinker_cookbook recommends for that model.
I used Tinker to LoRA fine-tune it with the settings in train.py:
| Setting | Value |
|---|---|
| Method | LoRA, rank 16 |
| Epochs / batch size | 4 / 4, so 36 steps |
| Learning rate | 1e-4, decaying to 1e-5 |
| Renderer |
qwen3_5 (matches get_recommended_renderer_name) |
| Training cost | about $0.38 of credit ($10.00 to $9.62) |
Every example is a (photo, correct JSON) pair.
epoch 1/4 step 1/36 loss/token 0.1466
epoch 1/4 step 9/36 loss/token 0.0185
epoch 2/4 step 18/36 loss/token 0.0050
epoch 3/4 step 27/36 loss/token 0.0013
epoch 4/4 step 36/36 loss/token 0.0008
I'm not treating that curve as a result. It's loss on training pages the model saw four times each, so it says the run was healthy, not that the model is good.
Step 4: the evaluation
ft/evaluate.py runs the untouched base model and the fine-tuned model on the five held-out pages and scores both with my own score.py. Each true row has to be matched to a predicted row, and then name, amount, date, type and crossed-out flag are checked. A row counts as "full" only when all five are right.
The base model scored 0%. I want to be precise about why. It did not read the pages badly. On all five pages it failed to produce valid JSON for my schema (wrong shape, no JSON object at all, or text after the JSON). score.py can only score what it can parse, so every page became "found 0 rows". That number measures instruction-following on one schema, not handwriting, so I'm not using it as my "before". Gemma is my baseline, because it produces valid output.
Fine-tuned Qwen against local Gemma, 50 rows, same pages, same checker:
| Gemma 4 e4b (local) | Qwen3.6 fine-tuned (Tinker) | |
|---|---|---|
| Full row correct | 41/50 (82%) | 44/50 (88%) |
| Name | 46/50 | 45/50 |
| Amount | 48/50 | 49/50 |
| Date | 47/50 | 49/50 |
| Type (credit vs payment) | 44/50 (88%) | 49/50 (98%) |
| Crossed out | 47/50 | 48/50 |
| Extra rows invented | 0 | 0 |
| Seconds per page | 159 s (laptop CPU) | 12 s (Tinker cloud GPU) |
Fully right rows by kind of page (this is the part I'd look at before trusting any single percentage):
| Page type | Rows | Gemma 4 e4b (local) | Qwen3.6 fine-tuned |
|---|---|---|---|
| Neat (page02, page09, page10) | 30 | 26 (87%) | 28 (93%) |
| Messy (page04) | 10 | 7 (70%) | 10 (100%) |
| Mixed Hindi/English (page06) | 10 | 8 (80%) | 6 (60%) |
What stands out:
- Credit versus payment went from 88% to 98%. In a ledger that is the field that matters most: mixing up udhar and jama flips the sign on someone's balance.
- Three more rows fully right out of 50 (82% to 88%), with dates, amounts and crossed-out flags all a little better and no invented rows.
- 12 seconds a page instead of 159. That compares a cloud GPU with my laptop CPU, so read it as "hosted versus on my machine", not as a model-size result.
- Where it's level or behind: names (45 against Gemma's 46) and my one mixed Hindi/English page (6 of 10 rows against 8). On the neat and messy pages it got 28 of 30 and 10 of 10 rows fully right.
With 50 test rows, a one or two row gap is noise, so I trust the type and speed results most.
Two details from walking through every row in compare.html:
- One label was my mistake. On page06, row 7, my ground-truth JSON said the row was crossed out, but I never crossed it out on the paper. Both models said "not crossed out", so both were marked wrong for being right. I corrected that one label after seeing the results and rescored both models, so the numbers above include the fix.
- One real miss: the fine-tuned model wrote "Pooja" in Latin letters where the ledger has the name in Devanagari. It's the same person, but the tool is meant to copy what's written, so I left it counted as wrong. Several of the other name misses were rows Gemma read correctly.
Step 5: seeing the mistakes, not just the percentages
ft/compare.py builds one HTML page: my photo, my ground truth, and every model side by side, with wrong fields in red and the correct value underneath. I used it to find that the mixed page was where the fine-tuned model lost rows.
Using it in the app
One environment variable decides who reads the page:
$env:LEDGERLY_EXTRACTOR = "tinker" # default is the local Gemma via Ollama
The default keeps every photo on the computer. With tinker, each photo is sent to Tinker's cloud to be read by the fine-tuned model, and the README says so. The API key lives in an environment variable and nowhere in the repo.
What I got wrong on the way
- I almost trained on three pages. I had put all eight new pages into the held-out list. That left the training set exactly where it started. Reading the dry-run's "36 examples" line is what caught the mistake.
-
My new photos were
.jpeg, and the pipeline looked for.jpg. They were skipped without a single warning. Renaming fixed it.What I learned
Label quality matters as much as quantity. Some of the mistakes I found in
compare.htmlwere mine, not the model's: a "crossed out" label on a row I never crossed out. A model can only be as consistent as the labels it is trained and judged on.Measure the baseline fairly. Base Qwen's 0% looked dramatic and meant "can't follow my JSON format". Gemma was the honest comparison.
Look at the rows, not just the percentages. The scoreboard said the fine-tuned model was a little better. The side-by-side page showed where: credit versus payment and the neat and messy pages, not the mixed one.
Dry-run first. The whole fine-tune cost about forty cents because I rendered everything for free before spending any credit.
What's next
-
Learn the neighbours' khata. Label a few pages in their layout and run the same pipeline. Every page they confirm in the app becomes training data (
--from-db). - More writers. My own handwriting is in all 14 pages. Pages from other hands will show how far the model carries.
- Publish the adapter on Hugging Face (Tinker can push a checkpoint straight there), so anyone can run it on their own hardware.
- Package it as an app so a shopkeeper can use it without a laptop.
The review screen is part of the design: the model proposes the rows, the shopkeeper confirms them, and nothing reaches the ledger unchecked.
Why Does Open Innovation Matter?
- It stays on the shopkeeper's computer. A khata is a list of who owes whom. With open weights the default path never uploads a photo, and the cloud path is opt-in and said out loud.
-
I could change the model and measure it. Tinker let me fine-tune open weights for about 38 cents, and I could run that against an open baseline on the same pages with the same scorer. Swapping the base model is one variable (
TINKER_BASE_MODEL). - I could see why it was wrong. The comparison page shows each wrong field next to the right value, so I could tell a reading error from a spelling variant instead of trusting one score.
-
The result is reproducible. Open weights, my labelled pages, my scorer and the saved outputs are all in the repo, so anyone can rerun the comparison.
Prize Categories
Best Use of Tinker: LoRA fine-tune of Qwen3.6-35B-A3B on 36 augmented examples, scored on 5 held-out pages against a local baseline, for about $0.38.
Best Use of Gemma: Gemma 4 e4b runs locally through Ollama as the private default and as the baseline the fine-tuned model has to beat.
Built with Claude and tea ☕.
Top comments (0)