Why Our 13 MB GST Invoice AI Failed — And How a 107 MB Desktop Engine Hit 95.95% Entity F1
We built an offline AI engine to read Indian GST invoices. Then we made it too small, and it broke.
This is the candid failure analysis: why we abandoned the phone-first constraint, what the 13 MB model actually got wrong on real bills, and what a 107 MB desktop engine changed.
Not a production-ready accounting product.
The Tally XML path is validated against a mock server, not a live TallyPrime install. We haven't run a live trial import, so we make no claim about voucher creation or automated ledger entries inside Tally. Everything else below is a measured benchmark on our own holdout set.
The Siren Song of Extreme Quantization
Part 1 of this series took a LayoutLM-style document model from 430 MB down to 13 MB so it would run on a ₹7,000 Android phone in a warehouse with no reliable internet. The promise was real: zero cloud cost, full data residency, and an OCR-to-voucher pipeline that never phones home.
We shipped that. In Part 1 we measured it at 64.2% Entity F1 on an 18-page evaluation set drawn from four vendor families the model had never seen. It looked like success.
The Brutal Truth: Where 13 MB Broke Down
Then we pointed it at real distributor invoices.
Re-measuring the same 13 MB model on a messier holdout that included genuinely real bills gave 62.6% Entity F1. At that accuracy, an accountant re-types roughly 4 of every 10 fields. The model was not saving time — it was generating a new review queue.
Three failure modes showed up repeatedly:
- Truncated line items. Multi-line descriptions with a subtotal row and a discount row lost the relationship between them.
- Column drift. When a vendor shifted a table by a few millimetres, the token classifier assigned cells to the wrong field.
-
Rate-to-amount confusion. Fields like
9%and18%scored approximately 0% F1 on unseen vendors in Part 1. Amounts survived; attribution did not.
The Product Epiphany: We Solved the Wrong Constraint
The failure wasn't a tuning problem. It was a constraint problem.
We optimised for a device nobody does accounting on. The bill book gets typed at a desk, by an accountant, who then needs the output to land in a Tally ledger. The phone was a capture convenience, not the compute target. We had let the smallest possible artifact dictate the whole architecture.
So we inverted it: capture on the phone, infer on the desktop.
The Upgrade: LayoutLM-Base Quantized to 107 MB
We moved to Microsoft LayoutLM-base (113M parameters), MIT-licensed for commercial use, and INT8-quantized it through ONNX Runtime CPU.
| Model | Size | Entity F1 | Notes |
|---|---|---|---|
| Part 1 quantized model | 13.17 MB | 62.6% | Phone-first, failed on real bills |
| LayoutLM-base INT8 | 107.66 MB | 95.95% | Desktop-first, offline |
We deliberately chose LayoutLM v1. LayoutLM v2 scores higher on document benchmarks and v3 scores higher still — but v3 carries non-commercial terms and v2's licensing differs. For a product intended for Indian accounting firms, an MIT licence is worth more than a leaderboard delta.
Dataset Breakthrough: Multi-Layout Fusion
The 13 MB model had been trained on a single dominant layout. The 107 MB model was trained on 1,357 annotated invoices spanning four Indian layouts — kirana store bills, distributor invoices, and two additional families.
One caveat we want to be precise about: most of that training set is either our own verified kirana data or programmatically generated synthetic data (CC BY 4.0, from the public Divya52/indian-gst-invoices set). The public dataset card itself warns that models trained on synthetic layouts alone are weaker on real photographs. The 95.95% holdout includes synthetic samples too. Expect a lower figure on genuinely messy real-world bills until we re-benchmark on a purely real holdout — that's on our list.
The Payoff: 95.95% Entity F1
On a 136-invoice holdout the model never saw during training, scored with strict Entity F1 requiring exact character-level match across all multi-word tokens:
95.95% Entity F1.
Latency on a single-core x86 CPU is ~792 ms per page — about half a minute for a 40-page bill book. The ~140 ms figure we quote on Apple Silicon is not what a ₹7,000 Android box or an old office PC will see. We report the pessimistic number because that's the one you'll feel.
Inference uses roughly 550–650 MB of RAM, so a 4 GB or 8 GB machine runs it comfortably.
The Cost Comparison: Tally's Docs by Ira vs a Local Desktop Engine
Tally's OCR product, Docs by Ira, is the first product under TallyIra. It requires TallyPrime Release 7.1 or above with an active TSS subscription, and it does not support handwritten documents.
| Docs by Ira | Local desktop engine | |
|---|---|---|
| Price | ₹200 + GST per 100 pages | ₹0.00 marginal |
| Effective per page | ₹2.36 | ₹0.00 |
| ~36,000 pages/year | ~₹85,000 | ₹0.00 |
| TallyPrime + TSS required | Yes (Release 7.1+) | No |
| Handwritten bills | Not supported | Not evaluated |
| Offline | Yes | Yes |
| Data leaves your machine | No | No |
| Live Tally import validated | Vendor-supported | No — mock server only |
On marginal extraction cost, the local engine wins at volume. The honest framing: Docs by Ira is a mature, supported product inside an accounting suite you're probably already paying for, and its per-page cost drops on larger packs. The local engine trades breadth of support for zero marginal cost and full data residency.
What's Still Hard: Honest Limitations
- Synthetic training data dominates the public set. Covered above — expect lower real-world F1.
- LayoutLM v1 is an older architecture. Chosen deliberately for the MIT licence.
- Latency on low-end x86 is real. ~792 ms/page single-core.
- Not a production-ready accounting product. The XML envelope is validated against a mock Tally server (ledger auto-create plus idempotent re-import), not a real TallyPrime install. We haven't run a live trial import, so we can't claim voucher creation or automated ledger entries work inside Tally. Treat the Tally output as an unverified export format until that test passes.
- Tax rate fields remain the weak spot. Amounts improved substantially; rate-to-amount attribution has not been re-measured on this run.
Key Engineering Takeaways
- Constraints choose your architecture. The 13 MB model wasn't badly trained — it was badly aimed. We picked the smallest artifact and let it dictate everything downstream.
- Real-world holdouts expose what random splits hide. 64.2% on a clean eval set, 62.6% on real bills. The second number is the honest one.
- Size is a cost, but not the only cost. 107 MB was the point where accuracy crossed the line where a human stops re-typing.
- Licence terms can outrank benchmark deltas. MIT over non-commercial, no contest for a commercial product.
- Disclose the unverified path. A mock server is not a live integration. We would rather publish a gap than let you find it in a live trial import.
What's next
- Re-benchmark on a purely real, un-synthetic holdout set and publish the delta
- Validate a live TallyPrime trial import end-to-end, not just against a mock server
- Re-measure tax rate field F1 on the 107 MB model
- Profile the quantised ONNX graph for a smaller footprint without regressing F1
Sources
- microsoft/layoutlm-base-uncased — Hugging Face model card — architecture and usage (checked 11 November 2026)
- microsoft/unilm — LayoutLM repository README — parameter counts (113M base) and IIT-CDIP pre-training details (checked 11 November 2026)
- LayoutLM — arXiv 1912.13318 — the original paper (checked 11 November 2026)
- LayoutLMv3 — arXiv 2204.08387 — non-commercial terms, which is why we did not use it (checked 11 November 2026)
- Divya52/indian-gst-invoices — Hugging Face dataset card — synthetic origin, CC BY 4.0 licence, stated limitations (checked 11 November 2026)
- ONNX Runtime — INT8 quantisation and CPU inference (checked 11 November 2026)
- TallyIra — Docs by Ira — scope, Release 7.1 and TSS requirements (checked 11 November 2026)
- The Total Cost of TallyPrime in India — Docs by Ira at ₹200 + GST per 100 pages (checked 11 November 2026)
- Tiny GST repository — GitHub — source code (checked 11 November 2026)
About this post: Written and fact-checked by San at PyBankers. Benchmark figures are our own internal measurements and have not been third-party reproduced. Vendor claims about Tally Solutions products and pricing are quoted from Tally's own published pages and were re-checked on 11 November 2026. A factual error in the original version of this article — regarding the Tally–AWS collaboration and the per-page pricing unit — has been corrected.
Read the full engineering article on PyBankers:
Why Our 13 MB GST AI Failed — And How a 107 MB Desktop Engine Hit 95.95%

Top comments (0)