DEV Community

Cover image for How I built a fully offline ID card OCR, trained on synthetic data
Luthfi Khusyasy
Luthfi Khusyasy

Posted on AI-assisted

How I built a fully offline ID card OCR, trained on synthetic data

I needed to read KTPs (Indonesian ID cards) for a sign-up form. OCR APIs charge per scan, and when I tried running OCR myself the results were pretty bad :/

So I made my own. The catch is that KTP data is personal data, so I couldn't train on real cards. It turns out synthetic data was enough, and here is my little journey.

Live demo | Source code

Setup

Everything here ran on my laptop, no GPU.

  • Windows 11, Intel i7-12700H, 16GB RAM
  • Python with PyTorch (CPU) for training, ONNX for the browser
  • One training run takes about an hour, so I could try a few ideas in one afternoon

Why in the browser

Mainly because I wanted it free. If the OCR runs on the user's phone, there's no API bill and no server doing the heavy work. Every scan costs me nothing.

Privacy was the other reason. A KTP photo is about the most sensitive thing a user can send you. If the reading happens on their phone, the photo doesn't have to go anywhere, and my server only gets the fields I actually need. Indonesia's data protection law (UU PDP) asks for exactly that, collect only what you need.

And as a bonus, once the model is downloaded it works offline too.

How it works

The pipeline has 4 steps.

  1. Find the card's 4 corners and straighten it (DocAligner)
  2. Find the text lines (PaddleOCR PP-OCRv6 det)
  3. Read each line (PP-OCRv6 tiny, this is the one I trained)
  4. Match the text to the KTP layout, fix it against official lists, and check it against the NIK

The open models were already good at finding the card and the text lines. Reading was the problem. The general PaddleOCR reader got only about half of the KTP lines exactly right, so that's the only part I trained myself. The others I just compressed.

The result is JSON with 21 fields, and using it is one HTML element.

<script type="module">import "ktp-kit/element";</script>
<ktp-capture base="/ktp-kit/" lang="en"></ktp-capture>
<script>
  document.querySelector("ktp-capture").addEventListener("ktp-result", e => console.log(e.detail));
</script>
Enter fullscreen mode Exit fullscreen mode

Synthetic data

Since I can't use real KTP photos and data, the only other way was collecting cards with consent, and that takes way too much time and effort. So synthetic data it is.

I wrote a generator that draws single KTP text lines, like labels, names, addresses and NIKs, in KTP fonts and layout. Then it degrades them on purpose so they look like phone photos, with blur, glare, fading, tilt and JPEG artifacts. Names are put together from common name parts, made-up syllables and random letters, so the model learns letters instead of memorizing names. The NIKs use sequence numbers no real person has.

Generated KTP lines and the text the model should read
Lines from the public ktp-kit KTP line benchmark (CC BY 4.0). Blank strips are there so the model learns to read nothing.

My first version was too clean, and the generated lines were too far from real printed cards. So I looked closely at my own KTP. The values are printed bolder than the labels, the letters are squeezed together, and the ink has tiny gaps in the strokes. I added all three to the generator, and that version is the one I shipped.

I also tried training more layers of the model. It got worse. The model got better at my fake lines and worse at real print. The real limit was how realistic the data was, not the model's size.

Training

I didn't train from scratch. I started from PaddleOCR's PP-OCRv6 tiny reader, which already knows how to read text in general.

The original model can output 6,906 different characters, because it covers Chinese too. A KTP only uses 46 (A to Z, 0 to 9 and a few symbols). So I cut the output down to those 46 plus one "blank", and kept the pretrained weights for them. Fewer characters means fewer ways to be wrong, and a smaller model, about 0.55M parameters.

Then I fine-tuned it on the generated lines, on CPU, about an hour per run.

Making it fit in the browser

I wanted the whole thing small enough that a phone on mobile data doesn't feel it. When I started, the download was 7.7 MB. Here's what got it under 5.

int8 weights. All three models store their weights as 8-bit integers instead of 32-bit floats. My reader didn't lose any accuracy. I also tried 4-bit on the line finder, but it started missing text, so it stayed at 8-bit.

A smaller ONNX runtime. The stock onnxruntime-web WebAssembly file is 11.25 MB. It can run almost any model, but mine only need a few operators. onnxruntime can be built with only the operators you list, so I built it from source with just mine. It came out at 3.75 MB, and 0.87 MB after compression.

The build wasn't smooth. CMake 4 refused some of the old dependencies, so I went back to CMake 3.31. Then building with 10 parallel jobs ran out of memory on my 16GB laptop, and 4 jobs worked.

Brotli. Every file is pre-compressed with brotli, so the download size is the same on any host.

Part Download
Runtime 0.87 MB
Line finder 1.71 MB
Card corners 1.14 MB
My reader 0.56 MB
Region list 0.53 MB
Total about 4.9 MB

After the first visit everything stays in the browser's cache, so it works offline.

Speed. WebAssembly can use several CPU threads, but only when the page sends two headers (COOP and COEP). With them a read is about 3x faster, around 2 to 4 seconds per card on a mid-range phone.

Results

I tested on 2,400 generated lines that were never used for training or for picking the model. The data and scripts are in the repo, so anyone can rerun it.

Reader NIK lines All text lines Blank strips Size
Mine 90.8% 81.5% 97.0% 0.7 MB
PP-OCRv6 small 91.6% 51.4% 70.5% 21.2 MB
PP-OCRv6 tiny (where I started) 87.8% 48.7% 71.1% 4.5 MB

On the NIK the general models are just as good, even a bit better. The difference is on names, addresses and other words, and on not "reading" text into the card's background pattern. That's the blank strips column.

But to be honest, these lines come from the same kind of generator my model learned from, so of course it does well here. I don't have real-card numbers I can publish yet. If you try it on your own card, I'd love to hear how it goes.

Limits

  • Names are the weakest field. The result has a perlu_dicek ("needs checking") list that flags fields the model was unsure about, so your form can ask the user to look again.
  • Laptop webcams are usually too blurry. Phones work much better.
  • The generator itself isn't open source, only the trained model and the benchmark.

Try it

This is my first time building a system like this, so if you know a better way or spot something wrong, please let me know in the comments!

Top comments (0)