Take a photo of a printed page and ask your computer to type it out. The job sounds like one step — "read it" — but a computer starts from a place so humble it's almost funny: all it has is a grid of millions of brightness numbers. Nothing in that grid says "letter." Nothing says "word." There is no text in a photo of text.
Optical character recognition — OCR — is the staircase from those raw numbers up to the sentence you can copy-paste. It has only three stairs, and each one is honest enough to explain completely. Let's climb them with a real example: a phone photo of meeting notes, slightly shadowed on one side, like every photo ever taken in a hurry.
Step 1 — Every pixel becomes ink or paper
First problem: the photo is a mess of grays. The paper isn't white (it's in shadow on the right), and the ink isn't black (cameras soften everything). Before any reading happens, OCR wants a brutally simple world: every pixel is either ink or paper. Nothing in between.
How do you pick the cutoff? Count the pixels. Make a histogram — a bar chart of "how many pixels have each brightness, from 0 (black) to 255 (white)." A photo of text always shows the same beautiful shape: two hills. A big hill of bright pixels (all that paper) and a smaller hill of dark pixels (the ink). Between them: a valley.
The cutoff should sit in that valley — and a method from 1979 called Otsu's method finds it with pure arithmetic: try every possible threshold from 0 to 255, and for each one, measure how cleanly it splits the pixels into two tight groups. Keep the threshold with the cleanest split. That's the whole algorithm — a for-loop, no AI. On our sample photo it lands on 104:
Every pixel darker than the valley becomes pure black; everything brighter becomes pure white. The shadow vanishes — because the shadow's pixels, dark as they were, still sat on the bright side of the valley. This step is why decent OCR tools shrug off bad lighting that would ruin a naive "is it dark?" rule.
Step 2 — Find the text before reading it
Now the page is black ink on white paper, but the computer still doesn't know where the words are. The fix is geometry, not intelligence.
Black pixels that touch each other get grouped into clumps — each clump is a letter (or a few letters stuck together). Clumps that sit close together on the same baseline become a word. Words sharing a baseline become a line. It's exactly how you'd organize spilled beads: things that are near each other and aligned probably belong together.
The output of this stair is a stack of little image strips, one per line of text, each handed to the reader with a note attached: "this strip contains exactly one line — read it."
Step 3 — Reading is a sequence, not a lookup
Here is where old OCR and modern OCR part ways, and the difference teaches a real lesson about machine learning.
Old OCR tried to cut each word into individual letters and match each letter against stencils — does this blob look like the "B" template? It worked on clean typewriter fonts and fell apart on everything else, because real text refuses to be cut: letters touch, fonts vary, and in cursive scripts the letters flow into each other.
Modern OCR — the engine in our tool is Tesseract, with its neural "LSTM" reader — stops trying to cut. Instead, a small network slides along the strip like a reading head, a few pixels at a time. At every step it shouts probabilities: "81% this is part of a d, 9% maybe cl…" — and crucially, it carries memory of what it has already seen. Having read "Budge…", it strongly expects a t next. (The memory cell it uses is called an LSTM — "long short-term memory" — which is exactly what it sounds like: a little notebook the network keeps as it reads.)
But wait — the head takes twelve little steps across a six-letter word, so its raw output is a stutter: "B B u u _ d d g _ e t t". A wonderfully simple trick called CTC turns the stutter into text: collapse the repeats, drop the silences. "BBuu_ddg_ett" becomes "Budget". The deep consequence: nobody ever has to tell the network where one letter ends and the next begins — it learns its own timing from examples, which is precisely what made letter-cutting unnecessary.
Why Arabic is the real test
Our tool reads English and Arabic — and Arabic is where the sequence approach proves itself. Arabic script is cursive by design: most letters connect, and a letter changes shape depending on whether it sits at the start, middle, or end of a word — the same letter can look four different ways. Stencil-matching has no chance. But a reader with memory, sliding along the line (right to left, this time), handles it the way a human reader does: shape-in-context. That's why the tool ships two trained models, one per language, each a few megabytes of learned weights — downloaded once to your browser, cached, and run locally.
And that sentence is doing a lot of work: locally. Receipts, contracts, IDs, prescriptions — photos of text are the most sensitive images people process online, and the standard OCR site uploads them to a server. Ours doesn't have a server to upload to. The staircase you just climbed — histogram, valley, clumps, boxes, sliding reader — runs inside your tab, and you can watch the network panel stay silent while it works.
Try it on your own messy photo — English, Arabic, or both at once: jigsawlabs.tools/tools/image-to-text — free, no signup, and the photo never leaves your device.
Originally published on the JigsawLabs Blog. This is the third post in a series on the engineering inside JigsawLabs tools: the 55 KB brain that formats documents and how an AI cuts you out of a photo.
Top comments (0)