OCR is not my area. I mostly read OCR comparisons from the outside, and a percentage in one of them tells me little until I know how it was scored. So I ran three engines over two clean printed pages, page 8 of the 1895 Time Machine and a 1986 Chinese official document, on a Mac (Apple M4) in a desktop browser, and wrote down the scoring choices that moved a number. The engines were Tesseract, the text recognition built into macOS, and ImgIng https://imging.ai/ , a browser tool that downloads PP-OCR models on first use. Every figure below is a single run on those two pages, so read them as examples of how scoring choices matter, not as engine rankings.
Edit distance and lost characters catch different failures
Edit distance counts every insertion, deletion and substitution needed to turn the output into the truth. It punishes order: when a line comes out in the wrong place, the moved line can cost almost as many edits as it has characters. Lost characters ignores order. Treat both as bags of characters and count what is left in the truth after subtracting the output. Both functions below are a simplified illustration written from the definitions; the figures in this post come from a fuller scoring script that normalizes the text first.
import collections
def levenshtein(truth, ocr):
prev = list(range(len(ocr) + 1))
for i, tc in enumerate(truth, 1):
cur = [i] + [0] * len(ocr)
for j, oc in enumerate(ocr, 1):
cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (tc != oc))
prev = cur
return prev[-1]
def missing_chars(truth, ocr):
left = collections.Counter(truth)
left.subtract(collections.Counter(ocr))
return sum(n for n in left.values() if n > 0)
Swap two lines in a toy string and the first climbs while the second stays at zero. On real pages the pair separates two problems. With the English page rotated 3°, ImgIng's output scored 400 edits against a 1,002-character truth, yet only 1 character was missing: each paragraph's first line had been pushed below the next one. That was a reading-order problem, and a report quoting only the 400 would hide it. On the Chinese page (557 characters) rotated 5°, the same tool gave 362 edits and 275 lost. That is real loss: of 14 lines, about half survived. The interface still said processing was complete, with no warning. Upright, that page came back with 2 errors.
The ground truth needs its own provenance
For the English page I used the Wikisource transcription of that exact page, marked Validated, which means two people proofread it. I added the running header and page number, since they are on the scan. That choice decided ImgIng's English score on the upright page: its only error was the missing page number 8. Leaving the page number out of the truth should have removed that error, but I did not re-score it that way.
The Chinese truth started from 556 characters checked sentence by sentence earlier, plus one visible full stop, making 557. One character, 雠, was re-checked under magnification. ImgIng's output gave the traditional form 讎, which I counted as an error. Someone else could count it as correct, so the rule belongs in the report.
Normalization rules move the score before any engine runs
For Chinese I removed all whitespace and folded full-width and half-width punctuation and brackets together. For English I unified quotes and apostrophes, merged hyphenated line breaks, collapsed whitespace and unified dashes. Marker lines the tools add, such as "[Image OCR]", were stripped first. Tesseract 5.5.3 made 11 errors on the upright English page, mostly opening quotes read as ' or «, plus "It" as "Tt". A stricter quote rule would change that number, and I did not rerun it to see by how much.
The engine settings belong in the row label
macOS built-in text recognition (the system Vision framework, not the Preview app) in accurate mode with language correction made 5 errors on the Chinese page. In fast mode the same page gave 526, and the English page went from 0 to 260. My guess is that fast mode skips Chinese, but that is inference. Tesseract here had only the English language pack installed, so its Chinese column is "not tested". That is not a zero.
Synthetic skew and noise are not a scanner
All skew here came from rotating the page onto a larger white canvas. Noise was Gaussian (σ 12 or 25) or 5% salt-and-pepper from a fixed seed. The English page already had about 0.3° of its own tilt, so "3°" measured as 3.3°. Format leaks in too: the 3° English page as JPEG at quality 88 cost Tesseract 23 errors, while a PNG 3° rotation scored 98.1%. I cannot say format is the only difference, so they stay in separate rows. One cell looks like a finding. At 5° with only a 3×3 median filter, Tesseract dropped to 34.0%. The output had the column cut into blocks with sentences interleaved, which is layout analysis breaking, not characters misread. Deskew the same image first and it reads 98.9.
So the header I now want above any OCR table has five lines: truth source and what it includes, normalization rules, edit distance next to lost characters, engine version with mode and language packs, and whether degradation is real or synthetic, with the image format. Real scanner skew, page curl and handwriting were not tested. On a small table sample I only scored character pairs, not whether the structure survived. I still have no scoring rule for structure that I trust.
Samples: Chinese page from the 1986 Table of Simplified Characters (National Language Commission), Internet Archive, Public Domain Mark 1.0. English page from H. G. Wells, The Time Machine, Heinemann 1895, Wikimedia Commons, public domain. Skewed and noisy versions are my own.
Top comments (0)