OCR for Hebrew Text With and Without Niqqud

#ai #machinelearning #llm #programming

A Hebrew page comes back from OCR looking almost right. The consonants are there, the line breaks are there, and every vowel point is gone. This is rarely a recognition failure. In most pipelines the points were deleted by image preprocessing, before the recogniser saw the page at all.

What niqqud is, in code points

Hebrew is an abjad: the twenty-two letters are consonants, and vowels are optional marks written around them. Modern Hebrew prints almost nothing with those marks. They survive in poetry, children’s books, dictionaries, liturgy and scripture, which is exactly the material anyone digitising Hebrew is likely to be working with.

In Unicode the marks live in the Hebrew block, U+0590–U+05FF, and they are combining characters that follow their base letter in the string. The vowel points proper run from U+05B0 (sheva) to U+05BB (qubuts), with U+05BC (dagesh or mapiq), U+05BD (meteg) and the shin and sin dots at U+05C1 and U+05C2 alongside them. A biblical text adds the cantillation marks, U+0591–U+05AF, above and below the same letters. A single consonant in a pointed Tanakh can carry a dagesh inside it, a vowel beneath it and an accent above it — four code points for one visual unit.

Two of those are unusual enough to matter for OCR. The dagesh is drawn inside the letter, not next to it, so it changes the letter’s own shape rather than adding ink beside it. The shin and sin dots sit on the letter’s upper arms and are the only thing distinguishing two different phonemes that share one base code point.

Where the points are lost

Four stages remove niqqud, and all of them run before the model does.

Resolution. A sheva is two dots. At 300 dpi, on 10-point type, each dot is roughly two to three pixels across. Scale the image down to the recogniser’s expected line height — which most pipelines do, because the model was trained on a fixed height — and the dots resample into a grey smudge or vanish.
Binarisation. A global threshold picks one cutoff for the whole page. The consonant strokes are thick and survive any reasonable cutoff; the points are thin, so their pixels sit closer to the paper value and fall on the wrong side of it wherever the scan is slightly light. Adaptive thresholding is a genuine fix here, not a refinement.
Despeckling. This is the big one. Noise removal drops connected components below a minimum area, and a vowel point is a small, isolated connected component that is not touching anything. It is, structurally, indistinguishable from dust. Many pipelines enable this by default.
Baseline assignment. Line segmentation fits one baseline per line and crops a band around it. Niqqud sits below the baseline, in the same vertical band as the descenders of the final forms ך, ן, ף, ץ and the letter ק. Crop tightly enough to separate adjacent lines and you crop the vowels off with the whitespace.

There is a fifth, quieter loss: the training data. A model trained on modern Hebrew newsprint has seen essentially no pointed text, so even when the ink survives every stage above, the recogniser has no class for it and will emit the bare consonant with high confidence. Softmax renormalises over the classes that exist; absence of a label does not show up as low confidence.

The pairs that get confused

Where points do survive, a predictable set of substitutions dominates the error profile, and knowing them tells you whether you are looking at a segmentation problem or a classifier problem.

Qamats and patah (U+05B8 vs U+05B7) — a horizontal bar, with and without a small downward tail. At low resolution the tail is the first thing to disappear, so errors run overwhelmingly in one direction, qamats read as patah.
Segol and sheva (U+05B6 vs U+05B0) — three dots in a triangle against two dots vertically. Lose one dot to despeckling and one becomes the other.
Holam and the shin dot (U+05B9 vs U+05C1) — both are a single dot above and to one side. They are told apart only by which letter they belong to and where exactly they sit, which is a spatial-assignment decision, not a classification one.
Dagesh and a printing artefact (U+05BC) — a dot in the middle of an otherwise open letter. A speck of foxing in the counter of a ב is the same object to the segmenter.

Assignment is a category of error on its own. The points sit under the letter, but Hebrew runs right to left while ink coordinates run left to right, so an engine that attaches each mark to the nearest base by horizontal centre will systematically shift marks by one letter wherever the type is tight or the scan is skewed. The output is well-formed Hebrew with every vowel one position out — which no spell-check catches and no confidence score flags.

Why you cannot restore them afterwards

The obvious workaround is to accept an unpointed transcription and add niqqud back with a diacritisation model. That is a real and useful tool for producing a reading edition, and it is not a substitute for OCR, because the mapping is genuinely many-to-one. The consonant skeleton דבר is davar (a thing), dever (a plague), diber (he spoke) and dibber (speech) depending only on the points. A restoration model picks the most probable reading in context. For a critical edition, a liturgical text or a manuscript study, the most probable reading is precisely the thing the document is evidence against.

So the decision to make early is what the transcription is for. If it feeds search or a summariser, strip the points deliberately and normalise, and the whole problem disappears — see how models handle unvocalised abjad text. If it is the artefact, the points have to come out of the image. The two paths also cost differently downstream, because each mark is its own code point and most tokenisers spend a token or more on it — Hebrew token costs covers what pointed text does to a bill.

A pipeline that keeps them

Scan or acquire at 600 dpi, greyscale, no compression. This is the single highest-leverage change and it costs nothing but disk.
Turn off despeckling and noise removal entirely. If the scan needs cleaning, clean it with a filter that preserves small components — not an area threshold.
Binarise adaptively (Sauvola or Niblack over a local window) rather than with a global cutoff, or feed greyscale directly if the recogniser accepts it.
Deskew before line segmentation, and give the line band generous padding below the baseline. Accept overlapping bands; resolve the overlap at assignment time rather than by cropping.
Use a model trained on pointed Hebrew, and check that its character set includes U+0591–U+05C7. A model whose label set stops at the consonants cannot fail visibly.
Normalise to NFC and check ordering. Hebrew combining marks carry distinct canonical combining classes, so canonical reordering does put dagesh, vowel and accent into a stable order — which means comparing your output against ground truth without normalising will report differences that are not errors.
Report a separate metric for marks. Count code points in the range U+0591–U+05C7 in the output and in the ground truth. If the ratio is near zero, no accuracy figure computed over the whole string is telling you anything.

Character error rate over the full string will look excellent on pointed material even when every point is missing, because the consonants outnumber the marks. Report mark recall separately or the aggregate will hide the failure completely.