OCR for Tamil and Malayalam Vowel Sign Placement

#ai #machinelearning #llm #programming

Tamil and Malayalam OCR usually gets the consonants right. The errors concentrate almost entirely in the vowel signs, and they concentrate there for three unrelated reasons that produce identical-looking damage: a mark in the wrong order, a mark reassembled from the wrong pieces, and a mark that was never a separate piece to begin with.

An abugida, and what that implies

Both scripts are abugidas. A consonant letter carries an inherent “a” vowel; any other vowel is written as a dependent sign attached to that consonant. Tamil occupies U+0B80–U+0BFF and Malayalam U+0D00–U+0D7F. In both, the string order is fixed and logical: consonant first, then the vowel sign, regardless of where the sign is drawn.

That last clause is the source of most of the trouble. In an alphabet, the order you read the ink in is the order the characters are stored in. In these scripts it is not, and an OCR engine that emits characters in the order it encountered them on the page produces strings that look correct when rendered by a lenient font stack and compare unequal to the ground truth, fail to match a search query, and embed to a different vector.

Five positions for one class of mark

Which side a vowel sign attaches to depends on the vowel, not on the consonant, and every position is populated:

To the right. Tamil U+0BBE (aa) and U+0BCB’s right component; Malayalam U+0D3E. The straightforward case.
Above. Tamil U+0BBF and U+0BC0 (i, ii) hook over the top right of the consonant.
Below. Tamil U+0BC1 and U+0BC2 (u, uu); Malayalam U+0D41–U+0D43. These are the ones that ligate, discussed below.
To the left. Tamil U+0BC6, U+0BC7, U+0BC8 (e, ee, ai) and Malayalam U+0D46, U+0D47, U+0D48 render before the consonant they follow in the string. This is the classic reordering case.
Both sides at once. Tamil U+0BCA, U+0BCB, U+0BCC and Malayalam U+0D4A, U+0D4B, U+0D4C are two-part signs: one code point that renders as ink on the left of the consonant and ink on the right.

The two-part signs are the hardest case in the pipeline. A connected-component segmenter sees three separate ink regions in a row — left component, consonant, right component — and has no way to know from geometry alone that the outer two are halves of a single character. The reassembly rule is script-specific and has to be encoded explicitly: a left component of a known set, followed by a consonant, followed by a right component of the matching set, collapses to one code point placed after the consonant.

Visual order is not storage order

A recogniser that walks the line left to right and appends what it sees produces, for the Tamil syllable கெ, the sequence sign-then-consonant. Rendered, that often still looks like கெ, because the shaping engine reorders anyway. It is a different string.

The consequences are all silent. Exact-match search misses. A diff against ground truth reports errors your eyes cannot see in the rendered output, so a reviewer signs off on a transcription that is wrong. Embeddings differ because the tokeniser sees different byte sequences, which is the same class of failure described in splitting Devanagari text for embedding. And Unicode normalisation does not fix it: NFC reorders combining marks by canonical combining class, and these vowel signs carry class 0, so the canonical algorithm leaves them exactly where you put them.

The check is cheap and worth adding to any Indic OCR pipeline: assert that no output string begins a grapheme cluster with a dependent vowel sign, and that no dependent vowel sign is preceded by another dependent vowel sign. Both conditions are illegal in well-formed text and both are exactly what visual-order emission produces.

When the vowel changes the consonant

The below-attaching u and uu vowels are a different problem again. In Tamil, க + u is not a க with a mark under it — the pair fuses into a single connected glyph, and the fusion differs per consonant, so கு, சு, டு and து are each a distinct shape rather than a common base with a common decoration. There is nothing to segment.

This forces the recogniser to learn consonant-vowel combinations as whole classes, which multiplies the label set: eighteen Tamil consonants times twelve vowels, before the Grantha letters used for Sanskrit loans and the numerals. Malayalam is considerably worse, because traditional orthography ligates far more and adds conjunct consonant clusters on top — the same combinatorial explosion described for Devanagari ligatures, with more of it. It is also where a genuine historical split appears: the script reform promoted in Kerala from 1971 detaches many of these signs into separate marks, so mid-century and modern printing use visually different forms of the same characters. A model trained on one performs poorly on the other, and the corpus you are digitising may contain both.

Malayalam adds one more encoding trap. The chillu letters — U+0D7A to U+0D7F, added in Unicode 5.1 — represent consonants in their pure, vowel-less form, and the same reader-visible text can also be encoded as the base consonant plus virama U+0D4D. Two encodings, one appearance. Ground truth and OCR output can disagree on every occurrence without a single visible difference, and your error rate will report it as a substitution. Decide which representation is canonical for your corpus and normalise both sides before comparing.

Measuring the thing that is actually broken

Normalise both output and ground truth to NFC, then apply your own chillu and split-vowel canonicalisation, before computing any metric.
Report grapheme-cluster error rate alongside character error rate. One wrong vowel sign breaks one cluster a reader perceives, but may be one code point out of four — the code-point figure understates it.
Compute a vowel-sign-only error rate: strip all dependent vowel signs from both strings, measure consonant accuracy, then measure the signs in isolation. If consonants are at 98% and signs at 70%, you know immediately that image quality is not your problem.
Add the well-formedness assertions above as a hard gate in the pipeline, not a warning. A reordering bug that reaches the index is far more expensive than one that fails the build.
Split evaluation by orthography era for Malayalam. An aggregate over pre- and post-reform material hides which one the model handles.

Confusable pairs also cluster in the signs rather than the consonants: Tamil U+0BBE (the aa sign) against the letter ர, and the i and ii signs against each other, differ by a stroke length that survives neither downsampling nor thin paper. Expect them in the error table and treat their absence as a sign your test set is too clean.