An abjad writes consonants and leaves most vowels to the reader. That is not a deficiency of the script — it works because Semitic morphology puts the lexical meaning in the consonants — but it means an unvocalised written word frequently corresponds to several spoken words, and any system reading that word has to choose.
What an abjad leaves out
Arabic and Hebrew both work from a consonantal root, typically three consonants, carrying a broad semantic field. Vowel patterns applied to that root produce the specific words: agent, action, place, plural, passive, tense. In everyday writing the consonants are written and the vowel pattern is not.
Both scripts have full vowel notation available — Arabic harakat, Hebrew niqqud — and both restrict it to a specific set of contexts: sacred texts, poetry, children’s books, dictionaries, language teaching, and occasionally a single word in ordinary prose whose reading would otherwise be genuinely unclear. Newspapers, websites, contracts, chat messages and the overwhelming majority of the training corpus carry none. So a model trained on Arabic or Hebrew has learned from text where the vowels are absent, and asking it to produce vocalised output is asking it for something rare in its training distribution.
One root, several words
The Arabic root k-t-b concerns writing. Written unvocalised as three letters, the string supports at least these readings:
كتب (k-t-b) unvocalised
kataba he wrote (past, active)
kutiba it was written (past, passive)
kutub books (plural noun)
kutub write! (dialectal / regional imperative forms vary)
Hebrew behaves the same way. The root s-p-r concerns counting and recounting, and the unvocalised three-letter string supports:
ספר (s-p-r) unvocalised
sefer a book
sofer a scribe / writer
safar he counted
sapar a barber
sippur a story (with different vocalisation)
These are not obscure readings dug up to make a point — they are all common words. The reader disambiguates instantly from context and almost never notices doing it. The model does the same job, from the same information, with the same failure mode when context is thin.
One important qualification, because “abjads do not write vowels” is a useful simplification that is not quite true. Both scripts write long vowels using consonant letters pressed into service as vowel signs — the matres lectionis. Arabic uses alef, waw and ya for long a, u and i; Hebrew uses aleph, vav, yod and he the same way, and modern Israeli spelling (ktiv male) adds extra vav and yod precisely to reduce ambiguity in unvocalised text. So the ambiguity is partial rather than total: it is the short vowels, which carry most of the grammatical information, that go unwritten. That is why the surviving ambiguity falls so heavily on tense, voice and number rather than on which root is meant.
What decides which reading the model picks
Not a rule, and not a lookup. The model is producing a distribution over continuations, so the reading it commits to is whichever is most probable given everything else in the context. In practice three things dominate:
- Syntactic position. A word following a preposition is a noun; a word in a verbal slot is a verb. This resolves the noun-versus-verb ambiguity most of the time, and it is why longer context resolves better than shorter.
- Collocation. Adjacent words pull hard. The presence of a bookshop, a library or a title nearby makes “books” far more probable than “it was written”.
- Corpus frequency, when the first two are silent. With an isolated word — a form field, a search query, a table cell, a proper name — there is no syntax and no collocation, and the model falls back to whichever reading was commonest in training. That is the case where it will be confidently wrong, and it is exactly the case that short-input pipelines produce.
A fourth factor is worth naming for Arabic specifically: the same consonantal string can be Modern Standard Arabic or one of several regional varieties with different vocalisations and different meanings. Dialect is another axis the model has to guess along, which is why regional dialect confusion and vowel ambiguity so often show up as one bug rather than two.
The tasks this actually breaks
Generation is mostly fine, because the model produces unvocalised text and readers vocalise it as they always have. The failures cluster in tasks that need the vowels made explicit:
- Text to speech. A speech system must commit to a pronunciation. Arabic and Hebrew TTS therefore need an explicit diacritisation step, and its errors are audible: the wrong word, said confidently, in the middle of a fluent sentence.
- Transliteration and romanisation. Writing an Arabic or Hebrew name in Latin script requires the vowels. This is why one name yields many spellings, and why transliterating rather than translating names still leaves a one-to-many mapping.
- Exact matching and deduplication. Two records for the same person can differ by the presence of diacritics, by the choice of hamza carrier, or by alef variants. String equality says they are different people.
- Search. A user typing an unvocalised query against a vocalised index, or the reverse, matches nothing without normalisation on both sides.
- OCR. Recognising niqqud or harakat is a separate and harder recognition problem than recognising the consonants, covered on OCR of Hebrew with niqqud.
Handling it
The practical pattern is the same one that works for tone marks: keep the original as written, and derive a normalised form for comparison. For Arabic, a standard normalisation removes harakat and the tatweel elongation character, and folds the alef variants and the final ya and alef maqsura to canonical forms. For Hebrew, strip niqqud and cantillation marks. Index the normalised form, store and display the original.
The Arabic normalisation is worth spelling out because it is where the duplicate-record bugs actually come from, and it is not the same operation as stripping vowels. Alef appears as bare alef, alef with hamza above, alef with hamza below and alef with madda, and writers use them inconsistently in informal text; final ya and alef maqsura are routinely interchanged in Egyptian usage; and ta marbuta and final ha are confused the same way. A normaliser that folds those families and removes tatweel will collapse most of the spelling variation for the same word without touching the letters that distinguish different words. Apply it symmetrically to the query and the index, and never to the stored original.
Where you need the vowels — speech, transliteration, teaching material — treat diacritisation as an explicit step whose output is a guess, not a fact. Give the model the largest context you have, since syntax and collocation are the only signals available to it, and never write the diacritised form back over the source. And in any interface where a single word arrives with no surrounding text, expect the training-frequency default and design for a correction rather than for being right.
Top comments (0)