Transliterating Hindi Text From Devanagari to Latin Script

#ai #machinelearning #tutorial #llm

राम is Rāma and it is Ram. The first is what the letters say, the second is what the word sounds like in Hindi, and the gap between them is a single grammatical feature that decides which transliteration system you should be asking for.

Why Rama and Ram are both correct

In Devanagari, a consonant letter is not a bare consonant. It carries an inherent vowel — the schwa, a — unless something on the page cancels it. So र म is literally ra + ma, and a strict letter-by-letter transliteration gives Rāma. In Sanskrit that final vowel is pronounced and the transliteration is also the pronunciation.

Hindi deletes it. The rule is called schwa deletion, it applies at the end of a word and in some medial positions, and it is not marked in the script at all — a reader knows the word and therefore knows which schwas are silent. So the same three letters that a Sanskritist writes Rāma, a Hindi speaker says Rām.

Medially it is worse, because the deletion is not simply word-final:

Devanagari   letter-by-letter   spoken Hindi
भारत          bhārata            Bhārat
सरकार         sarakāra           sarkār
धड़कन          dhaṛakana          dhaṛkan
नमस्ते         namaste            namaste   (no deletion here)

There is no way to derive the second column from the first without knowing the word. That is the whole reason this is a task worth giving to a language model rather than to a character-mapping library: the library can only produce column two, and column three is what people expect to read. It is also why two runs can differ on an unusual name — the model is recalling a pronunciation, not applying a table.

The systems you are choosing between

IAST (International Alphabet of Sanskrit Transliteration) is the academic standard for Sanskrit. Fully reversible, diacritic-heavy, script-faithful: it keeps every inherent vowel. कृष्ण is kṛṣṇa.
ISO 15919 extends the same idea across all Indic scripts so that Hindi, Bengali, Tamil and Gurmukhi romanise into one comparable system. It differs from IAST in small, visible ways: the anusvara is ṁ rather than ṃ, vocalic ऋ is r̥ rather than ṛ, and it distinguishes long ē and ō because some Indic languages need the contrast.
Hunterian is the Government of India’s official system, used by the Survey of India for place names and reflected in official documents. It is diacritic-light and pronunciation-oriented, which is why maps say Varanasi and not Vārāṇasī.
ITRANS and Harvard-Kyoto are ASCII-only reversible schemes that use capitalisation and punctuation to carry the diacritics — kRSNa in Harvard-Kyoto. Built for typing Sanskrit on a keyboard that has no diacritics, and still common in older corpora.
Ordinary Hinglish — Krishna, Sharma, Vishwanath — has no standards body, drops every diacritic, applies schwa deletion, and is what appears on every form your users fill in.

For names in a product database the answer is nearly always Hinglish, with the Devanagari kept as the canonical field. For a scholarly corpus, ISO 15919. Mixing them within one column is the failure mode, and it happens because the request said “transliterate” without saying which.

The letters that collapse without diacritics

Hinglish is cheap to read and expensive in information. These are the distinctions it loses, and knowing them tells you what you can and cannot recover later.

Retroflex against dental. ट ठ ड ढ ण are made with the tongue curled back; त थ द ध न are dental. IAST writes ṭ ṭh ḍ ḍh ṇ against t th d dh n. Hinglish writes t and d for all of them, so Tandon could begin with either. This is the single largest source of ambiguity in romanised Indian names.
Aspiration, and the digraph it creates. ख is kh, थ is th, and now th in a Hinglish string is either the single letter थ or the cluster त् + ह. Thakur (ठाकुर) and a hypothetical त्हाकुर are indistinguishable in ASCII.
Long against short vowels. इ and ई, उ and ऊ, अ and आ. IAST marks them with a macron; Hinglish sometimes doubles the letter (Aarti for आरती) and sometimes does not (Arti), which is why the same name arrives spelled two ways in one dataset.
The anusvara. The dot ं marks a nasal whose actual consonant is determined by what follows. हिंदी is hiṁdī in ISO 15919 and Hindi in practice; the model has to resolve the nasal to n, m or ng from context.
Nukta letters. The subscript dot creates क़ q, ख़ x, ग़ ġ, ज़ z, फ़ f, ड़ ṛ. These matter for Urdu-origin vocabulary and names, and the nukta is frequently omitted in typed Hindi even when the word requires it — so ज and ज़ arrive as the same code point sometimes and not others.

A prompt for a batch of names

Batch consistency comes from stating the system, the schwa rule and the diacritic policy separately, because they are three independent decisions that a single word like “transliterate” leaves open.

SYSTEM
You transliterate Hindi personal names written in Devanagari into
Latin script for a customer database.

Target convention: common Indian English spelling (Hunterian-style).
- Apply Hindi schwa deletion. Write Ram, not Rama. Bharat, not Bharata.
- Use no diacritics. ASCII letters only.
- Retroflex and dental consonants both map to t / d / n.
- Aspirated consonants keep the h: kh, gh, chh, jh, th, dh, ph, bh.
- Resolve the anusvara to n, m or ng by the following consonant.
- Nukta letters map to q, kh, g, z, f and r respectively.
- Capitalise the first letter of each name part only.
- Do not translate the meaning of a name. Transliterate it.

Also return the ISO 15919 form in a second field, with full diacritics.

Output JSON: [ { "devanagari": "...", "latin": "...", "iso15919": "..." } ]

Run ten names you can verify, including at least one ending in a consonant cluster and one containing a nukta letter.
Check that the latin field is pure ASCII with a regular expression. A single stray ā means the diacritic instruction was partially ignored and the rest of the batch is suspect.
Check that stripping the diacritics from iso15919 produces something close to latin — not identical, because schwa deletion means it will not be, but with the same consonant skeleton. A different skeleton means the model read the name two ways.
Store the Devanagari. It is the only lossless field, and it is what you regenerate from when you later decide the convention was wrong.
Run the batch in chunks with the same system prompt on each, rather than one long conversation. Later chunks in a long context drift back towards the model’s default.

Validating a batch you cannot read

Most of the useful checks do not need Hindi. Assert the character class of each field. Assert that the number of output rows equals the number of input rows — a batch transliteration that silently merges two rows is common and catastrophic. Group by the ISO 15919 field and look for groups where the latin field varies: that is the model spelling one name two ways, and it is exactly what you are trying to prevent.

Round-tripping is a weaker test than it looks. Asking a model to convert the Latin back to Devanagari and comparing will produce mismatches on every name where schwa deletion or a dropped retroflex removed information, and those mismatches are expected rather than wrong. The check is only informative in the ISO 15919 column, which is reversible by design.

If the corpus is Sanskrit rather than Hindi, the schwa instruction above is wrong and will corrupt it — Sanskrit does not delete the inherent vowel. Sanskrit transliteration and its diacritics is a separate problem with a separate prompt.

Two adjacent issues are worth knowing about before you build on this: a Devanagari grapheme can span several code points, so splitting Devanagari text at the wrong boundary breaks clusters that must stay whole, and the same clusters are the hard part for recognising Devanagari ligatures from a scan.