The problem
Usually transliteration libraries handle only transformation of non-Latin letters into Latin/English letters, e.g.:
Chinese / Japanese / Korean / Hindi / Arabic / Greek / Russian > Latin / English
This is only one side of transliteration which is called romanization. Libraries for the opposite direction are much harder to find, e.g. if you need to transliterate not to, but from a Latin-script language:
English / Spanish / Portuguese / German / French / Italian > Chinese / Japanese / Korean / Hindi / Arabic / Greek / Russian
Moreover the transliteration libraries often don’t handle contexts, and transform in a straightforward way letter by letter. The results can be hard to read, and can violate all possible language rules. However the idea of such Unicode-to-Latin libraries is to simply represent any Unicode non-Latin character with its closest Latin equivalent.
E.g. this is how the popular transliteration library handles Hokkaido written in Japanese Kanji:
北海道 > Bei Hai Dao
And this is how it handles Baghdad written in Arabic:
بغداد > bGdd
It does the job if you only need a rough Latin representation of any non-Latin character, e.g. for slugification purposes. This Frankenstein Latin outcome can be put in URLs where no human will ever read it.
We could also use Google Translate API, but there are some caveats:
- In some situations it might translate individual words instead of transliterating them
- In some situations it might transliterate incorrectly (e.g. it transforms the Maltese town of Marsaxlokk into Greek as Μαρσαξλόκ, whereas the correct form is Μαρσασλοκκ)
- We have to wait for the result from an external API each time
- It is not free
We could also use AI for this task, but there are caveats here too:
- It can hallucinate and produce different results for one and the same input text
- We have to wait for the result from an external API each time
- It is not free
What if you need a free self-hosted library that consistently produces predictable results? What if you need an accurate, human-readable transliterations? What if you need to handle not only romanization, but transliteration between a wide variety of writing systems and languages?
An accurate transliteration
More technically I’ll be talking about language-specific orthographic transcription. Pure transliteration usually describes the script conversion, but not necessarily the linguistic rules governing how the sounds/letters are represented. A useful conceptual hierarchy is:
- Script-to-script transliteration
- Language-specific transliteration
- Phonetic transcription: representing the pronunciation, e.g. with IPA
- Language-specific orthographic transcription: representing pronunciation of the source language using orthographic conventions of the target language
The latter is what I’m aiming for. My ultimate goal is to make a person speaking the target language to be able to correctly pronounce a word in the source language:
Source language > Target language
In practice, the boundaries between transliteration and transcription can get blurry, especially for proper nouns.
Although linguistically the term transcription is more correct here, nowadays it is mostly used when talking about transcribing audio or video into text. Therefore, to avoid confusion regarding the library's real functionality, from now on I will be using only the term transliteration.
Exonyms
The library does not handle translations. This includes exonyms – established traditional translations of foreign proper nouns (e.g. city names or person’s names).
E.g. the Greece’s capital Αθήνα has historically established English translation Athens, while the transliteration from Greek to English is Athina:
Αθήνα > English translation > Athens
Αθήνα > English transliteration > Athina
The first is how the city is known among English speakers. The second is how the city name is pronounced by Greek speakers, and how this pronunciation would be written in English.
Apart from such exceptions, the translations of proper nouns between different languages usually are transliterations rather than translations.
ICU
ICU (International Components for Unicode) is a mature, widely used open-source library providing Unicode and Globalization support for software applications, including text formatting, translation rules, and regional language features.
ICU’s Transforms is an exceptionally good tool for handling text transformations. It’s designed to handle case mapping, normalization, transliteration and bidirectional text converters using contexts and regex-style matching sets. It also conveniently has built-in sets of rules for transforming from one script to another – although this is not what I’m using due to their extremely rough transformation rules with lack of context.
ICU has implementations in C/C++ and Java. Since I’m building a JS library, I created my own JS implementation. There is no need to re-create the whole ICU’s Transforms functionality, therefore I created only the specific part I’m using – the Rule-Based Transliterator (RBT).
AI
When it comes to writing rules for transliterating between many languages, the most convenient part here is that I can use AI for this purpose. Unfortunately, the drawbacks here are AI hallucinations (even when using heavier/smarter/pro models) and AI confusions regarding how ICU actually works (e.g. it can confuse } and { when handling contexts; or it can skip ::Null when there is a need for a new pass).
Therefore human review of all AI suggestions is a must – even if I don’t speak the languages from a given language pair, I can still check the generated rules according to ICU principles.
To confirm the generated rules I always check the rules by using set of examples (which consist of 20 names of cities + 20 full names of people + 20 names of companies, which I have for each supported language). I apply the generated rules to these examples, and then ask AI to evaluate the outcome.
This is the process I’m following to generate valid set of ICU rules:
- Initial generation of ICU rules.
- Manual review of the rules.
- Application of the rules to my 60 examples.
- Evaluation of the results (0-100).
- Generation of fixes for the rules (if needed).
- Manual review of the fixes.
- Application of the fixes.
- Repeat steps 3-7 until the evaluation hits 100. If it takes more than 20-30 cycles, then any evaluation above 95 is considered sufficient.
IPA
Creating direct transliteration rules between 2 languages is justified when these languages are historically and/or linguistically bound (e.g. Cyrillic languages, Asian languages, Semitic languages, Indo-Aryan languages, etc.). But creating direct transliterators between all 50 languages would be cumbersome, and a better general approach is needed.
Languages fall into two main categories when borrowing proper nouns: orthographic (spelling-based) and phonetic (sound-based).
Scripts that do not share a common alphabetic ancestor with Latin or Cyrillic almost exclusively borrow foreign names based on how they sound, entirely ignoring how they are spelled. Such languages include Chinese, Japanese, Korean, Hindi, Bengali, Urdu, Persian, Arabic and Hebrew. For these languages I’m using IPA (International Phonetic Alphabet) as an intermediary. Therefore I created rules for transforming almost all supported languages to IPA (I skipped a few languages – English, Chinese, Japanese, Korean – and used external libraries instead). The transliteration pipeline for these languages now looks like this:
Any language > IPA > Chinese / Japanese / Korean / Hindi / Bengali / Urdu / Persian / Arabic / Hebrew
There is another case where IPA should be used when transliterating from certain Latin-script languages like English or French. These languages suffer from one problem: they have deep orthographies (heavy use of silent letters, historical spellings, and massive vowel reductions), or they use the Latin alphabet in ways that completely violate standard European phonetic norms. So the transliteration pipeline for these languages looks like this:
Deep Latin > IPA > Any language
Other use cases where IPA is used as an intermediary:
Urdu / Persian / Arabic / Hebrew / Armenian / Georgian > IPA > Any language
Chinese / Japanese / Korean > IPA > Armenian / Georgian
Hindi / Bengali > IPA > Armenian / Georgian / Cyrillic / Shallow Latin
Standardized Latin
While mapping sounds to IPA and then mapping IPA to target language does work for the above language pairs, for the rest of them I’m using Standardized Latin as the pivot script:
Chinese / Japanese / Korean / Greek / Cyrillic / Armenian / Georgian / Shallow Latin > Standardized Latin > Any Latin
Hindi / Bengali > Standardized Latin > Deep Latin
Greek / Armenian / Georgian / Shallow Latin > Standardized Latin > Cyrillic
Any language > Standardized Latin > Greek
Greek / Cyrillic / Shallow Latin > Standardized Latin > Armenian / Georgian
Standardized Latin hub is entirely exhaustive. It accounts for every unique phonetic consonant, digraph, and vowel variation produced by the source languages so that the final target rules can map them flawlessly.
For the cases when the text is already in Latin (like Polish or Italian – shallow orthographies), the Standardized Latin works as a “normalizer” which only targets the language-specific exceptions, e.g.:
- convert Italian ci to č
- convert Polish c to ts
- convert Turkish c to dž
Romanization
I also generated dedicated sets of rules for transliterating to English since it’s the international lingua franca. It’s also widely used as a backup language in many systems.
Transliteration of non-Latin-script languages to English usually has multiple systems in every language. The systems I’m using for each non-Latin script language are:
- Chinese: Pinyin
- Japanese: Hepburn romanization system
- Korean: Revised Romanization of Korean (RR)
- Hindi/Devanagari, Bengali, Urdu: Hunterian transliteration system
- Greek: ELOT 743, 2nd ed.
- Armenian, Georgian: BGN/PCGN romanization system
- Russian, Kazakh: Wikipedia-style romanization
- Ukrainian: Ukrainian National (2010) transliteration system
- Bulgarian: Streamlined System with -ia-exception (2009 Transliteration Act)
- Macedonian: Digraph system ICAO Doc 9303
For Romanization of the languages where letters represent only consonants (abjad languages – Arabic, Persian and Hebrew) I’m using IPA:
Arabic/Persian/Hebrew > IPA > English
Cyrillization
Cyrillization faces a major linguistic roadblock: the Cyrillic alphabet is highly fragmented. Different countries use entirely different letters to represent the exact same sounds. Because of this, a word cyrillized for a Russian reader will look broken, misspelled, or be completely mispronounced by a Bulgarian or Macedonian reader.
There are various Cyrillic languages: some of them are highly used, some of them are barely used, some of them have Latin equivalent. For now I decided to support the following Cyrillic languages:
- Russian. Covers the massive East Slavic demographic. Will naturally serve as a fallback for Belarusian and Central Asian Cyrillic readers.
- Kazakh. Kazakhstan has been undergoing a state-mandated transition from Cyrillic to the Latin alphabet. However the timeline has faced delays, and Cyrillic remains in use.
- Ukrainian. Has distinct vowels (І, Ї, Є) and consonants (Ґ) that other Cyrillic languages lack.
- Bulgarian. Represents the South Slavic standard. It ensures the crucial Ъ vowel is used properly.
- Macedonian. Integrates Serbian-style single letters (Ј, Љ, Њ, Џ) but remains structurally unique (using Ѓ and Ќ).
Regarding other prominent Cyrillic languages:
- Serbian. Only Latin script is supported. Serbians are fluent in both Cyrillic and Latin scripts. In fact, for digital interfaces, web browsing, and casual typing, the Latin script (Gaj’s Latin alphabet) is often the preferred choice. Also, Serbian Latin can always be converted to Cyrillic and vice versa using this library.
- Belarusian. While it has its own distinct Cyrillic orthography (featuring the unique Ў / short U), the population is overwhelmingly bilingual, and Russian is heavily dominant in Belarus.
When transliterating to Cyrillic, I’m using IPA or Standardized Latin as a pivot for most of the languages. However Chinese, Japanese and Korean demand individual approach because they have established transcription systems:
- from Chinese (Pinyin):
- to Bulgarian: official Bulgarian transcription system for Chinese (established by Sofia University’s Sinology Department in the 1990s)
- to Macedonian: adapted version of the Serbian transcription system
- to Russian: Palladius System
- to Kazakh: adapted version of Palladius System
to Ukrainian: Academic System (Ukrainianized Palladius)
from Japanese (Rōmaji – Hepburn):
to Bulgarian: official Bulgarian transcription system (established by Sofia University’s Japanese Studies Department)
to Macedonian: adapted version of the Serbian transcription system
to Russian: Polivanov System
to Kazakh: adapted version of Polivanov System
to Ukrainian: Bondarenko System
from Korean (Romaja – Revised Romanization):
to Bulgarian: official Bulgarian transcription system (established by Sofia University’s Korean Studies Department)
to Macedonian: adapted version of the Serbian transcription system
to Russian: Kontsevich system
to Kazakh: adapted version of Kontsevich system
to Ukrainian: Ukrainianized Kontsevich System (refined by scholars at Taras Shevchenko National University and KNLU)
Forbidden letter combinations
Once all transformations are ready, the output can contain unusual letter combinations which would look weird to a native speaker. Almost each language has established rules for specific forbidden letter combinations and how they should be handled. Sometimes these rules are applied exclusively to transliterated foreign proper nouns.
In order to comply with language norms I generated fixing forbidden letter combinations rules for the supported languages (where applicable).
Currently the library supports 50 languages – they result in 2450 language combinations. You can play around with it in your browser – the library is compatible with Chromium and WebKit, although it’s primarily designed for Node.js usage.
Top comments (2)
The distinction between script conversion and language-aware transcription is crucial for users. I’d expose it in the API as an explicit mode with examples and confidence or fallback metadata, so callers can choose predictable slug behavior without mistaking it for a pronunciation-safe result.
Good point Alex! I'll update the readme. Thanks!