DEV Community

Jayesh Patel
Jayesh Patel

Posted on

Why “reversible transliteration” isn't always actually reversible

#unicode #i18n #opensource #sanskrit

I built an Indic transliteration library that can recover the exact original Unicode source

I've been working on lipimala, an open-source Indic transliteration library for:

  • IAST / extended Indic Latin
  • Devanagari
  • Gujarati
  • Plain English / Hunterian

with native implementations for:

  • Dart
  • JavaScript / Node.js
  • Python
  • PHP

Repository: https://github.com/jayeshmepani/indic-script-converter

The problem I wanted to solve

The interesting part is not simply converting:

Kṛṣṇa → कृष्ण
Enter fullscreen mode Exit fullscreen mode

There are already excellent projects for Indic transliteration. The problem I wanted to solve was different:

What happens when I need to convert the text back and recover exactly what the user originally supplied?

Canonical reverse is not exact reverse

Consider these three different source strings:

Kṛṣṇa
kṛṣṇa
Kr̥ṣṇa
Enter fullscreen mode Exit fullscreen mode

Different source strings can legitimately produce the same visible Devanagari representation:

कृष्ण
Enter fullscreen mode Exit fullscreen mode

Once that happens, the visible target string alone cannot tell us:

  • the original casing
  • whether the source was NFC or NFD
  • whether a precomposed or decomposed character was used
  • which accepted alias was entered
  • the original combining-mark ordering
  • exact whitespace or punctuation

A traditional reverse transliteration can return a canonical Latin representation. But canonical source is not necessarily exact original source. That distinction became the central design principle behind lipimala.

Two ways to preserve the original

lipimala provides two exact-source mechanisms.

1. TransliterationResult

Instead of treating transliteration only as string → string, the higher-level API returns a structured result containing concepts such as:

  • original
  • normalizedInput
  • rendered
  • profile
  • inputNormalization
  • outputNormalization
  • renderingIsInjective
  • issues
  • originalCodePoints

The original can then be recovered independently of how lossy the visible representation is.

Conceptually:

source
  │
  ├── original provenance
  │
  └── rendered representation
Enter fullscreen mode Exit fullscreen mode

This is useful for editors, databases, APIs, archival systems, and other applications where the rendered text and original source need to coexist.

2. Invisible exact-source metadata

Sometimes carrying a separate object isn't convenient.

lipimala can therefore optionally attach checksummed exact-source metadata using invisible Unicode Tag characters.

Conceptually:

Kṛṣṇa
   ↓
कृष्ण + invisible source metadata
   ↓
Kṛṣṇa
Enter fullscreen mode Exit fullscreen mode

The displayed Brahmic text remains normal while the original source can travel with the string.

This is intended for Unicode-preserving environments. For durable interchange, the structured result envelope remains the safer option because some systems may strip unusual Unicode characters.

Canonical, smart and exact reverse APIs

Another thing I wanted to make explicit was what "reverse" actually means.

lipimala distinguishes three modes:

Canonical

visible Devanagari
        ↓
canonical IAST
Enter fullscreen mode Exit fullscreen mode

Smart

exact metadata exists?
    │
    ├─ yes → restore exact source
    └─ no  → canonical reverse
Enter fullscreen mode Exit fullscreen mode

Exact

exact metadata required
        ↓
restore original or fail
Enter fullscreen mode Exit fullscreen mode

This avoids pretending that every visible transliteration mapping is mathematically one-to-one.

Direct Devanagari ↔ Gujarati

The library also contains a direct Devanagari/Gujarati converter.

It doesn't need to bounce through IAST:

Devanagari  ↔  Gujarati
Enter fullscreen mode Exit fullscreen mode

The converter handles:

  • nukta combinations
  • Unicode normalization
  • digits
  • unknown-character policy
  • whitespace policy
  • Vedic marks
  • optional source provenance

Where the two Unicode repertoires collapse distinctions, canonical conversion remains canonical and exact provenance is handled separately.

Vedic text

Vedic accents were another area where I didn't want them to be an afterthought.

The converter preserves supported encoded Vedic marks and handles their placement/storage ordering with the surrounding vowel signs, bindu/visarga, and svara marks.

There are dedicated Vedic fixtures in the shared verification suite.

Four native implementations

lipimala isn't a JavaScript library with thin wrappers around it.

The repository contains native implementations for:

  • Dart
  • JavaScript
  • Python
  • PHP

The goal is that all four implement the same transliteration contract.

The current shared verification material contains:

  • 497 transliteration cases
  • 22 Vedic fixtures

and the runtimes are tested against shared expected behavior/golden outputs.

That matters for applications such as:

Flutter client
      ↓
Node service
      ↓
Python processing
      ↓
PHP/Laravel backend
Enter fullscreen mode Exit fullscreen mode

where different parts of the same system should not transliterate the same source differently.

Installation

JavaScript 

npm install lipimala
Enter fullscreen mode Exit fullscreen mode

Python 

pip install lipimala
Enter fullscreen mode Exit fullscreen mode

Dart 

dart pub add lipimala
Enter fullscreen mode Exit fullscreen mode

PHP 

composer require jayeshmepani/lipimala
Enter fullscreen mode Exit fullscreen mode

A deliberately narrow scope

lipimala is not trying to compete with projects such as Aksharamukha or Sanscript on the number of scripts supported. Those projects are much broader.

The goal here is different: deterministic, Unicode-aware, source-preserving transliteration for a focused script set.

Today that focus is mainly:

  • IAST / extended Indic
  • Devanagari
  • Gujarati
  • Plain English / Hunterian

If your application needs 100+ scripts, another converter is probably a better choice.

If it needs to know exactly what source produced the rendered representation, that's the problem lipimala is designed around.

Links

I'd especially appreciate feedback from people working with Unicode, Sanskrit/Vedic texts, archival data, transliteration systems, or multi-runtime applications.

Top comments (0)