How We Cut Devanagari LLM Token Costs by 33.8% via Brahmi Token Injection

#ai #machinelearning #python #opensource

Introducing Bharat-Tiny-LLM v2: Cutting Devanagari LLM Token Costs by 33.8%

Multilingual transformer models built on English-dominant training datasets impose a severe "Token Tax" on non-Latin scripts. When processing Devanagari (Hindi) script, standard Byte-Pair Encoding (BPE) tokenizers break whole words into fragmented sub-byte sequences.

For developers deploying Indic LLMs in production, this token bloat causes:

3x Higher Memory Bandwidth Consumption
3x Slower Token-per-Second Throughput
Inflated Cloud API Costs
Premature Context Window Exhaustion

To solve this efficiency barrier, we engineered Brahmi Token Injection and released open weights for Bharat-Tiny-LLM v2.

Here is a look at the achievement, benchmarks, and how to run the open-weights model family locally.

The Breakthrough: 33.8% Token Compression

Consider the Hindi phrase:

"ज़रूरी बात है क्या करते हो"

On a standard 1.5B multilingual base model, encoding this prompt requires 26 tokens.

With Bharat-Tiny-LLM v2, the exact same phrase requires only 11 tokens — representing a 58% reduction in token count for conversational text. Across broad evaluation corpora, Bharat-Tiny-LLM v2 averages a 33.8% overall token reduction.

[Standard Tokenizer]   ज़रूरी बात है क्या करते हो  ➔ 26 Tokens 🔴
[Bharat-v2 Tokenizer]  ज़रूरी बात है क्या करते हो  ➔ 11 Tokens 🟢 (58% Savings)

Performance & Quality Benchmarks

By optimizing vocabulary efficiency, Bharat-Tiny-LLM v2 achieves significant throughput and quality gains while keeping the model size down to an ultra-compact 880 MB:

Benchmark Metric	Standard Base Model	Bharat-Tiny-LLM v2	Achievement Delta
Tokens per 1,000 Hindi Chars	~950 tokens	~630 tokens	33.8% Token Reduction
Inference Speed (Apple M-Series)	50 tok/s	68 tok/s	+36% Faster Throughput
Validation Loss (Hindi Corpus)	2.776	1.837	52.5% Loss Reduction (Perplexity: 16.1 ➔ 6.3)
Model Size (Q4 Quantized)	880 MB	880 MB	Runs in <3.8 GB Peak RAM

Prompt Compression Examples

Hindi Prompt	Standard Tokens	Bharat-v2 Tokens	Token Savings
`"ज़रूरी बात है क्या करते हो"`	26 tokens	11 tokens	58% Savings
`"नमस्ते, आप कैसे हैं?"`	15 tokens	7 tokens	53% Savings
`"भारत की राजधानी नई दिल्ली है"`	22 tokens	14 tokens	36% Savings
`"bhai aaj ka weather kaisa hai?"`	12 tokens	8 tokens	33% Savings

High-Level Innovation: Brahmi Token Injection

Standard tokenizers lack representation for high-frequency Indic subwords. Brahmi Token Injection solves this by:

Surgically Expanding Vocabulary: Ingesting script-specific subwords directly into the tokenizer dictionary (expanding vocabulary from 151,936 to 152,236 tokens).
Targeted Embedding & Attention Alignment: Aligning newly introduced token representations while fine-tuning attention projection layers to maintain fluency and semantic reasoning.

(Note: Detailed mathematical formulations, training logs, and ablation studies will be published in our upcoming research paper).

Quickstart: Run Open Weights Locally

Option A: PyTorch & Transformers (Linux / Windows / CUDA GPUs)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "eulogik/Bharat-Tiny-LLM-v2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="auto"
)

prompt = "भारत के प्रमुख त्यौहारों के बारे में लिखिए:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Option B: Apple Silicon MLX (Mac / iPhone / iPad)

from mlx_lm import load, generate

model, tokenizer = load("eulogik/Bharat-Tiny-LLM-v2-MLX")
response = generate(model, tokenizer, prompt="नमस्ते! आज का दिन कैसा है?", max_tokens=100)
print(response)