How We Built a Sub-200ms Multilingual Chat System Translating 100+ Languages with Our Own LLM

iRoom — Sun, 10 May 2026 12:15:00 +0000

How We Built a Sub-200ms Multilingual Chat System Translating 100+ Languages with Our Own LLM

A guest from Tokyo checks into a hotel in Istanbul. They want to ask about breakfast. The receptionist speaks Turkish and English. The guest writes in Japanese.

For decades this meant confused gestures, dictionary apps, and often guests giving up entirely.

We spent two years building the infrastructure to remove this friction completely. The result runs in production across 700+ hotels in 50+ countries, translating live conversations between guests and staff with median latency under 200 milliseconds. This is how the system works under the hood.

Why Generic Translation APIs Failed for Us

We started where most teams would: piping messages through commercial translation APIs. Within three months in production we hit three walls that pushed us to build our own model.

The first wall was tone. Hospitality is a register-sensitive domain. A polite Japanese request like 「お湯をいただけますか」 came out as flat imperative English. The reverse was worse — neutral English requests landed in Japanese, Korean, and German as informal or even rude. Asian languages with explicit honorific systems and Slavic languages with ty/vy distinctions were consistently mistranslated. Guest satisfaction scores in those languages were measurably lower.

The second wall was domain vocabulary. Hospitality has its own lexicon. Concierge, do not disturb, continental breakfast, valet parking, early check-in, late checkout, no-show, compendium, turn-down service — these have established equivalents across hospitality but generic models translate them literally. A Russian guest asking about "услуга вечерней подготовки номера" should translate to turn-down service. Generic APIs returned things like "evening room preparation service".

The third wall was operational. Per-character pricing scaled unpredictably as our network grew. Upstream rate limits caused tail latency spikes during peak hours when hotels needed reliability most. We had no control over model updates that could change behavior overnight, sometimes breaking carefully tuned hospitality phrasings without notice.

By month four we had decided to build our own. By month eight we had a working prototype. Today, two years in, iRoom LLM handles every message that flows through our network.

iRoom LLM: A Two-Year Engineering Investment

iRoom LLM is a transformer-based translation and reasoning model fine-tuned specifically on hospitality conversational data. It runs entirely on our own infrastructure across multi-region GPU clusters, with no dependency on third-party inference APIs.

The training corpus took eighteen months to assemble. We built it from four sources:

A base of multilingual web text filtered for hospitality and travel domain content
A curated dataset of professionally-translated hotel collateral across 47 languages — welcome letters, in-room directories, service menus, signage, FAQ pages from luxury hotel chains
Real conversational chat data from our own platform, with guest consent and full anonymization
A synthetic dataset of hospitality dialogue scenarios we generated and then had native-speaker reviewers correct for naturalness in 30+ target languages

Training was iterative. We started from a 7B-parameter open-weight base, ran continued pretraining on our hospitality corpus, then progressively fine-tuned through supervised instruction-tuning and reinforcement learning from human feedback using hotel staff and native speakers as raters. The model was evaluated weekly against a held-out benchmark of 12,000 real guest-staff exchanges across the 25 most common language pairs in our network, scored on three axes: semantic accuracy, register preservation, and domain vocabulary fidelity.

The result is a model that consistently outperforms commercial translation APIs on hospitality benchmarks while running 4-6× faster on our own hardware because we control the inference stack end-to-end.

The Production Architecture

The full message path:

When a guest scans the QR code in their room, the frontend opens a Progressive Web App. Before the chat interface even renders, three things happen in parallel: the browser language is captured, IP-based country and likely-language are inferred from a cached GeoIP database we maintain in-edge, and the room's stored guest profile is queried for any language preference set during check-in. These three signals collapse into a single confirmed guest_locale that gets locked in for the session.

When the guest types a message, the client sends the raw UTF-8 text plus the guest_locale and a session token to our edge nodes over a persistent WebSocket. We never trust client-side language detection — clients can be wrong, and WebViews on certain Android devices misreport language entirely.

At the edge, the message hits a router that does three things atomically:

Writes the raw message to our primary store with the original language tagged
Publishes the message ID to a translation queue keyed by (target_locale_set)
Acknowledges the client

Acknowledgement happens within 20-40ms on the p50 because the original-language storage write is the only blocking operation.

The translation queue is consumed by inference workers running iRoom LLM. Each worker processes messages with the following pipeline:

Cache lookup against normalized message representation
→ Unicode NFC, casefold, punctuation strip, whitespace collapse
→ Named entities (guest names, room numbers) replaced with typed placeholders
→ Hit: return cached translation in <5ms
→ Miss: continue
Prompt construction
→ Hospitality system instruction
→ Conversation context window (last 3-5 messages)
→ Source language + target language tags
→ Source message
iRoom LLM inference
→ p50: 95ms warm GPU
→ p95: 240ms
Result storage
→ Cache by normalized form
→ Link to original message ID
→ Emit on WebSocket bus

When staff reads a message, the frontend requests rendering in their staff_locale. The backend returns a payload with both the original and the translated text. Staff sees the translated version by default, with a tap or hover to reveal the original. Staff replies follow the same pipeline reversed.

The Five Engineering Decisions That Made This Work

1. Original-as-source-of-truth, translation-as-derived-data

Every message is stored exactly as the sender wrote it, in the source language, with the language explicitly tagged. Translations are derivative artifacts linked by message ID. We never overwrite a stored message with its translation.

This sounds trivial but pays dividends repeatedly. When iRoom LLM was retrained at version 1.4 and again at 2.0, we re-translated portions of historical conversations to backfill improved quality without touching the canonical message store. When a hotel reported a translation error six months after the fact, we could trace the exact model version that produced it and deploy a corrected mapping. When a property requested an export of all guest conversations for legal compliance, we produced it in source-of-truth form, no language ambiguity.

2. Pivot through English, but smarter

We pivot non-English ↔ non-English translations through English. Naive direct translation between every pair would require N×(N-1) optimized model paths. With pivoting, we maintain N high-quality paths to and from English.

The naive concern is that pivoting compounds errors: Turkish → English → Korean should be worse than direct Turkish → Korean. In practice we found the opposite for most pairs because iRoom LLM's training data is much denser for non-English ↔ English pairs than for non-English ↔ non-English pairs. The English representation acts as a high-quality semantic intermediate.

We do bypass the pivot for a small set of high-volume pairs where we have enough native data to make direct translation reliably better — Japanese ↔ Korean, Spanish ↔ Portuguese, Russian ↔ Ukrainian, Arabic ↔ Farsi. The bypass list is data-driven and adjusted quarterly based on benchmark performance.

3. Aggressive normalized caching

Hospitality conversation is more repetitive than developers expect. "What time is breakfast?" "Is the gym open?" "Where is the pool?" "Can I get extra towels?" These exact phrases — across thousands of variant spellings, capitalizations, and punctuations — flow through the network millions of times per month.

Our cache layer stores translations keyed by (normalized_text, source_lang, target_lang). Normalization runs as a deterministic function: Unicode NFC normalization, casefold, punctuation stripping, whitespace collapse, and named-entity substitution. "Hi!" "hi" "HI!!!" all hit the same entry. "Can I get an extra towel for room 412?" and "Can I get an extra towel for room 815?" both normalize to "can i get an extra towel for room ROOM_NUM" and share a cache slot.

Production hit rate is 47% network-wide, with translation cost cut roughly in half and p50 latency on cache hits at 4ms. The cache is sharded across regions, eventually consistent, with TTLs tuned per language pair based on observed translation drift between iRoom LLM versions.

4. Lazy fan-out, never eager translation

A conversation might have one guest writing in Japanese and three staff members watching in Turkish, English, and Russian. Naive eager translation generates three translations per guest message. Lazy translation generates one per staff member, only when they actually read.

Most staff don't read most messages in real-time — they read the queue when they sit down at the front desk between guest interactions. Lazy translation cut our compute budget by approximately 70% versus eager generation, with no perceptible impact on perceived latency because translation happens during the read request, which already has expected network latency anyway.

Implementation-wise, the message bus emits a message-arrived event that subscribed staff clients receive as a "new message" notification. The client requests the translated rendering only when the staff member focuses the conversation. Cache hit rate on these requests is even higher than on guest writes because the same message is often read by multiple staff members in different languages — second and third reads are nearly always cached.

5. Domain context as system prompt, conversation history as context window

iRoom LLM receives more than just the message to translate. Each inference call constructs a prompt containing:

The hospitality system instruction (preserve register, use formal honorifics where appropriate, recognize hospitality terminology, avoid literal translation of cultural idioms)
The last 3-5 messages of conversation context
Source language and target language tags
The message itself

The conversation context window is the difference between coherent threading and isolated mistranslation. "Yes, please" translated standalone is ambiguous in many languages. With the previous message visible ("Would you like extra pillows?"), the translation becomes correct in every target language we support.

We tuned the context window length empirically. Three messages is the sweet spot — longer windows added latency and noise from older off-topic exchanges; shorter windows lost critical referent information. The window is also right-truncated when the conversation exceeds 5 turns, retaining only the most recent turns.

Production Numbers

Two years in, here is what the system delivers in steady state across the network:

Metric	Value
End-to-end p50 latency (send → translated display)	180ms
End-to-end p95 latency	480ms
End-to-end p99 latency	950ms
iRoom LLM inference p50 (warm GPU)	95ms
iRoom LLM inference p95	240ms
Cache hit rate (network-wide)	47%
Cache hit rate (high-volume properties)	70%+
Translation cost per active room per month	~$0.04
Hospitality benchmark score (BLEU-4 + COMET + register)	0.91
System availability (12-month rolling)	99.97%

For comparison, commercial translation APIs we benchmarked against scored 0.74-0.81 on the same hospitality benchmark.

What We Got Wrong on the Way Here

Plenty.

Our first translation cache implementation used unnormalized message text as the key. Hit rate was 8%. Adding deterministic normalization and named-entity substitution lifted it to 47%. We could have done this in week one but didn't think repetition would matter that much. It mattered enormously.

We initially built without a conversation context window. iRoom LLM was translating each message in isolation. Quality on multi-turn dialogues was significantly worse than on standalone messages, and we didn't catch it for months because our automated benchmarks tested isolated message pairs. Adding a 3-message context window fixed it; building the benchmark for multi-turn dialogues should have come first.

We over-engineered the streaming inference path early on. Token-by-token streaming of translations to the client felt like a good idea — staff would see translation appear as it generated. In practice, hotel staff prefer to see the complete translated message at once because partial messages look broken on their dashboards. We removed streaming and saved the implementation complexity.

Our first multi-region deployment routed all inference to a single primary region. Tail latency for Asia-Pacific hotels was awful. We migrated to region-local inference clusters with model weight replication, which cut p99 latency in half and increased cost only marginally.

Where This Runs

This entire system powers the chat layer of iRoom, a digital guest service platform used by 700+ hotels worldwide. The chat is one feature among many — there's a QR-based service ordering system, a real-time room status dashboard, a multi-channel notification fanout to web, desktop, and Telegram, an AI receptionist that handles routine questions before they reach human staff — but the multilingual chat is the piece that took the most engineering investment and is the one that most clearly removes a problem hotels could not solve before.

If you're building anything that involves real-time cross-lingual communication at scale, hopefully some of these decisions save you time. We learned most of them by getting them wrong first.

Happy to discuss any of these tradeoffs in the comments.

DEV Community: iRoom

How We Built a Sub-200ms Multilingual Chat System Translating 100+ Languages with Our Own LLM

How We Built a Sub-200ms Multilingual Chat System Translating 100+ Languages with Our Own LLM

Why Generic Translation APIs Failed for Us

iRoom LLM: A Two-Year Engineering Investment

The Production Architecture

The Five Engineering Decisions That Made This Work

1. Original-as-source-of-truth, translation-as-derived-data

2. Pivot through English, but smarter

3. Aggressive normalized caching

4. Lazy fan-out, never eager translation

5. Domain context as system prompt, conversation history as context window

Production Numbers

What We Got Wrong on the Way Here

Where This Runs