How We Built Instant Translation Help (即时翻译帮助) with Python and LLMs

Balancing speed and context with a hybrid glossary + LLM caching system

The Need for Instant Translation Help

At LectuLibre, we use LLMs to translate entire books into different languages. But even after a high-quality translation, readers sometimes stumble upon an unfamiliar word or want to see the original phrase for clarity. We envisioned a feature we called 即时翻译帮助 (instant translation help): clicking on any word in the translated text would immediately show a contextual explanation or alternative translation, right inside the reading interface.

The core requirement was speed. The popup had to feel instantaneous—under 500ms. A spinning wheel disrupts the reading flow. However, making a full LLM call for every click was out of the question: latency ranged from 2–5 seconds, and at scale it would be expensive.

Breaking Down the Problem

We needed:

• Low-latency responses for common words
• Context awareness (the same word can mean different things in different sentences)
• Coverage: handle any word or short phrase the user might click
• Cost efficiency: minimize LLM calls

Our first prototype simply sent the selected word and its surrounding sentence to Claude, but the 3‑second average wait was unacceptable. We had to get creative.

The Hybrid Approach: Glossary + LLM Fallback

We decided to preprocess each book after the main translation to build a bilingual glossary of key terms. At query time, we would first check this local glossary for a match. If found, we could return the result instantly. If not, we would fall back to a faster LLM (DeepSeek) and cache the response for future lookups.

In essence, the glossary acts as a permanent, read‑optimised cache, while the LLM covers the long tail and handles rare words.

Implementation Details

1. Preprocessing Pipeline

After a book is translated, a background task (we use Celery) runs a pipeline that extracts important phrases from the source text and aligns them with their translations. For a first version, we focused on noun phrases, as those are most likely to be clicked for clarification.

We used spaCy for POS tagging and noun chunking, and SentenceTransformers to help align source phrases with target phrases by cosine similarity.

import spacy
from sentence_transformers import SentenceTransformer, util

nlp_source = spacy.load("en_core_web_sm")
nlp_target = spacy.load("es_core_news_sm")  # example target
model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')

def extract_phrases(sent, nlp):
    doc = nlp(sent)
    return [chunk.text for chunk in doc.noun_chunks]

def align_phrases(src_sent, tgt_sent):
    src_phrases = extract_phrases(src_sent, nlp_source)
    tgt_phrases = extract_phrases(tgt_sent, nlp_target)
    if not src_phrases or not tgt_phrases:
        return []
    src_embs = model.encode(src_phrases)
    tgt_embs = model.encode(tgt_phrases)
    scores = util.cos_sim(src_embs, tgt_embs)
    pairs = []
    for i, src in enumerate(src_phrases):
        best_idx = scores[i].argmax()
        if scores[i][best_idx] > 0.7:
            pairs.append((src, tgt_phrases[best_idx]))
    return pairs

This approach misses many verbs and adjectives, but it gave us a solid starting point. We stored the glossary in PostgreSQL with the source sentence to provide context later.

CREATE TABLE glossary (
    id SERIAL PRIMARY KEY,
    book_id INT NOT NULL,
    source_phrase TEXT NOT NULL,
    target_phrase TEXT NOT NULL,
    context_sentence TEXT,
    created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_glossary_book_phrase ON glossary (book_id, source_phrase);

The context_sentence column became crucial for disambiguation—more on that next.

2. Resolving Ambiguity with Embeddings

A simple glossary lookup on the surface form can be ambiguous: “bank” could be a river bank or a financial institution. To reduce mis-hits, we added a context‑matching step that compares the embedding of the user’s surrounding sentence with the gloss entry’s context sentence. Only if the similarity is high enough do we serve the glossary answer; otherwise, we fall through to the LLM.

def is_context_match(user_context, gloss_context):
    if not gloss_context or not user_context:
        return True  # not enough data, trust the match
    emb1 = model.encode(user_context)
    emb2 = model.encode(gloss_context)
    return util.cos_sim(emb1, emb2) > 0.6

This simple check improved precision by about 30% in our tests.

3. The FastAPI Endpoint

The core of the feature is a FastAPI endpoint that receives the book ID, selected text, and the surrounding sentence (grabbed by the frontend). We keep a local in‑memory cache (via cachetools.TTLCache) for LLM fallback results to avoid extra network trips.

from fastapi import FastAPI, Depends
from sqlalchemy.orm import Session
from cachetools import TTLCache
from pydantic import BaseModel

app = FastAPI()
llm_cache = TTLCache(maxsize=10000, ttl=3600)

class HelpRequest(BaseModel):
    book_id: int
    selected_text: str
    context_sentence: str = ""

@app.post("/translate-help")
async def translate_help(req: HelpRequest, db: Session = Depends(get_db)):
    normalized = req.selected_text.strip().lower()

    # 1. Check glossary with context
    gloss_entry = db.query(Glossary).filter(
        Glossary.book_id == req.book_id,
        Glossary.source_phrase.ilike(normalized)
    ).first()
    if gloss_entry and is_context_match(req.context_sentence, gloss_entry.context_sentence):
        return {"translation": gloss_entry.target_phrase, "source": "glossary"}

    # 2. Check LLM cache
    cache_key = f"{req.book_id}:{normalized}"
    if cache_key in llm_cache:
        return {"translation": llm_cache[cache_key], "source": "llm_cache"}

    # 3. Fallback to LLM (DeepSeek)
    prompt = (
        f"Translate the word '{req.selected_text}' in this context:\n"
        f"{req.context_sentence}\n"
        "Provide only the translation, no explanation."
    )
    translation = await call_deepseek(prompt, timeout=2.0)
    if translation:
        llm_cache[cache_key] = translation
        return {"translation": translation, "source": "llm"}
    else:
        return {"translation": "Translation not available", "source": "error"}

We chose DeepSeek for the fallback because its API response time for short prompts averaged 1.2 seconds, compared to 2-4 seconds for Claude. For the cache, we started with a simple in‑memory TTLCache to avoid Redis serialization overhead for small strings. When we later scaled to multiple workers, we added Redis as a shared second‑level cache, but the local one still handles ~80% of cache reads.

Performance and Results

We measured performance over a week on a sample of 100 books (mix of fiction and non‑fiction):

• Glossary hit rate: 45% of queries (highest for technical books, lowest for poetry)
• LLM cache hit rate: 20%
• LLM fallback rate: 35%
• p95 overall latency: 120 ms (with hits from glossary/cache returning in <10 ms)
• LLM fallback average latency: 1.3 s

User feedback was positive—the occasional 1.5‑second wait for an obscure word was acceptable. Most readers never noticed any delay.

Lessons Learned & Trade‑offs

• Glossary coverage: Our noun‑phrase‑only extraction left gaps (verbs, adjectives, idioms). A better word‑aligner (like awesome-align) could improve recall. We’re also experimenting with using the LLM itself during translation to output explicit translation pairs.
• Cold start: The first readers of a newly translated book see more fallback calls. A background job could pre‑seed the cache with the top ~1000 words.
• LLM reliability: Sometimes the LLM returns a full sentence instead of a single word. We added output validation: if the result is longer than 5 words, we reject it and show a generic message.
• Caching invalidation: When a user reports a bad translation and we improve our prompt, we must clear the cache. We version our cache keys with a translation version number.
• Cost: Even though 35% of queries hit the LLM, the per‑call cost is tiny (DeepSeek is very cheap). We’ve spent less than $5/month on this feature.

What’s Next?

We’re now testing a small, fine‑tuned T5 model deployed directly on our VPS to replace the DeepSeek fallback. Early results show ~200ms latency for most words, which would make the feature feel truly instant for every click.

Building 即时翻译帮助 reminded us that hybrid systems—combining fast heuristics with modern AI—often deliver the best user experience. Start simple, measure your cache hit rates, and iterate on the glossary quality. And never underestimate the power of a good cache!

We’d love to hear how others are tackling real‑time AI features. What architectures have worked for you?