Streaming Instant Translation with Cultural Insights: The Engineering Behind LectuLibre’s 即时翻译与文字解读

How we built a feature that translates e-book text in real time and explains cultural nuances, all while keeping latency under 2 seconds with Python and FastAPI.

The Problem: More Than Just Words

At LectuLibre, we translate entire books using AI. But during development, we discovered that readers often get stuck on specific passages—idioms, cultural references, puns—that static translation glosses over. They wanted a quick way, while reading, to get not just a translation of a tricky sentence but also an explanation of its deeper meaning. So we built 即时翻译与文字解读 (Instant Translation & Text Interpretation). The goal: select any text in a book, click a button, and within seconds see a translation plus insightful commentary.

From an engineering perspective, this meant:

• Must be fast (<2 seconds for most passages) to keep the reading flow.
• Must handle many concurrent users (LectuLibre has thousands of active readers).
• Must control LLM costs, as each request uses API credits.
• Must deliver results incrementally so users feel instant feedback.

Our Approach: Single LLM Call with Streaming and Caching

We decided to use a single LLM call (Anthropic’s Claude 3.5 Sonnet) to generate both the translation and the interpretation. This reduces latency compared to two sequential calls. To speed up perceived performance, we stream the LLM’s response back to the frontend using Server-Sent Events (SSE). The stream first delivers the translation chunk by chunk, then the interpretation. So the translation appears quickly, while the more elaborate interpretation loads shortly after.

We also implemented aggressive caching with Redis, because many sentences in a book (or across books) are repeated or similar. A cache hit completely bypasses the LLM, dropping latency to near-zero.

Implementation: Backend in Python FastAPI

Our backend is built with FastAPI. Let’s dive into the core endpoint.

1. API Design

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
import hashlib
from slowapi import Limiter
from slowapi.util import get_remote_address
import redis.asyncio as redis

app = FastAPI()
redis_client = redis.Redis(decode_responses=True)
limiter = Limiter(key_func=get_remote_address)

class TranslateRequest(BaseModel):
    text: str
    source_lang: str
    target_lang: str
    context_prev: str | None = None
    context_next: str | None = None
    book_id: str | None = None

@app.post("/translate_interpret")
@limiter.limit("10/minute")
async def translate_interpret(request: Request, body: TranslateRequest):
    # Create a cache key from text + language pair
    raw_key = f"{body.text}|{body.source_lang}|{body.target_lang}"
    cache_key = hashlib.sha256(raw_key.encode()).hexdigest()

    cached = await redis_client.get(cache_key)
    if cached:
        # For cache hits, we return the stored SSE-formatted string directly
        return StreamingResponse(
            iter([cached]),
            media_type="text/event-stream"
        )

    # Otherwise, call the LLM and stream the processed response
    prompt = build_prompt(body)
    return StreamingResponse(
        stream_llm(prompt, cache_key, body),
        media_type="text/event-stream"
    )

We use slowapi for rate limiting to prevent abuse. The cache key is a SHA256 hash of the normalized input to avoid huge keys.

2. Prompt Engineering for Structured Streaming

Our prompt instructs the LLM to output the translation first, wrapped in a special marker, then the interpretation. For example:

def build_prompt(req: TranslateRequest) -> str:
    context = ""
    if req.context_prev:
        context += f"Previous sentence: {req.context_prev}\n"
    if req.context_next:
        context += f"Next sentence: {req.context_next}\n"

    return f"""
You are a literary translation assistant. Translate the following text from {req.source_lang} to {req.target_lang}. Then, provide a brief interpretation that explains any cultural references, idioms, or nuances.

{context}
Text: {req.text}

Respond **exactly** in this format:
[TRANSLATION]
<your translation here>
[INTERPRETATION]
<your interpretation here>
"""

In the streaming generator, we parse the raw LLM stream for the markers [TRANSLATION] and [INTERPRETATION] and emit SSE events accordingly.

3. Streaming Generator with Marker Parsing

We use httpx to make an asynchronous streaming request to the Anthropic API (or any LLM with streaming). The generator looks for the markers and yields SSE events.

import httpx
import asyncio
import json

LLM_API_URL = "https://api.anthropic.com/v1/messages"
HEADERS = {
    "x-api-key": "your-api-key",
    "anthropic-version": "2023-06-01"
}

async def stream_llm(prompt: str, cache_key: str, req: TranslateRequest):
    data = {
        "model": "claude-3-5-sonnet-20240620",
        "max_tokens": 1024,
        "stream": True,
        "messages": [{"role": "user", "content": prompt}]
    }

    async with httpx.AsyncClient() as client:
        async with client.stream("POST", LLM_API_URL, headers=HEADERS, json=data) as response:
            buffer = ""
            current_marker = None
            full_response = ""
            async for chunk in response.aiter_text():
                if chunk.startswith("data: "):
                    payload = chunk[6:]
                    if payload == "[DONE]":
                        break
                    try:
                        obj = json.loads(payload)
                        text = obj.get("delta", {}).get("text", "")
                    except:
                        continue
                    full_response += text
                    buffer += text

                    # Check for markers in buffer
                    if "[TRANSLATION]" in buffer and current_marker is None:
                        idx = buffer.find("[TRANSLATION]")
                        before = buffer[:idx]
                        if before.strip():
                            yield f"event: unknown\ndata: {before}\n\n"
                        buffer = buffer[idx+len("[TRANSLATION]:"):]
                        current_marker = "translation"
                        yield "event: translation_start\ndata: \n\n"
                        continue
                    elif "[INTERPRETATION]" in buffer and current_marker == "translation":
                        idx = buffer.find("[INTERPRETATION]")
                        translation_end = buffer[:idx]
                        if translation_end.strip():
                            yield f"event: translation\ndata: {translation_end}\n\n"
                        buffer = buffer[idx+len("[INTERPRETATION]:"):]
                        current_marker = "interpretation"
                        yield "event: interpretation_start\ndata: \n\n"
                        continue
                    # Emit partial data based on current marker
                    if current_marker == "translation" and buffer:
                        yield f"event: translation\ndata: {buffer}\n\n"
                        buffer = ""
                    elif current_marker == "interpretation" and buffer:
                        yield f"event: interpretation\ndata: {buffer}\n\n"
                        buffer = ""
            # After streaming ends, store full response in cache
            await redis_client.setex(cache_key, 86400, full_response)

This is a simplified version; in production we handle edge cases like markers appearing split across chunks, which requires a more robust parser. We fell back to buffering until a newline, but it worked well in practice because the markers were at the start of new lines.

4. Caching Strategy

We cache the raw LLM output (the full string including markers) in Redis. On a cache hit, we can simply replay the stream as if it were coming from the LLM. However, for simplicity, our endpoint returns the entire cached content as a single SSE event containing both translation and interpretation. A better approach would parse the cached string and emit chunked events, but we found that a single event still felt fast enough (since the data is already available client-side). We might improve this later.

Performance and Lessons Learned

• Latency: Uncached requests to Claude 3.5 Sonnet averaged 1.8 seconds until the first translation chunk, and 2.5 seconds for the complete response. With caching (roughly 40% hit rate for a typical novel due to repeated phrases), the p50 latency dropped to 80ms.
• Cost: Each request costs ~$0.003 with Claude. Caching saved us around $500 per month for 10,000 daily active users.
• Translation quality: Providing one sentence of context before and after dramatically reduced misinterpretation of ambiguous words. We also experimented with sending the entire paragraph if the selected text was a fragment, but that increased token usage.
• Streaming parsing pain: The marker-based approach is fragile. If the LLM decides to insert an extra space or newline, the marker detection can fail. We added fallback by detecting common patterns with regex, but we’re planning to switch to tool use / function calling with structured output once the LLMs support streamable JSON. For now, it serves 99% of cases.

Frontend Integration

The frontend is a React-based reader. It listens to SSE events: translation_start, translation (chunks), interpretation_start, interpretation. It renders the translation in a side panel as it arrives, and the interpretation appears below after. We used the EventSource API with a polyfill for broad compatibility.

Open Question for the Community

How are you handling streaming structured output from LLMs without hacks? Has anyone found a reliable library or technique to parse a partial JSON object as it streams? We’d love to hear your solutions.

---

Takeaway: Building a real-time translation feature with cultural insights required balancing latency, cost, and accuracy. By streaming from a single LLM call and caching aggressively, we delivered a responsive experience that keeps readers in the flow. The marker-parsing hack got the job done, but we’re keen to evolve as the ecosystem matures.