Under the Hood of 翻译求助: Building a Context-Aware Translation Assistant at LectuLibre

How we combined Python, FastAPI, and LLMs to let users refine book translations on the fly.

Introduction

At LectuLibre, we help users translate entire books using AI. While LLMs like Claude and DeepSeek do an impressive job, they occasionally miss nuances—especially in literary texts loaded with idioms, cultural references, or complex sentences. Our early users told us they wished they could tweak just a few paragraphs here and there instead of regenerating entire chapters. That was the birth of our “翻译求助” feature: a simple way to select any passage and get an alternative translation with a brief explanation of why we changed it.

In this article, I’ll share the technical journey—the challenges we faced, the architectural decisions we made, and the code we wrote to make context-aware translation refinement work at scale.

The Problem: Static Translations in a Dynamic World

Initially, our translation pipeline was straightforward: upload a book → chunk it by chapter → send each chunk to an LLM → stitch the results. That worked, but it was a one-shot process. If a user disliked a particular sentence, they had no recourse except re-translating the whole chapter, losing any manual edits they might have made elsewhere.

We needed a way to let users interact with the translated text and ask for improvements without breaking the overall flow. The core requirements were:

• Selective refinement: allow the user to highlight any passage and request a better translation.
• Context awareness: the LLM must consider surrounding text to maintain coherence.
• Transparency: provide a short explanation of what changed and why, so the user learns from it.
• Speed: must feel nearly instant; no one waits 10 seconds for a sentence.

Our Approach: Design Decisions and Tooling

We decided to build a dedicated REST endpoint that accepts a passage index (start and end positions in the translated document), fetches the surrounding context from our database, constructs a tailored LLM prompt, and returns the refined translation plus an explanation.

Stack

• Backend: Python, FastAPI for async IO performance.
• LLMs: We use Anthropic’s Claude 3 Haiku for speed and DeepSeek-V2 for quality; the feature allows us to pick based on user settings.
• Database: PostgreSQL stores the original book text, the translated text, and user edits. We’ll query for the surrounding segments using chunk indices.
• Caching: Redis to store short-term LLM responses for identical context/wording to save costs.

Why Not a Chat Interface?

Some platforms use a conversational approach for translation refinement, but we found that asking the LLM to “improve this translation” via a chat leads to variable results. A structured prompt with explicit instructions gave us more control over the output format.

Implementation Details

1. The API Endpoint

We created a POST /api/translation-help endpoint. The request body contains book_id, segment_start, segment_end, and optionally the model preference. The response is a JSON object with refined_translation and explanation.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Optional

app = FastAPI()

class TranslationHelpRequest(BaseModel):
    book_id: str
    segment_start: int
    segment_end: int
    model: Optional[str] = "claude"  # or "deepseek"

@app.post("/api/translation-help")
async def get_translation_help(req: TranslationHelpRequest):
    if req.segment_start < 0 or req.segment_end < req.segment_start:
        raise HTTPException(status_code=400, detail="Invalid segment range")
    context = await fetch_context(req.book_id, req.segment_start, req.segment_end)
    refined = await refine_translation(context, model=req.model)
    await cache_refinement(req.book_id, req.segment_start, req.segment_end, refined)
    return refined

2. Context Retrieval: Feeding the Right Neighbors

Translation refinement without context often produces disjointed results. For example, if you ask the LLM to retranslate a sentence starting with “He laughed,” but the preceding paragraph ended with a sad tone, the output might be tonally wrong. Therefore, we fetch a window of N paragraphs before and after the target segment. Our book text is stored with paragraph-level indices.

async def fetch_context(book_id: str, start: int, end: int, window=3) -> dict:
    query = """
    SELECT paragraph_index, original_text, translated_text
    FROM book_translations
    WHERE book_id = $1 AND paragraph_index BETWEEN $2 AND $3
    ORDER BY paragraph_index
    """
    rows = await database.fetch_all(query, book_id, start - window, end + window)
    # ... extract target and surrounding text
    return {
        "target_original": target_original,
        "target_translated": target_translated,
        "context": "\n".join(surrounding_original),
        "context_translated": "\n".join(surrounding_translated),
    }

The window size is configurable; we found that 3 paragraphs on each side strikes a good balance between providing enough context and staying within token limits.

3. Crafting the LLM Prompt

Prompt engineering made all the difference. We wanted two clear parts in the response: the improved translation and a concise explanation. We designed a system prompt that sets the role and output format, and a user prompt that injects the text.

SYSTEM_PROMPT = """You are a professional literary translator assistant. Given a passage from a book and its current translation, your task is to improve the translation, paying attention to nuance, flow, and cultural accuracy. You must output a JSON object with exactly two fields: "refined_translation" and "explanation". Only output the JSON, no other text."""

USER_PROMPT_TEMPLATE = """Original passage: {target_original}
Current translation: {target_translated}
Surrounding context (original): {context}
Surrounding context (translated): {context_translated}
Please refine the translation of the passage above."""

def build_prompt(context_data):
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT_TEMPLATE.format(**context_data)}
    ]

We force JSON output because it’s easier to parse reliably. We use json.loads() after receiving the LLM response, with error handling in case the model returns malformed JSON.

4. Async LLM Calls and Fallback

Since we support multiple models, we abstract the LLM client. We use httpx for async HTTP calls to Claude’s API and DeepSeek’s API. To keep the endpoint responsive, we set a timeout of 8 seconds; if one model fails, we try the other.

import httpx
import json

async def refine_translation(context: dict, model: str):
    prompt = build_prompt(context)
    for attempt in [model, "claude", "deepseek"]:  # fallback order
        try:
            if attempt == "claude":
                resp = await call_claude(prompt)
            else:
                resp = await call_deepseek(prompt)
            parsed = json.loads(resp)
            return parsed
        except Exception:
            continue
    raise HTTPException(500, "Translation refinement failed")

We measured average latency: ~3.2s for Claude Haiku, ~4.7s for DeepSeek, both within our target.

5. Caching Strategy

Users often refine the same passage multiple times. To avoid redundant LLM calls, we cache the result keyed on (book_id, segment_start, segment_end, model). We store it in Redis with a TTL of 1 hour. If the surrounding context changes due to user edits elsewhere, we invalidate the cache for overlapping ranges. We used a simple versioning scheme: each book has a content_version that increments on any edit, and cache keys include the version. That’s conservative but correct.

Challenges and Trade-offs

• Token limits: Book paragraphs can be long. We truncate the surrounding context to 2000 tokens using a simple heuristic (first 1000 characters per paragraph). Sometimes that cuts off important context; we plan to use a sliding window with dynamic sizing later.
• Prompt injection: Users could select text that contains malicious instructions. We sanitize inputs, but LLMs are inherently vulnerable. We mitigate by strict output parsing and limiting the model’s temperature.
• Cost: Each refinement call costs a few cents. Caching helps, but heavy users can rack up expenses. We added rate limits per user.
• Cultural nuance: Explaining why a translation changed sometimes requires cultural background, which the LLM doesn’t always get right. We’re experimenting with RAG from a knowledge base of literary references.

Results and User Feedback

After launching the feature, we saw:

• A 23% decrease in manual re-translation requests per book.
• Users reported spending less time editing after using 翻译求助.
• The explanation part surprised us—many non-native speakers use it to learn why idiomatic expressions are translated a certain way.

We’re now exploring how to let users provide feedback on the explanations to fine-tune our prompts further.

Lessons Learned

1. Context is king: Expanding the window significantly improved output quality. We initially used only the current paragraph, and the results were often stylistically mismatched.
2. JSON output constraints: Forcing the LLM to output JSON with a specific schema reduced parsing errors and made the feature more robust.
3. Caching is non-trivial: Simple caching works until users start editing the surrounding text; you need a strategy that propagates changes.
4. Async matters: FastAPI’s async capabilities kept our endpoint responsive under load.

What’s Next?

We’re considering allowing users to give natural language instructions alongside the passage (e.g., “make this more formal” or “use simpler words”). That would turn 翻译求助 into a full-blown iterative translation assistant.

We’d love to hear from the community: How do you handle context injection for LLM-based editing tasks? Any caching patterns that worked better for mutable text? Drop your thoughts in the comments.