How We Translate Entire Books with LLMs Without Losing Context

Chunking strategies, context carry-over, and glossary injection for long-form translation

The Problem: Translating Books is Not Like Translating Tweets

At LectuLibre, we translate entire books using LLMs like Claude and DeepSeek. Early on, we discovered that feeding a whole book into the API wasn't just expensive—it was impossible. A 300-page novel can be 120,000 tokens, but even the best models max out at 200k context (and the quality degrades near the limit). So we had to chunk.

Our first attempt was naive: split text into fixed-size chunks of 4,000 characters. The result? Sentences cut off mid-thought, paragraphs split across chunks, and worst of all, the model lost track of the story. Characters' names changed, tone shifted, and the translation felt disjointed.

We needed a better way: chunk by structure, not just size, and carry context across chunks.

Our Chunking Strategy: Paragraph-Aware with Overlap

We decided to split on paragraph boundaries using Python's textwrap? Actually we needed more control. We used nltk for sentence tokenization and then grouped sentences into chunks that fit within a token budget.

Here's the core function we wrote:

import nltk
from typing import List

nltk.download('punkt')

def chunk_by_paragraphs(text: str, max_tokens: int = 6000, overlap_sentences: int = 3) -> List[str]:
    # Split into paragraphs, then sentences
    paragraphs = text.split('\n\n')
    sentences = []
    for para in paragraphs:
        sentences.extend(nltk.sent_tokenize(para))

    chunks = []
    current_chunk = []
    current_tokens = 0

    for sentence in sentences:
        # Estimate tokens (roughly 4 chars per token)
        sentence_tokens = len(sentence) // 4 + 1
        if current_tokens + sentence_tokens > max_tokens and current_chunk:
            chunks.append(' '.join(current_chunk))
            # Keep overlap sentences for context continuity
            overlap = current_chunk[-overlap_sentences:] if overlap_sentences > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(s) // 4 + 1 for s in current_chunk)
        current_chunk.append(sentence)
        current_tokens += sentence_tokens

    if current_chunk:
        chunks.append(' '.join(current_chunk))

    return chunks

Key decisions:

• Split on paragraphs first, then sentence tokenization using NLTK (offline, reliable).
• Overlap of 3 sentences between chunks: this bridges context and prevents the model from losing the thread at boundaries.
• We set max_tokens to 6000 as a safe limit for Claude 3.5 Sonnet, leaving room for the prompt and response.

But chunking alone wasn't enough. The model still forgot what happened in earlier chapters.

Carrying Context: Chapter Summaries and Sliding Window

We experimented with three approaches:

1. No context: Just translate each chunk independently. Result: inconsistent terminology, story drift, poor quality.
2. Previous chunk as context: Include the last chunk's translated text (or source) as a prefix. Helped at boundaries but didn't solve long-range dependencies.
3. Hierarchical summarization: Before translating a chapter, generate a summary of the previous chapter(s) and inject that as a "story so far" context.

We settled on a hybrid: for each chapter, we generate a summary of the previous chapter (or the last few if short) and include it in the system prompt. For within-chapter chunks, we use the overlapping sentences plus a short local context.

Here's the summary generation code:

import anthropic

client = anthropic.Anthropic(api_key="...")

def generate_chapter_summary(chapter_text: str, max_summary_tokens: int = 500) -> str:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=max_summary_tokens,
        system="You are a literary summarizer. Create a concise summary of the given chapter, focusing on plot points, character introductions, and key events. Preserve names and terminology exactly.",
        messages=[{"role": "user", "content": f"Summarize this chapter:\n\n{chapter_text}"}]
    )
    return response.content[0].text

Then when translating a chunk, we build a prompt like:

def translate_chunk(chunk: str, context_summary: str, glossary: dict) -> str:
    system_prompt = f"""You are a professional literary translator. Translate the following text from {source_lang} to {target_lang}.

    Context from previous chapters:
    {context_summary}

    Maintain consistency with the provided glossary:
    {glossary}

    Preserve formatting, tone, and style. Do not add explanations."""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=len(chunk) * 3,  # generous output tokens
        system=system_prompt,
        messages=[{"role": "user", "content": chunk}]
    )
    return response.content[0].text

Glossary Injection for Terminology Consistency

One of the biggest challenges in book translation is keeping character names, place names, and invented terms consistent. LLMs are prone to transliterating names differently or using synonyms. We solved this by building a dynamic glossary.

We first run a lightweight NER (Named Entity Recognition) pass using spaCy to extract potential entities from the source text. Then we ask the LLM to filter and translate those entities into the target language (or keep original if it's a name). We store the mapping and inject it into every translation prompt.

Here's our glossary extraction:

import spacy

nlp = spacy.load("en_core_web_sm")

def extract_entities(text: str, limit: int = 50) -> List[str]:
    doc = nlp(text)
    entities = set()
    for ent in doc.ents:
        if ent.label_ in ["PERSON", "GPE", "LOC", "ORG", "PRODUCT"]:
            entities.add(ent.text)
    return list(entities)[:limit]

Then we ask the LLM to create a glossary:

def build_glossary(entities: List[str], target_lang: str) -> dict:
    prompt = f"Given these entities, provide a translation into {target_lang}. If the entity should remain unchanged (e.g., a proper name), return it as is. Output as JSON.\n\nEntities: {entities}"
    # call LLM, parse JSON
    ...

We then pass this glossary to every translation call. This reduced name inconsistencies from about 15% to less than 2% in our tests.

Real Numbers: Cost and Quality

We translated a 120,000-token book (about 350 pages) using this pipeline. On Claude 3.5 Sonnet:

• Total API calls: ~40 chunks + 10 summary calls + glossary calls ≈ 55 requests.
• Cost: $6.50 (input tokens dominated by context carry-over).
• Time: ~3 minutes with async concurrency of 5 requests.
• Quality: Human reviewer rated consistency 8.5/10 vs 5/10 for naive chunking.

The biggest cost factor was the context summary injection, which added about 30% more input tokens. We could reduce that by using smaller summaries or only summarizing every few chapters.

Lessons Learned and Trade-offs

• Chunk size matters: Too small loses context, too large risks truncation and slow responses. We found 4000-6000 tokens per chunk to be the sweet spot.
• Overlap is essential: Even 2-3 sentences of overlap dramatically improved boundary quality.
• Summaries are expensive but worth it: For long books, a hierarchical summary approach saved us from narrative drift. But we had to balance cost; for cheaper models like DeepSeek, we use shorter summaries.
• Glossary injection is non-negotiable: Without it, character names would change every few chapters.
• One failure: We tried to use the entire previous chapter as context, but that blew the context window and the model started ignoring the current chunk. So we always use concise summaries, not raw text.

Open Question for the Community

We're still wrestling with cross-chapter references: when a character refers to an event from 10 chapters ago, the summary might miss it. Has anyone tried using a retrieval-augmented approach for book translation? Maybe embedding chunks and retrieving relevant context on demand? We'd love to hear your ideas.

You can find more about what we're building at LectuLibre (lectulibre.com), where we put these techniques to work translating real books for readers.

---

Practical takeaway: Don't just chunk—think about what context each chunk needs to make sense. Use structural boundaries, overlap, and a compact memory of the story so far.