Translating Entire Books with LLMs: Our Chunking Strategy for Context Preservation

How we built a Python pipeline to translate long-form EPUBs and PDFs without losing narrative context across chunk boundaries.

At LectuLibre, we translate entire books using large language models. Users upload an EPUB or PDF, and our backend (Python/FastAPI) orchestrates calls to Claude and DeepSeek to produce a high-quality translation. The core challenge? Books are long—hundreds of thousands of tokens—while LLMs have limited context windows. If we naively translate paragraph by paragraph, we lose critical context: pronoun references, character names, terminology consistency, and narrative flow. Here’s how we solved it with a chunking strategy that preserves context without blowing up cost or latency.

The Problem: Long Documents vs. Context Windows

An average novel contains 80,000–120,000 words, which translates to roughly 100,000–150,000 tokens. Even the largest context windows (200k tokens for Claude 3, 128k for GPT-4 Turbo) can’t hold an entire book in one prompt, and even if they could, the cost and latency would be prohibitive. Translating sentence by sentence or paragraph by paragraph leads to:

• Inconsistent terminology: A character’s name or a technical term might be translated differently each time.
• Broken references: Pronouns like “he” or “she” lose their antecedent when the previous paragraph isn’t in context.
• Style drift: The tone of translation can shift between chunks, making the book feel disjointed.

We needed a way to chunk the text into manageable pieces while carrying over enough context to maintain coherence.

Our Approach: Overlapping Chunks with Contextual Memory

We use a hierarchical chunking strategy that works at two levels:

1. Book structure: We respect natural boundaries like chapters (from EPUB) or detected headings (from PDF).
2. Within a chapter: We split paragraphs into chunks that fit within a safe token limit, but we include two types of contextual information:
   • Overlap paragraphs: The last few paragraphs from the previous chunk are included as untranslated context (or their already-translated version).
   • Running glossary: A dictionary of key terms and their translations, built incrementally as we translate.

We also send a system prompt that describes the book’s genre, target language, and translation guidelines.

Why This Works

Including the previous chunk’s translated text (or a summary) gives the LLM the immediate narrative context. The glossary ensures consistency for names, places, and domain-specific terms. By keeping chunks relatively small (2,000–4,000 tokens), we stay well within the model’s comfort zone and keep API calls fast.

Implementation Details

Here’s the core of our Python pipeline. We use tiktoken for fast token estimation, ebooklib for EPUB parsing, and pypdf for PDFs. API calls are made asynchronously with httpx and the Anthropic/DeepSeek SDKs.

Token Counting

We approximate token counts using tiktoken’s cl100k_base encoding. It’s not exact for Claude’s tokenizer, but it’s close enough for chunk sizing with a safety margin.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

Chunking Paragraphs

We split a chapter’s paragraphs into chunks that stay under max_tokens. To preserve continuity, we include the last overlap_paragraphs (usually 2) from the previous chunk as untranslated context.

def chunk_paragraphs(paragraphs: list[str], max_tokens: int = 3500, overlap_paragraphs: int = 2) -> list[list[str]]:
    chunks = []
    current_chunk = []
    current_tokens = 0

    for para in paragraphs:
        para_tokens = count_tokens(para)
        if current_tokens + para_tokens > max_tokens and current_chunk:
            chunks.append(current_chunk)
            # Start new chunk with overlap paragraphs from previous chunk
            overlap = current_chunk[-overlap_paragraphs:] if overlap_paragraphs else []
            current_chunk = overlap.copy()
            current_tokens = sum(count_tokens(p) for p in overlap)
        current_chunk.append(para)
        current_tokens += para_tokens

    if current_chunk:
        chunks.append(current_chunk)
    return chunks

Important: We do not translate the overlap paragraphs again. They are included only as context; the LLM is instructed to translate only the new paragraphs appended after the overlap. This prevents duplication.

Building the Glossary

During translation, we extract proper nouns and domain terms from the source text and track their translations. We use a simple regex-based extractor for capitalized words and a manually curated list for common terms.

import re
from collections import defaultdict

glossary = defaultdict(str)

def update_glossary(source_text: str, translated_text: str):
    # Extract potential proper nouns (crude but works for many books)
    candidates = re.findall(r'\b[A-Z][a-z]+\b', source_text)
    for term in candidates:
        if term not in glossary:
            # Ask LLM for translation of this term (or use a separate call)
            glossary[term] = translate_term(term)  # simplified

In practice, we don’t make a separate API call for every candidate; instead, we include the glossary in the prompt and ask the model to suggest translations for new terms as part of its output. We parse those suggestions from the response with a lightweight JSON format.

Translating a Chunk

Here’s the async function that translates one chunk. It takes the chunk (list of paragraphs), the previous chunk’s translated text as context, and the current glossary.

async def translate_chunk(
    chunk: list[str],
    context: str,          # previous chunk's translated text (empty for first)
    glossary: dict,
    model: str = "claude-3-haiku-20240307"
) -> str:
    # Build prompt
    prompt = f"""You are translating a book from English to Spanish.\n\
Maintain the original tone, style, and formatting.\n\
Use the following glossary for consistency (term: translation):\n\
{glossary}\n\
\n\
Context from the previous section (already translated):\n\
{context}\n\
\n\
Now translate the following paragraphs. Only translate the paragraphs after the context.\n\
Do not repeat the context.\n\
\n\
{''.join(chunk)}"""

    # API call (async)
    if model.startswith("claude"):
        response = await anthropic_client.messages.create(
            model=model,
            max_tokens=4096,
            messages=[{"role": "user", "content": prompt}]
        )
        return response.content[0].text
    else:
        # DeepSeek or other OpenAI-compatible
        response = await deepseek_client.chat.completions.create(
            model=model,
            max_tokens=4096,
            messages=[{"role": "user", "content": prompt}]
        )
        return response.choices[0].message.content

Orchestrating the Full Book

We process chapters sequentially but translate chunks within a chapter asynchronously to maximize throughput. For each chapter, we:

1. Extract paragraphs.
2. Chunk them (with overlap).
3. Initialize context = "" and glossary = {}.
4. For each chunk in order:
   • Call translate_chunk(chunk, context, glossary).
   • Update context with the translated output (truncated to last 500 tokens).
   • Update glossary with any new terms from the output.
5. Concatenate translated chunks, removing the overlap duplicates.

We use asyncio.gather to translate multiple chunks concurrently, but we must be careful: context depends on the previous chunk’s output, so true parallelism isn’t possible within a chapter. However, we can pre-translate a chapter’s chunks without context in parallel to get a rough draft, then refine sequentially with context? We found that sequential translation with context gave better quality, so we accept the latency. For speed, we run multiple books concurrently on our VPS.

Performance Numbers

For a typical 100,000-token novel (about 80k words):

• Number of chunks: ~30 chunks (average 3,300 tokens per chunk including overlap).
• API calls: 30 sequential calls.
• Cost: Using Claude 3 Haiku ($0.25 / 1M input tokens, $1.25 / 1M output tokens) and DeepSeek for some chapters, total cost per book ≈ $1.50–$3.00.
• Latency: With async calls and retries, total translation time ≈ 4–6 minutes per book.
• Quality: In a side-by-side comparison with naive paragraph-by-paragraph translation, our context-aware approach reduced pronoun errors by ~70% and terminology inconsistencies by ~80% (measured on a sample of 10 books).

Lessons Learned and Trade-offs

• Overlap size matters: Too little overlap (1 paragraph) causes context breaks; too much (5+ paragraphs) leads to repetitive phrasing because the model sometimes re-translates the overlap. We settled on 2 paragraphs for narrative text, 1 for technical books.
• Context format: We initially tried sending a summary of the previous chunk (generated by another LLM call) instead of the actual translation. The summary saved tokens but lost style and voice. Using the actual translated text as context preserved the author’s tone better, even if it cost more tokens.
• Glossary maintenance: The glossary is critical for consistency, but building it automatically is tricky. We started with a regex for capitalized words, but that missed many terms. We now use a combination of spaCy named entity recognition and a small curated list. The LLM also suggests translations for new terms, which we review asynchronously.
• PDF vs EPUB: EPUBs are far easier because they contain structural markup. For PDFs, we use pypdf to extract text, but we lose paragraph boundaries. We then use heuristics (double newlines, indentation) to reconstruct paragraphs. This works acceptably but sometimes merges dialogue lines incorrectly.
• Rate limiting and retries: LLM APIs have rate limits. We implemented exponential backoff with tenacity and a semaphore to cap concurrent calls. On a VPS with 4 cores, we can translate 3–4 books concurrently without hitting limits.

What’s Next?

Our current approach works well for novels and non-fiction, but there’s an open question: how to handle cross-chapter references? A character introduced in chapter 1 might be mentioned again in chapter 10. Our glossary helps with names, but subtle narrative callbacks are lost. We’re experimenting with a book-level summary that is updated after each chapter and included as high-level context in the system prompt. Early results are promising but increase token usage by ~10%.

If you’re building a similar system, start with overlapping chunks and a glossary—it covers 80% of the consistency issues. The remaining 20% requires more sophisticated memory, but for most books, the trade-off isn’t worth the complexity.

Open question for the community: How do you handle translation of poetry or highly formatted text (tables, footnotes) without losing layout? We currently strip them or translate separately, but it’s not ideal. We’d love to hear your approaches.