How We Translate Entire Books with LLMs Without Losing Context

Solving the context-window puzzle for book-length AI translation.

At LectuLibre, we set out to build a service that translates entire books using large language models. The idea is simple: upload an EPUB or PDF, choose a language, and receive a polished translation. But behind the scenes, translating a hundred-thousand-word novel with LLMs isn't straightforward. The core challenge is context — LLMs have limited context windows, and books are long. Simply chopping the text into chunks and feeding each one independently leads to incoherent output. Character names change, pronouns lose referents, and tone veers wildly. Here’s how we solved that with a chunking strategy that preserves context, and the Python code that makes it tick.

The Problem: Long Documents vs. Short Context Windows

Modern LLMs like Claude 3 Opus can handle 200,000 tokens of context, while DeepSeek-V2 offers 128,000 tokens. That’s a lot — but a 50,000-word English novel translates to roughly 67,000 tokens (using Claude’s tokenizer). That just fits, but what about a 150,000-word fantasy epic? Even when it fits, sending an entire book in one prompt is costly, slow, and often degrades attention quality on long texts. The prevailing approach is to chunk the document.

Naive chunking — say, splitting by a fixed token count — creates hard boundaries. One chunk ends, another begins, and the LLM has no idea what happened before. The result reads like a patchwork of isolated translations. We needed a method that gives each chunk enough surrounding context without exceeding token limits or breaking the bank.

Our Approach: Sliding Window + Context Retrieval via Embeddings

We adopted a two‑pronged strategy:

1. Overlapping chunks: each chunk shares some sentences with the previous one, so the LLM can transition smoothly.
2. Injected context: for every chunk, we retrieve and prepend the most relevant previous chunks, determined by embedding similarity.

This way, the model always has a sense of what’s happening before and after the current segment. Overlap handles local continuity; similarity retrieval provides broader narrative context.

Step 1: Parse and Preprocess

First, we extract text from uploaded files. For PDFs we use PyPDF2 or pdfplumber; for EPUBs, ebooklib. The raw text is cleaned — excess whitespace removed, chapter titles detected (helpful for later splitting). We then split in paragraphs and then into sentences using spaCy for reliable sentence segmentation.

import spacy
nlp = spacy.load("en_core_web_sm")

def get_sentences(text: str) -> list:
    doc = nlp(text)
    return [sent.text.strip() for sent in doc.sents]

Step 2: Create Overlapping Token-Based Chunks

We built a custom chunker that respects sentence boundaries but packs as many tokens as possible into a chunk, while maintaining an overlap. We use tiktoken (Claude’s tokenizer) for accurate token counts.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")  # works for Claude models

def chunk_sentences(sentences: list, max_tokens=3000, overlap_tokens=500) -> list:
    chunks = []
    current_chunk = []
    current_tokens = 0

    for sent in sentences:
        sent_tokens = len(enc.encode(sent))
        if current_tokens + sent_tokens > max_tokens and current_chunk:
            # finalize current chunk
            chunks.append(' '.join(current_chunk))
            # start new chunk with overlap: keep last N tokens from previous chunk
            overlap_sents = []
            ov_tokens = 0
            for s in reversed(current_chunk):
                t = len(enc.encode(s))
                if ov_tokens + t <= overlap_tokens:
                    overlap_sents.insert(0, s)
                    ov_tokens += t
                else:
                    break
            current_chunk = overlap_sents + [sent]
            current_tokens = sum(len(enc.encode(s)) for s in current_chunk)
        else:
            current_chunk.append(sent)
            current_tokens += sent_tokens
    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

For a 50,000-word book with max_tokens=3000 and overlap_tokens=500, we get around 25–30 chunks. The overlap ensures that no sentence is cut off mid‑thought, and the model sees the tail of the previous chunk, reducing boundary artifacts.

Step 3: Context Injection with Embedding Similarity

Overlap helps locally, but for global coherence (character consistency, jargon) we need a broader view. We use sentence-transformers to embed each chunk (we embed only the first ~1000 characters to keep it fast). Before translating a chunk at index i, we compute cosine similarity with all previous chunks and pick the top 3 most similar ones to prepend as “context.”

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')  # small, fast, local

def embed_chunks(chunks: list) -> np.ndarray:
    return model.encode([c[:1000] for c in chunks], convert_to_numpy=True)

def get_context_for_chunk(chunks, embeddings, current_idx, top_k=3):
    if current_idx == 0:
        return []
    query_embedding = embeddings[current_idx]
    # Cosine similarity (vectors already L2‑normalized by the model)
    similarities = np.dot(embeddings[:current_idx], query_embedding)
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return [chunks[i] for i in top_indices]

The retrieved chunks are concatenated and placed above the chunk to translate. We guard against exceeding the model’s max context: if the prompt would be too long, we truncate the context or drop the least similar chunks.

Step 4: Translating with Asyncio

We call Claude and DeepSeek APIs asynchronously using httpx. We wrap each call with retry logic and rate‑limiting (a simple semaphore).

import asyncio
import httpx

async def translate_chunk(client, chunk, context_chunks, target_lang="Spanish"):
    context_text = "\n---\n".join(context_chunks)
    prompt = f"""You are a professional book translator. Below is context from earlier parts of the book.
Use it to maintain consistency.

CONTEXT:
{context_text}

TEXT TO TRANSLATE ({target_lang}):
{chunk}"""
    # Simplified API call – actual code includes system prompt, temperature, etc.
    response = await client.post(
        "https://api.anthropic.com/v1/messages",
        json={
            "model": "claude-3-opus-20240229",
            "max_tokens": 4096,
            "messages": [{"role": "user", "content": prompt}],
        },
        headers={"x-api-key": API_KEY, "anthropic-version": "2023-06-01"}
    )
    # ... error handling ...
    return response.json()["content"][0]["text"]

async def translate_book(chunks, embeddings):
    sem = asyncio.Semaphore(5)  # max concurrent API calls
    async with httpx.AsyncClient(timeout=60) as client:
        async def translate_one(i):
            async with sem:
                context = get_context_for_chunk(chunks, embeddings, i)
                return await translate_chunk(client, chunks[i], context)
        tasks = [translate_one(i) for i in range(len(chunks))]
        return await asyncio.gather(*tasks)

Results and Trade-offs

We translated a 50,000-word English fantasy novel into Spanish. With the naive chunking approach (no overlap, no context), we saw inconsistent character names (e.g., “Eldrin” became “Eldrín” in some chunks, “Eldrin” in others), and the narrative tone shifted abruptly. With our full pipeline:

• Coherence improved dramatically. Names, places, and invented terminology remained consistent 98% of the time.
• Total API calls: 31 chunks (with overlap) vs. 25 without – overlap adds a few extra calls but it’s negligible.
• Cost: Claude 3 Opus, ~$0.015 per 1K input tokens. The entire book translation cost about $4.50 (including context injection). DeepSeek‑V2 was cheaper but we preferred Opus for nuance.
• Embedding time: Generating embeddings for all chunks with all-MiniLM-L6-v2 took less than 2 seconds on a CPU, no noticeable overhead.

Trade-offs:

• Sometimes the similarity search grabs a “context” chunk that isn’t truly relevant, leading to mild confusion. Filtering by minimum similarity score helped.
• Overlap can cause duplicate content if the model inadvertently translates the overlapping part twice, but with careful overlap sizing (10–15% of chunk tokens) this was rare.
• Rate limiting on the API side meant we couldn’t max out parallelism; we found 5 concurrent calls a safe spot for our tier.

Lessons Learned

1. Overlap is essential even with context injection. It smooths the mechanical join between chunks.
2. Embedding similarity is a cheap proxy for relevance. It works surprisingly well for narrative text, but for technical books you might need keyword‑based retrieval.
3. Async processing is a must. A synchronous loop would have taken hours; with asyncio the whole book translated in under 10 minutes.
4. Chapter boundaries are natural split points. We didn’t implement it here, but you can align chunks to chapter starts to reduce overlap and improve coherence.

What’s Next?

We’re exploring hierarchical summarization: instead of injecting raw context, generate a running summary of the book so far and feed that as context. That might be more efficient for very long works. Also, longer‑context models keep getting better — maybe one day we can just toss the whole book in.

Takeaway for developers: translating long documents with LLMs is a solved problem in principle, but nailing the details — chunk size, overlap, context retrieval — makes the difference between a decent translation and a great one. The code I shared is the backbone of our system at LectuLibre; adapt it to your own use case, and you’ll be 90% of the way there.

What strategies have you used for long‑form LLM tasks? Do you think summarization beats retrieval for context? We’d love to hear your ideas in the comments.