Translating Entire Books with LLMs: A Chunking Strategy That Doesn't Lose Context

How we built a Python pipeline to chunk books, preserve context, and maintain consistent terminology across hundreds of chapters.

The Problem: Translating a Whole Book, Not Just a Page

At LectuLibre, we let users upload an EPUB or PDF and get back a professionally translated book. The tricky part isn't calling an LLM—it's doing it across 100,000+ words without losing the plot, literally.

Naively, you might split the book into chunks that fit the LLM's context window and translate each independently. We tried that first. The result: names changed spelling halfway through, terminology was inconsistent, and dialogue references to earlier events sometimes made no sense. The story fell apart.

Context windows are huge now (Claude's 200k tokens, GPT-4 Turbo's 128k), but even those aren't enough for a full novel—and even if they were, cost and quality degrade with massive inputs. So we needed a chunking strategy that respects the document's structure and carries context across chunks.

Our Approach: Structure-Aware Chunking with Overlap and a Running Glossary

We broke the problem into three parts:

1. Split by document structure, not arbitrary token counts. Books are organized into chapters, sections, paragraphs. We extract those first.
2. Overlap chunks to give the model a few sentences of surrounding context for continuity.
3. Maintain a glossary of character names, place names, and technical terms that gets injected into every translation prompt.

Here's the pipeline in Python:

# Extract text from EPUB using ebooklib
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup

def extract_chapters(epub_path):
    book = epub.read_epub(epub_path)
    chapters = []
    for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
        soup = BeautifulSoup(item.get_content(), 'html.parser')
        # Extract paragraphs, join with newlines
        text = '\n\n'.join(p.get_text() for p in soup.find_all('p'))
        if text.strip():
            chapters.append(text.strip())
    return chapters

That gives us a list of chapters. But chapters can be long, so we further split them into chunks of roughly 4,000 tokens (a sweet spot we found for quality vs. cost). We use tiktoken to count tokens, and we split at paragraph boundaries to avoid cutting sentences.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")  # Works for many models

def count_tokens(text):
    return len(enc.encode(text))

def chunk_chapter(chapter_text, max_tokens=4000, overlap_tokens=200):
    paragraphs = chapter_text.split('\n\n')
    chunks = []
    current_chunk = ""
    current_tokens = 0

    for para in paragraphs:
        para_tokens = count_tokens(para)
        # If a single paragraph exceeds max_tokens (rare for normal prose), hard split it
        if para_tokens > max_tokens:
            # Hard split by sentences if a paragraph is too long
            sentences = para.replace('\n', ' ').split('. ')
            temp = ""
            for sent in sentences:
                sent_tokens = count_tokens(sent)
                if current_tokens + sent_tokens > max_tokens:
                    chunks.append(current_chunk)
                    # Add overlap from previous chunk end
                    if chunks:
                        overlap_text = ' '.join(chunks[-1].split()[-overlap_tokens:])
                        current_chunk = overlap_text + " " + sent
                        current_tokens = count_tokens(current_chunk)
                    else:
                        current_chunk = sent
                        current_tokens = sent_tokens
                else:
                    current_chunk += " " + sent if current_chunk else sent
                    current_tokens += sent_tokens
            continue

        if current_tokens + para_tokens > max_tokens:
            # Finish current chunk
            chunks.append(current_chunk)
            # Start new chunk with overlap from previous chunk's tail
            overlap_text = ' '.join(current_chunk.split()[-overlap_tokens:])
            current_chunk = overlap_text + " " + para
            current_tokens = count_tokens(current_chunk)
        else:
            current_chunk += "\n\n" + para if current_chunk else para
            current_tokens += para_tokens

    if current_chunk:
        chunks.append(current_chunk)
    return chunks

The overlap ensures the model sees the last ~200 tokens of the previous chunk, which helps it maintain flow and avoid abrupt style changes. We tested different overlap sizes: 100 tokens was sometimes not enough for a sentence to complete; 500 tokens added cost without quality gain. 200 seemed the best compromise.

Carrying Context with a Dynamic Glossary

Overlap alone doesn't solve terminology consistency. A character introduced in chapter 1 as "Elara" might become "Elena" in chapter 20 because the model doesn't remember the earlier choice. So we maintain a glossary.

After translating each chunk, we run a lightweight extraction on the translation to pull out proper nouns and key terms. We use a simple heuristic with spaCy for named entity recognition, then ask the LLM to verify and suggest a consistent translation for each entity. The glossary is stored as a dictionary, and we inject it into the system prompt of every translation call.

import spacy
nlp = spacy.load("en_core_web_sm")  # or target language model

def extract_entities(text, lang='en'):
    doc = nlp(text)
    entities = set()
    for ent in doc.ents:
        if ent.label_ in ['PERSON', 'GPE', 'ORG', 'PRODUCT']:
            entities.add(ent.text)
    return entities

# After translating a chunk, we call this:
def update_glossary(translated_chunk, glossary):
    # Extract candidate entities from the translated text
    candidates = extract_entities(translated_chunk)
    # For each candidate, ask the LLM to provide a canonical translation
    # (we batch them in one prompt for efficiency)
    if candidates:
        prompt = f"Given these terms from a book: {list(candidates)}. Provide a consistent translation for each in the target language. Return JSON."
        # Call LLM (example with Anthropic)
        import anthropic
        client = anthropic.Anthropic()
        response = client.messages.create(
            model="claude-3-haiku-20240307",
            max_tokens=300,
            system="You are a terminology manager for book translation.",
            messages=[{"role": "user", "content": prompt}]
        )
        # Parse response and update glossary
        # (pseudo-code, actual parsing omitted for brevity)
        new_terms = parse_llm_json(response.content[0].text)
        glossary.update(new_terms)
    return glossary

The glossary is then included in the system prompt of the next chunk's translation:

def translate_chunk(chunk, glossary, target_lang='es', source_lang='en'):
    system_prompt = f"Translate the following text from {source_lang} to {target_lang}. Maintain the style of a novel. Use these translations for recurring terms: {glossary}"
    # Call LLM...

We also keep a context summary—a one-paragraph summary of the plot so far—that gets appended to the system prompt after every few chapters. We generate it periodically by feeding the LLM the last few translated chunks and asking for a summary. This summary acts as a high-level memory without bloating the prompt.

def summarize_progress(translated_chunks, max_tokens=1000):
    # Join last few chunks, ask for a summary
    recent = " ".join(translated_chunks[-5:])
    prompt = f"Summarize the key events, characters, and style of the following book excerpt in 3-4 sentences:\n{recent}"
    # Call LLM, return summary

Results, Costs, and Trade-offs

With this approach, we translated a 120,000-word novel to Spanish. Here are the concrete numbers:

• Total chunks: ~140 (average chunk size 3,800 tokens after overlap).
• Cost: Using Claude 3 Haiku for translation and glossary updates, total API cost was $4.20. Using Sonnet for final quality pass on selected chapters added another $3.50. So about $0.06 per 1,000 words—cheap enough for a subscription service.
• Time: The whole pipeline ran in under 20 minutes on a single VPS, including EPUB parsing, chunking, translation, glossary updates, and reassembly.
• Quality: Compared to our naive independent-chunk baseline, the glossary + overlap + summary improved consistency dramatically. We measured a 78% reduction in inconsistent term usage (based on manual review of 50 random name occurrences). The flow was also much smoother; beta readers reported fewer "jarring" transitions.

But there are trade-offs:

• Overlap costs extra tokens: The 200-token overlap adds about 5% more tokens per chunk. For very long books, that's not negligible.
• Complexity: The pipeline is more complex than a simple map-reduce. We had to handle edge cases like footnotes, images, and dialogue formatting. Our parser sometimes struggles with PDFs that have complex layouts; we recommend EPUB for best results.
• Glossary extraction is imperfect: spaCy misses some entities, and the LLM sometimes suggests inconsistent translations. We mitigated by having a human review step for premium users, but it's not perfect.
• Long-range dependencies: Even with a summary, the model can forget details from 50 chapters ago. We experimented with a vector store (using FAISS) to retrieve relevant paragraphs from earlier chapters, but it increased latency and cost without a clear quality boost for most books. We may revisit for non-fiction with heavy cross-references.

Lessons Learned

1. Respect the document's structure. Arbitrary token windows break paragraphs and chapters. Use natural boundaries.
2. A small overlap goes a long way. Just 200 tokens of previous context significantly improves cohesion.
3. Terminology management is non-negotiable. For any long-form translation, you need a glossary. It's the difference between a professional book and a garbled mess.
4. Iterate on chunk size. We tested 2k, 4k, 8k tokens. 4k gave the best trade-off between context and cost for our use case, but your mileage may vary.

What's Next?

We're working on better handling of cultural references and idioms—things that don't translate literally and need extra context. An open question for the community: How do you handle footnotes and endnotes in LLM-based translation? We currently strip them and reinsert manually, but that's clunky. Would love to hear ideas.