Chunking 300-Page Books for Claude: How We Beat Token Limits in AI Translation

A practical guide to slicing long documents, preserving context, and avoiding context-window explosions.

When we first set out to build LectuLibre, our AI-powered book translation service, the promise was simple: upload an EPUB or PDF, and get a professionally translated book back. The reality? A 300-page book contains roughly 90,000 words—around 120,000 tokens—and while Claude's 200K context window can hold that, doing so in a single API call is expensive, slow, and often produces lower-quality translations because the model loses focus.

So we had to answer a fundamental question: how do you translate a 300-page book with an LLM without blowing up the context window or your budget?

Here's how we solved it—the chunking strategy, the code, the trade-offs, and the lessons we learned the hard way.

The Problem: Long Texts vs. Context Windows

Claude 3.5 Sonnet offers a 200,000-token context window. That sounds like a lot—it is—but for a complete book, it's still not enough. And even if it were, there are three practical reasons we avoid sending an entire book in one prompt:

1. Cost: Token-based pricing means a single call with 120k tokens would cost several dollars. Doing that for every book translation is unsustainable.
2. Latency: A 120k-token input takes minutes to process. Users expect a translation in minutes, not hours.
3. Quality: LLMs tend to "skim" when given extremely long inputs. We found that translations of later chapters suffered from dropped details and inconsistent terminology.

So chunking was mandatory. But naive chunking—splitting by fixed character count or by paragraph without context—led to another problem: broken coherence. Sentences that referenced previous paragraphs, character names, or plot points became disjointed. We needed a chunking approach that preserved enough context for the model to translate accurately while keeping each chunk small enough to be fast and cheap.

Our Approach: Overlapping Window Chunking with Continuation Memory

We chose a hybrid strategy:

• Split the book into semantic units (paragraphs) using a lightweight parser.
• Group paragraphs into chunks of approximately 4,000 tokens each, with a 500-token overlap from the previous chunk.
• Before translating each chunk, prepend a continuation memory—a short summary of the previous chunk's translation, generated automatically by the model itself.
• Use tiktoken (the same tokenizer OpenAI uses, which is compatible with Claude's token counting for English text) to estimate token counts accurately.

Why 4,000 tokens? It's small enough to keep latency low (typically under 10 seconds per chunk) and cost manageable, while still providing enough context for coherent paragraphs. The 500-token overlap ensures that sentences split at chunk boundaries can be resolved with the missing context. The continuation memory helps maintain consistency across chunks, especially for character names and specialized terminology.

Implementation Details: The Python Code

Our backend is Python/FastAPI, and we use the official anthropic Python SDK. For PDF/EPUB parsing, we use pymupdf for PDFs and ebooklib for EPUBs, extracting text into a list of paragraphs. Here's the core chunking function:

import tiktoken
from typing import List, Tuple

def chunk_paragraphs(
    paragraphs: List[str],
    max_tokens: int = 4000,
    overlap_tokens: int = 500
) -> List[Tuple[int, int, str]]:
    """
    Group paragraphs into chunks with token-based limits and overlap.
    Returns list of (start_index, end_index, chunk_text).
    """
    enc = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0
    start_idx = 0

    for i, para in enumerate(paragraphs):
        para_tokens = len(enc.encode(para))

        # If adding this paragraph exceeds max_tokens, finalize current chunk
        if current_tokens + para_tokens > max_tokens and current_chunk:
            # Save current chunk
            chunk_text = "\n\n".join(current_chunk)
            chunks.append((start_idx, i - 1, chunk_text))

            # Start new chunk with overlap from the end of previous chunk
            overlap_start = max(start_idx, i - overlap_tokens)
            overlap_paragraphs = paragraphs[overlap_start:i]
            current_chunk = overlap_paragraphs.copy()
            current_tokens = sum(len(enc.encode(p)) for p in current_chunk)
            start_idx = overlap_start

        current_chunk.append(para)
        current_tokens += para_tokens

    # Add the last chunk
    if current_chunk:
        chunk_text = "\n\n".join(current_chunk)
        chunks.append((start_idx, len(paragraphs) - 1, chunk_text))

    return chunks

A few notes on this code:

• We use cl100k_base encoding, which is the tokenizer for GPT-4 and also works well for Claude because both models use similar byte-pair encoding. It's not perfect for all languages, but for English source text it's accurate enough.
• Overlap is implemented by keeping the last overlap_tokens worth of paragraphs from the previous chunk at the beginning of the new chunk. This means some paragraphs are translated twice, but the second translation has the benefit of context from the first.
• The function returns indices so we can later assemble the final translation in order, discarding duplicate translations from overlap sections.

Next, we handle the actual translation call. We use a function that takes a chunk, the continuation memory, and returns the translated text, with retry logic for API errors:

import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential

client = anthropic.Anthropic()

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def translate_chunk(chunk_text: str, memory: str, target_lang: str) -> str:
    """Translate a single chunk using Claude, with continuation memory."""
    prompt = f"""You are a professional book translator. Translate the following text into {target_lang}.

Continuation context from previous section (for consistency):
{memory}

Text to translate:
{chunk_text}

Translation (only the translated text, no explanations):"""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        temperature=0.3,
        system="You are a literary translator. Preserve tone, style, and terminology. Keep names and specialized terms consistent.",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

After each chunk is translated, we generate a brief summary of that translation to serve as memory for the next chunk. This is done by a lightweight call to Claude with the translated text:

def generate_memory(translated_text: str) -> str:
    """Generate a short summary of key terms and style from translated text."""
    prompt = f"""Summarize the following translated text in 3-5 sentences, focusing on character names, specialized terminology, and stylistic choices. Keep it concise.

{translated_text}

Summary:"""
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",  # cheaper model for summarization
        max_tokens=500,
        temperature=0.0,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

We use claude-3-5-haiku for memory generation because it's fast and cheap, and the summarization doesn't require the full power of Sonnet.

The Assembly Problem: Merging Chunks Without Duplicates

Because of the overlap, we end up translating some paragraphs twice. To assemble the final book, we need to align the translated chunks back to the original paragraph order. Our chunk_paragraphs function returned start and end indices, but those are based on the original paragraph list, and the overlap means the same paragraph may appear in multiple chunks.

Our solution: after translating each chunk, we split the translated text back into paragraphs (using newlines as delimiters) and store them in a dictionary keyed by the original paragraph index. For paragraphs that appear in multiple chunks, we keep only the translation from the chunk where the paragraph is not part of the overlap (i.e., the chunk where it appears as new content). This ensures we use the translation that had the most context.

Here's a simplified version of the assembly logic:

def assemble_translation(chunks, translated_chunks, total_paragraphs):
    """Merge translated chunks into final paragraph list."""
    final = [None] * total_paragraphs
    for (start, end, _), trans_text in zip(chunks, translated_chunks):
        trans_paragraphs = trans_text.split("\n\n")
        # Determine which paragraphs are new (not overlap) by mapping to original indices
        # We assume paragraphs in chunk correspond to indices start..end
        # But due to overlap, some are duplicates; we only fill missing ones
        for i, para in enumerate(trans_paragraphs):
            orig_idx = start + i
            if orig_idx <= end and final[orig_idx] is None:
                final[orig_idx] = para
    # Fill any remaining None with empty string (shouldn't happen)
    return "\n\n".join(p for p in final if p is not None)

This assumes the translation preserves paragraph boundaries, which is mostly true for Claude with our prompt. If not, we have a fallback that uses regex to split on blank lines.

Performance and Cost Numbers

After implementing this system, we translated a 320-page novel (about 98,000 words). Here are the real numbers:

• Total tokens processed: ~140,000 input tokens across all chunks (including overlaps and memory prompts), ~110,000 output tokens.
• Number of chunks: 42 chunks (average ~3,300 tokens per chunk after trimming).
• Total cost: Approximately $1.20 using Claude 3.5 Sonnet for translation and Haiku for memory. A single-call approach with full context would have cost roughly $5-7 and taken much longer.
• Total translation time: 12 minutes end-to-end, including API latency and post-processing.
• Quality: The continuation memory eliminated most terminology inconsistencies. We saw a 90% reduction in name variations compared to naive chunking without overlap or memory.

Lessons Learned and Trade-offs

1. Token counting is never exact. tiktoken works great for English, but we had to adjust for languages with different token densities. We added a 10% safety margin on chunk sizes to avoid exceeding Claude's per-request token limit.

2. Overlap size matters. Too little overlap (e.g., 100 tokens) still caused broken context; too much overlap (2000 tokens) increased cost and duplicate work. 500 tokens (about 1-2 paragraphs) was the sweet spot for book text.

3. Continuation memory is a game-changer. Without it, the model sometimes translated the same character name differently across chapters. With a simple summary of key terms, consistency improved dramatically. We considered using a vector database for long-term memory, but for a single book, a rolling summary is sufficient and simpler.

4. Footnotes and special formatting are tricky. Books with heavy formatting (code blocks, poetry, tables) broke our paragraph-based splitting. We had to implement a pre-processing step that extracts and protects these elements, then re-inserts them after translation.

5. The assembly step can be error-prone. If the model merges or splits paragraphs unexpectedly, the index mapping fails. We added a validation step that checks the number of translated paragraphs matches the expected count and flags discrepancies for manual review.

Open Question for the Community

We're still exploring better ways to maintain global consistency across an entire book without drastically increasing cost. Some ideas we're considering:

• Using a two-pass approach: first pass generates a glossary and style guide from the full book (using a cheaper model on summaries), second pass translates chunks with that glossary as part of the system prompt.
• Experimenting with Claude's prompt caching to reuse the continuation memory across chunks, reducing input tokens.
• For books with many recurring characters, perhaps a knowledge graph updated as translation progresses.

How have you handled long-form text with LLMs? Any tips for maintaining consistency across thousands of tokens? We'd love to hear your approaches in the comments.

If you're building something similar, our main takeaway is this: chunking is necessary, but how you chunk and what context you carry between chunks determines the quality of the final output. Overlap + memory is a simple but effective pattern that works well for book-length documents.