Translating 300-Page Books with Claude: Taming Token Limits and Chunking Strategies

How we built a reliable pipeline to split long texts for LLM translation without losing context or breaking the bank

At LectuLibre, we translate entire books using Claude. The challenge: a 300-page book is roughly 90,000–120,000 words, which translates to 120,000–160,000 tokens. While Claude 3 models have a 200k context window, sending an entire book in one API call is impractical. It's slow, expensive, and often degrades translation quality due to attention dilution. We needed a robust chunking strategy that preserved context and stayed within token limits.

The Problem: One Book, Too Many Tokens

When we first started building LectuLibre, we naively assumed we could just pass the whole book to Claude and get a translation back. We quickly hit three walls:

1. Rate limits: A single request with 150k tokens triggered API timeouts and 429 errors.
2. Cost: Even if it worked, processing 150k tokens per request with Opus would cost over $13 per book, and most of the input would be wasted on repeated context.
3. Quality: Long contexts tend to make the model "forget" early chapters, leading to inconsistent character names and terminology.

Clearly, chunking was necessary. But how do you split a book without losing narrative flow?

First Attempt: Naive Splitting by Paragraphs

Our initial approach was simple: split the text into chunks of roughly 10,000 tokens by paragraphs. We used a regex to split on double newlines and then concatenated paragraphs until we hit the token limit.

import re

def split_into_paragraphs(text: str) -> list[str]:
    return re.split(r'\n\s*\n', text)

def chunk_by_paragraphs(paragraphs: list[str], max_tokens: int = 10000) -> list[str]:
    chunks = []
    current_chunk = []
    current_tokens = 0
    for para in paragraphs:
        # Estimate tokens using character count / 4 (quick and dirty)
        para_tokens = len(para) // 4
        if current_tokens + para_tokens > max_tokens and current_chunk:
            chunks.append('\n\n'.join(current_chunk))
            current_chunk = []
            current_tokens = 0
        current_chunk.append(para)
        current_tokens += para_tokens
    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

This worked for small documents, but for books it produced broken sentences at chunk boundaries. A single sentence could be split across two chunks, causing the model to translate the halves differently. Worse, context from the previous chapter was completely lost, leading to inconsistent names and terminology.

Token Counting with Tiktoken

Before improving chunking, we needed accurate token counts. Character-based estimates were unreliable, especially for non-English languages. We switched to tiktoken, the tokenizer used by OpenAI and compatible with Claude's tokenizer.

import tiktoken

def count_tokens(text: str) -> int:
    encoding = tiktoken.get_encoding("cl100k_base")
    return len(encoding.encode(text))

Using tiktoken gave us exact token counts, which was crucial for staying within limits and estimating costs.

Chunking with Overlap and Context Windows

The naive split ignored context between chunks. We fixed this by introducing overlap—each chunk would include a few paragraphs from the previous chunk to provide continuity. Additionally, we added a preamble to each chunk containing the book's title, author, and a glossary of recurring terms.

def chunk_paragraphs_with_overlap(
    paragraphs: list[str],
    max_tokens: int = 10000,
    overlap_paragraphs: int = 2,
) -> list[str]:
    encoding = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for i, para in enumerate(paragraphs):
        para_tokens = len(encoding.encode(para))
        if current_tokens + para_tokens > max_tokens and current_chunk:
            # Finalize current chunk
            chunks.append('\n\n'.join(current_chunk))
            # Keep last few paragraphs for overlap
            overlap = current_chunk[-overlap_paragraphs:] if overlap_paragraphs > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(encoding.encode(p)) for p in current_chunk)
        current_chunk.append(para)
        current_tokens += para_tokens

    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

This simple overlap improved consistency at boundaries. However, we still had issues with sentence splits. A paragraph might be very long, and the token limit could cut it mid-sentence. We needed to split at sentence boundaries within paragraphs when necessary.

Respecting Sentence Boundaries

We used a lightweight sentence splitter (nltk.sent_tokenize was too slow for large books, so we used a regex-based splitter) to break paragraphs into sentences first, then chunk sentences with overlap. This ensured no sentence was ever split across chunks.

import re

def split_into_sentences(text: str) -> list[str]:
    # Simple regex for sentence boundaries (handles periods, exclamations, questions)
    sentences = re.split(r'(?<=[.!?])\s+', text)
    return [s.strip() for s in sentences if s.strip()]

def chunk_sentences_with_overlap(
    sentences: list[str],
    max_tokens: int = 10000,
    overlap_sentences: int = 3,
) -> list[str]:
    encoding = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for sent in sentences:
        sent_tokens = len(encoding.encode(sent))
        if current_tokens + sent_tokens > max_tokens and current_chunk:
            chunks.append(' '.join(current_chunk))
            overlap = current_chunk[-overlap_sentences:] if overlap_sentences > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(encoding.encode(s)) for s in current_chunk)
        current_chunk.append(sent)
        current_tokens += sent_tokens

    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

Now our chunks were mostly clean, with no broken sentences and enough overlap to keep context.

The Translation Pipeline

With chunking in place, we built an asynchronous pipeline to translate each chunk using the Anthropic Python client. We added retry logic with exponential backoff for rate limits and used a semaphore to limit concurrency.

import anthropic
import asyncio
import random

client = anthropic.AsyncAnthropic(api_key="YOUR_API_KEY")

async def translate_chunk(
    chunk: str,
    glossary: str,
    model: str = "claude-3-haiku-20240307",
    max_retries: int = 5,
) -> str:
    system_prompt = (
        "You are a professional book translator. "
        "Translate the given text to Spanish. "
        "Preserve formatting, tone, and style.\n\n"
        f"Glossary:\n{glossary}"
    )
    for attempt in range(max_retries):
        try:
            response = await client.messages.create(
                model=model,
                max_tokens=4096,
                temperature=0.2,
                system=system_prompt,
                messages=[{"role": "user", "content": chunk}],
            )
            return response.content[0].text
        except anthropic.RateLimitError:
            wait = 2 ** attempt + random.random()
            await asyncio.sleep(wait)
        except Exception as e:
            print(f"Error: {e}, retrying...")
            await asyncio.sleep(1)
    raise Exception("Max retries exceeded")

async def translate_book(chunks: list[str], glossary: str) -> str:
    semaphore = asyncio.Semaphore(3)  # Limit concurrent requests
    async def translate_one(chunk):
        async with semaphore:
            return await translate_chunk(chunk, glossary)
    tasks = [translate_one(chunk) for chunk in chunks]
    translated_chunks = await asyncio.gather(*tasks)
    return '\n\n'.join(translated_chunks)

We used claude-3-haiku for cost efficiency during development, but in production we allow users to choose their preferred model. Haiku is extremely fast and cheap, while Opus provides the highest quality for complex literary works.

Context Preservation: Glossaries and Style

Even with overlap, long-range consistency (e.g., a character introduced in chapter 1 appearing in chapter 20) was still a challenge. We addressed this by extracting a glossary of key terms and character names before translation.

We built a simple extraction step: we send a few sample chapters to Claude and ask it to list important terms, names, and their preferred translations. That glossary is then included in the system prompt of every chunk translation.

async def extract_glossary(sample_text: str) -> str:
    response = await client.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=1000,
        temperature=0.1,
        system="Extract a glossary of important terms, character names, and places from the text. Format as 'Original: Translation' pairs.",
        messages=[{"role": "user", "content": sample_text[:20000]}],
    )
    return response.content[0].text

This glossary reduced inconsistencies dramatically. For example, a character named "John" would always be translated as "Juan" instead of sometimes "Jean" or "Giovanni".

Results and Trade-offs

After implementing the full pipeline, here's what we observed for a typical 300-page novel (approx. 100k tokens):

• Chunk size: 8,000–12,000 tokens per chunk worked best. Smaller chunks lost too much context; larger chunks approached the limits of quality degradation.
• Overlap: 300–500 tokens (2–3 sentences) of overlap virtually eliminated boundary artifacts.
• Cost: Translating with Haiku cost about $0.15 per book, Sonnet about $1.50, and Opus about $9.00. This is a fraction of what a single full-context call would cost, and quality is much better.
• Speed: With concurrency of 3, a 100k token book translated in under 2 minutes using Haiku, and about 5 minutes with Sonnet. A single full-context call would often time out.
• Failure: We initially tried using Claude's long context to translate an entire book in one go. Not only did we hit API timeouts, but the translation quality dropped noticeably after the first 10k tokens—the model would start skipping paragraphs and mixing up characters.

One trade-off we accepted: the overlap means we translate some text twice. That's a small redundancy (about 5% extra tokens) but worth it for consistency.

Lessons Learned and Next Steps

• Always count tokens accurately with tiktoken—don't guess.
• Overlap is a cheap insurance policy against context loss.
• Glossaries are essential for long-form translation; they keep terminology consistent across chunks.
• Respect rate limits with exponential backoff and concurrency limits.

We're still exploring how to handle cross-chapter references more gracefully. One idea: generate a summary of each chapter as it's translated and include that summary in subsequent chunks, effectively building a "memory" for the model. We'd love to hear how others have tackled this.

Open question: How do you preserve narrative voice and style across hundreds of chunks? We've found temperature and prompt engineering help, but there's room for improvement. Share your thoughts in the comments!