Handling Token Limits When Translating 300-Page Books with Claude

How we chunked long-form content, preserved context, and managed output limits to build a reliable book translation pipeline.

When we started building LectuLibre, our AI-powered book translation service, we knew the core challenge wouldn't be the translation quality itself—it would be feeding an entire 300-page book to a large language model without hitting token limits.

Claude's context window is huge (200k tokens for Claude 3 Opus), but that's still not enough for many full-length books. And even if it were, the output token limit—the maximum number of tokens the model can generate in a single response—is far smaller (4,096 tokens for Claude 3 Sonnet and Opus). If you ask Claude to translate an entire book, it will happily start and then abruptly stop mid-sentence once it hits that output ceiling.

Here's how we solved it with a chunking strategy that preserves context, respects output limits, and still delivers a coherent final translation.

The Problem: Long Text vs. Hard Limits

A typical 300-page novel contains about 80,000–90,000 words. In token terms, that's roughly 100,000–130,000 tokens. While that fits inside Claude's input context, the model's output limit means it can only produce a tiny fraction of that in one go. Even if you tried a streaming approach, the max output tokens remains a hard stop.

Our first naive attempt was to send a whole chapter (maybe 20–30 pages) at once. That failed for two reasons:

1. Input token overflow on some chapters that were longer than expected (especially with PDF extraction producing messy whitespace).
2. Output truncation — the model would translate beautifully for a few paragraphs, then stop because it reached the 4,096-token output limit.

We needed a way to split the book into pieces small enough that the model could translate each piece completely, but large enough to maintain context and coherence.

Our Approach: Paragraph-Aware Chunking with Context Overlap

The core idea is simple: split the book into chunks based on paragraph boundaries and token counts, translate each chunk separately, and stitch the results back together. But the devil is in the details.

We chose paragraph-level chunking rather than splitting by fixed token size or sentence count because it respects the natural rhythm of prose. Splitting mid-paragraph often leads to awkward translations or duplicated context.

We also added a context overlap — each chunk includes the last few sentences of the previous chunk as untranslated context. This helps the model maintain narrative flow and avoid repeating introductions or forgetting character names.

Finally, we used a running glossary of key terms (character names, places, technical jargon) that gets passed to the model with each chunk. This ensures consistency across chunks—something that pure overlap alone doesn't guarantee.

Implementation Details

1. Token Counting with the Anthropic SDK

We use the official anthropic Python SDK. It provides a convenient count_tokens method that gives us an accurate token count for a given text.

import anthropic

client = anthropic.Anthropic(api_key="YOUR_API_KEY")

def count_tokens(text: str) -> int:
    """Return token count for a text using Claude's tokenizer."""
    response = client.count_tokens(text)
    return response.tokens

This is more accurate than using OpenAI's tiktoken approximation, and it only costs a tiny fraction of a penny per call. For chunking decisions, precision matters because we want to get as close to the input limit as possible without exceeding it.

2. Chunking Function

The chunker takes a list of paragraphs (extracted from EPUB or PDF) and returns a list of chunks. Each chunk is a tuple: (context, text_to_translate). The context is the last N sentences from the previous chunk (if any), and it is not translated—it's only there to give the model context.

from typing import List, Tuple

MAX_INPUT_TOKENS = 3000   # conservative limit to stay under output limit
                         # and leave room for system prompt + context
CONTEXT_SENTENCES = 3     # number of previous sentences to include as context

def chunk_paragraphs(
    paragraphs: List[str],
    max_tokens: int = MAX_INPUT_TOKENS,
    context_sentences: int = CONTEXT_SENTENCES
) -> List[Tuple[str, str]]:
    chunks = []
    current_text = ""
    current_tokens = 0
    previous_context = ""

    for para in paragraphs:
        para_tokens = count_tokens(para)

        # If a single paragraph is too large, split it further (rare)
        if para_tokens > max_tokens:
            # Fallback: split by sentences
            sentences = re.split(r'(?<=[.!?]) +', para)
            for sent in sentences:
                sent_tokens = count_tokens(sent)
                if current_tokens + sent_tokens > max_tokens:
                    chunks.append((previous_context, current_text.strip()))
                    # Update context for next chunk
                    previous_context = " ".join(
                        current_text.split()[-context_sentences*10:]  # rough sentence proxy
                    )
                    current_text = sent
                    current_tokens = sent_tokens
                else:
                    current_text += " " + sent
                    current_tokens += sent_tokens
            continue

        if current_tokens + para_tokens > max_tokens:
            # Finalize current chunk
            chunks.append((previous_context, current_text.strip()))
            # Update context for next chunk
            previous_context = " ".join(current_text.split()[-context_sentences*10:])
            current_text = para
            current_tokens = para_tokens
        else:
            current_text += " " + para
            current_tokens += para_tokens

    if current_text:
        chunks.append((previous_context, current_text.strip()))

    return chunks

This code is simplified—in production we also handle special cases like chapter boundaries (prefer to start a new chunk at a chapter heading) and avoid splitting code blocks or poetry.

3. Translation Loop with Streaming and Retries

For each chunk, we call the Claude API with a system prompt that includes translation instructions and our running glossary. We use streaming to show progress to the user in real time, and we implement exponential backoff for rate limit errors.

import asyncio
from anthropic import AsyncAnthropic

client = AsyncAnthropic(api_key="YOUR_API_KEY")

SYSTEM_PROMPT_TEMPLATE = """
You are a professional book translator. Translate the following text from {source_lang} to {target_lang}.

Rules:
- Translate only the text after the marker "---TEXT---".
- The text before "---TEXT---" is context from the previous chunk. Do NOT translate it.
- Maintain the original formatting (paragraphs, line breaks).
- Use the glossary below for consistent terminology.

Glossary:
{glossary}
"""

async def translate_chunk(
    context: str,
    text: str,
    glossary: dict,
    source_lang: str,
    target_lang: str,
    model: str = "claude-3-sonnet-20240229"
) -> str:
    prompt = SYSTEM_PROMPT_TEMPLATE.format(
        source_lang=source_lang,
        target_lang=target_lang,
        glossary="\n".join(f"{k}: {v}" for k, v in glossary.items())
    )
    messages = [
        {"role": "user", "content": f"{context}\n---TEXT---\n{text}"}
    ]

    translated = ""
    max_retries = 5
    for attempt in range(max_retries):
        try:
            async with client.messages.stream(
                model=model,
                max_tokens=4096,  # max output tokens
                system=prompt,
                messages=messages
            ) as stream:
                async for event in stream:
                    if event.type == "content_block_delta":
                        translated += event.delta.text
            break
        except anthropic.RateLimitError:
            await asyncio.sleep(2 ** attempt)
        except anthropic.APIError as e:
            print(f"API error: {e}, retrying...")
            await asyncio.sleep(2 ** attempt)
    else:
        raise Exception("Max retries exceeded")

    return translated.strip()

In production, we also update the glossary after each chunk by asking the model for a list of new terms it encountered, but that's beyond the scope of this article.

4. Stitching It All Together

The main loop walks through the chunks, translates each one, and collects the outputs. We store the final translation in a database (PostgreSQL) and also allow the user to download as EPUB or PDF.

async def translate_book(paragraphs: List[str], source_lang: str, target_lang: str):
    chunks = chunk_paragraphs(paragraphs)
    glossary = {}
    translated_parts = []

    for i, (context, text) in enumerate(chunks):
        print(f"Translating chunk {i+1}/{len(chunks)}...")
        translated = await translate_chunk(
            context, text, glossary, source_lang, target_lang
        )
        translated_parts.append(translated)

        # Optional: update glossary from translated text
        # (we use a separate call to extract new terms)

    return "\n\n".join(translated_parts)

Results and Lessons Learned

After implementing this pipeline, we successfully translated several 300-page books from English to Spanish and French. Here are some concrete numbers for a typical novel:

• Book size: ~85,000 words, ~110,000 tokens extracted
• Number of chunks: 38 (average chunk size ~2,800 tokens)
• Total processing time: ~8 minutes with sequential streaming (using Claude 3 Sonnet)
• Cost: approximately $3.20 per book (Sonnet pricing)
• Translation quality: significantly better than our earlier attempts without overlap and glossary

What went wrong initially:

1. No context overlap → the model would translate each chunk as if it were a standalone text, leading to inconsistent character names and repeated exposition. Adding just 3–5 sentences of previous context fixed most of this.
2. Ignoring output token limit → we sized chunks to fit the input context (200k) but output would truncate. The realization that output tokens are usually the binding constraint was a turning point.
3. Unbounded retries → we initially retried on every error without backoff, causing us to hit rate limits hard. Exponential backoff solved it.

Trade-offs:

• Smaller chunks → safer output, but more API calls (cost and latency) and potential loss of long-range context.
• Larger chunks → better context, but higher risk of hitting output limits or truncation, especially for languages that expand (e.g., German often produces 20–30% more tokens).
• We settled on a dynamic chunk size based on the source language's expansion ratio—for languages like Spanish (which expands slightly) we keep input tokens around 2,500; for Japanese (which can shrink) we go up to 3,500.

Final Thoughts and an Open Question

The key takeaway for anyone building long-form text processing with LLMs is: token limits are not just about input size—output limits often matter more, and chunking must be context-aware. A naive split by token count works for summarization but fails for translation, where narrative continuity is crucial.

We're still exploring better ways to maintain consistency across chunks. One idea is to use Claude's prompt caching (introduced in beta) to cache the system prompt and glossary, reducing latency and cost for subsequent chunks. Another is to use a vector database to retrieve relevant context from previous chapters instead of a simple fixed overlap.

Open question for the community: How do you handle languages with extreme expansion ratios? We've experimented with asking the model to summarize the chunk before translating, but that adds latency. Any better ideas?

---

LectuLibre is an AI-powered book translation service that lets users upload EPUB or PDF files and receive a fully translated book. The backend is Python/FastAPI with PostgreSQL, and it runs on a simple VPS. This article reflects our real-world experience building the translation pipeline.