DEV Community: 龚旭东

Translating 300-Page Books with Claude: Taming Token Limits and Chunking Strategies

龚旭东 — Sat, 05 Sep 2026 03:02:12 +0000

How we built a reliable pipeline to split long texts for LLM translation without losing context or breaking the bank

At LectuLibre, we translate entire books using Claude. The challenge: a 300-page book is roughly 90,000–120,000 words, which translates to 120,000–160,000 tokens. While Claude 3 models have a 200k context window, sending an entire book in one API call is impractical. It's slow, expensive, and often degrades translation quality due to attention dilution. We needed a robust chunking strategy that preserved context and stayed within token limits.

The Problem: One Book, Too Many Tokens

When we first started building LectuLibre, we naively assumed we could just pass the whole book to Claude and get a translation back. We quickly hit three walls:

Rate limits: A single request with 150k tokens triggered API timeouts and 429 errors.
Cost: Even if it worked, processing 150k tokens per request with Opus would cost over $13 per book, and most of the input would be wasted on repeated context.
Quality: Long contexts tend to make the model "forget" early chapters, leading to inconsistent character names and terminology.

Clearly, chunking was necessary. But how do you split a book without losing narrative flow?

First Attempt: Naive Splitting by Paragraphs

Our initial approach was simple: split the text into chunks of roughly 10,000 tokens by paragraphs. We used a regex to split on double newlines and then concatenated paragraphs until we hit the token limit.

import re

def split_into_paragraphs(text: str) -> list[str]:
    return re.split(r'\n\s*\n', text)

def chunk_by_paragraphs(paragraphs: list[str], max_tokens: int = 10000) -> list[str]:
    chunks = []
    current_chunk = []
    current_tokens = 0
    for para in paragraphs:
        # Estimate tokens using character count / 4 (quick and dirty)
        para_tokens = len(para) // 4
        if current_tokens + para_tokens > max_tokens and current_chunk:
            chunks.append('\n\n'.join(current_chunk))
            current_chunk = []
            current_tokens = 0
        current_chunk.append(para)
        current_tokens += para_tokens
    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

This worked for small documents, but for books it produced broken sentences at chunk boundaries. A single sentence could be split across two chunks, causing the model to translate the halves differently. Worse, context from the previous chapter was completely lost, leading to inconsistent names and terminology.

Token Counting with Tiktoken

Before improving chunking, we needed accurate token counts. Character-based estimates were unreliable, especially for non-English languages. We switched to tiktoken, the tokenizer used by OpenAI and compatible with Claude's tokenizer.

import tiktoken

def count_tokens(text: str) -> int:
    encoding = tiktoken.get_encoding("cl100k_base")
    return len(encoding.encode(text))

Using tiktoken gave us exact token counts, which was crucial for staying within limits and estimating costs.

Chunking with Overlap and Context Windows

The naive split ignored context between chunks. We fixed this by introducing overlap—each chunk would include a few paragraphs from the previous chunk to provide continuity. Additionally, we added a preamble to each chunk containing the book's title, author, and a glossary of recurring terms.

def chunk_paragraphs_with_overlap(
    paragraphs: list[str],
    max_tokens: int = 10000,
    overlap_paragraphs: int = 2,
) -> list[str]:
    encoding = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for i, para in enumerate(paragraphs):
        para_tokens = len(encoding.encode(para))
        if current_tokens + para_tokens > max_tokens and current_chunk:
            # Finalize current chunk
            chunks.append('\n\n'.join(current_chunk))
            # Keep last few paragraphs for overlap
            overlap = current_chunk[-overlap_paragraphs:] if overlap_paragraphs > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(encoding.encode(p)) for p in current_chunk)
        current_chunk.append(para)
        current_tokens += para_tokens

    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

This simple overlap improved consistency at boundaries. However, we still had issues with sentence splits. A paragraph might be very long, and the token limit could cut it mid-sentence. We needed to split at sentence boundaries within paragraphs when necessary.

Respecting Sentence Boundaries

We used a lightweight sentence splitter (nltk.sent_tokenize was too slow for large books, so we used a regex-based splitter) to break paragraphs into sentences first, then chunk sentences with overlap. This ensured no sentence was ever split across chunks.

import re

def split_into_sentences(text: str) -> list[str]:
    # Simple regex for sentence boundaries (handles periods, exclamations, questions)
    sentences = re.split(r'(?<=[.!?])\s+', text)
    return [s.strip() for s in sentences if s.strip()]

def chunk_sentences_with_overlap(
    sentences: list[str],
    max_tokens: int = 10000,
    overlap_sentences: int = 3,
) -> list[str]:
    encoding = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for sent in sentences:
        sent_tokens = len(encoding.encode(sent))
        if current_tokens + sent_tokens > max_tokens and current_chunk:
            chunks.append(' '.join(current_chunk))
            overlap = current_chunk[-overlap_sentences:] if overlap_sentences > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(encoding.encode(s)) for s in current_chunk)
        current_chunk.append(sent)
        current_tokens += sent_tokens

    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

Now our chunks were mostly clean, with no broken sentences and enough overlap to keep context.

The Translation Pipeline

With chunking in place, we built an asynchronous pipeline to translate each chunk using the Anthropic Python client. We added retry logic with exponential backoff for rate limits and used a semaphore to limit concurrency.

import anthropic
import asyncio
import random

client = anthropic.AsyncAnthropic(api_key="YOUR_API_KEY")

async def translate_chunk(
    chunk: str,
    glossary: str,
    model: str = "claude-3-haiku-20240307",
    max_retries: int = 5,
) -> str:
    system_prompt = (
        "You are a professional book translator. "
        "Translate the given text to Spanish. "
        "Preserve formatting, tone, and style.\n\n"
        f"Glossary:\n{glossary}"
    )
    for attempt in range(max_retries):
        try:
            response = await client.messages.create(
                model=model,
                max_tokens=4096,
                temperature=0.2,
                system=system_prompt,
                messages=[{"role": "user", "content": chunk}],
            )
            return response.content[0].text
        except anthropic.RateLimitError:
            wait = 2 ** attempt + random.random()
            await asyncio.sleep(wait)
        except Exception as e:
            print(f"Error: {e}, retrying...")
            await asyncio.sleep(1)
    raise Exception("Max retries exceeded")

async def translate_book(chunks: list[str], glossary: str) -> str:
    semaphore = asyncio.Semaphore(3)  # Limit concurrent requests
    async def translate_one(chunk):
        async with semaphore:
            return await translate_chunk(chunk, glossary)
    tasks = [translate_one(chunk) for chunk in chunks]
    translated_chunks = await asyncio.gather(*tasks)
    return '\n\n'.join(translated_chunks)

We used claude-3-haiku for cost efficiency during development, but in production we allow users to choose their preferred model. Haiku is extremely fast and cheap, while Opus provides the highest quality for complex literary works.

Context Preservation: Glossaries and Style

Even with overlap, long-range consistency (e.g., a character introduced in chapter 1 appearing in chapter 20) was still a challenge. We addressed this by extracting a glossary of key terms and character names before translation.

We built a simple extraction step: we send a few sample chapters to Claude and ask it to list important terms, names, and their preferred translations. That glossary is then included in the system prompt of every chunk translation.

async def extract_glossary(sample_text: str) -> str:
    response = await client.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=1000,
        temperature=0.1,
        system="Extract a glossary of important terms, character names, and places from the text. Format as 'Original: Translation' pairs.",
        messages=[{"role": "user", "content": sample_text[:20000]}],
    )
    return response.content[0].text

This glossary reduced inconsistencies dramatically. For example, a character named "John" would always be translated as "Juan" instead of sometimes "Jean" or "Giovanni".

Results and Trade-offs

After implementing the full pipeline, here's what we observed for a typical 300-page novel (approx. 100k tokens):

Chunk size: 8,000–12,000 tokens per chunk worked best. Smaller chunks lost too much context; larger chunks approached the limits of quality degradation.
Overlap: 300–500 tokens (2–3 sentences) of overlap virtually eliminated boundary artifacts.
Cost: Translating with Haiku cost about $0.15 per book, Sonnet about $1.50, and Opus about $9.00. This is a fraction of what a single full-context call would cost, and quality is much better.
Speed: With concurrency of 3, a 100k token book translated in under 2 minutes using Haiku, and about 5 minutes with Sonnet. A single full-context call would often time out.
Failure: We initially tried using Claude's long context to translate an entire book in one go. Not only did we hit API timeouts, but the translation quality dropped noticeably after the first 10k tokens—the model would start skipping paragraphs and mixing up characters.

One trade-off we accepted: the overlap means we translate some text twice. That's a small redundancy (about 5% extra tokens) but worth it for consistency.

Lessons Learned and Next Steps

Always count tokens accurately with tiktoken—don't guess.
Overlap is a cheap insurance policy against context loss.
Glossaries are essential for long-form translation; they keep terminology consistent across chunks.
Respect rate limits with exponential backoff and concurrency limits.

We're still exploring how to handle cross-chapter references more gracefully. One idea: generate a summary of each chapter as it's translated and include that summary in subsequent chunks, effectively building a "memory" for the model. We'd love to hear how others have tackled this.

Open question: How do you preserve narrative voice and style across hundreds of chunks? We've found temperature and prompt engineering help, but there's room for improvement. Share your thoughts in the comments!

Chunking 300-Page Books for Claude: How We Beat Token Limits in AI Translation

龚旭东 — Wed, 02 Sep 2026 03:01:56 +0000

A practical guide to slicing long documents, preserving context, and avoiding context-window explosions.

When we first set out to build LectuLibre, our AI-powered book translation service, the promise was simple: upload an EPUB or PDF, and get a professionally translated book back. The reality? A 300-page book contains roughly 90,000 words—around 120,000 tokens—and while Claude's 200K context window can hold that, doing so in a single API call is expensive, slow, and often produces lower-quality translations because the model loses focus.

So we had to answer a fundamental question: how do you translate a 300-page book with an LLM without blowing up the context window or your budget?

Here's how we solved it—the chunking strategy, the code, the trade-offs, and the lessons we learned the hard way.

The Problem: Long Texts vs. Context Windows

Claude 3.5 Sonnet offers a 200,000-token context window. That sounds like a lot—it is—but for a complete book, it's still not enough. And even if it were, there are three practical reasons we avoid sending an entire book in one prompt:

Cost: Token-based pricing means a single call with 120k tokens would cost several dollars. Doing that for every book translation is unsustainable.
Latency: A 120k-token input takes minutes to process. Users expect a translation in minutes, not hours.
Quality: LLMs tend to "skim" when given extremely long inputs. We found that translations of later chapters suffered from dropped details and inconsistent terminology.

So chunking was mandatory. But naive chunking—splitting by fixed character count or by paragraph without context—led to another problem: broken coherence. Sentences that referenced previous paragraphs, character names, or plot points became disjointed. We needed a chunking approach that preserved enough context for the model to translate accurately while keeping each chunk small enough to be fast and cheap.

Our Approach: Overlapping Window Chunking with Continuation Memory

We chose a hybrid strategy:

Split the book into semantic units (paragraphs) using a lightweight parser.
Group paragraphs into chunks of approximately 4,000 tokens each, with a 500-token overlap from the previous chunk.
Before translating each chunk, prepend a continuation memory—a short summary of the previous chunk's translation, generated automatically by the model itself.
Use tiktoken (the same tokenizer OpenAI uses, which is compatible with Claude's token counting for English text) to estimate token counts accurately.

Why 4,000 tokens? It's small enough to keep latency low (typically under 10 seconds per chunk) and cost manageable, while still providing enough context for coherent paragraphs. The 500-token overlap ensures that sentences split at chunk boundaries can be resolved with the missing context. The continuation memory helps maintain consistency across chunks, especially for character names and specialized terminology.

Implementation Details: The Python Code

Our backend is Python/FastAPI, and we use the official anthropic Python SDK. For PDF/EPUB parsing, we use pymupdf for PDFs and ebooklib for EPUBs, extracting text into a list of paragraphs. Here's the core chunking function:

import tiktoken
from typing import List, Tuple

def chunk_paragraphs(
    paragraphs: List[str],
    max_tokens: int = 4000,
    overlap_tokens: int = 500
) -> List[Tuple[int, int, str]]:
    """
    Group paragraphs into chunks with token-based limits and overlap.
    Returns list of (start_index, end_index, chunk_text).
    """
    enc = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0
    start_idx = 0

    for i, para in enumerate(paragraphs):
        para_tokens = len(enc.encode(para))

        # If adding this paragraph exceeds max_tokens, finalize current chunk
        if current_tokens + para_tokens > max_tokens and current_chunk:
            # Save current chunk
            chunk_text = "\n\n".join(current_chunk)
            chunks.append((start_idx, i - 1, chunk_text))

            # Start new chunk with overlap from the end of previous chunk
            overlap_start = max(start_idx, i - overlap_tokens)
            overlap_paragraphs = paragraphs[overlap_start:i]
            current_chunk = overlap_paragraphs.copy()
            current_tokens = sum(len(enc.encode(p)) for p in current_chunk)
            start_idx = overlap_start

        current_chunk.append(para)
        current_tokens += para_tokens

    # Add the last chunk
    if current_chunk:
        chunk_text = "\n\n".join(current_chunk)
        chunks.append((start_idx, len(paragraphs) - 1, chunk_text))

    return chunks

A few notes on this code:

We use cl100k_base encoding, which is the tokenizer for GPT-4 and also works well for Claude because both models use similar byte-pair encoding. It's not perfect for all languages, but for English source text it's accurate enough.
Overlap is implemented by keeping the last overlap_tokens worth of paragraphs from the previous chunk at the beginning of the new chunk. This means some paragraphs are translated twice, but the second translation has the benefit of context from the first.
The function returns indices so we can later assemble the final translation in order, discarding duplicate translations from overlap sections.

Next, we handle the actual translation call. We use a function that takes a chunk, the continuation memory, and returns the translated text, with retry logic for API errors:

import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential

client = anthropic.Anthropic()

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def translate_chunk(chunk_text: str, memory: str, target_lang: str) -> str:
    """Translate a single chunk using Claude, with continuation memory."""
    prompt = f"""You are a professional book translator. Translate the following text into {target_lang}.

Continuation context from previous section (for consistency):
{memory}

Text to translate:
{chunk_text}

Translation (only the translated text, no explanations):"""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        temperature=0.3,
        system="You are a literary translator. Preserve tone, style, and terminology. Keep names and specialized terms consistent.",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

After each chunk is translated, we generate a brief summary of that translation to serve as memory for the next chunk. This is done by a lightweight call to Claude with the translated text:

def generate_memory(translated_text: str) -> str:
    """Generate a short summary of key terms and style from translated text."""
    prompt = f"""Summarize the following translated text in 3-5 sentences, focusing on character names, specialized terminology, and stylistic choices. Keep it concise.

{translated_text}

Summary:"""
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",  # cheaper model for summarization
        max_tokens=500,
        temperature=0.0,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

We use claude-3-5-haiku for memory generation because it's fast and cheap, and the summarization doesn't require the full power of Sonnet.

The Assembly Problem: Merging Chunks Without Duplicates

Because of the overlap, we end up translating some paragraphs twice. To assemble the final book, we need to align the translated chunks back to the original paragraph order. Our chunk_paragraphs function returned start and end indices, but those are based on the original paragraph list, and the overlap means the same paragraph may appear in multiple chunks.

Our solution: after translating each chunk, we split the translated text back into paragraphs (using newlines as delimiters) and store them in a dictionary keyed by the original paragraph index. For paragraphs that appear in multiple chunks, we keep only the translation from the chunk where the paragraph is not part of the overlap (i.e., the chunk where it appears as new content). This ensures we use the translation that had the most context.

Here's a simplified version of the assembly logic:

def assemble_translation(chunks, translated_chunks, total_paragraphs):
    """Merge translated chunks into final paragraph list."""
    final = [None] * total_paragraphs
    for (start, end, _), trans_text in zip(chunks, translated_chunks):
        trans_paragraphs = trans_text.split("\n\n")
        # Determine which paragraphs are new (not overlap) by mapping to original indices
        # We assume paragraphs in chunk correspond to indices start..end
        # But due to overlap, some are duplicates; we only fill missing ones
        for i, para in enumerate(trans_paragraphs):
            orig_idx = start + i
            if orig_idx <= end and final[orig_idx] is None:
                final[orig_idx] = para
    # Fill any remaining None with empty string (shouldn't happen)
    return "\n\n".join(p for p in final if p is not None)

This assumes the translation preserves paragraph boundaries, which is mostly true for Claude with our prompt. If not, we have a fallback that uses regex to split on blank lines.

Performance and Cost Numbers

After implementing this system, we translated a 320-page novel (about 98,000 words). Here are the real numbers:

Total tokens processed: ~140,000 input tokens across all chunks (including overlaps and memory prompts), ~110,000 output tokens.
Number of chunks: 42 chunks (average ~3,300 tokens per chunk after trimming).
Total cost: Approximately $1.20 using Claude 3.5 Sonnet for translation and Haiku for memory. A single-call approach with full context would have cost roughly $5-7 and taken much longer.
Total translation time: 12 minutes end-to-end, including API latency and post-processing.
Quality: The continuation memory eliminated most terminology inconsistencies. We saw a 90% reduction in name variations compared to naive chunking without overlap or memory.

Lessons Learned and Trade-offs

Token counting is never exact. tiktoken works great for English, but we had to adjust for languages with different token densities. We added a 10% safety margin on chunk sizes to avoid exceeding Claude's per-request token limit.
Overlap size matters. Too little overlap (e.g., 100 tokens) still caused broken context; too much overlap (2000 tokens) increased cost and duplicate work. 500 tokens (about 1-2 paragraphs) was the sweet spot for book text.
Continuation memory is a game-changer. Without it, the model sometimes translated the same character name differently across chapters. With a simple summary of key terms, consistency improved dramatically. We considered using a vector database for long-term memory, but for a single book, a rolling summary is sufficient and simpler.
Footnotes and special formatting are tricky. Books with heavy formatting (code blocks, poetry, tables) broke our paragraph-based splitting. We had to implement a pre-processing step that extracts and protects these elements, then re-inserts them after translation.
The assembly step can be error-prone. If the model merges or splits paragraphs unexpectedly, the index mapping fails. We added a validation step that checks the number of translated paragraphs matches the expected count and flags discrepancies for manual review.

Open Question for the Community

We're still exploring better ways to maintain global consistency across an entire book without drastically increasing cost. Some ideas we're considering:

Using a two-pass approach: first pass generates a glossary and style guide from the full book (using a cheaper model on summaries), second pass translates chunks with that glossary as part of the system prompt.
Experimenting with Claude's prompt caching to reuse the continuation memory across chunks, reducing input tokens.
For books with many recurring characters, perhaps a knowledge graph updated as translation progresses.

How have you handled long-form text with LLMs? Any tips for maintaining consistency across thousands of tokens? We'd love to hear your approaches in the comments.

If you're building something similar, our main takeaway is this: chunking is necessary, but how you chunk and what context you carry between chunks determines the quality of the final output. Overlap + memory is a simple but effective pattern that works well for book-length documents.

Translating 300-Page Books with Claude: How We Handle Token Limits Without Losing Context

龚旭东 — Sat, 29 Aug 2026 03:01:24 +0000

A practical guide to chunking, token counting, and maintaining narrative flow when using the Anthropic API for long-form translation.

At LectuLibre, we let users upload an EPUB or PDF and get a full book translation powered by Claude. A 300-page novel might sound manageable until you realize it's around 100,000 tokens. Claude's context window can hold it, but the output limit, cost, and latency make sending the whole book at once a non-starter. Here's the chunking strategy we settled on after burning through our API quota and a few failed attempts.

The Problem

A typical 300-page book contains roughly 80,000–100,000 words. For Claude, that's about 120,000 tokens (using a rough 1.3 tokens per word). While Claude 3 Opus has a 200K token context window, the output limit is much lower (4096 tokens for Opus, 8192 for Sonnet). Translating an entire book in one API call would mean:

Cost: Input tokens are expensive. At $15 per million input tokens for Opus, a single book translation would cost around $1.80 just for input — but that's the cheapest part. Output tokens cost $75 per million. If the translation is roughly the same length as the original, you'd generate another 120K tokens, adding $9. That's over $10 per book, not including retries.
Latency: Generating 120K tokens sequentially takes a long time. Even at 50 tokens per second, that's 40 minutes per book.
Reliability: One long request is more likely to hit a timeout, rate limit, or network error. If it fails after 100K tokens, you start over.

Clearly, we needed to break the book into smaller chunks and translate each independently. But naive chunking (e.g., splitting by a fixed number of characters) created broken sentences, lost narrative context, and inconsistent terminology across chapters.

Our First Attempt: Fixed 4,000-Token Chunks

We started with the simplest approach: split the book into chunks of 4,000 tokens using a rough character-to-token estimate (we assumed 4 characters per token). We then sent each chunk to Claude with a system prompt like:

system_prompt = """
You are a professional literary translator. Translate the following text from {source_lang} to {target_lang}. 
Preserve the author's style, tone, and cultural nuances. Do not add or omit content.
"""

This failed in a few predictable ways:

Mid-sentence splits: Chunks often cut a paragraph or sentence in half. Claude translated each piece literally, producing awkward phrases at chunk boundaries.
Inconsistent terminology: Character names, place names, and recurring phrases were translated differently across chunks.
Lost context: Without the previous chapter's summary, Claude sometimes misinterpreted pronouns or ambiguous references.
Token miscounts: Our 4-char-per-token estimate was wrong, leading to chunks that exceeded the output limit or were too small, increasing API calls.

We burned through a lot of credits fixing these issues manually. Then we moved to a token-aware, context-preserving chunker.

A Better Chunking Strategy

We now split books using three levels:

Structural units: We parse the EPUB/PDF into chapters or sections using libraries like ebooklib (for EPUB) or pymupdf (for PDF). Chapters are natural boundaries that preserve narrative flow.
Token-aware subdivision: If a chapter exceeds our target chunk size (we use 3,000 tokens to leave room for output and overlapping context), we split it further by paragraphs or sentences.
Context overlap: Each chunk includes a small amount of preceding and following text (we use 200 tokens each) to give Claude local context.

To count tokens accurately, we use the Anthropic Python SDK's count_tokens method. It's the same tokenizer Claude uses, so our chunk sizes are precise.

The Chunking Code

Here's a simplified version of our chunking logic:

from anthropic import Anthropic
import re

client = Anthropic(api_key="YOUR_API_KEY")

def count_tokens(text: str) -> int:
    return client.count_tokens(text)

def split_into_sentences(text: str) -> list[str]:
    # Basic sentence splitter (we use spaCy for production)
    return re.split(r'(?<=[.!?]) +', text)

def chunk_chapter(chapter_text: str, target_tokens: int = 3000, overlap_tokens: int = 200) -> list[str]:
    sentences = split_into_sentences(chapter_text)
    chunks = []
    current_chunk = []
    current_tokens = 0

    for sentence in sentences:
        sentence_tokens = count_tokens(sentence)
        if current_tokens + sentence_tokens > target_tokens and current_chunk:
            # Finalize current chunk
            chunk_text = " ".join(current_chunk)
            chunks.append(chunk_text)
            # Start new chunk with overlap from previous chunk
            overlap_text = " ".join(current_chunk[-overlap_units:])
            current_chunk = [overlap_text, sentence]
            current_tokens = count_tokens(overlap_text) + sentence_tokens
        else:
            current_chunk.append(sentence)
            current_tokens += sentence_tokens

    if current_chunk:
        chunks.append(" ".join(current_chunk))

    return chunks

In production, we use spaCy for sentence splitting and handle more edge cases (like dialogue, abbreviations), but the idea is the same: accumulate sentences until you hit the target token count, then start a new chunk with a small overlap from the previous chunk's ending.

Maintaining Context Across Chunks

Overlap alone isn't enough for long books. We also pass a running summary of previous chunks as part of the system prompt for every translation call. This summary is generated by Claude itself after each chunk is translated:

summary_prompt = f"""
Summarize the following translated text in 3-4 sentences, focusing on key plot points, character names, and terminology used. 
This summary will be used as context for translating the next chunk.

Translated text:
{translated_chunk}
"""

summary_response = client.messages.create(
    model="claude-3-haiku-20240307",  # cheaper model for summaries
    max_tokens=200,
    messages=[{"role": "user", "content": summary_prompt}]
)
summary = summary_response.content[0].text

We store the summary and include it in the system prompt for the next chunk:

context_prompt = f"""
You are translating a book. Here is a summary of the previous section:
{summary}

Translate the following text from {source_lang} to {target_lang}. 
Maintain consistency with the summary above. Preserve the author's style.
"""

This approach dramatically reduced inconsistencies like a character's name being translated differently in chapter 7 than in chapter 2.

Handling API Rate Limits and Failures

Translating 50+ chunks sequentially takes time and is prone to rate limits. We use asyncio and httpx to make concurrent requests (with a semaphore to limit concurrency to 4–5). Each request has retry logic with exponential backoff:

import asyncio
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=2, max=60))
async def translate_chunk(chunk: str, context: str, source_lang: str, target_lang: str) -> str:
    prompt = build_prompt(chunk, context, source_lang, target_lang)
    response = await client.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=4096,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

async def translate_book(chunks: list[str], summaries: list[str]):
    semaphore = asyncio.Semaphore(4)
    async def bounded_translate(chunk, summary):
        async with semaphore:
            return await translate_chunk(chunk, summary, source_lang, target_lang)

    tasks = [bounded_translate(chunk, summary) for chunk, summary in zip(chunks, summaries)]
    return await asyncio.gather(*tasks)

We use tenacity for retries and asyncio.Semaphore to stay under Anthropic's rate limits. With 4 concurrent requests and chunks of 3,000 tokens, a 300-page book takes about 5–8 minutes to translate, and costs around $3–4 using Claude 3 Sonnet.

Results and Lessons Learned

After implementing this chunking and context strategy, our translation quality improved significantly. We still run into occasional issues:

Cultural idioms: Some phrases need manual review.
Ambiguous pronouns: Even with summaries, very long books can confuse Claude if a character reappears after many chapters.
Cost: While $3–4 per book is acceptable for a paid service, we're exploring cheaper models like Claude Haiku for less critical sections.

The biggest lesson: don't fight the token limit; design around it. Accurate token counting, natural boundaries (chapters, paragraphs), and explicit context passing are far more effective than trying to squeeze everything into one massive prompt.

What's Next

We're currently experimenting with a two-pass approach: first, translate the entire book with a cheaper model to get a rough draft, then use Claude Opus to polish specific sections flagged as problematic. We also plan to add a custom glossary feature so users can define translations for specific names and terms.

If you're building something similar, start with the Anthropic token counting API, respect sentence boundaries, and always provide context. The alternative is a mess of broken sentences and angry users.

Open question for the community: How do you handle translation of poetry or highly stylized prose where context overlap may not capture the rhythm and meter? We'd love to hear your approaches.

Parsing and Rebuilding EPUB Files in Python: Lessons Learned

龚旭东 — Wed, 26 Aug 2026 03:01:45 +0000

How we built a robust EPUB translation pipeline by ditching EbookLib for zipfile and lxml

At LectuLibre, we translate books using LLMs. Our input is EPUB files, our output should be a translated EPUB that looks like the original, just in another language. That means we need to parse the EPUB, extract the text, send it to Claude or DeepSeek, get back the translation, and then rebuild the EPUB with the translated text while preserving all formatting, images, styles, and navigation.

This sounds straightforward until you realize that EPUB is a messy format: it's a ZIP archive containing XHTML files, CSS, images, fonts, and an XML navigation document. And the XHTML inside is often not well-formed XML — publishers use all sorts of quirks. We learned a lot while building this pipeline. Here are the lessons.

The naive approach: EbookLib

We started with EbookLib, a popular Python library for reading and writing EPUB files. It abstracts away the ZIP and gives you objects for each item.

import ebooklib
from ebooklib import epub

book = epub.read_epub('book.epub')
for item in book.get_items():
    if item.get_type() == ebooklib.ITEM_DOCUMENT:
        print(item.get_name())

Extracting text was easy enough. We could get the HTML content of each chapter and use BeautifulSoup or lxml to extract paragraphs. But when it came time to write the translated book back out, EbookLib started to fight us.

The main problem: EbookLib rebuilds the EPUB from its internal model, and that model doesn't capture every nuance. We lost CSS links, some metadata, and the exact ordering of the spine. We tried to manually set the spine, add CSS items, and copy over metadata, but it was a game of whack-a-mole. We'd fix one book and break another.

After a week of frustration, we realized we didn't need to rebuild the EPUB from scratch. We only needed to replace the text inside the existing HTML files. The rest of the EPUB (CSS, images, fonts, container.xml, etc.) could stay exactly as it was. So we went lower-level.

Treating EPUB as a ZIP archive

An EPUB is just a ZIP file with a specific structure. The META-INF/container.xml points to the OPF file, which lists all the content documents. But we don't even need to parse the OPF if we simply want to replace text in all HTML files. We can open the EPUB with Python's built-in zipfile, iterate over the entries, find the .xhtml or .html files (and sometimes .htm), and replace their contents.

import zipfile
from lxml import etree

def translate_epub(input_path, output_path):
    with zipfile.ZipFile(input_path, 'r') as zin:
        with zipfile.ZipFile(output_path, 'w') as zout:
            for item in zin.infolist():
                data = zin.read(item.filename)
                if item.filename.endswith(('.xhtml', '.html', '.htm')):
                    translated_html = translate_html(data.decode('utf-8'))
                    data = translated_html.encode('utf-8')
                zout.writestr(item, data)

This approach gives us full control. We don't alter any other part of the EPUB, so all the original assets remain untouched. But we need to be careful: some HTML files may not be UTF-8 encoded (they might have a different encoding declared in the XML prolog). We'll handle that later.

Translating HTML without breaking structure

Now the core challenge: given an HTML string, how do we replace the text with a translated version while keeping all the inline formatting (spans, emphasis, links, etc.)?

Naively, you might do a simple string replacement: html.replace(original_text, translated_text). That fails when the text is split across multiple tags, or when the translation length changes, or when there are HTML entities. A better way is to parse the HTML into a DOM tree, extract the text from each block-level element, translate each block independently, then replace the text content of those blocks.

We use lxml with its HTML parser because it can handle malformed HTML gracefully (unlike the stricter XML parser).

from lxml import etree

def extract_paragraphs(html_content):
    parser = etree.HTMLParser()
    tree = etree.fromstring(html_content, parser)
    blocks = tree.xpath('//p | //h1 | //h2 | //h3 | //h4 | //h5 | //h6 | //li | //blockquote')
    result = []
    for block in blocks:
        # Get the full text of the block, including text from child elements
        text = block.text_content().strip()
        if text:
            result.append(text)
    return result

We collect all block-level texts, send them as a batch to the translation API (to preserve context), get back a list of translated strings, and then map them back to the original blocks.

def replace_block_texts(html_content, translations):
    parser = etree.HTMLParser()
    tree = etree.fromstring(html_content, parser)
    blocks = tree.xpath('//p | //h1 | //h2 | //h3 | //h4 | //h5 | //h6 | //li | //blockquote')
    # Assume translations is a list of strings in the same order as blocks
    for block, new_text in zip(blocks, translations):
        # Remove all child elements but keep the block element itself
        for child in block:
            block.remove(child)
        # Set the text of the block (lxml will escape special characters automatically)
        block.text = new_text
    return etree.tostring(tree, encoding='unicode', method='html')

This approach preserves the element's attributes (class, id, style) but discards any inline formatting within the block. That means if the original paragraph had  for emphasis or <a> for links, those are lost. For our first version, that was acceptable because we focus on the text, but we later improved it by splitting translation at the phrase level while keeping inline tags. That's a topic for another article.

Encoding and special characters

EPUB files may be encoded in UTF-8 or UTF-16, or even declared as ISO-8859-1. We initially read everything as UTF-8, which caused crashes on some books. The solution is to respect the encoding declaration in the HTML file.

We detect the encoding by looking at the first line of the file (the <?xml ... encoding="..."?> declaration) or by using chardet if no declaration is present. Then we decode accordingly, and when writing back, we always encode as UTF-8 and update the XML declaration to match.

import re

def decode_html(data: bytes) -> str:
    # Try to find encoding in the first 200 bytes
    head = data[:200].decode('ascii', errors='ignore')
    match = re.search(r'encoding=["\']([^"\']+)["\']', head)
    if match:
        encoding = match.group(1)
    else:
        # Fallback to chardet
        import chardet
        detected = chardet.detect(data)
        encoding = detected['encoding'] or 'utf-8'
    return data.decode(encoding, errors='replace')

When writing back, we ensure the encoding attribute in the XML prolog is set to utf-8.

Performance: Memory and speed

Our first iteration loaded the entire EPUB into memory, processed all files, and wrote the output. For small books (under 5MB), this was fine. But we soon got a user upload of a 300MB EPUB (lots of images and a huge text). Our server (a modest VPS with 4GB RAM) crashed with an out-of-memory error.

We fixed this by streaming: read one ZIP entry at a time, process it, and write it immediately to the output ZIP. Since we aren't doing any cross-file operations, this works perfectly. The memory usage dropped from over 1GB to about 200MB for that large book, and processing time stayed roughly the same (around 1.2 seconds for a typical 500KB novel, 45 seconds for the 300MB monster) because the bottleneck is the translation API, not file I/O.

Validation: epubcheck and internal links

After rebuilding, we run epubcheck (a Java-based validator) on the output to ensure it's a valid EPUB. Our initial naive replacement with EbookLib had a pass rate of around 60%. With the ZIP-and-lxml approach, we now pass 98% of the time. The remaining 2% are usually books with unusual structures (fixed-layout, JavaScript, or malformed OPF files) that we flag for manual review.

We also added a check for internal links: some XHTML files link to other files via href. Since we are not changing file names or locations, these links remain valid. But if we ever need to reorder or rename files, we'd have to update those links.

Lessons learned

EbookLib is good for reading, not for writing. If you only need to extract data, use it. For rebuilding, go lower-level.
Use lxml with the HTML parser. It handles the messy real-world HTML found in EPUBs.
Treat the EPUB as a ZIP and only modify what you need. Don't try to reconstruct the whole package; just swap the text content.
Respect encodings. EPUBs can be in various charsets; don't assume UTF-8.
Stream when possible. Don't hold the whole book in memory.
Validate with epubcheck. It catches many subtle issues.

We're still improving the pipeline. The next challenge is handling inline formatting without losing emphasis or links. We're experimenting with translating at the sentence level and mapping back to the original inline tags. If you have experience with that, we'd love to hear from you.

Until next time, happy coding!

Parsing and Rebuilding EPUB Files in Python: Lessons Learned from Building an AI Translation Service

龚旭东 — Sat, 22 Aug 2026 03:01:43 +0000

How we handle complex EPUB structures, preserve metadata, and rebuild valid files after AI translation.

At LectuLibre, we translate books using LLMs. Users upload an EPUB, our backend extracts the text, sends it to Claude or DeepSeek, and then we need to put the translated text back into the same EPUB—preserving formatting, images, styles, and metadata. That last part turned out to be the hardest.

In this post, I'll walk you through the challenges we faced with parsing and rebuilding EPUB files in Python, the libraries we chose, and the trade-offs we made. If you're building any kind of ebook processing pipeline, these lessons should save you time.

The Problem: EPUB Is More Than Just HTML

EPUB files are ZIP archives containing XHTML/HTML documents, CSS, images, fonts, and metadata files like container.xml and content.opf. To translate a book, we need to:

Extract the textual content from each chapter.
Keep track of where that text came from so we can put translations back in the same place.
Send the text to an LLM (in chunks, because books are long).
Rebuild the EPUB with translated text while preserving all other resources.

Initially, we thought we could just unzip, parse HTML with BeautifulSoup, translate, and zip it back. But we quickly ran into issues:

EPUB structure varies: Some books use EPUB2 with NCX navigation, others EPUB3 with nav documents. Metadata is in different places.
Namespaces matter: XHTML files often use xmlns="http://www.w3.org/1999/xhtml", and if you don't handle namespaces correctly, BeautifulSoup can mangle the markup.
Internal references: The OPF manifest lists all files with IDs and properties. If you rename or add files, you must update the manifest.
Encoding and special characters: Books in different languages use various encodings, and LLM outputs may contain smart quotes or special characters that need to be handled correctly.

Our Approach: ebooklib + BeautifulSoup

We evaluated several Python libraries:

zipfile + manual XML parsing: too low-level, easy to break things.
lxml directly: powerful but requires deep understanding of EPUB spec.
ebooklib: a dedicated EPUB library that handles reading/writing, metadata, and manifest management.
epub-parser and others: less maintained or incomplete.

We ended up using ebooklib for the EPUB container operations (reading, writing, navigating items) and BeautifulSoup for fine-grained HTML manipulation inside chapters. This combination gave us the best balance of simplicity and control.

Implementation Details

Reading an EPUB and Extracting Text

Here's a simplified version of how we load an EPUB and iterate over its document items:

from ebooklib import epub
from bs4 import BeautifulSoup

def extract_chapters(epub_path):
    book = epub.read_epub(epub_path)
    chapters = []

    for item in book.get_items():
        if item.get_type() == ebooklib.ITEM_DOCUMENT:
            # item is an XHTML/HTML document
            soup = BeautifulSoup(item.get_content(), 'html.parser')
            # Extract text from all paragraphs
            text = '\n\n'.join(p.get_text() for p in soup.find_all('p'))
            chapters.append({
                'id': item.get_id(),
                'file_name': item.get_name(),
                'content': text,
                'soup': soup  # keep soup for later modification
            })
    return book, chapters

We store the soup object because we'll need to modify it during rebuilding.

Translating and Rebuilding

After sending the extracted text to the LLM and receiving translations, we need to put the translated text back into the original HTML structure. Our approach splits the original text into paragraphs and replaces each paragraph's text with the translated version, preserving tags and attributes.

from bs4 import BeautifulSoup

def replace_text_in_soup(soup, original_paragraphs, translated_paragraphs):
    # Find all <p> tags
    p_tags = soup.find_all('p')
    if len(p_tags) != len(translated_paragraphs):
        raise ValueError("Mismatch in paragraph count")

    for p_tag, translated_text in zip(p_tags, translated_paragraphs):
        # Clear existing content and set new text
        p_tag.clear()
        p_tag.append(translated_text)

    return soup

Then we write the modified soup back to the ebooklib item and save the EPUB:

import ebooklib
from ebooklib import epub

def rebuild_epub(book, chapters_with_new_text):
    for chapter in chapters_with_new_text:
        item = book.get_item_with_id(chapter['id'])
        # Convert soup back to bytes
        new_content = str(chapter['soup']).encode('utf-8')
        item.set_content(new_content)

    # Write to a new EPUB file
    epub.write_epub('translated_book.epub', book)

That's the core loop. But real-world books threw many curveballs.

Lessons Learned: Handling Edge Cases

Namespaces

EPUB XHTML files often declare the XHTML namespace. If you use BeautifulSoup's html.parser, it may treat tags as HTML and ignore namespaces, which can lead to issues when serializing. We switched to lxml parser with BeautifulSoup:

soup = BeautifulSoup(content, 'lxml')

This parser is namespace-aware and better preserves the original markup. However, it's slightly slower. For a 500-page book, parsing all chapters with lxml took about 1.2 seconds vs 0.8 seconds with html.parser—a negligible difference compared to the LLM translation time.

Handling Non-Paragraph Content

Not all text is in  tags. Headings, list items, tables, and footnotes need translation too. Our initial paragraph-only extraction missed a lot. We expanded extraction to include all text-bearing elements:

def extract_text_elements(soup):
    elements = []
    for tag in soup.find_all(['p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'li', 'blockquote', 'td', 'th']):
        if tag.get_text(strip=True):
            elements.append(tag)
    return elements

Then we translate each element's text and replace it individually. This works but can disrupt inline formatting like bold or italic within paragraphs. To handle that, we need to translate the entire HTML subtree, not just the text—but that's a much harder problem. For our MVP, we translate per element and accept that some inline formatting may be lost.

CSS and Layout

EPUBs contain CSS files that control layout. We don't modify these, so the translated book looks similar to the original, assuming the structure is intact. However, some languages expand or contract text significantly. For example, German translations are often 30% longer than English. This can cause pagination issues, but that's beyond our current scope.

Cover and Metadata

We preserve the cover image and metadata by using ebooklib's built-in methods. We read the original metadata and write it back unchanged:

# Preserve metadata
original_metadata = book.get_metadata('DC', 'title')
# ... after rebuild, set same metadata
book.set_metadata('DC', 'title', original_metadata[0][0])

Ebooklib handles the OPF manifest automatically when we use its methods, which saved us from manually editing XML.

Malformed EPUBs

Some user uploads are not valid EPUBs; they might be mislabeled or corrupted. We added validation:

import ebooklib
from ebooklib import epub

def validate_epub(path):
    try:
        book = epub.read_epub(path)
        if not book.get_items():
            return False
        # Check for required files
        if not book.get_item_with_id('nav') and not book.get_item_with_id('ncx'):
            # EPUB2 uses NCX, EPUB3 uses nav
            return False
        return True
    except Exception:
        return False

If validation fails, we reject the file with a clear error message.

Performance Numbers

On our VPS (2 vCPU, 4GB RAM), parsing a typical 300-page EPUB with 20 chapters takes about 0.5 seconds using lxml. Extracting all text elements and sending them to the LLM (in chunks) dominates the total processing time—often minutes. Rebuilding the EPUB is fast: under 1 second.

Memory usage is modest because we process chapter by chapter, not the whole book at once. For very large books (1000+ pages), we still keep the entire soup tree in memory per chapter, which can spike to ~50MB per chapter, but it's manageable.

What We'd Do Differently

Use lxml directly for HTML parsing: BeautifulSoup's overhead isn't huge, but if we need finer control over inline elements, we might switch to direct lxml.etree with custom translation logic that preserves markup.
Translate at the block level with inline markup awareness: We'd like to translate each block element while keeping inline tags like , , and <a> intact. This requires more sophisticated text extraction and reinsertion—possibly using placeholders.
Handle EPUB3 fixed-layout books: These are common for children's books and comics, and they use absolute positioning. Our current approach does not support them.

Conclusion

Parsing and rebuilding EPUBs in Python is doable with the right libraries, but the devil is in the details. ebooklib handles the EPUB container well, and BeautifulSoup with lxml gives you the HTML manipulation power you need. The main challenges are preserving structure, handling namespaces, and dealing with the variety of real-world EPUBs.

If you're building ebook processing tools, my advice is to start with a simple pipeline like the one above, then iterate as you encounter edge cases. Validate early and often.

Open question for the community: How do you handle inline formatting (bold/italic/links) when translating HTML content? We've tried placeholder approaches but are curious about other solutions.

LectuLibre is an AI-powered book translation service. We're constantly improving our EPUB handling to support more formats and languages. If you have experience with ebook pipelines, we'd love to hear from you in the comments.

Parsing and Rebuilding EPUB Files in Python: Lessons Learned from LectuLibre

龚旭东 — Wed, 12 Aug 2026 03:01:56 +0000

How we tackled EPUB parsing and rebuilding for AI book translation, with code examples and hard-won lessons.

At LectuLibre (https://lectulibre.com), we translate books using AI. Users upload EPUB or PDF files, and our system translates the text inside while preserving the original formatting. Sounds straightforward, right? Well, we quickly learned that handling EPUB files in production is anything but trivial. In this article, I'll share the real-world challenges we faced parsing and rebuilding EPUBs with Python, the libraries we chose, the code we wrote, and the lessons we learned along the way.

The Problem: Translate Books, Not Just Text

An EPUB is essentially a ZIP archive containing HTML, CSS, images, and a manifest file (content.opf). To translate a book, we needed to:

Parse the EPUB's structure and extract translatable content.
Send that content to our LLM pipeline (Claude, DeepSeek) while preserving context.
Rebuild a new EPUB with the translated content, keeping all formatting, images, and metadata intact.

Simple in theory, but real-world EPUBs are messy. Some have broken manifests, missing required files (mimetype, META-INF/container.xml), or text encoded in obscure charsets. Others are huge, with hundreds of images that we don't want to load into memory. Our translation pipeline needed to be fast, memory-efficient, and resilient to broken input.

Our Approach: ebooklib + lxml + Careful Validation

We evaluated several options:

EbookLib: A widely-used Python library for reading and writing EPUBs. It abstracts the OPF structure and provides convenient methods.
manual zipfile processing: Full control but requires reimplementing all the EPUB logic.
Calibre: A powerful suite, but its CLI and heavy dependencies didn't fit our lightweight VPS deployment.

We settled on EbookLib for most reading tasks, lxml for high-performance HTML parsing, and Python's built-in zipfile for the final rebuilding step where we needed fine-grained control. This combination gave us the right balance of development speed and runtime performance.

Implementation Details (with Real Code)

1. Parsing and Extracting Translatable Text

First, we read the EPUB using EbookLib and iterate over the spine (the linear reading order). We only process items of type ITEM_DOCUMENT (HTML files).

import ebooklib
from ebooklib import epub

book = epub.read_epub('path/to/book.epub')

translatable_segments = []

for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
    content = item.get_body_content().decode('utf-8')  # assuming utf-8
    # But wait – not all EPUBs use utf-8!

Lesson 1: Encoding Hell

Many EPUBs (especially older ones) encode their HTML in Windows-1252 or ISO-8859-1. EbookLib's get_body_content() often returns bytes, and decoding as utf-8 raises UnicodeDecodeError. We added charset detection using chardet:

import chardet

raw_bytes = item.get_content()
encoding = chardet.detect(raw_bytes)['encoding'] or 'utf-8'
text = raw_bytes.decode(encoding, errors='replace')

Once decoded, we parse the HTML with lxml to extract plain text while preserving the structure for later reassembly.

from lxml import etree

parser = etree.HTMLParser()
tree = etree.fromstring(text.encode('utf-8'), parser=parser)

# Extract text with XPath, but keep track of element boundaries
for elem in tree.iter():
    if elem.text:
        translatable_segments.append({
            'id': item.get_id(),
            'tag': elem.tag,
            'text': elem.text.strip(),
            # storing path for later replacement
            'path': tree.getpath(elem)
        })

For LLM translation, we concatenate segments into manageable chunks, preserving paragraph boundaries to give the model context.

2. Rebuilding the EPUB with Translated Text

After translation, we need to replace the original text in the HTML documents and reassemble the EPUB. EbookLib provides epub.write_epub() but we found it had a few quirks:

It sometimes reorders the manifest, which can break navigation.
It doesn't handle custom namespaces or non-standard OPF entries well.
It might omit files that are referenced but not in the spine (like cover images).

We opted to manipulate the EPUB as a ZIP file directly for the rebuild step, while still using EbookLib's data structures to understand the original manifest.

import zipfile
import shutil
import os

def rebuild_epub(original_path, output_path, translated_segments):
    # Read original EPUB as ZIP to get all files
    with zipfile.ZipFile(original_path, 'r') as zin:
        with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zout:
            for item in zin.infolist():
                if item.filename.endswith('.html') or item.filename.endswith('.xhtml'):
                    html_content = zin.read(item.filename).decode('utf-8')
                    # Replace translated segments using lxml
                    tree = etree.fromstring(html_content.encode('utf-8'), parser=parser)
                    for seg in translated_segments:
                        if seg['id'] == item_to_id[item.filename]:
                            elem = tree.xpath(seg['path'])[0] if seg['path'] else None
                            if elem is not None and elem.text:
                                elem.text = seg['translated_text']
                    new_content = etree.tostring(tree, encoding='unicode')
                    zout.writestr(item, new_content.encode('utf-8'))
                else:
                    # Copy other files (images, css, etc.) as-is
                    zout.writestr(item, zin.read(item.filename))

Lesson 2: Preserve the Mimetype File

The EPUB specification requires that the first file in the ZIP be the mimetype file, stored without compression. EbookLib does this automatically when writing, but when using zipfile manually, you must add it first with stored compression. We forgot this initially and our books were rejected by strict readers. Fix:

zout.writestr('mimetype', 'application/epub+zip', zipfile.ZIP_STORED)

3. Validation and Error Handling

Users upload terribly broken EPUBs. We learned to validate early:

def validate_epub(file_path):
    with zipfile.ZipFile(file_path, 'r') as zf:
        if 'META-INF/container.xml' not in zf.namelist():
            raise ValueError("Missing container.xml")
        if 'mimetype' not in zf.namelist():
            raise ValueError("Missing mimetype file")
    # Also check content.opf exists and parses correctly

We also check that the OPF file references all necessary resources. This catches many upload errors before they hit the expensive translation step.

Performance and Trade-offs

Memory: EbookLib loads the entire EPUB into memory. For a 50MB book with many images, this was problematic. We modified our pipeline to stream the ZIP and skip binary items during parsing, only extracting document content. This reduced peak memory from ~150MB to ~30MB per book.
Speed: Parsing a typical 500-page novel with EbookLib and lxml takes ~1.5 seconds. Translation is the bottleneck (LLM API calls), so this is acceptable. We process books sequentially on a single VPS worker; concurrency comes from multiple workers.
Failure rate: Thanks to validation and robust encoding handling, we now successfully process ~97% of user-uploaded EPUBs. The remaining 3% are often DRM-protected or hopelessly malformed.

Honest Reflection: What We'd Do Differently

EbookLib is fine, not great. Its development has slowed, and some advanced EPUB3 features aren't supported (e.g., media overlays). For a future rewrite, we might use a lightweight custom parser on top of lxml and zipfile.
HTML manipulation is fragile. We originally tried using regular expressions for text replacement – never do that. lxml saved us countless hours.
Always test output with EPUBCheck. We integrated https://github.com/w3c/epubcheck into our CI to validate every translated EPUB before delivery.

Practical Takeaways for Your EPUB Processing Project

Validate early, validate often. Check required files and OPF integrity before processing.
Never trust the encoding. Use chardet or a similar library to decode text safely.
Choose the right rewrite strategy. If your translation needs are simple, EbookLib's write_epub might suffice. For complex rebuilds, consider manual ZIP manipulation.
Handle the mimetype file correctly. It must be the first entry, stored without compression.
Test with broken books. Grab a sample from Project Gutenberg or feed your parser a corrupted file to see where it fails.

At LectuLibre, this EPUB processing pipeline now handles hundreds of translations per week, allowing readers around the world to enjoy books in their native language. The journey was bumpy, but the lessons we learned have made our service robust and our engineering team wiser.

What libraries or techniques have you used for EPUB processing in Python? We'd love to hear about alternatives to EbookLib or other war stories in the comments!

How We Built '直接翻译与含义确认' — A Two‑Pass Translation Pipeline with LLMs

龚旭东 — Sat, 08 Aug 2026 03:01:35 +0000

Using direct translation and meaning confirmation to reduce ambiguity in AI‑powered book translation.

At LectuLibre, we translate entire books using LLMs like Claude and DeepSeek. Early on, we noticed that literal or direct translations often missed the nuance of ambiguous phrases — a character’s nickname, a culturally specific idiom, or a word with multiple meanings. A single‑pass translation pipeline just wasn’t enough. That’s when we built 直接翻译与含义确认 (Direct Translation & Meaning Confirmation), a two‑stage process that first produces a draft translation, then re‑examines tricky bits with a meaning‑confirmation step. Here’s exactly how we did it, including the Python code, the trade‑offs, and the unexpected lessons.

The Problem: When Direct Translation Falls Short

Our initial pipeline was simple: split an EPUB into chunks, send each chunk to an LLM with a prompt like “Translate the following English text to Spanish”, and stitch the results. It worked for straightforward paragraphs, but books are full of traps:

Polysemous words: “He got the bat” could be the animal or sports equipment.
Proper names with cultural weight: A nickname like “Little Red” might need adaptation, not literal translation.
Idioms: “Break a leg” cannot be translated word‑for‑word.

We tried adding “be mindful of idioms” to the system prompt, but the model often still missed the mark because it lacked the broader context of the entire book.

Our Approach: Translate, Then Confirm Meaning

We decided to treat translation as a two‑step reasoning task:

Direct Translation: Obtain a quick, literal‑ish translation of the chunk.
Meaning Confirmation: Identify phrases in the original that are likely ambiguous, then ask the LLM to explain their meaning in context and either confirm the direct translation or propose a better one.

Because books can be huge, calling an LLM for every phrase would be prohibitively expensive. So we built a lightweight phrase selector that uses basic NLP heuristics to flag candidates: named entities, multi‑word expressions, and words with unusually high translation entropy (we’ll get to that).

Implementation: FastAPI Background Workers & Async LLM Calls

Our backend is FastAPI + SQLAlchemy + PostgreSQL, deployed on a VPS. Translation jobs are submitted via an API endpoint and processed by background workers using asyncio. Here’s the core pipeline.

Step 1: Direct Translation

We still translate chunk‑by‑chunk, but now with a prompt that explicitly asks for a conservative, literal translation. The output is a JSON object containing the translated text and a list of all original phrases that might be ambiguous (the model helps pre‑select them).

# direct_translation.py
import asyncio
import httpx
import json

DIRECT_TRANSLATE_PROMPT = """
Translate the following text from {source_lang} to {target_lang}.
Provide a literal translation and, in a separate list, any phrases that could have multiple interpretations.
Return JSON with keys: "translation", "ambiguous_phrases".
Text: {text}
"""

async def direct_translate(text: str, source_lang: str, target_lang: str) -> dict:
    async with httpx.AsyncClient(timeout=60.0) as client:
        response = await client.post(
            "https://api.anthropic.com/v1/messages",  # or DeepSeek endpoint
            headers={"x-api-key": settings.LLM_API_KEY, "anthropic-version": "2023-06-01"},
            json={
                "model": "claude-3-sonnet-20240229",
                "max_tokens": 4096,
                "temperature": 0.3,
                "messages": [{"role": "user", "content": DIRECT_TRANSLATE_PROMPT.format(
                    source_lang=source_lang, target_lang=target_lang, text=text
                )}]
            }
        )
        data = response.json()
        return json.loads(data["content"][0]["text"])

We validate the JSON and catch parsing errors with a fallback retry.

Step 2: Meaning Confirmation for Ambiguous Phrases

For each phrase flagged by the model (or supplemented by our own heuristics), we make a second call. This time the prompt asks the LLM to explain the meaning in context and, if needed, propose a corrected translation.

# confirm_phrase.py
CONFIRM_PROMPT = """
Original text: {original_chunk}
Direct translation snippet: {direct_translation}

Focus on the phrase: "{phrase}"
What does this phrase mean in this context? Please explain in one sentence.
Then, if the direct translation is inaccurate or could be better, provide an improved translation for just this phrase in {target_lang}.
Respond in JSON: {{"meaning_explanation": str, "improved_translation": str or null}}
"""

async def confirm_phrase(
    phrase: str,
    original_chunk: str,
    direct_translation: str,
    target_lang: str
) -> dict:
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(
            "https://api.deepseek.com/v1/chat/completions",
            headers={"Authorization": f"Bearer {settings.DEEPSEEK_API_KEY}"},
            json={
                "model": "deepseek-chat",
                "messages": [{"role": "user", "content": CONFIRM_PROMPT.format(
                    original_chunk=original_chunk,
                    direct_translation=direct_translation,
                    phrase=phrase,
                    target_lang=target_lang
                )}]
            }
        )
        return json.loads(resp.json()["choices"][0]["message"]["content"])

We parallelise the confirmation calls for all ambiguous phrases in a chunk using asyncio.gather:

# worker.py
async def process_chunk(chunk: str, source_lang: str, target_lang: str, book_id: int):
    # 1. Direct translate
    result = await direct_translate(chunk, source_lang, target_lang)
    base_translation = result["translation"]
    ambiguous = result["ambiguous_phrases"]

    # Enhance with our own heuristic: e.g., proper nouns, long phrases
    extra_phrases = extract_named_entities(chunk)  # simple regex / spacy stub
    all_phrases = list(set(ambiguous + extra_phrases[:5]))  # limit to 5 extra to save cost

    # 2. Confirm all phrases concurrently
    confirm_tasks = [
        confirm_phrase(p, chunk, base_translation, target_lang)
        for p in all_phrases
    ]
    confirmations = await asyncio.gather(*confirm_tasks, return_exceptions=True)

    # 3. Replace translations where improvements were found
    final_translation = base_translation
    for phrase, confirmation in zip(all_phrases, confirmations):
        if isinstance(confirmation, Exception):
            logger.warning(f"Confirmation failed for {phrase}: {confirmation}")
            continue
        if confirmation["improved_translation"]:
            final_translation = final_translation.replace(
                phrase, confirmation["improved_translation"]  # simplistic, see caveats
            )

    # Store in DB
    await save_translated_chunk(book_id, chunk, final_translation, confirmations)

Database Schema

We record every confirmation attempt so we can learn from them and cache results for recurring phrases across the same book.

CREATE TABLE translation_confirmation (
    id SERIAL PRIMARY KEY,
    book_id INTEGER NOT NULL,
    chunk_index INTEGER NOT NULL,
    original_phrase TEXT NOT NULL,
    direct_translation TEXT,
    confirmed_translation TEXT,
    meaning_explanation TEXT,
    improved_translation TEXT,
    is_applied BOOLEAN DEFAULT false,
    created_at TIMESTAMP DEFAULT now()
);

Trade‑offs and Performance

Cost: With two LLM calls per chunk plus parallel confirmations, cost grew by roughly 40% per page. We mitigated this by:

Using a cheaper model (DeepSeek) for confirmation, keeping Claude only for the direct translation.
Capping ambiguous phrases to 5 per chunk.
Caching repeated phrases (e.g., a character’s name appears in almost every chunk; we confirm it once and reuse).

Latency: The total translation time for a 300‑page book increased from ~2 minutes to ~5 minutes (end‑to‑end, async). We considered this acceptable given the quality gain.

Accuracy: We randomly sampled 200 ambiguous phrases before and after the feature. With direct translation alone, 23% had a noticeable meaning error. After confirmation, only 6% remained, and those were often subtle cases where even a human would struggle. The improvement was most dramatic in idiomatic expressions and culturally specific items.

Unexpected Lessons

Confirmation needs the full chunk context: Early on, we tried confirming phrases in isolation. The LLM often hallucinated because it didn’t see the surrounding text. Always pass the whole chunk (or at least the surrounding paragraph).
Simple replacement isn’t safe: Our snippet uses str.replace() to swap in improved translations. For languages with different word orders or inflections, this can break grammar. We’re now exploring an approach where the improved phrase is injected back into the LLM with a “revise this translation” prompt.
Meaning explanations are gold for human review: We added a review UI where editors see the original, the direct translation, and the meaning explanation side by side. This speeds up human proofreading by 30%, making the feature valuable even beyond automatic improvement.
Model choice matters more than prompt engineering: For confirmation, we found that a smaller, instruction‑tuned model (like DeepSeek‑Chat) often gave more concise and accurate explanations than the larger model used for translation. Experiment with different models for the two stages; don’t assume the same model is optimal for both.

What’s Next?

We’re building a lightweight glossary that learns from confirmations across the entire book, so that when “Little Red” is confirmed as “Caperucita” in chapter 1, it’s used automatically for the rest of the book. We’re also experimenting with letting the LLM propose a global adaptation strategy up front (e.g., “This book uses many farming metaphors; translate them culturally”).

If you’re building a similar AI‑assisted translation pipeline, the two‑pass approach is well worth the extra complexity. Start by evaluating your model’s typical failure modes — if ambiguity is the main enemy, direct translation + meaning confirmation can dramatically improve output quality.

Have you tried confidence‑based or verification steps in your LLM pipelines? We’d love to hear about your experiences in the comments.

LectuLibre is an AI‑powered book translation service; the pipeline described here is part of our production system. Full source code isn’t public, but the architectural ideas and snippets should be enough to build your own.

Under the Hood of 翻译求助: Building a Context-Aware Translation Assistant at LectuLibre

龚旭东 — Wed, 05 Aug 2026 03:01:46 +0000

How we combined Python, FastAPI, and LLMs to let users refine book translations on the fly.

Introduction

At LectuLibre, we help users translate entire books using AI. While LLMs like Claude and DeepSeek do an impressive job, they occasionally miss nuances—especially in literary texts loaded with idioms, cultural references, or complex sentences. Our early users told us they wished they could tweak just a few paragraphs here and there instead of regenerating entire chapters. That was the birth of our “翻译求助” feature: a simple way to select any passage and get an alternative translation with a brief explanation of why we changed it.

In this article, I’ll share the technical journey—the challenges we faced, the architectural decisions we made, and the code we wrote to make context-aware translation refinement work at scale.

The Problem: Static Translations in a Dynamic World

Initially, our translation pipeline was straightforward: upload a book → chunk it by chapter → send each chunk to an LLM → stitch the results. That worked, but it was a one-shot process. If a user disliked a particular sentence, they had no recourse except re-translating the whole chapter, losing any manual edits they might have made elsewhere.

We needed a way to let users interact with the translated text and ask for improvements without breaking the overall flow. The core requirements were:

Selective refinement: allow the user to highlight any passage and request a better translation.
Context awareness: the LLM must consider surrounding text to maintain coherence.
Transparency: provide a short explanation of what changed and why, so the user learns from it.
Speed: must feel nearly instant; no one waits 10 seconds for a sentence.

Our Approach: Design Decisions and Tooling

We decided to build a dedicated REST endpoint that accepts a passage index (start and end positions in the translated document), fetches the surrounding context from our database, constructs a tailored LLM prompt, and returns the refined translation plus an explanation.

Stack

Backend: Python, FastAPI for async IO performance.
LLMs: We use Anthropic’s Claude 3 Haiku for speed and DeepSeek-V2 for quality; the feature allows us to pick based on user settings.
Database: PostgreSQL stores the original book text, the translated text, and user edits. We’ll query for the surrounding segments using chunk indices.
Caching: Redis to store short-term LLM responses for identical context/wording to save costs.

Why Not a Chat Interface?

Some platforms use a conversational approach for translation refinement, but we found that asking the LLM to “improve this translation” via a chat leads to variable results. A structured prompt with explicit instructions gave us more control over the output format.

Implementation Details

1. The API Endpoint

We created a POST /api/translation-help endpoint. The request body contains book_id, segment_start, segment_end, and optionally the model preference. The response is a JSON object with refined_translation and explanation.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Optional

app = FastAPI()

class TranslationHelpRequest(BaseModel):
    book_id: str
    segment_start: int
    segment_end: int
    model: Optional[str] = "claude"  # or "deepseek"

@app.post("/api/translation-help")
async def get_translation_help(req: TranslationHelpRequest):
    if req.segment_start < 0 or req.segment_end < req.segment_start:
        raise HTTPException(status_code=400, detail="Invalid segment range")
    context = await fetch_context(req.book_id, req.segment_start, req.segment_end)
    refined = await refine_translation(context, model=req.model)
    await cache_refinement(req.book_id, req.segment_start, req.segment_end, refined)
    return refined

2. Context Retrieval: Feeding the Right Neighbors

Translation refinement without context often produces disjointed results. For example, if you ask the LLM to retranslate a sentence starting with “He laughed,” but the preceding paragraph ended with a sad tone, the output might be tonally wrong. Therefore, we fetch a window of N paragraphs before and after the target segment. Our book text is stored with paragraph-level indices.

async def fetch_context(book_id: str, start: int, end: int, window=3) -> dict:
    query = """
    SELECT paragraph_index, original_text, translated_text
    FROM book_translations
    WHERE book_id = $1 AND paragraph_index BETWEEN $2 AND $3
    ORDER BY paragraph_index
    """
    rows = await database.fetch_all(query, book_id, start - window, end + window)
    # ... extract target and surrounding text
    return {
        "target_original": target_original,
        "target_translated": target_translated,
        "context": "\n".join(surrounding_original),
        "context_translated": "\n".join(surrounding_translated),
    }

The window size is configurable; we found that 3 paragraphs on each side strikes a good balance between providing enough context and staying within token limits.

3. Crafting the LLM Prompt

Prompt engineering made all the difference. We wanted two clear parts in the response: the improved translation and a concise explanation. We designed a system prompt that sets the role and output format, and a user prompt that injects the text.

SYSTEM_PROMPT = """You are a professional literary translator assistant. Given a passage from a book and its current translation, your task is to improve the translation, paying attention to nuance, flow, and cultural accuracy. You must output a JSON object with exactly two fields: "refined_translation" and "explanation". Only output the JSON, no other text."""

USER_PROMPT_TEMPLATE = """Original passage: {target_original}
Current translation: {target_translated}
Surrounding context (original): {context}
Surrounding context (translated): {context_translated}
Please refine the translation of the passage above."""

def build_prompt(context_data):
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT_TEMPLATE.format(**context_data)}
    ]

We force JSON output because it’s easier to parse reliably. We use json.loads() after receiving the LLM response, with error handling in case the model returns malformed JSON.

4. Async LLM Calls and Fallback

Since we support multiple models, we abstract the LLM client. We use httpx for async HTTP calls to Claude’s API and DeepSeek’s API. To keep the endpoint responsive, we set a timeout of 8 seconds; if one model fails, we try the other.

import httpx
import json

async def refine_translation(context: dict, model: str):
    prompt = build_prompt(context)
    for attempt in [model, "claude", "deepseek"]:  # fallback order
        try:
            if attempt == "claude":
                resp = await call_claude(prompt)
            else:
                resp = await call_deepseek(prompt)
            parsed = json.loads(resp)
            return parsed
        except Exception:
            continue
    raise HTTPException(500, "Translation refinement failed")

We measured average latency: ~3.2s for Claude Haiku, ~4.7s for DeepSeek, both within our target.

5. Caching Strategy

Users often refine the same passage multiple times. To avoid redundant LLM calls, we cache the result keyed on (book_id, segment_start, segment_end, model). We store it in Redis with a TTL of 1 hour. If the surrounding context changes due to user edits elsewhere, we invalidate the cache for overlapping ranges. We used a simple versioning scheme: each book has a content_version that increments on any edit, and cache keys include the version. That’s conservative but correct.

Challenges and Trade-offs

Token limits: Book paragraphs can be long. We truncate the surrounding context to 2000 tokens using a simple heuristic (first 1000 characters per paragraph). Sometimes that cuts off important context; we plan to use a sliding window with dynamic sizing later.
Prompt injection: Users could select text that contains malicious instructions. We sanitize inputs, but LLMs are inherently vulnerable. We mitigate by strict output parsing and limiting the model’s temperature.
Cost: Each refinement call costs a few cents. Caching helps, but heavy users can rack up expenses. We added rate limits per user.
Cultural nuance: Explaining why a translation changed sometimes requires cultural background, which the LLM doesn’t always get right. We’re experimenting with RAG from a knowledge base of literary references.

Results and User Feedback

After launching the feature, we saw:

A 23% decrease in manual re-translation requests per book.
Users reported spending less time editing after using 翻译求助.
The explanation part surprised us—many non-native speakers use it to learn why idiomatic expressions are translated a certain way.

We’re now exploring how to let users provide feedback on the explanations to fine-tune our prompts further.

Lessons Learned

Context is king: Expanding the window significantly improved output quality. We initially used only the current paragraph, and the results were often stylistically mismatched.
JSON output constraints: Forcing the LLM to output JSON with a specific schema reduced parsing errors and made the feature more robust.
Caching is non-trivial: Simple caching works until users start editing the surrounding text; you need a strategy that propagates changes.
Async matters: FastAPI’s async capabilities kept our endpoint responsive under load.

What’s Next?

We’re considering allowing users to give natural language instructions alongside the passage (e.g., “make this more formal” or “use simpler words”). That would turn 翻译求助 into a full-blown iterative translation assistant.

We’d love to hear from the community: How do you handle context injection for LLM-based editing tasks? Any caching patterns that worked better for mutable text? Drop your thoughts in the comments.

Building Real-Time AI Translation Assistance with FastAPI, Claude, and Server-Sent Events

龚旭东 — Sat, 01 Aug 2026 03:02:37 +0000

How we added an on-demand translation help feature to our book translation platform, streaming LLM suggestions for tricky passages.

At LectuLibre, our AI-powered book translation service allows users to upload EPUB or PDF files and get translations generated by large language models like Claude and DeepSeek. But we quickly noticed a pain point: automated translations, while fast, sometimes produced awkward or ambiguous results for culturally specific phrases, idioms, or technical jargon. Users wanted a way to get instant, contextual help for these tricky passages without leaving the platform. That’s when we set out to build the 翻译与转录求助 (Translation Assistance) feature — an interactive side panel where users can select any sentence or paragraph and receive alternative translations, explanations, and stylistic suggestions from an LLM in real time.

In this article, I’ll walk you through the engineering challenge, the architecture we chose, and the specific code and trade-offs that made it work smoothly under production constraints.

The Problem: Real-Time, Context-Aware Translation Help

The core requirement was simple: a user highlights a piece of text in the translated book and clicks “Get Assistance”. Immediately, the system should stream back multiple translation options, a brief explanation of differences, and stylistic notes — all aware of the surrounding context, the author’s style, and the target language.

Under the hood, this meant:

Low latency: Users expect a response in under 2 seconds.
Streaming: The LLM output can be long, so we needed to stream tokens as they are generated.
Context awareness: We must include enough surrounding text from the book to ground the model’s response.
No blocking: The main translation pipeline shouldn’t be affected; the assistance feature should exist as an independent async service.
Cost efficiency: Avoid re-processing the entire book each time a user asks for help.

Our Approach: Async FastAPI + SSE + Rate Limiting

We run a Python/FastAPI backend on a VPS, with the existing translation pipeline making batch calls to Claude’s API for full-book translation. For the assistance feature, we built a separate endpoint that accepts a book ID, a start/end offset, and streams back assistance via Server-Sent Events (SSE). The flow:

The frontend sends a POST request with the selected text offsets.
The backend retrieves the surrounding paragraphs from our PostgreSQL database (already chunked during ingestion).
A carefully engineered prompt is composed, including the selected text, its context, and instructions for multiple translation variants.
We call Claude’s streaming API (messages.create(stream=True)) and forward each chunk as an SSE event.

Database and Ingestion Setup

When a book is uploaded, we parse the EPUB/PDF, extract text, and store it as chunks (paragraphs) with their position and metadata. In PostgreSQL, we have a table book_chunks with columns book_id, chunk_index, text, and lang. This allows us to quickly fetch the chunk containing the selected offset plus a few neighbors with a simple query:

async def get_context_chunks(book_id: str, chunk_index: int, context_size: int = 2):
    query = """
        SELECT chunk_index, text
        FROM book_chunks
        WHERE book_id = :book_id
          AND chunk_index BETWEEN :start_idx AND :end_idx
        ORDER BY chunk_index
    """
    values = {
        "book_id": book_id,
        "start_idx": max(0, chunk_index - context_size),
        "end_idx": chunk_index + context_size,
    }
    return await database.fetch_all(query, values)

This gives us the surrounding text without pulling the entire book into memory.

The Streaming Endpoint

We used FastAPI’s StreamingResponse with a generator that yields SSE-formatted messages. The trick is that the Claude SDK’s stream=True returns an async iterator, so we can await each chunk and yield it immediately:

from fastapi import APIRouter, Request
from fastapi.responses import StreamingResponse
import anthropic

router = APIRouter()
client = anthropic.AsyncAnthropic()

@router.post("/assist/stream")
async def stream_assistance(request: Request):
    data = await request.json()
    book_id = data["book_id"]
    chunk_index = data["chunk_index"]
    selected_text = data["selected_text"]

    context_chunks = await get_context_chunks(book_id, chunk_index)
    context = "\n".join([c["text"] for c in context_chunks])

    prompt = build_assistance_prompt(context, selected_text)

    async def event_generator():
        try:
            async with client.messages.stream(
                model="claude-3-5-sonnet-20240620",
                max_tokens=1024,
                temperature=0.7,
                messages=[{"role": "user", "content": prompt}],
            ) as stream:
                async for text in stream.text_stream:
                    yield f"data: {text}\n\n"
                yield "data: [DONE]\n\n"
        except anthropic.APIStatusError as e:
            yield f"data: ERROR: {e.message}\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

This gives the frontend a real-time, word-by-word streaming experience. The event_generator is an async generator thanks to stream.text_stream, which is a core feature of the Anthropic SDK.

Prompt Engineering for Multiple Variants

To get useful output, we crafted a prompt that asks for three distinct translation options, each with a brief note. We also instruct the model to preserve context and provide a style guide. A simplified version:

def build_assistance_prompt(context: str, selected: str) -> str:
    return f"""You are a professional literary translator. Given the following context from a book:

{context}

The user has selected this passage for translation help:
"{selected}"

Provide:
1. Three alternative translations into the target language, labeled Option A, B, C.
2. For each option, a one-sentence explanation of the stylistic or semantic choice.
3. A final recommendation with justification.

Respond in JSON format with keys: options (list of dicts with translation, explanation), recommendation.
"""

We then parse the JSON on the frontend to display the alternatives cleanly. The streaming text is accumulated and displayed as it arrives, giving the impression of immediate response.

Trade-Offs and Challenges

Latency vs. Completeness

Streaming helps perceived performance, but the time to first token is still around 1-2 seconds for a 150-token context. We considered pre-fetching context embeddings to speed up retrieval, but the overhead of the LLM call dominates. So we optimized by:

Using the async SDK to avoid blocking the event loop.
Implementing a simple cache in Redis for identical queries (same book, same chunk, same selected text) with a 5-minute TTL. This cut repeated requests by about 30%.
Choosing a relatively small max_tokens (1024) because the response is short.

Cost Management

Each assistance call burns tokens. With Claude 3.5 Sonnet, input tokens are $3/MTok, output $15/MTok. We track usage per user and implemented a daily limit of 50 assistance requests. If a user exceeds it, they get a polite message. This was done using a simple rate limiter with Redis:

import aioredis
import time

redis = aioredis.from_url("redis://localhost")

async def check_rate_limit(user_id: str, limit: int = 50, window: int = 86400) -> bool:
    key = f"assist_rate:{user_id}"
    current = await redis.incr(key)
    if current == 1:
        await redis.expire(key, window)
    return current <= limit

Model Choice

We initially used DeepSeek because it’s cheaper, but the translation quality for literary nuance was noticeably lower, and the streaming API was less stable. Claude’s output was richer, so we settled on that despite the cost. For non-literary works, we might offer a toggle to DeepSeek.

Context Window and Token Limits

We limited context to 3 paragraphs (roughly 500 tokens) to stay well within the model’s 200K context window while keeping prompt costs low. That’s usually enough for the model to understand tone and style.

Results and Lessons Learned

After deploying, the “翻译与转录求助” feature quickly became one of the most used parts of LectuLibre. Users loved the immediacy and the ability to see multiple translation options side by side. Average response time from click to first token is 1.4 seconds, and a complete response (3 options) takes about 5-7 seconds, which users found acceptable.

Key takeaways:

Streaming is non-negotiable for real-time LLM apps; always use SSE or WebSockets.
Caching pays off: simple query-level caching can significantly reduce load and costs.
Async truly matters: using asyncio and async SDKs prevents thread exhaustion under load.
Prompt structure controls quality and frontend parsing: asking for JSON output simplifies UI rendering but may constrain the model’s creativity; we found a middle ground by instructing JSON format but allowing free-form explanations inside.

What’s Next?

We’re considering adding user feedback to fine-tune the assistance model, and possibly exposing the feature as a standalone API for translators. We’d love to hear how other teams handle real-time LLM streaming at scale — especially around batching requests and reducing token costs. Drop your thoughts in the comments!

Parsing and Rebuilding EPUB Files in Python: Lessons from Building an AI Book Translator

龚旭东 — Wed, 29 Jul 2026 03:02:19 +0000

How we used ebooklib and Beautiful Soup to process thousands of EPUBs without losing metadata, formatting, or our sanity.

The Dream: One-Click Book Translation

When we started building LectuLibre, an AI-powered book translation service, we knew the core challenge wouldn't be the translation itself. LLMs like Claude and DeepSeek are remarkably good at handling text. The real headache was the book container: EPUB files. Users upload an EPUB, we translate it, and they download a perfectly formatted translated book. Sound simple? So did we—until we actually tried it.

EPUB is a deceptively complex format. It's a ZIP archive containing XHTML chapters, CSS, images, fonts, and a few XML control files (notably the content.opf and toc.ncx). To translate a book, you must:

Extract all text from the XHTML files while preserving the surrounding markup.
Translate only the text nodes, leaving tags, anchors, and images untouched.
Rebuild the EPUB with the same metadata, spine order, and resources.

This is a parsing and restructuring problem that sits somewhere between web scraping and document assembly. Here's how we tackled it with Python, and what we learned along the way.

Why Not Just Use Calibre?

Calibre is the Swiss Army knife of ebooks, but it's a desktop application, not a library we could embed in a FastAPI service. We needed something lightweight, scriptable, and open-source. Enter ebooklib, a pure-Python library that reads and writes EPUB files. It does the job, but as we discovered, it has quirks you'll only find when you throw thousands of real-world EPUBs at it.

Our Parsing Pipeline: ebooklib + Beautiful Soup

The basic workflow with ebooklib looks like this:

from ebooklib import epub

book = epub.read_epub('book.epub')

That one-liner unpacks the ZIP, parses the XML, and gives you a Book object. But the devil is in the details. Here's the actual production code we landed on:

import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup

def parse_epub(filepath: str):
    book = epub.read_epub(filepath, options={'ignore_ncx': False})
    items = list(book.get_items_of_type(ebooklib.ITEM_DOCUMENT))

    chapters = []
    for item in items:
        content = item.get_content().decode('utf-8')
        soup = BeautifulSoup(content, 'lxml-xml')  # Use XML parser to avoid HTML closing tag issues
        # Extract text nodes while preserving structure
        chapters.append({
            'id': item.get_id(),
            'soup': soup,
            'href': item.get_name()
        })
    return book, chapters

We used lxml-xml as the BeautifulSoup parser because EPUB XHTML is often XML, not HTML. With the default HTML parser, self-closing tags like   got mangled. A subtle but critical choice.

Navigating the OPF Spine

The content.opf defines the reading order via the <spine> element. Ebooklib represents this as book.spine. Each spine item has an idref that points to a manifest item. So to walk the book in order:

spine_order = []
for spine_item in book.spine:
    item = book.get_item_with_id(spine_item[0])
    if item.get_type() == ebooklib.ITEM_DOCUMENT:
        spine_order.append(item)

But we found that some EPUBs have spine entries pointing to non-existent IDs. Our approach: silently skip them and log a warning. Robustness over strictness.

Translating In-Place Without Breaking Markup

After extracting the text, we send it to an LLM in chunks (respecting token limits). The tricky part is replacing the original text with the translated version while keeping every tag, class, and attribute intact.

We traverse the BeautifulSoup tree and only replace NavigableString nodes:

def translate_soup(soup: BeautifulSoup, translation_map: dict) -> BeautifulSoup:
    for text_node in soup.find_all(string=True):
        stripped = text_node.strip()
        if stripped and stripped in translation_map:
            text_node.replace_with(translation_map[stripped])
    return soup

This works for simple cases, but we quickly ran into issues when the translation changed the number of paragraphs (e.g., a single  became two). The solution? We stitch translations back by splitting on paragraph boundaries and reassembling manually. It's not perfect, but it handles 95% of cases.

Rebuilding the EPUB: Writing Back Without Losing Your Mind

Once all soup objects have been translated, we rebuild the EPUB:

def save_epub(book, chapters, output_path):
    for ch in chapters:
        item = book.get_item_with_id(ch['id'])
        # Update content with translated soup
        item.set_content(str(ch['soup']).encode('utf-8'))

    # Write to disk
    epub.write_epub(output_path, book)

Seems simple, but ebooklib's write_epub can create invalid archives if you’ve added or removed items without updating the manifest and spine. Our rule: never add or remove items; only modify existing XHTML documents. This preserves the existing content.opf and avoids UUID mismatches.

A Word on Memory

Processing a 200 MB EPUB with hundreds of images can spike memory to over 1 GB with naive loading. We avoid that by selectively loading only document items and not images:

book = epub.read_epub(filepath, options={'ignore_ncx': False, 'expand_css': False})
# Don't load images or fonts unless needed
for item in book.get_items():
    if item.get_type() != ebooklib.ITEM_DOCUMENT:
        item.content = b''  # Clear binary content to free memory early

This dropped memory usage by 60% on large files.

Dealing with Concurrency

In our FastAPI backend, we handle uploads asynchronously. We stream the file to a temporary location, then hand it off to a translation worker. Because the translation step is CPU-bound and memory-heavy, we use a ThreadPoolExecutor with a limited number of workers (max cpu_count()). This keeps our VPS responsive and avoids OOM kills.

Wrestling with Malformed EPUBs

Real-world EPUBs are a mess. We've encountered:

Missing content.opf → fall back to looking for any .opf file in the ZIP.
Non-UTF-8 encodings → we use chardet to detect encoding before decoding.
Cyclic references in the manifest → we added a visited set to our parsing loop.
Empty spine → we treat all XHTML files as a reading order.
CSS with @import → we inline critical styles because Kindle doesn't support imports.
External fonts loaded from CDNs → we swap in a local fallback.
CDATA sections wrapping entire documents → we had to pre-process the XML to unwrap them.

We implemented a preflight validation step that checks for these issues and tries to auto-heal where possible. For the tough cases, we notify the user they uploaded a "quirky" file, and our AI will still do its best.

The Table of Contents Conundrum

Many translators overlook the navigation. We initially left the toc.ncx (or nav.xhtml for EPUB3) untouched—but then users complained that the table of contents was still in the original language. We're now developing a separate step that translates the nav labels (the <navLabel> elements) using the same LLM, but with a simpler context. The challenge is that some labels are just numbers or chapter abbreviations, and over-translating breaks the user experience. We're exploring a hybrid approach: only translate labels that contain natural language.

Performance and Scalability Numbers

Our backend is a FastAPI app on a 4‑core VPS. A typical translation workflow takes:

Parsing: 0.2–0.5 seconds for a 300‑page novel.
Translation (LLM call): 30–50 seconds (the bottleneck by far).
Rebuilding EPUB: 0.5–1.5 seconds depending on complexity.

We can handle 8 concurrent translations before hitting the LLM API rate limit or memory pressure. Our worker pool uses asyncio to manage these, with a semaphore limiting parallelism.

Testing with a Corpus of Horrors

To ensure we don't regress, we built a corpus of 500 reference EPUBs—including many deliberately broken ones. Before every deploy, we run integration tests that:

Check that the spine order is preserved.
Verify that the number of XHTML files remains unchanged.
Compare the translated output against a human‑reviewed gold standard for a subset.
Validate the output EPUB with epubcheck (the IDPF tool).

This saved us multiple times when a library update changed parsing behavior.

Lessons Learned

Never trust an EPUB. Always assume the XML might be invalid.
Separation of concerns: Keep parsing, translation, and rebuilding as separate, testable functions.
ebooklib is great but not bulletproof. For advanced features like media overlays or EPUB3 audio, you may need to manipulate the ZIP directly.
Memory matters. Don't load binary blobs you don't need.
Test with a corpus. We maintain a set of 500 reference EPUBs (including broken ones) to run before every deploy.
Navigation is part of the content. Translating the book but not the TOC leads to an inconsistent experience.

The Future: PDFs and Streaming

PDF parsing is next on our roadmap, with an entirely different set of challenges (we're eyeing pymupdf). But the EPUB pipeline taught us that ebooks are closer to web pages than to documents, and using web-native tools (BeautifulSoup, CSS inlining) pays off.

If you're building something similar, I'd love to hear your approach. Did you stick with ebooklib or roll your own? How do you handle RTL languages? Join the discussion in the comments.

—

LectuLibre is in private beta at lectulibre.com. We built this pipeline because we love books and believe language shouldn't be a barrier.

How We Built Precise Translation and Language Identification for AI Book Translation

龚旭东 — Sat, 25 Jul 2026 03:01:44 +0000

How we tackled 精准翻译与语言识别 (precise translation and language identification) for AI-powered book translation.

The Problem: Garbage In, Garbage Out

When we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation.

Users upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's langdetect library was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits.

We needed something far more robust—what we internally call 精准翻译与语言识别 (precise translation and language identification). Here’s how we built it.

The Language Detection Pipeline: From 85% to 98% Accuracy

Our first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language (e.g., French phrases in an English novel) would throw off chunk-level detection.

We realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation.

Combining Multiple Detectors with Voting

We created a LanguageDetector class that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are:

fastText with the official lid.176.bin model (loaded once, not per request)
langdetect, which is lightweight and good for long texts
cld3 (Compact Language Detector 3) from Google, which works well on short snippets
A custom fine-tuned fastText model on 10,000 book excerpts we curated from multilingual EPUBs (we'll open source this soon)

For a given text, we take the top-1 prediction from each detector, and if at least three of them agree, we trust that result. If there's a tie, we fall back to the one with the highest overall confidence (from the model's probability). If the user explicitly set a source language during upload, we honor that, but we still run detection and warn if a mismatch is strong.

Here's a simplified version of our detector:

import fasttext
import langdetect
import cld3

class LanguageDetector:
    def __init__(self):
        self.ft_model = fasttext.load_model('lid.176.bin')

    def detect(self, text, user_lang=None):
        # Clean text: replace newlines, keep only first 10000 chars
        clean = text.strip().replace('\n', ' ')[:10000]
        votes = []
        # fastText
        pred_ft = self.ft_model.predict(clean, k=1)
        lang_ft = pred_ft[0][0].replace('__label__', '')
        conf_ft = pred_ft[1][0]
        votes.append((lang_ft, conf_ft))
        # langdetect
        try:
            lang_ld = langdetect.detect(clean)
            votes.append((lang_ld, 1.0))  # langdetect doesn't give confidence
        except Exception:
            votes.append(('unknown', 0.0))
        # cld3
        pred_cld = cld3.get_language(clean)
        if pred_cld.is_reliable:
            votes.append((pred_cld.language, pred_cld.probability))
        else:
            votes.append(('unknown', 0.0))

        # Voting: count occurrences, pick max with tie-break on confidence
        lang_counts = {}
        for lang, conf in votes:
            lang_counts[lang] = lang_counts.get(lang, 0) + 1
        majority_lang = max(lang_counts, key=lambda l: (lang_counts[l], 
                            sum(c for v_l, c in votes if v_l == l)))
        if majority_lang != 'unknown' and lang_counts[majority_lang] >= 3:
            # High confidence, possibly override user_lang with warning
            if user_lang and user_lang != majority_lang:
                # log warning, but use detection
                return majority_lang
            return majority_lang
        # If no consensus, fallback to user setting or the highest confidence from all
        if user_lang:
            return user_lang
        # Pick the highest confidence individually
        best = max(votes, key=lambda x: x[1])
        return best[0]

In production, we load the fastText model at startup to avoid per-request overhead. We also cache detection results per book (based on a hash of the first 10k characters plus random sampling) to speed up subsequent operations.

Handling Multilingual Books

For books that contain multiple languages (e.g., a language textbook with side-by-side translation), we don't require detection at the book level. Instead, we first try to identify the dominant language (the one with >80% of the pages). If the user requests translation, we only translate chunks that match the source language and leave the rest untouched. This requires per-chunk language identification. Our pipeline runs detection on each chunk (around 1000 tokens) before sending it to the LLM, and skips chunks that are clearly not the target language. That prevents the LLM from halting on foreign phrases.

Chunking for LLMs: Preserving Context Without Breaking the Bank

Large language models are great at translation, but they have token limits and cost per token. Books are long, and sending them whole would bust context windows and API budgets. We needed a chunking strategy that keeps enough context for accurate translation while minimizing token waste.

Dynamic Chunking with Overlap

We use TikToken (OpenAI's tokenizer) to count tokens (also works for Claude's tokenizer which is similar). We set a maximum chunk size of 3,500 tokens (to leave headroom for the system prompt and response). We also maintain a 200-token overlap with the previous chunk to provide context continuity. This is crucial for sentences that span paragraph boundaries.

import tiktoken

def chunk_text(text, max_tokens=3500, overlap=200):
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunk = tokens[start:end]
        # if we're not at the beginning, prepend overlap from previous chunk
        if start > 0:
            overlap_start = max(0, start - overlap)
            prepend = tokens[overlap_start:start]
            chunk = prepend + chunk
        chunks.append(enc.decode(chunk))
        start = end
    return chunks

We then pass each chunk to the LLM with a system prompt that includes the source and target languages, and instructions to preserve formatting (markdown, special characters). We use Anthropic's Claude 3 Haiku for cost efficiency, but fall back to Claude Sonnet for complex passages when the model returns a confidence score below a threshold.

Async Translation Pipeline with Rate Limiting

Translating a full book might require 50+ chunks. We run these in parallel using asyncio, but we must respect API rate limits (for Anthropic, it's 5 requests per second on our tier). We implemented a token bucket rate limiter and semaphore:

import asyncio
from aiolimiter import AsyncLimiter

rate_limiter = AsyncLimiter(5, 1)  # 5 requests per 1 second
sem = asyncio.Semaphore(10)  # max 10 in-flight

async def translate_chunk(chunk, src_lang, dst_lang):
    async with sem:
        async with rate_limiter:
            # Call Claude API
            ...

We also batch large books into groups of chunks and process them with a small number of workers. This keeps the API usage smooth and prevents 429 errors.

Quality Control and Fallback

Even with good language detection and chunking, LLMs sometimes produce translations that are literal and miss literary nuance. For academic books, that might be fine, but for novels, we needed a way to improve fluency. We added a post-processing step where we run each translated chunk through a second LLM call (a smaller model like Claude Haiku with a higher temperature) to "polish" the language only if the user selected the "literary" translation mode. This adds cost but significantly improves readability. We found that around 30% of users opt for literary mode.

Results and Lessons Learned

After deploying the multi-model language detector, our language identification accuracy on a test set of 5,000 book excerpts jumped from 85% to 98.2%. Misidentifications dropped to rare edge cases (e.g., very short texts in dialect). In production, false language detection errors fell to near zero, and user complaints about garbled translations ceased.

The chunking strategy with overlap reduced API retries due to incomplete sentences by 40%, and the literary post-polish improved average user readability ratings (from A/B tests) by 15% on fiction books.

We also saved about 20% on API costs because the better language detection meant we weren't wasting credits on incorrect translations, and the dynamic chunking allowed us to pack more text per API call.

Key Takeaways for Developers

Don't trust a single detector; ensemble methods are far more reliable for language ID.
Always run per-chunk verification before translating to catch mixed-language content.
Overlap is essential when chunking for LLM translations—200 tokens of context made a big difference in our tests.
Rate limiting and async are your friends; they turn a fragile pipeline into a robust one.

Open Questions

We're still exploring: How to handle ancient languages or highly specialized jargon? Is there a way to dynamically adjust chunk size based on sentence boundaries? We'd love to hear from the community—especially anyone who's tackled translation of technical manuals or poetry.

LectuLibre's translation pipeline continues to evolve, and we're planning to open-source our language detection ensemble and curated dataset. If you're building something similar, reach out!

How We Translate Entire Books with LLMs Without Losing Context

龚旭东 — Wed, 22 Jul 2026 03:02:04 +0000

Solving the context-window puzzle for book-length AI translation.

At LectuLibre, we set out to build a service that translates entire books using large language models. The idea is simple: upload an EPUB or PDF, choose a language, and receive a polished translation. But behind the scenes, translating a hundred-thousand-word novel with LLMs isn't straightforward. The core challenge is context — LLMs have limited context windows, and books are long. Simply chopping the text into chunks and feeding each one independently leads to incoherent output. Character names change, pronouns lose referents, and tone veers wildly. Here’s how we solved that with a chunking strategy that preserves context, and the Python code that makes it tick.

The Problem: Long Documents vs. Short Context Windows

Modern LLMs like Claude 3 Opus can handle 200,000 tokens of context, while DeepSeek-V2 offers 128,000 tokens. That’s a lot — but a 50,000-word English novel translates to roughly 67,000 tokens (using Claude’s tokenizer). That just fits, but what about a 150,000-word fantasy epic? Even when it fits, sending an entire book in one prompt is costly, slow, and often degrades attention quality on long texts. The prevailing approach is to chunk the document.

Naive chunking — say, splitting by a fixed token count — creates hard boundaries. One chunk ends, another begins, and the LLM has no idea what happened before. The result reads like a patchwork of isolated translations. We needed a method that gives each chunk enough surrounding context without exceeding token limits or breaking the bank.

Our Approach: Sliding Window + Context Retrieval via Embeddings

We adopted a two‑pronged strategy:

Overlapping chunks: each chunk shares some sentences with the previous one, so the LLM can transition smoothly.
Injected context: for every chunk, we retrieve and prepend the most relevant previous chunks, determined by embedding similarity.

This way, the model always has a sense of what’s happening before and after the current segment. Overlap handles local continuity; similarity retrieval provides broader narrative context.

Step 1: Parse and Preprocess

First, we extract text from uploaded files. For PDFs we use PyPDF2 or pdfplumber; for EPUBs, ebooklib. The raw text is cleaned — excess whitespace removed, chapter titles detected (helpful for later splitting). We then split in paragraphs and then into sentences using spaCy for reliable sentence segmentation.

import spacy
nlp = spacy.load("en_core_web_sm")

def get_sentences(text: str) -> list:
    doc = nlp(text)
    return [sent.text.strip() for sent in doc.sents]

Step 2: Create Overlapping Token-Based Chunks

We built a custom chunker that respects sentence boundaries but packs as many tokens as possible into a chunk, while maintaining an overlap. We use tiktoken (Claude’s tokenizer) for accurate token counts.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")  # works for Claude models

def chunk_sentences(sentences: list, max_tokens=3000, overlap_tokens=500) -> list:
    chunks = []
    current_chunk = []
    current_tokens = 0

    for sent in sentences:
        sent_tokens = len(enc.encode(sent))
        if current_tokens + sent_tokens > max_tokens and current_chunk:
            # finalize current chunk
            chunks.append(' '.join(current_chunk))
            # start new chunk with overlap: keep last N tokens from previous chunk
            overlap_sents = []
            ov_tokens = 0
            for s in reversed(current_chunk):
                t = len(enc.encode(s))
                if ov_tokens + t <= overlap_tokens:
                    overlap_sents.insert(0, s)
                    ov_tokens += t
                else:
                    break
            current_chunk = overlap_sents + [sent]
            current_tokens = sum(len(enc.encode(s)) for s in current_chunk)
        else:
            current_chunk.append(sent)
            current_tokens += sent_tokens
    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

For a 50,000-word book with max_tokens=3000 and overlap_tokens=500, we get around 25–30 chunks. The overlap ensures that no sentence is cut off mid‑thought, and the model sees the tail of the previous chunk, reducing boundary artifacts.

Step 3: Context Injection with Embedding Similarity

Overlap helps locally, but for global coherence (character consistency, jargon) we need a broader view. We use sentence-transformers to embed each chunk (we embed only the first ~1000 characters to keep it fast). Before translating a chunk at index i, we compute cosine similarity with all previous chunks and pick the top 3 most similar ones to prepend as “context.”

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')  # small, fast, local

def embed_chunks(chunks: list) -> np.ndarray:
    return model.encode([c[:1000] for c in chunks], convert_to_numpy=True)

def get_context_for_chunk(chunks, embeddings, current_idx, top_k=3):
    if current_idx == 0:
        return []
    query_embedding = embeddings[current_idx]
    # Cosine similarity (vectors already L2‑normalized by the model)
    similarities = np.dot(embeddings[:current_idx], query_embedding)
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return [chunks[i] for i in top_indices]

The retrieved chunks are concatenated and placed above the chunk to translate. We guard against exceeding the model’s max context: if the prompt would be too long, we truncate the context or drop the least similar chunks.

Step 4: Translating with Asyncio

We call Claude and DeepSeek APIs asynchronously using httpx. We wrap each call with retry logic and rate‑limiting (a simple semaphore).

import asyncio
import httpx

async def translate_chunk(client, chunk, context_chunks, target_lang="Spanish"):
    context_text = "\n---\n".join(context_chunks)
    prompt = f"""You are a professional book translator. Below is context from earlier parts of the book.
Use it to maintain consistency.

CONTEXT:
{context_text}

TEXT TO TRANSLATE ({target_lang}):
{chunk}"""
    # Simplified API call – actual code includes system prompt, temperature, etc.
    response = await client.post(
        "https://api.anthropic.com/v1/messages",
        json={
            "model": "claude-3-opus-20240229",
            "max_tokens": 4096,
            "messages": [{"role": "user", "content": prompt}],
        },
        headers={"x-api-key": API_KEY, "anthropic-version": "2023-06-01"}
    )
    # ... error handling ...
    return response.json()["content"][0]["text"]

async def translate_book(chunks, embeddings):
    sem = asyncio.Semaphore(5)  # max concurrent API calls
    async with httpx.AsyncClient(timeout=60) as client:
        async def translate_one(i):
            async with sem:
                context = get_context_for_chunk(chunks, embeddings, i)
                return await translate_chunk(client, chunks[i], context)
        tasks = [translate_one(i) for i in range(len(chunks))]
        return await asyncio.gather(*tasks)

Results and Trade-offs

We translated a 50,000-word English fantasy novel into Spanish. With the naive chunking approach (no overlap, no context), we saw inconsistent character names (e.g., “Eldrin” became “Eldrín” in some chunks, “Eldrin” in others), and the narrative tone shifted abruptly. With our full pipeline:

Coherence improved dramatically. Names, places, and invented terminology remained consistent 98% of the time.
Total API calls: 31 chunks (with overlap) vs. 25 without – overlap adds a few extra calls but it’s negligible.
Cost: Claude 3 Opus, ~$0.015 per 1K input tokens. The entire book translation cost about $4.50 (including context injection). DeepSeek‑V2 was cheaper but we preferred Opus for nuance.
Embedding time: Generating embeddings for all chunks with all-MiniLM-L6-v2 took less than 2 seconds on a CPU, no noticeable overhead.

Trade-offs:

Sometimes the similarity search grabs a “context” chunk that isn’t truly relevant, leading to mild confusion. Filtering by minimum similarity score helped.
Overlap can cause duplicate content if the model inadvertently translates the overlapping part twice, but with careful overlap sizing (10–15% of chunk tokens) this was rare.
Rate limiting on the API side meant we couldn’t max out parallelism; we found 5 concurrent calls a safe spot for our tier.

Lessons Learned

Overlap is essential even with context injection. It smooths the mechanical join between chunks.
Embedding similarity is a cheap proxy for relevance. It works surprisingly well for narrative text, but for technical books you might need keyword‑based retrieval.
Async processing is a must. A synchronous loop would have taken hours; with asyncio the whole book translated in under 10 minutes.
Chapter boundaries are natural split points. We didn’t implement it here, but you can align chunks to chapter starts to reduce overlap and improve coherence.

What’s Next?

We’re exploring hierarchical summarization: instead of injecting raw context, generate a running summary of the book so far and feed that as context. That might be more efficient for very long works. Also, longer‑context models keep getting better — maybe one day we can just toss the whole book in.

Takeaway for developers: translating long documents with LLMs is a solved problem in principle, but nailing the details — chunk size, overlap, context retrieval — makes the difference between a decent translation and a great one. The code I shared is the backbone of our system at LectuLibre; adapt it to your own use case, and you’ll be 90% of the way there.

What strategies have you used for long‑form LLM tasks? Do you think summarization beats retrieval for context? We’d love to hear your ideas in the comments.