DEV Community: 龚旭东

Parsing and Rebuilding EPUB Files in Python: Lessons Learned from Building an AI Book Translator

龚旭东 — Sat, 17 Oct 2026 03:01:36 +0000

How we used ebooklib and lxml to extract, translate, and reconstruct EPUBs while preserving formatting and structure.

We started building LectuLibre, an AI-powered book translation service, with a simple premise: users upload an EPUB or PDF, we translate it using large language models (Claude, DeepSeek), and they download a translated copy. Sounds straightforward—but the devil is in the details, especially when you promise to preserve the original formatting, fonts, images, and structure. Here’s what we learned about parsing and rebuilding EPUB files in Python, and how we solved the trickiest parts.

The Problem: Translating Books While Keeping Everything Else Intact

EPUB is essentially a ZIP archive containing XHTML files, CSS, images, and metadata. To translate a book, we needed to:

Extract the text from each XHTML file in reading order.
Send the text to an LLM for translation (in chunks, but that's another story).
Replace the original text with the translated text, preserving the DOM structure, inline tags, and styling.
Rebuild the EPUB with updated metadata, navigation, and all assets intact.

Naively, you might think you can just unzip, regex-replace the text, and zip it back. But that breaks as soon as you encounter nested tags, split sentences across tags, or special characters. We needed a proper parser.

Evaluating Libraries: Why We Chose ebooklib + lxml

We looked at several options:

ebooklib: High-level library for reading/writing EPUBs. Handles the ZIP container, manifest, spine, and TOC. It gives you EpubBook and EpubItem objects but doesn't parse the HTML content—you get raw bytes/strings.
epubfile: Another library, but less maintained and less flexible.
Calibre's ebook-convert: Powerful but designed as a command-line tool, not for programmatic control inside a web service.
Direct zipfile + lxml: Full control but we'd have to implement the EPUB spec ourselves (manifest, spine, etc.).

We settled on ebooklib for the container-level work and lxml for parsing and modifying the XHTML content. This combination gave us the best balance of convenience and control.

Reading and Iterating Over EPUB Content

Here's a minimal example of loading an EPUB and iterating over its spine items (documents in reading order):

from ebooklib import epub
from lxml import etree

book = epub.read_epub('input.epub')

for item in book.get_items_of_type(epub.ITEM_DOCUMENT):
    # item.content is bytes
    content = item.content.decode('utf-8')
    tree = etree.fromstring(content.encode('utf-8'))
    # Now work with tree...

But there's a catch: namespaces. XHTML files declare a default namespace (xmlns="http://www.w3.org/1999/xhtml"), and lxml requires you to handle that. If you just do tree.xpath('//p'), you'll get nothing. You need to map the namespace.

ns = {'xhtml': 'http://www.w3.org/1999/xhtml'}
paragraphs = tree.xpath('//xhtml:p', namespaces=ns)

This was our first big "gotcha". We wrote a helper to strip namespaces or always use the namespace map.

Extracting Text in Paragraph Order

For translation, we needed to extract text chunks that made sense. Instead of grabbing all text nodes indiscriminately (which would flatten everything and lose structure), we walked the DOM and collected text per block-level element (p, h1, h2, etc.) while preserving inline tags.

Here's a simplified version of our extraction function:

def extract_paragraphs(tree):
    ns = {'x': 'http://www.w3.org/1999/xhtml'}
    blocks = []
    for elem in tree.xpath('//x:p | //x:h1 | //x:h2 | //x:h3 | //x:h4 | //x:h5 | //x:h6', namespaces=ns):
        text = elem.text_content().strip()
        if text:
            blocks.append(text)
    return blocks

But this flattened inline tags—losing  or  information. For translation, we actually wanted to preserve those tags because they carry meaning (emphasis, quotes, etc.). So we devised a more granular approach: we serialized the inner content of each block element to a compact markup string, sent that to the LLM with instructions to translate text but keep tags, and then parsed the result back.

That was complex and error-prone, so for v1 we settled on a simpler method: we translate the plain text of each paragraph, then insert the translated text into the original block element, stripping all inline tags. This meant losing italics/bold in translated text, which was acceptable for MVP but we knew we'd need to improve later. It's a trade-off we documented clearly.

Replacing Text Without Breaking the DOM

To replace text while preserving the block-level structure, we used lxml to set the text content of each block element. For a paragraph Hello world!, if we want to replace with "Bonjour le monde !", we could do:

new_p = etree.SubElement(parent, '{http://www.w3.org/1999/xhtml}p')
new_p.text = translated_text
# Replace old_p with new_p
old_p.getparent().replace(old_p, new_p)

But that removes any attributes or classes on the original paragraph. Better approach: keep the element, but remove all children and set .text:

for child in old_p:
    old_p.remove(child)
old_p.text = translated_text

This preserves the tag, attributes, and tail text (though tail text needed careful handling; we often used etree.strip_elements or manual tail merging).

Batch Translation and Reconstruction

In our pipeline, we don't translate paragraph by paragraph; that would be too slow and lose context. Instead, we extract all paragraphs from the entire book (in spine order), then send them to the LLM in batches (e.g., 50 paragraphs per request). The LLM returns translated paragraphs in the same order, and we map them back.

Here's how we apply the translations back to the document tree:

def replace_paragraphs(tree, translated_texts):
    ns = {'x': 'http://www.w3.org/1999/xhtml'}
    blocks = tree.xpath('//x:p | //x:h1 | //x:h2 | //x:h3 | //x:h4 | //x:h5 | //x:h6', namespaces=ns)
    for elem, new_text in zip(blocks, translated_texts):
        # Remove all children (inline tags) but keep the element itself
        for child in elem:
            elem.remove(child)
        elem.text = new_text
    return tree

We keep a global list of all block elements across all XHTML files, in order, and after translation, we iterate over the files again and update each element with the corresponding translated text. This ensures the reading order is preserved.

Handling Images and CSS

Images and CSS are separate EPUB items. We didn't need to modify them, just make sure they are preserved when rebuilding. ebooklib does this automatically: when you read an EPUB, all items are stored in the book object, and when you write, they are included. However, we had to ensure that the items we modified (the XHTML documents) maintained their correct names and paths.

One issue: when we modified an item's content, we had to set it back:

item.content = etree.tostring(tree, encoding='utf-8', xml_declaration=True)

We also had to be careful about the xml:lang attribute and the DOCTYPE declaration. ebooklib sometimes added or removed them inconsistently. We ended up post-processing the serialized string to fix it.

Rebuilding the Navigation (TOC)

EPUB3 uses a nav.xhtml file for the table of contents. ebooklib has a book.toc property, but it's often out of sync or not updated when you change document structure. Since we weren't changing the structure (just text inside paragraphs), we didn't need to modify the TOC. But for some books, we had to add language metadata or update the title.

We used ebooklib's metadata API:

book.set_title('Translated Title')
book.set_language('fr')
book.add_metadata('DC', 'description', 'Translated by LectuLibre')

Performance Numbers

For a typical 300-page novel (~100,000 words), our parsing and rebuilding pipeline (excluding LLM translation time) takes about 2-3 seconds on a standard VPS. The bottleneck is actually the epub.read_epub and epub.write_epub operations, which involve ZIP compression/decompression. The lxml DOM operations are fast (milliseconds per file). Since translation via LLM takes minutes, the overhead is negligible.

Challenges and Workarounds

Namespaces: As mentioned, always use namespace maps.
Self-closing tags: lxml sometimes converts  to , which can break some EPUB readers. We had to force full closing tags by using etree.tostring(..., method='html') or using the HTML parser. Actually, for XHTML we should use XML serialization, but we found that many EPUBs are not strictly valid XML, so we used the HTML parser for resilience.
Encoding: Most EPUBs use UTF-8, but occasionally we saw Windows-1252 or other encodings. ebooklib doesn't handle this well; we added detection using chardet.
Fixed-layout EPUBs: Some EPUBs are fixed-layout (images per page). Our text-based approach doesn't work for those. We detect them and fall back to a different pipeline (OCR or PDF-like processing). This is still an open area.
Broken EPUBs: Many EPUBs in the wild are not well-formed. We had to implement fallbacks: try strict XML parsing, then fall back to lxml.html parser, which is more forgiving.

Lessons Learned

Use ebooklib for container management, lxml for content manipulation. Don't try to reinvent the ZIP/OPF layer.
Always test with a diverse set of EPUBs. We collected a corpus of 50+ books from different sources and used them for regression testing.
Preserve the original DOM as much as possible. Even small changes to the structure (like removing empty spans) can break the layout.
Plan for inline tags early. Stripping them was a temporary solution; we're now working on a tag-preserving translation using a custom markup language and XML-aware replacement.

Open Question for the Community

How do you handle translation of text with inline formatting (bold, italics, links) without losing the semantics? We've experimented with sending serialized HTML to the LLM and asking it to return translated HTML, but reliability is hit-or-miss. If you've solved this elegantly, we'd love to hear about it.

Scaling Multilingual Search in a Book Translation Platform: FastAPI + PostgreSQL Full-Text Search Architecture

龚旭东 — Wed, 14 Oct 2026 03:01:29 +0000

Lessons learned implementing multilingual search and translation management in a production FastAPI app.

The Problem: Translating Books Means Storing and Searching Many Languages

At LectuLibre, we built a platform where users upload EPUB or PDF books, and our system translates them using LLMs like Claude and DeepSeek. The result is a multilingual library: the same book can exist in Spanish, French, German, and more. We needed a backend that could handle:

Storing original and translated text efficiently.
Performing fast full-text search across all languages, letting users find books or passages in their preferred language.
Managing asynchronous translation jobs without blocking the API.

We chose Python/FastAPI for the API layer and PostgreSQL for persistence. This article focuses on how we designed the database schema and search functionality to support multiple languages without adding complex infrastructure.

Our Approach: Keep It Simple, Leverage PostgreSQL

After evaluating options like Elasticsearch, we decided to stick with PostgreSQL’s built-in full-text search (FTS). For a single VPS deployment with a few hundred thousand documents, PostgreSQL’s FTS is more than capable and avoids operational overhead. We use SQLAlchemy 2.0 (async) with asyncpg driver, and Alembic for migrations.

The key design decisions:

Normalized translation table: each translation is a separate row with a language code, not a JSON blob.
Generated tsvector columns for each translation language, with a GIN index.
Language-specific text search configurations (e.g., english, spanish, french) and a simple fallback.
pg_trgm extension for fuzzy matching on book titles.
FastAPI async endpoints that run search queries directly against PostgreSQL.

This approach let us implement multilingual search with minimal code and excellent performance.

Code Walkthrough: Database Models and Search Endpoint

We'll show the essential parts.

Database Models

We have three core tables: books, chapters, and translations. Each translation belongs to a chapter and has a language code (ISO 639-1) and the translated text.

# models.py
from sqlalchemy import String, Text, ForeignKey, Index, text
from sqlalchemy.dialects.postgresql import TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class Book(Base):
    __tablename__ = "books"
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(500))
    # ... other metadata

class Chapter(Base):
    __tablename__ = "chapters"
    id: Mapped[int] = mapped_column(primary_key=True)
    book_id: Mapped[int] = mapped_column(ForeignKey("books.id"))
    order: Mapped[int]
    # ...

class Translation(Base):
    __tablename__ = "translations"
    id: Mapped[int] = mapped_column(primary_key=True)
    chapter_id: Mapped[int] = mapped_column(ForeignKey("chapters.id"))
    language: Mapped[str] = mapped_column(String(10))  # 'en', 'es', etc.
    content: Mapped[str] = mapped_column(Text)
    # Generated tsvector column (created via migration)
    content_tsv: Mapped[str] = mapped_column(TSVECTOR, nullable=True)

We create the content_tsv column as a generated column in the migration, not directly in the model. This keeps the model simple and lets PostgreSQL handle the generation.

Migration snippet (Alembic):

# migrations/versions/xxxx_add_tsvector.py
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql

def upgrade():
    op.add_column('translations', sa.Column('content_tsv', postgresql.TSVECTOR()))
    # Generate tsvector from content using language-specific config
    op.execute("""
        UPDATE translations SET content_tsv = 
            to_tsvector(
                CASE 
                    WHEN language = 'en' THEN 'english'
                    WHEN language = 'es' THEN 'spanish'
                    WHEN language = 'fr' THEN 'french'
                    ELSE 'simple'
                END,
                content
            )
    """)
    op.create_index('ix_translations_content_tsv', 'translations', ['content_tsv'], postgresql_using='gin')

We also create a trigger to keep content_tsv in sync when content changes. We use a PostgreSQL trigger function:

CREATE OR REPLACE FUNCTION translations_tsv_trigger() RETURNS trigger AS $$
BEGIN
    NEW.content_tsv := to_tsvector(
        CASE 
            WHEN NEW.language = 'en' THEN 'english'
            WHEN NEW.language = 'es' THEN 'spanish'
            WHEN NEW.language = 'fr' THEN 'french'
            ELSE 'simple'
        END,
        NEW.content
    );
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_translations_tsv BEFORE INSERT OR UPDATE ON translations
FOR EACH ROW EXECUTE FUNCTION translations_tsv_trigger();

This ensures content_tsv is always up-to-date.

Search Endpoint

We expose a search endpoint that accepts a query, optional language filter, and uses PostgreSQL's websearch_to_tsquery for user-friendly syntax (supports quotes, OR, etc.).

# api/search.py
from fastapi import APIRouter, Query
from sqlalchemy import select, func, text
from sqlalchemy.ext.asyncio import AsyncSession
from models import Translation, Book, Chapter

router = APIRouter()

@router.get("/search")
async def search(
    q: str = Query(..., min_length=1),
    lang: str = Query("en"),
    limit: int = Query(20, le=100),
    db: AsyncSession = Depends(get_db),
):
    # Map language code to PostgreSQL text search config
    config_map = {"en": "english", "es": "spanish", "fr": "french", "de": "german"}
    ts_config = config_map.get(lang, "simple")

    # Build query: join translations with chapters and books, filter by tsvector
    stmt = (
        select(
            Translation.id,
            Translation.language,
            Chapter.order,
            Book.title,
            # Highlight snippet
            func.ts_headline(
                ts_config,
                Translation.content,
                func.websearch_to_tsquery(ts_config, q),
                'StartSel=<mark>, StopSel=</mark>, MaxWords=50, MinWords=20'
            ).label("snippet"),
        )
        .join(Chapter, Translation.chapter_id == Chapter.id)
        .join(Book, Chapter.book_id == Book.id)
        .where(
            Translation.language == lang,
            Translation.content_tsv @@ func.websearch_to_tsquery(ts_config, q)
        )
        .order_by(
            func.ts_rank(Translation.content_tsv, func.websearch_to_tsquery(ts_config, q)).desc()
        )
        .limit(limit)
    )

    results = await db.execute(stmt)
    return {"results": [dict(row) for row in results.mappings()]}

The endpoint uses ts_headline to generate a snippet with highlighted matches, websearch_to_tsquery for query parsing, and ts_rank for relevance ranking. All done in a single SQL query.

Fuzzy Title Search

For book titles, we added trigram similarity to allow misspellings. We enable the pg_trgm extension and create a GIN index on title using gin_trgm_ops.

# migration
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
op.create_index('ix_books_title_trgm', 'books', ['title'], postgresql_using='gin', postgresql_ops={'title': 'gin_trgm_ops'})

Then in a search endpoint for titles, we can use:

stmt = select(Book).where(
    func.similarity(Book.title, q) > 0.3
).order_by(func.similarity(Book.title, q).desc()).limit(10)

Performance Numbers and Trade-offs

Before adding the GIN index and generated tsvector, a search query across 10,000 translations took ~800ms because it had to scan and cast text on the fly. After the index, the same query dropped to ~20-40ms, even with 100,000 translations. The pg_trgm index on titles improved fuzzy title search from 1.2s to 15ms.

The biggest trade-off was choosing PostgreSQL FTS over Elasticsearch. For our scale (a few hundred thousand rows), PostgreSQL is simpler to operate, requires no extra services, and integrates perfectly with our existing relational data. However, we know that if we grow to millions of documents or need advanced features like faceting, we may need to revisit Elasticsearch.

Another trade-off: language-specific configurations. PostgreSQL supports many languages out of the box, but for languages like Chinese or Japanese, full-text search requires different tokenization (e.g., pg_jieba or zhparser). We haven't tackled that yet; currently, we fall back to the simple config, which splits on whitespace—not ideal for those languages. That's an area for future work.

We also learned that storing translations in a JSON column was a mistake initially. Querying into JSON for search was painful and slow. Normalizing into a separate table made everything cleaner and allowed us to use PostgreSQL's FTS directly.

Lessons Learned

Leverage PostgreSQL's built-in features before adding new infrastructure. It saved us from managing Elasticsearch and kept our deployment simple.
Use generated columns and triggers to keep search vectors in sync. It eliminates the need for application-level index updates.
Async SQLAlchemy with FastAPI works well, but be mindful of session management. We use async_sessionmaker and dependency injection to ensure sessions are closed after each request.
Always test with real data volume. Our initial search was fine with 1,000 rows, but degraded quickly; adding indexes and generated columns was a straightforward fix once we measured.

Open Question for the Community

How do you handle full-text search for languages that PostgreSQL doesn't support natively (e.g., CJK)? We're considering integrating a tokenizer like pg_jieba for Chinese, but we'd love to hear about your experiences or alternatives.

Building a Multilingual Book Search with FastAPI and PostgreSQL Full-Text Search

龚旭东 — Sat, 10 Oct 2026 03:01:59 +0000

How we store, index, and search book translations across 30+ languages using PostgreSQL's built-in full-text search.

At LectuLibre, we let users upload books in any language and get AI-powered translations into dozens of languages. That means our database stores not just the original text, but multiple translated versions of every paragraph. When a user searches for a phrase, they expect results across all languages—and they expect it fast. Here’s how we built that search system using FastAPI and PostgreSQL’s full-text search, and what we learned along the way.

The Problem: Searching Across Languages

Our platform ingests EPUB and PDF files, extracts the text, splits it into paragraphs, and then runs each paragraph through an LLM translation pipeline. A single book can produce translations in 30+ languages, meaning every original paragraph spawns dozens of translated rows. We needed to:

Store millions of paragraphs and their translations efficiently.
Provide keyword search across all content, regardless of language.
Return relevant results with sensible ranking.
Keep query times under 100ms for a responsive UI.

Initially we considered Elasticsearch, but our scale (hundreds of thousands of paragraphs, not billions) didn’t justify the operational overhead. PostgreSQL’s built-in full-text search promised enough features for our needs: language-specific stemming, ranking, and GIN indexes. So we decided to go all-in on PostgreSQL.

Our Approach: Normalized Tables + Per-Language tsvector Columns

Data Model

We use SQLAlchemy 2.0 with asyncpg for async database access. Our core tables are books, chapters, paragraphs, and translations. Here’s the simplified model:

from sqlalchemy import String, ForeignKey, Text, Index, UniqueConstraint
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
from sqlalchemy.dialects.postgresql import TSVECTOR

class Base(DeclarativeBase):
    pass

class Book(Base):
    __tablename__ = "books"
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(500))
    source_language: Mapped[str] = mapped_column(String(10))

class Chapter(Base):
    __tablename__ = "chapters"
    id: Mapped[int] = mapped_column(primary_key=True)
    book_id: Mapped[int] = mapped_column(ForeignKey("books.id"))
    order: Mapped[int]

class Paragraph(Base):
    __tablename__ = "paragraphs"
    id: Mapped[int] = mapped_column(primary_key=True)
    chapter_id: Mapped[int] = mapped_column(ForeignKey("chapters.id"))
    order: Mapped[int]
    original_text: Mapped[str] = mapped_column(Text)

class Translation(Base):
    __tablename__ = "translations"
    id: Mapped[int] = mapped_column(primary_key=True)
    paragraph_id: Mapped[int] = mapped_column(ForeignKey("paragraphs.id"))
    language: Mapped[str] = mapped_column(String(10))
    translated_text: Mapped[str] = mapped_column(Text)
    # We'll add a tsvector column per language later
    __table_args__ = (
        UniqueConstraint("paragraph_id", "language", name="uq_paragraph_language"),
    )

We chose a normalized structure rather than storing translations in a JSONB column. The reason: we wanted to create separate GIN indexes for each language’s full-text search vector. If we had a single JSONB column, PostgreSQL couldn’t efficiently index the individual language fields for FTS.

Generating tsvector Columns

The key to fast full-text search is precomputing a tsvector column for each translation. PostgreSQL 12+ supports generated columns, which are automatically updated when the source column changes. We use a generated column that applies the language-specific text search configuration:

from sqlalchemy import Computed, text

class Translation(Base):
    # ... existing fields ...
    search_vector: Mapped[str] = mapped_column(
        TSVECTOR,
        Computed(
            "to_tsvector(language::regconfig, translated_text)",
            persisted=True
        ),
        nullable=True
    )

But here’s a catch: language is stored as a string like 'english', 'spanish', 'french'. PostgreSQL’s to_tsvector expects a regconfig value, which is an OID. We use the cast language::regconfig to convert the language name to its configuration. This works as long as the language name exactly matches a built-in configuration (english, spanish, french, german, etc.). For languages that don’t have a built-in configuration (e.g., Chinese, Japanese), we handle them separately—more on that later.

We then create a GIN index on search_vector per language to speed up queries:

CREATE INDEX idx_translations_search_vector_english ON translations USING GIN (search_vector) WHERE language = 'english';
CREATE INDEX idx_translations_search_vector_spanish ON translations USING GIN (search_vector) WHERE language = 'spanish';
-- ... repeat for all supported languages

This partial index approach keeps each index smaller and more efficient than a single huge GIN index on the whole table.

Searching with SQLAlchemy

Our FastAPI endpoint receives a query string and an optional target language filter. We use websearch_to_tsquery to parse user input (supports quotes, OR, minus, etc.) and join translations with paragraphs and books. Here’s a simplified async query:

from sqlalchemy import select, func, or_, and_
from sqlalchemy.dialects.postgresql import websearch_to_tsquery, ts_rank

async def search_books(query: str, language: str | None = None, limit: int = 20):
    tsquery = websearch_to_tsquery(language or 'simple', query)
    stmt = (
        select(
            Book.id,
            Book.title,
            Paragraph.order,
            Translation.language,
            Translation.translated_text,
            ts_rank(Translation.search_vector, tsquery).label('rank')
        )
        .select_from(Translation)
        .join(Paragraph, Translation.paragraph_id == Paragraph.id)
        .join(Chapter, Paragraph.chapter_id == Chapter.id)
        .join(Book, Chapter.book_id == Book.id)
        .where(Translation.search_vector.op('@@')(tsquery))
        .order_by(desc('rank'))
        .limit(limit)
    )
    if language:
        stmt = stmt.where(Translation.language == language)

    result = await session.execute(stmt)
    return result.all()

Important note: We pass the language to websearch_to_tsquery as the first argument (config name). If the user searches without specifying a language, we default to 'simple', which doesn’t apply stemming but still tokenizes properly. We could also search across all languages by using multiple tsquery configurations, but that would require separate queries and merging results. For now, we let users pick a language or search in the original language.

Handling CJK Languages

PostgreSQL’s built-in full-text search works great for European languages with word boundaries and stemming, but Chinese, Japanese, and Korean (CJK) don’t use spaces between words. The default simple configuration treats each character as a separate token, which is useless for phrase search. We tried a few approaches:

pg_trgm trigram similarity – Good for fuzzy matching, but not true full-text search. We use it for ILIKE-style queries.
Custom text search parser – e.g., zhparser for Chinese, but it requires compiling and installing a PostgreSQL extension, which we wanted to avoid on our managed VPS.
External tokenization – We use the jieba Python library during ingestion to segment Chinese text into words, then store the segmented text in a separate column (translated_text_segmented) and run FTS on that with the simple configuration.

We went with option 3 because it keeps everything inside PostgreSQL and only adds a preprocessing step in Python. For Japanese, we use fugashi with mecab-ipadic. For Korean, we use mecab-ko. The segmented text column has its own generated tsvector column and index.

Performance Numbers

We benchmarked with 1.2 million translation rows across 40 languages (mostly English, Spanish, French, German).

Before GIN indexes: a typical search query took 2–4 seconds because PostgreSQL had to scan and compute tsvectors on the fly.
After adding per-language GIN indexes on generated columns: median query time dropped to 35ms, p95 at 80ms. That’s well within our 100ms target.
Index size: the GIN indexes added about 15% overhead to the table size, which was acceptable.

We also tested using a single composite GIN index on search_vector without partial indexes. Performance was similar but index creation time was longer and the index was larger.

Lessons and Trade-offs

PostgreSQL FTS is powerful but not a replacement for Elasticsearch if you need advanced features like synonym dictionaries, custom analyzers, or distributed search. For our current scale, it’s perfect.
Generated columns simplify maintenance but you must be careful when changing the text. Since the vector is persisted, updates to translated_text automatically update the vector and index (with a small write overhead).
CJK requires extra work outside the database. We originally tried to use PostgreSQL’s built-in simple config with trigram indexes and found it lacking for phrase search. The Python tokenization step added ~50ms per paragraph during ingestion, which is negligible compared to LLM translation time.
Language configuration names must match exactly – typos lead to errors. We validate language codes at ingestion time and store only supported configurations.
Async SQLAlchemy with asyncpg gave us excellent concurrency for search endpoints, easily handling hundreds of requests per second on a single VPS.

What’s Next

We’re considering moving to a dedicated search engine like Meilisearch or Typesense for more advanced relevance tuning and typo tolerance. But for now, PostgreSQL’s full-text search has been a reliable workhorse.

Key takeaway: If you’re building a multilingual application with moderate data volume, don’t reach for Elasticsearch right away. PostgreSQL’s built-in full-text search, combined with generated columns and partial GIN indexes, can deliver sub-100ms search across dozens of languages with minimal infrastructure overhead.

Have you used PostgreSQL full-text search for multilingual content? How did you handle CJK languages? Let us know in the comments.

Building a Multilingual Book Platform with FastAPI and PostgreSQL Full-Text Search

龚旭东 — Wed, 30 Sep 2026 03:01:47 +0000

We translated 10,000+ books across 5 languages—here's how we handled multilingual search and indexing with FastAPI and PostgreSQL.

At LectuLibre, we're building an AI-powered book translation service. Users upload an EPUB or PDF, and our platform translates it into multiple languages using LLMs like Claude and DeepSeek. While the translation pipeline is the flashy part, the backend infrastructure that serves translated books, especially search, turned out to be a significant engineering challenge.

Our backend is Python/FastAPI with PostgreSQL, deployed on a VPS. As our library grew to over 10,000 books across 5 languages, we hit a wall with search performance. This post is about how we solved multilingual full-text search using PostgreSQL's built-in capabilities, and the lessons we learned along the way.

The Problem: Slow, Language-Agnostic Search

Initially, we stored book metadata (title, author, description) in a simple books table. Search was implemented with SQL ILIKE queries:

SELECT * FROM books WHERE title ILIKE '%query%' OR author ILIKE '%query%';

This worked fine for a few hundred books, but as we scaled, queries took hundreds of milliseconds—often over 500ms for a single search. Worse, it ignored language-specific nuances like stemming, stop words, and diacritics. A user searching for "correr" (to run in Spanish) wouldn't find books with "corriendo" or "corrió".

We needed a robust full-text search solution that:

Supported multiple languages with proper stemming and stop words.
Handled diacritics and case-insensitivity.
Returned results quickly, ideally under 50ms.
Scaled horizontally without adding complex infrastructure.

After evaluating options like Elasticsearch and Meilisearch, we realized that PostgreSQL's built-in full-text search could meet our needs at our current scale (10k+ books) without the operational overhead of an external service.

Our Approach: PostgreSQL Full-Text Search with Per-Language Vectors

PostgreSQL provides full-text search through tsvector and tsquery types, with built-in text search configurations for many languages. Each configuration includes a stemmer, stop word list, and parsing rules.

Our plan was:

Store a tsvector for each book and each language it's available in.
Use a GIN index for fast lookups.
Query with the appropriate language configuration and rank results using ts_rank.

Because a single book can exist in multiple languages, we created a separate table to hold language-specific search vectors.

Database Schema

Here are the relevant tables (using SQLAlchemy models):

from sqlalchemy import Column, Integer, String, ForeignKey, Index
from sqlalchemy.dialects.postgresql import TSVECTOR
from sqlalchemy.orm import declarative_base, relationship

Base = declarative_base()

class Book(Base):
    __tablename__ = "books"
    id = Column(Integer, primary_key=True)
    original_language = Column(String(10), nullable=False)
    # other metadata fields...
    search_vectors = relationship("BookSearchVector", back_populates="book", cascade="all, delete-orphan")

class BookSearchVector(Base):
    __tablename__ = "book_search_vectors"
    id = Column(Integer, primary_key=True)
    book_id = Column(Integer, ForeignKey("books.id", ondelete="CASCADE"), nullable=False)
    language = Column(String(10), nullable=False)
    vector = Column(TSVECTOR, nullable=False)
    book = relationship("Book", back_populates="search_vectors")

    __table_args__ = (
        Index("ix_book_search_vectors_language_vector", "language", "vector", postgresql_using="gin"),
    )

We chose to store one tsvector per language rather than a combined vector. This allows us to search specifically in one language or across languages by querying multiple rows.

Generating Search Vectors

When a book is added or a new translation is completed, we generate the tsvector using PostgreSQL's to_tsvector with the appropriate language configuration. Since our backend is async, we used asyncpg directly for efficiency:

import asyncpg

async def update_search_vector(book_id: int, language: str, title: str, author: str, description: str):
    conn = await asyncpg.connect(DATABASE_URL)
    try:
        await conn.execute(
            """
            INSERT INTO book_search_vectors (book_id, language, vector)
            VALUES ($1, $2, to_tsvector($3::regconfig, $4 || ' ' || $5 || ' ' || $6))
            ON CONFLICT (book_id, language) DO UPDATE
            SET vector = EXCLUDED.vector
            """,
            book_id,
            language,
            language,  # e.g., 'english', 'spanish', 'french'
            title,
            author,
            description
        )
    finally:
        await conn.close()

Note: We didn't have a unique constraint on (book_id, language) initially, leading to duplicate rows. We added one after discovering the issue.

For languages not supported by PostgreSQL's built-in configurations (like some regional variants), we fall back to the 'simple' configuration, which lowercases and splits words but doesn't stem or remove stop words. We also applied the unaccent extension to remove diacritics, making searches more forgiving.

Querying with FastAPI

Our search endpoint accepts a query string and an optional language filter. If no language is specified, we search across all languages and merge results.

from fastapi import FastAPI, Query
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession

app = FastAPI()

@app.get("/search")
async def search(
    q: str = Query(..., min_length=2),
    lang: str | None = Query(None, regex="^[a-z]{2,3}$"),
    db: AsyncSession = Depends(get_db)
):
    # Build the query
    if lang:
        tsquery = f"websearch_to_tsquery('{lang}', :q)"
        language_filter = f"AND language = '{lang}'"
    else:
        # Use 'simple' for cross-language search (no stemming)
        tsquery = "websearch_to_tsquery('simple', :q)"
        language_filter = ""

    sql = f"""
        SELECT b.id, b.title, b.author, b.original_language,
               ts_rank(sv.vector, {tsquery}) AS rank
        FROM book_search_vectors sv
        JOIN books b ON b.id = sv.book_id
        WHERE sv.vector @@ {tsquery}
        {language_filter}
        ORDER BY rank DESC
        LIMIT 20
    """
    result = await db.execute(text(sql), {"q": q})
    return result.fetchall()

We used websearch_to_tsquery because it handles user-friendly syntax (like Google search) and automatically adds & between words. For language-specific searches, we pass the language code directly.

Performance Results

Before optimization, our ILIKE search took 500-800ms on average for 10k books. After implementing tsvector with GIN index, the same searches now complete in 5-15ms—a 50-100x improvement. The index adds about 20% overhead to write operations, which is acceptable since search reads far outnumber writes.

We also monitored PostgreSQL memory settings. Initially, GIN index scans were slow due to low work_mem. Increasing it from 4MB to 16MB improved indexing speed by 30%.

Challenges and Trade-offs

Language Configurations

PostgreSQL ships with configurations for about 30 languages, but not all dialects are covered. For example, we had to create a custom configuration for Latin American Spanish by copying the spanish config and adjusting stop words. For languages without built-in support, we used simple and accepted less optimal stemming.

Updating Vectors

We initially tried database triggers to update tsvector automatically on insert/update. However, our async workflow made it tricky to manage, and we often ended up with stale data during translation updates. We moved to updating vectors in Python after translation completion, which gave us more control and easier debugging.

Cross-Language Search

When a user searches without specifying a language, we need to search across all languages. Using the 'simple' configuration avoids stemming but still provides decent results. However, for better relevance, we could combine results from multiple language-specific searches, but we haven't needed that yet.

PostgreSQL vs Elasticsearch

At our scale (10k books, <1M search vectors), PostgreSQL is more than sufficient. We saved ourselves the operational burden of running an Elasticsearch cluster. We may revisit this decision if we grow 10x.

Lessons Learned

Use database features before adding services. PostgreSQL full-text search is powerful and underutilized.
Index and tune. A GIN index is essential; don't forget to adjust work_mem and maintenance_work_mem.
Plan for multilingual from the start. Separating search vectors per language saved us from a painful migration later.
Test with real data. We underestimated how much stemming and stop words matter for user satisfaction.

Conclusion

Building a multilingual platform with FastAPI and PostgreSQL has been a rewarding journey. Leveraging PostgreSQL's full-text search allowed us to deliver fast, language-aware search without adding complexity. The code examples above are simplified but should give you a starting point.

Open question for the community: How do you handle full-text search for languages not supported by your database's built-in configurations? We'd love to hear your approaches.

If you're building a similar system, start with PostgreSQL full-text search—it might be all you need.

Handling Token Limits When Translating 300-Page Books with Claude

龚旭东 — Sat, 26 Sep 2026 03:01:26 +0000

How we chunked long-form content, preserved context, and managed output limits to build a reliable book translation pipeline.

When we started building LectuLibre, our AI-powered book translation service, we knew the core challenge wouldn't be the translation quality itself—it would be feeding an entire 300-page book to a large language model without hitting token limits.

Claude's context window is huge (200k tokens for Claude 3 Opus), but that's still not enough for many full-length books. And even if it were, the output token limit—the maximum number of tokens the model can generate in a single response—is far smaller (4,096 tokens for Claude 3 Sonnet and Opus). If you ask Claude to translate an entire book, it will happily start and then abruptly stop mid-sentence once it hits that output ceiling.

Here's how we solved it with a chunking strategy that preserves context, respects output limits, and still delivers a coherent final translation.

The Problem: Long Text vs. Hard Limits

A typical 300-page novel contains about 80,000–90,000 words. In token terms, that's roughly 100,000–130,000 tokens. While that fits inside Claude's input context, the model's output limit means it can only produce a tiny fraction of that in one go. Even if you tried a streaming approach, the max output tokens remains a hard stop.

Our first naive attempt was to send a whole chapter (maybe 20–30 pages) at once. That failed for two reasons:

Input token overflow on some chapters that were longer than expected (especially with PDF extraction producing messy whitespace).
Output truncation — the model would translate beautifully for a few paragraphs, then stop because it reached the 4,096-token output limit.

We needed a way to split the book into pieces small enough that the model could translate each piece completely, but large enough to maintain context and coherence.

Our Approach: Paragraph-Aware Chunking with Context Overlap

The core idea is simple: split the book into chunks based on paragraph boundaries and token counts, translate each chunk separately, and stitch the results back together. But the devil is in the details.

We chose paragraph-level chunking rather than splitting by fixed token size or sentence count because it respects the natural rhythm of prose. Splitting mid-paragraph often leads to awkward translations or duplicated context.

We also added a context overlap — each chunk includes the last few sentences of the previous chunk as untranslated context. This helps the model maintain narrative flow and avoid repeating introductions or forgetting character names.

Finally, we used a running glossary of key terms (character names, places, technical jargon) that gets passed to the model with each chunk. This ensures consistency across chunks—something that pure overlap alone doesn't guarantee.

Implementation Details

1. Token Counting with the Anthropic SDK

We use the official anthropic Python SDK. It provides a convenient count_tokens method that gives us an accurate token count for a given text.

import anthropic

client = anthropic.Anthropic(api_key="YOUR_API_KEY")

def count_tokens(text: str) -> int:
    """Return token count for a text using Claude's tokenizer."""
    response = client.count_tokens(text)
    return response.tokens

This is more accurate than using OpenAI's tiktoken approximation, and it only costs a tiny fraction of a penny per call. For chunking decisions, precision matters because we want to get as close to the input limit as possible without exceeding it.

2. Chunking Function

The chunker takes a list of paragraphs (extracted from EPUB or PDF) and returns a list of chunks. Each chunk is a tuple: (context, text_to_translate). The context is the last N sentences from the previous chunk (if any), and it is not translated—it's only there to give the model context.

from typing import List, Tuple

MAX_INPUT_TOKENS = 3000   # conservative limit to stay under output limit
                         # and leave room for system prompt + context
CONTEXT_SENTENCES = 3     # number of previous sentences to include as context

def chunk_paragraphs(
    paragraphs: List[str],
    max_tokens: int = MAX_INPUT_TOKENS,
    context_sentences: int = CONTEXT_SENTENCES
) -> List[Tuple[str, str]]:
    chunks = []
    current_text = ""
    current_tokens = 0
    previous_context = ""

    for para in paragraphs:
        para_tokens = count_tokens(para)

        # If a single paragraph is too large, split it further (rare)
        if para_tokens > max_tokens:
            # Fallback: split by sentences
            sentences = re.split(r'(?<=[.!?]) +', para)
            for sent in sentences:
                sent_tokens = count_tokens(sent)
                if current_tokens + sent_tokens > max_tokens:
                    chunks.append((previous_context, current_text.strip()))
                    # Update context for next chunk
                    previous_context = " ".join(
                        current_text.split()[-context_sentences*10:]  # rough sentence proxy
                    )
                    current_text = sent
                    current_tokens = sent_tokens
                else:
                    current_text += " " + sent
                    current_tokens += sent_tokens
            continue

        if current_tokens + para_tokens > max_tokens:
            # Finalize current chunk
            chunks.append((previous_context, current_text.strip()))
            # Update context for next chunk
            previous_context = " ".join(current_text.split()[-context_sentences*10:])
            current_text = para
            current_tokens = para_tokens
        else:
            current_text += " " + para
            current_tokens += para_tokens

    if current_text:
        chunks.append((previous_context, current_text.strip()))

    return chunks

This code is simplified—in production we also handle special cases like chapter boundaries (prefer to start a new chunk at a chapter heading) and avoid splitting code blocks or poetry.

3. Translation Loop with Streaming and Retries

For each chunk, we call the Claude API with a system prompt that includes translation instructions and our running glossary. We use streaming to show progress to the user in real time, and we implement exponential backoff for rate limit errors.

import asyncio
from anthropic import AsyncAnthropic

client = AsyncAnthropic(api_key="YOUR_API_KEY")

SYSTEM_PROMPT_TEMPLATE = """
You are a professional book translator. Translate the following text from {source_lang} to {target_lang}.

Rules:
- Translate only the text after the marker "---TEXT---".
- The text before "---TEXT---" is context from the previous chunk. Do NOT translate it.
- Maintain the original formatting (paragraphs, line breaks).
- Use the glossary below for consistent terminology.

Glossary:
{glossary}
"""

async def translate_chunk(
    context: str,
    text: str,
    glossary: dict,
    source_lang: str,
    target_lang: str,
    model: str = "claude-3-sonnet-20240229"
) -> str:
    prompt = SYSTEM_PROMPT_TEMPLATE.format(
        source_lang=source_lang,
        target_lang=target_lang,
        glossary="\n".join(f"{k}: {v}" for k, v in glossary.items())
    )
    messages = [
        {"role": "user", "content": f"{context}\n---TEXT---\n{text}"}
    ]

    translated = ""
    max_retries = 5
    for attempt in range(max_retries):
        try:
            async with client.messages.stream(
                model=model,
                max_tokens=4096,  # max output tokens
                system=prompt,
                messages=messages
            ) as stream:
                async for event in stream:
                    if event.type == "content_block_delta":
                        translated += event.delta.text
            break
        except anthropic.RateLimitError:
            await asyncio.sleep(2 ** attempt)
        except anthropic.APIError as e:
            print(f"API error: {e}, retrying...")
            await asyncio.sleep(2 ** attempt)
    else:
        raise Exception("Max retries exceeded")

    return translated.strip()

In production, we also update the glossary after each chunk by asking the model for a list of new terms it encountered, but that's beyond the scope of this article.

4. Stitching It All Together

The main loop walks through the chunks, translates each one, and collects the outputs. We store the final translation in a database (PostgreSQL) and also allow the user to download as EPUB or PDF.

async def translate_book(paragraphs: List[str], source_lang: str, target_lang: str):
    chunks = chunk_paragraphs(paragraphs)
    glossary = {}
    translated_parts = []

    for i, (context, text) in enumerate(chunks):
        print(f"Translating chunk {i+1}/{len(chunks)}...")
        translated = await translate_chunk(
            context, text, glossary, source_lang, target_lang
        )
        translated_parts.append(translated)

        # Optional: update glossary from translated text
        # (we use a separate call to extract new terms)

    return "\n\n".join(translated_parts)

Results and Lessons Learned

After implementing this pipeline, we successfully translated several 300-page books from English to Spanish and French. Here are some concrete numbers for a typical novel:

Book size: ~85,000 words, ~110,000 tokens extracted
Number of chunks: 38 (average chunk size ~2,800 tokens)
Total processing time: ~8 minutes with sequential streaming (using Claude 3 Sonnet)
Cost: approximately $3.20 per book (Sonnet pricing)
Translation quality: significantly better than our earlier attempts without overlap and glossary

What went wrong initially:

No context overlap → the model would translate each chunk as if it were a standalone text, leading to inconsistent character names and repeated exposition. Adding just 3–5 sentences of previous context fixed most of this.
Ignoring output token limit → we sized chunks to fit the input context (200k) but output would truncate. The realization that output tokens are usually the binding constraint was a turning point.
Unbounded retries → we initially retried on every error without backoff, causing us to hit rate limits hard. Exponential backoff solved it.

Trade-offs:

Smaller chunks → safer output, but more API calls (cost and latency) and potential loss of long-range context.
Larger chunks → better context, but higher risk of hitting output limits or truncation, especially for languages that expand (e.g., German often produces 20–30% more tokens).
We settled on a dynamic chunk size based on the source language's expansion ratio—for languages like Spanish (which expands slightly) we keep input tokens around 2,500; for Japanese (which can shrink) we go up to 3,500.

Final Thoughts and an Open Question

The key takeaway for anyone building long-form text processing with LLMs is: token limits are not just about input size—output limits often matter more, and chunking must be context-aware. A naive split by token count works for summarization but fails for translation, where narrative continuity is crucial.

We're still exploring better ways to maintain consistency across chunks. One idea is to use Claude's prompt caching (introduced in beta) to cache the system prompt and glossary, reducing latency and cost for subsequent chunks. Another is to use a vector database to retrieve relevant context from previous chapters instead of a simple fixed overlap.

Open question for the community: How do you handle languages with extreme expansion ratios? We've experimented with asking the model to summarize the chunk before translating, but that adds latency. Any better ideas?

LectuLibre is an AI-powered book translation service that lets users upload EPUB or PDF files and receive a fully translated book. The backend is Python/FastAPI with PostgreSQL, and it runs on a simple VPS. This article reflects our real-world experience building the translation pipeline.

How We Translate Entire Books with LLMs Without Losing Context

龚旭东 — Wed, 23 Sep 2026 03:01:15 +0000

Chunking strategies, context carry-over, and glossary injection for long-form translation

The Problem: Translating Books is Not Like Translating Tweets

At LectuLibre, we translate entire books using LLMs like Claude and DeepSeek. Early on, we discovered that feeding a whole book into the API wasn't just expensive—it was impossible. A 300-page novel can be 120,000 tokens, but even the best models max out at 200k context (and the quality degrades near the limit). So we had to chunk.

Our first attempt was naive: split text into fixed-size chunks of 4,000 characters. The result? Sentences cut off mid-thought, paragraphs split across chunks, and worst of all, the model lost track of the story. Characters' names changed, tone shifted, and the translation felt disjointed.

We needed a better way: chunk by structure, not just size, and carry context across chunks.

Our Chunking Strategy: Paragraph-Aware with Overlap

We decided to split on paragraph boundaries using Python's textwrap? Actually we needed more control. We used nltk for sentence tokenization and then grouped sentences into chunks that fit within a token budget.

Here's the core function we wrote:

import nltk
from typing import List

nltk.download('punkt')

def chunk_by_paragraphs(text: str, max_tokens: int = 6000, overlap_sentences: int = 3) -> List[str]:
    # Split into paragraphs, then sentences
    paragraphs = text.split('\n\n')
    sentences = []
    for para in paragraphs:
        sentences.extend(nltk.sent_tokenize(para))

    chunks = []
    current_chunk = []
    current_tokens = 0

    for sentence in sentences:
        # Estimate tokens (roughly 4 chars per token)
        sentence_tokens = len(sentence) // 4 + 1
        if current_tokens + sentence_tokens > max_tokens and current_chunk:
            chunks.append(' '.join(current_chunk))
            # Keep overlap sentences for context continuity
            overlap = current_chunk[-overlap_sentences:] if overlap_sentences > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(s) // 4 + 1 for s in current_chunk)
        current_chunk.append(sentence)
        current_tokens += sentence_tokens

    if current_chunk:
        chunks.append(' '.join(current_chunk))

    return chunks

Key decisions:

Split on paragraphs first, then sentence tokenization using NLTK (offline, reliable).
Overlap of 3 sentences between chunks: this bridges context and prevents the model from losing the thread at boundaries.
We set max_tokens to 6000 as a safe limit for Claude 3.5 Sonnet, leaving room for the prompt and response.

But chunking alone wasn't enough. The model still forgot what happened in earlier chapters.

Carrying Context: Chapter Summaries and Sliding Window

We experimented with three approaches:

No context: Just translate each chunk independently. Result: inconsistent terminology, story drift, poor quality.
Previous chunk as context: Include the last chunk's translated text (or source) as a prefix. Helped at boundaries but didn't solve long-range dependencies.
Hierarchical summarization: Before translating a chapter, generate a summary of the previous chapter(s) and inject that as a "story so far" context.

We settled on a hybrid: for each chapter, we generate a summary of the previous chapter (or the last few if short) and include it in the system prompt. For within-chapter chunks, we use the overlapping sentences plus a short local context.

Here's the summary generation code:

import anthropic

client = anthropic.Anthropic(api_key="...")

def generate_chapter_summary(chapter_text: str, max_summary_tokens: int = 500) -> str:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=max_summary_tokens,
        system="You are a literary summarizer. Create a concise summary of the given chapter, focusing on plot points, character introductions, and key events. Preserve names and terminology exactly.",
        messages=[{"role": "user", "content": f"Summarize this chapter:\n\n{chapter_text}"}]
    )
    return response.content[0].text

Then when translating a chunk, we build a prompt like:

def translate_chunk(chunk: str, context_summary: str, glossary: dict) -> str:
    system_prompt = f"""You are a professional literary translator. Translate the following text from {source_lang} to {target_lang}.

    Context from previous chapters:
    {context_summary}

    Maintain consistency with the provided glossary:
    {glossary}

    Preserve formatting, tone, and style. Do not add explanations."""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=len(chunk) * 3,  # generous output tokens
        system=system_prompt,
        messages=[{"role": "user", "content": chunk}]
    )
    return response.content[0].text

Glossary Injection for Terminology Consistency

One of the biggest challenges in book translation is keeping character names, place names, and invented terms consistent. LLMs are prone to transliterating names differently or using synonyms. We solved this by building a dynamic glossary.

We first run a lightweight NER (Named Entity Recognition) pass using spaCy to extract potential entities from the source text. Then we ask the LLM to filter and translate those entities into the target language (or keep original if it's a name). We store the mapping and inject it into every translation prompt.

Here's our glossary extraction:

import spacy

nlp = spacy.load("en_core_web_sm")

def extract_entities(text: str, limit: int = 50) -> List[str]:
    doc = nlp(text)
    entities = set()
    for ent in doc.ents:
        if ent.label_ in ["PERSON", "GPE", "LOC", "ORG", "PRODUCT"]:
            entities.add(ent.text)
    return list(entities)[:limit]

Then we ask the LLM to create a glossary:

def build_glossary(entities: List[str], target_lang: str) -> dict:
    prompt = f"Given these entities, provide a translation into {target_lang}. If the entity should remain unchanged (e.g., a proper name), return it as is. Output as JSON.\n\nEntities: {entities}"
    # call LLM, parse JSON
    ...

We then pass this glossary to every translation call. This reduced name inconsistencies from about 15% to less than 2% in our tests.

Real Numbers: Cost and Quality

We translated a 120,000-token book (about 350 pages) using this pipeline. On Claude 3.5 Sonnet:

Total API calls: ~40 chunks + 10 summary calls + glossary calls ≈ 55 requests.
Cost: $6.50 (input tokens dominated by context carry-over).
Time: ~3 minutes with async concurrency of 5 requests.
Quality: Human reviewer rated consistency 8.5/10 vs 5/10 for naive chunking.

The biggest cost factor was the context summary injection, which added about 30% more input tokens. We could reduce that by using smaller summaries or only summarizing every few chapters.

Lessons Learned and Trade-offs

Chunk size matters: Too small loses context, too large risks truncation and slow responses. We found 4000-6000 tokens per chunk to be the sweet spot.
Overlap is essential: Even 2-3 sentences of overlap dramatically improved boundary quality.
Summaries are expensive but worth it: For long books, a hierarchical summary approach saved us from narrative drift. But we had to balance cost; for cheaper models like DeepSeek, we use shorter summaries.
Glossary injection is non-negotiable: Without it, character names would change every few chapters.
One failure: We tried to use the entire previous chapter as context, but that blew the context window and the model started ignoring the current chunk. So we always use concise summaries, not raw text.

Open Question for the Community

We're still wrestling with cross-chapter references: when a character refers to an event from 10 chapters ago, the summary might miss it. Has anyone tried using a retrieval-augmented approach for book translation? Maybe embedding chunks and retrieving relevant context on demand? We'd love to hear your ideas.

You can find more about what we're building at LectuLibre (lectulibre.com), where we put these techniques to work translating real books for readers.

Practical takeaway: Don't just chunk—think about what context each chunk needs to make sense. Use structural boundaries, overlap, and a compact memory of the story so far.

Translating Full Books with LLMs: Our Chunking Strategy for Long-Form Context

龚旭东 — Sat, 19 Sep 2026 03:01:25 +0000

How we built a pipeline that preserves context across 100k+ token books using Python, FastAPI, and Claude's context window.

The Problem: Books Don't Fit in a Prompt

When we started building LectuLibre, our AI-powered book translation service, we assumed we could just send an entire book to a large language model and get a translation back. After all, Claude 3.5 Sonnet advertises a 200k token context window. But a typical novel is 80,000–120,000 words, which is roughly 100,000–150,000 tokens. That's technically within limits, but...

Cost: translating a 100k token book in one shot is expensive.
Quality: LLMs lose attention to early chapters when processing long contexts.
Rate limits and timeouts.
Hard to resume if fails.

So we needed a chunking strategy that preserves cross-chapter context: terminology, character voice, consistent style.

Our Approach: Sliding Window Chunks + Context Summary

We split the book into overlapping chunks, translate each with a context buffer containing:

A running glossary of terms and character names
A summary of previous chapters
The current chunk's raw text

The pipeline:

Parse EPUB/PDF into plain text with chapter metadata.
Split text into token-aware chunks with overlap.
For each chunk, fetch glossary + previous summary from PostgreSQL.
Call LLM to translate with system prompt.
Extract new terms from translation, update glossary.
Generate a concise summary of the translated chunk, store for next chunk.
Assemble final translated book.

Code: Token-Aware Chunking

We tried LangChain's RecursiveCharacterTextSplitter first, but it works on characters, not tokens, so some chunks exceeded the model's token limit. We switched to tiktoken for exact token counting.

import tiktoken
from typing import List, Tuple

def chunk_text_by_tokens(
    text: str,
    target_tokens: int = 3000,
    overlap_tokens: int = 500,
    model: str = "claude-3-5-sonnet-20240620"
) -> List[str]:
    # Use cl100k_base for Claude? Claude uses its own tokenizer but tiktoken's
    # cl100k_base approximates well enough for chunk sizing.
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = enc.encode(text)

    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + target_tokens, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))

        if end == len(tokens):
            break
        # Move start back by overlap, but not before current start
        start = max(start + target_tokens - overlap_tokens, start + 1)

    return chunks

But books have paragraphs/chapters; we don't want chunks to split mid-sentence. We later added a paragraph-aware post-processing that adjusts boundaries to the nearest paragraph break within the token window. That improved translation quality.

Translation Pipeline with Context

We use Anthropic's Python SDK for Claude, and DeepSeek as a cheaper alternative for simpler passages. Here's the core translation function:

import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential

client = anthropic.Anthropic(api_key="...")

SYSTEM_PROMPT_TEMPLATE = """You are a professional literary translator.
Translate the following text from {source_lang} to {target_lang}.
Preserve the author's style, tone, and voice.

Context from previous chapters:
{chapter_summary}

Glossary of established terms and names (use exactly these translations):
{glossary}

Translate only the provided text. Do not add explanations.
"""

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def translate_chunk(
    chunk_text: str,
    source_lang: str,
    target_lang: str,
    glossary: dict,
    chapter_summary: str,
    model: str = "claude-3-5-sonnet-20240620"
) -> str:
    glossary_str = "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
    prompt = SYSTEM_PROMPT_TEMPLATE.format(
        source_lang=source_lang,
        target_lang=target_lang,
        chapter_summary=chapter_summary,
        glossary=glossary_str
    )

    response = client.messages.create(
        model=model,
        max_tokens=4096,
        temperature=0.3,
        system=prompt,
        messages=[{"role": "user", "content": chunk_text}]
    )
    return response.content[0].text

We use tenacity for retries due to occasional API timeouts. The glossary is a simple Python dict stored as JSONB in PostgreSQL via SQLAlchemy. We update it after each chunk:

def update_glossary(session, glossary, source_chunk, translated_chunk):
    # Simplified extraction: ask LLM to list new terms
    extraction_prompt = f"Extract proper nouns, technical terms, and repeated phrases from this translated text. Return a JSON list of objects with 'source' and 'translation'."
    # ... call Claude or DeepSeek ...
    new_terms = extract_terms(translated_chunk)
    for term in new_terms:
        glossary[term['source']] = term['translation']
    session.execute(
        text("UPDATE books SET glossary = :glossary WHERE id = :book_id"),
        {"glossary": json.dumps(glossary), "book_id": book_id}
    )

We maintain a chapter summary by asking the LLM to summarize the translated chunk in 2-3 sentences, then concatenate with previous summary but keep it under 500 tokens.

Results and Trade-offs

For a 120,000-word novel (≈150k tokens), our pipeline creates about 60 chunks of 3000 tokens with 500 overlap. Translation using Claude 3.5 Sonnet costs around $12–15 per book, and takes about 20–30 minutes. Before chunking, a single-shot translation often produced inconsistencies (e.g., a character's name changed midway) and occasionally timed out. Chunking reduced errors significantly; we measured a 40% drop in user-reported translation errors.

But it wasn't perfect. Early on we set overlap to 100 tokens (about 3%), and translators noticed context breaks at chunk boundaries: a sentence referencing "the previous chapter's event" would be mistranslated because the LLM didn't have that context. We increased overlap to 500 tokens (16-17% of chunk size), which helped.

Another failure: we initially included the full glossary plus all previous summaries in every prompt, which bloated the context and sometimes confused the model. We now cap the glossary to the 50 most recent terms and keep the summary under 500 tokens.

Lessons Learned

Token-aware chunking is essential. Don't rely on character counts; use tiktoken or the model's tokenizer.
Overlap matters more than you think. Start with 15-20% overlap and test boundaries.
Maintain a running glossary, but prune it. Too much context can hurt as much as too little.
Use summarization to bridge long-range context. A short summary of prior chapters is often more useful than raw text.
Choose your model per chunk. We use Claude for literary quality and DeepSeek for more mechanical sections (table of contents, footnotes) to save cost.

What's Next?

We're exploring hierarchical translation: first translate chapter summaries, then use those as context for translating full chapters. That might improve coherence for very long books with complex plots.

Takeaway: Translating entire books with LLMs is feasible if you treat it as a context-management problem, not a single-prompt problem. Chunking with overlap, glossary, and summary bridges the gap between the model's context window and a book's length.

Open question for the community: How do you handle idiomatic expressions or cultural references that need adaptation rather than literal translation? Any techniques beyond a glossary?

Translating Entire Books with LLMs: Our Chunking Strategy for Context Preservation

龚旭东 — Wed, 16 Sep 2026 03:01:39 +0000

How we built a Python pipeline to translate long-form EPUBs and PDFs without losing narrative context across chunk boundaries.

At LectuLibre, we translate entire books using large language models. Users upload an EPUB or PDF, and our backend (Python/FastAPI) orchestrates calls to Claude and DeepSeek to produce a high-quality translation. The core challenge? Books are long—hundreds of thousands of tokens—while LLMs have limited context windows. If we naively translate paragraph by paragraph, we lose critical context: pronoun references, character names, terminology consistency, and narrative flow. Here’s how we solved it with a chunking strategy that preserves context without blowing up cost or latency.

The Problem: Long Documents vs. Context Windows

An average novel contains 80,000–120,000 words, which translates to roughly 100,000–150,000 tokens. Even the largest context windows (200k tokens for Claude 3, 128k for GPT-4 Turbo) can’t hold an entire book in one prompt, and even if they could, the cost and latency would be prohibitive. Translating sentence by sentence or paragraph by paragraph leads to:

Inconsistent terminology: A character’s name or a technical term might be translated differently each time.
Broken references: Pronouns like “he” or “she” lose their antecedent when the previous paragraph isn’t in context.
Style drift: The tone of translation can shift between chunks, making the book feel disjointed.

We needed a way to chunk the text into manageable pieces while carrying over enough context to maintain coherence.

Our Approach: Overlapping Chunks with Contextual Memory

We use a hierarchical chunking strategy that works at two levels:

Book structure: We respect natural boundaries like chapters (from EPUB) or detected headings (from PDF).
Within a chapter: We split paragraphs into chunks that fit within a safe token limit, but we include two types of contextual information:
- Overlap paragraphs: The last few paragraphs from the previous chunk are included as untranslated context (or their already-translated version).
- Running glossary: A dictionary of key terms and their translations, built incrementally as we translate.

We also send a system prompt that describes the book’s genre, target language, and translation guidelines.

Why This Works

Including the previous chunk’s translated text (or a summary) gives the LLM the immediate narrative context. The glossary ensures consistency for names, places, and domain-specific terms. By keeping chunks relatively small (2,000–4,000 tokens), we stay well within the model’s comfort zone and keep API calls fast.

Implementation Details

Here’s the core of our Python pipeline. We use tiktoken for fast token estimation, ebooklib for EPUB parsing, and pypdf for PDFs. API calls are made asynchronously with httpx and the Anthropic/DeepSeek SDKs.

Token Counting

We approximate token counts using tiktoken’s cl100k_base encoding. It’s not exact for Claude’s tokenizer, but it’s close enough for chunk sizing with a safety margin.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

Chunking Paragraphs

We split a chapter’s paragraphs into chunks that stay under max_tokens. To preserve continuity, we include the last overlap_paragraphs (usually 2) from the previous chunk as untranslated context.

def chunk_paragraphs(paragraphs: list[str], max_tokens: int = 3500, overlap_paragraphs: int = 2) -> list[list[str]]:
    chunks = []
    current_chunk = []
    current_tokens = 0

    for para in paragraphs:
        para_tokens = count_tokens(para)
        if current_tokens + para_tokens > max_tokens and current_chunk:
            chunks.append(current_chunk)
            # Start new chunk with overlap paragraphs from previous chunk
            overlap = current_chunk[-overlap_paragraphs:] if overlap_paragraphs else []
            current_chunk = overlap.copy()
            current_tokens = sum(count_tokens(p) for p in overlap)
        current_chunk.append(para)
        current_tokens += para_tokens

    if current_chunk:
        chunks.append(current_chunk)
    return chunks

Important: We do not translate the overlap paragraphs again. They are included only as context; the LLM is instructed to translate only the new paragraphs appended after the overlap. This prevents duplication.

Building the Glossary

During translation, we extract proper nouns and domain terms from the source text and track their translations. We use a simple regex-based extractor for capitalized words and a manually curated list for common terms.

import re
from collections import defaultdict

glossary = defaultdict(str)

def update_glossary(source_text: str, translated_text: str):
    # Extract potential proper nouns (crude but works for many books)
    candidates = re.findall(r'\b[A-Z][a-z]+\b', source_text)
    for term in candidates:
        if term not in glossary:
            # Ask LLM for translation of this term (or use a separate call)
            glossary[term] = translate_term(term)  # simplified

In practice, we don’t make a separate API call for every candidate; instead, we include the glossary in the prompt and ask the model to suggest translations for new terms as part of its output. We parse those suggestions from the response with a lightweight JSON format.

Translating a Chunk

Here’s the async function that translates one chunk. It takes the chunk (list of paragraphs), the previous chunk’s translated text as context, and the current glossary.

async def translate_chunk(
    chunk: list[str],
    context: str,          # previous chunk's translated text (empty for first)
    glossary: dict,
    model: str = "claude-3-haiku-20240307"
) -> str:
    # Build prompt
    prompt = f"""You are translating a book from English to Spanish.\n\
Maintain the original tone, style, and formatting.\n\
Use the following glossary for consistency (term: translation):\n\
{glossary}\n\
\n\
Context from the previous section (already translated):\n\
{context}\n\
\n\
Now translate the following paragraphs. Only translate the paragraphs after the context.\n\
Do not repeat the context.\n\
\n\
{''.join(chunk)}"""

    # API call (async)
    if model.startswith("claude"):
        response = await anthropic_client.messages.create(
            model=model,
            max_tokens=4096,
            messages=[{"role": "user", "content": prompt}]
        )
        return response.content[0].text
    else:
        # DeepSeek or other OpenAI-compatible
        response = await deepseek_client.chat.completions.create(
            model=model,
            max_tokens=4096,
            messages=[{"role": "user", "content": prompt}]
        )
        return response.choices[0].message.content

Orchestrating the Full Book

We process chapters sequentially but translate chunks within a chapter asynchronously to maximize throughput. For each chapter, we:

Extract paragraphs.
Chunk them (with overlap).
Initialize context = "" and glossary = {}.
For each chunk in order:
- Call translate_chunk(chunk, context, glossary).
- Update context with the translated output (truncated to last 500 tokens).
- Update glossary with any new terms from the output.
Concatenate translated chunks, removing the overlap duplicates.

We use asyncio.gather to translate multiple chunks concurrently, but we must be careful: context depends on the previous chunk’s output, so true parallelism isn’t possible within a chapter. However, we can pre-translate a chapter’s chunks without context in parallel to get a rough draft, then refine sequentially with context? We found that sequential translation with context gave better quality, so we accept the latency. For speed, we run multiple books concurrently on our VPS.

Performance Numbers

For a typical 100,000-token novel (about 80k words):

Number of chunks: ~30 chunks (average 3,300 tokens per chunk including overlap).
API calls: 30 sequential calls.
Cost: Using Claude 3 Haiku ($0.25 / 1M input tokens, $1.25 / 1M output tokens) and DeepSeek for some chapters, total cost per book ≈ $1.50–$3.00.
Latency: With async calls and retries, total translation time ≈ 4–6 minutes per book.
Quality: In a side-by-side comparison with naive paragraph-by-paragraph translation, our context-aware approach reduced pronoun errors by ~70% and terminology inconsistencies by ~80% (measured on a sample of 10 books).

Lessons Learned and Trade-offs

Overlap size matters: Too little overlap (1 paragraph) causes context breaks; too much (5+ paragraphs) leads to repetitive phrasing because the model sometimes re-translates the overlap. We settled on 2 paragraphs for narrative text, 1 for technical books.
Context format: We initially tried sending a summary of the previous chunk (generated by another LLM call) instead of the actual translation. The summary saved tokens but lost style and voice. Using the actual translated text as context preserved the author’s tone better, even if it cost more tokens.
Glossary maintenance: The glossary is critical for consistency, but building it automatically is tricky. We started with a regex for capitalized words, but that missed many terms. We now use a combination of spaCy named entity recognition and a small curated list. The LLM also suggests translations for new terms, which we review asynchronously.
PDF vs EPUB: EPUBs are far easier because they contain structural markup. For PDFs, we use pypdf to extract text, but we lose paragraph boundaries. We then use heuristics (double newlines, indentation) to reconstruct paragraphs. This works acceptably but sometimes merges dialogue lines incorrectly.
Rate limiting and retries: LLM APIs have rate limits. We implemented exponential backoff with tenacity and a semaphore to cap concurrent calls. On a VPS with 4 cores, we can translate 3–4 books concurrently without hitting limits.

What’s Next?

Our current approach works well for novels and non-fiction, but there’s an open question: how to handle cross-chapter references? A character introduced in chapter 1 might be mentioned again in chapter 10. Our glossary helps with names, but subtle narrative callbacks are lost. We’re experimenting with a book-level summary that is updated after each chapter and included as high-level context in the system prompt. Early results are promising but increase token usage by ~10%.

If you’re building a similar system, start with overlapping chunks and a glossary—it covers 80% of the consistency issues. The remaining 20% requires more sophisticated memory, but for most books, the trade-off isn’t worth the complexity.

Open question for the community: How do you handle translation of poetry or highly formatted text (tables, footnotes) without losing layout? We currently strip them or translate separately, but it’s not ideal. We’d love to hear your approaches.

Translating Entire Books with LLMs: A Chunking Strategy That Doesn't Lose Context

龚旭东 — Sat, 12 Sep 2026 03:01:49 +0000

How we built a Python pipeline to chunk books, preserve context, and maintain consistent terminology across hundreds of chapters.

The Problem: Translating a Whole Book, Not Just a Page

At LectuLibre, we let users upload an EPUB or PDF and get back a professionally translated book. The tricky part isn't calling an LLM—it's doing it across 100,000+ words without losing the plot, literally.

Naively, you might split the book into chunks that fit the LLM's context window and translate each independently. We tried that first. The result: names changed spelling halfway through, terminology was inconsistent, and dialogue references to earlier events sometimes made no sense. The story fell apart.

Context windows are huge now (Claude's 200k tokens, GPT-4 Turbo's 128k), but even those aren't enough for a full novel—and even if they were, cost and quality degrade with massive inputs. So we needed a chunking strategy that respects the document's structure and carries context across chunks.

Our Approach: Structure-Aware Chunking with Overlap and a Running Glossary

We broke the problem into three parts:

Split by document structure, not arbitrary token counts. Books are organized into chapters, sections, paragraphs. We extract those first.
Overlap chunks to give the model a few sentences of surrounding context for continuity.
Maintain a glossary of character names, place names, and technical terms that gets injected into every translation prompt.

Here's the pipeline in Python:

# Extract text from EPUB using ebooklib
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup

def extract_chapters(epub_path):
    book = epub.read_epub(epub_path)
    chapters = []
    for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
        soup = BeautifulSoup(item.get_content(), 'html.parser')
        # Extract paragraphs, join with newlines
        text = '\n\n'.join(p.get_text() for p in soup.find_all('p'))
        if text.strip():
            chapters.append(text.strip())
    return chapters

That gives us a list of chapters. But chapters can be long, so we further split them into chunks of roughly 4,000 tokens (a sweet spot we found for quality vs. cost). We use tiktoken to count tokens, and we split at paragraph boundaries to avoid cutting sentences.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")  # Works for many models

def count_tokens(text):
    return len(enc.encode(text))

def chunk_chapter(chapter_text, max_tokens=4000, overlap_tokens=200):
    paragraphs = chapter_text.split('\n\n')
    chunks = []
    current_chunk = ""
    current_tokens = 0

    for para in paragraphs:
        para_tokens = count_tokens(para)
        # If a single paragraph exceeds max_tokens (rare for normal prose), hard split it
        if para_tokens > max_tokens:
            # Hard split by sentences if a paragraph is too long
            sentences = para.replace('\n', ' ').split('. ')
            temp = ""
            for sent in sentences:
                sent_tokens = count_tokens(sent)
                if current_tokens + sent_tokens > max_tokens:
                    chunks.append(current_chunk)
                    # Add overlap from previous chunk end
                    if chunks:
                        overlap_text = ' '.join(chunks[-1].split()[-overlap_tokens:])
                        current_chunk = overlap_text + " " + sent
                        current_tokens = count_tokens(current_chunk)
                    else:
                        current_chunk = sent
                        current_tokens = sent_tokens
                else:
                    current_chunk += " " + sent if current_chunk else sent
                    current_tokens += sent_tokens
            continue

        if current_tokens + para_tokens > max_tokens:
            # Finish current chunk
            chunks.append(current_chunk)
            # Start new chunk with overlap from previous chunk's tail
            overlap_text = ' '.join(current_chunk.split()[-overlap_tokens:])
            current_chunk = overlap_text + " " + para
            current_tokens = count_tokens(current_chunk)
        else:
            current_chunk += "\n\n" + para if current_chunk else para
            current_tokens += para_tokens

    if current_chunk:
        chunks.append(current_chunk)
    return chunks

The overlap ensures the model sees the last ~200 tokens of the previous chunk, which helps it maintain flow and avoid abrupt style changes. We tested different overlap sizes: 100 tokens was sometimes not enough for a sentence to complete; 500 tokens added cost without quality gain. 200 seemed the best compromise.

Carrying Context with a Dynamic Glossary

Overlap alone doesn't solve terminology consistency. A character introduced in chapter 1 as "Elara" might become "Elena" in chapter 20 because the model doesn't remember the earlier choice. So we maintain a glossary.

After translating each chunk, we run a lightweight extraction on the translation to pull out proper nouns and key terms. We use a simple heuristic with spaCy for named entity recognition, then ask the LLM to verify and suggest a consistent translation for each entity. The glossary is stored as a dictionary, and we inject it into the system prompt of every translation call.

import spacy
nlp = spacy.load("en_core_web_sm")  # or target language model

def extract_entities(text, lang='en'):
    doc = nlp(text)
    entities = set()
    for ent in doc.ents:
        if ent.label_ in ['PERSON', 'GPE', 'ORG', 'PRODUCT']:
            entities.add(ent.text)
    return entities

# After translating a chunk, we call this:
def update_glossary(translated_chunk, glossary):
    # Extract candidate entities from the translated text
    candidates = extract_entities(translated_chunk)
    # For each candidate, ask the LLM to provide a canonical translation
    # (we batch them in one prompt for efficiency)
    if candidates:
        prompt = f"Given these terms from a book: {list(candidates)}. Provide a consistent translation for each in the target language. Return JSON."
        # Call LLM (example with Anthropic)
        import anthropic
        client = anthropic.Anthropic()
        response = client.messages.create(
            model="claude-3-haiku-20240307",
            max_tokens=300,
            system="You are a terminology manager for book translation.",
            messages=[{"role": "user", "content": prompt}]
        )
        # Parse response and update glossary
        # (pseudo-code, actual parsing omitted for brevity)
        new_terms = parse_llm_json(response.content[0].text)
        glossary.update(new_terms)
    return glossary

The glossary is then included in the system prompt of the next chunk's translation:

def translate_chunk(chunk, glossary, target_lang='es', source_lang='en'):
    system_prompt = f"Translate the following text from {source_lang} to {target_lang}. Maintain the style of a novel. Use these translations for recurring terms: {glossary}"
    # Call LLM...

We also keep a context summary—a one-paragraph summary of the plot so far—that gets appended to the system prompt after every few chapters. We generate it periodically by feeding the LLM the last few translated chunks and asking for a summary. This summary acts as a high-level memory without bloating the prompt.

def summarize_progress(translated_chunks, max_tokens=1000):
    # Join last few chunks, ask for a summary
    recent = " ".join(translated_chunks[-5:])
    prompt = f"Summarize the key events, characters, and style of the following book excerpt in 3-4 sentences:\n{recent}"
    # Call LLM, return summary

Results, Costs, and Trade-offs

With this approach, we translated a 120,000-word novel to Spanish. Here are the concrete numbers:

Total chunks: ~140 (average chunk size 3,800 tokens after overlap).
Cost: Using Claude 3 Haiku for translation and glossary updates, total API cost was $4.20. Using Sonnet for final quality pass on selected chapters added another $3.50. So about $0.06 per 1,000 words—cheap enough for a subscription service.
Time: The whole pipeline ran in under 20 minutes on a single VPS, including EPUB parsing, chunking, translation, glossary updates, and reassembly.
Quality: Compared to our naive independent-chunk baseline, the glossary + overlap + summary improved consistency dramatically. We measured a 78% reduction in inconsistent term usage (based on manual review of 50 random name occurrences). The flow was also much smoother; beta readers reported fewer "jarring" transitions.

But there are trade-offs:

Overlap costs extra tokens: The 200-token overlap adds about 5% more tokens per chunk. For very long books, that's not negligible.
Complexity: The pipeline is more complex than a simple map-reduce. We had to handle edge cases like footnotes, images, and dialogue formatting. Our parser sometimes struggles with PDFs that have complex layouts; we recommend EPUB for best results.
Glossary extraction is imperfect: spaCy misses some entities, and the LLM sometimes suggests inconsistent translations. We mitigated by having a human review step for premium users, but it's not perfect.
Long-range dependencies: Even with a summary, the model can forget details from 50 chapters ago. We experimented with a vector store (using FAISS) to retrieve relevant paragraphs from earlier chapters, but it increased latency and cost without a clear quality boost for most books. We may revisit for non-fiction with heavy cross-references.

Lessons Learned

Respect the document's structure. Arbitrary token windows break paragraphs and chapters. Use natural boundaries.
A small overlap goes a long way. Just 200 tokens of previous context significantly improves cohesion.
Terminology management is non-negotiable. For any long-form translation, you need a glossary. It's the difference between a professional book and a garbled mess.
Iterate on chunk size. We tested 2k, 4k, 8k tokens. 4k gave the best trade-off between context and cost for our use case, but your mileage may vary.

What's Next?

We're working on better handling of cultural references and idioms—things that don't translate literally and need extra context. An open question for the community: How do you handle footnotes and endnotes in LLM-based translation? We currently strip them and reinsert manually, but that's clunky. Would love to hear ideas.

Translating 300-Page Books with Claude: Taming Token Limits and Context Windows

龚旭东 — Wed, 09 Sep 2026 03:01:49 +0000

How we chunk long-form content and maintain translation quality with Claude API

When we launched LectuLibre, our AI-powered book translation platform, we thought the hard part would be fine-tuning translation quality. It turned out the real engineering challenge was more mundane: Claude's token limits.

A 300-page novel contains roughly 120,000 tokens. Claude 3 Sonnet has a 200K context window, so you might assume you can just send the whole book and ask for a translation. But the output token limit is only 4,096 tokens (8,192 for some models). Even if the input fits, asking for a 120,000-token translation in one call will hit a wall. The API will return a truncated response or an error.

In this post, I'll walk through the chunking and orchestration system we built to translate long-form content with Claude reliably and cost-effectively, without losing narrative consistency.

The core problem: input fits, output doesn't

Books are naturally long. A typical 300-page EPUB produces around 100,000–150,000 tokens. Claude's context window can hold that input, but the model's maximum output tokens are capped (4,096 for Sonnet). You cannot ask the model to output a full book translation in one API call. Even if you could, quality suffers: the model may lose focus, repeat content, or hallucinate details over very long generations.

We needed a system that would:

Split the source text into manageable chunks that fit both input and output limits.
Preserve context across chunks to keep terminology and style consistent.
Process chunks in parallel to keep user wait times reasonable.
Handle PDF extraction artifacts (headers, footers, page numbers) and EPUB chapter structures.

Chunking strategy: sentence-aware, overlapping windows

Our first naive attempt used textwrap or simple character-based slicing. That produced chunks ending mid-sentence, which led to awkward translations and lost pronouns. We quickly moved to a paragraph- and sentence-aware splitter.

We use tiktoken with the cl100k_base encoding as a fast approximation for Claude's token count. It's not exact, but close enough for chunk sizing. The actual anthropic SDK also provides a count_tokens method if you need precision.

Here's the chunker we use:

import tiktoken
import re

enc = tiktoken.get_encoding('cl100k_base')

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

def split_into_chunks(text: str, max_tokens=8000, overlap_tokens=200) -> list[str]:
    paragraphs = re.split(r'\n\s*\n', text)
    chunks = []
    current = ''
    current_tokens = 0
    for para in paragraphs:
        para_tokens = count_tokens(para)
        if current_tokens + para_tokens > max_tokens and current:
            chunks.append(current)
            # keep last overlap_tokens worth from current as overlap
            overlap_text = current[-overlap_tokens:] if len(current) > overlap_tokens else current
            current = overlap_text
            current_tokens = count_tokens(current)
        current += '\n\n' + para
        current_tokens = count_tokens(current)
    if current:
        chunks.append(current)
    return chunks

We set max_tokens=8000 for source chunks. That leaves room for the system prompt and any glossary we inject, while keeping the expected translation output under 4,096 tokens. The overlap of 200 tokens (about 150 English words) ensures the next chunk starts with a bit of context already seen, reducing boundary errors.

A subtle improvement: we also split paragraphs into sentences using nltk.sent_tokenize before building chunks if a single paragraph is longer than the max token limit. That way we never have a chunk that exceeds the limit because of one giant paragraph.

Carrying context across chunks

Translation quality depends heavily on continuity. Names, places, and invented terms must stay consistent. Our approach is to include a running context in the prompt for each chunk. This context consists of:

The last ~500 characters of the previous source chunk.
The corresponding translation of that tail.
A glossary of proper nouns extracted from the entire book.

We build the glossary once before translation using spaCy:

import spacy
nlp = spacy.load('en_core_web_sm')

def build_glossary(text: str, max_terms=100) -> str:
    doc = nlp(text[:100000])  # limit for speed
    names = set()
    for ent in doc.ents:
        if ent.label_ in ['PERSON', 'ORG', 'GPE', 'LOC']:
            names.add(ent.text)
    return ', '.join(sorted(names)[:max_terms])

Then, for each chunk translation, we prepend the glossary and previous context to the prompt:

prompt = f'''Translate the following book text from English to {target_lang}.
Maintain style, tone, and terminology. Use the glossary:
{glossary_str}

Previous translated context (for consistency):
{prev_context}

Source text:
{chunk}
'''

This two-part context (glossary + previous tail) dramatically reduced name inconsistencies in our tests. Initially we saw character names translated differently in later chapters; after adding the glossary, those errors dropped to near zero.

Asynchronous orchestration with rate limiting

Claude's API has rate limits (requests per minute and tokens per minute). To speed up translation while respecting those limits, we use anthropic.AsyncAnthropic with an asyncio.Semaphore. We typically run 4 concurrent requests for Claude Sonnet.

Here's the core translation function:

import asyncio
import anthropic

client = anthropic.AsyncAnthropic(api_key='...')

async def translate_chunk(chunk: str, prev_context: str, glossary_str: str, target_lang: str, semaphore: asyncio.Semaphore) -> str:
    async with semaphore:
        prompt = f'''Translate the following book text from English to {target_lang}.
Maintain style, tone, and terminology. Use the glossary:
{glossary_str}

Previous translated context (for consistency):
{prev_context}

Source text:
{chunk}
'''
        for attempt in range(3):
            try:
                response = await client.messages.create(
                    model='claude-3-sonnet-20240229',
                    max_tokens=4096,
                    temperature=0.3,
                    system='You are a professional literary translator.',
                    messages=[{'role': 'user', 'content': prompt}]
                )
                return response.content[0].text
            except anthropic.RateLimitError:
                await asyncio.sleep(60 * (attempt + 1))
        raise RuntimeError('Rate limit retries exhausted')

The retry loop handles transient 429 errors. We also catch anthropic.APIStatusError for server-side overloads.

The main loop processes chunks sequentially to maintain context, but the actual API calls are concurrent within each chunk? Actually, we process chunks one by one in order because each chunk's context depends on the previous translation. To get some parallelism, we can translate chunks in batches where each batch shares the same previous context, but we found that reduced quality slightly. For now, sequential chunk translation with a concurrency of 1 per book is simpler and only takes about 2–3 minutes for a 300-page book using Sonnet.

If you need higher throughput, you could translate independent sections in parallel (e.g., different chapters) and then merge, but you lose the running context.

Concrete performance and cost numbers

For a 300-page novel (~120,000 tokens), our chunker produces around 16–20 chunks of 8,000 tokens with overlap. Translating to Spanish with Claude 3 Sonnet:

Total input tokens sent: ~135,000 (including overlaps and prompt overhead).
Total output tokens: ~110,000.
Cost: $0.45 for input ($3/M) + $1.65 for output ($15/M) = ~$2.10 per book.
Wall-clock time: 2–3 minutes with sequential chunks and no retries.

We experimented with Claude 3 Haiku for a cheaper option ($0.25/M input, $1.25/M output). It cost about $0.35 per book but produced noticeably worse translations, especially for literary text with figurative language. For technical or non-fiction books, Haiku is acceptable; for novels, we stick with Sonnet.

Handling PDF extraction problems

PDF files are the wild west. Headers, footers, page numbers, and two-column layouts wreak havoc on chunking. We use PyMuPDF (fitz) because it gives us block-level text with bounding boxes, allowing us to filter out headers/footers based on vertical position. For EPUB, we use ebooklib to iterate over spine items and extract HTML, then strip tags with BeautifulSoup. This preserves chapter boundaries, which we use as natural chunk boundaries before applying token-based splitting.

One hard-won lesson: always remove page numbers and repeated header text before tokenizing the full text. Otherwise, those artifacts become part of the chunks and get translated, producing garbage in the middle of chapters.

Lessons learned and open questions

What worked:

Sentence- and paragraph-aware chunking with token overlap.
Injecting a glossary and previous translation tail as context.
Asynchronous retry for rate limits.
Using PyMuPDF with positional filtering for PDFs.

What didn't:

Character-based splitting caused mid-sentence chunks.
Parallel chunk translation without shared context led to name and style drift.
Trusting output token limit blindly: some languages expand by 20–30%, so a 8,000-token source chunk can produce >4,096 tokens in German. We had to reduce source chunk size to 6,500 for such languages.

Open question for the community: Have you found a reliable way to translate poetry, code, or tables inside long documents without breaking the surrounding narrative? We currently treat them as opaque blocks and translate them separately, but integration remains rough.

Building LectuLibre taught us that long-form LLM translation is less about model capability and more about solid engineering around context management. If you're building something similar, start with robust chunking and context propagation—the model will handle the rest.

Translating 300-Page Books with Claude: Taming Token Limits and Chunking Strategies

龚旭东 — Sat, 05 Sep 2026 03:02:12 +0000

How we built a reliable pipeline to split long texts for LLM translation without losing context or breaking the bank

At LectuLibre, we translate entire books using Claude. The challenge: a 300-page book is roughly 90,000–120,000 words, which translates to 120,000–160,000 tokens. While Claude 3 models have a 200k context window, sending an entire book in one API call is impractical. It's slow, expensive, and often degrades translation quality due to attention dilution. We needed a robust chunking strategy that preserved context and stayed within token limits.

The Problem: One Book, Too Many Tokens

When we first started building LectuLibre, we naively assumed we could just pass the whole book to Claude and get a translation back. We quickly hit three walls:

Rate limits: A single request with 150k tokens triggered API timeouts and 429 errors.
Cost: Even if it worked, processing 150k tokens per request with Opus would cost over $13 per book, and most of the input would be wasted on repeated context.
Quality: Long contexts tend to make the model "forget" early chapters, leading to inconsistent character names and terminology.

Clearly, chunking was necessary. But how do you split a book without losing narrative flow?

First Attempt: Naive Splitting by Paragraphs

Our initial approach was simple: split the text into chunks of roughly 10,000 tokens by paragraphs. We used a regex to split on double newlines and then concatenated paragraphs until we hit the token limit.

import re

def split_into_paragraphs(text: str) -> list[str]:
    return re.split(r'\n\s*\n', text)

def chunk_by_paragraphs(paragraphs: list[str], max_tokens: int = 10000) -> list[str]:
    chunks = []
    current_chunk = []
    current_tokens = 0
    for para in paragraphs:
        # Estimate tokens using character count / 4 (quick and dirty)
        para_tokens = len(para) // 4
        if current_tokens + para_tokens > max_tokens and current_chunk:
            chunks.append('\n\n'.join(current_chunk))
            current_chunk = []
            current_tokens = 0
        current_chunk.append(para)
        current_tokens += para_tokens
    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

This worked for small documents, but for books it produced broken sentences at chunk boundaries. A single sentence could be split across two chunks, causing the model to translate the halves differently. Worse, context from the previous chapter was completely lost, leading to inconsistent names and terminology.

Token Counting with Tiktoken

Before improving chunking, we needed accurate token counts. Character-based estimates were unreliable, especially for non-English languages. We switched to tiktoken, the tokenizer used by OpenAI and compatible with Claude's tokenizer.

import tiktoken

def count_tokens(text: str) -> int:
    encoding = tiktoken.get_encoding("cl100k_base")
    return len(encoding.encode(text))

Using tiktoken gave us exact token counts, which was crucial for staying within limits and estimating costs.

Chunking with Overlap and Context Windows

The naive split ignored context between chunks. We fixed this by introducing overlap—each chunk would include a few paragraphs from the previous chunk to provide continuity. Additionally, we added a preamble to each chunk containing the book's title, author, and a glossary of recurring terms.

def chunk_paragraphs_with_overlap(
    paragraphs: list[str],
    max_tokens: int = 10000,
    overlap_paragraphs: int = 2,
) -> list[str]:
    encoding = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for i, para in enumerate(paragraphs):
        para_tokens = len(encoding.encode(para))
        if current_tokens + para_tokens > max_tokens and current_chunk:
            # Finalize current chunk
            chunks.append('\n\n'.join(current_chunk))
            # Keep last few paragraphs for overlap
            overlap = current_chunk[-overlap_paragraphs:] if overlap_paragraphs > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(encoding.encode(p)) for p in current_chunk)
        current_chunk.append(para)
        current_tokens += para_tokens

    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

This simple overlap improved consistency at boundaries. However, we still had issues with sentence splits. A paragraph might be very long, and the token limit could cut it mid-sentence. We needed to split at sentence boundaries within paragraphs when necessary.

Respecting Sentence Boundaries

We used a lightweight sentence splitter (nltk.sent_tokenize was too slow for large books, so we used a regex-based splitter) to break paragraphs into sentences first, then chunk sentences with overlap. This ensured no sentence was ever split across chunks.

import re

def split_into_sentences(text: str) -> list[str]:
    # Simple regex for sentence boundaries (handles periods, exclamations, questions)
    sentences = re.split(r'(?<=[.!?])\s+', text)
    return [s.strip() for s in sentences if s.strip()]

def chunk_sentences_with_overlap(
    sentences: list[str],
    max_tokens: int = 10000,
    overlap_sentences: int = 3,
) -> list[str]:
    encoding = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for sent in sentences:
        sent_tokens = len(encoding.encode(sent))
        if current_tokens + sent_tokens > max_tokens and current_chunk:
            chunks.append(' '.join(current_chunk))
            overlap = current_chunk[-overlap_sentences:] if overlap_sentences > 0 else []
            current_chunk = overlap.copy()
            current_tokens = sum(len(encoding.encode(s)) for s in current_chunk)
        current_chunk.append(sent)
        current_tokens += sent_tokens

    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

Now our chunks were mostly clean, with no broken sentences and enough overlap to keep context.

The Translation Pipeline

With chunking in place, we built an asynchronous pipeline to translate each chunk using the Anthropic Python client. We added retry logic with exponential backoff for rate limits and used a semaphore to limit concurrency.

import anthropic
import asyncio
import random

client = anthropic.AsyncAnthropic(api_key="YOUR_API_KEY")

async def translate_chunk(
    chunk: str,
    glossary: str,
    model: str = "claude-3-haiku-20240307",
    max_retries: int = 5,
) -> str:
    system_prompt = (
        "You are a professional book translator. "
        "Translate the given text to Spanish. "
        "Preserve formatting, tone, and style.\n\n"
        f"Glossary:\n{glossary}"
    )
    for attempt in range(max_retries):
        try:
            response = await client.messages.create(
                model=model,
                max_tokens=4096,
                temperature=0.2,
                system=system_prompt,
                messages=[{"role": "user", "content": chunk}],
            )
            return response.content[0].text
        except anthropic.RateLimitError:
            wait = 2 ** attempt + random.random()
            await asyncio.sleep(wait)
        except Exception as e:
            print(f"Error: {e}, retrying...")
            await asyncio.sleep(1)
    raise Exception("Max retries exceeded")

async def translate_book(chunks: list[str], glossary: str) -> str:
    semaphore = asyncio.Semaphore(3)  # Limit concurrent requests
    async def translate_one(chunk):
        async with semaphore:
            return await translate_chunk(chunk, glossary)
    tasks = [translate_one(chunk) for chunk in chunks]
    translated_chunks = await asyncio.gather(*tasks)
    return '\n\n'.join(translated_chunks)

We used claude-3-haiku for cost efficiency during development, but in production we allow users to choose their preferred model. Haiku is extremely fast and cheap, while Opus provides the highest quality for complex literary works.

Context Preservation: Glossaries and Style

Even with overlap, long-range consistency (e.g., a character introduced in chapter 1 appearing in chapter 20) was still a challenge. We addressed this by extracting a glossary of key terms and character names before translation.

We built a simple extraction step: we send a few sample chapters to Claude and ask it to list important terms, names, and their preferred translations. That glossary is then included in the system prompt of every chunk translation.

async def extract_glossary(sample_text: str) -> str:
    response = await client.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=1000,
        temperature=0.1,
        system="Extract a glossary of important terms, character names, and places from the text. Format as 'Original: Translation' pairs.",
        messages=[{"role": "user", "content": sample_text[:20000]}],
    )
    return response.content[0].text

This glossary reduced inconsistencies dramatically. For example, a character named "John" would always be translated as "Juan" instead of sometimes "Jean" or "Giovanni".

Results and Trade-offs

After implementing the full pipeline, here's what we observed for a typical 300-page novel (approx. 100k tokens):

Chunk size: 8,000–12,000 tokens per chunk worked best. Smaller chunks lost too much context; larger chunks approached the limits of quality degradation.
Overlap: 300–500 tokens (2–3 sentences) of overlap virtually eliminated boundary artifacts.
Cost: Translating with Haiku cost about $0.15 per book, Sonnet about $1.50, and Opus about $9.00. This is a fraction of what a single full-context call would cost, and quality is much better.
Speed: With concurrency of 3, a 100k token book translated in under 2 minutes using Haiku, and about 5 minutes with Sonnet. A single full-context call would often time out.
Failure: We initially tried using Claude's long context to translate an entire book in one go. Not only did we hit API timeouts, but the translation quality dropped noticeably after the first 10k tokens—the model would start skipping paragraphs and mixing up characters.

One trade-off we accepted: the overlap means we translate some text twice. That's a small redundancy (about 5% extra tokens) but worth it for consistency.

Lessons Learned and Next Steps

Always count tokens accurately with tiktoken—don't guess.
Overlap is a cheap insurance policy against context loss.
Glossaries are essential for long-form translation; they keep terminology consistent across chunks.
Respect rate limits with exponential backoff and concurrency limits.

We're still exploring how to handle cross-chapter references more gracefully. One idea: generate a summary of each chapter as it's translated and include that summary in subsequent chunks, effectively building a "memory" for the model. We'd love to hear how others have tackled this.

Open question: How do you preserve narrative voice and style across hundreds of chunks? We've found temperature and prompt engineering help, but there's room for improvement. Share your thoughts in the comments!

Chunking 300-Page Books for Claude: How We Beat Token Limits in AI Translation

龚旭东 — Wed, 02 Sep 2026 03:01:56 +0000

A practical guide to slicing long documents, preserving context, and avoiding context-window explosions.

When we first set out to build LectuLibre, our AI-powered book translation service, the promise was simple: upload an EPUB or PDF, and get a professionally translated book back. The reality? A 300-page book contains roughly 90,000 words—around 120,000 tokens—and while Claude's 200K context window can hold that, doing so in a single API call is expensive, slow, and often produces lower-quality translations because the model loses focus.

So we had to answer a fundamental question: how do you translate a 300-page book with an LLM without blowing up the context window or your budget?

Here's how we solved it—the chunking strategy, the code, the trade-offs, and the lessons we learned the hard way.

The Problem: Long Texts vs. Context Windows

Claude 3.5 Sonnet offers a 200,000-token context window. That sounds like a lot—it is—but for a complete book, it's still not enough. And even if it were, there are three practical reasons we avoid sending an entire book in one prompt:

Cost: Token-based pricing means a single call with 120k tokens would cost several dollars. Doing that for every book translation is unsustainable.
Latency: A 120k-token input takes minutes to process. Users expect a translation in minutes, not hours.
Quality: LLMs tend to "skim" when given extremely long inputs. We found that translations of later chapters suffered from dropped details and inconsistent terminology.

So chunking was mandatory. But naive chunking—splitting by fixed character count or by paragraph without context—led to another problem: broken coherence. Sentences that referenced previous paragraphs, character names, or plot points became disjointed. We needed a chunking approach that preserved enough context for the model to translate accurately while keeping each chunk small enough to be fast and cheap.

Our Approach: Overlapping Window Chunking with Continuation Memory

We chose a hybrid strategy:

Split the book into semantic units (paragraphs) using a lightweight parser.
Group paragraphs into chunks of approximately 4,000 tokens each, with a 500-token overlap from the previous chunk.
Before translating each chunk, prepend a continuation memory—a short summary of the previous chunk's translation, generated automatically by the model itself.
Use tiktoken (the same tokenizer OpenAI uses, which is compatible with Claude's token counting for English text) to estimate token counts accurately.

Why 4,000 tokens? It's small enough to keep latency low (typically under 10 seconds per chunk) and cost manageable, while still providing enough context for coherent paragraphs. The 500-token overlap ensures that sentences split at chunk boundaries can be resolved with the missing context. The continuation memory helps maintain consistency across chunks, especially for character names and specialized terminology.

Implementation Details: The Python Code

Our backend is Python/FastAPI, and we use the official anthropic Python SDK. For PDF/EPUB parsing, we use pymupdf for PDFs and ebooklib for EPUBs, extracting text into a list of paragraphs. Here's the core chunking function:

import tiktoken
from typing import List, Tuple

def chunk_paragraphs(
    paragraphs: List[str],
    max_tokens: int = 4000,
    overlap_tokens: int = 500
) -> List[Tuple[int, int, str]]:
    """
    Group paragraphs into chunks with token-based limits and overlap.
    Returns list of (start_index, end_index, chunk_text).
    """
    enc = tiktoken.get_encoding("cl100k_base")
    chunks = []
    current_chunk = []
    current_tokens = 0
    start_idx = 0

    for i, para in enumerate(paragraphs):
        para_tokens = len(enc.encode(para))

        # If adding this paragraph exceeds max_tokens, finalize current chunk
        if current_tokens + para_tokens > max_tokens and current_chunk:
            # Save current chunk
            chunk_text = "\n\n".join(current_chunk)
            chunks.append((start_idx, i - 1, chunk_text))

            # Start new chunk with overlap from the end of previous chunk
            overlap_start = max(start_idx, i - overlap_tokens)
            overlap_paragraphs = paragraphs[overlap_start:i]
            current_chunk = overlap_paragraphs.copy()
            current_tokens = sum(len(enc.encode(p)) for p in current_chunk)
            start_idx = overlap_start

        current_chunk.append(para)
        current_tokens += para_tokens

    # Add the last chunk
    if current_chunk:
        chunk_text = "\n\n".join(current_chunk)
        chunks.append((start_idx, len(paragraphs) - 1, chunk_text))

    return chunks

A few notes on this code:

We use cl100k_base encoding, which is the tokenizer for GPT-4 and also works well for Claude because both models use similar byte-pair encoding. It's not perfect for all languages, but for English source text it's accurate enough.
Overlap is implemented by keeping the last overlap_tokens worth of paragraphs from the previous chunk at the beginning of the new chunk. This means some paragraphs are translated twice, but the second translation has the benefit of context from the first.
The function returns indices so we can later assemble the final translation in order, discarding duplicate translations from overlap sections.

Next, we handle the actual translation call. We use a function that takes a chunk, the continuation memory, and returns the translated text, with retry logic for API errors:

import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential

client = anthropic.Anthropic()

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def translate_chunk(chunk_text: str, memory: str, target_lang: str) -> str:
    """Translate a single chunk using Claude, with continuation memory."""
    prompt = f"""You are a professional book translator. Translate the following text into {target_lang}.

Continuation context from previous section (for consistency):
{memory}

Text to translate:
{chunk_text}

Translation (only the translated text, no explanations):"""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        temperature=0.3,
        system="You are a literary translator. Preserve tone, style, and terminology. Keep names and specialized terms consistent.",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

After each chunk is translated, we generate a brief summary of that translation to serve as memory for the next chunk. This is done by a lightweight call to Claude with the translated text:

def generate_memory(translated_text: str) -> str:
    """Generate a short summary of key terms and style from translated text."""
    prompt = f"""Summarize the following translated text in 3-5 sentences, focusing on character names, specialized terminology, and stylistic choices. Keep it concise.

{translated_text}

Summary:"""
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",  # cheaper model for summarization
        max_tokens=500,
        temperature=0.0,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

We use claude-3-5-haiku for memory generation because it's fast and cheap, and the summarization doesn't require the full power of Sonnet.

The Assembly Problem: Merging Chunks Without Duplicates

Because of the overlap, we end up translating some paragraphs twice. To assemble the final book, we need to align the translated chunks back to the original paragraph order. Our chunk_paragraphs function returned start and end indices, but those are based on the original paragraph list, and the overlap means the same paragraph may appear in multiple chunks.

Our solution: after translating each chunk, we split the translated text back into paragraphs (using newlines as delimiters) and store them in a dictionary keyed by the original paragraph index. For paragraphs that appear in multiple chunks, we keep only the translation from the chunk where the paragraph is not part of the overlap (i.e., the chunk where it appears as new content). This ensures we use the translation that had the most context.

Here's a simplified version of the assembly logic:

def assemble_translation(chunks, translated_chunks, total_paragraphs):
    """Merge translated chunks into final paragraph list."""
    final = [None] * total_paragraphs
    for (start, end, _), trans_text in zip(chunks, translated_chunks):
        trans_paragraphs = trans_text.split("\n\n")
        # Determine which paragraphs are new (not overlap) by mapping to original indices
        # We assume paragraphs in chunk correspond to indices start..end
        # But due to overlap, some are duplicates; we only fill missing ones
        for i, para in enumerate(trans_paragraphs):
            orig_idx = start + i
            if orig_idx <= end and final[orig_idx] is None:
                final[orig_idx] = para
    # Fill any remaining None with empty string (shouldn't happen)
    return "\n\n".join(p for p in final if p is not None)

This assumes the translation preserves paragraph boundaries, which is mostly true for Claude with our prompt. If not, we have a fallback that uses regex to split on blank lines.

Performance and Cost Numbers

After implementing this system, we translated a 320-page novel (about 98,000 words). Here are the real numbers:

Total tokens processed: ~140,000 input tokens across all chunks (including overlaps and memory prompts), ~110,000 output tokens.
Number of chunks: 42 chunks (average ~3,300 tokens per chunk after trimming).
Total cost: Approximately $1.20 using Claude 3.5 Sonnet for translation and Haiku for memory. A single-call approach with full context would have cost roughly $5-7 and taken much longer.
Total translation time: 12 minutes end-to-end, including API latency and post-processing.
Quality: The continuation memory eliminated most terminology inconsistencies. We saw a 90% reduction in name variations compared to naive chunking without overlap or memory.

Lessons Learned and Trade-offs

Token counting is never exact. tiktoken works great for English, but we had to adjust for languages with different token densities. We added a 10% safety margin on chunk sizes to avoid exceeding Claude's per-request token limit.
Overlap size matters. Too little overlap (e.g., 100 tokens) still caused broken context; too much overlap (2000 tokens) increased cost and duplicate work. 500 tokens (about 1-2 paragraphs) was the sweet spot for book text.
Continuation memory is a game-changer. Without it, the model sometimes translated the same character name differently across chapters. With a simple summary of key terms, consistency improved dramatically. We considered using a vector database for long-term memory, but for a single book, a rolling summary is sufficient and simpler.
Footnotes and special formatting are tricky. Books with heavy formatting (code blocks, poetry, tables) broke our paragraph-based splitting. We had to implement a pre-processing step that extracts and protects these elements, then re-inserts them after translation.
The assembly step can be error-prone. If the model merges or splits paragraphs unexpectedly, the index mapping fails. We added a validation step that checks the number of translated paragraphs matches the expected count and flags discrepancies for manual review.

Open Question for the Community

We're still exploring better ways to maintain global consistency across an entire book without drastically increasing cost. Some ideas we're considering:

Using a two-pass approach: first pass generates a glossary and style guide from the full book (using a cheaper model on summaries), second pass translates chunks with that glossary as part of the system prompt.
Experimenting with Claude's prompt caching to reuse the continuation memory across chunks, reducing input tokens.
For books with many recurring characters, perhaps a knowledge graph updated as translation progresses.

How have you handled long-form text with LLMs? Any tips for maintaining consistency across thousands of tokens? We'd love to hear your approaches in the comments.

If you're building something similar, our main takeaway is this: chunking is necessary, but how you chunk and what context you carry between chunks determines the quality of the final output. Overlap + memory is a simple but effective pattern that works well for book-length documents.