DEV Community: 龚旭东

How We Built Instant Translation Help (即时翻译帮助) with Python and LLMs

龚旭东 — Wed, 08 Jul 2026 03:02:28 +0000

Balancing speed and context with a hybrid glossary + LLM caching system

The Need for Instant Translation Help

At LectuLibre, we use LLMs to translate entire books into different languages. But even after a high-quality translation, readers sometimes stumble upon an unfamiliar word or want to see the original phrase for clarity. We envisioned a feature we called 即时翻译帮助 (instant translation help): clicking on any word in the translated text would immediately show a contextual explanation or alternative translation, right inside the reading interface.

The core requirement was speed. The popup had to feel instantaneous—under 500ms. A spinning wheel disrupts the reading flow. However, making a full LLM call for every click was out of the question: latency ranged from 2–5 seconds, and at scale it would be expensive.

Breaking Down the Problem

We needed:

Low-latency responses for common words
Context awareness (the same word can mean different things in different sentences)
Coverage: handle any word or short phrase the user might click
Cost efficiency: minimize LLM calls

Our first prototype simply sent the selected word and its surrounding sentence to Claude, but the 3‑second average wait was unacceptable. We had to get creative.

The Hybrid Approach: Glossary + LLM Fallback

We decided to preprocess each book after the main translation to build a bilingual glossary of key terms. At query time, we would first check this local glossary for a match. If found, we could return the result instantly. If not, we would fall back to a faster LLM (DeepSeek) and cache the response for future lookups.

In essence, the glossary acts as a permanent, read‑optimised cache, while the LLM covers the long tail and handles rare words.

Implementation Details

1. Preprocessing Pipeline

After a book is translated, a background task (we use Celery) runs a pipeline that extracts important phrases from the source text and aligns them with their translations. For a first version, we focused on noun phrases, as those are most likely to be clicked for clarification.

We used spaCy for POS tagging and noun chunking, and SentenceTransformers to help align source phrases with target phrases by cosine similarity.

import spacy
from sentence_transformers import SentenceTransformer, util

nlp_source = spacy.load("en_core_web_sm")
nlp_target = spacy.load("es_core_news_sm")  # example target
model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')

def extract_phrases(sent, nlp):
    doc = nlp(sent)
    return [chunk.text for chunk in doc.noun_chunks]

def align_phrases(src_sent, tgt_sent):
    src_phrases = extract_phrases(src_sent, nlp_source)
    tgt_phrases = extract_phrases(tgt_sent, nlp_target)
    if not src_phrases or not tgt_phrases:
        return []
    src_embs = model.encode(src_phrases)
    tgt_embs = model.encode(tgt_phrases)
    scores = util.cos_sim(src_embs, tgt_embs)
    pairs = []
    for i, src in enumerate(src_phrases):
        best_idx = scores[i].argmax()
        if scores[i][best_idx] > 0.7:
            pairs.append((src, tgt_phrases[best_idx]))
    return pairs

This approach misses many verbs and adjectives, but it gave us a solid starting point. We stored the glossary in PostgreSQL with the source sentence to provide context later.

CREATE TABLE glossary (
    id SERIAL PRIMARY KEY,
    book_id INT NOT NULL,
    source_phrase TEXT NOT NULL,
    target_phrase TEXT NOT NULL,
    context_sentence TEXT,
    created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_glossary_book_phrase ON glossary (book_id, source_phrase);

The context_sentence column became crucial for disambiguation—more on that next.

2. Resolving Ambiguity with Embeddings

A simple glossary lookup on the surface form can be ambiguous: “bank” could be a river bank or a financial institution. To reduce mis-hits, we added a context‑matching step that compares the embedding of the user’s surrounding sentence with the gloss entry’s context sentence. Only if the similarity is high enough do we serve the glossary answer; otherwise, we fall through to the LLM.

def is_context_match(user_context, gloss_context):
    if not gloss_context or not user_context:
        return True  # not enough data, trust the match
    emb1 = model.encode(user_context)
    emb2 = model.encode(gloss_context)
    return util.cos_sim(emb1, emb2) > 0.6

This simple check improved precision by about 30% in our tests.

3. The FastAPI Endpoint

The core of the feature is a FastAPI endpoint that receives the book ID, selected text, and the surrounding sentence (grabbed by the frontend). We keep a local in‑memory cache (via cachetools.TTLCache) for LLM fallback results to avoid extra network trips.

from fastapi import FastAPI, Depends
from sqlalchemy.orm import Session
from cachetools import TTLCache
from pydantic import BaseModel

app = FastAPI()
llm_cache = TTLCache(maxsize=10000, ttl=3600)

class HelpRequest(BaseModel):
    book_id: int
    selected_text: str
    context_sentence: str = ""

@app.post("/translate-help")
async def translate_help(req: HelpRequest, db: Session = Depends(get_db)):
    normalized = req.selected_text.strip().lower()

    # 1. Check glossary with context
    gloss_entry = db.query(Glossary).filter(
        Glossary.book_id == req.book_id,
        Glossary.source_phrase.ilike(normalized)
    ).first()
    if gloss_entry and is_context_match(req.context_sentence, gloss_entry.context_sentence):
        return {"translation": gloss_entry.target_phrase, "source": "glossary"}

    # 2. Check LLM cache
    cache_key = f"{req.book_id}:{normalized}"
    if cache_key in llm_cache:
        return {"translation": llm_cache[cache_key], "source": "llm_cache"}

    # 3. Fallback to LLM (DeepSeek)
    prompt = (
        f"Translate the word '{req.selected_text}' in this context:\n"
        f"{req.context_sentence}\n"
        "Provide only the translation, no explanation."
    )
    translation = await call_deepseek(prompt, timeout=2.0)
    if translation:
        llm_cache[cache_key] = translation
        return {"translation": translation, "source": "llm"}
    else:
        return {"translation": "Translation not available", "source": "error"}

We chose DeepSeek for the fallback because its API response time for short prompts averaged 1.2 seconds, compared to 2-4 seconds for Claude. For the cache, we started with a simple in‑memory TTLCache to avoid Redis serialization overhead for small strings. When we later scaled to multiple workers, we added Redis as a shared second‑level cache, but the local one still handles ~80% of cache reads.

Performance and Results

We measured performance over a week on a sample of 100 books (mix of fiction and non‑fiction):

Glossary hit rate: 45% of queries (highest for technical books, lowest for poetry)
LLM cache hit rate: 20%
LLM fallback rate: 35%
p95 overall latency: 120 ms (with hits from glossary/cache returning in <10 ms)
LLM fallback average latency: 1.3 s

User feedback was positive—the occasional 1.5‑second wait for an obscure word was acceptable. Most readers never noticed any delay.

Lessons Learned & Trade‑offs

Glossary coverage: Our noun‑phrase‑only extraction left gaps (verbs, adjectives, idioms). A better word‑aligner (like awesome-align) could improve recall. We’re also experimenting with using the LLM itself during translation to output explicit translation pairs.
Cold start: The first readers of a newly translated book see more fallback calls. A background job could pre‑seed the cache with the top ~1000 words.
LLM reliability: Sometimes the LLM returns a full sentence instead of a single word. We added output validation: if the result is longer than 5 words, we reject it and show a generic message.
Caching invalidation: When a user reports a bad translation and we improve our prompt, we must clear the cache. We version our cache keys with a translation version number.
Cost: Even though 35% of queries hit the LLM, the per‑call cost is tiny (DeepSeek is very cheap). We’ve spent less than $5/month on this feature.

What’s Next?

We’re now testing a small, fine‑tuned T5 model deployed directly on our VPS to replace the DeepSeek fallback. Early results show ~200ms latency for most words, which would make the feature feel truly instant for every click.

Building 即时翻译帮助 reminded us that hybrid systems—combining fast heuristics with modern AI—often deliver the best user experience. Start simple, measure your cache hit rates, and iterate on the glossary quality. And never underestimate the power of a good cache!

We’d love to hear how others are tackling real‑time AI features. What architectures have worked for you?

Building Instant Translation Assistance for Book Translations with Python and LLMs

龚旭东 — Sat, 04 Jul 2026 03:01:48 +0000

How we integrated real-time phrase translation feedback into our AI-powered book translation workflow, and what we learned about latency, context, and prompt engineering.

When we launched LectuLibre, our AI-powered book translation platform, users loved the quality of full-chapter translations. But they kept asking for something else: while reading a partially translated book, they'd stumble on an untranslated phrase or an awkward auto-translation and want to quickly get a better version without leaving the page. So we built 即时翻译求助 (Instant Translation Help)—a feature that lets readers highlight any phrase and get a context-aware, human-quality translation within seconds, along with a brief explanation of tricky parts.

Here's how we built it, the technical challenges we faced, and the lessons we learned about stitching LLMs into a real-time reading experience.

Problem: Real-time, Context-Aware Translation Inside a Book

Most web apps offer generic translation via API calls—send a sentence to Google Translate, get a result. But that doesn't work for literary texts. A phrase like "She let the cat out of the bag" needs to be translated idiomatically, and the appropriate rendering depends heavily on the surrounding paragraphs (is the tone formal? sarcastic? part of a metaphor chain?). Our existing translation pipeline processes entire chapters in bulk with carefully crafted prompts, but for instant help, we needed sub-second latency while preserving that same depth of context.

Our Approach: Server‑Sent Events and a Smart Prompt Buffer

We chose Server-Sent Events (SSE) over WebSockets because the communication is one-directional (server pushes translation tokens) and SSE is simpler to implement with FastAPI. The client (a React app) sends a POST request with:

The phrase to translate
The book ID and the exact location (chapter/paragraph index)
The target language

Our backend retrieves the surrounding text from PostgreSQL (we store the original book in chunks), feeds a carefully assembled prompt to the LLM (Claude 3 Haiku for speed), and streams the response back token-by-token.

Implementation Deep Dive

1. Background Context Retrieval

We index each paragraph with its position. Given a highlighted phrase, we grab the paragraph containing it, plus one paragraph before and after. This usually provides enough narrative context without blowing up the prompt size.

async def get_context(book_id: str, para_index: int, db: AsyncSession):
    # Fetch surrounding paragraphs
    stmt = (
        select(BookParagraph)
        .where(
            BookParagraph.book_id == book_id,
            BookParagraph.index.between(para_index - 1, para_index + 1)
        )
        .order_by(BookParagraph.index)
    )
    result = await db.execute(stmt)
    paragraphs = result.scalars().all()
    return "\n".join(p.text for p in paragraphs)

2. Prompt Engineering for Instant Help

We needed a prompt that instructs the LLM to:

Translate the given phrase in the exact tone and style of the surrounding text
If the phrase contains an idiom or cultural reference, provide a natural equivalent in the target language, with a short explanation
Return the result as a clean Markdown snippet (translation + explanation)
Keep it concise (we display in a small popover)

Here's the core prompt template:

INSTANT_HELP_PROMPT = """
You are a literary translator. Below is the source text surrounding a highlighted phrase, the phrase itself, and the target language.
Translate the highlighted phrase into {target_lang} in a way that fits the style of the surrounding text.
If the phrase contains an idiom, metaphor, or cultural reference, provide a natural equivalent and a one-sentence explanation in parentheses.
Output format:
**Translation:** [your translation]
**Note:** [explanation if needed]

Surrounding text:
{context}

Highlighted phrase:
"{phrase}"

Translation:
"""

We found that Claude 3 Haiku respects this format almost always, and the "Note" part is omitted when not needed.

3. Streaming the Response with FastAPI and SSE

We built an async endpoint that yields SSE chunks. The client can start rendering the translation as tokens arrive, which feels instant.

from fastapi import APIRouter, Request
from fastapi.responses import StreamingResponse
import json
import asyncio

router = APIRouter()

@router.post("/api/instant-help")
async def instant_help(request: Request):
    data = await request.json()
    phrase = data["phrase"]
    book_id = data["bookId"]
    para_index = data["paraIndex"]
    target_lang = data["targetLang"]

    async def event_generator():
        async with async_session() as db:
            context = await get_context(book_id, para_index, db)
        prompt = INSTANT_HELP_PROMPT.format(
            target_lang=target_lang,
            context=context,
            phrase=phrase
        )
        # Stream from Claude using the official Anthropic Python SDK
        async with anthropic.AsyncAnthropic() as client:
            stream = await client.messages.create(
                model="claude-3-haiku-20240307",
                max_tokens=300,
                temperature=0.3,
                messages=[{"role": "user", "content": prompt}],
                stream=True
            )
            async for event in stream:
                if event.type == "content_block_delta":
                    data = json.dumps({"text": event.delta.text})
                    yield f"data: {data}\n\n"
                elif event.type == "message_stop":
                    yield "data: [DONE]\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

On the frontend, we use EventSource to consume these events. The whole round-trip from click to first token appears in about 400–600ms for typical phrases.

Trade-offs and Hard Decisions

Latency vs. Quality

Haiku is fast but not always perfect. We tried DeepSeek-V2 (slower but better with idioms) but its latency crossed 2 seconds, killing the "instant" feel. We settled on Haiku for now, with a secondary more detailed translation available on demand (which uses Claude 3 Opus in the background).

Cost Management

Each instant help call costs about $0.002 (input + output tokens). With thousands of users, that adds up. We implemented a local cache keyed on (book_id, para_index, phrase, target_lang) using Redis. Repeated requests for the same phrase (e.g., multiple users reading the same book) are served from cache instantly, reducing LLM calls by ~30% in our beta.

Prompt Buffer Size

Experimentally, more context (2 paragraphs) significantly improved quality without adding too many tokens. But including an entire chapter led to slower responses and occasional off-topic interpretations. We keep the context at ~500 tokens on average.

Results and What We Learned

User happiness: Readers now translate 3x more phrases than when they had to copy-paste to another tool. The inline Explanation often teaches them new idioms, which they love.
Engineering takeaway: Server-Sent Events are underrated for LLM streaming. They work perfectly over HTTP/2 and are trivial to debug compared to WebSockets.
Prompt sensitivity: The exact wording Output format: **Translation:** ... **Note:** ... reduced malformed responses by 90%. Small tweaks matter.
Caching is critical: With Redis, we kept extra LLM costs in check and improved perceived performance for popular books.

Where We Might Go Next

We're exploring a context window expansion that uses the entire chapter, but with aggressive summarization of preceding paragraphs via a cheap model call. Also, fine-tuning a small open-source model on our translation style could bring costs close to zero. If you've built similar inline AI features, how did you handle the cost/latency/quality triangle? We'd love to hear your approach in the comments.

Building LectuLibre has taught us that AI-powered tools shine when they fit seamlessly into the user's workflow. Instant translation help is that seam—a small feature that feels like magic because it respects the reader's flow.

How We Translate 300-Page Books Using Claude Without Hitting Token Limits

龚旭东 — Wed, 01 Jul 2026 03:01:18 +0000

Breaking long documents into overlapping chunks, preserving context, and reassembling with FastAPI

At LectuLibre, we’ve built an AI‑powered platform that translates entire books—EPUBs and PDFs—using large language models. When we first hooked up Claude’s API, we naively fed it a 300‑page PDF in one request. It failed immediately. Claude 3 Opus has a 200K token window, but a 300‑page book can easily run to 300K tokens or more. Even if we squeezed it in, the output would be truncated and the quality would degrade at the extremes of the context window.

So we faced a classic long‑document problem: how do you translate a book that’s larger than the model’s context window? Here’s the real approach we ended up with, the code we wrote, and the lessons we learned.

The Problem: Token Limits Are Real

Claude 3 Opus and Haiku models (and most LLMs) have a maximum context length—200,000 tokens for Opus. A token is roughly ¾ of a word. A 300‑page novel with ~75,000 words translates to about 100K tokens, so it should fit, right? But translations from English to Spanish can expand by 15–20%, and the prompt instructions, system message, and the user message itself all eat into that budget. Plus, we needed to send the entire source text in every call to give the model full context. That’s not feasible.

We could have tried a simple split: cut the book at arbitrary page boundaries and translate piecemeal. That fails spectacularly. Narrative breaks mid‑sentence, and phrases like “the previous chapter” lose their referents. We needed a more intelligent chunking strategy.

Our Approach: Sliding Window with Overlapping Paragraphs

We settled on a sliding window chunking algorithm based on paragraphs, with a generous overlap. Here’s the idea:

Split the source text into paragraphs (using \n\n).
Build chunks of max_chunk_tokens (we used 180,000 to keep a safety margin), adding paragraphs one by one and counting tokens with tiktoken.
When the chunk exceeds the limit, we start a new chunk but we include the last few paragraphs of the previous chunk as context. This overlap (we used 5 paragraphs) gives the model continuity across chunk boundaries.
We translate each chunk independently, then stitch them back together, removing the overlap.

This isn’t perfect—some chapters may still be split—but it preserves far more context than any fixed‑size split.

Implementation in Python with FastAPI

We built our translation pipeline inside a FastAPI background task. Here’s the core chunking function:

import tiktoken
from typing import List
from langchain_text_splitters import RecursiveCharacterTextSplitter

def chunk_by_paragraphs(text: str, max_tokens: int = 180000, overlap_paragraphs: int = 5) -> List[str]:
    """
    Split text into chunks of at most `max_tokens` tokens,
    using paragraphs as atomic units and overlapping the last
    `overlap_paragraphs` from the previous chunk.
    """
    enc = tiktoken.get_encoding("cl100k_base")  # Claude's tokenizer
    paragraphs = text.split('\n\n')
    chunks = []
    current_chunk = []
    current_token_count = 0

    for para in paragraphs:
        para_tokens = len(enc.encode(para))
        # If a single paragraph exceeds the limit (rare), split it further
        if para_tokens > max_tokens:
            # Fallback to sentence splitting
            para_texts = RecursiveCharacterTextSplitter(
                chunk_size=max_tokens, chunk_overlap=100,
                length_function=lambda x: len(enc.encode(x))
            ).split_text(para)
            for p in para_texts:
                p_tokens = len(enc.encode(p))
                if current_token_count + p_tokens > max_tokens and current_chunk:
                    chunks.append('\n\n'.join(current_chunk))
                    overlap = current_chunk[-overlap_paragraphs:] if len(current_chunk) >= overlap_paragraphs else current_chunk
                    current_chunk = overlap.copy()
                    current_token_count = sum(len(enc.encode(p)) for p in overlap)
                current_chunk.append(p)
                current_token_count += p_tokens
        else:
            if current_token_count + para_tokens > max_tokens and current_chunk:
                chunks.append('\n\n'.join(current_chunk))
                # Keep overlapping paragraphs
                overlap = current_chunk[-overlap_paragraphs:] if len(current_chunk) >= overlap_paragraphs else current_chunk
                current_chunk = overlap.copy()
                current_token_count = sum(len(enc.encode(p)) for p in overlap)
            current_chunk.append(para)
            current_token_count += para_tokens

    if current_chunk:
        chunks.append('\n\n'.join(current_chunk))
    return chunks

Then we translate each chunk using Anthropic’s Python SDK, with back‑pressure and retry logic to handle rate limits:

from anthropic import Anthropic, RateLimitError
import asyncio
from tenacity import retry, stop_after_attempt, wait_exponential

async def translate_chunk(client: Anthropic, chunk: str, target_lang: str) -> str:
    system_prompt = f"You are a professional translator. Translate the following text from English to {target_lang}. Preserve all formatting, line breaks, and special characters. Do not add commentary."

    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=60))
    async def _call():
        try:
            response = await asyncio.to_thread(
                client.messages.create,
                model="claude-3-opus-20240229",
                max_tokens=4096,
                system=system_prompt,
                messages=[{"role": "user", "content": chunk}]
            )
            return response.content[0].text
        except RateLimitError:
            # Let tenacity handle the retry
            raise
    return await _call()

We use asyncio.to_thread because the Anthropic SDK is synchronous; in a FastAPI app we don’t want to block the event loop. The tenacity library gives us exponential backoff for rate limits. After translating all chunks in parallel with asyncio.gather, we merge them:

def merge_chunks(translated_chunks: List[str], overlap_paragraphs: int = 5) -> str:
    """
    Concatenate translated chunks, removing the overlapping paragraphs
    except from the first chunk.
    """
    if not translated_chunks:
        return ""
    result = translated_chunks[0]
    for i in range(1, len(translated_chunks)):
        # Each subsequent chunk starts with 5 overlap paragraphs; skip them
        chunk_paragraphs = translated_chunks[i].split('\n\n')
        # We assume the translation preserved paragraph boundaries
        main_text = chunk_paragraphs[overlap_paragraphs:] if len(chunk_paragraphs) > overlap_paragraphs else chunk_paragraphs
        result += '\n\n' + '\n\n'.join(main_text)
    return result

Parallel Translation and Performance

We run all chunk translations concurrently. For a 300‑page book, we typically get 5–8 chunks of ~180K tokens each. With Claude 3 Opus, each chunk takes about 15–30 seconds to translate. We impose a concurrency limit of 4 simultaneous calls to avoid hitting Anthropic’s rate caps. Overall, a full‑book translation completes in 2–5 minutes.

Cost: Claude 3 Opus is expensive. At $15 per million input tokens, a 300‑page book (~100K input tokens per chunk, ~8 chunks) costs around $12–15. We mitigated this by offering Claude 3 Haiku (cheaper, faster, but lower quality) and DeepSeek as alternatives. Users can choose.

Quality trade‑offs: The overlap strategy works well for most texts, but sometimes a chapter ends exactly at a chunk boundary and the narrative flow feels a bit disjointed. We experimented with dynamic overlap based on chapter markers (e.g., force a split only at chapter headings), but that added complexity and didn’t always align with token limits. We’re sticking with paragraph‑level overlap for now.

Lessons Learned

Token counting is tricky. tiktoken’s cl100k_base is close to Claude’s tokenizer but not identical. We saw a 5% discrepancy in token counts, so we kept a safety margin of 20K tokens below the limit.
Overlap size matters. Too little overlap and you lose context; too much wastes tokens and money. Five paragraphs proved a sweet spot for most books.
Rate limits forced us to build robust retries. Anthropic’s API will 429 you aggressively if you fire too many concurrent requests. tenacity and a concurrency semaphore saved us.
The merge step must handle formatting. Splitting and rejoining on \n\n works for prose, but tables, lists, and code blocks get mangled. We’re now exploring a markdown‑aware splitter.
Cost transparency is crucial. Users understand that translating a 300‑page book isn’t free. We show an upfront cost estimate based on token counts.

Where We Are Now

LectuLibre’s translation pipeline currently handles EPUBs and PDFs up to ~1000 pages. We’ve translated novels, technical manuals, and even a PhD thesis. The chunking approach has held up surprisingly well, but there’s room for improvement: dynamic overlap detection, better table handling, and perhaps a two‑stage translation where we first summarize each chunk’s context.

If you’re building a similar system, don’t underestimate the merge logic. The chunking is easy; making the final output read like a single, coherent book is the real challenge.

What’s your experience with long‑form AI translation? Have you found a better chunking heuristic? We’d love to hear your thoughts in the comments.

Parsing and Rebuilding EPUB Files in Python: Lessons Learned

龚旭东 — Sat, 27 Jun 2026 03:00:46 +0000

How we handle complex EPUB structures for AI translation without breaking navigation and metadata

At LectuLibre, we built an AI‑powered book translation service. Users upload an EPUB, and our pipeline translates the text using LLMs like Claude and DeepSeek. That sounds straightforward until you have to parse and rebuild a valid EPUB without mangling the table of contents, internal links, or styles.

I’m sharing the real‑world challenge we faced, how we chose our tooling, and the ugly corners we discovered when dealing with real‑world EPUB files.

The Problem: EPUB is a Messy Zip File

An EPUB is essentially a ZIP archive containing XHTML, CSS, images, and an OPF manifest. It’s a well‑defined standard (EPUB 3.2), but in practice publishers produce files that bend the rules: missing container.xml, inline styles that break after translation, and structural quirks that make parsing fragile.

Our translation process needed to:

Accept any EPUB the user throws at us.
Extract all text content while preserving the exact structure.
Send each paragraph to an LLM for translation.
Re‑insert the translated text into the original XHTML files.
Repackage everything into a new, valid EPUB.

Step 4 is the tricky part: the translated text can be longer or shorter, it may contain characters that need escaping, and the surrounding markup must remain intact.

Our Approach: Use `ebooklib` with a Dose of Defensive Coding

We evaluated several Python libraries:

epub (pypub) – too simple, no editing support.
lxml + manual zip – too much boilerplate.
ebooklib – full read/write with a clean API.

We went with ebooklib. It provides an object‑oriented model of the EPUB structure, allows us to iterate over documents, and can write a new EPUB from the modified objects. The downside: its documentation is sparse and it can choke on malformed files. We had to layer on a lot of validation.

Step 1: Loading and Validating the EPUB

import ebooklib
from ebooklib import epub

def load_epub(epub_path: str) -> epub.EpubBook:
    book = epub.read_epub(epub_path, {"ignore_ncx": True})
    # Force title to be a string (some books have list titles)
    if isinstance(book.title, list):
        book.title = " ".join(book.title)
    return book

But we quickly learned that read_epub can fail silently if the book’s metadata is corrupted. We added a custom validation step that checks for a valid OPF and at least one spine item.

def validate_epub(book: epub.EpubBook):
    if not book.opf:
        raise ValueError("Missing OPF metadata")
    if len(list(book.spine)) == 0:
        raise ValueError("No spine items found – EPUB is unreadable")

Step 2: Extracting Readable Text from XHTML Documents

An EPUB’s content is stored in epub.EpubHtml objects. We iterate over all items in reading order (spine) and parse the body content with BeautifulSoup (lxml parser) because ebooklib’s own get_body_content() returns raw bytes, and we need to extract text paragraph‑by‑paragraph while keeping the HTML structure.

from bs4 import BeautifulSoup
import html

def extract_paragraphs(item: epub.EpubHtml) -> list[dict]:
    soup = BeautifulSoup(item.get_body_content(), "html.parser")
    paragraphs = []
    for tag in soup.find_all(["p", "h1", "h2", "h3", "h4", "h5", "h6", "li"]):
        clean_text = tag.get_text(strip=True)
        if clean_text:
            paragraphs.append({
                "tag": tag,
                "original": clean_text,
                "translated": None
            })
    return paragraphs

We keep a reference to the original BeautifulSoup tag object so we can later replace its text. This is memory‑heavy for large books but works for books under 10 MB (our VPS limit).

Step 3: Translating with an LLM (and Controlling Length)

For each paragraph we call our translation API (Claude or DeepSeek). The tricky part is that some paragraphs are very short (headers) or contain entity references. We escape HTML entities before sending, and decode them afterward.

import requests

def translate_text(text: str, source_lang: str, target_lang: str) -> str:
    escaped = html.escape(text, quote=False)
    response = requests.post(
        "https://api.lectulibre.com/v1/translate",  # simplified
        json={"text": escaped, "source": source_lang, "target": target_lang},
        headers={"Authorization": f"Bearer {API_KEY}"}
    )
    translated = response.json()["translated"]
    return html.unescape(translated)

We found that LLMs can sometimes add extra spaces or punctuation. We apply a light post‑processing: trim, normalize spaces, and ensure the translated text doesn’t break the containing tag’s structure.

Step 4: Rebuilding the XHTML with Translated Text

Back in the extract_paragraphs output, we replace the tag.string with the translated text. Since tag.string might be a NavigableString containing child elements, we must be careful. If the tag contains only a string, we replace it. If it contains mixed content, we replace the first text node only, which is a simplification that works for most books.

def replace_text(tag, new_text: str):
    if tag.string is not None and not tag.find_all(text=False):
        # Simple case: tag has only a single text node
        tag.string.replace_with(new_text)
    else:
        # Find the first text node and replace it
        for child in tag.children:
            if isinstance(child, str) and child.strip():
                child.replace_with(new_text)
                break

After all replacements, we set the item’s body content back to the modified HTML.

def update_item(item: epub.EpubHtml, paragraphs: list[dict]):
    for p in paragraphs:
        if p["translated"]:
            replace_text(p["tag"], p["translated"])
    # Rebuild the HTML
    html_str = p["tag"].prettify()  # or extract the full soup
    item.set_body_content(html_str.encode("utf-8"))

A problem here: set_body_content expects bytes, and we must ensure the encoding is UTF‑8. Also, if the original file had a XML declaration or namespaces, we might lose them. We handle that by preserving the item.media_type and other metadata.

Step 5: Writing the Translated EPUB

Once all items are updated, we write the book to a new file. We also add a modified‑date and update the language metadata.

def save_book(book: epub.EpubBook, output_path: str):
    book.set_identifier("urn:uuid:" + str(uuid.uuid4()))
    book.add_metadata("DC", "language", "fr")  # target language
    epub.write_epub(output_path, book, {})

We learned the hard way that epub.write_epub may fail if items reference resources (images, fonts) that aren’t properly registered in the manifest. We iterate all items from the original book and add them to the manifest early to avoid missing dependency errors.

Real‑World Pitfalls and How We Solved Them

Broken Table of Contents: After translation, the NCX/NAV files pointed to old file names or anchors that no longer existed because we had renamed items. We now never rename items; we only modify their content in-place. If we must add new items (e.g., for footnotes), we update the TOC manually using ebooklib.epub.Link objects.
Inline CSS Overwrites: Some books use inline styles like font-size: 12pt. When a translated paragraph becomes longer, it can overflow fixed‑height containers. We don’t modify CSS, but we added a warning for books with rigid styling and offer a “clean” version without fixed heights.
Performance: For a 500‑page novel, the entire pipeline (parse, translate, rebuild) takes about 90 seconds on our VPS (4 vCPU, 8 GB RAM). The LLM calls dominate; we batch paragraphs of up to 5 together to reduce API overhead, trading off a slight translation quality dip.
Memory: Loading the entire EPUB and keeping BeautifulSoup trees in memory can spike to 300 MB for large books. We process one book at a time and use a queue to avoid concurrency issues.

Lessons Learned

ebooklib is great but fragile – always validate the EPUB structure yourself; don’t assume all fields are present.
Preserve the original item order and names – renaming breaks internal links.
Escape/unescape HTML entities when moving text between HTML and plain text.
Translation quality depends on context – we’re experimenting with sending the entire chapter instead of individual paragraphs, but that increases latency.

What’s Next?

We’re exploring pandoc for pre‑conversion to a simpler intermediate format that’s easier to manipulate. However, the rebuild step becomes more complex. For now, ebooklib + BeautifulSoup serves our needs.

If you’re building an EPUB processing tool in Python, I hope these real‑world insights save you some of the debugging hours we spent. Got a better approach? I’d love to hear it in the comments!

Happy coding!

How We Built a Robust EPUB Parsing and Rebuilding Pipeline in Python

龚旭东 — Wed, 24 Jun 2026 03:02:01 +0000

Dealing with broken markup, embedded fonts, and namespace chaos while building LectuLibre's translation engine

At LectuLibre, we needed to translate entire EPUB books while preserving their exact visual structure. The core challenge: parse the EPUB, extract all translatable text, send it to an LLM, then reassemble the book with the translated content—images, CSS, fonts, and layout untouched. This turned out to be much harder than it looked. Here’s how we solved it, what broke, and what we learned.

The Problem: EPUBs Are Zip Files of Chaos

An EPUB is a ZIP archive containing XHTML, CSS, images, and a few XML control files (like container.xml and the OPF manifest). In theory, it’s a clean format. In practice, real‑world EPUBs are a mess:

XHTML with invalid markup, unclosed tags, or missing namespace declarations.
Embedded fonts, SVG chapters, and MathML that must be passed through untouched.
Text split across multiple inline elements (Hello World), requiring sentence‑aware translation.
The EPUB 3 spec is huge, and many books were generated by tools that barely follow it.

We needed a pipeline that could handle 90%+ of books without manual intervention, run fast enough for an interactive web service, and survive the most broken inputs we’d inevitably receive.

First Attempt: The High‑Level Library Approach

We reached for ebooklib—a dedicated Python library for reading and writing EPUB files. It gives you a nice object model: an EpubBook with items (documents, images, stylesheets), a spine, table of contents, and metadata. The code to open a book and grab all XHTML files looks deceptively simple:

import ebooklib
from ebooklib import epub

book = epub.read_epub('the-old-man-and-the-sea.epub')
for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
    content = item.get_content().decode('utf-8')
    # translate content ...
    item.set_content(translated_content.encode('utf-8'))

epub.write_epub('translated.epub', book)

This works for many clean EPUBs. But when we stress‑tested it with 100 public‑domain books, we quickly hit walls:

Performance: ebooklib uses xml.dom.minidom internally; reading a 20 MB book with many XHTML files took over 6 seconds, and writing it back took even longer. Memory usage would spike to 1 GB+ because the entire DOM was held in memory.
Namespace handling: Some EPUB3 books use explicit XHTML namespaces everywhere (<html xmlns="http://www.w3.org/1999/xhtml">). ebooklib’s XML serialization would sometimes drop these namespaces, producing output that failed validation.
No fine‑grained text traversal: To replace only the displayed text while preserving markup, we had to parse the XHTML ourselves anyway.

Clearly, we needed something lower‑level for the actual content manipulation.

The Hybrid Solution: `ebooklib` for Metadata, `lxml` for XHTML

We settled on a hybrid architecture:

Use ebooklib to read and write the EPUB structure: the manifest, spine, TOC, and binary files (images, fonts). This saved us from having to reimplement the ZIP juggling and OPF generation.
For every XHTML file, we parse the content with lxml.etree (which is fast, namespaces‑aware, and can recover from broken markup). We walk the tree, extract translatable text segments, translate them, and then inject the translations back into the tree.

Here’s the core extraction logic:

from lxml import etree

def extract_translatable_blocks(html_bytes):
    parser = etree.HTMLParser(recover=True, encoding='utf-8')
    tree = etree.HTML(html_bytes, parser)
    # We only care about text that appears in the body.
    body = tree.find('.//body')
    if body is None:
        return []
    segments = []
    for element in body.iter():
        # Skip script, style, and void elements
        if element.tag in ('script', 'style', 'br', 'hr', 'img'):
            continue
        text = (element.text or '').strip()
        if text:
            segments.append((element, 'text', text))
        tail = (element.tail or '').strip()
        if tail:
            segments.append((element.getparent(), 'tail', tail))
    return segments

Notice we track both element.text and element.tail—this is critical because in HTML like Hello World, the word “World” is actually the tail of the  element.

Before translation, we group adjacent text segments into sentences. Our sentencizer (a lightweight regex‑based splitter) joins text across inline tags, so we send a single unit "Hello World" to the AI instead of two separate fragments. After translation, we split the result back across the original boundaries, taking care to preserve leading/trailing whitespace.

Rebuilding the XHTML

Once the tree is modified, we serialize it back with namespace preservation:

def serialize_html(root_node):
    # lxml's etree.tostring handles namespaces correctly if you pass the tree with nsmap
    html_str = etree.tostring(
        root_node,
        method='html',
        encoding='unicode',
        xml_declaration=False,
        pretty_print=True
    )
    # Wrap back into a full XHTML document if needed
    return f'<?xml version="1.0" encoding="utf-8"?>\n<!DOCTYPE html>\n{html_str}'

We then call item.set_content(serialized_html.encode('utf-8')) on the ebooklib item and write the book back out.

Dealing with the Hard Stuff

1. Embedded Fonts and Binary Resources

ebooklib handled images and fonts transparently as ITEM_IMAGE and ITEM_OTHER. We simply skip translation for non‑XHTML items. However, we discovered that some books rely on font‑face declarations in CSS that must remain valid after rebuild. We don’t modify CSS (translating content: "Chapter 1" would be suicidal), but we do parse each CSS to check for font‑face src URLs and ensure they are preserved as relative paths in the rebuilt EPUB.

2. Validation with `epubcheck`

We run every rebuilt EPUB through epubcheck (the official Java validator) as a final sanity check. Initially, 30% of our output files failed—mostly because ebooklib would omit the mimetype file entry at the beginning of the ZIP, or because we inadvertently stripped xml:lang attributes. We patched our write routine to always inject the mimetype file first, and we now preserve all XML namespaces and attributes during the lxml manipulation.

3. Performance Tuning

Processing a 500‑page novel end‑to‑end (parse, translate, rebuild) takes roughly:

EPUB read & XHTML extraction: 2–3 s CPU
Translation API calls: 8–12 s (dominated by LLM latency)
XHTML rebuild & EPUB write: 3–4 s CPU

We parallelise XHTML file processing with asyncio (asyncio.to_thread for lxml work) because each chapter is independent. This brings wall‑clock time down to about 10 seconds for a typical book—acceptable for a real‑time web service.

Memory usage stays stable at ~150–200 MB by avoiding loading huge DOMs simultaneously.

Lessons Learned (The Hard Way)

High‑level libraries are a great start, but you’ll eventually need to understand the spec. ebooklib saved us weeks of work on the ZIP container and manifest. But debugging why a book wouldn’t open on iBooks meant reading the EPUB 3 spec and checking the OPF line by line.
Test on real‑world garbage. We assembled a corpus of 100 public‑domain EPUBs from Project Gutenberg, Standard Ebooks, and random indie publications. About 15% were seriously broken (e.g., XHTML with three opening <body> tags). lxml’s recover=True was a lifesaver.
Always validate after rebuild. Even if the book “looks fine” in Calibre, hidden structural errors will cause issues on other readers. Automate epubcheck in your CI.
Namespaces will ruin your day. Always use lxml’s nsmap when parsing; never assume default namespace prefixes.
Don’t translate everything. CSS content, <pre> formatted blocks, and math should be left alone. We filter out elements based on a configurable allow‑list.

Open Questions for the Community

We’re still not 100% satisfied with our pipeline. ebooklib is slow on large files due to its DOM‑based approach; rewriting the ZIP and OPF ourselves with zipfile + lxml could be faster, but it’s a lot of code. Are there other Python EPUB libraries that offer more granular control without the overhead? Would it make sense to fork ebooklib and swap out the XML backends for lxml? We’d love to hear your war stories—especially if you’ve built a similar translation or conversion pipeline.

This article was written by the LectuLibre engineering team. We’re building an AI‑powered book translation service—if you wrestle with EPUBs too, let’s talk!

Parsing and Rebuilding EPUB Files in Python: Lessons Learned from Building an AI Translation Service

龚旭东 — Sat, 20 Jun 2026 03:01:16 +0000

How we extract, translate, and reconstruct entire ebooks with Python while preserving every detail

At LectuLibre, we built a service that translates entire books using large language models. Our users upload EPUB files, and our backend pipeline parses them, extracts the text, sends it to an LLM for translation, and then rebuilds the EPUB with the translated content—all while preserving the original formatting, images, and metadata. This sounded straightforward until we looked inside a real EPUB.

EPUB is essentially a ZIP file containing a structured set of XHTML, CSS, and XML files. The content.opf file defines the reading order (spine), metadata, and manifest. The toc.ncx holds the table of contents. The actual text lives in XHTML documents, often split per chapter. To translate a book, we needed to: 1) reliably parse the EPUB, 2) locate all translatable text, 3) send it chunk by chunk to the LLM, and 4) rebuild the EPUB with the translated text while keeping every byte of the formatting intact.

The Problem with Off-the-Shelf Libraries

We initially reached for ebooklib, the most popular Python library for EPUB manipulation. It worked great for simple EPUBs—until we threw a few hundred real-world files at it. We quickly hit issues:

Metadata loss: ebooklib didn’t fully preserve custom metadata or namespace-prefixed properties in the OPF.
Namespace handling: When modifying XHTML, it could strip or mangle xmlns attributes, breaking rendering on some devices.
TOC and spine sync: After rebuilding, the table of contents and spine often got out of sync unless we manually repaired them.
Large files: Processing a 200‑chapter book consumed surprising memory because ebooklib loaded everything at once.

We could have used a heavyweight tool like Calibre’s command-line interface, but that introduced external dependencies and wasn’t as programmatically flexible. Instead, we decided to stick with ebooklib for high-level book structure and augment it with lxml for precise XML control.

Our Parsing and Rebuilding Pipeline

Here’s the core approach we landed on:

Read the EPUB with ebooklib to get a list of items (documents, images, CSS).
Identify translatable content – usually ITEM_DOCUMENT (XHTML) and sometimes ITEM_NAVIGATION (NCX for titles).
Parse each XHTML document with lxml, extract text, while keeping a map of each text node to its parent element.
Send blocks of text to the LLM for translation, preserving order and context.
Rebuild the XHTML by replacing original text nodes with their translations using the saved mapping.
Write the new EPUB with ebooklib, manually ensuring the OPF and spine are correct.

Let’s dive into the code.

Step 1: Reading and Filtering Items

import ebooklib
from ebooklib import epub

book = epub.read_epub('original.epub')

translatable_items = []
for item in book.get_items():
    if item.get_type() == ebooklib.ITEM_DOCUMENT:
        translatable_items.append(item)
    # Some books use NCX for chapter titles
    elif item.get_type() == ebooklib.ITEM_NAVIGATION:
        translatable_items.append(item)

We ignore images, fonts, and CSS—they don’t contain translatable text.

Step 2: Extracting Text with Context

We need to extract text while remembering exactly where it came from. We use lxml.etree to parse the XHTML and walk the tree, collecting text nodes and their XPath locations:

from lxml import etree

def extract_text_with_xpath(content):
    parser = etree.HTMLParser()
    root = etree.fromstring(content, parser)
    tree = etree.ElementTree(root)

    text_mapping = []  # list of (xpath, original_text, parent_element)
    for elem in root.iter():
        if elem.text and elem.text.strip():
            xpath = tree.getpath(elem)
            text_mapping.append((xpath, elem.text, elem))
        if elem.tail and elem.tail.strip():
            # tail text belongs to the parent, but logically follows the element
            parent = elem.getparent()
            xpath = tree.getpath(parent) if parent is not None else None
            if xpath:
                text_mapping.append((xpath, elem.tail, elem))
    return text_mapping

Pay attention to tail text—it’s the text that follows a closing tag, common in interleaved markup. Missing it leads to lost sentences.

Step 3: Translating in Chunks

We batch the collected text nodes into chunks that respect LLM token limits. For instance, we group consecutive text from the same XHTML document, aiming for ~3000 tokens per batch. We then send each chunk to our translation model (e.g., Claude 3.5 Sonnet) and receive a block of translated text. We split the translated block back into individual strings by comparing lengths (advanced: we use a diff algorithm to align original and translated sentences). This is simplified here for brevity.

Step 4: Replacing Text in the Original XHTML

Now we map translations back:

for (xpath, original, elem), translated_text in zip(text_mapping, translations):
    # Use xpath to locate the element again (parsed fresh from original)
    # but we cached the element objects, so we can just update them
    if elem.text and elem.text == original:
        elem.text = translated_text
    elif elem.tail and elem.tail == original:
        elem.tail = translated_text

# Serialize back to string
new_content = etree.tostring(root, encoding='unicode', method='html')

We return the modified XHTML as a string, ready to replace the item’s content in the EPUB.

Step 5: Rebuilding the EPUB

Here’s where ebooklib shines. We create a new EpubBook, set the same metadata (title, author, language), and add items:

new_book = epub.EpubBook()
new_book.set_identifier(original_book.get_metadata('DC', 'identifier')[0][0])
new_book.set_title(original_book.get_metadata('DC', 'title')[0][0])
new_book.set_language(original_book.get_metadata('DC', 'language')[0][0])

# Add all original items, replacing document content where needed
for item in original_book.get_items():
    if item.get_name() in modified_content_map:
        # Replace with translated XHTML
        new_content = modified_content_map[item.get_name()]
        new_item = epub.EpubItem(
            uid=item.get_id(),
            file_name=item.get_name(),
            media_type=item.get_type(),
            content=new_content.encode('utf-8')
        )
    else:
        # Copy image, CSS, etc. as-is
        new_item = item
    new_book.add_item(new_item)

# Replicate the spine and table of contents
new_book.spine = original_book.spine
new_book.toc = original_book.toc

# Write out
epub.write_epub('translated.epub', new_book, {})

But wait—this naive approach can corrupt the OPF. We found that ebooklib sometimes rewrites the spine order incorrectly if the original had complex nesting. To fix this, we manually post-process the written EPUB’s content.opf using lxml:

import zipfile
from lxml import etree

# Open the new EPUB as a ZIP
with zipfile.ZipFile('translated.epub', 'a') as zf:
    with zf.open('content.opf', 'r') as f:
        opf = etree.parse(f)
    # Ensure itemref order matches original spine
    spine = opf.find('.//{http://www.idpf.org/2007/opf}spine')
    # Reorder based on original spine list
    # ... custom correction logic ...
    zf.writestr('content.opf', etree.tostring(opf, xml_declaration=True, encoding='UTF-8'))

Yes, it’s ugly, but it saved us from countless validation errors.

Performance and Real-World Numbers

We benchmarked on a typical novel: 50 chapters, 350KB uncompressed. Parsing and extracting text: ~0.2 seconds. Rebuilding after translation: ~0.3 seconds. The LLM translation step dominates (around 45 seconds for the whole book), so we worked on parallelism for that part instead.

However, with larger educational texts containing hundreds of images and complex tables, memory usage spiked to over 500MB. We mitigated this by processing documents one by one and releasing them immediately.

Key Lessons Learned

Namespaces are the devil: Always preserve xmlns="http://www.w3.org/1999/xhtml" and any custom namespaces on the <html> tag. Lxml’s etree.tostring() with method='html' can drop them unless you explicitly add them back.
Validate, validate, validate: After rebuilding, we run epubcheck (via Python subprocess) to catch issues. False positives from custom metadata? We whitelist them after manual review.
Don’t trust the library for everything: ebooklib is great for reading, but for writing, we ended up doing a lot of OPF and NCX manipulation ourselves to ensure compliance.
Handle encoding upfront: Some old EPUBs use Latin-1. We transcode everything to UTF-8 early in the pipeline to avoid crashes later.
DRM is a dead end: We detect encrypted books by checking the <encryption> element in META-INF/encryption.xml and gracefully reject them.

The Open Question for the Community

We’d love to know how others are managing complex EPUB manipulation in production. Have you found a more robust library than ebooklib? How do you deal with interactive EPUB3 elements (Javascript, form fields) when translating? We’re still iterating on our pipeline and would appreciate any battle stories.

If you’re tackling similar problems or want to try translating your own eBooks, you can see the result of this work at LectuLibre. But most importantly, we hope this deep dive saves you a few late nights the next time you need to mess with EPUB internals.

How We Translate Entire Books with LLMs Without Losing Context

龚旭东 — Thu, 18 Jun 2026 23:19:00 +0000

Our chunking strategy that keeps chapters coherent, respects context windows, and handles multi-lingual books.

The problem: books don’t fit in a prompt

At LectuLibre, we translate entire books — novels, technical manuals, poetry — using large language models. It sounds simple: feed each paragraph to an LLM, concatenate results, done. But the moment we tried a 300‑page EPUB, chaos ensued. Chapters bled into each other, sentences were chopped mid‑word, and the translation of chapter 5 had no idea what happened in chapter 4.

LLMs have limited context windows. Even the massive 200K token window of Claude 3 can’t hold a whole 150K‑word book. And even if it could, the cost and latency would be absurd. We needed a way to split the book into manageable chunks while preserving enough context so that the translation remains coherent across thousands of pages.

Here’s how we designed a chunking pipeline that respects your wallet, the context window, and the book’s narrative flow.

Step 1: extract structure, not just text

Naively splitting by character count is a recipe for disaster. Instead, we first parse the document to understand its logical units: chapters, sections, headings. For EPUB, we use ebooklib; for PDF, pdfplumber. Both give us a stream of items (paragraphs, headings) that we then organize into a tree of chapters and sub‑sections.

import ebooklib
from ebooklib import epub

def get_chapters(epub_path):
    book = epub.read_epub(epub_path)
    chapters = []
    for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
        # Simplified: each document is a chapter
        content = item.get_content().decode('utf-8')
        chapters.append(content)
    return chapters

In practice, we use BeautifulSoup to extract <body> text and identify heading tags (<h1>–<h6>) to build a table of contents. This way, even if a chapter is 20,000 tokens, we keep it together as a single unit until later splitting.

Step 2: sentence‑aware splitting with token budgets

A chapter still needs to be broken down to fit the model’s context window. But we never split mid‑sentence. We use spaCy to tokenize the text into sentences, then greedily group them until we hit a token limit.

Why not simple character‑based splitting? Because sentences carry semantic boundaries. Breaking inside a sentence occasionally produces artefacts like “He walked to the sta‑” / “‑tion.” LLMs are forgiving but not that forgiving.

import spacy
from transformers import AutoTokenizer  # for accurate token count

nlp = spacy.load("en_core_web_sm")
tokenizer = AutoTokenizer.from_pretrained("claude-tokenizer")  # custom tokenizer for Claude

def sentence_split(text):
    doc = nlp(text)
    return [sent.text for sent in doc.sents]

def chunk_sentences(sentences, max_tokens=1800, overlap_sentences=5):
    chunks = []
    current_chunk = []
    current_token_count = 0

    for i, sent in enumerate(sentences):
        sent_tokens = len(tokenizer.encode(sent))
        if current_token_count + sent_tokens > max_tokens:
            # Store chunk with a sliding overlap
            chunks.append(current_chunk)
            # Overlap: take last `overlap_sentences` from the chunk just concluded
            current_chunk = sentences[i - overlap_sentences : i] if i - overlap_sentences > 0 else []
            current_token_count = sum(len(tokenizer.encode(s)) for s in current_chunk)
        current_chunk.append(sent)
        current_token_count += sent_tokens
    if current_chunk:
        chunks.append(current_chunk)
    return chunks

We set max_tokens to 1800, leaving room for the system prompt, context from previous chunks, and the model’s response. That’s for Claude Haiku, which has a 32K context window. For longer‑context models we’d scale up, but keeping chunks smaller also means faster, cheaper API calls.

Step 3: passing context across chunks

The real magic is what we do between chunks. A standalone translation of chunk #5 has no clue that the protagonist just entered a dark cave in chunk #4. Two techniques solved this:

Sliding window of previous sentences — we include the last 5–10 sentences from the preceding chunk directly in the prompt as “context left.”
A running summary — after translating a chunk, we ask the LLM to generate a one‑sentence summary of that chunk. This summary is accumulated and fed into every subsequent prompt, so the model remembers high‑level events.

def build_prompt(chunk, previous_context_sentences, summary_so_far):
    context_left = " ".join(previous_context_sentences)
    prompt = f"""You are translating a book. Here is a summary of the story so far:
    {summary_so_far}

    And the previous text (for immediate context):
    "{context_left}"

    Now translate the following text to Spanish, preserving tone and style:
    {chunk}"""
    return prompt

The summary is generated using a separate, cheap call (we use DeepSeek for summaries, even if the main translation uses Claude). This keeps the context token usage minimal while still giving long‑range coherence.

Why not just include the entire previous chunk? That doubles the token count per call. On a 200K‑word book, that adds up to hundreds of dollars. Summaries cut that cost by ~80% with negligible quality loss.

The translation loop then looks like this:

overall_summary = ""
previous_context = []
full_translation = []

for chapter_chunks in all_chunks_by_chapter:
    chapter_summary = ""
    for i, chunk in enumerate(chapter_chunks):
        prompt = build_prompt(
            " ".join(chunk),
            previous_context,
            chapter_summary + "\n" + overall_summary if i > 0 else ""
        )
        translated = call_llm(prompt)
        full_translation.append(translated)

        # Update context: keep last 5 sentences of the translated chunk as next context
        trans_sents = sentence_split(translated)
        previous_context = trans_sents[-5:]

        # Generate chunk summary asynchronously to save time
        chunk_summary = call_llm(f"Summarize this passage in one sentence: {chunk}")
        chapter_summary += chunk_summary + " "
    overall_summary += chapter_summary

We process chunks concurrently using asyncio and httpx to keep translation times reasonable.

Real‑world results and trade‑offs

Translating a 120K‑word Spanish novel (“El Quijote”) into English took about 4 minutes end‑to‑end with Claude 3 Haiku. Total API cost: $0.67. The translation was surprisingly fluid — chapters felt connected, and the occasional flashback or pronoun reference (“she” referring to a character introduced three pages earlier) was correctly resolved. Without the context pipeline, the same book would have been riddled with inconsistencies.

We experimented with other models: DeepSeek‑V3 gave similar quality at half the price but with higher latency, making it better for batch jobs where speed isn’t critical. GPT‑4 Turbo reproduced stylistic flourishes more naturally, but its 16K context window forced us to use even smaller chunks, which sometimes fragmented dialogue. Claude struck the best balance.

But it’s not perfect. Humor and idioms still occasionally fall flat because the summary can’t encapsulate a running joke. Code blocks and tables inside technical books need special handling — we’re working on a parser that detects them and wraps them in [CODE] markers so the LLM doesn’t try to translate variable names. And poetry, with its line breaks and meter, remains a challenge; we’re considering a dedicated poetry‑aware chunker.

The key takeaway

If you’re building long‑document translation using LLMs, invest in a pipeline that:

Respects document structure (chapters, paragraphs) before splitting.
Splits on sentences, and always leaves room for context.
Provides both immediate context (last few sentences) and global context (summaries) to each chunk.
Uses separate, cheap models for auxiliary tasks like summarization to keep costs down.

Our code is not open‑source yet, but we plan to release the core chunking library once we’ve battle‑tested it on more formats.

How do you handle context in LLM translations? We’re especially curious about handling highly technical books with equations, footnotes, and cross‑references. Drop your ideas in the comments — let’s figure this out together.

The Hidden Masterpieces of Japanese Crime Fiction: Novels Still Waiting for English Translation

龚旭东 — Thu, 18 Jun 2026 08:42:59 +0000

Discover the works of Seishi Yokomizo, Soji Shimada, and other giants of the genre that remain out of reach for English-only readers.

If you’re a fan of Japanese mystery novels in English, you’ve probably noticed a golden age of translation in recent years. Publishers like Pushkin Vertigo, Bento Books, and Locked Room International have brought us long-awaited classics by Seishi Yokomizo, Soji Shimada, and other masters of Japanese crime fiction. But for every Honjin Murders or Tokyo Zodiac Murders that finally lands on our shelves, dozens of equally brilliant works remain hidden behind the language barrier.

I still remember the frustration of finishing Yokomizo’s The Inugami Curse and discovering that only a handful of his 77 Kosuke Kindaichi novels had been translated. The same goes for Shimada’s intricate locked-room puzzles—the English-speaking world has tasted only a fraction of his genius. This article is for readers who want to go deeper, who refuse to let language be a wall between them and the stories they crave. We’ll explore the untranslated jewels of Japanese mystery, and I’ll show you practical ways to access them now.

Why Japanese Crime Fiction Captivates the World

Japanese mystery novels aren’t just puzzles with a Tokyo backdrop. They blend meticulous plotting with a literary sensibility that often feels more adult, more psychologically nuanced than their Western counterparts. Classic honkaku (orthodox) mysteries by authors like Yokomizo and Shimada treat the whodunit as a rigorous intellectual game, where every clue is laid out fairly for the reader. Yet the atmosphere is soaked in history, folklore, and post-war tension—elements that give these stories a depth beyond the puzzle.

Meanwhile, the social crime novels of Seicho Matsumoto and the psychological thrillers of Natsuo Kirino have already proven their international appeal. But beneath the well-translated surface lies a vast reservoir of mid-century and contemporary works that Anglophone readers have never seen. Many of these are considered landmarks in Japan, yet they remain invisible abroad simply because no publisher has taken a chance on them.

The Translation Gap: A Numbers Game

Let’s talk numbers. Seishi Yokomizo alone wrote over 70 novels featuring the scruffy, brilliant detective Kosuke Kindaichi. So far, Pushkin Vertigo has translated about 10 of them—impressive, but just the tip of the iceberg. Soji Shimada, the godfather of the logic-obsessed shin-honkaku movement, has had only a handful of his dozens of novels appear in English. And these are just the most famous names.

The reasons are complex. Japanese-to-English literary translation is costly and time-consuming. Publishers are often risk-averse, preferring to invest in authors with proven international sales records. Kulturkampf titles that require background knowledge of Japanese society can seem daunting to market. But the result is a frustrating feast-and-famine cycle: readers devour the few available translations and are left hungry for more.

The Authors Whose Best Work Remains Locked Away

Seishi Yokomizo: More Than the Honjin Murders

Yokomizo’s popularity in the West has soared since the translation of his debut, The Honjin Murders. Fans adore his macabre imagination, his rural settings steeped in family curses, and his detective Kindaichi, with his stutter and unassuming charm. But many of Yokomizo’s most acclaimed novels have never been translated.

For example, Akuma no Temari Uta (The Devil’s Hand-Tapping Song) is considered one of the finest locked-room mysteries in the Japanese canon. It involves a series of murders linked to an old nursery rhyme, and its denouement is both shocking and emotionally devastating. Another untranslated masterpiece is Byoinzaka no Kubikukuri no Ie (The House of the Hanging on Hospital Hill), a labyrinthine tale of family secrets and ritualized death. While we can hope Pushkin Vertigo will eventually bring these to light, for now they exist only in Japanese.

Soji Shimada: Puzzles Waiting to Be Solved

Soji Shimada’s Mitarai Kiyoshi series revolutionized the genre in the 1980s by embracing absurdly complex tricks and a Holmes-like detective. The Tokyo Zodiac Murders and Murder in the Crooked House have already achieved cult status in English. Yet some of Shimada’s most daring puzzles remain untranslated.

Consider Knight of the Underground Temple (Chikashitsuden no Kishi), in which a body is found inside a sealed underground chamber that appears both physically and magically locked. Or The Kobe Beef Murder Case (Kōbe Gyū Satsujin Jiken), a deliciously titled story involving a impossible crime on a train. Shimada’s ability to stretch plausibility to its breaking point while still playing fair is on full display in these works. Reading them is like watching a master magician reveal his secrets.

Other Giants of Untranslated Japanese Mystery

Beyond the two most famous names, the well of Japanese crime fiction goes deep. Tetsuya Ayukawa, known for his train alibi puzzles, has a vast oeuvre almost entirely untouched by English publishers. His detective, Inspector Onitsura, stars in dozens of stories that manipulate timetables with devilish precision. Shizuko Natsuki, a female pioneer of psychological suspense, wrote chilling novels like The Third Lady—but most of her bibliography, including her award-winning debut, Whispering Room, remains inaccessible. And Akimitsu Takagi’s later works, which blend horror with detection, also await discovery.

Specific Untranslated Novels You Should Know About

Here are five extraordinary Japanese mysteries that English readers can’t legally purchase in translation—and why they deserve to be on your radar.

Akuma no Temari Uta (The Devil’s Hand-Tapping Song) by Seishi Yokomizo

Why it matters: Often ranked among the greatest Japanese locked-room mysteries, this novel uses a macabre children’s song as the pattern for a series of murders. The solution is both ingenious and emotionally resonant, cementing Yokomizo’s reputation as a master of plotting.
Underground Temple Knight (Chikashitsuden no Kishi) by Soji Shimada

Why it matters: A quintessential Shimada locked-room extravaganza featuring a killing in a chamber sealed with magical symbols. The trick is characteristically wild yet logical, and the novel explores themes of obsession and illusion.
The Village of Doctor Ayakazu (Ayakazu-sense no Mura) by Tetsuya Ayukawa

Why it matters: An intricate alibi puzzle set in a remote village where a doctor’s murder forces Inspector Onitsura to untangle a web of train schedules and lies. Ayukawa’s work is a must for fans of Crofts or Freeman.
Whispering Room (Sasayaku heya) by Shizuko Natsuki

Why it matters: A tense psychological thriller about a woman whose new marriage hides dark secrets. Natsuki’s debut won the prestigious Edogawa Rampo Prize, yet it has never been published in English.
The Invisible Mask (Mienai Kamen) by Akimitsu Takagi

Why it matters: A later Takagi novel where a Noh mask seems to hold the key to an impossible disappearance. Blending Japanese tradition with pure detection, it shows a master at the height of his powers.

How to Read These Novels Now (Without Waiting for Publishers)

So what can you do if you’re dying to read these stories but don’t have strong Japanese? The good news is that technology and community have opened doors that didn’t exist a decade ago.

1. AI-Powered Translation Tools

If you can obtain a digital Japanese text (e.g., from an ebook store like BookWalker or a digital library), tools like LectuLibre let you upload the file and receive a full machine translation in minutes. While AI translations aren’t perfect—they can miss nuance, especially with cultural references or idiomatic dialogue—they have improved dramatically and can now produce readable, coherent prose. Many fans have already used these tools to enjoy untranslated manga and novels. If you’re patient and willing to gloss over occasional awkward phrasing, services like LectuLibre can be your personal key to the locked treasure chest of Japanese mystery.

2. Learn Japanese Through Your Love of Books

Obviously, learning Japanese is the ultimate solution. The journey from zero to reading a Yokomizo novel is long, but it’s a rewarding hobby that pays dividends across all your interests. Start with graded readers, move on to young adult mysteries, and gradually work your way up. The honkaku style, with its repetitive vocabulary around crime scenes and alibis, can actually be a helpful entry point for genre-focused learners.

3. Join Fan Translation Communities

Online forums like Reddit’s r/JapaneseMystery and dedicated Discord servers bring together bilingual fans who sometimes produce unofficial translations. While the quality varies, some are excellent and keep the torch burning for forgotten classics. Just remember to support official releases when they finally arrive.

4. Advocate and Wait

The more we talk about these books, the more likely publishers are to take notice. Review translated works, request them at your library, and mention untranslated titles on social media. Pushkin Vertigo and others have shown they listen to reader demand—The Honjin Murders itself was published after years of fan campaigning.

The Future of Japanese Mystery in English

The recent translation wave gives us hope. Yokomizo’s entire Kindaichi series may eventually see the light of day. Shimada might finally get the broader recognition his genius deserves. And new voices like Keigo Higashino continue to build bridges between cultures.

But there will always be more stories than there are translators and publishing slots. That’s where passion comes in. Whether you use AI tools, start learning Japanese, or simply voice your enthusiasm, you’re part of a global conversation that brings these novels across borders.

In the meantime, I’ll be over here trying to decipher the scene of the crime in Akuma no Temari Uta with a machine translation and a kanji dictionary. Won’t you join me?

Happy reading—or, should I say, happy detecting.

DEV Community: 龚旭东

How We Built Instant Translation Help (即时翻译帮助) with Python and LLMs

The Need for Instant Translation Help

Breaking Down the Problem

The Hybrid Approach: Glossary + LLM Fallback

Implementation Details

1. Preprocessing Pipeline

2. Resolving Ambiguity with Embeddings

3. The FastAPI Endpoint

Performance and Results

Lessons Learned & Trade‑offs

What’s Next?

Building Instant Translation Assistance for Book Translations with Python and LLMs

Problem: Real-time, Context-Aware Translation Inside a Book

Our Approach: Server‑Sent Events and a Smart Prompt Buffer

Implementation Deep Dive

1. Background Context Retrieval

2. Prompt Engineering for Instant Help

3. Streaming the Response with FastAPI and SSE

Trade-offs and Hard Decisions

Latency vs. Quality

Cost Management

Prompt Buffer Size

Results and What We Learned

Where We Might Go Next

How We Translate 300-Page Books Using Claude Without Hitting Token Limits

The Problem: Token Limits Are Real

Our Approach: Sliding Window with Overlapping Paragraphs

Implementation in Python with FastAPI

Parallel Translation and Performance

Lessons Learned

Where We Are Now

Parsing and Rebuilding EPUB Files in Python: Lessons Learned

The Problem: EPUB is a Messy Zip File

Our Approach: Use ebooklib with a Dose of Defensive Coding

Step 1: Loading and Validating the EPUB

Step 2: Extracting Readable Text from XHTML Documents

Step 3: Translating with an LLM (and Controlling Length)

Step 4: Rebuilding the XHTML with Translated Text

Step 5: Writing the Translated EPUB

Real‑World Pitfalls and How We Solved Them

Lessons Learned

What’s Next?

How We Built a Robust EPUB Parsing and Rebuilding Pipeline in Python

The Problem: EPUBs Are Zip Files of Chaos

First Attempt: The High‑Level Library Approach

The Hybrid Solution: ebooklib for Metadata, lxml for XHTML

Rebuilding the XHTML

Dealing with the Hard Stuff

1. Embedded Fonts and Binary Resources

2. Validation with epubcheck

3. Performance Tuning

Lessons Learned (The Hard Way)

Open Questions for the Community

Parsing and Rebuilding EPUB Files in Python: Lessons Learned from Building an AI Translation Service

The Problem with Off-the-Shelf Libraries

Our Parsing and Rebuilding Pipeline

Step 1: Reading and Filtering Items

Step 2: Extracting Text with Context

Step 3: Translating in Chunks

Step 4: Replacing Text in the Original XHTML

Step 5: Rebuilding the EPUB

Performance and Real-World Numbers

Key Lessons Learned

The Open Question for the Community

How We Translate Entire Books with LLMs Without Losing Context

The problem: books don’t fit in a prompt

Step 1: extract structure, not just text

Step 2: sentence‑aware splitting with token budgets

Step 3: passing context across chunks

Real‑world results and trade‑offs

The key takeaway

The Hidden Masterpieces of Japanese Crime Fiction: Novels Still Waiting for English Translation

Why Japanese Crime Fiction Captivates the World

The Translation Gap: A Numbers Game

The Authors Whose Best Work Remains Locked Away

Seishi Yokomizo: More Than the Honjin Murders

Soji Shimada: Puzzles Waiting to Be Solved

Other Giants of Untranslated Japanese Mystery

Specific Untranslated Novels You Should Know About

Our Approach: Use `ebooklib` with a Dose of Defensive Coding

The Hybrid Solution: `ebooklib` for Metadata, `lxml` for XHTML

2. Validation with `epubcheck`