Parsing and Rebuilding EPUB Files in Python: Lessons from Building an AI Book Translator

How we used ebooklib and Beautiful Soup to process thousands of EPUBs without losing metadata, formatting, or our sanity.

The Dream: One-Click Book Translation

When we started building LectuLibre, an AI-powered book translation service, we knew the core challenge wouldn't be the translation itself. LLMs like Claude and DeepSeek are remarkably good at handling text. The real headache was the book container: EPUB files. Users upload an EPUB, we translate it, and they download a perfectly formatted translated book. Sound simple? So did we—until we actually tried it.

EPUB is a deceptively complex format. It's a ZIP archive containing XHTML chapters, CSS, images, fonts, and a few XML control files (notably the content.opf and toc.ncx). To translate a book, you must:

1. Extract all text from the XHTML files while preserving the surrounding markup.
2. Translate only the text nodes, leaving tags, anchors, and images untouched.
3. Rebuild the EPUB with the same metadata, spine order, and resources.

This is a parsing and restructuring problem that sits somewhere between web scraping and document assembly. Here's how we tackled it with Python, and what we learned along the way.

Why Not Just Use Calibre?

Calibre is the Swiss Army knife of ebooks, but it's a desktop application, not a library we could embed in a FastAPI service. We needed something lightweight, scriptable, and open-source. Enter ebooklib, a pure-Python library that reads and writes EPUB files. It does the job, but as we discovered, it has quirks you'll only find when you throw thousands of real-world EPUBs at it.

Our Parsing Pipeline: ebooklib + Beautiful Soup

The basic workflow with ebooklib looks like this:

from ebooklib import epub

book = epub.read_epub('book.epub')

That one-liner unpacks the ZIP, parses the XML, and gives you a Book object. But the devil is in the details. Here's the actual production code we landed on:

import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup

def parse_epub(filepath: str):
    book = epub.read_epub(filepath, options={'ignore_ncx': False})
    items = list(book.get_items_of_type(ebooklib.ITEM_DOCUMENT))

    chapters = []
    for item in items:
        content = item.get_content().decode('utf-8')
        soup = BeautifulSoup(content, 'lxml-xml')  # Use XML parser to avoid HTML closing tag issues
        # Extract text nodes while preserving structure
        chapters.append({
            'id': item.get_id(),
            'soup': soup,
            'href': item.get_name()
        })
    return book, chapters

We used lxml-xml as the BeautifulSoup parser because EPUB XHTML is often XML, not HTML. With the default HTML parser, self-closing tags like <br/> got mangled. A subtle but critical choice.

Navigating the OPF Spine

The content.opf defines the reading order via the <spine> element. Ebooklib represents this as book.spine. Each spine item has an idref that points to a manifest item. So to walk the book in order:

spine_order = []
for spine_item in book.spine:
    item = book.get_item_with_id(spine_item[0])
    if item.get_type() == ebooklib.ITEM_DOCUMENT:
        spine_order.append(item)

But we found that some EPUBs have spine entries pointing to non-existent IDs. Our approach: silently skip them and log a warning. Robustness over strictness.

Translating In-Place Without Breaking Markup

After extracting the text, we send it to an LLM in chunks (respecting token limits). The tricky part is replacing the original text with the translated version while keeping every tag, class, and attribute intact.

We traverse the BeautifulSoup tree and only replace NavigableString nodes:

def translate_soup(soup: BeautifulSoup, translation_map: dict) -> BeautifulSoup:
    for text_node in soup.find_all(string=True):
        stripped = text_node.strip()
        if stripped and stripped in translation_map:
            text_node.replace_with(translation_map[stripped])
    return soup

This works for simple cases, but we quickly ran into issues when the translation changed the number of paragraphs (e.g., a single <p> became two). The solution? We stitch translations back by splitting on paragraph boundaries and reassembling manually. It's not perfect, but it handles 95% of cases.

Rebuilding the EPUB: Writing Back Without Losing Your Mind

Once all soup objects have been translated, we rebuild the EPUB:

def save_epub(book, chapters, output_path):
    for ch in chapters:
        item = book.get_item_with_id(ch['id'])
        # Update content with translated soup
        item.set_content(str(ch['soup']).encode('utf-8'))

    # Write to disk
    epub.write_epub(output_path, book)

Seems simple, but ebooklib's write_epub can create invalid archives if you’ve added or removed items without updating the manifest and spine. Our rule: never add or remove items; only modify existing XHTML documents. This preserves the existing content.opf and avoids UUID mismatches.

A Word on Memory

Processing a 200 MB EPUB with hundreds of images can spike memory to over 1 GB with naive loading. We avoid that by selectively loading only document items and not images:

book = epub.read_epub(filepath, options={'ignore_ncx': False, 'expand_css': False})
# Don't load images or fonts unless needed
for item in book.get_items():
    if item.get_type() != ebooklib.ITEM_DOCUMENT:
        item.content = b''  # Clear binary content to free memory early

This dropped memory usage by 60% on large files.

Dealing with Concurrency

In our FastAPI backend, we handle uploads asynchronously. We stream the file to a temporary location, then hand it off to a translation worker. Because the translation step is CPU-bound and memory-heavy, we use a ThreadPoolExecutor with a limited number of workers (max cpu_count()). This keeps our VPS responsive and avoids OOM kills.

Wrestling with Malformed EPUBs

Real-world EPUBs are a mess. We've encountered:

• Missing content.opf → fall back to looking for any .opf file in the ZIP.
• Non-UTF-8 encodings → we use chardet to detect encoding before decoding.
• Cyclic references in the manifest → we added a visited set to our parsing loop.
• Empty spine → we treat all XHTML files as a reading order.
• CSS with @import → we inline critical styles because Kindle doesn't support imports.
• External fonts loaded from CDNs → we swap in a local fallback.
• CDATA sections wrapping entire documents → we had to pre-process the XML to unwrap them.

We implemented a preflight validation step that checks for these issues and tries to auto-heal where possible. For the tough cases, we notify the user they uploaded a "quirky" file, and our AI will still do its best.

The Table of Contents Conundrum

Many translators overlook the navigation. We initially left the toc.ncx (or nav.xhtml for EPUB3) untouched—but then users complained that the table of contents was still in the original language. We're now developing a separate step that translates the nav labels (the <navLabel> elements) using the same LLM, but with a simpler context. The challenge is that some labels are just numbers or chapter abbreviations, and over-translating breaks the user experience. We're exploring a hybrid approach: only translate labels that contain natural language.

Performance and Scalability Numbers

Our backend is a FastAPI app on a 4‑core VPS. A typical translation workflow takes:

• Parsing: 0.2–0.5 seconds for a 300‑page novel.
• Translation (LLM call): 30–50 seconds (the bottleneck by far).
• Rebuilding EPUB: 0.5–1.5 seconds depending on complexity.

We can handle 8 concurrent translations before hitting the LLM API rate limit or memory pressure. Our worker pool uses asyncio to manage these, with a semaphore limiting parallelism.

Testing with a Corpus of Horrors

To ensure we don't regress, we built a corpus of 500 reference EPUBs—including many deliberately broken ones. Before every deploy, we run integration tests that:

• Check that the spine order is preserved.
• Verify that the number of XHTML files remains unchanged.
• Compare the translated output against a human‑reviewed gold standard for a subset.
• Validate the output EPUB with epubcheck (the IDPF tool).

This saved us multiple times when a library update changed parsing behavior.

Lessons Learned

1. Never trust an EPUB. Always assume the XML might be invalid.
2. Separation of concerns: Keep parsing, translation, and rebuilding as separate, testable functions.
3. ebooklib is great but not bulletproof. For advanced features like media overlays or EPUB3 audio, you may need to manipulate the ZIP directly.
4. Memory matters. Don't load binary blobs you don't need.
5. Test with a corpus. We maintain a set of 500 reference EPUBs (including broken ones) to run before every deploy.
6. Navigation is part of the content. Translating the book but not the TOC leads to an inconsistent experience.

The Future: PDFs and Streaming

PDF parsing is next on our roadmap, with an entirely different set of challenges (we're eyeing pymupdf). But the EPUB pipeline taught us that ebooks are closer to web pages than to documents, and using web-native tools (BeautifulSoup, CSS inlining) pays off.

If you're building something similar, I'd love to hear your approach. Did you stick with ebooklib or roll your own? How do you handle RTL languages? Join the discussion in the comments.

—

LectuLibre is in private beta at lectulibre.com. We built this pipeline because we love books and believe language shouldn't be a barrier.

Comments

[Alex Shev]

EPUB translation is a great example of AI being only one part of the system. Preserving structure, references, images, formatting, and chapter boundaries is the real product work. The translation model matters, but the rebuild pipeline is what users feel.