DEV Community

龚旭东
龚旭东

Posted on

How We Built a Robust EPUB Parsing and Rebuilding Pipeline in Python

#ai #python #webdev #tutorial

Dealing with broken markup, embedded fonts, and namespace chaos while building LectuLibre's translation engine

At LectuLibre, we needed to translate entire EPUB books while preserving their exact visual structure. The core challenge: parse the EPUB, extract all translatable text, send it to an LLM, then reassemble the book with the translated content—images, CSS, fonts, and layout untouched. This turned out to be much harder than it looked. Here’s how we solved it, what broke, and what we learned.

The Problem: EPUBs Are Zip Files of Chaos

An EPUB is a ZIP archive containing XHTML, CSS, images, and a few XML control files (like container.xml and the OPF manifest). In theory, it’s a clean format. In practice, real‑world EPUBs are a mess:

  • XHTML with invalid markup, unclosed tags, or missing namespace declarations.
  • Embedded fonts, SVG chapters, and MathML that must be passed through untouched.
  • Text split across multiple inline elements (<b>Hello</b> <i>World</i>), requiring sentence‑aware translation.
  • The EPUB 3 spec is huge, and many books were generated by tools that barely follow it.

We needed a pipeline that could handle 90%+ of books without manual intervention, run fast enough for an interactive web service, and survive the most broken inputs we’d inevitably receive.

First Attempt: The High‑Level Library Approach

We reached for ebooklib—a dedicated Python library for reading and writing EPUB files. It gives you a nice object model: an EpubBook with items (documents, images, stylesheets), a spine, table of contents, and metadata. The code to open a book and grab all XHTML files looks deceptively simple:

import ebooklib
from ebooklib import epub

book = epub.read_epub('the-old-man-and-the-sea.epub')
for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
    content = item.get_content().decode('utf-8')
    # translate content ...
    item.set_content(translated_content.encode('utf-8'))

epub.write_epub('translated.epub', book)
Enter fullscreen mode Exit fullscreen mode

This works for many clean EPUBs. But when we stress‑tested it with 100 public‑domain books, we quickly hit walls:

  • Performance: ebooklib uses xml.dom.minidom internally; reading a 20 MB book with many XHTML files took over 6 seconds, and writing it back took even longer. Memory usage would spike to 1 GB+ because the entire DOM was held in memory.
  • Namespace handling: Some EPUB3 books use explicit XHTML namespaces everywhere (<html xmlns="http://www.w3.org/1999/xhtml">). ebooklib’s XML serialization would sometimes drop these namespaces, producing output that failed validation.
  • No fine‑grained text traversal: To replace only the displayed text while preserving markup, we had to parse the XHTML ourselves anyway.

Clearly, we needed something lower‑level for the actual content manipulation.

The Hybrid Solution: ebooklib for Metadata, lxml for XHTML

We settled on a hybrid architecture:

  • Use ebooklib to read and write the EPUB structure: the manifest, spine, TOC, and binary files (images, fonts). This saved us from having to reimplement the ZIP juggling and OPF generation.
  • For every XHTML file, we parse the content with lxml.etree (which is fast, namespaces‑aware, and can recover from broken markup). We walk the tree, extract translatable text segments, translate them, and then inject the translations back into the tree.

Here’s the core extraction logic:

from lxml import etree

def extract_translatable_blocks(html_bytes):
    parser = etree.HTMLParser(recover=True, encoding='utf-8')
    tree = etree.HTML(html_bytes, parser)
    # We only care about text that appears in the body.
    body = tree.find('.//body')
    if body is None:
        return []
    segments = []
    for element in body.iter():
        # Skip script, style, and void elements
        if element.tag in ('script', 'style', 'br', 'hr', 'img'):
            continue
        text = (element.text or '').strip()
        if text:
            segments.append((element, 'text', text))
        tail = (element.tail or '').strip()
        if tail:
            segments.append((element.getparent(), 'tail', tail))
    return segments
Enter fullscreen mode Exit fullscreen mode

Notice we track both element.text and element.tail—this is critical because in HTML like <p><b>Hello</b> <i>World</i></p>, the word “World” is actually the tail of the <b> element.

Before translation, we group adjacent text segments into sentences. Our sentencizer (a lightweight regex‑based splitter) joins text across inline tags, so we send a single unit "Hello World" to the AI instead of two separate fragments. After translation, we split the result back across the original boundaries, taking care to preserve leading/trailing whitespace.

Rebuilding the XHTML

Once the tree is modified, we serialize it back with namespace preservation:

def serialize_html(root_node):
    # lxml's etree.tostring handles namespaces correctly if you pass the tree with nsmap
    html_str = etree.tostring(
        root_node,
        method='html',
        encoding='unicode',
        xml_declaration=False,
        pretty_print=True
    )
    # Wrap back into a full XHTML document if needed
    return f'<?xml version="1.0" encoding="utf-8"?>\n<!DOCTYPE html>\n{html_str}'
Enter fullscreen mode Exit fullscreen mode

We then call item.set_content(serialized_html.encode('utf-8')) on the ebooklib item and write the book back out.

Dealing with the Hard Stuff

1. Embedded Fonts and Binary Resources

ebooklib handled images and fonts transparently as ITEM_IMAGE and ITEM_OTHER. We simply skip translation for non‑XHTML items. However, we discovered that some books rely on font‑face declarations in CSS that must remain valid after rebuild. We don’t modify CSS (translating content: "Chapter 1" would be suicidal), but we do parse each CSS to check for font‑face src URLs and ensure they are preserved as relative paths in the rebuilt EPUB.

2. Validation with epubcheck

We run every rebuilt EPUB through epubcheck (the official Java validator) as a final sanity check. Initially, 30% of our output files failed—mostly because ebooklib would omit the mimetype file entry at the beginning of the ZIP, or because we inadvertently stripped xml:lang attributes. We patched our write routine to always inject the mimetype file first, and we now preserve all XML namespaces and attributes during the lxml manipulation.

3. Performance Tuning

Processing a 500‑page novel end‑to‑end (parse, translate, rebuild) takes roughly:

  • EPUB read & XHTML extraction: 2–3 s CPU
  • Translation API calls: 8–12 s (dominated by LLM latency)
  • XHTML rebuild & EPUB write: 3–4 s CPU

We parallelise XHTML file processing with asyncio (asyncio.to_thread for lxml work) because each chapter is independent. This brings wall‑clock time down to about 10 seconds for a typical book—acceptable for a real‑time web service.

Memory usage stays stable at ~150–200 MB by avoiding loading huge DOMs simultaneously.

Lessons Learned (The Hard Way)

  1. High‑level libraries are a great start, but you’ll eventually need to understand the spec. ebooklib saved us weeks of work on the ZIP container and manifest. But debugging why a book wouldn’t open on iBooks meant reading the EPUB 3 spec and checking the OPF line by line.
  2. Test on real‑world garbage. We assembled a corpus of 100 public‑domain EPUBs from Project Gutenberg, Standard Ebooks, and random indie publications. About 15% were seriously broken (e.g., XHTML with three opening <body> tags). lxml’s recover=True was a lifesaver.
  3. Always validate after rebuild. Even if the book “looks fine” in Calibre, hidden structural errors will cause issues on other readers. Automate epubcheck in your CI.
  4. Namespaces will ruin your day. Always use lxml’s nsmap when parsing; never assume default namespace prefixes.
  5. Don’t translate everything. CSS content, <pre> formatted blocks, and math should be left alone. We filter out elements based on a configurable allow‑list.

Open Questions for the Community

We’re still not 100% satisfied with our pipeline. ebooklib is slow on large files due to its DOM‑based approach; rewriting the ZIP and OPF ourselves with zipfile + lxml could be faster, but it’s a lot of code. Are there other Python EPUB libraries that offer more granular control without the overhead? Would it make sense to fork ebooklib and swap out the XML backends for lxml? We’d love to hear your war stories—especially if you’ve built a similar translation or conversion pipeline.

This article was written by the LectuLibre engineering team. We’re building an AI‑powered book translation service—if you wrestle with EPUBs too, let’s talk!

Top comments (1)

Collapse
 
nazar-boyko profile image
Nazar Boyko •

The catch about text versus tail is one of those things you only learn by getting burned, so calling it out is doing readers a favor. The part I keep poking at is splitting the translated sentence back across the original inline boundaries, because word order moves between languages. If "Hello World" has the bold on "Hello" and the target language flips the two, the bold should now wrap a different word, and a naive split back by position would land the emphasis on the wrong one. How are you mapping the inline tags onto the translated text, or is keeping that styling roughly in place good enough in practice?