DEV Community

nan nan
nan nan

Posted on

Ebook Voice Player Document Parsing Engine Deep Dive: Unified Multi-Format Abstraction & Smart Chapter Detection

Product Link: Microsoft Store - Ebook Voice Player

Introduction

Ebook Voice Player supports TXT, DOCX, PDF, EPUB, HTML — five major e-book formats. This article deep-dives into the document parsing engine in reader.py, covering unified interface design, auto-encoding detection, dual-engine PDF fallback, EPUB semantic parsing, and the core smart chapter detection algorithm.


1. Unified Abstraction Interface Design

1.1 Single Entry Point, Format Differences Hidden

SUPPORTED_SUFFIXES = {".txt", ".docx", ".doc", ".pdf", ".epub", ".html", ".htm"}

def read_file(filepath):
    """Unified entry: file path → paragraph list"""
    path = Path(filepath)
    suffix = path.suffix.lower()
    if suffix == ".txt":
        return read_txt(filepath)
    elif suffix in (".docx", ".doc"):
        return read_docx(filepath)
    elif suffix == ".pdf":
        return read_pdf(filepath)
    elif suffix == ".epub":
        return read_epub(filepath)
    else:
        raise ValueError(f"Unsupported format: {suffix}")
Enter fullscreen mode Exit fullscreen mode

Design Principles:

  • Zero Awareness Upstream: Caller only needs paragraphs = read_file(path), no format knowledge required
  • Unified Exceptions: Consistent ValueError / ImportError for uniform error handling
  • Extensible: New formats = add elif branch + parser function

1.2 Batch Folder Reading with Source Tracking

def read_folder(directory):
    """Read all supported files in directory, merge paragraphs, track source"""
    entries = sorted(os.listdir(directory), key=str.lower)
    files = []
    for name in entries:
        if is_auxiliary_file(name):  # Skip auxiliary files like 术语表.txt
            continue
        full = os.path.join(directory, name)
        if os.path.isfile(full) and Path(name).suffix.lower() in SUPPORTED_SUFFIXES:
            files.append(full)

    paragraphs = []
    source_index = []          # Start index of each file in paragraph list
    prev = 0
    for f in files:
        try:
            new = read_file(f)
        except Exception:
            new = []  # Single file failure doesn't block batch
        source_index.append(prev)
        paragraphs.extend(new)
        prev = len(paragraphs)

    # Tag each paragraph with source file (for UI chapter attribution)
    source_of = []
    for i, start in enumerate(source_index):
        end = source_index[i + 1] if i + 1 < len(source_index) else len(paragraphs)
        source_of.extend([os.path.basename(files[i])] * (end - start))

    return paragraphs, files, source_of
Enter fullscreen mode Exit fullscreen mode

2. TXT: Auto-Encoding Detection & Robust Reading

2.1 Multi-Encoding Probe Strategy

def read_txt(filepath, encoding=None):
    # Try UTF-8 with BOM first, then common CJK encodings
    encodings = encoding or ["utf-8-sig", "utf-8", "gbk", "gb2312", "gb18030",
                             "big5", "utf-16", "shift-jis", "euc-jp"]
    if isinstance(encodings, str):
        encodings = [encodings]

    for enc in encodings:
        try:
            with open(filepath, "r", encoding=enc) as f:
                text = f.read()
            break  # Success → exit loop
        except (UnicodeDecodeError, UnicodeError):
            continue
    else:
        # Fallback: replace mode guarantees no exception
        with open(filepath, "r", encoding="utf-8", errors="replace") as f:
            text = f.read()

    # Strip BOM
    if text.startswith("\ufeff"):
        text = text[1:]

    # Split by line, filter empty
    paragraphs = [p.strip() for p in text.split("\n") if p.strip()]
    return paragraphs
Enter fullscreen mode Exit fullscreen mode

2.2 Encoding Probe Priority Rationale

Priority Encoding Use Case
1 utf-8-sig UTF-8 with BOM (Windows Notepad default)
2 utf-8 Standard UTF-8 no BOM
3 gbk/gb2312/gb18030 Simplified Chinese legacy
4 big5 Traditional Chinese legacy
5 utf-16 Unicode double-byte
6 shift-jis Japanese legacy
7 euc-jp Japanese legacy
Fallback utf-8, errors="replace" Unknown encoding, replace undecodable with �

3. DOCX: Structured Paragraph Extraction

def read_docx(filepath):
    import docx
    doc = docx.Document(filepath)
    paragraphs = []
    for p in doc.paragraphs:
        text = p.text.strip()
        if text:
            paragraphs.append(text)
    return paragraphs
Enter fullscreen mode Exit fullscreen mode

Characteristics:

  • Preserves Word's native paragraph structure
  • Auto-skips empty paragraphs
  • Tables, headers/footers, images auto-skipped (extensible if needed)

4. PDF: Dual-Engine Fallback & Text Extraction Strategy

4.1 PyMuPDF (fitz) First: Fast & Accurate

def read_pdf(filepath):
    paragraphs = []
    try:
        import fitz  # PyMuPDF
        doc = fitz.open(filepath)
        for page in doc:
            text = page.get_text().strip()
            if text:
                for p in text.split("\n"):
                    p = p.strip()
                    if p:
                        paragraphs.append(p)
        doc.close()
    except ImportError:
        # Fallback to pdfplumber
        try:
            import pdfplumber
            with pdfplumber.open(filepath) as pdf:
                for page in pdf.pages:
                    text = page.extract_text()
                    if text:
                        for p in text.split("\n"):
                            p = p.strip()
                            if p:
                                paragraphs.append(p)
        except ImportError:
            raise ImportError("Please install PyMuPDF or pdfplumber: pip install PyMuPDF pdfplumber")
    return paragraphs
Enter fullscreen mode Exit fullscreen mode

4.2 Dual-Engine Comparison & Selection

Dimension PyMuPDF (fitz) pdfplumber
Parse Speed ⚡ Blazing (C++ core) Slower
Text Accuracy High High
Table Extraction Basic Strong (structured tables)
Dependency Size Smaller Larger (includes pdfminer)
Chinese Support Excellent Excellent
Scanned PDF ❌ No text layer ❌ No text layer

Strategy: Default PyMuPDF, auto-fallback to pdfplumber if missing — user-transparent.

4.3 Scanned PDF Limitation Note

Important: Both engines do not support scanned/image-based PDFs (no text layer). Such files require OCR (Adobe Acrobat, ABBYY, PaddleOCR) to generate searchable PDF first, then import.


5. EPUB: Semantic HTML Parsing

5.1 Standard EPUB Structure Traversal

def read_epub(filepath):
    try:
        import ebooklib
        from ebooklib import epub
        from bs4 import BeautifulSoup
    except ImportError:
        raise ImportError("Please install ebooklib: pip install EbookLib")

    book = epub.read_epub(filepath)
    paragraphs = []

    for item in book.get_items():
        if item.get_type() == ebooklib.ITEM_DOCUMENT:
            content = item.get_content()
            soup = BeautifulSoup(content, "html.parser")
            # Preserve semantic tag hierarchy: headings, paragraphs, div, span
            for p in soup.find_all(["p", "h1", "h2", "h3", "h4", "h5", "h6", "div", "span"]):
                text = p.get_text().strip()
                if text:
                    paragraphs.append(text)
    return paragraphs
Enter fullscreen mode Exit fullscreen mode

5.2 Semantic Tag Preservation Value

Tag Semantics TTS Value
h1-h6 Chapter headings Chapter detection, navigation
p Body paragraphs Core reading units
div/span Containers/inline Fallback for non-standard markup

Note: EPUBs contain CSS/JS/fonts; ebooklib auto-filters to ITEM_DOCUMENT only.


6. Smart Chapter Detection Algorithm: Core Highlight

6.1 Multi-Pattern Regex System

def detect_chapters(paragraphs):
    # Primary patterns: standard chapter markers
    primary_patterns = [
        r"^第[一二三四五六七八九十百千万零〇\d]+[章卷部篇]",  # 第N章/卷/部/篇
        r"^第\s*\d+\s*[章卷部篇]",                          # 第 N 章
        r"^Chapter\s+\d+",                                   # Chapter N
        r"^CHAPTER\s+\d+",                                   # CHAPTER N
    ]

    # Secondary patterns: sub-sections/parts/volumes
    secondary_patterns = [
        r"^第[一二三四五六七八九十百千万零〇\d]+[话节集]",
        r"^第\s*\d+\s*[话节集]",
        r"^[Pp]art\s+\d+",
        r"^[Ss]ection\s+\d+",
        r"^[Vv]olume\s+\d+",
    ]

    primary = [re.compile(p) for p in primary_patterns]
    secondary = [re.compile(p) for p in secondary_patterns]
    ...
Enter fullscreen mode Exit fullscreen mode

6.2 Two-Level Heading Fusion Strategy

raw_chapters = []
for i, para in enumerate(paragraphs):
    stripped = para.strip()
    if not stripped:
        continue
    matched = False
    # Try primary first
    for cp in primary:
        if cp.match(stripped):
            raw_chapters.append((i, stripped[:60], "primary"))
            matched = True
            break
    # Fallback to secondary
    if not matched:
        for cp in secondary:
            if cp.match(stripped):
                raw_chapters.append((i, stripped[:60], "secondary"))
                break

# Fusion logic: secondary within ≤3 paras of primary → merge, not separate chapter
if not raw_chapters:
    if paragraphs:
        return [(0, "全文 / Full Text")]
    return []

chapters = [(raw_chapters[0][0], raw_chapters[0][1])]
for i in range(1, len(raw_chapters)):
    pos, title, kind = raw_chapters[i]
    prev_pos, prev_title, prev_kind = raw_chapters[i - 1]
    gap = pos - prev_pos
    # Secondary after primary with gap ≤3 → merge
    if kind == "secondary" and prev_kind == "primary" and gap <= 3:
        continue
    chapters.append((pos, title))

return chapters
Enter fullscreen mode Exit fullscreen mode

6.3 Algorithm Design Rationale

Paragraph Index:     0    1    2    3    4    5    6    7    8    9   10   11
Content:           Ch1  text text Sec1 text Ch2  text Sec2.1 text  Ch3  ...
Match Type:        P1   -    -    P2   -    P1   -    P2     -    P1
Gap:               -    -    -    3    -    2    -    1      -    -
Fusion Result:     ✓Ch1       (merged)     ✓Ch2     (merged)      ✓Ch3
Enter fullscreen mode Exit fullscreen mode

Core Rules:

  1. Primary First: Chapter N / 第N章 strong signals take priority
  2. Secondary Proximity Merge: Section / 第N节 within 3 paras of primary → subordinate, not separate chapter
  3. Fallback: No matches → entire book as single chapter [(0, "Full Text")]

6.4 Chapter Segment Retrieval

def get_chapter_paragraphs(paragraphs, chapter_index, chapters):
    if not chapters:
        return paragraphs
    start = chapters[chapter_index][0]
    if chapter_index + 1 < len(chapters):
        end = chapters[chapter_index + 1][0]
        return paragraphs[start:end]
    else:
        return paragraphs[start:]  # Last chapter to end
Enter fullscreen mode Exit fullscreen mode

7. Term Replacement: Personalized Pronunciation Correction

7.1 Term File Format

# 名词转换.txt example
原词<TAB>替换词
哈利波特    Harry Potter
伏地魔   Voldemort
霍格沃茨    Hogwarts
# Comment lines start with #
特殊空白    (空白符)  # Replace with empty string
Enter fullscreen mode Exit fullscreen mode

7.2 Parsing & Application

def parse_term_file(filepath):
    term_map = {}
    text = Path(filepath).read_text(encoding="utf-8")
    for raw in text.splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        parts = [p.strip() for p in line.split("\t")]
        parts = [p for p in parts if p]
        if len(parts) < 2:
            continue
        key = parts[0]
        val = parts[-1]
        if key == val:
            continue
        if val == "(空白符)":  # Special marker: replace with empty
            val = ""
        term_map[key] = val
    return term_map

def apply_term_map(text, term_map):
    """Longest-first replacement: 'Sophilia' before 'Sophi'"""
    if not term_map or not text:
        return text
    for key in sorted(term_map, key=len, reverse=True):
        text = text.replace(key, term_map[key])
    return text
Enter fullscreen mode Exit fullscreen mode

Use Cases:

  • Proper noun pronunciation correction
  • Abbreviation expansion: CPU → central processing unit
  • Dialect/colloquial standardization
  • Sensitive word masking (replace with empty)

8. Performance & Memory Optimization

Optimization Implementation Effect
Streaming Read PDF/EPUB page-by-page/item-by-item, no full DOM load Peak memory < 50MB
Error Isolation Single file parse failure doesn't block batch folder read Robustness
Encoding Cache Reuse detected encoding for multiple TXT in same dir Faster re-open
Paragraph Filter filter_short_paragraphs(min_length=2) denoising Reduces useless synthesis

9. Extending New Formats Guide

9.1 Steps to Add Format

# 1. Add suffix to SUPPORTED_SUFFIXES
SUPPORTED_SUFFIXES = {".txt", ".docx", ".doc", ".pdf", ".epub", ".html", ".htm", ".mobi"}

# 2. Implement parser
def read_mobi(filepath):
    # Use mobi lib or calibre toolchain
    ...

# 3. Add dispatch branch in read_file
elif suffix == ".mobi":
    return read_mobi(filepath)
Enter fullscreen mode Exit fullscreen mode

9.2 Recommended Libraries

Format Library Notes
MOBI mobi / kindle-unpack Handle DRM restrictions
AZW3 calibre Heavy dep, prefer EPUB conversion
FB2 lxml direct XML parse Simple structure
RTF striprtf Plain text extraction

10. Troubleshooting Common Issues

Symptom Cause Fix
PDF no text/garbled Scanned/encrypted/font embedding OCR / decrypt / print to PDF
EPUB chapters missing Non-standard markers (e.g., class="chapter") Edit EPUB or add custom patterns
TXT garbled Encoding not in probe list Specify encoding param or save as UTF-8
DOCX tables lost Only paragraphs extracted Extend to traverse doc.tables
Chapter detection off Regex doesn't match specific style Add to primary/secondary_patterns

11. Conclusion

Ebook Voice Player's document parsing engine embodies pragmatic engineering balancing UX:

  1. Unified Abstraction lets upstream business ignore format differences
  2. Dual-Engine Fallback maximizes PDF parse availability
  3. Smart Chapter Detection covers CN/EN/JP/KR writing conventions, two-level fusion prevents fragmentation
  4. Term Replacement solves proper noun pronunciation pain points with personalization
  5. Error Isolation & Streaming ensures stability on large files/batch reads

This engine powers millions of document imports for tens of thousands of users in production.


Try It: Microsoft Store - Ebook Voice Player

Source Code: reader.py / tts_app/reader.py

Top comments (0)