Product Link: Microsoft Store - Ebook Voice Player
Introduction
Ebook Voice Player supports TXT, DOCX, PDF, EPUB, HTML — five major e-book formats. This article deep-dives into the document parsing engine in reader.py, covering unified interface design, auto-encoding detection, dual-engine PDF fallback, EPUB semantic parsing, and the core smart chapter detection algorithm.
1. Unified Abstraction Interface Design
1.1 Single Entry Point, Format Differences Hidden
SUPPORTED_SUFFIXES = {".txt", ".docx", ".doc", ".pdf", ".epub", ".html", ".htm"}
def read_file(filepath):
"""Unified entry: file path → paragraph list"""
path = Path(filepath)
suffix = path.suffix.lower()
if suffix == ".txt":
return read_txt(filepath)
elif suffix in (".docx", ".doc"):
return read_docx(filepath)
elif suffix == ".pdf":
return read_pdf(filepath)
elif suffix == ".epub":
return read_epub(filepath)
else:
raise ValueError(f"Unsupported format: {suffix}")
Design Principles:
-
Zero Awareness Upstream: Caller only needs
paragraphs = read_file(path), no format knowledge required -
Unified Exceptions: Consistent
ValueError/ImportErrorfor uniform error handling -
Extensible: New formats = add
elifbranch + parser function
1.2 Batch Folder Reading with Source Tracking
def read_folder(directory):
"""Read all supported files in directory, merge paragraphs, track source"""
entries = sorted(os.listdir(directory), key=str.lower)
files = []
for name in entries:
if is_auxiliary_file(name): # Skip auxiliary files like 术语表.txt
continue
full = os.path.join(directory, name)
if os.path.isfile(full) and Path(name).suffix.lower() in SUPPORTED_SUFFIXES:
files.append(full)
paragraphs = []
source_index = [] # Start index of each file in paragraph list
prev = 0
for f in files:
try:
new = read_file(f)
except Exception:
new = [] # Single file failure doesn't block batch
source_index.append(prev)
paragraphs.extend(new)
prev = len(paragraphs)
# Tag each paragraph with source file (for UI chapter attribution)
source_of = []
for i, start in enumerate(source_index):
end = source_index[i + 1] if i + 1 < len(source_index) else len(paragraphs)
source_of.extend([os.path.basename(files[i])] * (end - start))
return paragraphs, files, source_of
2. TXT: Auto-Encoding Detection & Robust Reading
2.1 Multi-Encoding Probe Strategy
def read_txt(filepath, encoding=None):
# Try UTF-8 with BOM first, then common CJK encodings
encodings = encoding or ["utf-8-sig", "utf-8", "gbk", "gb2312", "gb18030",
"big5", "utf-16", "shift-jis", "euc-jp"]
if isinstance(encodings, str):
encodings = [encodings]
for enc in encodings:
try:
with open(filepath, "r", encoding=enc) as f:
text = f.read()
break # Success → exit loop
except (UnicodeDecodeError, UnicodeError):
continue
else:
# Fallback: replace mode guarantees no exception
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
text = f.read()
# Strip BOM
if text.startswith("\ufeff"):
text = text[1:]
# Split by line, filter empty
paragraphs = [p.strip() for p in text.split("\n") if p.strip()]
return paragraphs
2.2 Encoding Probe Priority Rationale
| Priority | Encoding | Use Case |
|---|---|---|
| 1 | utf-8-sig |
UTF-8 with BOM (Windows Notepad default) |
| 2 | utf-8 |
Standard UTF-8 no BOM |
| 3 |
gbk/gb2312/gb18030
|
Simplified Chinese legacy |
| 4 | big5 |
Traditional Chinese legacy |
| 5 | utf-16 |
Unicode double-byte |
| 6 | shift-jis |
Japanese legacy |
| 7 | euc-jp |
Japanese legacy |
| Fallback | utf-8, errors="replace" |
Unknown encoding, replace undecodable with � |
3. DOCX: Structured Paragraph Extraction
def read_docx(filepath):
import docx
doc = docx.Document(filepath)
paragraphs = []
for p in doc.paragraphs:
text = p.text.strip()
if text:
paragraphs.append(text)
return paragraphs
Characteristics:
- Preserves Word's native paragraph structure
- Auto-skips empty paragraphs
- Tables, headers/footers, images auto-skipped (extensible if needed)
4. PDF: Dual-Engine Fallback & Text Extraction Strategy
4.1 PyMuPDF (fitz) First: Fast & Accurate
def read_pdf(filepath):
paragraphs = []
try:
import fitz # PyMuPDF
doc = fitz.open(filepath)
for page in doc:
text = page.get_text().strip()
if text:
for p in text.split("\n"):
p = p.strip()
if p:
paragraphs.append(p)
doc.close()
except ImportError:
# Fallback to pdfplumber
try:
import pdfplumber
with pdfplumber.open(filepath) as pdf:
for page in pdf.pages:
text = page.extract_text()
if text:
for p in text.split("\n"):
p = p.strip()
if p:
paragraphs.append(p)
except ImportError:
raise ImportError("Please install PyMuPDF or pdfplumber: pip install PyMuPDF pdfplumber")
return paragraphs
4.2 Dual-Engine Comparison & Selection
| Dimension | PyMuPDF (fitz) | pdfplumber |
|---|---|---|
| Parse Speed | ⚡ Blazing (C++ core) | Slower |
| Text Accuracy | High | High |
| Table Extraction | Basic | Strong (structured tables) |
| Dependency Size | Smaller | Larger (includes pdfminer) |
| Chinese Support | Excellent | Excellent |
| Scanned PDF | ❌ No text layer | ❌ No text layer |
Strategy: Default PyMuPDF, auto-fallback to pdfplumber if missing — user-transparent.
4.3 Scanned PDF Limitation Note
Important: Both engines do not support scanned/image-based PDFs (no text layer). Such files require OCR (Adobe Acrobat, ABBYY, PaddleOCR) to generate searchable PDF first, then import.
5. EPUB: Semantic HTML Parsing
5.1 Standard EPUB Structure Traversal
def read_epub(filepath):
try:
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup
except ImportError:
raise ImportError("Please install ebooklib: pip install EbookLib")
book = epub.read_epub(filepath)
paragraphs = []
for item in book.get_items():
if item.get_type() == ebooklib.ITEM_DOCUMENT:
content = item.get_content()
soup = BeautifulSoup(content, "html.parser")
# Preserve semantic tag hierarchy: headings, paragraphs, div, span
for p in soup.find_all(["p", "h1", "h2", "h3", "h4", "h5", "h6", "div", "span"]):
text = p.get_text().strip()
if text:
paragraphs.append(text)
return paragraphs
5.2 Semantic Tag Preservation Value
| Tag | Semantics | TTS Value |
|---|---|---|
h1-h6
|
Chapter headings | Chapter detection, navigation |
p |
Body paragraphs | Core reading units |
div/span
|
Containers/inline | Fallback for non-standard markup |
Note: EPUBs contain CSS/JS/fonts; ebooklib auto-filters to ITEM_DOCUMENT only.
6. Smart Chapter Detection Algorithm: Core Highlight
6.1 Multi-Pattern Regex System
def detect_chapters(paragraphs):
# Primary patterns: standard chapter markers
primary_patterns = [
r"^第[一二三四五六七八九十百千万零〇\d]+[章卷部篇]", # 第N章/卷/部/篇
r"^第\s*\d+\s*[章卷部篇]", # 第 N 章
r"^Chapter\s+\d+", # Chapter N
r"^CHAPTER\s+\d+", # CHAPTER N
]
# Secondary patterns: sub-sections/parts/volumes
secondary_patterns = [
r"^第[一二三四五六七八九十百千万零〇\d]+[话节集]",
r"^第\s*\d+\s*[话节集]",
r"^[Pp]art\s+\d+",
r"^[Ss]ection\s+\d+",
r"^[Vv]olume\s+\d+",
]
primary = [re.compile(p) for p in primary_patterns]
secondary = [re.compile(p) for p in secondary_patterns]
...
6.2 Two-Level Heading Fusion Strategy
raw_chapters = []
for i, para in enumerate(paragraphs):
stripped = para.strip()
if not stripped:
continue
matched = False
# Try primary first
for cp in primary:
if cp.match(stripped):
raw_chapters.append((i, stripped[:60], "primary"))
matched = True
break
# Fallback to secondary
if not matched:
for cp in secondary:
if cp.match(stripped):
raw_chapters.append((i, stripped[:60], "secondary"))
break
# Fusion logic: secondary within ≤3 paras of primary → merge, not separate chapter
if not raw_chapters:
if paragraphs:
return [(0, "全文 / Full Text")]
return []
chapters = [(raw_chapters[0][0], raw_chapters[0][1])]
for i in range(1, len(raw_chapters)):
pos, title, kind = raw_chapters[i]
prev_pos, prev_title, prev_kind = raw_chapters[i - 1]
gap = pos - prev_pos
# Secondary after primary with gap ≤3 → merge
if kind == "secondary" and prev_kind == "primary" and gap <= 3:
continue
chapters.append((pos, title))
return chapters
6.3 Algorithm Design Rationale
Paragraph Index: 0 1 2 3 4 5 6 7 8 9 10 11
Content: Ch1 text text Sec1 text Ch2 text Sec2.1 text Ch3 ...
Match Type: P1 - - P2 - P1 - P2 - P1
Gap: - - - 3 - 2 - 1 - -
Fusion Result: ✓Ch1 (merged) ✓Ch2 (merged) ✓Ch3
Core Rules:
-
Primary First:
Chapter N/第N章strong signals take priority -
Secondary Proximity Merge:
Section/第N节within 3 paras of primary → subordinate, not separate chapter -
Fallback: No matches → entire book as single chapter
[(0, "Full Text")]
6.4 Chapter Segment Retrieval
def get_chapter_paragraphs(paragraphs, chapter_index, chapters):
if not chapters:
return paragraphs
start = chapters[chapter_index][0]
if chapter_index + 1 < len(chapters):
end = chapters[chapter_index + 1][0]
return paragraphs[start:end]
else:
return paragraphs[start:] # Last chapter to end
7. Term Replacement: Personalized Pronunciation Correction
7.1 Term File Format
# 名词转换.txt example
原词<TAB>替换词
哈利波特 Harry Potter
伏地魔 Voldemort
霍格沃茨 Hogwarts
# Comment lines start with #
特殊空白 (空白符) # Replace with empty string
7.2 Parsing & Application
def parse_term_file(filepath):
term_map = {}
text = Path(filepath).read_text(encoding="utf-8")
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
parts = [p.strip() for p in line.split("\t")]
parts = [p for p in parts if p]
if len(parts) < 2:
continue
key = parts[0]
val = parts[-1]
if key == val:
continue
if val == "(空白符)": # Special marker: replace with empty
val = ""
term_map[key] = val
return term_map
def apply_term_map(text, term_map):
"""Longest-first replacement: 'Sophilia' before 'Sophi'"""
if not term_map or not text:
return text
for key in sorted(term_map, key=len, reverse=True):
text = text.replace(key, term_map[key])
return text
Use Cases:
- Proper noun pronunciation correction
- Abbreviation expansion:
CPU→central processing unit - Dialect/colloquial standardization
- Sensitive word masking (replace with empty)
8. Performance & Memory Optimization
| Optimization | Implementation | Effect |
|---|---|---|
| Streaming Read | PDF/EPUB page-by-page/item-by-item, no full DOM load | Peak memory < 50MB |
| Error Isolation | Single file parse failure doesn't block batch folder read | Robustness |
| Encoding Cache | Reuse detected encoding for multiple TXT in same dir | Faster re-open |
| Paragraph Filter |
filter_short_paragraphs(min_length=2) denoising |
Reduces useless synthesis |
9. Extending New Formats Guide
9.1 Steps to Add Format
# 1. Add suffix to SUPPORTED_SUFFIXES
SUPPORTED_SUFFIXES = {".txt", ".docx", ".doc", ".pdf", ".epub", ".html", ".htm", ".mobi"}
# 2. Implement parser
def read_mobi(filepath):
# Use mobi lib or calibre toolchain
...
# 3. Add dispatch branch in read_file
elif suffix == ".mobi":
return read_mobi(filepath)
9.2 Recommended Libraries
| Format | Library | Notes |
|---|---|---|
| MOBI |
mobi / kindle-unpack
|
Handle DRM restrictions |
| AZW3 | calibre |
Heavy dep, prefer EPUB conversion |
| FB2 |
lxml direct XML parse |
Simple structure |
| RTF | striprtf |
Plain text extraction |
10. Troubleshooting Common Issues
| Symptom | Cause | Fix |
|---|---|---|
| PDF no text/garbled | Scanned/encrypted/font embedding | OCR / decrypt / print to PDF |
| EPUB chapters missing | Non-standard markers (e.g., class="chapter") |
Edit EPUB or add custom patterns |
| TXT garbled | Encoding not in probe list | Specify encoding param or save as UTF-8 |
| DOCX tables lost | Only paragraphs extracted |
Extend to traverse doc.tables
|
| Chapter detection off | Regex doesn't match specific style | Add to primary/secondary_patterns
|
11. Conclusion
Ebook Voice Player's document parsing engine embodies pragmatic engineering balancing UX:
- Unified Abstraction lets upstream business ignore format differences
- Dual-Engine Fallback maximizes PDF parse availability
- Smart Chapter Detection covers CN/EN/JP/KR writing conventions, two-level fusion prevents fragmentation
- Term Replacement solves proper noun pronunciation pain points with personalization
- Error Isolation & Streaming ensures stability on large files/batch reads
This engine powers millions of document imports for tens of thousands of users in production.
Try It: Microsoft Store - Ebook Voice Player
Source Code:
reader.py/tts_app/reader.py
Top comments (0)