Getting Clean Text Out of PDF, DOCX, and HTML

ivyjsu — Sat, 25 Jul 2026 08:10:28 +0000

Anything that turns a document into audio, a summary, or embeddings has the same unglamorous bottleneck: getting clean, correctly-ordered text out of the file. I underestimated this badly. Generation is a solved-ish problem you can buy. Extraction is where the engineering lives.

PDF has no concept of a paragraph, and this ruins everything

A PDF is a drawing program's output. The content stream says "place this glyph at these coordinates in this font." No paragraph, no reading order, no notion that this run is a footnote and that one a heading. Libraries handing you a page.get_text() string are guessing, usually "sort by y, then x."

That dies on the first two-column academic paper: line one of column A, then line one of column B, then line two of column A — interleaved garbage. Fed into anything narrative, it produces confident nonsense.

Detect columns geometrically before ordering anything. Project every text block's horizontal extent onto the x-axis and look for a sustained gap:

def detect_columns(blocks, page_width, min_gap_ratio=0.04):
    """blocks: [(x0, y0, x1, y1, text), ...]. Returns x-cut positions."""
    occupied = [False] * 100  # 1% buckets across the page
    for x0, _, x1, _, _ in blocks:
        lo, hi = int(x0 / page_width * 100), int(x1 / page_width * 100)
        for b in range(lo, min(hi, 99) + 1):
            occupied[b] = True

    cuts, start = [], None
    for i, filled in enumerate(occupied):
        if not filled and start is None:
            start = i
        elif filled and start is not None:
            if (i - start) >= min_gap_ratio * 100:
                cuts.append((start + i) / 2 / 100 * page_width)
            start = None
    return cuts

def reading_order(blocks, cuts):
    """Within each column top-to-bottom, then columns left-to-right."""
    col = lambda b: sum(1 for c in cuts if b[0] > c)
    return sorted(blocks, key=lambda b: (col(b), round(b[1], 1), b[0]))

Two caveats. Run this per page, not per document — papers are often single-column for the abstract and two-column afterward. And a whitespace gap can be a real column boundary or just a figure with text either side; require it to persist across most of the page height before trusting it.

Headers, footers, and the boilerplate you must delete

Every page carries running heads, page numbers, journal names, DOIs. Extracted naively, these splice mid-sentence into your text: "...the distribution is 14 Journal of Irreproducible Results, Vol. 3 skewed toward..." Once in, it's nearly impossible to remove downstream, and it will end up read aloud or embedded. The reliable detector is repetition across pages, not position:

def find_boilerplate(pages, band=0.10, min_ratio=0.6):
    """Text in the top/bottom band of >=60% of pages is chrome."""
    counts = Counter()
    for page in pages:
        h = page.height
        for _, y0, _, y1, text in page.blocks:
            if (y1 < h * band or y0 > h * (1 - band)) and len(text) < 120:
                counts[normalize(text)] += 1   # digits -> '#', casefold
    return {t for t, c in counts.items()
            if c >= max(2, int(len(pages) * min_ratio))}

Normalizing digits to a placeholder before counting makes page numbers collapse into one repeated pattern instead of looking unique on every page. Guard it behind a page-count minimum — on a two-page document, "appears on 60% of pages" means "appears twice," which catches real content.

Footnotes need similar treatment for a different reason: legitimate content, but splicing them mid-sentence into the body is wrong. Detect them by font size (typically 1–3 pt smaller than body) plus lower-region position, then pull them into a separate stream. Dropping or appending them is a product decision; extraction's job is only to not interleave them.

Tables deserve the same honesty. For ruled tables you can recover cells from the drawing operations, intersecting line segments into a grid. For borderless tables — most of them — you're inferring structure from whitespace alone, and reliability collapses with merged cells, wrapped text, and nested headers. Don't solve this generically: classify the region, attempt structured extraction, and if the result fails a sanity check (consistent column count, plausible cell density), fall back to a natural-language summary. "Table 2 compares four models across three benchmarks" beats a soup of misaligned numbers.

DOCX and HTML: different problems, both underestimated

DOCX is a zip of XML and far more tractable — paragraph structure is explicit. The traps are elsewhere: text lives in w:t runs that split mid-word at formatting boundaries (<w:t>hyper</w:t><w:t>link</w:t> must be joined before tokenizing), tracked changes leave deleted text in w:del elements you must exclude, and text boxes and headers sit outside the main flow.

HTML from a URL is a boilerplate-stripping problem: nav, cookie banners, related-articles, comments, modals. The classic Readability-style approach scores DOM nodes on text density — text length against link density, weighted by tag semantics — then takes the highest-scoring subtree. Good default; fails predictably when the article is fragmented across many small containers.

The cheap win most people skip: check application/ld+json first. A large share of publishers emit an Article schema with articleBody populated, and when it's there it's cleaner than anything heuristics will give you.

TYPES = ("Article", "NewsArticle", "BlogPosting")

def extract_article(html):
    for block in find_jsonld_blocks(html):
        for node in flatten_graph(json.loads(block)):
            if node.get("@type") in TYPES and (b := node.get("articleBody")):
                return b
    return readability_extract(html)   # fall back to density scoring

Whatever the source, you land in the same normalization pass: de-hyphenate line-broken words (but keep genuine hyphenates — check whether the joined form is more plausible than the split), collapse the ligatures PDF fonts love (ﬁ, ﬂ), normalize Unicode quotes and dashes, and rebuild paragraph boundaries from line spacing rather than trusting newlines.

Building the ingestion side of DuoCast, which takes PDF/DOCX/URL input, what surprised me was how much of the perceived output quality traced back to this layer. A model handed interleaved two-column text produces plausible-sounding wrong content — and nothing in the output reveals extraction as the culprit. Which is why this stage should emit diagnostics: column count, boilerplate removed, footnote count, unresolved regions. When quality drops, that's the only way to know where to look.

Making Two TTS Voices Sound Like an Actual Conversation

ivyjsu — Fri, 24 Jul 2026 19:20:55 +0000

Rendering a document as a two-host podcast is not just alternating two TTS voices line by line. Do that and you get something that sounds like two people reading unrelated scripts in the same room.

The problem is prosody, not voices

Human conversation carries information in timing. A reply that arrives instantly reads as agreement. A short gap reads as consideration. Speakers overlap slightly at turn boundaries. Pitch tends to fall at the end of a statement and rise before a handoff.

Concatenated TTS has none of this. Each utterance is synthesised in isolation with neutral prosody and identical gaps, and the result sits in an uncanny valley — clearly speech, clearly not conversation.

Things that measurably help

Vary inter-turn gaps by turn type. Not one fixed pause:

Agreement or continuation: short, 150-250ms
Topic shift: longer, 400-600ms
Answer to a question: short, the responder was already primed
After a complex explanation: longer, it reads as processing time

A lookup keyed on turn type gets you most of the way, and it is cheap to implement.

Write handoffs into the script. The transition matters more than the voice. "Right — so what happens when the file is large?" carries turn-taking structure that "The file size limit is 10MB" does not. This is a scripting problem before it is an audio problem.

Do not centre both voices. A few degrees of stereo separation makes two speakers legible to the ear. Hard-panning is disorienting on headphones; subtle offset is not consciously noticed but does the work.

Chunking is where quality actually dies

Most TTS APIs have input length limits, so long documents get split. Naive splitting at the limit cuts mid-sentence, and the resulting prosody discontinuity is jarring — the pitch contour restarts mid-thought.

Split on semantic boundaries — paragraph, then sentence — and never mid-clause. If a single sentence exceeds the limit, split at a comma and accept a small artefact rather than a mid-word cut.

Multilingual complicates the pause table

Turn-taking norms are not universal. Japanese conversation tolerates longer pauses than English; treating a 600ms gap as "awkward" is an English-specific assumption. If you support multiple languages, the pause table needs to be per-language, not global.

I build this into DuoCast — paste text, a PDF, or a URL, get a two-host podcast with the conversational layer handled.

Caveat

None of this makes synthetic conversation indistinguishable from a real recording. It moves it from "obviously robotic" to "pleasant enough to listen to on a commute," which is a lower but more achievable bar.

DEV Community: ivyjsu