Stitching Hundreds of TTS Segments Into an Audiobook Without Seams

elenaashford — Sat, 25 Jul 2026 08:08:36 +0000

Character attribution gets all the attention in multi-voice audiobook work, and fair enough — it's the interesting ML problem. But what makes a generated audiobook feel amateur is almost never the voice casting. It's the seams. A click between paragraphs. A chapter that starts mid-copyright-notice. Volume that jumps when a different speaker takes over. Losing your place because the app re-generated and every segment boundary shifted.

That's assembly engineering. It's unglamorous, and it's where perceived quality actually lives.

EPUB structure lies to you

An EPUB is a zip with an OPF manifest, a spine giving reading order, and usually a nav document (nav.xhtml in EPUB 3, toc.ncx in EPUB 2). Naively, spine items are chapters. They are not.

Real-world breakage, all of which I've hit: one spine item containing five chapters (common in older conversions where the whole book is book.xhtml); one chapter split across four spine items because the converter chunked on file size; spine entries that are cover images, blank pages, or ads for the publisher's other titles; and nav docs pointing at fragment IDs (chapter3.xhtml#ch3), so chapter boundaries live inside files rather than at file boundaries.

So chapter detection is a merge of three signals, in priority order: nav document entries (including fragments), heading structure in the content, and spine order as fallback.

def detect_chapters(epub):
    nav = parse_nav(epub)                    # strongest signal
    if nav and len(nav.entries) >= 3:
        return split_at_anchors(epub, nav.entries)

    # fall back to headings across the flattened spine
    doc = flatten_spine(epub)
    heads = [h for h in doc.headings if h.level <= 2]
    if plausible_chapter_rhythm(heads):
        return split_at(doc, heads)

    return [Chapter(from_spine=item) for item in epub.spine]

plausible_chapter_rhythm is a sanity check worth writing: if "chapters" average 200 words, you split on scene-break headings. If one is 60% of the book, you missed the real boundaries. Reject and fall through.

Front and back matter need explicit handling because nobody wants to listen to a copyright page. Classify by position plus title matching plus length: leading items titled cover/title/copyright/dedication, trailing items titled acknowledgments/about the author/index. Don't silently delete them — mark them skippable and default them off. Some listeners want the author's note. Nobody wants ISBN digits read aloud.

The other thing that must not reach TTS: stranded page numbers, running headers repeated at every page break, and footnote markers. The result was clear.7 becomes "the result was clear seven," and it sounds broken every time.

Stitching without seams

You will generate hundreds of segments — sentence or paragraph level, sometimes across different voices. Concatenating them is where three specific artifacts come from.

1. Sample rate mismatch. Different voices, or the same engine on a different day, can hand back different sample rates. Concatenating 24kHz and 22.05kHz PCM without resampling produces pitch-shifted, chipmunked segments. Normalize everything on ingest — decode to float32, resample to one working rate, consistent channel count — and only encode at the very end.

2. Clicks at boundaries. A click is a discontinuity: segment A ends at amplitude 0.3, segment B starts at -0.2, and that instantaneous jump is broadband noise your ear hears as a tick. Fix by trimming to zero-crossings and applying a short fade:

FADE_MS = 8   # inaudible as a fade, sufficient to kill the click

def join(a, b, sr):
    a = trim_trailing_silence(a, threshold_db=-45)
    b = trim_leading_silence(b, threshold_db=-45)
    a = apply_fade_out(a, ms=FADE_MS, sr=sr)
    b = apply_fade_in(b,  ms=FADE_MS, sr=sr)
    return concat(a, b)

Trim the engine's own leading and trailing silence first — it's inconsistent between segments, and leaving it in produces erratic gaps that read as hesitation. Then re-insert silence deliberately.

3. Wrong pacing. Silence is punctuation. Fixed gaps everywhere sound robotic; no gaps sound breathless. What has worked for me:

within a paragraph, sentence to sentence   ~250-400 ms
paragraph to paragraph                     ~600-900 ms
scene break (blank line / *** )            ~1200-1500 ms
speaker change in dialogue                 ~150-300 ms  (tighter!)
chapter boundary                           ~1500-2000 ms + optional title read

Speaker changes being tighter than paragraph breaks is the counterintuitive one. Real dialogue overlaps and interrupts; a long pause between two characters trading lines destroys the sense of exchange. Multi-voice with generous gaps sounds like two people reading in separate rooms — because that's literally what it is.

Loudness normalization is not peak normalization. Different voices have genuinely different perceived loudness at the same peak amplitude, and peak-normalizing each segment makes it worse. Use an integrated loudness measure (EBU R128 / LUFS), and measure per voice across a whole chapter rather than per segment — per-segment normalization flattens intentional dynamics and makes quiet lines shout. Around −18 LUFS integrated with true peak under about −1.5 dBTP is a reasonable starting point for speech, but audiobook distributors publish their own required windows, so check before mastering.

Getting this stack right end-to-end — EPUB parse, front-matter classification, per-voice loudness, silence grammar — is most of the work behind VoxForge; the character casting was genuinely the easier half.

Playback position that survives regeneration

Last one, and it's the one that gets discovered too late to fix cheaply.

If your playback position is a byte offset or an absolute timestamp into a concatenated file, it is invalid the moment anything upstream changes. Re-cast one character, fix one mispronunciation, swap a TTS model — every downstream offset shifts and your listener is thrown to the wrong place in a nine-hour file.

Anchor position to the text, not the audio:

{
  "chapter_id": "ch07",
  "segment_id": "ch07-p012-s03",
  "offset_ms": 1420,
  "text_hash": "9f2ab1c4"
}

Segment IDs derive from document structure, so they're stable across regeneration. On resume: look up the segment, seek to its current start, add the offset. If text_hash doesn't match, the text itself changed — seek to segment start and accept losing a couple of seconds rather than landing somewhere arbitrary.

This same index is what makes chapter navigation, sentence-level scrubbing, and read-along highlighting possible later. Build it during assembly. Reconstructing text↔audio alignment afterward means forced alignment over hours of audio, and you'll wish you had written the offsets down while you had them.

Multi-Voice Audiobooks: The Character Attribution Problem

elenaashford — Fri, 24 Jul 2026 19:02:22 +0000

Turning a novel into a multi-voice audiobook sounds like a TTS problem. It is mostly a parsing problem, and the parsing is harder than the synthesis.

Attribution is not solved by quotation marks

The naive approach: find text in quotes, that is dialogue, assign a voice. It breaks immediately on real prose.

"I told you," she said, "and you did not listen."

One utterance, split across two quoted spans, with the attribution wedged in the middle. Naive splitting produces two separate lines and often assigns them different speakers.

Then there is the harder case — no attribution at all:

"Are you coming?"
"In a minute."
"You said that an hour ago."

Three lines, zero speaker tags. A human tracks the alternation from context. A parser has to model turn-taking, and it has to know when the alternation breaks because a third character walked in.

What actually helps

Attribution verb lexicons beat regex. said / asked / whispered / muttered plus their inflections cover the majority of tagged dialogue. The pattern is usually "..." <verb> <entity> or <entity> <verb>, "..." — worth handling both orderings explicitly rather than hoping one regex catches everything.

Coreference is unavoidable. "she said" only resolves if you know who "she" is in this scene. Without a coref pass you get correct attribution to the wrong voice, which sounds worse than a monotone read because listeners notice the inconsistency.

Alternation as a prior, not a rule. In untagged runs, alternating between the last two active speakers is right most of the time. Treat it as a default that named attribution overrides — not as ground truth.

The consistency constraint nobody warns you about

Once a voice is assigned to a character, it must stay assigned. Chapter 14 cannot re-cast the protagonist because your attribution confidence dipped.

That means voice assignment is a document-level decision made after full-text analysis, not a streaming one made per chapter. Which in turn means you cannot start rendering audio until you have parsed the entire book — an architectural constraint that surprises people building this incrementally.

Practical shape that works:

Full-text parse -> character inventory with mention counts
Assign voices to the top N by frequency, narrator to everything else
Persist the mapping
Render chapters against the frozen mapping

I built this pipeline into VoxForge — epub or txt in, auto-cast characters, chaptered audiobook out.

Where it still fails

First-person narration where the narrator is also a character. Epistolary novels. Books that switch POV mid-chapter without a section break. All three break the character-inventory assumption, and I have not found a clean general solution — currently they need manual overrides.

DEV Community: elenaashford