Multi-Voice Audiobooks: The Character Attribution Problem

elenaashford — Fri, 24 Jul 2026 19:02:22 +0000

Turning a novel into a multi-voice audiobook sounds like a TTS problem. It is mostly a parsing problem, and the parsing is harder than the synthesis.

Attribution is not solved by quotation marks

The naive approach: find text in quotes, that is dialogue, assign a voice. It breaks immediately on real prose.

"I told you," she said, "and you did not listen."

One utterance, split across two quoted spans, with the attribution wedged in the middle. Naive splitting produces two separate lines and often assigns them different speakers.

Then there is the harder case — no attribution at all:

"Are you coming?"
"In a minute."
"You said that an hour ago."

Three lines, zero speaker tags. A human tracks the alternation from context. A parser has to model turn-taking, and it has to know when the alternation breaks because a third character walked in.

What actually helps

Attribution verb lexicons beat regex. said / asked / whispered / muttered plus their inflections cover the majority of tagged dialogue. The pattern is usually "..." <verb> <entity> or <entity> <verb>, "..." — worth handling both orderings explicitly rather than hoping one regex catches everything.

Coreference is unavoidable. "she said" only resolves if you know who "she" is in this scene. Without a coref pass you get correct attribution to the wrong voice, which sounds worse than a monotone read because listeners notice the inconsistency.

Alternation as a prior, not a rule. In untagged runs, alternating between the last two active speakers is right most of the time. Treat it as a default that named attribution overrides — not as ground truth.

The consistency constraint nobody warns you about

Once a voice is assigned to a character, it must stay assigned. Chapter 14 cannot re-cast the protagonist because your attribution confidence dipped.

That means voice assignment is a document-level decision made after full-text analysis, not a streaming one made per chapter. Which in turn means you cannot start rendering audio until you have parsed the entire book — an architectural constraint that surprises people building this incrementally.

Practical shape that works:

Full-text parse -> character inventory with mention counts
Assign voices to the top N by frequency, narrator to everything else
Persist the mapping
Render chapters against the frozen mapping

I built this pipeline into VoxForge — epub or txt in, auto-cast characters, chaptered audiobook out.

Where it still fails

First-person narration where the narrator is also a character. Epistolary novels. Books that switch POV mid-chapter without a section break. All three break the character-inventory assumption, and I have not found a clean general solution — currently they need manual overrides.

DEV Community: elenaashford

Multi-Voice Audiobooks: The Character Attribution Problem

Attribution is not solved by quotation marks

What actually helps

The consistency constraint nobody warns you about

Where it still fails