Catching wrong-voice lines in AI-generated video with 60 lines of Python

Every few clips, our video model gives a line to the wrong character. The uncle speaks with the investor's voice, or a narrator turns up mid-scene. Watching every clip doesn't scale once you publish every day, so we built a check that does it for us. It's about 60 lines of Python.

The setup: we run two animated series, Hummingtree, a nursery rhyme channel, and a 3D comedy cast on Instagram with three recurring characters: Ethan, Uncle Rick and Chad, an investor. Clips come out of an image-to-video model that renders dialogue and lip-sync together. Faces stay on model. Voices don't always.

Here's the pipeline that catches the wrong-voice lines and swaps them out before the edit.

The idea

1. Keep one clean reference recording per character (10 to 20 seconds of that character talking).
2. Transcribe each generated clip with word timestamps, so you know when each line starts and ends.
3. Embed every line and every reference with a speaker encoder, then compare with cosine similarity.
4. If a line is closer to another character's reference than to the expected speaker's, flag it.
5. Mute the flagged span and drop in a clean TTS line, loudness matched to the clip.

Nothing here is model-specific. It works on any clip with speech in it.

Step 1: references

# pip install resemblyzer faster-whisper soundfile numpy
from resemblyzer import VoiceEncoder, preprocess_wav
import numpy as np

enc = VoiceEncoder()
REFS = {
    "ethan": enc.embed_utterance(preprocess_wav("refs/ethan.wav")),
    "rick": enc.embed_utterance(preprocess_wav("refs/rick.wav")),
    "chad": enc.embed_utterance(preprocess_wav("refs/chad.wav")),
}

embed_utterance returns a 256-dimension vector that is already L2-normalised, so cosine similarity is just a dot product.

Step 2: find the lines

from faster_whisper import WhisperModel
import soundfile as sf

asr = WhisperModel("small", compute_type="int8")

def lines(clip_wav):
    segs, _ = asr.transcribe(clip_wav, word_timestamps=True, language="en")
    return [(s.start, s.end, s.text.strip()) for s in segs]

We convert every clip to 16 kHz mono first (ffmpeg -i clip.mp4 -ac 1 -ar 16000 clip.wav). Whisper and the encoder both expect that.

Step 3: score each line

def who_said_it(clip_wav, start, end):
    audio, sr = sf.read(clip_wav)
    span = audio[int(start * sr):int(end * sr)]
    if len(span) < sr * 0.6:          # too short to judge reliably
        return None, {}
    emb = enc.embed_utterance(preprocess_wav(span, source_sr=sr))
    scores = {name: float(np.dot(emb, ref)) for name, ref in REFS.items()}
    return max(scores, key=scores.get), scores

def audit(clip_wav, expected):
    """expected: list of character names, one per line in script order"""
    report = []
    for (start, end, text), want in zip(lines(clip_wav), expected):
        got, scores = who_said_it(clip_wav, start, end)
        ok = got is None or got == want
        report.append({"t": (round(start, 2), round(end, 2)), "text": text,
                       "want": want, "got": got, "ok": ok,
                       "margin": round(scores.get(want, 0) - max(scores.values(), default=0), 3)})
    return report

Two things we learned the hard way:

• Skip spans under about 0.6 seconds. A one-word "Whoa!" doesn't carry enough voice for a stable embedding, so you get noise.
• Look at the margin, not only at the winner. If the expected speaker loses by 0.01, it's usually a voice that moved slightly between clips, which is fine. If it loses by 0.1 or more, someone else is talking.

Step 4: the second failure mode, invented lines

The same check finds a quieter bug. If you ask for an 8-second clip and the script line takes 3 seconds, the model fills the rest. Sometimes that's a smile or a nod. Sometimes it's a line nobody wrote: a mumble, a repeated name, or an extra "Uncle Rick!" at the start of a scene.

The fix is simple. Whisper returns more segments than the script has lines, so anything outside the scripted spans gets cut in the edit or muted:

def extra_speech(report_lines, scripted_count):
    return report_lines[scripted_count:]   # everything past the script is invented

It's even better to stop it happening: keep each generated clip close to the length of its dialogue, and write a visual beat into the prompt for any silence ("he grins and raises his glass, no dialogue").

Step 5: patch the line

For a flagged line, we synthesise the correct voice and splice it in at the same loudness as the surrounding clip:

import subprocess

def patch(clip_mp4, out_mp4, start, end, tts_wav):
    # zero the bad span, then mix the new line in at `start`
    filt = (
        f"[0:a]volume=enable='between(t,{start},{end})':volume=0[a0];"
        f"[1:a]adelay={int(start*1000)}|{int(start*1000)}[a1];"
        f"[a0][a1]amix=inputs=2:normalize=0:duration=first[a]"
    )
    subprocess.run(["ffmpeg", "-y", "-i", clip_mp4, "-i", tts_wav,
                    "-filter_complex", filt, "-map", "0:v", "-map", "[a]",
                    "-c:v", "copy", "-c:a", "aac", out_mp4], check=True)

Before mixing, scale the TTS file so its RMS matches the clip's speech. Otherwise the patched line jumps out. Lip-sync on a replaced line is never perfect, so we prefer to replace lines where the character is facing away, or a reaction shot is on screen.

Results

On our last two comedy episodes the audit flagged wrong voices in 5 of 14 clips. In one clip every line came out in the wrong voice. In the others it was a single line, including the closing call to action. All five were fixed in the edit without regenerating any video. Regenerating a clip takes minutes of GPU time and often introduces a new problem. A 2-second audio patch costs nothing.

Where this sits in the pipeline

Voice audit is one of four automatic checks we run on every clip before a person watches it:

1. Frame sheet: 12 evenly spaced frames on one image, to catch warped hands, extra characters or the camera wandering off.
2. Transcript: word timestamps against the script, to find missing and invented lines.
3. Voice match: the check above.
4. Layout check: every caption and overlay drawn on the real frames, so no text box ever sits on a face.

Each check takes a few seconds. Together they're why a daily schedule is possible at all.

If you'd like this kind of series without building the pipeline yourself, it's what our AI animation studio does: every episode as a YouTube video, a Short and a Reel, with music, voices, retakes, covers and captions included.