The narrator voice for one of my Italian channels is a clone made with a local open-source model. It sounds good most of the time. Sometimes it opens a sentence with a syllable that belongs to no language, swallows the first word, or garbles one in the middle. A viewer notices that within seconds, so I built a checking step that every sentence has to pass before it goes into a video.
The script is split into sentences, and each sentence is generated as a separate take. Two speech recognizers of different sizes then transcribe the take, and I compare their output with the script word by word. A take is rejected for a missing word, an extra word or a word the recognizer is unsure about. A close match is not enough, because a sentence that is 93 percent right can still contain the one word that sounds wrong.
The take is then trimmed, and the trimmed audio goes through both recognizers again. It is accepted only when both return exactly the script, with no difference at all.
Two details came from real failures. Recognizers ignore short sounds that are not words, but a listener does not. So I measure the audio level in the pauses between sentences, and any isolated burst of sound gets cut. The second detail is numbers. A script that says "9 minutes" comes back from the recognizer as the digit 9, while the voice says "nove", so I convert digits to words on both sides before comparing.
Sometimes the model swallows the same word on every attempt. One sentence started with "Immergiti" and lost it in all six takes. I replaced the verb with "Scendi" and the next take passed. After eight failed attempts a sentence is split into two shorter ones and tried again.
The check makes production slower, because a short video with five sentences can need a dozen takes or more. I accept that cost. The voice is the first thing a viewer judges, and the computer time is cheap.
If you work with cloned voices, the part I would copy first is the second pass on the trimmed audio. Checking the raw take is easy to do, but the trimming step can introduce its own errors, and the final audio is the one that gets published.
Disclosure: drafted with a local AI model, then reviewed and approved by me. Originally published at https://bbal-blog.kairo-app.workers.dev/checking-a-cloned-voice-word-by-word/
Top comments (0)