I run a reading site for language learners. Every line of every story has audio, generated with OpenAI's gpt-4o-mini-tts. Russian is the hard case: stress is free, unmarked in normal writing, and it changes meaning (за́мок is a castle, замо́к a lock). A native Russian teacher told me the audio got the stress wrong often enough to be a problem.
The bug
My text already carried the stress. I add an acute accent (U+0301) after the stressed vowel for learners. But a "clean the text before speech" function stripped every combining mark before sending it to the voice. On top of that, the corpus wrote е where the word needs ё, as everyday Russian printing does. So the model guessed the stress on every single word, and it often guessed wrong on ё.
What I measured
I rendered 25 test items several ways and checked them with a phone-level recogniser (wav2vec2 xlsr-53 espeak):
- shipped audio: 14 of 21 checkable items right
- same voice, acute marks + ё restored + one sentence in the voice instructions saying what the marks mean: 18 of 21
- an Azure neural voice, marked or not: 11 to 14. The marks barely changed anything.
With the marks, ё words were right 4 times out of 4 rolls. True context homographs (до́ма vs дома́) stayed around 50% per roll even when marked.
Marking 39,465 texts
Stress for each word came from three sources, in this order: my own overrides, the glossary the site already shows learners, then RuAccent (an open-source accent model). Where they disagreed, or the tool was unsure, an LLM reviewer read the word in its sentence. About 12,000 tokens went through that, and roughly 8% of the tool's proposals were wrong, mostly case forms like окна́ / о́кна.
The render gate
The model is sampled, so the same input can come out differently. Each clip passes a length check, and single words also pass a stress check from the phone recogniser: vowel quality first, duration as a tie-break. A clip that fails gets re-rolled up to three times. For a stubborn word, the last take respells the unstressed о as а in the copy sent to the voice. That is akanye, the way Russian actually sounds, and it fixed за́мок.
One trap cost me a whole pilot: ffprobe reports "N/A" for the duration of an MP3 read from a pipe, so my length gate rejected every clip. Decode the audio to measure it, and test one clip before a batch.
Result
About 36,000 clips re-recorded for roughly $35 of API time. On my check set, wrong ё went from about 40% to about 2%, and other stress errors from 13% to under 1%. 82% of clips passed on the first take. 142 single words still failed after three takes; those go to a human ear.
What a machine still can't certify is a 3-second clip of one word. If you work on Russian TTS and have a better scorer, I'd love to hear about it.
The site: https://languagethroughliterature.com
Tom
Top comments (0)