Most open-weight TTS models treat pitch like seasoning. Stress a syllable here, drop your voice at the end of a question, and English listeners still understand you.
Yorùbá does not work like that. Pitch is part of the spelling. Take the letters o-k-o (with underdots ọkọ):
- ọkọ (mid-mid) means husband
- ọkọ̀ (mid-low) means car or vehicle
- ọ̀kọ̀ (low-low) means spear
If an English-trained model guesses a rising or falling melody on ọkọ, it says the wrong word.
Over the past few weeks, I adapted the Kokoro / StyleTTS2 architecture to speak Yorùbá using a single Google Colab NVIDIA L4 GPU (22 GB VRAM) and 2.70 hours of cleaned audio. A lot of things broke along the way: dropped tones, raspy noise, speaker mix-ups, PyTorch 2.6 crashes, flat robotic speech, and even a 402 Payment Required surprise when I tried to deploy a free demo.
This is the exact path I took, including the mistakes I made and where v1 still falls short.
Building oro-g2p and hunting for unused token slots
Before touching any neural network weights, I needed a phonemizer that actually understood Yorùbá orthography. English phonemizers fail immediately here because gb is a single sound (/ɡb/), p is pronounced /kp/ (kp), and consonants like n or m nasalize vowels that follow them. Syllabic nasals (m, n, ŋ) also shift based on the next consonant, including stacked cases like nǹkan.
I did not want people to install heavy system packages like espeak-ng, so I wrote oro-g2p, a zero-dependency Python library (pip install oro-g2p):
from oro_g2p import g2p
print(g2p("Ẹ kú àárọ̀"))
# 'ɛˉ kuˊ aˋaˊɾɔˋ'
Getting the text into IPA with tone marks (ˊ High, ˉ Mid, ˋ Low) was the easy part. Feeding it into Kokoro was where I got stuck.
My first attempt emitted the vowel and its tone as two separate tokens, like ["a", "ˋ"]. That sounded terrible in StyleTTS2. The duration predictor treated the tone mark like an extra letter, stretching trailing tones at the end of phrases or losing the link between a vowel and the pitch it was supposed to hold.
I did not want to resize Kokoro's embedding table (n_token = 178) because doing that breaks direct weight loading from pretrained checkpoints. So I printed out BASE_KOKORO_VOCAB to see what was actually inside those 178 IDs (0 to 177).
Slot 0 is reserved (PAD/BOS/EOS), and 114 slots are already taken by Kokoro's base tokens. That left 63 completely empty slots sitting unused inside 1..177.
So I stopped sending vowels and tones separately. I packed every Yorùbá (vowel + tone) pair, (syllabic_nasal + tone) pair, and digraph directly into those 63 spare slots as single atomic IDs ("kp": 122, "ɡb": 124, "dʒ": 82, "aˊ": 7, "aˋ": 26, "ũˋ": 105, etc.), matched with a greedy longest-first tokenizer. Because n_token stayed 178, all pretrained tensor shapes lined up without touching the model architecture.
Half of my "model bugs" were actually data bugs
Early training runs sounded awful, and I spent too long blaming hyperparameters before inspecting my dataset.
The biggest culprit was StyleTTS2's default dataloader crop. By default, it crops long audio down to max_len=120 (around 1.44 seconds) while still feeding the full transcript. A lot of my source clips were 6 to 11 seconds long. I was basically asking the model to listen to 1.4 seconds of audio while guessing how it aligned to a 10-second sentence. Once I trimmed leading and trailing silence (top_db=35) and filtered my training set down to clips between 1.2s and 5.8s, alignment snapped into place.
The second mess came from speaker IDs. StyleTTS2 grabs another random clip from the same speaker_id during training to estimate pitch baseline and voice style. Early on, I had multiple voices lumped under speaker_id = 0, which meant the pitch predictor was getting conflicting references on every batch. Separating speaker IDs strictly cleared up a lot of random pitch jumps.
I also learned a strange quirk about iSTFTNet the hard way at inference time: if you try to strip predicted silence frames before passing features into the decoder, InstanceNorm1d calibration gets thrown off and the audio comes out covered in raspy static. Waveform silence trimming only works after the decoder produces audio.
After cleaning, my final Gold training split came out to 2.70 hours (2,168 clips) drawn from AfriSpeech/open-bible-speech-african (CC BY-SA 4.0) and google/WaxalNLP (CC-BY / CC-BY-SA-4.0).
Patching StyleTTS2 for PyTorch 2.6+ and Colab
Trying to run Stage 2 training on Google Colab with modern PyTorch 2.6+ threw a handful of upstream crashes unrelated to Yorùbá:
-
models.pyfailed to load checkpoints withoutweights_only=False, choked onmodule.DataParallel key prefixes, and threwspectral_normerrors if.eval()ran onstyle_encoderorpredictor_encoderbefore a 1-step.train()warmup. -
Modules/slmadv.pyhad 17.squeeze()calls (wav.detach().squeeze(),y_pred.squeeze(),y_rec_gt.detach().squeeze(),y_rec_gt_pred.detach().squeeze(),real_GP.detach().squeeze(), etc.) that dropped the batch dimension completely ifbatch_size=1, crashing tensor shapes. I replaced those with a_wav2d(x)helper (x.reshape(x.shape[0], -1)) and added 1D/3D shape guards inModules/istftnet.py. -
train_second.pycalledpdb.set_trace()on NaN/Inf batches. If a batch blew up overnight on Colab, the script did not skip it or crash loudly, it just sat waiting for keyboard input while burning GPU units. I disabledset_detect_anomaly(True), wrappedy_rec_gt_predintorch.no_grad(), addedclip_grad_norm_(..., 5.0)onbert,predictor, anddiffusion, and skipped NaN batches safely.
I pushed these fixes upstream in semidark/StyleTTS2#4 (+47 / -19 across 4 files) and left a breakdown in semidark/kikiri-tts#46.
Why Round 1 sounded like a robot
My Stage 2 runs on the L4 (batch_size=4) proved just how much diffusion and style loss weights matter for tonal speech:
-
Round 1: I accidentally left
lambda_diff=0.0andlambda_sty=0.0. MyF0 lossstalled at2.245and the voice sounded completely flat. -
Round 2: Turned both back on (
lambda_diff=1.0,lambda_sty=1.0,lambda_F0=3.0,bert_lr=5e-5). Validation loss dropped to0.313andF0 lossfell to1.921. -
Round 3: Ran 15 epochs with
lambda_F0=4.5,lambda_sty=1.5,lambda_diff=1.0, andbert_lr=4e-5.F0 lossreached a low of1.758along the way, finishing Epoch 15 (epoch_2nd_00014.pth) atValidation loss: 0.305, Dur loss: 0.974, F0 loss: 1.823.
Shrinking the checkpoint, stopping vocal tremor, and deploying for $0
epoch_2nd_00014.pth was 2,091.5 MB on disk because it carried training discriminators and optimizer state. Filtering down strictly to INFERENCE_MODULES compressed the release checkpoint (kokoro_yoruba_gold.pth) to 526.0 MB.
During listening tests, I still noticed two annoying glitches:
-
Vocal shaking on multi-tone vowels like láàárín. Locking diffusion parameters to
alpha=0.0,beta=0.55,embedding_scale=1.15,diffusion_steps=15, andspeed=1.0stopped the tremor. -
Rushing on long text. Passing 400+ character paragraphs all at once made the model speed up unnaturally. Splitting text by punctuation (
0.18spause on commas,0.35spause on periods, plus appending a trailing.if missing) fixed the pacing.
In Python, running inference looks like this:
from tts_yoruba import YorubaTTS
tts = YorubaTTS(styletts2_dir="./StyleTTS2")
tts.speak(
"Ẹ kú àárọ̀ o! Báwo ni àwọn ará ilé?",
voice="spk8",
out_path="greeting.wav",
)
When I tried to deploy a live Gradio Space on Hugging Face's free cpu-basic tier, it threw 402 Payment Required. Since I did not want to pay for idle GPU hosting or keep people from testing it, I split the public demo into two pieces: a 24/7 static HTML audio showcase on Hugging Face Spaces so anyone can hear real v1 outputs right away, paired with a 1-click Google Colab Gradio notebook (which I redesigned after checking live browser screenshots so all 9 preset phrases fit without scrolling or clipped sliders).
Where v1 still breaks
If you try this model today, keep its v1 limits in mind:
-
Small, lopsided training data (2.70h total):
Speaker 8(BibleTTS) makes up ~92% of the audio and matches the reference voice closely (~98% in my A/B listening checks).Speaker 2(Waxal) is only ~1% of the data, so that voice is noticeably weaker with slight static. -
English-pretrained
PLBERT: Because the text encoder never saw native Yorùbá during pretraining, rare 4-to-6 syllable compound words (like Ṣàwòròidẹ or ògbóǹtarìgì) can lose tone accuracy. - Full diacritics required: There is no auto-diacritizer in v1 yet. Plain ASCII Yorùbá without tone marks defaults to mid-tones.
-
Pure Yorùbá only: Raw English words get read with Yorùbá spelling rules (for example,
"phone"mapspto/kp/askphoˉneˉ).
How I split the repos
Rather than leaving everything jammed inside one repo, I broke the project into three standalone parts:
-
oro-g2p(MIT) — Just the pure-Python Yorùbá G2P and Kokoro atomic vocab (pip install oro-g2p). -
kokoro-tonal-kit(MIT) — The starter kit (patch_styletts2.py,vocab_template.py,export_release.py, Stage 2 config notes) in case you want to adapt Kokoro / StyleTTS2 for another tonal language like Igbo, Twi, Hausa, or Vietnamese. -
kokoro-yoruba— The Yorùbá inference wrapper (tts_yoruba.py) and 1-click Colab Gradio demo wired up to theCC BY-SA 4.0weights.
Links
- Audio Showcase (HF Space): huggingface.co/spaces/Akintunde102/kokoro-yoruba-demo
- Live Gradio Demo (Colab): Open in Google Colab
-
Model Weights (
CC BY-SA 4.0): huggingface.co/Akintunde102/kokoro-yoruba-v1 - Yorùbá TTS Repo: github.com/Akintunde102/kokoro-yoruba
- Tonal Starter Kit: github.com/Akintunde102/kokoro-tonal-kit
- Yorùbá G2P: github.com/Akintunde102/oro-g2p | PyPI
- Upstream StyleTTS2 PR: semidark/StyleTTS2#4
For v2, I want to scale past 2.70h toward 15h+ of balanced multi-speaker audio, train a native Yorùbá PLBERT so long compounds hold their contour tones, and add auto-diacritization plus English code-switching before G2P runs. If you are building TTS for a tonal or low-resource language and hit similar walls, drop a comment below.
Top comments (0)