If your app needs a few minutes of narration now and then, a GPU is the wrong tool. Between two voice-overs it sits idle, and you still pay for it. We needed English and Spanish narration for demos and walkthroughs, so we chose Kyutai's Pocket TTS, a text-to-speech (TTS) model designed for an ordinary processor. Setup is pip plus the CPU-only flavour of PyTorch, with no graphics card and no GPU drivers when you generate audio.
English was fine from the first try. Spanish did not sound like Spain. Here are the decisions that fixed it, for anyone adapting a voice model to another accent. The listening samples, the full speed table and every command live in our write-up on fine-tuning Pocket TTS for Castilian Spanish.
Pick the model size before you pick the voice
Kyutai ships seven languages (English, Spanish, French, German, Italian, Portuguese, Dutch), and every one comes in two sizes: 6 layers and 24 layers. Layers are the stacked blocks of the network; more layers means more work per second of audio. The 24-layer models run about three times slower, and that shapes your pipeline more than anything else.
Speed here is measured against "real time": 2x real time means 10 seconds of audio take 5 seconds to generate. On our development box (a shared virtual machine on an 8-core Ryzen 7 5700X, three server processes with two threads apiece, six everyday sentences per model), the English 6-layer model hit 3.1x. The Spanish 24-layer base model reached 1.2x, and our fine-tune 1.05x. Each running copy of a 24-layer model holds about 2.1 GB in RAM. Above 1x works for a voice-over; a long audiobook on a CPU is another matter. The Pocket TTS CPU speed measurements list audio length and generation time per model.
Fine-tuning means taking a trained model and continuing its training on your own data. Ours keeps the network's shape and only changes the numbers inside it, so it runs at the speed of the base 24-layer model, and switching to it means pointing the server at another config:
pip install pocket-tts
pocket-tts serve --config hf://inferencekey/pocket-tts-spanish-24l-castilian/config.yaml
serve loads the model once and then answers over HTTP. We start a few of those processes and hand each one its own sentence.
The problem was the accent, not the words
The base Spanish model was perfectly intelligible. The accent gave it away: foreign-sounding, with seseo, where the Castilian "z" and soft "c" collapse into an "s" as in much of Latin America. In Spain that is obvious from the first word with a "z".
We started from spanish_24l (about 316 million parameters, the learned numbers, in the part that produces speech) and trained it further on roughly 2,000 recordings of Castilian speech, using a single AMD GPU with ROCm, AMD's counterpart to CUDA. The result is on Hugging Face as inferencekey/pocket-tts-spanish-24l-castilian.
The trap: training longer made it worse
Validation loss is the error the model makes on held-out clips it never trains on; lower is better. Ours hit its minimum at step 600 (0.4951) and had drifted up to 0.5095 by step 1500. By then the model had gone through the training set about 47 times, and its output started losing words and inventing others.
| Checkpoint | Validation loss | Test clips with complete sentences (ASR) |
|---|---|---|
| Step 600 | 0.4951 | 6 of 6 |
| Step 1500 | 0.5095 | 3 of 6 |
A checkpoint is the weights saved at one training step. Step 600 scored like the base model; step 1500 lost half. Grab the last checkpoint and you ship the broken one.
From step 600 we released the EMA weights instead of the raw ones. EMA (exponential moving average) blends the weights over recent steps, so it moves further from the base model more gently. That suited us: the base model already read Spanish well, and we only wanted to touch the accent.
What to measure, and what your metric cannot see
The natural automatic check is ASR (automatic speech recognition): transcribe every generated clip and compare it with the input text. We used Qwen3-ASR, and it does one job well, spotting words that vanished or appeared. It cannot hear the thing we were fixing. "Caza" and "casa" come out as the same transcript, so a model with the accent fix and one without it get identical scores.
So the evaluation has two halves, and neither replaces the other:
- ASR on every clip, to catch dropped or invented words.
- A person listening to every clip, to judge the accent.
The ASR pass paid off beyond Spanish. With Kyutai's own 6-layer French model, "sept heures et quart" was misread in two of three attempts, and the transcript flagged it. For any sentence you care about, transcribe it and regenerate when it fails.
Two input rules that change how it sounds
First, send complete sentences or short paragraphs. The model ends each chunk with a falling, final intonation, so a fragmented paragraph sounds stop-and-go. With a server pool, parallelize per sentence, never per fragment.
Second, spell numbers out. The model handles "siete y cuarto" every time; "7:15" sometimes trips it.
What it does not show
The speed figures come from one 8-core development VM and six sentences per model, not a broad benchmark. The accent was judged by ear, the only judge that works here and also the one that does not scale. And at 24 layers, this is a CPU narration tool, not an audiobook pipeline.
Credit where it belongs: Kyutai built Pocket TTS, its multilingual models and the voice cloning that gives us one voice (alba) in every language. Qwen3-ASR caught the missing words. Our part was the Castilian fine-tune, choosing the checkpoint and checking every sample.
To hear the base model next to the fine-tune and copy the generate command, read the full Pocket TTS Castilian Spanish article on inferencekey.com.
Top comments (0)