Teaching a 35B voice model to speak Turkish on one 16 GB card

Kasparov Abi — Wed, 21 Oct 2026 19:05:59 +0000

Update, 22 October 2026. I spent a day on speed with the same weights. Short sentences now run at 2.7x real time in disk mode (0.86x before) and paragraphs at about 3.3x, and Freya-TR-Eval WER is 2.27% (2.78% before, most of that gap is sampling noise). Four changes mattered. The per-frame step is compiled with torch.compile before it is captured in a CUDA graph, logits are computed only for the 4,097 audio tokens instead of the 252k vocabulary, the 6,000 most used experts stay in pinned RAM, and each expert loads with one copy instead of five, which turned out to be the real limit while reading the prompt. FP8 for the dense layers was 8% faster but cost a point of WER, so it stays off. Details and numbers are in the model card and the repository README.

Fish Audio's Drama 3 is a 35 billion parameter mixture of experts model that turns text into speech and follows stage directions written in square brackets. You can write [whispering] or [angry] in front of a sentence and it acts the line. It also needs about 41.5 GB just for its weights, which rules out every consumer graphics card on the market.

I wanted it in my desk robot, in Turkish, on an RTX 5070 Ti with 16 GB. This post describes what worked, what I measured, and which parts other people had already published before me.

A 30 second demo with the samples is at huggingface.co/kasparovabi/drama-3-tr-speaker.

Reading and speaking are different jobs

A text to speech model like Drama 3 does two things. First it reads the prompt (the system text, a reference recording of the voice and the sentence to say). Then it produces audio frames one by one, about 21.5 per second.

I counted which experts each phase uses on Turkish speech. Reading touches almost all 256 experts in every layer. Speaking concentrates on a much smaller set. So I kept the original model for reading and trained a separate speaker that holds only 32 experts per layer, 12.5% of the original, for every audio frame. The two meet through the attention cache and the DeltaNet recurrent states, so the speaker continues exactly where the reader stopped.

Later I found that SlimWise (arXiv 2609.34117) had published the same split for language models a few weeks earlier. The idea is theirs as well, I arrived at it separately for speech.

Training the speaker on its own mistakes

Each layer of the speaker is trained on the hidden states produced by the speaker layers below it, which have already drifted a little from the original model, while the target is the original model's next hidden state on the clean path. In my ablation this beat plain output matching at every size (KL 0.109 against 0.153 at 32 experts). GPTAQ (ICML 2025) uses the same asymmetric objective for quantization.

Training took about 85 minutes on 3,800 utterances from the ISSAI Turkish Speech Corpus.

Streaming experts from the drive

Reading still needs the full model. Instead of keeping 41.5 GB in RAM, each 3 MB expert is read on demand from the NVMe drive with unbuffered I/O into pinned memory. The process then needs about 4.4 GB of system RAM while generating, with a peak of 9.3 GB in the first seconds of loading. Offloading experts to disk is an established technique and I claim nothing new here. The cost is speed on short sentences, the numbers are below.

Results

On Freya-TR-Eval, the public Turkish benchmark from Freya, with their own scoring script, the system reaches 2.68% WER and 0.70% CER over 495 sentences with the experts in RAM, and 2.78% and 0.67% with the experts on disk and direction guidance on. FreyaTTS is a 183M model built for running on device, so this is not a like for like comparison. What it shows is that the small speaker keeps Drama 3's Turkish intelligible.

With the experts in RAM the system runs at 1.2 times real time on short sentences and 1.7 to 1.9 times on paragraphs, and with streaming codec decode the first audio arrives after about 1.15 s. Disk mode is slower on short sentences (0.86 times real time with guidance on), because reading one sentence pulls about 8 GB of experts from the drive. That traffic is the part I most want to cut next.

The finding I did not expect

I also tried shrinking the reader. A reader with 16 experts per layer on text kept the words clear but the voice went flat, and directions like [whispering] stopped working. Shrinking only the speaker did not cause this. It is a single experiment checked by listening, so I treat it as a lead. In this model, emotion and style seem to get written into the cache while the model reads, and the frames that follow mostly carry them out.

The bug I only heard after publishing

A day after the release I listened to the samples again and the second one sounded like a different person. My first suspect was the direction guidance at strength 4, which pushes the brackets harder. To test it I generated the same sentences at several strengths and scored each one against the reference voice with a speaker embedding (CAM++). Similarity fell to between 0.02 and 0.49, far too low to blame on guidance.

The cause was in my engine. When a single process switched from unguided to guided generation, the attention caches were reallocated for a batch of two, which wiped the cached reference recording, while my prefix cache still believed it was in place. The model was speaking with no reference voice at all. With the fix, and each strength run in its own process, plain sentences score 0.81 to 0.87 at every strength and bracketed ones 0.56 to 0.78. Guidance at 4 was fine, and the published sample had drawn an unlucky seed, so I regenerated every sample with seed 0. The weights and the Freya numbers were not affected.

Working with Claude Code

I ran the whole project with Claude Code. I decided what to try next and what would count as evidence, and Claude wrote most of the code, ran the experiments and searched the literature. Two habits mattered most. Every claim had to come with a measurement, which is how the cache bug showed up as a number I could chase. And before calling an idea new I asked for a search aimed at that specific idea, which is how SlimWise and GPTAQ turned up before this post went out. Several ideas I liked ended there too, such as steering emotion directly in the hidden states, which SteerSpeech and EmoSteer-TTS already cover.

Links

Weights are at huggingface.co/kasparovabi/drama-3-tr-speaker and code at github.com/kasparovabi/drama-3-tr. Built with Fish Audio. The weights fall under the Fish Audio Community License.

Ahmet Kazankaya

DEV Community: Kasparov Abi