For a take-home assignment from e3 Group, I got a question I couldn't stop thinking about: how fast can text-to-speech start speaking if you refuse to launch a thousand CUDA kernels?
The answer, on an RTX 5090: the megakernel-backed decoder reached ~25 ms time-to-first-chunk, running at RTF 0.12 (it generates audio tokens ~8× faster than real time). Repo: e3-megakernel-tts-takehome.
Honest scope note before anything else: this was a take-home focused on the talker-decode path. The codec stage was partly stubbed, so I make no claims about production-quality end-to-end audio — the interesting engineering (and the numbers) live in the decoder.
The idea: one kernel that never exits
A normal LLM/TTS decode step launches a long chain of kernels — embedding lookup, attention, MLP, norms, sampling — and every launch pays fixed overhead. For big-batch training that overhead amortizes to nothing. For latency-critical, batch-of-one decoding it is the cost: microseconds of launch latency and pipeline bubbles between every op, dozens of times per token.
A persistent megakernel inverts this: launch one kernel that owns the GPU and loops over decode steps internally, keeping weights hot and skipping the launch/teardown cycle entirely. AlpinDale wrote a brilliant ~1,200-line persistent CUDA megakernel for Qwen decoding — my job was to make it decode a different model family.
Porting it to Qwen3-TTS-1.7B
The megakernel was written for a Qwen text model. Qwen3-TTS-1.7B's talker looks similar on paper and disagrees in exactly the places a hand-tuned kernel cares about:
- Model constants. Hidden size, intermediate (MLP) size, and vocab size are baked into the kernel's tiling and launch configuration as compile-time constants. Retuning them isn't find-and-replace — the tile shapes have to keep the occupancy and shared-memory budget the kernel's design assumes.
- Untied embeddings. The text model tied input embeddings and the output head; the TTS talker doesn't. That splits one weight buffer into two and changes what the final projection reads.
- Multi-section rotary embeddings (MRoPE). Instead of one rotary frequency band over the whole head dimension, the TTS talker splits position encoding into sections. The kernel's inline RoPE math had to be reworked to apply per-section rotations without breaking its memory-access pattern.
The debugging loop was the classic one: diff the megakernel's hidden states against a reference HuggingFace forward pass, layer by layer, until the divergence pointed at whichever assumption I hadn't found yet.
Making it talk end-to-end
A decoder alone isn't a voice agent, so the pipeline runs through Pipecat:
Deepgram (speech-to-text) → Groq (LLM reply) → Qwen3-TTS talker on the megakernel (streaming audio tokens out chunk by chunk).
Streaming is the point of the whole exercise. Time-to-first-chunk is what a human perceives as "it answered instantly" — and TTFC is exactly where per-step kernel-launch overhead hurts most, because the first chunk can't hide behind pipelining. ~25 ms to first chunk makes the TTS stage effectively free next to the LLM and network hops.
What I took away
- Kernel launch overhead is a latency tax, not a throughput tax. Persistent kernels are the difference in interactive, batch-of-one workloads; they're the wrong tool for big offline batches.
- "Similar architecture" is a trap. Tied vs. untied embeddings and single vs. multi-section RoPE are one-line differences in a config file and days of work in a hand-tuned kernel.
- Layer-by-layer diffing against a reference implementation is the only sane way to port numerical code. Trust nothing end-to-end until every layer matches.
- Reading someone else's excellent CUDA (AlpinDale's kernel is genuinely instructive) teaches you more about GPU programming than writing mediocre CUDA of your own.
Code: github.com/pratham7711/e3-megakernel-tts-takehome
I'm Pratham Sharma, a software engineer (full-stack & AI) at Leegality. I write about the engineering behind things I actually shipped. Portfolio: prathamsharma.in · GitHub: pratham7711 · LinkedIn: pratham-sharma-a555b8207
Top comments (0)