DEV Community: Pratham Sharma

Porting a 1,200-line persistent CUDA megakernel to Qwen3-TTS: ~25 ms to first audio chunk

Pratham Sharma — Mon, 06 Jul 2026 06:59:58 +0000

For a take-home assignment from e3 Group, I got a question I couldn't stop thinking about: how fast can text-to-speech start speaking if you refuse to launch a thousand CUDA kernels?

The answer, on an RTX 5090: the megakernel-backed decoder reached ~25 ms time-to-first-chunk, running at RTF 0.12 (it generates audio tokens ~8× faster than real time). Repo: e3-megakernel-tts-takehome.

Honest scope note before anything else: this was a take-home focused on the talker-decode path. The codec stage was partly stubbed, so I make no claims about production-quality end-to-end audio — the interesting engineering (and the numbers) live in the decoder.

The idea: one kernel that never exits

A normal LLM/TTS decode step launches a long chain of kernels — embedding lookup, attention, MLP, norms, sampling — and every launch pays fixed overhead. For big-batch training that overhead amortizes to nothing. For latency-critical, batch-of-one decoding it is the cost: microseconds of launch latency and pipeline bubbles between every op, dozens of times per token.

A persistent megakernel inverts this: launch one kernel that owns the GPU and loops over decode steps internally, keeping weights hot and skipping the launch/teardown cycle entirely. AlpinDale wrote a brilliant ~1,200-line persistent CUDA megakernel for Qwen decoding — my job was to make it decode a different model family.

Porting it to Qwen3-TTS-1.7B

The megakernel was written for a Qwen text model. Qwen3-TTS-1.7B's talker looks similar on paper and disagrees in exactly the places a hand-tuned kernel cares about:

Model constants. Hidden size, intermediate (MLP) size, and vocab size are baked into the kernel's tiling and launch configuration as compile-time constants. Retuning them isn't find-and-replace — the tile shapes have to keep the occupancy and shared-memory budget the kernel's design assumes.
Untied embeddings. The text model tied input embeddings and the output head; the TTS talker doesn't. That splits one weight buffer into two and changes what the final projection reads.
Multi-section rotary embeddings (MRoPE). Instead of one rotary frequency band over the whole head dimension, the TTS talker splits position encoding into sections. The kernel's inline RoPE math had to be reworked to apply per-section rotations without breaking its memory-access pattern.

The debugging loop was the classic one: diff the megakernel's hidden states against a reference HuggingFace forward pass, layer by layer, until the divergence pointed at whichever assumption I hadn't found yet.

Making it talk end-to-end

A decoder alone isn't a voice agent, so the pipeline runs through Pipecat:

Deepgram (speech-to-text) → Groq (LLM reply) → Qwen3-TTS talker on the megakernel (streaming audio tokens out chunk by chunk).

Streaming is the point of the whole exercise. Time-to-first-chunk is what a human perceives as "it answered instantly" — and TTFC is exactly where per-step kernel-launch overhead hurts most, because the first chunk can't hide behind pipelining. ~25 ms to first chunk makes the TTS stage effectively free next to the LLM and network hops.

What I took away

Kernel launch overhead is a latency tax, not a throughput tax. Persistent kernels are the difference in interactive, batch-of-one workloads; they're the wrong tool for big offline batches.
"Similar architecture" is a trap. Tied vs. untied embeddings and single vs. multi-section RoPE are one-line differences in a config file and days of work in a hand-tuned kernel.
Layer-by-layer diffing against a reference implementation is the only sane way to port numerical code. Trust nothing end-to-end until every layer matches.
Reading someone else's excellent CUDA (AlpinDale's kernel is genuinely instructive) teaches you more about GPU programming than writing mediocre CUDA of your own.

Code: github.com/pratham7711/e3-megakernel-tts-takehome

I'm Pratham Sharma, a software engineer (full-stack & AI) at Leegality. I write about the engineering behind things I actually shipped. Portfolio: prathamsharma.in · GitHub: pratham7711 · LinkedIn: pratham-sharma-a555b8207

How I made a scroll-scrubbed video portfolio fast (Next.js 15 + GSAP + canvas)

Pratham Sharma — Mon, 06 Jul 2026 06:58:33 +0000

How do you ship a portfolio that plays 45 seconds of video, scrubbed by scroll, without shipping 12 MB of video?

That was the problem with my portfolio, prathamsharma.in. It tells a story — a volleyball rally, three touches, three pitches — and every beat is footage that plays as you scroll. The first deployed version worked, but it was slow and glitchy: 12.8 MB of media fetched eagerly, 210 kB of First Load JS, and scroll jank on anything weaker than a desktop GPU.

Here is exactly what I changed, with numbers. Everything below is live on the site right now.

1. Videos → WebP frame sequences on a canvas

Scroll-scrubbing a <video> element is unreliable: seeking is async, keyframe intervals cause visible stutter, and mobile Safari fights you on autoplay and inline playback.

So there are no video files at all. Each clip is extracted to 61 frames at 12 fps, 1280×720, and drawn onto a <canvas>:

ffmpeg -i clip.mp4 -vf "fps=12,scale=1280:-2" /tmp/frames/%03d.jpg
for f in /tmp/frames/*.jpg; do
  cwebp -q 68 -m 6 "$f" -o "public/media/seq/dig/$(basename "${f%.jpg}").webp"
done

GSAP's ScrollTrigger drives a single progress value from 0 to 1; the component maps it to a frame index and draws it. Scrubbing becomes deterministic — frame N is frame N, no seeking, no codec state.

Two details that mattered:

Quality sweep, not guesswork. I encoded the same sequence at several cwebp quality levels and diffed visually. The source footage was already compressed, so -q 68 -m 6 was the sweet spot: ~20% smaller than the JPEG frames with no visible loss on moving, partially-masked footage. Total media went 12.8 MB → 9.8 MB.
DPR cap. The canvas backing store is capped at devicePixelRatio 1.5. On a 3× phone screen you cannot see the difference on moving footage, and it's less than half the pixels to paint.

2. Load frames like you mean it

Having 244 frames doesn't mean fetching 244 frames up front. Each sequence lazy-loads through two gates:

const io = new IntersectionObserver(start, { rootMargin: '800px 0px' })

An IntersectionObserver with an 800px margin starts a sequence only when the user is approaching it.
A requestIdleCallback (with a setTimeout fallback) defers decoding off the critical path.
A priority prop marks the hero sequence: the first beat loads immediately with default fetch priority, and every other sequence gets img.fetchPriority = 'low' so it never competes with the hero for bandwidth.

Result: the first paint needs one sequence, not four. The rest stream in while you read.

3. Code-split below the fold

The page is one route, but it doesn't have to be one bundle. Everything below the fold became a next/dynamic import — the acts, the marquee, the projects rail, the footer. Purely decorative client-only components (easter eggs, a flash-cut effect) got ssr: false so they don't even render on the server.

First Load JS: 210 kB → 162 kB. The above-the-fold experience — nav, hero sequence, custom cursor — is all that blocks.

4. Cache immutable things immutably

Frame sequences never change; when they do, the filename changes. So they're served with:

Cache-Control: public, max-age=31536000, immutable

via a headers() entry in next.config.ts. Second visit: zero media requests.

5. Stop animating when nothing moves

Small one, but it shows up in profiles: the custom cursor ring used a permanent requestAnimationFrame loop, easing toward the pointer at rest — 60 fps of work to move 0 pixels. Now the loop kills itself when the ring settles within 0.1px and restarts on the next mousemove. Idle CPU when the page is idle.

What I'd tell past me

Scroll-scrubbed video is a frame-sequence problem, not a video problem.
Sweep encoder quality against your actual footage. The right number for masked, moving frames (68) is far below what you'd pick for stills.
"Lazy" needs teeth: an observer margin, an idle callback, and an explicit fetch priority — any one alone still contends with the hero.
Measure before and after. 12.8→9.8 MB and 210→162 kB are boring numbers that produce a non-boring difference on a mid-range phone.

The site is live at prathamsharma.in — scroll it with the network tab open and you can watch the staging happen.