9 Triton kernels in 9 weeks: what PyTorch taught me by winning

RV42 — Thu, 08 Oct 2026 17:32:02 +0000

I'm Richael (dh8116), a Year 11 student in Auckland. Since late July I've written one Triton kernel a week and benchmarked each against PyTorch, publishing the numbers even when PyTorch won, which was often. Here's what nine weeks taught me, kernel by kernel.

The ones that won

Fused softmax (week 2). Naive softmax makes three passes over memory. Fusing them into one took it from ~55 GB/s to ~230 GB/s, a 4x jump, and it held a tighter line than PyTorch's own kernel. Lesson one: on a memory-bound op, the passes you don't make are the speedup.

Fused cross-entropy (week 7). Forward and backward in one pass, with the gradient written straight over the logits buffer, so PyTorch's second [N, V] tensor never exists. On a T4 at vocab 131,072 in fp16: 15.90 ms vs 24.03 ms (1.51x), and 1.67x less peak memory.

The first version of this benchmark said 2.5x. That was wrong: I was comparing my fused kernel against an unfused PyTorch baseline. Comparing fused against fused is how I caught myself handicapping the baseline.

RoPE (week 6). Not the speedup, the elegance: backward reuses the exact same kernel as forward. You just flip the sign on sin.

The ones that lost, and why that was the useful part

Flash attention (week 3). Correct, but about 25x slower than PyTorch on my T4.

Matmul (week 4). The output matched, but throughput sat flat around 1 TFLOPS while cuBLAS climbed to 38. Reading the compiled PTX showed zero tensor-core instructions: the compiler never emitted them for my layout.

Fused linear + cross-entropy (week 8). Chunking the lm_head projection into the loss means the [batch, vocab] logits never exist at all. That's about 10x less activation memory, and it was slower at every single size. Sometimes memory is the thing you're buying, and you pay for it in time.

Fused SwiGLU MLP (week 9). Level with torch.compile on time. The win is memory: it holds two big activation tensors between forward and backward, where eager holds four. A compiler will fuse for you, but it won't throw an activation away.

What I'd tell anyone starting

Benchmark fused against fused. An unfused baseline makes every kernel look like a win.
Read the PTX when the numbers are flat. The matmul answer was sitting in the compiled output the whole time.
Memory is a result too. Two of my best kernels "lost" on time and won on memory, and that trade decides batch size.
Publish the losses. They taught me more, and they're the posts people actually argue with.

Code: https://github.com/dh8116/triton-kernels
Every write-up, with methods: https://dh8116.github.io/blog

Next on my list is revisiting flash attention on Ampere, because the T4 result deserves a second look.

I fine-tuned Qwen3-14B for my AI companion app for $1.30. The hard part wasn't the model

RV42 — Sat, 03 Oct 2026 09:29:26 +0000

I built Soulor (soulor.app) independently as a 16y/o high school student. It helps reading social situations with multiple perspectives, and helps users rehearse hard conversations by simulating others with analyses and suggestions, there are also companions that knows the full context if users choose to share that can chat with them naturally. It can be both serious and entertaining, by switching different world views, and it fully respects users' privacy by keeping all texts encrypted. This post is about the model underneath: a LoRA fine-tune of Qwen3-14B, served with vLLM on Modal, and the four things that mattered more than I expected.

1. The first adapter failed on format, not voice

The app doesn't want free text from the model. Every reply comes back in a JSON envelope the frontend parses: the message, plus fields the app acts on. My first adapter learned the persona perfectly and couldn't produce that envelope at all. It sounded right, and the app couldn't use any of it.

The fix was the training data, not the hyperparameters. I rewrote the corpus so every example was written inside the envelope, the same shape production asks for. The retrained adapter scores 15/15 on companion chat and 15/15 on feed comments at production prompt size.

Training cost about $1.30 and took half an hour.

Lesson: if your app parses the model's output, the output format is part of the behaviour you're fine-tuning, so train on it.

2. Latency was memory bandwidth, not a vLLM flag

Replies took 7.9s. I spent a while on serving flags before measuring properly. Decoding a 14B model is bound by memory bandwidth, so each token means reading the weights again, and no flag changes how fast the card can read them. Moving to an H100 took it to 2.0s.

The hosted Qwen fallback had a different problem. With thinking enabled it took 14.6s, and the reasoning was eating the token budget and truncating replies. Turning enable_thinking off gave 5.7s and complete answers.

3. Scale-to-zero needs a gate in front of it

The Modal endpoint scales to zero, which is what keeps it affordable. A cold start takes 108s, though, and nobody waits 108s for a chat reply.

So the gateway asks one question before every request: is a container warm? If yes, the fine-tune goes first in the provider chain. If not, it skips straight to the hosted models. The probe doubles as the warm-up, so the first message of a conversation is answered by a hosted model and the rest can go to the fine-tune. The voice can shift once, early, but I don't pay for an idle GPU between conversations.

4. Two bug shapes caused four different-looking outages

Signups leaving unreachable accounts, cleared chats coming back, companions forgetting the last few turns, and chat hanging for a minute all looked unrelated. They came from just two patterns:

A write fired without await, with its error swallowed. On serverless the function often returns before the write happens, so the work silently never runs.
A network call with no timeout. A fallback chain can fall through an error, but it can't fall through a hang.

Both are now fixed where they can't be forgotten: writes are awaited, and every provider call has a timeout.

What Simulation Mode does with all this

The same engine runs the companion and the panel. In Simulation Mode, one situation goes to five analysts: an Optimist, a Cynic, a Mentor, a Status Observer and a Gossiper. Because it's the same app, the panel already knows the context you gave your companion. You can rehearse a hard conversation before having it for real, then switch back mid-conversation without starting over.

Try Soulor: https://soulor.app/

I'm happy to go deeper on any of this in the comments: the envelope corpus, the warm gate, or the eval harness. More of what I build, including a weekly Triton kernel series benchmarked against PyTorch, is at https://dh8116.github.io/

DEV Community: RV42