I started on 8-bit machines, writing games where you drew the triangles yourself and counted clock cycles in Intel's V-Tune until the inner loop stopped hurting. The whole job was getting under the abstraction and making the metal do exactly what you asked. Then compilers got good, machines got fast, and for twenty-odd years that skill went quiet.
It came back, because a language model spends almost all its time doing the one operation I used to hand-tune on a 386: multiply a matrix by a matrix, add up the results, move on. So I worked on an inference engine by hand, in WGSL, running a real model in a browser tab, on kernels I can read line by line. Here's the road to it, and why it tells you something about your prompts.
2008: a model is a pile of numbers you can save to disk
The instinct pointed me at neural networks in 2008, on a contract at the British Library. I wrote a 4 layer network for a project, starting it out in LINQPad (as I did with most throw away tests), in C#, to sort academic paper titles into Dewey Decimal subjects: physics, medicine, agriculture etc. Four layers. Turn the title into a bag of words, multiply by a matrix of weights, bend the result through a sigmoid so it can do more than draw straight lines, multiply again, read off the winner with a softmax cross entropy layer.
Two things stuck. The maths is a matmul and a curve, over and over. And the knowledge it learns is nothing but those weight matrices. In my pre-school AI level knowledge, I called them synapses. Once trained, you serialise them to a file and that file is the model, the same idea as the multi-gigabyte files on Hugging Face today, only mine was a few hundred kilobytes. The toy became real at the library: instead of my 62 sample titles I trained it on 1.6 million PDF titles, which took 18 days on a 2008 Core i7 CPU. Expensive training makes the weights, a cheap forward pass uses them. That split is the whole LLM game, then and it's the same now.
The HLSL detour that paid off later
Around then I built a small library of WPF pixel-shader effects in HLSL, the shader language for DirectX: colourise, desaturate, a skin-shade for softening portraits. Nothing to do with machine learning, mostly an excuse to get back to assembly-era fiddling, this time on a GPU. It put HLSL in my hands a decade before I actually needed it. Skills you pick up for one thing sit on the shelf and pay off in another. I love keeping the saw sharp.
2019: I wanted to run one on the GPU myself
After the library I kept building nets for the fun of it. MNIST, the handwritten-digits, Hello World of Neural Networks. Then a CNN to spot a dog fouling a footpath, PoopScanner, which I was going to run on a Raspberry Pi and sell to councils with an audible "You're being watched. Clean that up.". The councils were not ready. Finally a network for recognising vehicle panel damage, this was ever so slightly before the frontier multimodal models came into being.
But...they're all the same shape underneath: multiply by a matrix, apply a curve, do it again. And from the shader work I already knew how to drive a GPU, rustily, slowly & badly, but enough to make progress. So in 2019 I put the two together and started an inference engine in HLSL, to watch the kernels do the work rather than call someone else's library (there's a Donald Knuth line at the end that sums up why). I got a chunk working, then life / job / contracting happened and it sat half-finished for nearly a decade. I have a whole bunch of dumpster fire code sat rotting away just like this one, that might be useful to resurrect if Claude can help take the strain ;)
Finishing it in a week, with an agent
A few weeks ago I picked it up again, and since I'm pretty much AI first I'll be straight about how. I sat down with Claude and we finished it together, rewritten from HLSL into WGSL so it runs in a browser tab. The design is mine and decades old, the kernels are hand-written and I can read you almost every line of tghe HLSL, but I won't pretend I typed every character anymore. Claude did a lot of the typing. The English I write to steer a model is now as real a language as the HLSL I wrote for the GPU, and this project wanted both, for the cost of a few hundred million tokens out of my 20x subscription.
The model I used in the demo is Qwen2.5-0.5B, downloaded from Hugging Face: a 265MB file of weights, the same species as my 2008 "synapses". Mine had a few thousand numbers, this has half a billion. Mine knew a few hundred words, this carries 151,936 pieces of text. Open it up and the inner loop is my C# toy again, 24 times over.
Tokens
You type words. The model sees tokens, chunks of text with a number attached. "Paris" might be one token, "unbelievably" three. The tokeniser that splits them is a lookup table of about 151,000 pieces plus a set of merge rules. I had written mine in 2018 in plain C# with the help of a pre-built stemmer - Stemmers.Net. Claude re-wrote it in JavaScript and tested it against the real Qwen one until they matched on every character, because if they disagree by a single token the model produces confident gibberish and you lose a day blaming the maths. So 50% mine I guess, but the important part was that I could write it if I had to.
Each token becomes a list of 896 numbers, its vector embedding. A prompt is a stack of those vectors, and everything after is arithmetic on the stack. The model does one job forever: score all 151,936 possible next tokens, take the highest, append it, run again. That word-by-word streaming in every chatbot isn't an animation. It's the real speed showing that loop! Every word is a full pass through the model.
The inner loop is still a matmul
In our tiny demo case, each pass runs the same block of maths 24 times: attention, then per-token crunching. Every step is dominated by one operation, a matrix multiply. The attention projections, the crunch, the final scoring against a 151,936-wide table, all matmul. So I wrote the matmul kernel by hand, and two old tricks from my assembly days make it fast.
Tiling. Memory is slow and there are millions of multiplies, so you don't fetch from GPU main memory each time. You grab a 16 by 16 block, park it in a scratchpad next to the compute units, and reuse it hard. Same instinct as hoisting a value out of a loop. This was similar to the 3d polygon filling with small textures on the CPU and using V-Tune to make sure that the cache never got thrown out.
Four-bit weights. The weights ship as 16-bit numbers, half a billion of them, about a gigabyte. Moving a gigabyte through the GPU per word is a real bottleneck, the fetching, not the multiplying. So I store each weight in 4 bits: take 32 of them, set a shared scale from the biggest, store the rest as small integers against it. A gigabyte becomes 265MB. You never unpack them back into memory, you unpack each one inside the loop at the moment you multiply. Cheap arithmetic to save expensive memory, the oldest trade there is.
On the LLM scale we're still on the tardigrade level - if you imagine GPT3 is ~175 billion parameters using 16bit floats, that's 2 bytes each amounting to ~350GB. That model is old, we're now entering the age of trillion weight models!
Attention is you, steering. Steering IS your job.
So, here's the part most explanations skip, and the one that connects the maths to what you do all day: writing prompts.
Every layer, each token looks back and works out how much each earlier token matters right now. Each makes a query and a key, you multiply one token's query by another's key, a big number means relevant, softmax turns the scores into percentages, and you blend the earlier tokens in those proportions.
So a prompt is you loading those weights by hand. Write "explain this simply" and the word "simply" sits in the context, and every word the model generates glances back at it and gets pulled towards something short and plain. Swap it for "rigorously" and the same weights write something denser, because the attention lands elsewhere.
You didn't change the model, you changed what it looks at.
That's the whole skill of prompt wording, no magic, just understanding the maths.
And because I wrote the attention kernel, I've got those weights in hand ready to surface in something you can see. Usually the model throws them away after each word. I don't. So the demo can draw them: run a prompt and watch the earlier tokens light up to show what each new word leaned on and see the actual % numbers, change one word and watch the lights move. It'll even knock out each word of your prompt in turn when you check the MEASURE STEERING checkbox and measure which ones genuinely steer the answer, so you can watch a word do work or watch it get ignored.
A 0.5B model with two attention heads is a small window so don't expect it to solve any complex problems beyond writing a haiku, and the big models will attend in subtler ways across many heads. But the mechanism is the mechanism, and seeing it once on something you can hold in your head changes how you read the large ones.
Honest numbers
I raced my version against transformers.js, Hugging Face's tuned browser runtime, same model and machine, both on WebGPU. I know I can't win, but want to know if I'm still employable ;)
I win the first word: first token on screen in about 450ms against their 590, because my prefill is lean with no framework on top. They win sustained speed: warmed up, about 20 tokens a second to my 12, because a large team has spent far longer than a fortnight tuning it. Arguably the most important stat. Ask both "capital of France, one word" and they give the identical token, Paris. Ask something open-ended and they drift apart after a sentence, because my four-bit rounding and theirs aren't bit-for-bit the same. They agree where there's a right answer and drift where there isn't.
That's the four-bit tax, visibly measured. Four bits is just a plaything here because I'm not bothered about showing 'correct' answers, only how we reach them
The point was never to win a benchmark. It was to understand every line, and get close enough to a tuned engine that "close" is the interesting word. I can live with my failure to match a team of 20.
Two traps, because you'll hit them
My output came out as pure garbage, all zeros then noise, and only once the whole thing was assembled, though every kernel passed its own test. The cause was one missing flag on a GPU buffer, the one that says "you may copy out of this". Without it, the copy filling the attention cache silently did nothing, so the model attended over zeros, which becomes divide-by-zero, which becomes NaN, which spreads through the network in a single pass, like a forest fire started with a dumb ass disposable barbecue. No crash, no console error, because WebGPU reports that class of mistake down a side channel you have to go and ask for. One flag, half a day. This never gets old.
The other: I set a starting value one notch past the largest 32-bit float. The shader compiler rejected the whole file for that one constant, silently, and handed me a kernel that ran and did nothing. If you write GPU code, the compiler failing in silence will cost you the most hours. THE MOST HOURS. Check your shaders compiled at all. My fault again. I think I preferred C++ STL compiler errors to this.
Finally, coding isn't fun if you can't open the box
There's a line from Knuth I keep coming back to.
The problem is that coding isn't fun if all you can do is call things out of a library, if you can't write the library yourself.
He wasn't gatekeeping dependencies. He was mourning the job turning into plumbing between black boxes you're never allowed to open, and pointing out that once you can see inside one you can usually improve it. The "don't use what you couldn't write" version is the folk edit. For most libraries I'd pass it anyway, which is how I know it was never the point.
Doing tasks like this is what keeps me sane in a world where 90% of the profession is exactly that, parameter plumbing using other's code. Dull, yes, but it pays the mortgage, so no one complains, and it's not your employer's job to make it interesting.
I could have just called transformers.js in four lines and had a chatbot by lunchtime, and for a product that's 100% the right call. But I wanted to open the box, to see what happens between the prompt and the words at the level of the arithmetic, because you can't tune what you can't see. Half a billion parameters, and at the bottom of the well it's still just a matmul I can read, and it's taken me nearly 20 years to get here.
Anyway, it's live: type a prompt, watch the tokens land, race it against transformers.js, and turn on the attention view to watch your words steer it, at attentionspan.fortitude-omnis.group. It's one of the builds from the Fortitude Omnis R&D lab. Forty years in, and understanding the inner loop is still as important as ever. Who'd have thought? Are we redundant yet?
Top comments (0)