Muon gets written up as the optimizer that finally beats AdamW. Reading Keller Jordan's original post, that framing falls apart in the first section — and what's actually going on is more interesting.
It was never meant to cover the whole model
Here's the definition, from the post that introduced it:
Muon is an optimizer for 2D parameters of neural network hidden layers.
Hidden layers. 2D. That's the scope, stated up front. What about everything else?
When training a neural network with Muon, scalar and vector parameters of the network, as well as the input and output layers, should be optimized by a standard method such as AdamW.
So your biases, your LayerNorm gains, your scalars — AdamW. And the input and output layers too, which is the part I found genuinely surprising:
Empirically, we find that it is also important to optimize input and output parameters using AdamW, even though these are typically 2D.
Read that again. The embedding matrix and the classifier head are 2D. They fit Muon's stated scope perfectly. And you should still use AdamW on them, because it works better. Jordan is explicit that for transformers, AdamW "should be used for the embedding and final classifier head layers in order to attain the best performance".
A transformer's embedding and unembedding are a serious fraction of its parameters. So "we switched from AdamW to Muon" almost always means "we run both, and Muon takes the middle."
Then it borrowed AdamW's defining feature
This is the part that made me laugh.
AdamW exists because of one idea: decouple weight decay from the gradient update. That's the whole paper — it's in the title, Decoupled Weight Decay Regularization. Take that away and you have Adam.
Now here's Moonshot AI's paper on scaling Muon to real model sizes. Their two contributions:
We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale.
Contribution number one is weight decay. Muon needed AdamW's signature idea bolted on before it would scale.
That isn't a criticism of Muon — it's how optimizer research works, and the result is good. But it does make "Muon replaces AdamW" a strange thing to say about an optimizer that runs alongside AdamW and adopted its main contribution.
What it actually buys you
The numbers are real and worth knowing precisely, because they get inflated in the retelling.
From Jordan's results:
- CIFAR-10 to 94% accuracy: speed record improved from 3.3 to 2.6 A100-seconds
- NanoGPT speedrunning to 3.28 val loss: faster by a factor of 1.35x
- A 1.5B transformer to GPT-2 XL level on HellaSwag: 10 8xH100-hours, against 13.3 for AdamW
And from Moonshot at larger scale, with scaling-law experiments behind it:
Muon achieves ∼ 2× computational efficiency compared to AdamW with compute optimal training
They trained Moonlight, a 3B/16B MoE, on 5.7T tokens with it. So this isn't a toy-scale result any more.
But notice the spread: 1.35x on NanoGPT, roughly 1.33x on the 1.5B run, around 2x in Moonshot's scaling-law setting. "2x faster than AdamW" is the number that travels, and it's the top of the range, from one paper's specific setup.
The mechanism, briefly
AdamW keeps two running averages per parameter and rescales each weight individually. Every parameter is treated as its own little scalar problem.
Muon treats a weight matrix as a matrix. It takes the momentum update and orthogonalises it using a Newton-Schulz iteration, so the update doesn't collapse toward a few dominant directions. That's why it only makes sense for 2D hidden layers — the whole idea depends on there being a meaningful matrix structure to condition.
Which also explains the embedding exception. An embedding table is 2D in shape, but it isn't doing matrix-multiply work in the same sense, and empirically it prefers AdamW.
What I'd take away
Muon is a hidden-layer optimizer, not a whole-model one. If someone says they replaced AdamW, ask what's optimizing their embeddings.
The headline number is a range, not a constant. 1.35x on speedrunning, around 2x in Moonshot's scaling-law experiments.
It's additive, not a replacement. The current best setup is Muon on 2D hidden layers plus AdamW on everything else — which is exactly what the original post recommends.
Longer version with every quote sourced at diffstudy.com. Sources: Keller Jordan's Muon post, arXiv 2502.16982 (Moonshot AI), arXiv 1711.05101 (AdamW).
Top comments (0)