A single-author preprint published this week shows that in a mixture-of-experts model, the expert layers can be trained with no momentum at all -- and that doing so cuts persistent optimizer state from 50.55 gigabytes to 1.29, a 97.4% reduction, while final model quality barely moves. The paper, arXiv:2607.19058 by Nuemaan Malik with matching code on GitHub, calls the method SkewAdam.
Key facts
- The headline number: persistent optimizer state falls from 50.55 GiB to 1.29 GiB on a 6.784-billion-parameter mixture-of-experts model.
- The catch: peak training memory falls from 81.4 GiB to 31.3 GiB -- about 61%, not 97%.
- Who: Nuemaan Malik, a self-funded single-author preprint, not peer-reviewed, submitted July 2026.
- Primary source: the paper and the repository.
Anyone who has tried to train a large model has hit the wall this addresses. The optimizer -- typically AdamW -- keeps two extra numbers for every parameter: a running average of recent gradients, called momentum, and a running average of their squares, used to scale the step size. That bookkeeping is often larger than the model itself, and it is the reason a model that fits comfortably in memory for inference will not fit for training.
SkewAdam's insight is that in a mixture-of-experts model, not every parameter has earned that bookkeeping. It assigns state by role:
- Dense backbone (about 5% of parameters): full 32-bit momentum plus a compressed second-moment estimate.
- Experts (about 95%): compressed second moment only -- no momentum.
- Router (under 0.01%): full-precision second moment, no momentum.
The reasoning is clean once stated. Every token that passes through the model updates the dense backbone, so its momentum is a running average over consecutive, recent information. But with top-2-of-128 routing, any individual expert is touched sparsely and irregularly. Its "recent gradient average" is an average over updates that may be dozens of steps apart. That is not momentum in any useful sense -- it is a stale number occupying four bytes per parameter across 95% of the model.
The analogy is a company where the core team meets daily and 128 specialists get consulted occasionally. Keeping detailed running notes on what each specialist said last time is reasonable if they speak every day. If they speak once a month, the notes are mostly archaeology.
The compression trick underneath is not new -- the factored variance estimator and RMS clipping come from Adafactor (original paper), which stores row and column summaries instead of a full per-parameter variance tensor. SkewAdam's contribution is deciding who gets what, not inventing new machinery.
Now the number, scoped honestly. On the paper's 6.784-billion-parameter two-block model, persistent optimizer state drops from 50.55 GiB to 1.29 GiB. But peak allocated training memory falls from 81.4 GiB to 31.3 GiB, roughly 61.5%, because weights, gradients, activations and temporaries are all still there. The correct sentence is "persistent optimizer state falls from 50.55 to 1.29 gigabytes" -- never "a 6.7-billion-parameter model trains in 1.29 gigabytes." Of the remaining 1.29 GiB, about 1.27 is the backbone's momentum; the experts' compressed variance is around 12 megabytes and the router's roughly 2.
Throughput, which almost nobody quotes, lands at about 5,000 tokens per second: 6.6% faster than AdamW and 1.5% slower than Lion, the fastest baseline tested. So the tiering is not paid for in speed relative to AdamW. But the comparison that would settle the question -- SkewAdam against the twenty-times-larger uniform momentum-plus-factored configuration on identical hardware -- is not reported, so "tiering is free" is not established.
And then there is the result that reframes the whole paper. Restoring momentum for the experts raises state from 1.29 to 25.29 GiB and changes final validation perplexity by 0.2. A uniform momentum-plus-compressed policy matches the tiered version within single-run variation. Read plainly, the contribution is memory at parity rather than a better optimizer. In this configuration, momentum on sparsely updated experts is dead weight -- which is a genuinely useful negative result, and more honest than the headline number suggests.
The limits are substantial and the author states them. A deliberately shallow two-block model, 128-token contexts, 82 million training tokens, mostly single runs, and downstream evaluations near chance level. The comparison ran on a 141-gigabyte H200 so that AdamW would fit at all, meaning the 31.3-gigabyte result was never actually measured on a 40-gigabyte card. State sizes are analytic counts rather than measurements. And weight decay was effectively inert under 16-bit rounding in every configuration tested -- a detail that quietly undermines the regularization side of all the baselines too.
This is a promising role-aware allocation policy for heavily expert-skewed models, published without peer review, on a small stress test. It is not a demonstrated replacement for AdamW at frontier scale, and the paper does not claim to be one.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)