DEV Community

jamilxt
jamilxt

Posted on

Mixture of Experts (MoE): Why Big AI Models Are Cheaper to Run Than They Look

#ai #machinelearning #llm #deeplearning

DeepSeek-V3 has 671 billion parameters. When it processes your prompt, it uses about 37 billion of them per token. The rest sit idle.

That is not a typo, and it is not a trick. It is an architecture called Mixture of Experts, or MoE. Once you understand it, a lot of confusing things about modern AI models start making sense: why a "600B" model can respond in real time, why some models cost a fraction of others to serve, and why model cards list two different parameter counts.

The problem with dense models

Most language models you know of are dense. Every token you feed in flows through every single parameter. A 70B dense model uses all 70 billion parameters for every word it generates.

That simplicity is nice. It is also expensive. Double the parameters, double the compute per token. If you want a smarter model, you pay for its full size on every single request, forever.

For years, this was the deal. Bigger models meant proportionally bigger inference bills. MoE breaks that link.

What a MoE layer actually does

In a transformer, each layer has two parts: an attention block and a feed-forward network (FFN). The FFN holds most of the parameters. MoE replaces that single FFN with several parallel FFNs, called experts.

Here is the part people get wrong. The experts are not departments. There is no "code expert" or "math expert" you can point at. Instead, a small router network looks at each token, scores all the experts, and picks the top two (in Mixtral's case, two out of eight). Only those two FFNs process the token. Their outputs get combined and passed to the next layer.

The key detail: the choice happens per token, and it can change from one token to the next. Writing code might route a token through experts 3 and 7. The very next token might go through experts 1 and 5. Every token still has access to the full model, but each one only touches a slice of it.

So you get two numbers instead of one. Total parameters tell you how much knowledge the model can hold. Active parameters per token tell you what inference actually costs.

The idea is older than the transformer

MoE did not start with LLMs. The original paper, "Adaptive Mixtures of Local Experts" by Jacobs, Jordan, Nowlan, and Hinton, came out in 1991. The setup there was small-scale: several simple networks, each learning to handle a subset of the training cases, with a gating network deciding who handles what.

The idea then sat in the research literature for decades. Transformers arrived in 2017, and researchers quickly realized the FFN layers were a natural place to apply it. A 2017 paper by Shazeer and colleagues brought MoE into language modeling, but the models were hard to train and the engineering overhead kept most teams away.

Switch Transformer made it practical

Google's Switch Transformer paper (2021) is where MoE became usable at scale. The team simplified routing so each token goes to exactly one expert instead of several. That cut routing computation and communication cost, and the paper reported up to 7x faster pre-training than a comparable dense T5 model at the same compute budget.

It also pushed parameter counts somewhere new: the largest Switch model reached 1.6 trillion parameters across 2,048 experts, with a fixed FLOPs cost per token regardless of expert count. That result established the pattern every later MoE model follows: grow parameters along the expert axis, keep compute per token flat.

The same paper is honest about the pain. Large sparse models were unstable to train, and getting them to work took careful tricks around precision and initialization. That pain did not go away later; teams just got better at managing it.

Mixtral: MoE goes open

In December 2023, Mistral AI released Mixtral 8x7B, an open-weight MoE model. Its numbers are the cleanest illustration of the total-vs-active split you will find:

  • 47B total parameters
  • 13B active per token (2 of 8 experts, per layer)
  • Matched or beat Llama 2 70B on most benchmarks while using roughly 5x fewer active parameters
  • Apache 2.0 license, so anyone could inspect and run it

Mixtral also produced one of the more useful findings about experts. The paper's routing analysis found experts specialize more by syntax than by topic. The router cares about token-level patterns, not whether your prompt is about law or biology. Keep that in mind whenever someone describes MoE experts as if they were human specialists. They are not.

DeepSeek-V3: the economics

DeepSeek-V3 (December 2024) took the total-vs-active ratio to an extreme: 671B total parameters, 37B active per token. Its technical report says full training took 2.788 million H800 GPU hours, about $5.576 million at a $2 per GPU-hour rental rate, for a model that scored 88.5 on MMLU.

Two honesty notes on that famous number. First, it covers the final training run's compute only. It excludes research salaries, prior failed runs, and infrastructure. Second, the low figure is not luck; it is partly the architecture. With only 37B of 671B parameters active per token, both training and inference compute stay far below what a dense 671B model would need. MoE plus FP8 training and other engineering made the number possible.

Whether or not you care about training economics, the inference side matters to you directly. When you pay per token for an API call, a MoE backend is a big part of why aggressive pricing is possible at all.

The catches

MoE is a trade, not a free lunch. Four things you should know before repeating the "it's cheaper" line:

Memory does not shrink. Inference compute scales with active parameters, but all 47B of Mixtral's weights still need to sit in memory. To run it locally you need the RAM or VRAM for the full model, not the active slice. MoE saves compute, not memory. This is the single most common misunderstanding.

Routing needs babysitting. If the router sends most tokens to a few experts, the rest waste away. Every serious MoE model uses load-balancing tricks during training to keep expert usage even. DeepSeek even developed an "auxiliary-loss-free" balancing method to avoid the quality penalty that balancing losses can cause.

Fine-tuning is touchier. Sparse models are known to be more sensitive during fine-tuning than dense ones. If you plan to fine-tune a model yourself, a dense model at the same active size is the safer starting point.

Benchmarks compare differently. A MoE model matching a dense model is not "same size beats same size." Mixtral beating Llama 2 70B is more precisely: 13B active parameters with 47B worth of stored knowledge beat 70B active parameters. Knowledge capacity scales with total parameters, so a MoE model has an inherent capacity advantage at equal compute. That is the whole point, but it means raw parameter comparisons between architectures can mislead you.

What this means for you

When you read a model card now, look for two numbers:

  • Total parameters: what the model knows, and what you need in memory to run it
  • Active parameters per token: what each token costs you in compute, latency, and API price

If a card says "671B" with no active count, be skeptical of any latency claim. If it says "37B active," you know the inference bill will look more like a 37B model than a 671B one.

The second thing: the MoE era explains the current market. Open-weight models that compete with closed ones at a fraction of the serving cost, API prices that keep falling, and local model communities obsessing over which quantized MoE fits in which GPU. None of that is marketing. It is mostly this one architectural decision doing its job.

My take

MoE is one of those rare ideas where the 1991 version and the 2024 version differ mostly in scale, not concept. Hinton's group wanted small networks dividing up a vowel task. Mistral and DeepSeek route tokens through hundreds of FFNs. The core move is identical: stop paying for every parameter on every input.

I find the honest framing is this: MoE did not make models cheaper. It made a particular kind of bigger model affordable, and almost every frontier lab decided that trade is worth it. The dense frontier has not stood still, but the current open-weight leaders are nearly all MoE. For now, that tells you where the industry landed.

Sources:

Top comments (0)