MiMo-V2-Flash: How Xiaomi Built a 309B MoE Model That Tops SWE-Bench Without Burning Through Compute

Xiaomi's AI research team recently released MiMo-V2-Flash, a 309-billion-parameter Mixture-of-Experts language model that currently holds the top spot among open-source models on both SWE-Bench Verified and SWE-Bench Multilingual. What makes it worth examining isn't just the benchmark number — it's the combination of architectural choices and a post-training method called Multi-Teacher On-Policy Distillation (MOPD) that together let the team achieve frontier-level coding performance at a fraction of the usual compute cost.

The Architecture: Hybrid Attention and Built-In Speculative Decoding

MiMo-V2-Flash has 309B total parameters but only 15B active per token — a standard MoE tradeoff that keeps inference costs manageable relative to a dense model of equivalent capacity. The more interesting architectural decisions are in how it handles attention and decoding speed.

Hybrid attention (SWA + GA at 5:1). Most of the attention layers use Sliding Window Attention with a 128-token window, which dramatically reduces the KV-cache footprint for long sequences. Every sixth layer uses Global Attention to preserve long-range dependencies. This 5:1 interleaving cuts KV-cache storage by roughly 6× compared to full global attention across all layers, while a learnable attention sink bias keeps the model from losing track of distant context.

Multi-Token Prediction as a native draft mechanism. The model includes a 3-layer MTP module — lightweight dense FFNs with their own SWA — that predicts multiple future tokens in parallel. This functions as a built-in speculative decoding draft head, delivering a 2.0–2.6× decoding speedup without requiring a separate draft model. The MTP module adds relatively few parameters (0.33B per block) but meaningfully changes the throughput profile.

256K context window. Pre-training used a native 32K sequence length on 27 trillion tokens, then extended to 256K. The NIAH-Multi retrieval benchmark at 256K scores 96.7, suggesting the extension holds up in practice for long-document and multi-turn agent tasks.

Multi-Teacher On-Policy Distillation: The Post-Training Story

The benchmark result that stands out — 73.4% on SWE-Bench Verified — comes largely from the post-training approach rather than the base model alone. The team developed MOPD to address a real problem with standard RL-based post-training: it's expensive, and using a single teacher model limits how well the student can specialize across different domains.

How MOPD works. Instead of training a single teacher and distilling from it, the team trains multiple domain-specialized teacher models — one focused on math, one on code, and so on — each using large-scale reinforcement learning. The student (MiMo-V2-Flash) then learns from all of them simultaneously, but the key difference from standard knowledge distillation is that the process is formulated as an RL problem:

• The student generates responses on-policy (from its own current distribution), which eliminates the exposure bias that plagues offline distillation.
• Teachers provide dense, token-level reward signals rather than sparse sequence-level feedback. This gives the student much more informative gradients.
• Because the student is learning from its own generations rather than teacher demonstrations, the training stays stable even as the student improves.

The efficiency claim is striking: the team reports that MOPD requires less than 1/50th of the compute of a comparable SFT + RL pipeline while matching or exceeding teacher performance. That's a significant reduction if it holds across different model scales and domains.

Why on-policy matters here. Standard knowledge distillation has a well-known problem: the student is trained on teacher-generated data, but at inference time it generates from its own distribution. The gap between these two distributions grows as the student diverges from the teacher, leading to compounding errors. By keeping the student on-policy throughout distillation, MOPD avoids this — the student always sees data from its own current distribution, so the training signal stays relevant.

Benchmark Results in Context

The headline numbers from the post-training evaluation:

Benchmark	Score
SWE-Bench Verified	73.4%
SWE-Bench Multilingual	#1 open-source
GPQA-Diamond	83.7%
MMLU-Pro	84.9%
Arena-Hard (Creative Writing)	86.2%

SWE-Bench Verified tests whether a model can resolve real GitHub issues — it requires understanding a codebase, identifying the relevant files, writing a patch, and passing the existing test suite. A 73.4% score puts MiMo-V2-Flash on par with several closed-source frontier models on this specific task.

The GPQA-Diamond score (83.7%) is also notable — this benchmark tests graduate-level science reasoning, and it's not a task the model was explicitly optimized for, suggesting the MOPD approach generalizes beyond coding.

Deployment Considerations

The 309B parameter count means hardware requirements are substantial. All expert weights need to fit in VRAM regardless of how many are active per token:

• FP16/BF16: ~618 GB VRAM — roughly 10× H100 80GB GPUs
• FP8: ~309 GB — 4× H100 80GB or 3× H200 141GB
• INT4: ~155 GB — 2× H200 141GB

The model is compatible with vLLM (v0.8.0+) and SGLang, both of which support MoE expert parallelism. For the 256K context window, the recommendation is to start with --max-model-len 32768 to verify stability before scaling up, since KV-cache requirements grow significantly at full context length.

The model weights are available on Hugging Face under an open license, and the GitHub repository includes the technical report with full training details.

What's Worth Watching

Two things stand out about MiMo-V2-Flash beyond the benchmark numbers.

First, the MOPD approach is a concrete answer to a real scaling problem. Post-training with RL is expensive, and the standard recipe (SFT then RL with a single reward model) doesn't scale gracefully to multiple domains. If the 1/50th compute claim holds up under scrutiny, it's a meaningful contribution to how the field thinks about efficient post-training.

Second, the hybrid SWA+GA attention pattern — combined with built-in MTP for speculative decoding — represents a coherent set of engineering choices aimed at making a large MoE model actually deployable. The 6× KV-cache reduction and 2–2.6× decoding speedup aren't just nice-to-haves; they're what makes a 309B model viable for agentic workflows where you're running many sequential inference calls.

The technical report goes into considerably more depth on the training pipeline, including the RL curriculum and how the domain-specialized teachers were constructed. Worth reading if you're working on post-training methods or large-scale MoE deployment.