AMD Instella-MoE: How a Fully Open 16B MoE Model Proves You Don't Need NVIDIA to Train at Scale

When AMD released Instella-MoE-16B-A3B in September 2026, it did something that most large model releases don't: it published everything. Not just the weights, but the training code, data mixtures, configuration files, Docker environments, and checkpoints from every stage of the pipeline. That level of transparency is rare even among models that call themselves "open source." It also happens to be trained entirely on AMD hardware — a deliberate proof-of-concept that the ROCm ecosystem can handle frontier-scale MoE training from scratch.

This post walks through what Instella-MoE actually does differently, why its two architectural innovations matter, and what the fully open release means for researchers who want to reproduce or build on it.

What "Fully Open" Actually Means Here

The distinction between "open weight" and "fully open" has become increasingly important. Many models release weights under permissive licenses but withhold training recipes, data pipelines, or intermediate checkpoints — making true reproduction impossible.

Instella-MoE takes a different approach. AMD released the complete training recipe via the AMD-AGI GitHub repository and Hugging Face, including:

• Model checkpoints at every major training stage (pre-training, mid-training, long-context extension, SFT, DPO, and RL)
• Training configurations and launch scripts for each stage
• Data preparation tools and data mixture descriptions
• Docker images for both training and inference environments
• The Primus training framework and Miles RL framework used throughout

The license is ResearchRAIL, which restricts commercial use but allows academic and research applications. For practitioners who want to study how a competitive MoE is actually built — not just how it performs — this is a meaningful step beyond what most labs provide.

Architecture: Two Innovations Worth Understanding

Instella-MoE is a decoder-only MoE with 27 layers (26 of which are sparsely activated MoE layers), a hidden size of 2048, 64 routed experts plus 2 shared experts, and 6 experts activated per token. Total parameters: 16B. Active parameters per token: 2.8B. That's a familiar efficiency ratio — similar to DeepSeek-V2's design philosophy of keeping active compute low while maintaining a large parameter pool.

What distinguishes Instella-MoE are two specific architectural choices.

Gated Multi-head Latent Attention (Gated MLA)

Standard Multi-head Latent Attention (MLA), introduced by DeepSeek, compresses the KV cache by projecting keys and values into a low-dimensional latent space. This reduces memory bandwidth during inference significantly. Instella-MoE extends this with a lightweight, input-conditioned sigmoid gate applied to the attention output channels.

The gate learns to selectively attenuate low-utility attention responses — essentially adding a learned filter that decides which attention outputs are worth passing through at full strength. The result is improved model expressivity without a significant increase in parameter count or compute. It's a targeted modification rather than a wholesale architectural change, which makes it easier to analyze and potentially adopt in other models.

FarSkip-Collective

Expert parallelism in MoE models introduces a communication bottleneck: tokens must be routed to the correct expert, which may live on a different device, and the results must be gathered back. This all-to-all communication typically creates "bubbles" — idle periods where compute units wait for data to arrive.

FarSkip-Collective addresses this by overlapping expert-parallel communication with independent computation. Rather than waiting for all expert outputs before proceeding, the system schedules communication to run concurrently with computations that don't depend on those outputs. According to AMD's technical report, this yields a 12.7% improvement in pre-training throughput and up to a 39.2% reduction in time-to-first-token (TTFT) during inference when using SGLang with expert parallelism.

That TTFT improvement is particularly relevant for deployment: faster first-token latency directly affects user-perceived responsiveness in interactive applications.

The Training Pipeline

Instella-MoE was trained using AMD's Primus framework, an open-source training system built on top of Megatron-LM, TorchTitan, and JAX MaxText backends. Primus handles the full training lifecycle and includes MoE-specific optimizations like Turbo Grouped GEMM (fused kernel launches for all experts), DeepEP acceleration for expert token dispatch, and Sync-Free MoE for asynchronous pipeline execution.

The training pipeline ran in six stages:

1. Pre-training on 7.1 trillion tokens of web text, code, and mathematics
2. Mid-training on high-quality STEM and reasoning-focused data, with model souping to merge multiple variants
3. Long-context extension from 4K to 64K tokens using YaRN positional encoding and document masking
4. Supervised fine-tuning (SFT) with a feedback-driven data curation pipeline that identifies and targets model weaknesses
5. Direct Preference Optimization (DPO) with router bias updates disabled to maintain training stability
6. Reinforcement learning using the Miles framework, combining instruction-following RL with Multi-Teacher On-Policy Distillation (MOPD) to preserve reasoning performance while improving instruction adherence

The MOPD step is worth noting. A common failure mode in post-training RL is that gains on instruction-following come at the cost of regression on reasoning benchmarks. MOPD addresses this by distilling from multiple teacher models simultaneously, maintaining a broader performance profile across task types.

Benchmark Results

The base checkpoint (Instella-MoE-16B-A3B-Base) achieved an average score of 76.7 on standard pre-training benchmarks, outperforming fully open models like OLMo-3-7B and SmolLM3-3B, and remaining competitive with open-weight models like Moonlight-16B-A3B and Qwen3.5-4B.

The final "Think" checkpoint (post-RL) scored 73.2 on a combined suite of reasoning, math, coding, and chat benchmarks — surpassing OLMo-3-7B-Think (72.0), Gemma-4-E4B (70.5), and Qwen3.5-4B (69.7). These are meaningful comparisons because they're all in the same active-parameter range, making the efficiency story coherent.

It's worth being clear about what these numbers don't show: Instella-MoE is not competing with frontier closed models or even the largest open-weight releases. Its significance is in the fully open category, where the combination of competitive performance and complete reproducibility is genuinely uncommon.

What This Means for Practitioners

For researchers studying MoE training dynamics, Instella-MoE offers something most papers don't: a complete, reproducible pipeline on non-NVIDIA hardware. The availability of intermediate checkpoints means you can study how the model evolves across training stages, not just evaluate the final artifact.

For teams considering AMD hardware for training workloads, the Primus framework and the FarSkip-Collective results provide concrete data points on what's achievable with MI300X GPUs. The 39.2% TTFT improvement from FarSkip-Collective is a hardware-software co-optimization result that's directly applicable to inference deployments using SGLang.

For the broader open-source ecosystem, Instella-MoE continues a trend of fully open releases that include training infrastructure — a trend that makes the field more reproducible and lowers the barrier for groups without access to proprietary training pipelines.

The Gated MLA and FarSkip-Collective innovations are both modular enough to be adopted independently. If either proves durable across different model families, they could show up in future open-source releases from other groups — which is exactly the kind of knowledge transfer that fully open releases enable.

---

Primary source: Instella-MoE technical report on arXiv. Additional details from the AMD ROCm blog, Hugging Face model page, and Primus framework repository.