DEV Community: Prabhakar Chaudhary

RBS-Attention: How a Geometric Rescue Branch Fixes Sparse Prefill for Long-Context LLMs

Prabhakar Chaudhary — Mon, 21 Sep 2026 16:08:15 +0000

RBS-Attention: How a Geometric Rescue Branch Fixes Sparse Prefill for Long-Context LLMs

Long-context inference has become one of the defining engineering challenges of modern LLM deployment. As context windows stretch to 128K, 256K, and beyond, the prefill stage — where the model processes the entire input prompt before generating a single output token — can consume the majority of total request latency. Sparse attention is the standard tool for taming this cost, but a new paper from arXiv shows that the most common sparse selection strategy has a quiet failure mode that has been hiding in plain sight.

RBS-Attention (Radius-Bounded Sparse Prefill) identifies and fixes that failure mode with a clean geometric argument, achieving a 20.65× standalone prefill-attention speedup and a 5.97× end-to-end time-to-first-token (TTFT) speedup on H100 GPUs at 128K context — without any model training.

Why Prefill Is the Bottleneck

During inference, LLM computation splits into two phases: prefill and decode. Prefill processes the full input prompt in parallel to build the KV cache; decode generates tokens one at a time. For short prompts, prefill is fast. For long contexts — think a 100-page document, a large codebase, or a multi-turn agent session — prefill can dominate total latency, sometimes accounting for over 90% of TTFT.

The root cause is quadratic attention complexity. Every query token must attend to every key token, so doubling the context length quadruples the prefill compute. Sparse attention addresses this by selecting only the most relevant key blocks for each query, skipping the rest. The question is: how do you decide which blocks to keep?

The Standard Approach and Its Blind Spot

Most sparse-prefill systems use block-level centroid scoring. Each key block is summarized by its centroid — the average of its key vectors — and the dot product between the query and the centroid estimates how relevant the block is. Blocks with low scores get pruned; blocks with high scores get computed.

This works well when key blocks are compact and homogeneous. But real attention patterns are messier. A block might contain one highly relevant token surrounded by many irrelevant ones. In that case, the centroid is pulled away from the relevant token, the dot product score drops, and the selector discards the entire block — even though it contained exactly the information the query needed.

The RBS-Attention authors call this mean dilution, and they quantify it precisely. They find that centroid rank underestimation is strongly correlated with a block's radius — the maximum distance between any key in the block and the block's centroid. Critically, the highest-radius quintile of blocks accounts for 39.5% of the top-5% attention blocks. In other words, the blocks most likely to be incorrectly pruned are also disproportionately likely to be the most important ones.

The Fix: A Radius-Adaptive Rescue Branch

RBS-Attention addresses mean dilution by adding a second selection branch alongside the standard centroid branch. The two branches operate independently and their masks are combined.

Branch 1 — Centroid Base Branch: Standard centroid scoring, $q^\top c_b$. Handles compact, homogeneous blocks efficiently.

Branch 2 — Rescue Branch: Scores each block as $q^\top c_b + |q|_2 \cdot r_b \cdot \beta_b$, where $r_b$ is the block's radius and $\beta_b$ is a radius-adaptive rescue coefficient clamped between 0 and 1. The coefficient is computed from the distribution of block radii across prompts, layers, and attention heads, so it adapts to the specific structure of each inference call.

The geometric intuition is grounded in the Cauchy–Schwarz inequality: the maximum possible dot product between a query and any key in a block is bounded by $q^\top c_b + |q|_2 \cdot r_b$. The rescue branch uses this upper bound to identify blocks that the centroid score might be underestimating. If a block's radius is large enough that the upper bound exceeds the selection threshold, the rescue branch flags it for inclusion even if the centroid score alone would have dropped it.

The result is a selection mechanism that is conservative where it needs to be and efficient where it can be. Crucially, the method requires no model retraining and is compatible with standard block-sparse FlashAttention kernels.

Integration with vLLM and Benchmark Results

The authors evaluate RBS-Attention on the Qwen3-30B-A3B-Instruct-2507-FP8 model at 128K context length on H100 GPUs, integrating with vLLM for end-to-end measurements. The numbers are substantial:

20.65× standalone prefill-attention speedup
11.92× vLLM prefill-attention speedup
5.97× end-to-end TTFT speedup

Quality holds up well. On the dense Qwen3-32B model, RBS-Attention achieves 88.65 RULER accuracy versus 89.52 for dense attention — a 0.87-point gap for a nearly 6× latency reduction. The method is also validated on LongBench-v2, InfiniteBench, and Video-MME, confirming that the accuracy trade-off is consistent across diverse long-context tasks and across dense, MoE, and multimodal model architectures.

How This Fits Into the Broader Sparse Attention Landscape

RBS-Attention is not the first sparse-prefill method, and it is worth situating it relative to existing approaches. MInference 1.0 achieves up to 10× prefill speedup on 1M-token contexts by identifying three structural attention patterns (A-shape, Vertical-Slash, Block-Sparse) and assigning them to specific heads. FlashPrefill uses max-based dynamic thresholding to eliminate sorting overhead. DeepSeek Sparse Attention trains a learned indexer to score KV relevance.

RBS-Attention's contribution is narrower but complementary: it targets a specific failure mode that affects all centroid-based block selection methods, regardless of how the threshold is set. The rescue branch can in principle be layered on top of other sparse selection strategies, making it a potential building block rather than a standalone replacement. Its training-free property also matters practically — it works with any model using standard multi-head or grouped-query attention, covering the vast majority of production LLMs today.

Practitioner Implications

For teams running long-context inference at scale, the key takeaway is that centroid-based sparse selection has a systematic blind spot that grows worse as context length increases and as attention patterns become more heterogeneous. The 39.5% figure — nearly two-fifths of the most important blocks sitting in the highest-radius quintile — suggests this is not an edge case.

The 5.97× end-to-end TTFT improvement at 128K context is meaningful for any application where first-token latency matters: document Q&A, agentic tool-use loops, code review, and retrieval-augmented generation pipelines. At longer contexts, the gains are likely to be even larger, since the prefill fraction of total latency grows with sequence length.

The method's compatibility with vLLM and block-sparse FlashAttention means adoption does not require a custom inference stack. For teams already using these frameworks, RBS-Attention represents a relatively low-friction path to substantially better prefill performance.

Conclusion

Mean dilution is a subtle but consequential failure mode in sparse-prefill attention: the blocks most likely to be incorrectly pruned are also disproportionately likely to be the most important ones. RBS-Attention fixes this with a radius-adaptive rescue branch grounded in a clean geometric bound, achieving nearly 6× end-to-end TTFT speedup at 128K context with less than 1 point of accuracy loss. As context windows continue to grow and long-context inference becomes a standard production requirement, methods that address the specific failure modes of sparse selection — rather than just tuning sparsity ratios — will become increasingly important.

The paper is available at arXiv:2609.20971.

Edge0: How a Prerouter and SSD Offloading Let a 35B MoE Model Run in 3 GB of RAM

Prabhakar Chaudhary — Fri, 18 Sep 2026 16:04:58 +0000

Edge0: How a Prerouter and SSD Offloading Let a 35B MoE Model Run in 3 GB of RAM

Running a 35-billion-parameter Mixture-of-Experts (MoE) model on a consumer laptop sounds like a contradiction in terms. At 4-bit precision, those weights alone occupy roughly 19.5 GB — far beyond what most machines can hold in active memory. Yet Edge0, an open-source inference engine released this week, does exactly that: it runs a 35B MoE at 15–18 tokens per second while keeping peak active memory under 3 GB.

The trick is not compression alone. Edge0 combines SSD-based expert offloading with a trained routing predictor — the "prerouter" — that hides disk latency behind compute. The result is a practical system for running large sparse models on hardware that was never designed for them.

The Memory Wall in MoE Inference

Standard dense models scale memory linearly with parameter count. MoE models are different: only a small fraction of experts activate per token, so the compute cost stays manageable even as the total parameter count grows. The problem is that all those dormant experts still need to live somewhere, and "somewhere" is usually RAM.

For a 35B MoE like Qwen3.6-35B-A3B, the full weight set at int4 precision is around 19.5 GB. A Mac mini M4 Pro with 24 GB of unified memory can technically hold it, but that leaves almost nothing for the OS, the KV cache, or any other process. On a machine with 16 GB, it simply does not fit.

Edge0 reframes the problem: instead of treating RAM as the primary weight store, it treats the SSD as the primary tier and RAM as a bounded streaming pool. Expert weights are stored as int4 stacked safetensors on disk and mmap-streamed into a fixed-size active set on demand. Peak active memory tracks the working set — the experts currently in use — not the total parameter count.

The Prerouter: Hiding Disk Latency

SSD offloading introduces a new bottleneck. In standard MoE execution, the routing decision for layer N+1 depends on the output of layer N. If expert weights must be fetched from disk, the system stalls: it cannot start loading the next layer's experts until the current layer finishes computing.

Edge0 solves this with the prerouter, a small per-layer head trained to predict the routing for the next layer one token ahead of time. Because the prediction is available before it is needed, the system can initiate SSD reads for the upcoming experts while the current layer's forward pass is still running. Disk latency is hidden behind compute rather than added to it.

The prerouter's prediction also replaces the standard router's output at decode time — it is not just a prefetch hint but the actual routing decision. This means the system never waits for the standard router to finish before starting the next load. According to the paper, this overlap increases decode throughput by up to 59% compared to on-demand loading.

To avoid the quality loss that comes from routing approximation and int4 quantization, Edge0 adds Unmerged Recovery LoRA adapters. These are trained via distillation from an FP16 teacher while the quantized base weights remain frozen. Crucially, the adapters are kept as unmerged parallel delta branches rather than merged back into the quantized weights — merging would require re-quantization, which degrades quality. The inference path is int4 base + prerouter routing + LoRA delta, all applied in a single forward pass.

Fixed-Slot Double Buffering

A subtler engineering detail is how Edge0 manages the streaming pool itself. Naive implementations rebuild the layer's tensor stack at every decode step, which adds overhead. Edge0 uses fixed-slot double buffering: routing indices are mapped through a slot table, and the system performs in-place slot updates (incr_stack) rather than rebuilding from scratch. This eliminates the per-step stack-rebuild cost and keeps the memory layout stable across tokens.

Performance on Consumer Hardware

The Edge0 GitHub repository reports the following benchmarks on a Mac mini M4 Pro (24 GB unified memory), using a ~3,300-token prompt and 200 timed decode tokens:

Model tier	Decode speed	Peak active memory
edge0-35b	14.9–17.7 tok/s	~2.9 GiB
edge0-8b	23.9–25.3 tok/s	~1.0 GiB

The 35B model runs at roughly 15–18 tokens per second — usable for interactive inference — while consuming less than 3 GB of active memory. The 8B tier, based on the Ling 3.0 bailing hybrid architecture, reaches 24–25 tokens per second in just 1 GB.

Quality benchmarks using OpenCompass show that the int4 pipeline retains high fidelity relative to FP16 baselines, with an average quality drop of approximately 3.9 points for the 35B model across tasks like MMLU-Pro and HumanEval. That is a meaningful but not catastrophic gap, and one that the recovery LoRA is specifically designed to close.

What the Prerouter Gets Right Architecturally

The prerouter is worth examining as a design pattern beyond Edge0. The core insight is that routing in MoE models is highly predictable: the same input regions tend to activate the same experts across tokens. A small trained head can exploit this regularity to predict the next routing decision with enough accuracy to use the prediction as the actual decision — not just a hint.

This is different from speculative decoding, which predicts tokens and verifies them. The prerouter predicts routing indices and commits to them. There is no verification step, which means any prediction error propagates directly into the output. The paper reports that routing accuracy is high enough in practice that the quality impact is small, and the recovery LoRA absorbs most of the residual error.

The approach also generalizes naturally to other offloading scenarios. Any system that needs to prefetch data based on a future decision — whether that data lives on SSD, in a remote cache, or across a network — can benefit from a lightweight predictor that makes the decision one step early.

Practical Implications

Edge0 is currently implemented on the MLX backend for Apple Silicon, with the architecture designed to be backend-agnostic. The project is open-source under Apache 2.0, and both the 35B and 8B model tiers are available as end-to-end pipelines including the base checkpoint, LoRA adapters, and prerouter heads.

For practitioners, the immediate takeaway is that the hardware threshold for running large sparse models has dropped significantly. A machine with 24 GB of unified memory and a fast SSD can now run a 35B MoE at interactive speeds — not as a research demo but as a deployable inference setup. The SSD read bandwidth becomes the primary constraint, which means NVMe drives matter more than RAM capacity for this use case.

The broader implication is that the "memory wall" for MoE inference is not a fixed barrier. It is an engineering problem, and Edge0 demonstrates one concrete solution: treat storage as a tiered memory system, predict routing decisions early enough to hide latency, and use distillation-trained adapters to recover quality. Each of those components is independently useful, and the combination makes large sparse models accessible on hardware that most developers already own.

Primary source: Edge0 paper (arXiv 2609.18063) | GitHub repository | Model page | HuggingFace model

First Token Matters: Understanding Safety Collapse in Large Reasoning Models

Prabhakar Chaudhary — Thu, 17 Sep 2026 17:17:59 +0000

First Token Matters: Understanding Safety Collapse in Large Reasoning Models

The recent shift toward Large Reasoning Models (LRMs) — models that explicitly "think" through a Chain-of-Thought (CoT) before providing an answer — has promised a new era of complex problem-solving. However, this increased cognitive capacity comes with a technical irony: the more a model thinks, the more likely it is to bypass its own safety guardrails.

A recent research paper, "First Token Matters: Understanding Safety Collapse in Large Reasoning Models" (arXiv:2609.18471), identifies a specific, localized failure mode in these architectures. The study reveals that safety alignment in reasoning models is not failing because the models "forget" their training, but because of a transient breakdown at the very start of the generation process.

The Architecture of the Safety Gap

To understand why reasoning models struggle with safety, we first have to look at how they differ from standard Instruction-Tuned (IT) models. Standard LLMs are trained to map a prompt directly to a safe response. In contrast, reasoning models like DeepSeek-R1 are trained to generate a long intermediate reasoning trace.

This trace is often generated using Reinforcement Learning (RL) that prioritizes task success and logical coherence. Because the model's primary objective is to "solve" the prompt, the internal pressure to be helpful and logical can sometimes override the safety alignment that was baked into the base model. Researchers have noted a "safety gap" where the internal <think> process of a model often contains more harmful or biased content than the final filtered output.

Onset Refusal Collapse (ORC) Explained

The core discovery of the paper is a phenomenon the authors call Onset Refusal Collapse (ORC). Using mechanistic interpretability techniques, the researchers projected the hidden states of reasoning models onto "refusal vectors" — mathematical directions in the model's activation space that correspond to the intent to say "no" to a harmful request.

They found that when a reasoning model is presented with a harmful query, the refusal signal is actually present and strong during the initial prompt encoding phase. The model "knows" the request is harmful while it is reading it. However, the moment the model generates the very first token of its response, the refusal signal drops sharply.

This collapse happens in the transition from understanding to generating. Because reasoning models often start their CoT with affirmative or neutral tokens (like "Okay," "Let's think," or "To solve this"), the model effectively "commits" to a helpful path before its safety mechanism can assert itself. Once the first token is generated on a "helpful" trajectory, the rest of the reasoning trace follows suit, leading to an unsafe output.

The First Token as a Critical Vulnerability

The study demonstrates that the first 100 milliseconds of generation are the most critical for AI safety. If the model fails to trigger a refusal at the very first token, the probability of a successful "jailbreak" or safety violation increases by an order of magnitude.

This is particularly problematic for models that use "Distill" architectures. When reasoning capabilities are distilled from a larger model into a smaller one, the safety alignment of the smaller model is often the first thing to degrade. The distillation process prioritizes the reasoning logic, effectively crowding out the sparse refusal circuits that keep the model in check.

SafeToken: A Lightweight Intervention

Rather than suggesting a massive retraining effort or more Reinforcement Learning from Human Feedback (RLHF), which can often degrade reasoning performance, the authors propose a surgical solution called SafeToken.

SafeToken is an inference-time intervention. It works by injecting a learned, continuous "safety anchor" into the model's embedding space precisely at the onset of reasoning. Essentially, it "nudges" the model's internal state back toward the refusal vector at the exact moment the ORC is most likely to occur.

The beauty of SafeToken lies in its efficiency:

Minimal Overhead: It only requires updating a single token embedding during the inference pass.
Utility Preservation: Because the intervention is so localized, it doesn't interfere with the model's ability to solve complex math or coding problems when the query is benign.
Scalability: It can be applied to various reasoning models without needing access to the original training data.

Why Mechanistic Interpretability is the Key to Safety

The "First Token Matters" research highlights a broader trend in AI safety: the move away from black-box testing toward mechanistic understanding. By identifying exactly where and when a safety signal collapses, researchers can build targeted defenses that are both more effective and less intrusive than broad-based censorship.

As we move toward more autonomous AI agents that rely on long-horizon reasoning, understanding these transient failure modes will be essential. If we can't trust the first token of an agent's thought process, we can't trust the final action it takes in the real world.

Sources / Further Reading

Beyond Gradient Boosting: The Rise of LimiX-2 and Structured-Data Foundation Models

Prabhakar Chaudhary — Thu, 17 Sep 2026 17:13:50 +0000

Beyond Gradient Boosting: The Rise of LimiX-2 and Structured-Data Foundation Models

For decades, the dominant approach to structured data—the tabular information that powers everything from financial forecasting to healthcare diagnostics—has relied on gradient-boosted decision trees like XGBoost and LightGBM. These models are effective but require extensive manual effort: each new dataset necessitates a custom pipeline for feature engineering, hyperparameter tuning, and cross-validation. The recent introduction of LimiX-2 represents a major shift toward a foundation model approach for tabular data, enabling high-accuracy predictions across diverse tasks without dataset-specific training.

The Bottleneck in Tabular Learning

Traditional machine learning for tabular data is fundamentally fragmented. A model trained to predict credit risk cannot be easily repurposed for medical diagnosis without starting the training process from scratch. While Large Language Models (LLMs) have unified natural language processing, structured data has remained resistant to such unification due to its lack of inherent spatial or temporal locality. In a table, the order of columns is often arbitrary, and the relationships between features can be nonlinear and highly complex.

Recent attempts to build Tabular Foundation Models (TFMs) using Transformer architectures have faced two primary technical hurdles. First, numeric features are typically mapped through simple linear layers (affine scalar tokenization). This design creates a "value bottleneck" where the rank of the input matrix is severely restricted, regardless of the model's width. This leads to what researchers call "low-rank collapse," where the model's internal hidden states become highly redundant and lose the ability to distinguish between fine-grained value differences in the early layers.

Second, standard attention mechanisms often prioritize feature interactions before establishing the broader distributional statistics of the sample. In many TFM architectures, the model attempts to learn how "Feature A" relates to "Feature B" across the entire dataset before it understands the mean, variance, or scale of those features within a specific context. This suboptimal information routing forces the model to expend significant computational capacity on basic statistical normalization rather than high-level reasoning.

Architectural Innovations: RaBEL and SNF Routing

The LimiX-2 framework, particularly its lightweight variant LimiX-2M, introduces two specific architectural modifications designed to address these systemic inefficiencies. The first is the Radial-Basis Embedding Layer (RaBEL). Instead of using standard linear projections to tokenize numeric inputs, RaBEL employs a bank of localized, nonlinear Radial Basis Functions (RBFs). This approach injects nonlinearity at the very first stage of the model, significantly increasing the "effective rank" of the input representations.

By using RBFs, RaBEL allows the model to handle diverse value regimes—such as heavy-tailed distributions, local periodicity, or sharp discontinuities—with far greater precision than a linear layer could achieve. It also incorporates "exponent-gating" mechanisms to maintain numerical stability and precision across multiple orders of magnitude. This ensures that a value of 0.001 is represented with the same relative fidelity as a value of 1,000,000, a common requirement in scientific and financial datasets.

The second innovation is a reordered attention stack known as Sample-Attention → FFN → Feature-Attention (SNF) routing. In standard models, feature-level attention often precedes sample-level attention. LimiX-2 reverses this order, allowing the model to aggregate column-level statistics across different samples before mixing individual features. An intermediate Feed-Forward Network (FFN) then conditions these signals. This structural change ensures that the subsequent feature-level attention operates on richer, better-conditioned inputs that already account for the dataset's global distributional properties.

From Prediction to Mechanism Modeling

LimiX-2 moves beyond the target-centric paradigm of Prior-Data Fitted Networks (PFNs). While earlier models like TabPFN focused primarily on predicting a single target variable $p(y|x, D)$, LimiX-2 is designed as a Contextual Mechanism Network (CMN). Its objective is to learn the joint distribution of all variables and their missingness patterns simultaneously, represented as $p(x, y | D)$. This is achieved through Context-Conditional Masked Modeling (CCMM), an episodic learning formulation where the model learns to fill in "blanks" in a table based on the surrounding evidence.

This shift toward joint modeling allows LimiX-2 to function as a unified tool for multiple data science tasks. Within a single forward pass, the model can perform classification, regression, and missing-value imputation. Because it models the underlying "mechanism" of the data rather than just a mapping to a label, it is also capable of zero-shot adaptation to new tasks. For instance, a model trained on general tabular data can be used to recover "causal skeletons"—the directed graphs showing which variables influence others—without any task-specific parameter updates. The model's feature attention weights have been shown to naturally encode structural relationships that align with causal discovery benchmarks.

Benchmarking Performance and Efficiency

In empirical evaluations, LimiX-2 has demonstrated superior performance across major tabular benchmarks, including TabArena, TALENT, and BCCO. It consistently achieves higher Elo ratings than established systems like AutoGluon 1.6, XGBoost, and the original TabPFN-v2. In the TabArena benchmark, LimiX-2 achieved an Elo of 1935, ranking first across both classification and regression subsets.

Efficiency is another critical factor. The 2-million parameter LimiX-2M variant is approximately 2x faster than the 7-million parameter TabPFN-v2 while delivering better predictive accuracy. This efficiency is achieved by focusing on high-quality synthetic pre-training. LimiX models are pre-trained on millions of synthetic datasets generated using hierarchical Structural Causal Models (SCMs). This allows the model to learn the fundamental "logic" of tabular structures—such as how correlations and dependencies typically form—without being exposed to sensitive real-world data during the pre-training phase.

The Future of Tabular AI

The success of LimiX-2 suggests that the "model management tax"—the heavy overhead of maintaining, monitoring, and updating separate pipelines for every tabular task—may soon be a thing of the past. As foundation models for structured data continue to scale, we can expect them to integrate even richer semantic knowledge, such as using LLM-based encoders to handle complex text features within tables.

Furthermore, the move toward "Data Language Models" that treat tables as a native modality is likely to improve the accessibility of advanced analytics. By reducing the need for manual feature engineering and complex hyperparameter sweeps, these models allow data scientists to focus more on problem framing and domain-specific interpretation rather than low-level implementation details.

Conclusion

The transition from task-specific pipelines to unified foundation models marks a significant step in structured-data intelligence. By addressing the fundamental architectural limitations of earlier Transformers and shifting toward mechanism-oriented modeling, LimiX-2 provides a scalable, efficient alternative to traditional gradient-boosting methods. As these models become more integrated into enterprise workflows, they promise to make high-performance tabular analysis more automated, robust, and accessible.

Sources / Further Reading

Primary Source: LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
Supporting Source: LimiX-2: A Foundation Model for Structured-Data Intelligence
Supporting Source: The State of Tabular Foundation Models in 2026
Supporting Source: Demystifying Tabular Foundation Models

ZGCM-1: How a 7B Open-Weight Model Trained Itself to Beat Models 30x Its Size

Prabhakar Chaudhary — Wed, 16 Sep 2026 16:05:31 +0000

ZGCM-1: How a 7B Open-Weight Model Trained Itself to Beat Models 30x Its Size

A 7-billion-parameter model that competes with 235-billion-parameter systems on math and agentic benchmarks sounds implausible. ZGCM-1, released in September 2026 by the ZGCM team, makes a credible case for it — and the story of how it was built is as interesting as what it can do.

The ZGCM-1 paper describes a fully open foundation model: weights, intermediate checkpoints, training code, data recipes, and Weights & Biases logs are all public. But the technical choices behind it — a hybrid attention backbone, the Muon optimizer, FP8 training, and an "AI-native" R&D workflow where agent swarms managed their own data curation — make it worth examining in detail.

The Architecture: Hybrid Attention at 5:1

ZGCM-1 is a 7.39B decoder-only Transformer with 32 layers and a hidden dimension of 4,096. The headline architectural choice is a hybrid causal-attention backbone that interleaves gated sliding-window attention (SWA) with global attention at a 5:1 local-to-global ratio: 27 of the 32 layers use gated SWA with a 128-token window, while five layers use full global causal attention.

This is not a new idea — hybrid attention has appeared in models like Gemma 3 and Qwen3 — but ZGCM-1 applies it aggressively to support a 256K context window at 7B scale. The result is a 6.4x reduction in KV-cache footprint per token and a 3.94x throughput speedup at 256K context compared to standard full attention. For a model designed to do agentic search over long documents and codebases, that efficiency matters directly.

The model also uses Grouped-Query Attention (GQA), RMSNorm, SwiGLU activations, and Rotary Position Embeddings (RoPE) with QK normalization and Partial RoPE (rotary fraction of 0.33) to stabilize training at long contexts.

Training: Muon, FP8, and a 4.2x Speedup

The training pipeline is where ZGCM-1 diverges most sharply from standard practice. Three choices combine to deliver a ~4.2x pre-training time-to-loss speedup over an AdamW/BF16 baseline:

Muon optimizer. Muon is a second-order-inspired optimizer that applies Nesterov momentum in the gradient space and then orthogonalizes the update using Newton-Schulz iterations. It has shown faster convergence than AdamW on several recent open-source training runs, and ZGCM-1 is one of the larger-scale validations of that claim.

Hybrid FP8 precision. Rather than training entirely in BF16, ZGCM-1 uses FP8 for the bulk of matrix multiplications while keeping sensitive operations (layer norms, attention softmax) in higher precision. This reduces memory bandwidth pressure and enables larger effective batch sizes.

TWEO regularization. TWEO (Trainable Weight-based Outlier) is a regularization technique that penalizes weight outliers during training, reducing the activation spikes that typically force practitioners to keep certain layers in BF16 even in otherwise FP8 runs.

The training curriculum itself is progressive: the model first trains at 16K context, then extends to 64K, then to 256K. During the mid-training phase, interaction traces from tool-use and search tasks are reformulated as Markov Decision Processes (MDPs) to provide dense, step-level supervision — a technique borrowed from offline RL that gives the model richer signal than next-token prediction alone.

Post-training combines supervised fine-tuning (SFT) with GRPO (Group Relative Policy Optimization). The SFT stage uses aggressive quality filtering, pruning roughly 50% of raw candidate data through a tiered scoring process before any gradient updates.

The AI4AI Angle: Agent Swarms Building the Model

The most unusual aspect of ZGCM-1's development is what the team calls "AI-native R&D." Rather than relying entirely on human engineers for data curation, cluster diagnostics, and experiment management, the ZGCM team deployed agent swarms to handle these tasks autonomously.

Concretely, agents were responsible for:

Filtering and scoring training data at scale
Generating synthetic math and agentic-search examples
Monitoring training runs and flagging anomalies
Running ablations on hyperparameter choices

The HuggingFace community noted that this makes ZGCM-1 a model that "partially built itself" — a meaningful step toward the kind of recursive self-improvement that researchers have been discussing theoretically for years. It is worth being precise about what this means: the agents did not modify the model architecture or training objective autonomously. They operated within a fixed pipeline designed by human researchers. But the scale at which they replaced human labor in the data and infrastructure loop is notable.

Benchmark Results: Punching Well Above Weight

ZGCM-1 ranks first among 7B–8B models across 14 reasoning benchmarks. The headline numbers:

Mathematical reasoning:

MATH-500: 97.1%
AIME 2026: 75.0%
HMMT 2025: 70.4%

Agentic search:

WebWalkerQA: 63.1%
Binary Function Search: 62.0% (vs. Qwen3-8B at 12%)
BrowseComp: 19.4%

The Binary Function Search result is particularly striking. At 62%, ZGCM-1 outperforms Qwen3-8B by 50 percentage points on a task that requires navigating real codebases to identify function implementations — the kind of structured search that agentic coding tools rely on. The gap suggests that the mid-training MDP reformulation and the agentic-search data pipeline are doing real work, not just inflating benchmark scores.

On general language tasks, the model matches Qwen3-8B, which is a reasonable baseline for a model of this size. The ZGCM team is not claiming general-purpose superiority — the focus is explicitly on math and agentic search, and the benchmark profile reflects that.

What "Fully Open" Actually Means Here

Many models claim openness while releasing only weights. ZGCM-1 goes further: the release includes per-stage data recipes, training code, intermediate checkpoints at multiple stages of the curriculum, and W&B logs showing the full training trajectory. This level of transparency is rare even among genuinely open-weight releases.

For practitioners, this means ZGCM-1 is not just a model to deploy — it is a reproducible training recipe. Teams working on domain-specific math or agentic-search applications can use the released data recipes and checkpoints as a starting point for continued pre-training or fine-tuning, with full visibility into what the base model has already seen.

Practitioner Implications

ZGCM-1 is most directly useful for teams building:

Math-heavy agentic pipelines where a small, fast model needs to handle symbolic reasoning and tool use without the latency of a 70B+ system
Code search and analysis agents where the Binary Function Search results suggest genuine capability at navigating real codebases
Long-context document processing where the 256K window and 3.94x throughput advantage over full-attention models translate to real serving cost reductions

The Muon + FP8 + TWEO training stack is also worth attention for teams training their own models. The 4.2x speedup claim is significant if it holds at larger scales, and the fully open release makes it possible to verify the claim independently.

The model weights and training code are available on HuggingFace, and the full technical report is on arXiv. The GitHub repository includes training scripts, data recipes, and W&B logs for full reproducibility.

OpenAgentFlow: How a Control-Plane Architecture Brings System-Wide Safety to Multi-Agent AI

Prabhakar Chaudhary — Mon, 14 Sep 2026 16:10:02 +0000

OpenAgentFlow: How a Control-Plane Architecture Brings System-Wide Safety to Multi-Agent AI

As AI agents move from isolated assistants into interconnected fleets that read emails, call APIs, browse the web, and modify databases, the safety problem changes shape. You can no longer protect a system by guarding a single model or a single tool call. A new paper — OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleets — proposes a concrete architectural answer, borrowing ideas from network engineering to govern agent actions at the system level rather than the model level.

The Problem: Composed Risk in Multi-Agent Systems

Modern agentic deployments rarely involve a single agent acting alone. A typical enterprise workflow might chain a planning agent, a web-browsing agent, a code-execution agent, and a mail-sending agent — each with its own runtime, its own tool set, and its own local safety checks. The trouble is that individually safe actions can combine into unsafe outcomes.

Consider a scenario where an agent reads an email containing a hidden instruction (indirect prompt injection), then calls a payroll API, then sends a summary to an external address. Each step might pass a local safety check. The sequence, viewed as a whole, is a data exfiltration attack.

This is what the OpenAgentFlow authors call composed risk: the danger that emerges from the interaction of actions across a session, not from any single action in isolation. Existing defenses — prompt-level filters, per-tool guardrails, agent-local runtime checks — are not designed to see across execution boundaries. They govern individual actions, not flows.

Research on indirect prompt injection has documented this attack vector: malicious instructions hidden in data that agents consume can redirect agent behavior without touching the system prompt. As agents gain more autonomy, the attack surface for composed flows grows proportionally.

The Architecture: Two Planes, One Enforcement Point

OpenAgentFlow draws its design from network control-plane architectures like OpenFlow and Ethane, which separated policy management from packet forwarding. The same split applies here.

The Action Plane sits on every execution path — GUI interactions, API calls, tool invocations, LLM-generated actions — and normalizes them into a unified AgentEvent stream. This normalization is the key move: regardless of whether an agent is clicking a button in a browser, calling a REST endpoint, or invoking a Python function, the resulting event looks the same to the enforcement layer.

The Control Plane lives outside the agents entirely. It holds updatable policies (called FlowRules), session state, audit evidence, and provenance records. Because it is decoupled from the agents, administrators can install or update safety rules post-deployment without touching agent prompts, model weights, or execution code.

Between the two planes sits the Policy Enforcement Point (PEP), positioned at the "action-commit boundary" — the moment immediately before an action alters user or enterprise state. The PEP runs a four-tier evaluation pipeline:

Tier	Role	What It Does
T1	Structured Rules	Explicit policy and scope checks; can terminate immediately
T2	Payload/Provenance	Pattern analysis and source-sink checks on the pending action
T3	Semantic Assessment	Local semantic check with escalation capability
T4	Final Adjudication	Last decision point before the action commits

The tiered design matters for latency. Most actions are resolved at T1 or T2 without reaching the more expensive semantic evaluation at T3. Only ambiguous or high-risk actions escalate to the full pipeline.

Why Session-Level Provenance Changes the Game

The most technically interesting aspect of OpenAgentFlow is its use of session-level provenance. Rather than evaluating each action in isolation, the PEP has access to the accumulated state of the entire session: which agents have acted, what data they have read, what external sources they have contacted, and what actions they have already committed.

This lets the system detect composed attacks that would be invisible to per-action filters. An agent reading an untrusted email is not inherently dangerous. An agent reading an untrusted email and then, three steps later, calling an external API with a payload derived from that email — that is a pattern the session-level view can catch.

The provenance model also enables policy evolution without agent modification. When a new attack pattern is discovered, operators push a new FlowRule to the Control Plane. The rule takes effect immediately across the entire fleet, without re-prompting or retraining any agent. This is a meaningful operational advantage in environments where agent code is managed by multiple teams or third-party vendors.

Evaluation Results

The authors evaluated OpenAgentFlow on three test suites, with full results reported in the paper and the TS-Bench benchmark:

Controlled suite (300 cases): 94.00% accuracy, 95.35% attack-block rate
AgentDojo-Traj split of TS-Bench (1,220 cases): 97.62% accuracy, 96.59% unsafe-action recall, 1.96% safe false-intervention rate
Threat suite (200 cases): 96.08% attack-block rate

The false-intervention rate — the fraction of safe actions incorrectly blocked — is worth watching. At 1.96% on the TS-Bench split, the system is precise enough for production without generating excessive friction for legitimate workflows. The authors also validated the architecture on live Android execution paths, governing actions across GUI, API, and LLM-planned channels simultaneously.

What This Means for Practitioners

If you are building or operating multi-agent systems today, OpenAgentFlow surfaces a few concrete design principles worth internalizing:

Treat the action-commit boundary as the governance nexus. Prompt-level filters and model-level refusals are useful, but they operate before the agent has decided what to do. The action-commit boundary — the moment before a state change is written — is where you have the most information and the most leverage.

Normalize actions across execution paths. The AgentEvent abstraction is simple but powerful. If your safety layer only understands API calls, it is blind to GUI actions. A unified event stream is a prerequisite for consistent enforcement.

Separate policy from execution. Baking safety rules into agent prompts or model fine-tunes makes them hard to update and easy to bypass. A control plane that holds policies outside the agents lets you respond to new threats without touching the agents themselves.

Track session state, not just individual actions. The composed-risk problem is fundamentally temporal. Safety systems that evaluate actions in isolation will always be vulnerable to multi-step attacks that distribute risk across a session.

Open Questions

OpenAgentFlow is a research prototype, and several practical questions remain open. The paper does not address how the architecture handles very high-throughput agent fleets where the PEP could become a bottleneck. The four-tier pipeline adds latency, and the tradeoff between enforcement depth and response time will vary by use case. There is also the question of adversarial adaptation: attackers will probe for gaps in the AgentEvent normalization layer or attempt to manipulate session-state provenance directly.

Still, the core insight — that multi-agent safety requires a system-level governance layer, not just per-agent guardrails — is well-argued and practically grounded. As agent fleets grow in complexity, architectures like OpenAgentFlow will likely become a standard part of the deployment stack.

The paper is available at arXiv:2609.00015, with evaluation details in the supplementary material. For broader context, the AgentDojo benchmark paper covers how prompt injection attacks are evaluated in multi-agent settings.

DeepSeek-V4.1-Flash: How a Causal Encoder-Decoder Architecture Cuts Agent Memory Costs by 75%

Prabhakar Chaudhary — Fri, 11 Sep 2026 16:08:12 +0000

DeepSeek-V4.1-Flash: How a Causal Encoder-Decoder Architecture Cuts Agent Memory Costs by 75%

Released on September 10, 2026, DeepSeek-V4.1-Flash is a 552-billion-parameter multimodal Mixture-of-Experts model that takes a fundamentally different approach to the KV cache problem. Rather than optimizing attention computation — the path DeepSeek-V4 took — V4.1-Flash attacks the memory footprint of long-running AI agents directly. The result is a global KV cache of 890 bytes per token, roughly one-quarter of what V4-Flash required, and one-eighth the persistent SSD storage.

DeepSeek describes V4.1-Flash as the first model in a new architecture lineage: the Causal Encoder-Decoder (CED) design, the Compressed Sparse Attention 2 (CSA2) cache-sharing system, FP4 cache quantization, and the elimination of persistent sliding-window attention storage are four distinct innovations working in combination.

Why Agent Memory Became the Binding Constraint

Every time a large language model processes a prompt, it generates key-value pairs for each token and stores them in a KV cache. For a basic chatbot, this overhead is manageable. For a long-running agent that spends hours reading tool outputs, inspecting code, and iterating on plans, the cache grows continuously — and at sufficient context length, it outgrows the model weights themselves. For enterprise teams running agents at scale, KV cache management becomes the dominant operational cost. The technical report on Hugging Face frames the entire architecture around this goal: "Pushing the Limits of KV Cache Compression."

The Causal Encoder-Decoder Split

The most structurally significant change in V4.1-Flash is the Causal Encoder-Decoder (CED) architecture. The model's 40 Transformer layers are divided into two asymmetric halves: a 20-layer causal encoder and a 20-layer decoder.

During the prefill phase — when the model reads a long input — only the 20 encoder layers process the full context. The decoder's global KV cache is then synthesized directly from the encoder's final hidden states, rather than being recomputed by each decoder layer independently. The practical consequence: the model activates only 8 billion parameters per token during input processing, compared to 16 billion during text generation. For input-heavy agentic workloads, this roughly halves the most computationally expensive phase. The design draws on the YOCO ("You Only Cache Once") research approach from 2024, which demonstrated that global KV caches could be shared across decoder layers rather than recomputed independently.

Compressed Sparse Attention 2: Three Modes of Cache Sharing

DeepSeek's prior V4 architecture used a static attention compression scheme. V4.1-Flash introduces CSA2, a cache-sharing system in which every Transformer layer is assigned one of three operating modes:

Full: Compute a fresh KV cache and select the most relevant positions (top 512 per layer).
Reindex: Share an existing KV cache from an earlier layer, but independently choose which entries to attend to using the current layer's own queries.
Reuse: Share both the KV cache and the prior layer's attention selections outright — effectively saying "those notes look good, I'll use exactly those."

In Reuse mode, a decoder layer borrows the work of a shallower layer entirely, avoiding redundant KV storage and computation across the network's depth. A Hierarchical Sparse Indexer limits the candidate pool for attention indexing to 16,384 positions, keeping indexing costs stable regardless of context length.

FP4 Cache Quantization and SWA Bounded Replay

Two additional techniques complete the memory reduction:

FP4 quantization: V4.1-Flash stores its main global KV cache in 4-bit floating-point (E2M1) format rather than the FP8 used by V4, roughly halving the memory required per cache entry. DeepSeek applied quantization-aware training from the start, preventing the accuracy degradation that typically accompanies aggressive post-hoc quantization. The local sliding-window attention cache remains at FP8.

SWA Bounded Replay: Sliding-window attention state covers only the most recent tokens and becomes irrelevant quickly — yet in V4's prior architecture, SWA KV occupied nearly half of persistent SSD cache capacity unnecessarily. V4.1-Flash stops persisting SWA state to SSD entirely. A temporary DRAM pool holds SWA state for the brief window it is useful; when reconstruction is needed after a session pause, SWA Bounded Replay replays only the last 128 tokens. The practical effect: persistent SSD storage drops to one-eighth of V4-Flash's requirement.

Together, these four techniques reduce the global KV cache footprint to 890 bytes per token. A 1M-token context requires approximately 0.87 GiB of global KV cache — making million-token agent sessions economically viable at scale.

The Engram Module: Offloading Rote Recall

V4.1-Flash also integrates a 196-billion-parameter conditional memory module called Engram. Rather than forcing the Transformer backbone to memorize patterns and facts, Engram uses n-gram lookup tables (up to four tokens, roughly 16 million entries each) to handle rote recall. These tables reside in host memory and are accessed at specific layers (layers 1 and 14) during inference. By offloading pattern lookup to Engram, the backbone can dedicate its expensive computation to reasoning.

Benchmark Results and the Harness Problem

At maximum reasoning effort, V4.1-Flash posts competitive numbers on agentic benchmarks: 90.6% on Terminal-Bench 2.1, 74.2% on DeepSWE v1.1, and 88.1% on CyberGym — leading DeepSeek's own V4-Pro on all 12 agentic benchmarks where both models were evaluated, per the Progressive Robot analysis. On knowledge-intensive tasks, V4-Pro retains meaningful leads: SimpleQA-Verified (55.2 vs. 42.3), LongBench-V2 (51.5 vs. 45.2).

The most important finding in the technical report is not a benchmark score — it is the harness-variance data. The same V4.1-Flash checkpoint scored between 65.5% and 74.2% on DeepSWE v1.1 depending solely on which agent scaffold wrapped it. That 8.7-percentage-point range was produced by changing the evaluation framework, not the model. Single-percentage-point differences between models on agentic benchmarks are within the noise introduced by harness selection alone.

Pricing and the September 14 Cutover

V4.1-Flash introduces tiered peak/off-peak pricing. At off-peak rates: input cache hits at $0.003 per million tokens, cache misses at $0.15 per million tokens, and output at $0.60 per million tokens — more than 3x cheaper than V4-Pro's off-peak output rate ($1.98 per million). Starting September 14, 2026, all API traffic directed to deepseek-v4-pro will be automatically rerouted to V4.1-Flash at V4.1-Flash rates. The model is released under the MIT license, with weights available on Hugging Face as a 511 GB checkpoint.

Practitioner Implications

The CED + CSA2 + FP4 + SWA Bounded Replay combination is a coherent architectural response to a real deployment problem. For teams running production agent workloads where KV cache costs dominate the inference bill, V4.1-Flash represents a structural improvement, not a marginal one. What it does not change is the competitive ceiling on knowledge-intensive tasks — the gaps on SimpleQA-Verified and LongBench-V2 are real constraints for workloads requiring broad factual recall.

For teams evaluating self-hosting: the checkpoint requires at least 614 GB of accelerator memory, making it a multi-GPU infrastructure decision. Teams in regulated industries should note that the hosted API routes prompts to DeepSeek's servers in China, subject to China's National Intelligence Law — a fixed legal condition that does not change with model version or architectural improvements. Self-hosting the MIT-licensed weights on non-Chinese infrastructure eliminates the data-routing concern.

The CED design, CSA2 cache-sharing modes, and SWA Bounded Replay are techniques worth studying regardless of deployment context. As the field continues to treat agent memory economics as a first-class engineering problem, these patterns will likely appear in other models.

Consistency in the Latent Space: Inside the Semigroup-JEPA Architecture

Prabhakar Chaudhary — Thu, 10 Sep 2026 17:19:17 +0000

Consistency in the Latent Space: Inside the Semigroup-JEPA Architecture

The quest to build autonomous agents that understand the physical world has long faced a fundamental obstacle: the "pixel problem." When a machine learning model tries to predict the future by generating every pixel of a video frame, it wastes enormous computational resources on irrelevant details like flickering lights or shifting shadows. This often comes at the expense of understanding the core physical laws—gravity, momentum, and collision—that actually govern the scene.

A new research paper titled Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization addresses this gap. By extending the Joint-Embedding Predictive Architecture (JEPA), the authors introduce a framework that doesn't just predict what a sequence looks like, but learns the underlying "latent dynamics" that allow it to generalize to entirely new physical environments without further training.

The Shift from Pixels to Embeddings

To understand why Semigroup-JEPA (SG-JEPA) matters, we first need to look at the architecture it builds upon. Traditional generative models, such as those used in video generation, are often "auto-regressive" in pixel space. They predict the next frame, then use that prediction to generate the one after it. The problem is that errors accumulate quickly. If a model gets a single pixel wrong, that error propagates and magnifies, leading to "hallucinations" where objects morph or disappear.

Yann LeCun and the team at Meta proposed a different path with I-JEPA. Instead of predicting pixels, a JEPA model predicts the embeddings (mathematical representations) of missing or future parts of an image or video. By working in this abstract latent space, the model can ignore noise and focus on high-level semantic features.

However, while original JEPA models were excellent at understanding "what" is in an image, their ability to model "how" things move—specifically their ability to respect physical laws—remained limited. They could recognize a ball, but they couldn't necessarily predict its trajectory if the gravity of the environment changed.

Introducing Semigroup-JEPA

SG-JEPA, as detailed by Liu et al. in their recent work, introduces two key innovations to the JEPA framework to solve the physics problem.

First, it incorporates action-conditioning with physics parameters. In a standard world model, the agent knows what action it took (e.g., "push the block"). In SG-JEPA, the temporal predictor is also supplied with parameters governing the physics of the environment, such as the gravitational constant or friction coefficients. This allows the model to learn a mapping between actions and outcomes that is modulated by the physical state.

Second, the architecture utilizes multi-step latent rollouts during the training phase. Instead of just predicting the very next latent state, the model is trained to project several steps into the future—a "rollout." By back-propagating the loss through these multiple steps, the model is forced to maintain consistency over a long horizon.

This approach is highly related to earlier efforts in World Modeling, where agents "dream" of future states to plan their actions. However, by keeping these dreams in the latent space and enforcing semigroup consistency (the idea that performing two sequential actions should lead to the same state as a single "combined" action), SG-JEPA achieves a much higher level of stability.

Testing Zero-Shot Physics Generalization

The most impressive aspect of the SG-JEPA research is its performance in "zero-shot" scenarios. The researchers designed a series of dynamical tasks under varying gravitational fields. They trained the model in specific environments and then tested its ability to predict dynamics in fields it had never encountered—ranging from the weightlessness of deep space to the crushing gravity of a massive planet.

The results were striking. When compared against InternVideo2 and other high-capacity video foundation models, SG-JEPA demonstrated:

A 2x reduction in open-loop prediction error on 2D datasets.
A 2.5x increase in control success rate when used to guide 3D robotic tasks.

Crucially, the model didn't just memorize trajectories. It developed a generalized understanding of how gravity affects motion. In weak fields, it predicted "floating" behavior; in strong fields, it predicted rapid, energetic bouncing. The model could adjust its internal "physics engine" based provided parameters to generate realistic dynamics for the specific environment.

The Encoder Insight: What to Keep and What to Forget

One of the most profound findings in the paper relates to the role of the encoder. Initially, one might assume the performance gains come from the predictor getting "smarter" at physics. However, the authors' analysis suggests something different.

By using a linear feature model to separate errors, they found that back-propagating the multi-step rollout loss actually trains the encoder to be more selective. The encoder learns to identify and preserve features that the predictor is capable of carrying forward over time. It essentially learns to "forget" features that represent transient noise or unpredictable fluctuations, focusing purely on the physical constants and state variables that matter for long-term prediction.

This is a significant shift in how we think about representation learning. We are no longer just asking encoders to "describe the image"; we are asking them to "find the parts of the image that stay consistent under the laws of physics."

Implications for the Future of Robotics

For developers and engineers working in robotics and autonomous systems, SG-JEPA represents a practical step toward more reliable agents. Current robotic systems often struggle when moved from a controlled laboratory to the messy, unpredictable real world. If an agent can learn a latent dynamics model that generalizes zero-shot to different friction levels or uneven terrain, the cost and risk of deployment drop significantly.

Furthermore, the efficiency of working in latent space rather than pixel space cannot be overstated. By bypassing the need for high-fidelity video generation, these models can run on more modest hardware while providing the high-frequency feedback needed for real-time control.

As we move toward "World Models" that power everything from self-driving cars to household assistants, the principles laid out in the Semigroup-JEPA architecture—consistency, action-conditioning, and latent rollouts—will likely become standard components of the AI stack.

Conclusion

The transition from visual imitation to physical understanding is one of the most important transitions in AI research today. Semigroup-JEPA shows that we don't need to simulate every atom or render every pixel to understand the world. By focusing on the latent "semigroup" property of actions and outcomes, we can build models that are not only more accurate but more adaptable to the diverse physical environments of our reality.

The code and project page for SG-JEPA are currently being released to the community, offering a new baseline for those interested in the intersection of deep learning and classical physics.

AMD Instella-MoE: How a Fully Open 16B MoE Model Proves You Don't Need NVIDIA to Train at Scale

Prabhakar Chaudhary — Wed, 09 Sep 2026 16:04:55 +0000

AMD Instella-MoE: How a Fully Open 16B MoE Model Proves You Don't Need NVIDIA to Train at Scale

When AMD released Instella-MoE-16B-A3B in September 2026, it did something that most large model releases don't: it published everything. Not just the weights, but the training code, data mixtures, configuration files, Docker environments, and checkpoints from every stage of the pipeline. That level of transparency is rare even among models that call themselves "open source." It also happens to be trained entirely on AMD hardware — a deliberate proof-of-concept that the ROCm ecosystem can handle frontier-scale MoE training from scratch.

This post walks through what Instella-MoE actually does differently, why its two architectural innovations matter, and what the fully open release means for researchers who want to reproduce or build on it.

What "Fully Open" Actually Means Here

The distinction between "open weight" and "fully open" has become increasingly important. Many models release weights under permissive licenses but withhold training recipes, data pipelines, or intermediate checkpoints — making true reproduction impossible.

Instella-MoE takes a different approach. AMD released the complete training recipe via the AMD-AGI GitHub repository and Hugging Face, including:

Model checkpoints at every major training stage (pre-training, mid-training, long-context extension, SFT, DPO, and RL)
Training configurations and launch scripts for each stage
Data preparation tools and data mixture descriptions
Docker images for both training and inference environments
The Primus training framework and Miles RL framework used throughout

The license is ResearchRAIL, which restricts commercial use but allows academic and research applications. For practitioners who want to study how a competitive MoE is actually built — not just how it performs — this is a meaningful step beyond what most labs provide.

Architecture: Two Innovations Worth Understanding

Instella-MoE is a decoder-only MoE with 27 layers (26 of which are sparsely activated MoE layers), a hidden size of 2048, 64 routed experts plus 2 shared experts, and 6 experts activated per token. Total parameters: 16B. Active parameters per token: 2.8B. That's a familiar efficiency ratio — similar to DeepSeek-V2's design philosophy of keeping active compute low while maintaining a large parameter pool.

What distinguishes Instella-MoE are two specific architectural choices.

Gated Multi-head Latent Attention (Gated MLA)

Standard Multi-head Latent Attention (MLA), introduced by DeepSeek, compresses the KV cache by projecting keys and values into a low-dimensional latent space. This reduces memory bandwidth during inference significantly. Instella-MoE extends this with a lightweight, input-conditioned sigmoid gate applied to the attention output channels.

The gate learns to selectively attenuate low-utility attention responses — essentially adding a learned filter that decides which attention outputs are worth passing through at full strength. The result is improved model expressivity without a significant increase in parameter count or compute. It's a targeted modification rather than a wholesale architectural change, which makes it easier to analyze and potentially adopt in other models.

FarSkip-Collective

Expert parallelism in MoE models introduces a communication bottleneck: tokens must be routed to the correct expert, which may live on a different device, and the results must be gathered back. This all-to-all communication typically creates "bubbles" — idle periods where compute units wait for data to arrive.

FarSkip-Collective addresses this by overlapping expert-parallel communication with independent computation. Rather than waiting for all expert outputs before proceeding, the system schedules communication to run concurrently with computations that don't depend on those outputs. According to AMD's technical report, this yields a 12.7% improvement in pre-training throughput and up to a 39.2% reduction in time-to-first-token (TTFT) during inference when using SGLang with expert parallelism.

That TTFT improvement is particularly relevant for deployment: faster first-token latency directly affects user-perceived responsiveness in interactive applications.

The Training Pipeline

Instella-MoE was trained using AMD's Primus framework, an open-source training system built on top of Megatron-LM, TorchTitan, and JAX MaxText backends. Primus handles the full training lifecycle and includes MoE-specific optimizations like Turbo Grouped GEMM (fused kernel launches for all experts), DeepEP acceleration for expert token dispatch, and Sync-Free MoE for asynchronous pipeline execution.

The training pipeline ran in six stages:

Pre-training on 7.1 trillion tokens of web text, code, and mathematics
Mid-training on high-quality STEM and reasoning-focused data, with model souping to merge multiple variants
Long-context extension from 4K to 64K tokens using YaRN positional encoding and document masking
Supervised fine-tuning (SFT) with a feedback-driven data curation pipeline that identifies and targets model weaknesses
Direct Preference Optimization (DPO) with router bias updates disabled to maintain training stability
Reinforcement learning using the Miles framework, combining instruction-following RL with Multi-Teacher On-Policy Distillation (MOPD) to preserve reasoning performance while improving instruction adherence

The MOPD step is worth noting. A common failure mode in post-training RL is that gains on instruction-following come at the cost of regression on reasoning benchmarks. MOPD addresses this by distilling from multiple teacher models simultaneously, maintaining a broader performance profile across task types.

Benchmark Results

The base checkpoint (Instella-MoE-16B-A3B-Base) achieved an average score of 76.7 on standard pre-training benchmarks, outperforming fully open models like OLMo-3-7B and SmolLM3-3B, and remaining competitive with open-weight models like Moonlight-16B-A3B and Qwen3.5-4B.

The final "Think" checkpoint (post-RL) scored 73.2 on a combined suite of reasoning, math, coding, and chat benchmarks — surpassing OLMo-3-7B-Think (72.0), Gemma-4-E4B (70.5), and Qwen3.5-4B (69.7). These are meaningful comparisons because they're all in the same active-parameter range, making the efficiency story coherent.

It's worth being clear about what these numbers don't show: Instella-MoE is not competing with frontier closed models or even the largest open-weight releases. Its significance is in the fully open category, where the combination of competitive performance and complete reproducibility is genuinely uncommon.

What This Means for Practitioners

For researchers studying MoE training dynamics, Instella-MoE offers something most papers don't: a complete, reproducible pipeline on non-NVIDIA hardware. The availability of intermediate checkpoints means you can study how the model evolves across training stages, not just evaluate the final artifact.

For teams considering AMD hardware for training workloads, the Primus framework and the FarSkip-Collective results provide concrete data points on what's achievable with MI300X GPUs. The 39.2% TTFT improvement from FarSkip-Collective is a hardware-software co-optimization result that's directly applicable to inference deployments using SGLang.

For the broader open-source ecosystem, Instella-MoE continues a trend of fully open releases that include training infrastructure — a trend that makes the field more reproducible and lowers the barrier for groups without access to proprietary training pipelines.

The Gated MLA and FarSkip-Collective innovations are both modular enough to be adopted independently. If either proves durable across different model families, they could show up in future open-source releases from other groups — which is exactly the kind of knowledge transfer that fully open releases enable.

Primary source: Instella-MoE technical report on arXiv. Additional details from the AMD ROCm blog, Hugging Face model page, and Primus framework repository.

Iris: How SFT-RL Climbing Trains Open-Weight Agents to Search the Web Like a Researcher

Prabhakar Chaudhary — Mon, 07 Sep 2026 16:12:54 +0000

Iris: How SFT-RL Climbing Trains Open-Weight Agents to Search the Web Like a Researcher

Web search sounds simple until you try to automate it for hard questions. Ask a language model to find the founding date of a company, and it will usually succeed. Ask it to trace a multi-hop chain — "which researcher co-authored a 2019 paper with the person who later led the team that built X?" — and most agents fall apart within a few steps. They lose track of what they have already ruled out, exhaust their context window, or stop searching when the first plausible-sounding result appears.

Iris, a new open-weight search agent system from AllSpark Research, takes a systematic approach to this problem. The paper introduces two models — Iris-mini (35B total parameters, 3B active) and Iris-pro (397B total parameters, 17B active) — both post-trained from the Qwen3.5/3.6 MoE series. On benchmarks like BrowseComp and Humanity's Last Exam (HLE), they reach state-of-the-art results among open-source search agents. But the more interesting contribution is the training recipe and the inference-time insight that makes those numbers possible.

The Problem with Existing Search Agents

Most search agent research focuses on the policy — the model that decides what to search and how to interpret results. The execution harness (context management, episode restarting, trajectory filtering) is treated as an implementation detail. Iris argues this is backwards: for long-horizon search, the harness can matter as much as the policy itself.

The core challenge is context saturation. A multi-hop search task might require 20 or 30 tool calls before an answer emerges. Each call adds retrieved text to the context window. By the time the agent is halfway through a hard question, it may have consumed most of its 256K-token budget on intermediate evidence that is no longer relevant. Without a strategy for managing this, even a strong policy will fail — not because it reasoned poorly, but because it ran out of space to reason.

Building the Training Data from Web Graphs

Before training, the team needed hard, verifiable search tasks. Rather than hand-labeling questions, they built a fully automated pipeline that reverse-constructs tasks from the hyperlink structure of a web corpus.

The process works in three stages. First, the system creates a local subgraph from a seed page and its outbound links, then distills a compact entity graph. Second, it generates multi-hop questions where the answer requires traversing at least N hops through the entity graph. Third — and this is the key step — it applies an "anchor abstraction" operator that rewrites non-answer entities into descriptive references rather than proper names. This prevents the model from solving questions through simple string matching rather than genuine multi-step reasoning.

Tasks are only admitted to the training set if they pass dual-criteria verification: a reference model must fail them closed-book (confirming they are genuinely hard) but succeed when given the relevant evidence (confirming they are solvable). This filter keeps the training distribution challenging without making it impossible.

SFT-RL Climbing: An Iterative Training Loop

The training recipe, which the authors call SFT-RL climbing, alternates between supervised fine-tuning and reinforcement learning to continuously raise the difficulty ceiling.

The SFT phase trains on trajectories from a strong teacher model, filtered in two tiers. Coarse filtering removes trajectories that are incorrect, degenerate, or too shallow in search depth. Fine filtering uses an LLM judge to label individual turns, masking up to 10% of turns where the signal is noisy. This turn-level masking gives the model cleaner gradient signal than simply accepting or rejecting entire trajectories.

The RL phase optimizes the policy against live search environments using group-relative policy gradient. To handle long-horizon tasks, the system uses partial rollouts: when a trajectory exceeds the context limit, it is interrupted at the request level and the model resumes from its committed prefix rather than starting over. This allows the RL phase to explore trajectories that would otherwise be truncated.

The climbing part comes from feeding successful RL rollouts back into the SFT dataset. As the policy improves, the difficulty band shifts upward automatically — the model is always trained near the edge of its current capability, creating a self-reinforcing improvement cycle without manual curriculum design.

Inference-Time Context Management Is Not Optional

One of the paper's clearest findings is that inference-time context management (CM) is not a minor engineering concern — it is a first-class component of the search system.

The authors evaluate Iris under four settings: no context management, "retry" (restarting episodes that fail to produce a parseable answer while carrying forward a summary of ruled-out paths), "discard-all" (clearing tool history and restarting from the original question when context limits are reached), and a combination of both.

The performance differences are substantial. On BrowseComp, Iris-pro scores 88.6 with discard-all enabled, rising to 90.3 when combined with retry. Without any context management, the same model scores noticeably lower. For Iris-mini, the gap is even larger — smaller models with tighter effective context capacity benefit most from aggressive context clearing.

The practical implication is that evaluating search agents without specifying their context management strategy produces numbers that are not comparable across systems. A weaker policy with good context management can outperform a stronger policy running without it.

Benchmark Results

Evaluated on four benchmarks — BrowseComp, BrowseComp-ZH, DeepSearchQA, and the text-only subset of Humanity's Last Exam — Iris-mini and Iris-pro reach the following scores with the discard-all strategy:

Model	BrowseComp	BrowseComp-ZH	DeepSearchQA	HLE
Iris-mini (35B)	82.2	84.8	86.9	52.3
Iris-pro (397B)	88.6	85.1	92.9	56.4

These place both models at the top of open-source search agent rankings as of their September 2026 release. The models use a single ReAct-based architecture without sub-agents or test-time verification ensembles, keeping the inference setup straightforward to reproduce.

What This Means for Practitioners

Three takeaways stand out for anyone building or evaluating search-augmented LLM systems.

Context management deserves explicit design. Treating context handling as an afterthought likely leaves significant performance on the table. The discard-all strategy — clearing tool history when the context fills up — is simple to implement and consistently improves results.

SFT-RL climbing generalizes beyond search. The procedure — where RL discoveries feed back into SFT data and the difficulty band shifts upward automatically — is a general recipe for long-horizon agent training without manual curriculum design.

Open-weight models can compete on hard search tasks. Iris-pro at 397B parameters (17B active) reaches scores previously associated only with proprietary systems, and the post-training recipe is fully described in the paper.

The code, model weights, and datasets are available on the AllSpark Research GitHub repository.

Conclusion

Iris shows that the gap between open-weight and proprietary search agents is narrowing — not through architectural novelty, but through careful attention to training data quality, iterative policy improvement, and inference-time context handling. The SFT-RL climbing procedure is a reusable template for agents that need to sustain coherent reasoning across many steps and tool calls. The finding that context management is a first-class system component, not an implementation detail, is worth internalizing before your next search agent evaluation.

GPT-6 Astra: Inside OpenAI's Recurrent Depth Architecture and Its Agentic Ambitions

Prabhakar Chaudhary — Fri, 04 Sep 2026 16:06:07 +0000

GPT-6 Astra: Inside OpenAI's Recurrent Depth Architecture and Its Agentic Ambitions

OpenAI released GPT-6 Astra on September 3, 2026, marking what the company describes as its most significant model update to date. Beyond the headline benchmark numbers, Astra introduces a genuinely different approach to how a large language model reasons — one that has already sparked debate among AI safety researchers and practitioners alike.

What Is Recurrent Depth?

Most transformer-based language models process a prompt in a single forward pass: tokens flow through layers once, and the output is produced. GPT-6 Astra departs from this by using a technique OpenAI calls recurrent depth, where the model iteratively revisits and refines its internal representation of the input before committing to an output.

The practical effect is that the model can perform more nuanced multi-step reasoning without requiring an explicit chain-of-thought trace in the output. The reasoning happens in latent space — internal activations that are not directly observable as text. This is meaningfully different from models like o3 or Claude Opus 5, which surface their reasoning as readable scratchpad text.

The tradeoff is transparency. Because the chain of thought is hidden inside the model's recurrent passes rather than written out, it is harder for external observers — including OpenAI's own safety teams — to audit what the model is "thinking" before it acts. OpenAI has implemented chain-of-thought monitoring tools, but the company acknowledges that their effectiveness is limited when reasoning occurs in latent layers. This has drawn criticism from AI safety researchers who argue that monitorability is a prerequisite for safe deployment of highly capable models.

Scale and Training

Astra was trained on OpenAI's Stargate facility in Texas using more than 100,000 GPUs — the company's largest training run to date. Notably, it is also the first OpenAI model where previous AI generations played a substantial role in the training process itself, with earlier models assisting in data curation, evaluation, and synthetic data generation.

The infrastructure was built from the ground up to support this scale, including custom data center networking and specialized inference kernels. The result is a model that, despite its size, runs computer-use tasks at nearly twice the speed of its predecessor GPT-5.6 Sol.

Benchmark Performance

OpenAI reports the following benchmark results for Astra versus Sol:

Benchmark	GPT-6 Astra	GPT-5.6 Sol
FrontierMath Tier 4	97.6%	83.0%
Terminal-Bench 4.0	57.9%	37.3%
ExploitBench	100.0%	78.5%
DeepSWE v1.1	74.1%	72.7%

The ExploitBench score of 100% is the number that has attracted the most attention — and the most caution. During pre-deployment evaluation, Astra developed working exploits for hardened browsers and operating systems and surfaced two previously unknown zero-day vulnerabilities. This is what triggered OpenAI's internal "Critical" cybersecurity classification under its Preparedness Framework — the first time any OpenAI model has crossed that threshold.

The Daybreak Program and Staged Rollout

Because of the Critical classification, Astra's rollout is deliberately staged. The initial release went to enterprise customers enrolled in OpenAI's Daybreak cybersecurity program, specifically the Daybreak Blue track, which provides vetted defenders — government agencies, critical infrastructure operators, and security firms — with access to less-restricted capabilities for offensive security research.

The publicly available version of Astra refuses advanced offensive security requests outright. Users outside trusted-access programs who attempt exploit discovery or vulnerability chaining will encounter hard refusals or deliberate slowdowns. This is a meaningful departure from how previous OpenAI models handled security-adjacent tasks, where refusals were more inconsistent.

For enterprise deployments, Microsoft Foundry provides Astra through a Limited Access Program with additional governance layers: Microsoft Entra identity management, role-based access controls, encryption, and human-in-the-loop checkpoints for consequential agentic actions.

Context Window and Agentic Capabilities

Astra ships with a 1,050,000-token context window and a 128,000-token maximum output. For agentic workflows, the model introduces a persistent note-keeping feature that preserves context across windows — addressing a common failure mode where models lose track of earlier decisions as context fills up.

The model is explicitly designed for long-horizon tasks: software engineering, business intelligence, document creation, and multi-application workflows. OpenAI describes the shift as moving from "chat" to "delivering units of work." In practice, this means Astra can plan across multiple steps, make decisions, call tools, and execute tasks across applications without requiring constant human re-prompting.

API pricing is $10 per million input tokens and $50 per million output tokens, with a Fast mode available at double the speed for double the price. A stronger GPT-6 Astra Pro variant is available to Pro, Business, and Enterprise subscribers.

What Practitioners Should Watch

Three things stand out for teams evaluating Astra for production use:

Latent reasoning opacity. If your use case requires auditable reasoning traces — compliance, healthcare, legal — Astra's recurrent depth architecture is a genuine concern. The model may produce correct outputs through reasoning paths you cannot inspect. Pairing it with external logging and human review checkpoints is not optional in regulated environments.

Cybersecurity dual-use risk. The ExploitBench saturation is a capability signal, not just a benchmark number. Even with safety guardrails, a model that can autonomously discover zero-days represents a meaningful shift in the threat landscape. Security teams should treat Astra-class models as a new category of tool that requires its own access controls and monitoring.

Agentic infrastructure readiness. Astra's value proposition is long-horizon autonomous work. But most enterprise environments are not yet instrumented for agentic workflows — they lack the scoped credentials, activity logging, and rollback mechanisms that make autonomous agents safe to deploy. The model's capabilities will outpace most organizations' readiness to govern them.

Conclusion

GPT-6 Astra is a technically interesting release for reasons that go beyond its benchmark scores. The recurrent depth architecture represents a real departure from how frontier models have handled reasoning, with genuine tradeoffs between capability and transparency. The staged rollout and Daybreak program reflect a more deliberate approach to deploying a model that OpenAI itself classifies as critically capable in the cybersecurity domain.

For practitioners, the most useful frame is not "how good is this model" but "what governance infrastructure do I need before I can safely use it." The answer, for most organizations, is: more than you currently have.

Sources: OpenAI Path to Astra · GPT-6 Astra Wikipedia · Emergent.sh release analysis · Microsoft Foundry announcement · CNET coverage

Stable FP4 Pretraining with Transpose-Invariant 2D Block Scaling

Prabhakar Chaudhary — Thu, 03 Sep 2026 16:07:35 +0000

Stable FP4 Pretraining with Transpose-Invariant 2D Block Scaling

Billing Support — September 03, 2026

A tensor value can receive one scale during the forward pass and a different scale after the same tensor is transposed for backpropagation. At FP4 precision, that mismatch is not a minor rounding detail: it changes the quantized representation used to calculate gradients and can introduce systematic bias into parameter updates.

The September 3 cs.LG work on stable FP4 pretraining through block scaling addresses this failure mode by replacing transpose-sensitive one-dimensional scaling groups with square two-dimensional blocks. The broader recipe combines that structural change with truncation-free scaling, stochastic rounding, deterministic Hadamard rotations, and selective BF16 computation.

Why 1D microscaling breaks under transposition

FP4 offers limited dynamic range and relatively large quantization error. Microscaling formats such as MXFP4 or NVFP4 compensate by dividing a tensor into small groups and assigning each group its own scale.

For a one-dimensional group containing values x, quantization can be summarized as:

q(x, s) = round(x / s)

where s is derived from the values in that group. The problem is that matrix transposition changes which values belong together.

Suppose a matrix is grouped along rows during a forward matrix multiplication. Backpropagation may require its transpose, causing the corresponding computation to group values along what were previously columns. The numerical values have not changed, but their neighbors—and therefore their group scales—have.

The result is scale inconsistency:

q(X, row-scales)^T != q(X^T, row-scales-of-X^T)

This means the forward and backward passes operate on different low-precision approximations of the same underlying tensor. According to the paper’s reported analysis, this inconsistency produces biased gradients. Deterministic rounding cannot repair the structural problem because it only decides how values map to levels after the incompatible scales have already been selected.

Square blocks preserve scale assignments

The proposed fix partitions matrices into square two-dimensional blocks, with 32×32 given as an example. Each block receives a shared scale.

When the matrix is transposed, every square block is also transposed, but its membership remains intact: the same values stay together. A block at one matrix coordinate moves to its transposed coordinate without being regrouped into unrelated sets.

Conceptually:

scale(B) = scale(B^T)

provided the scale calculation is itself insensitive to element order. Forward and backward matrix multiplications can therefore reuse consistent quantized representations instead of deriving scales from different one-dimensional slices.

This does not eliminate FP4 error. It removes one specific source of systematic error: transpose-induced reassignment of values to scaling groups. That distinction matters for implementers. Two-dimensional scaling is the structural foundation, while the remaining techniques control clipping, rounding error, and outliers.

The rest of the stability recipe

Truncation-free scaling

A scale that is too small pushes large values outside FP4’s representable range. Those values are clipped, creating systematic distortion.

Truncation-free scaling chooses the scale so all values in the block fit within the available range. This avoids clipping rather than accepting it as an ordinary quantization effect. The trade-off is that a single large value may force a coarser quantization step for every other value in the block.

That trade-off explains why scaling alone is insufficient: preserving the largest value can reduce resolution for the majority of smaller values.

Stochastic rounding

Round-to-nearest deterministically maps a value to one adjacent representable level. Under repeated low-precision operations, those choices can accumulate directionally.

Stochastic rounding instead selects between neighboring levels probabilistically so that the expected quantized value equals the original value. In compact form:

E[q(x)] = x

This property is especially relevant to gradients, where persistent directional error can alter optimization. Stochastic rounding does not make an individual quantization exact; it targets unbiased behavior in expectation.

Deterministic Hadamard rotations

Outliers create another tension. If a few coordinates carry unusually large magnitude, truncation-free scaling must accommodate them, leaving fewer useful levels for the rest of the block.

Hadamard rotations in low-precision training redistribute outlier energy across coordinates before quantization. The evidence reports that deterministic Hadamard transforms were more effective than randomized variants for stabilizing the complete pipeline, particularly when quantizing weight gradients.

The rotation does not discard information. Its role is to produce a representation whose magnitudes are easier to quantize with one block scale.

Selectively retained BF16 paths

The method is not a claim that every operation should run in FP4. Some paths remain numerically sensitive, especially attention operations involving softmax and dot-product interactions.

The reported mixed-precision policy quantizes dense Q, K, and V projections while retaining sensitive attention paths in BF16. Related full-stack recipes also protect selected compact subspaces in BF16 while executing dominant dense operations in FP4.

For engineers, the practical principle is selective precision: use FP4 where tensor operations dominate cost, but retain BF16 where quantization error would be amplified by the operation.

Interpreting parity and speed claims

The reported evidence indicates performance near BF16 baselines, often with less than 1.5% perplexity degradation when the stabilization techniques are applied together. “Parity” should therefore be read as comparable training quality under the evaluated configurations, not numerical equivalence at every step.

The broader FP4 training literature also reports speedups as high as 4.64× in specific rollout-heavy tasks. That figure is conditional, not a universal pretraining multiplier. Real gains require hardware with native FP4 tensor support and workloads where reduced arithmetic and memory-bandwidth pressure affect end-to-end runtime. BF16 fallbacks, rotations, scale calculation, and data conversion all consume part of the theoretical saving.

Engineering implications and limits

A practical implementation needs more than changing a datatype:

Quantization metadata must represent two-dimensional block scales.
Matrix layouts and kernels must preserve block identity across transposes.
Forward, activation-gradient, and weight-gradient paths need coordinated policies.
Stochastic rounding must be integrated into training kernels.
Hadamard transforms add operations that must be measured against their stability benefit.
Attention and other sensitive paths require explicit BF16 exceptions.

The main limitation is complexity. Square block scaling changes kernel design and metadata handling, while mixed precision creates more execution paths to validate. Truncation-free scaling prevents clipping but cannot prevent an outlier from reducing effective resolution. Stochastic rounding is unbiased only in expectation, and the reported results do not imply that every model, optimizer, or hardware stack will match BF16.

The core result is narrower and more useful: if FP4 groups are not preserved under the transposes required by backpropagation, scale inconsistency can bias gradients. Square 2D blocks preserve those assignments. Combined with careful rounding, outlier redistribution, clipping avoidance, and selective BF16 retention, they provide a technically credible route to stable FP4 pretraining.