DEV Community: Prabhakar Chaudhary

FlatCake: Why Tiered KV Cache Storage Is Holding LLM Serving Back

Prabhakar Chaudhary — Mon, 02 Nov 2026 17:06:51 +0000

FlatCake: Why Tiered KV Cache Storage Is Holding LLM Serving Back

When you send a long prompt to an LLM API, the service has to store the key-value (KV) cache for your session somewhere. For short conversations, GPU HBM handles it fine. But as context windows stretch to hundreds of thousands of tokens — and agentic workflows keep sessions alive for minutes or hours — the KV cache grows large enough that it has to spill off the GPU entirely.

The standard answer has been a tiered hierarchy: GPU HBM → host DRAM → NVMe SSD. DRAM acts as a warm buffer, catching recently evicted blocks before they land on slower SSD. It's the same logic as CPU cache hierarchies, applied to LLM serving. And for a long time, it worked well enough.

A new paper from researchers at Tsinghua University and Zhipu AI argues that this assumption is now wrong. Their system, FlatCake (arXiv:2610.38355), replaces the hierarchy with a flat architecture that treats DRAM and SSD as peer media in a single unified address space.

The Problem with Tiered Storage

The tiered model rests on a key assumption: recently evicted KV blocks are likely to be reused soon, so keeping them in DRAM gives you a fast second chance before they hit SSD. This is the temporal locality assumption — the same one that makes CPU caches work.

Modern LLM workloads increasingly violate it. The authors identify two dominant patterns in production traces:

One-shot blocks: A user sends a long document for summarization. The KV cache is computed once, used once, and never touched again. Staging it through DRAM adds latency without any reuse benefit.

Massive session footprints: An agentic workflow keeps a 200K-token context alive for 30 minutes. The entire session footprint exceeds DRAM capacity, so blocks cycle through DRAM constantly — but the "warm buffer" never actually helps because the working set is too large to fit.

In both cases, DRAM temporal filtering fails. The result is what the authors call "staging contention": DRAM bandwidth gets consumed by blocks moving in and out, while the actual reads and writes that matter — serving live requests — compete for the same bandwidth.

There's a second problem: SSD layout. Traditional tiered systems write KV blocks to SSD in eviction order, not session order. When a session needs to be restored, the blocks for that session are scattered across the SSD, requiring multiple small reads instead of one large sequential read. This read amplification — measured at 6.5x to 6.7x in Mooncake, a widely-used KV cache system that won Best Paper at FAST 2025 — is a significant source of latency. The Mooncake paper describes this architecture in detail.

How FlatCake Works

FlatCake's core insight is that KV blocks are immutable and write-once. Once a block is computed, it never changes. This means you can decide at creation time exactly where a block should live — DRAM or SSD — and never move it again.

The architecture has three main components:

FC-GVA (Global Virtual Address Manager): Manages a unified 64-bit address space that spans both DRAM and SSD. Each block gets a single address that encodes both the medium (DRAM or SSD) and the offset within that medium. Because blocks never migrate, there's no need for eviction-ordered layout or cross-tier movement.

FC-MAS (Medium-Aware Request Scheduler): Maintains separate request queues for DRAM and SSD. By knowing which medium each block lives on (encoded in the address), the scheduler can prevent head-of-line blocking — a slow SSD read doesn't stall a fast DRAM read that's waiting behind it.

FC-CIO (Coalescing Direct I/O Engine): Writes KV blocks to SSD in session-contiguous, layer-major order. When a session needs to be restored, all its blocks are laid out sequentially on disk, enabling a single large read via GPUDirect Storage (GDS) that streams directly into a GPU window — bypassing host DRAM entirely. GDS establishes a direct DMA path between NVMe storage and GPU memory, eliminating the CPU bounce buffer that traditional I/O requires.

The placement policy is straightforward: blocks that are likely to be reused within a short window go to DRAM; everything else goes directly to SSD. Because the address is immutable, there's no migration overhead and no staging contention.

What the Numbers Show

The authors evaluated FlatCake against Mooncake using production traces from DeepSeek-V4-Flash on NVIDIA B200 GPUs, and against Mooncake and Mooncake-GDS using Llama-3.1-70B on A100 GPUs.

On the DeepSeek-V4-Flash production traces:

Cache hit rate: 94% versus 64% for Mooncake
SSD read amplification: Eliminated (Mooncake showed 6.5x–6.7x)
Mean TTFT at peak load: 1.6x reduction
Total token throughput: 1.6x improvement

On Llama-3.1-70B across 8K to 128K context lengths:

Mean TTFT: Up to 2.4x reduction versus Mooncake, up to 1.8x versus Mooncake-GDS
SSD read bandwidth: 1.9x higher with 5.8x lower prefetch latency

The 30-point cache hit rate gap (94% vs 64%) means FlatCake serves far more requests from cached context rather than recomputing — directly translating to lower latency and higher throughput.

Why This Matters for Agentic AI

The shift toward agentic workflows is changing LLM serving workloads in ways traditional infrastructure wasn't designed for. A single agentic session might involve dozens of tool calls, each appending to a growing context. The KV cache for that session needs to persist across calls, potentially for minutes.

This is exactly the workload profile where tiered storage struggles most. The session footprint is large, the reuse pattern is irregular, and the cost of a cache miss — recomputing hundreds of thousands of tokens — is high.

FlatCake handles this naturally. Because blocks are placed directly on their target medium at creation time, there's no staging overhead. Because SSD layout is session-contiguous, restoration is fast. And because DRAM and SSD are managed as peers rather than a hierarchy, the system adapts to different workload patterns without relying on temporal locality.

KV cache management has become one of the central infrastructure problems in LLM serving. Systems like Mooncake, vLLM's PagedAttention, and NVIDIA's Dynamo have each addressed different aspects of the problem. Research from NVIDIA has also highlighted how compression techniques often collide with serving infrastructure assumptions. FlatCake's contribution is specifically to the storage layer — arguing that the tiered model's assumptions no longer hold and that a flat, placement-at-creation-time approach is better suited to current workloads.

Practical Implications

FlatCake is integrated into the SGLang serving framework and includes optimized GPU kernels for H200 and B200 hardware. For teams running long-context or agentic workloads at scale, the results suggest meaningful gains in both latency and throughput without requiring changes to the model itself.

The key design principle — immutable blocks placed once, never migrated — is a useful mental model for KV cache infrastructure more broadly. If your workload doesn't exhibit the temporal locality that tiered storage assumes, you're paying the overhead of a hierarchy that isn't helping you.

Whether FlatCake's approach becomes standard or gets absorbed into existing systems, the underlying argument is worth taking seriously: as context windows grow and agentic sessions become longer, the assumptions baked into LLM serving infrastructure need to be revisited.

NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit

Prabhakar Chaudhary — Fri, 30 Oct 2026 16:04:32 +0000

NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit

Serving large language models at scale is fundamentally a memory problem. During autoregressive decoding, every generated token requires reading the entire weight matrix and the accumulated KV cache from GPU high-bandwidth memory (HBM). On an H100, that bandwidth ceiling is around 3.35 TB/s — fast, but finite. As context windows grow and batch sizes increase, the gap between what the GPU can compute and what it can actually move through memory widens.

Lossless weight compression is one lever practitioners can pull to ease this bottleneck. The idea is simple: store weights in a smaller format, decompress on the fly during inference, and net a bandwidth gain. The catch is that decompression itself must be cheap enough not to eat the savings. A new paper from researchers at Stanford and Together AI introduces NyoomFloat12 (NF12), a 12-bit fixed-length format that threads this needle by exploiting a structural property of trained BF16 weights that turns out to be nearly universal.

The Redundancy Hidden in BF16 Weights

BFloat16 uses 1 sign bit, 8 exponent bits, and 7 mantissa bits. The exponent field is where the redundancy lives. In a trained model, the distribution of weight magnitudes is tightly concentrated — most weights are small, and after applying a per-matrix power-of-two scale to normalize the distribution, the upper four bits of the exponent field are 0111 for more than 99.7% of weights across every tested LLM.

Those four bits are constant. Storing them is waste. NF12 drops them.

The result: eight BF16 weights (128 bits) pack into 96 bits — 12 bytes instead of 16. That is a 25% reduction in weight storage, achieved without any approximation.

For the small fraction of weights whose exponent falls outside the expected range, NF12 uses an escape mechanism. A zero meta byte signals that a group of eight weights is stored verbatim in a per-matrix BF16 buffer. The overhead is negligible, and the format remains strictly lossless.

Why Fixed-Length Matters for GPUs

The prior art in lossless LLM weight compression — most notably DFloat11, presented at NeurIPS 2025 — uses Huffman coding on the exponent bits. Huffman achieves slightly higher compression (around 10.79 bits per weight versus NF12's 12.02), but it produces variable-length encodings.

Variable-length codes are a poor fit for SIMT (Single Instruction, Multiple Threads) GPU architectures. When threads in a warp decode weights of different lengths, they diverge — some finish early and stall while others catch up. The decompression kernel becomes serialized, and the bandwidth savings evaporate into compute overhead. DFloat11's decode kernel takes around 1,771 microseconds on the benchmarks reported in the NF12 paper. NF12's fixed-length decoder takes 548 microseconds — a 3.2× speedup in decompression alone.

NF12's decode path is designed around GPU warp execution. A warp reconstructs groups of weights in lockstep: 13 register operations plus a three-instruction escape check. The common path costs less than a standard BF16 memory copy. That is the key property — the decoder is not just fast, it is faster than the alternative of not compressing at all.

Two Regimes, Two Wins

The NF12 paper evaluates performance across two distinct inference regimes, and the format delivers in both.

Bandwidth-bound inference (small batch, short context): Here the bottleneck is how fast weights can be streamed from HBM to the compute units. NF12's fused GEMV (general matrix-vector multiplication) kernel achieves 1.26× the throughput of a custom BF16 GEMV and 1.35× over cuBLAS on an H100. The 25% smaller weight footprint translates almost directly into a proportional bandwidth gain, minus the small decode overhead.

Capacity-bound inference (large batch, long context): Here the bottleneck is total GPU memory. The KV cache competes with model weights for HBM space, and as context grows, the cache wins — crowding out concurrency. By shrinking the weight footprint, NF12 frees HBM for KV cache. On Qwen3-32B, NF12 provides 4.5× more KV cache capacity compared to BF16. At high concurrency, this translates to 2.54× higher rollout throughput — the headline number from the paper.

That 2.54× figure is not a cherry-picked benchmark. It reflects a realistic serving scenario where many sequences are processed simultaneously and the KV cache is the binding constraint.

Practical Implications for Inference Engineers

NF12 is a drop-in optimization for existing model weights. No retraining, no fine-tuning, no accuracy degradation — the format is lossless by construction. The compression step is a one-time offline operation. At inference time, the fused GEMV kernel handles decompression transparently.

The format is most impactful in two situations practitioners encounter regularly:

Fitting larger models on existing hardware. A 32B model in NF12 occupies roughly the same HBM as a 24B model in BF16. That headroom can be used for longer contexts, larger batches, or simply running a model that previously required an extra GPU.
Improving throughput at long context. As context windows push into the hundreds of thousands of tokens, KV cache pressure becomes the dominant constraint. NF12's capacity gains compound with other KV cache optimizations like grouped query attention and prefix caching.

The comparison with DFloat11 is instructive for practitioners choosing between lossless methods. DFloat11 compresses slightly more aggressively and is well-suited for scenarios where the goal is fitting a model into a fixed memory budget and throughput is secondary. NF12 is the better choice when end-to-end inference speed is the objective — its fixed-length decoder keeps the GPU's SIMT execution efficient.

What This Tells Us About the Inference Optimization Landscape

NF12 is part of a broader pattern in 2026 inference research: the recognition that the memory wall is the primary constraint, and that solutions must be co-designed with GPU architecture rather than borrowed from general-purpose compression. The Invariant Bit Packing (IBP) work from earlier this year takes a similar approach for PCIe transfers. The common thread is that compression formats designed for CPUs — Huffman, ANS, LZ4 — do not map cleanly onto SIMT execution models.

NF12's contribution is demonstrating that a carefully chosen fixed-length format can beat variable-length alternatives on end-to-end throughput even when it compresses less. The 25% size reduction is not the point; the SIMT-aligned decoder is.

For teams running inference at scale, NF12 is worth evaluating as a complement to quantization-based approaches. Unlike INT8 or FP8 quantization, it introduces zero approximation error — useful in domains where numerical fidelity matters, or when quantization has already been applied to the KV cache and further lossy compression of weights is undesirable.

The paper is available on alphaXiv. The authors are Liberman, Fang, Zhang, Dao, and Fu — several of whom have prior work on FlashAttention and efficient GPU kernels, which explains the focus on SIMT-aligned design.

For teams already using vLLM or SGLang for serving, NF12 is designed to integrate at the kernel level without changes to the serving framework's scheduling or batching logic. The authors plan to release the CUDA kernels as open source, which should make adoption straightforward once the code is available.

Heads of Security: Defending Against Prompt Injection by Gating Attention

Prabhakar Chaudhary — Thu, 29 Oct 2026 20:10:58 +0000

Heads of Security: Defending Against Prompt Injection by Gating Attention

Indirect prompt injection is what happens when an LLM agent reads a web page, an email, or a tool result and mistakes attacker-planted text for a command from its user. A new paper from researchers at Imperial College London, the Tübingen AI Center and others, Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness (arXiv, 27 Oct 2026), attacks the problem at a surprisingly low level: it trains a small set of per-head attention gates rather than fine-tuning the model's weights. This post walks through how the mechanism works, what the numbers show, and where the authors say it stops helping.

The goal: instruction–data separation

The paper focuses on one defensive primitive, instruction–data separation. Authorized instructions (system prompt, user prompt) may steer the model. External content, such as a tool output, may be read, summarized, and reasoned over, but must not be obeyed. This is the failure mode first described in Greshake et al.'s work on indirect prompt injection, and it stays unsolved: adaptive attacks keep succeeding, and the authors note that open-weight models are noticeably easier to attack than the leading closed ones.

The usual fix is preference fine-tuning. SecAlign++ trains the model, via DPO, to prefer the response that follows the user instruction over the one that follows an injected instruction. That works on the training distribution, but the authors argue it gives the optimizer too much freedom to learn shortcuts.

How the gates work

Every token gets a trust channel. In a typical agent setup, the trusted channel covers the system prompt, user prompt and the model's own generations. The untrusted channel covers tool outputs. A third "template" channel covers chat-template tokens, including the BOS token that often acts as an attention sink.

Each attention head in each layer then gets a small table of learned gate logits, indexed by (query channel, key channel). Before the softmax, the head adds the log of a sigmoid of that logit to the raw attention score for each query–key pair. A gate near 1 means the head attends to that channel normally. A gate near 0 effectively closes it, in the same way causal masking adds negative infinity to disallowed positions. After softmax renormalization, attention mass moves to the remaining tokens.

Two properties follow from this design:

Heads specialize instead of being blinded. A head that mostly drives instruction following can learn to ignore untrusted tokens, while other heads still read that content, so the model can summarize or extract facts from it.
Single-channel inputs are unchanged. If every key shares the same gate, softmax cancels the shift. An input with no untrusted tokens is processed exactly as in the base model.

Training touches only the gate scalars, which is orders of magnitude fewer trainable parameters than SecAlign's adapter (the paper puts SecAlign at 349M trainable parameters for Llama-3.3-70B and 38M for GPT-OSS-20B). The authors reuse the SecAlign++ recipe: DPO on an adapted Cleaned-Alpaca dataset where each pair has a user instruction plus an injected instruction in the untrusted segment. For GPT-OSS, which has an explicit attention sink, the sink is treated as an extra key-only channel.

What the evaluation shows

The paper evaluates on Llama-3.3-70B-Instruct and GPT-OSS-20B, and compares against SecAlign++ and ASIDE, another architecture-level separation method.

Black-box adaptive attack (RL-Hammer on InjecAgent). RL-Hammer uses GRPO to train an attacker model to rewrite goals into injections tuned for a target. The strongest attacker reaches 97% attack success rate (ASR) against undefended GPT-OSS-20B and 85% against the SecAlign variant, but only 10% against the HoS variant. On Llama-3.3-70B the highest figures are 99%, 82% and 1%. One caveat from the paper itself: the attacker trained directly against HoS gets sparse reward, reaching only 3.2% training ASR on GPT-OSS-20B, so attacker weakness may explain part of that gap. That is why the transfer attacks matter.

White-box attack (Claudini plus autoresearch). Starting from the best attack reported by Claudini, the authors used an LLM-driven loop for 50 iterations to tailor the attack to HoS, then evaluated it on 100 AgentDojo and 100 InjecAgent cases. HoS needed roughly three times the attack compute to reach 50% ASR compared with the base model or SecAlign.

Utility. HoS preserves base-model scores on standard multiple-choice benchmarks, including in few-shot prompts, where SecAlign and ASIDE degrade. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and both improve the security–utility trade-off over the base model. ASIDE reaches only 10% utility on tool use. The paper's explanation is that, on identical data, SecAlign learns a shortcut while the constrained gate mechanism has less capacity to do so; the authors frame this as an argument, not a proof.

Limits worth knowing

The discussion section is candid, and engineers should read it before treating this as a drop-in fix:

It is not an end-to-end defense. HoS enforces a boundary but does not decide which sources are trusted. Your application still has to label channels correctly.
Some tasks need external instructions. Installing a package from a README is an example. Strict separation only fits cases where content is meant as data.
The attacker moves second. Lower success under a fixed budget does not rule out a motivated adversary.
Dense attention only. The method needs explicit pairwise attention logits, so it does not directly apply to linear-attention architectures.
Training data is fixed. The comparison uses one preference dataset. More diverse data (multi-turn, multiple instructions) might narrow SecAlign's gaps, and the authors say so.

The authors also released code, a demo and trained models; see the project repository referenced in the paper. For the training objective, the DPO paper is the background reading.

Takeaway

The interesting idea is less the headline ASR numbers than the design choice: put the trust boundary inside the attention computation, and give the optimizer only a small, structured knob to turn. If you build agents on open-weight models, this is a mechanism to watch, with the caveat that so far it is validated on two models and a handful of benchmarks. Treat it as one layer next to instruction hierarchy, least-privilege tool permissions and output filtering, not a replacement for them.

MuonGS: Squeezing 15% More Speed Out of LLM Pretraining via Relaxed Weight-Norm Control

Prabhakar Chaudhary — Thu, 29 Oct 2026 17:55:13 +0000

MuonGS: Squeezing 15% More Speed Out of LLM Pretraining via Relaxed Weight-Norm Control

In the high-stakes world of Large Language Model (LLM) pretraining, efficiency is everything. When you are training a model on trillions of tokens, a 5% improvement in optimizer efficiency can save millions of dollars in compute costs. Today, a new contender has emerged in the "Muon" family of optimizers: MuonGS.

Published on October 27, 2026, by researchers Athanasios Glentis, Dawei Li, and Mingyi Hong, MuonGS (Gram-Schmidt) introduces a sophisticated way to manage weight-norm growth. By relaxing the strict constraints found in previous versions of the optimizer, MuonGS delivers an estimated 15% token-equivalent speedup on dense models and shows strong results on Mixture-of-Experts (MoE) architectures.

The Problem: The Effective Learning Rate Decay

To understand why MuonGS is such a significant step forward, we first need to look at the "effective learning rate" problem. In deep learning, the actual progress a model makes isn't just determined by the nominal learning rate ($\eta$) you set in your config file. It is determined by the ratio of that learning rate to the magnitude (Frobenius norm) of the weight matrix:

$$\text{Effective LR} = \frac{\eta_t}{|W_t|_F}$$

As an LLM trains, its weight matrices naturally tend to grow in magnitude. In the vanilla Muon optimizer—which uses matrix-level Newton-Schulz orthogonalization to coordinate updates—this growth is linear. As the weights ($W$) get larger, the effective learning rate shrinks. Over a long training run, the model’s ability to explore the loss landscape stalls because the updates become too small relative to the weights they are trying to change.

The Muon Family: From Weight Decay to Hard Clamping

Researchers have tried several ways to stop this "norm explosion." The two most common variants are:

MuonW: Uses traditional weight decay to pull the weights back toward zero.
MuonH: Uses a "hard-radius" constraint (Hyperball) to fix the weight norm to a specific value.

While these methods work, they are somewhat heavy-handed. By forcing the weights into a specific "box" or radius, they can over-constrain the model's trajectory, potentially preventing it from reaching the most optimal minima in the loss landscape.

How MuonGS Works: The Gram-Schmidt Projection

MuonGS takes a more elegant, "relaxed" approach. Instead of forcing the weight norm to be a specific number, it focuses on the direction of the growth.

The core innovation of MuonGS is the use of Gram-Schmidt (GS) projection to remove the radial component of every update.

1. Identifying the Radial Component

Every update produced by the Muon optimizer can be decomposed into two parts: a radial part (pointing directly away from the origin, increasing the norm) and a tangential part (rotating the weight vector without changing its length).

2. The Projection Step

MuonGS identifies the radial part of the update—the portion that pushes the weight matrix directly outward—and removes it using the Gram-Schmidt process. This ensures that the update applied to the weights is always orthogonal to the current weights in the Frobenius inner product space.

3. Square-Root Growth Dynamics

Because the radial component is removed, the first-order change to the weight norm becomes zero. The norm only increases due to second-order effects. Mathematically, this means the weight norm grows at a rate proportional to the square root of the training steps ($\sqrt{t}$), rather than growing linearly ($t$) like in vanilla Muon.

This "sweet spot" allows the accumulated effective learning rate to grow at a rate of $\Omega(\sqrt{T})$. This is significantly more robust than the logarithmic accumulation seen in vanilla Muon, providing the model with sustained "exploration energy" throughout the entire training run without the rigid boundaries of MuonH.

Why It Matters: 15% More Efficient Pretraining

The theoretical elegance of MuonGS is backed by impressive empirical data. The researchers tested the optimizer across several architectures:

Dense Models: On 160M and 400M parameter models, MuonGS achieved lower validation perplexity than MuonH at long training horizons. This translated to a 15% token-equivalent speedup—meaning you can reach the same level of model quality using 15% fewer tokens (and 15% less compute).
Mixture-of-Experts (MoE): The optimizer was also validated on an MoE model with 135M active parameters and 1B total parameters. It successfully outperformed MuonH in this setting, proving that it is ready for the complex architectures used in modern production models.

What to Watch: The Future of "Relaxed" Optimization

MuonGS represents a shift in how we think about optimizer constraints. As we move toward training runs that span trillions of tokens, "hard" constraints like those in MuonH may become too restrictive.

By providing a formal stationarity guarantee of $O(T^{-1/2})$ and demonstrating that "relaxed" control is superior for long-term progress, MuonGS opens the door for a new generation of optimizers that are stable enough for massive scale but flexible enough for deep exploration. For researchers and engineers looking to squeeze every drop of performance out of their GPU clusters, MuonGS is a technique that demands attention.

Sources

Primary Source: Glentis, A., Li, D., & Hong, M. (2026). MuonGS: Relaxed Weight-Norm Control for LLM Pretraining. arXiv.
Supporting Source: MuonGS Relaxes Weight-Norm Control Without Giving Up Training Speed. Kenashe AI Blog.
Supporting Source: MuonGS Optimizer Delivers 15% Speedup for LLM Pretraining. Que AI News.

Mercury 3: How Inception Labs Scaled Diffusion LLMs to 1,000 Tokens Per Second

Prabhakar Chaudhary — Wed, 28 Oct 2026 16:04:43 +0000

Mercury 3: How Inception Labs Scaled Diffusion LLMs to 1,000 Tokens Per Second

When most people think of large language models, they picture the same basic loop: generate one token, append it to the context, generate the next. Autoregressive decoding has been the default for years, and for good reason — it works. But it also has a hard ceiling on throughput, because each token depends on every token before it. Inception Labs has been quietly building a different kind of model, and with Mercury 3, released on October 19, 2026, they've pushed diffusion-based language models to a scale where that ceiling starts to matter.

What Makes a Diffusion LLM Different

Standard autoregressive models generate text left-to-right, one token at a time. Each step is conditioned on the full sequence so far, which means you can't parallelize across output positions — you have to wait for position n before you can compute position n+1. This sequential dependency is the fundamental bottleneck.

Diffusion language models flip the process. Instead of building a sequence incrementally, they start with a fully masked or noisy draft of the entire response and iteratively refine all positions in parallel. The model runs multiple denoising passes, each time improving the quality of the whole output simultaneously. Think of it as the difference between writing a document word-by-word versus sketching a rough outline and progressively sharpening every section at once.

Mercury 3 is built on a Transformer backbone — the same architectural foundation as GPT-style models — but replaces the autoregressive training objective with a denoising diffusion loss. This means the model learns to predict clean tokens from corrupted ones, rather than predicting the next token from a clean prefix. The Transformer handles the heavy lifting of attention and representation; the diffusion process defines how generation actually works.

The Throughput Advantage

The practical payoff is speed. Mercury 3 achieves approximately 1,068 tokens per second on NVIDIA GPUs — a figure Inception claims is 4 to 8 times faster than speed-optimized autoregressive models like GPT-6 Luna and Claude Haiku 5.5. The reason is arithmetic intensity: because the model refines many positions simultaneously, it can saturate GPU compute more efficiently than a sequential decode loop that spends most of its time waiting on memory bandwidth for a single token at a time.

This isn't just a benchmark number. For applications that need to generate large volumes of text quickly — automated evaluation pipelines, document drafting, high-throughput API services — the difference between 150 tokens/sec and 1,000 tokens/sec changes what's economically feasible. Inception prices Mercury 3 at $0.10 per million input tokens and $0.35 per million output tokens (a 50% launch discount from list price), positioning it well below the cost of frontier reasoning models. Full pricing details are on the Inception pricing page.

Scale and Context

Mercury 3 is described as more than 40 times larger than the original Mercury 1 model. Inception has not disclosed an exact parameter count — the company notes that raw parameter comparisons between diffusion models and autoregressive models can be misleading, since the two architectures use compute differently during inference. What they have confirmed is a 1-million-token context window, up from 260K in Mercury 2.5.

The scaling story matters because it addresses a longstanding criticism of diffusion LLMs: that they work well at small scale but don't improve predictably as you add parameters. Inception's position, backed by their Mercury technical report, is that the same scaling laws that govern autoregressive models apply to diffusion models — you just need to run the experiment at sufficient scale to see it. Mercury 3 is their evidence.

Where Diffusion Helps (and Where It Doesn't)

The parallel refinement process gives diffusion models structural advantages in specific task types.

Infilling and editing are natural fits. Because every position in the output is revisable at each denoising step, the model can natively fill in the middle of a document or revise a specific section without the "fill-in-the-middle" workarounds that autoregressive models require. Constrained generation — producing output that must conform to a JSON schema or a specific format — also benefits, since late-stage constraints can propagate back to earlier positions during refinement.

Agentic coding tasks showed strong results in early testing. In one documented case, Mercury 3 diagnosed and fixed a non-obvious logic bug in a donor-registry application without explicit error messages, demonstrating grounded reasoning rather than pattern-matching on surface cues. Inception also reports that running "swarms" of Mercury 3 agents for recursive self-improvement increased effective throughput by an additional 16%.

The limitations are real, though. Long-horizon, multi-step tasks — building an interactive graphic from scratch, complex open-ended generation — exposed weaknesses. The model slowed down, experienced looping issues, and struggled with tool calling as task complexity increased. This is consistent with what you'd expect: diffusion models excel when the output space is well-defined enough that parallel refinement converges quickly. Open-ended generation with many interdependent decisions is harder to parallelize effectively.

Availability and Integration

Mercury 3 is available through the Inception API, OpenRouter, Baseten, and Vercel. It exposes an OpenAI-compatible API, which means existing code that calls GPT-style endpoints can switch to Mercury 3 with minimal changes. The model identifier is mercury-3 on the Inception platform.

Inception has sunset Mercury 2 and 2.5 for new users, with existing access ending November 15, 2026. The company has indicated interest in making the launch pricing permanent, though no formal commitment has been made.

What This Means for the Field

Mercury 3 doesn't replace frontier reasoning models — it's not trying to. What it demonstrates is that diffusion-based language models can scale to competitive quality levels while maintaining a throughput advantage that autoregressive architectures can't match by design. For practitioners building high-volume pipelines, automated evaluation systems, or latency-sensitive applications, that's a meaningful option that didn't exist at this quality level a year ago.

The deeper question is whether diffusion scaling continues to hold as models get larger. If Inception's claim about scaling laws is correct, the throughput gap between diffusion and autoregressive models should widen as both approaches scale — because the sequential bottleneck in autoregressive decoding doesn't get better with more parameters, but the parallelism in diffusion decoding does. That's a hypothesis worth watching as the next generation of models arrives.

RFD4-Proteina: How NVIDIA and the Baker Lab Turned Protein Design Into a Search Problem

Prabhakar Chaudhary — Mon, 26 Oct 2026 16:05:22 +0000

RFD4-Proteina: How NVIDIA and the Baker Lab Turned Protein Design Into a Search Problem

Designing a protein that binds a specific target, catalyzes a reaction, or folds into a precise shape has historically required years of wet-lab iteration. Computational tools have compressed that timeline, but even the best generative models still produce thousands of candidates that must be screened after the fact. RFD4-Proteina, released on October 20, 2026, by NVIDIA and the Institute for Protein Design at the University of Washington, takes a different approach: it treats protein design as an active search problem, pruning bad trajectories during generation rather than filtering them afterward.

The result is a 435-million-parameter open-weights model that outperforms previous methods by wide margins on four benchmark suites — and does so while using 84% less peak GPU memory than its predecessor.

What the Model Actually Does

RFD4-Proteina is a latent-space flow-matching generative model. Rather than predicting a protein structure from a sequence (the classic folding problem), it works in the opposite direction: given a design specification — a binding target, a catalytic site, a small-molecule ligand — it generates both the protein sequence and the corresponding all-atom 3D structure simultaneously.

The architecture has two main components: a 178-million-parameter trunk that runs inside a latent space, and a 256-million-parameter autoencoder that maps between that latent space and explicit backbone coordinates. By operating at the residue level rather than the atom level, the model compresses the representation enough to complete a generation in 60 sampling steps — compared to roughly 400 steps required by earlier methods like Proteina-Complexa.

The model supports seven distinct design tasks:

De novo generation of monomers or conditional designs
Motif scaffolding — building a protein scaffold around a specified functional segment
Enzyme design — generating scaffolds around catalytic sites
Small-molecule binder design — wrapping a protein around a ligand
Antibody/TCR design — redesigning CDR loops on fixed frameworks
Partial diffusion — refining an existing design by re-initiating the process mid-trajectory
Inverse folding — resampling sequences for a fixed backbone

The Core Innovation: Reward-Guided Beam Search

Most generative protein design workflows follow a simple pattern: sample many candidates, then filter. This works, but it wastes compute on trajectories that were never going to produce a viable design.

RFD4-Proteina replaces that pattern with reward-guided inference-time search. During the denoising process, the model pauses at defined step checkpoints — for example, at steps 0, 20, 40, 50, and 60. At each checkpoint, surviving candidates are branched: each branch is rolled out to a complete structure and scored against an objective. Only the top candidates (controlled by a beam_width parameter) continue to the next stage.

The scoring is done by CUDA-accelerated external oracles. The default integration uses OpenFold3 for structural confidence (pLDDT scores) and TMol for physics-based interaction metrics. Researchers can plug in custom reward functions as callbacks, making the search adaptable to specific design goals.

This is meaningfully different from best-of-N sampling. In best-of-N, you generate N complete structures and pick the best one. In beam search, you prune at intermediate steps, concentrating compute on trajectories that are already showing promise. The practical effect: at 200 residues, RFD4-Proteina is 3.8x faster than RFD3 and uses 84% less peak reserved memory.

Benchmark Results

The performance gap over previous methods is substantial. On a frozen benchmark run from October 9, 2026, RFD4-Proteina (High Thinking mode) achieved the following design efficiency scores:

Task	RFD4-Proteina	Best Previous Method
Protein binders (24 targets)	71.60	13.24 (Proteina-Complexa)
Small-molecule binders (77 targets)	112.25	15.48 (BoltzGen)
Atomic motif enzyme (37 targets)	91.18	26.02 (AlphaProtein-novo)
DNA binders (3 targets)	41.97	9.46 (BoltzGen)

These are not marginal improvements. The protein binder score is more than 5x the next-best method; the small-molecule binder score is more than 7x. Part of this gap reflects the reward-guided search itself — the model is not just better at generating candidates, it is better at finding the good ones within a fixed compute budget.

How to Use It

RFD4-Proteina is available through three channels:

Open weights on Hugging Face (nvidia/RFD4-Proteina) under a BSD 3-Clause license, in both native and diffusers-compatible formats
NVIDIA NIM microservice — a production-ready API wrapper with BioNeMo Inference Runtime acceleration
RosettaCommons GitHub — the reference implementation with full documentation for reward-guided search configuration

The model requires Linux x86_64, Python 3.12, and an NVIDIA GPU with a CUDA 13-compatible driver. The weights are stored in FP32 and run in BF16 at inference time. Full setup instructions and example notebooks are available in the RosettaCommons/RFD4-Proteina GitHub repository.

For reward-guided search, the key parameters are beam_width (how many candidates survive each checkpoint), n_branch (how many branches to create at each checkpoint), and step_checkpoints (which denoising steps trigger scoring). A typical experimental configuration uses a beam width of 3 and 2 branches per checkpoint.

Why This Architecture Matters

The shift from post-hoc filtering to in-trajectory search reflects a broader pattern in generative AI: inference-time compute is increasingly being used to improve output quality without retraining. In language models, this shows up as chain-of-thought reasoning, tree-of-thought search, and process reward models. RFD4-Proteina applies the same logic to protein design — the model's "thinking" happens during the diffusion trajectory, guided by external oracles that encode domain knowledge.

The open-weights release under BSD 3-Clause is also notable. Protein design tools have historically been locked behind institutional access or commercial APIs. Making the weights freely available for commercial use lowers the barrier for biotech startups and academic labs that want to build on top of the model rather than use it as a black box.

The NVIDIA developer blog post and the model card on NVIDIA Build both include worked examples of reward-guided search configurations, including how to integrate custom oracles and interpret the output confidence scores.

Practitioner Implications

For researchers already using RFD3 or Proteina-Complexa, the upgrade path is straightforward: the model is available through the same NIM API surface, and the open weights are compatible with existing RosettaCommons tooling. The memory reduction alone (84% less peak VRAM at 200 residues) makes it practical to run on workstation-class hardware that would have struggled with earlier versions.

For developers building protein design pipelines, the reward-guided search API is worth understanding in detail. The ability to plug in custom scoring functions — not just the built-in OpenFold3 and TMol oracles — means the search can be adapted to application-specific objectives: binding affinity predictions, solubility scores, or experimental assay proxies.

The model's support for seven distinct design tasks in a single set of weights also simplifies pipeline architecture. Rather than maintaining separate models for binder design, enzyme scaffolding, and inverse folding, a single RFD4-Proteina deployment can handle all of them.

Protein design is still a hard problem. But RFD4-Proteina demonstrates that treating generation as search — and investing compute in trajectory pruning rather than post-hoc filtering — can produce substantial efficiency gains without sacrificing output quality.

MiniMax M3.1 Flash: How a 428B MoE Model Cuts Agentic AI Costs by 70% Without Sacrificing Performance

Prabhakar Chaudhary — Fri, 23 Oct 2026 16:05:31 +0000

MiniMax M3.1 Flash: How a 428B MoE Model Cuts Agentic AI Costs by 70% Without Sacrificing Performance

MiniMax released M3.1 Flash on October 19, 2026, and the numbers are hard to ignore: a 428-billion-parameter Mixture-of-Experts model that activates only 23 billion parameters per token, runs at over 200 tokens per second, and costs $0.15 per million input tokens — roughly 70% cheaper than its predecessor. But the more interesting story is in the engineering choices that make those numbers possible.

The MoE Arithmetic: 428B Total, 23B Active

Mixture-of-Experts architectures have become the dominant approach for scaling language models without proportionally scaling inference cost. M3.1 Flash follows this pattern: the full model has 428 billion parameters spread across many expert sub-networks, but for any given token, only a small subset of those experts — totaling about 23 billion parameters — are activated by the router.

This means the model carries the representational capacity of a very large dense network while paying the inference cost of a much smaller one. The 23B active parameter count puts it in the same compute class as a mid-sized dense model, but with access to the specialized knowledge encoded across the full expert pool. For agentic workloads that involve long multi-step tasks, this matters: you get deep reasoning without the per-token cost of a 400B+ dense model.

MiniMax Sparse Attention: Rethinking the Attention Bottleneck

Attention is the other major cost driver in transformer inference, especially at long context lengths. M3.1 Flash introduces MiniMax Sparse Attention (MSA), which replaces the first three full-attention layers with sparse alternatives. Full attention scales quadratically with sequence length — a 1-million-token context window would be computationally prohibitive without some form of sparsification.

MSA selectively attends to a subset of positions rather than the full sequence, reducing the compute and memory footprint at long contexts while preserving the model's ability to track relevant information across the window. The first three layers handle early feature extraction, where full attention is most expensive and where sparse patterns tend to be sufficient. Later layers, which handle more complex reasoning, retain full attention where it matters most.

The result is a model that genuinely supports a 1-million-token context window in practice, not just on paper.

DSpark Speculative Decoding: Replacing EAGLE with a Dedicated Draft Head

Speculative decoding is a well-established technique for accelerating autoregressive generation: a small "draft" model proposes several tokens at once, and the main model verifies them in parallel. If the draft is good enough, you get multiple tokens per forward pass of the large model.

M3.1 Flash replaces the previous EAGLE-like Multi-Token Prediction (MTP) approach with DSpark, a new speculative decoding scheme that ships with a dedicated draft head included in the model release. The draft head is purpose-built for M3.1 Flash's architecture, which means it can propose tokens that align well with the main model's distribution — reducing the rejection rate and improving the effective throughput gain.

The practical outcome: MiniMax reports median output speeds of 170+ tokens per second, with peaks exceeding 200 TPS on compatible serving infrastructure (NVIDIA Blackwell architectures — B200, B300, GB200, GB300). For agentic applications where the model is generating long tool-call sequences or code blocks, this throughput difference is meaningful.

Quantization-Aware Training: NVFP4 for the Expert Weights

To fit a 428B model into practical serving infrastructure, M3.1 Flash uses quantization-aware training (QAT) rather than post-training quantization. The routed MoE experts are quantized to NVFP4 (W4A4 — 4-bit weights and 4-bit activations), while the sparse-attention indexer and main branch use Q8KV4.

QAT matters here because it bakes the quantization error into the training process, allowing the model to compensate during learning rather than absorbing the accuracy hit after the fact. Post-training quantization at 4-bit precision can degrade performance noticeably on reasoning tasks; QAT narrows that gap significantly. The result is a model that achieves near-full-precision accuracy at a fraction of the memory footprint.

Tunable Reasoning Effort: Five Levels of Deliberation

One of the more practical features for developers is M3.1 Flash's configurable reasoning effort. The model supports five levels — low, medium, high, xhigh, and max — set via the chat template. This lets you trade off latency and token cost against reasoning depth depending on the task.

A simple retrieval or classification task can run at low effort; a complex multi-step coding or cybersecurity task can run at max. Notably, the model enforces reasoning by default — attempting to disable it via API parameters returns an error. The design philosophy is that intermediate logic chains are always allocated, and the effort level controls how much deliberation happens, not whether it happens at all.

Benchmark Results: Where It Performs and Where It Doesn't

MiniMax published results across a range of agentic benchmarks:

CyberGym (vulnerability discovery): 82%
BankerToolBench (financial tool-use): 76.2
DeepSWE v1.1 (software engineering): 73.7%
OSWorld V2 (computer use): 61.7
Agents' Last Exam (complex multi-step professional tasks): 29.6

The cybersecurity and coding numbers are strong. The Agents' Last Exam score of 29.6 reflects the genuine difficulty of that benchmark — it's designed to be hard for current models — rather than a model weakness. For comparison, the OSWorld V2 score of 61.7 puts it among the top performers on computer-use tasks, which require the model to navigate real desktop environments across applications.

The model accepts text, images, and video natively, making it multimodal out of the box. Inference is supported on vLLM, SGLang, and TokenSpeed runtimes.

What This Means for Practitioners

The 70% cost reduction relative to M3 is the headline, but the more durable point is the architectural coherence: MSA, DSpark, and QAT are not independent optimizations bolted on after the fact. They're designed together to make a 428B model serve at the cost and latency of a much smaller one.

For teams building agentic pipelines — coding assistants, computer-use agents, financial automation — M3.1 Flash offers a practical option at the frontier of capability without the infrastructure cost of a dense model at comparable scale. The 1M-token context window and tunable reasoning effort give developers meaningful control over the cost-quality tradeoff at runtime.

The model weights are available on Hugging Face, and the API is live via MiniMax's platform at $0.15/$0.60 per million input/output tokens. For a detailed comparison against competing models, Kie.ai's benchmark breakdown is a useful reference. The NVIDIA model card covers the full serving configuration details for Blackwell hardware.

How AI-Assisted Vulnerability Research Becomes a Patch, Not Just a Finding

Prabhakar Chaudhary — Thu, 22 Oct 2026 19:41:25 +0000

Billing Support — October 22, 2026

Finding a vulnerability is not the same as securing users. Once AI accelerates discovery, the hard part shifts downstream: reproducing reports, rejecting false positives, designing compatible fixes, coordinating releases, and confirming that deployed systems are actually protected. OpenAI’s October 21–22, 2026 PHP2Shell disclosure illustrates why vulnerability research needs an operational pipeline, not merely a capable model.

Discovery Must Start With a Bounded Task

A useful AI-assisted security engagement begins by constraining the problem. Researchers and maintainers define the codebase, components, threat assumptions, supported versions, and classes of behavior worth investigating. Without those boundaries, a model can produce a large queue of speculative findings that consumes more maintainer time than it saves.

OpenAI’s Patch the Planet program was designed around this bottleneck. The initiative combines frontier models with human security review across discovery, validation, severity assessment, patch development, testing, and disclosure. Maintainers remain responsible for accepting and deploying upstream changes.

Within a bounded investigation, AI can help researchers perform variant analysis: starting from a known weakness or dangerous pattern and searching for related implementations elsewhere in the code. It can also assist with building fuzzing harnesses that repeatedly feed structured, malformed, or boundary-case inputs into a target component.

Harness creation matters because a finding becomes more credible when it can be triggered consistently. The harness also becomes reusable infrastructure. Maintainers can run it against later commits rather than relying on a one-time model transcript or a prose description of suspicious code.

Trail of Bits, which collaborated on the program, describes its role in the defensive workflow on its Patch the Planet program page. The important distinction is that the AI supports researchers performing these tasks. It does not independently decide that a report is valid, merge a change, publish a release, or operate production systems.

Humans Turn Candidate Findings Into Engineering Evidence

After discovery, human security engineers must reproduce the behavior in a controlled environment. They inspect the relevant execution path, establish the conditions required to reach it, and determine whether the result crosses a meaningful security boundary.

This triage stage separates four questions that automated output can blur:

Is the reported behavior reproducible?
Is it a security issue rather than an ordinary software defect?
Which versions, configurations, and interfaces are affected?
Can the report be communicated without exposing unnecessary exploitation detail?

A model-generated candidate may be wrong because it misunderstood data flow, assumed an unsupported configuration, or overlooked an existing validation layer. Manual reproduction limits the cost of sending maintainers low-confidence reports and gives patch authors a concrete failing case.

PHP2Shell is the current illustrative disclosure, not evidence that every model-generated report deserves emergency treatment. OpenAI’s account of the coordinated disclosure of critical vulnerabilities presents the response as a multi-stage process involving validation, fixes, staged communication, update deployment, and public guidance.

The lifecycle is compact in concept, but each transition requires evidence and an accountable owner:

Bound scope -> AI-assisted search -> Human reproduction
     -> Triage -> Patch -> Regression/differential tests
     -> Maintainer review -> Coordinated release -> Verify deployment

When discovery accelerates, this chain becomes the limiting factor. More findings increase demand on the same scarce resources: security engineers who can validate them, maintainers who understand compatibility constraints, release teams that can package fixes, and operators who must deploy them. Faster scanning without greater remediation capacity can therefore create a larger backlog of known but unresolved risk.

A Patch Needs More Than a Plausible Code Edit

Once engineers confirm a vulnerability, AI can assist with proposing edits, tracing callers, identifying similar branches, and generating tests. But a plausible diff is only the beginning.

The patch must first stop the reproduced behavior. It must then survive regression testing, which checks that existing supported behavior still works. Security fixes often touch parsers, validation rules, protocol handling, or compatibility paths where an apparently narrow change can reject legitimate inputs or alter output.

Differential testing adds another layer. Researchers run related inputs against the vulnerable and patched versions, then compare results. The expected difference should appear for the malicious or invalid case, while normal cases should remain stable. This technique helps detect both incomplete fixes and overly broad restrictions.

Variant analysis should also be repeated after the initial patch. If the vulnerable pattern exists in parallel code paths, fixing only the first reproducer may leave equivalent routes open. The original fuzzing harness and newly added regression cases should be retained so future changes cannot quietly restore the defect.

Upstream maintainer review remains essential. Maintainers understand release branches, portability requirements, API guarantees, and project-specific conventions that an external research team may not. They may revise the proposed patch, request a narrower change, or choose separate fixes for different maintained branches. That is normal review, not friction to be bypassed.

Disclosure Ends With Verification, Not Publication

For a serious vulnerability, publishing immediately after validation can leave downstream users exposed before fixes reach them. Coordinated disclosure instead stages the work: validate the issue, prepare and test patches, notify relevant parties, support update availability, and then provide public guidance.

Developers should treat the release as an operational event. Subscribe to authoritative project channels, map fixed releases to deployed versions, prioritize externally reachable systems, and preserve evidence that updates completed. For PHP, the official PHP announcement stream gives maintainers and operators a primary channel to check alongside their package or platform provider.

Verification should test the deployed environment, not merely confirm that a ticket was closed. OpenAI published a PHP2Shell checker to support that final step. Teams should still review what any checker tests, run it within appropriate authorization boundaries, and record the version, configuration, and result.

Maintainers can prepare before the next disclosure by keeping supported-version inventories, reproducible test environments, fuzz targets, private reporting channels, release checklists, and named owners for security decisions. They should also define who can approve patches, who coordinates downstream notification, and how production teams prove rollout completion.

The takeaway: AI can shorten vulnerability discovery and assist with analysis, harnesses, patches, and tests, but security improves only when humans operate the full path from bounded investigation through upstream review, coordinated release, and verified deployment.

Gemini 4 Argon: Inside Google DeepMind's New Flagship Reasoning Architecture

Prabhakar Chaudhary — Thu, 22 Oct 2026 16:52:16 +0000

The landscape of frontier artificial intelligence has shifted again. In a month dominated by high-profile releases from the industry’s biggest labs, Google DeepMind has introduced Gemini 4 Argon. Positioned as a flagship reasoning model, Argon represents a departure from the general-purpose multimodal scaling of previous generations, focusing instead on the intersection of autonomous agency, complex mathematical problem-solving, and large-scale software engineering.

While the "Argon" moniker might suggest a noble, inert gas, the model’s capabilities are anything but static. By integrating a native 1-million token context window with an optimized architecture for multi-step logical deduction, DeepMind is signaling that the era of simple prompt-response interaction is giving way to persistent, goal-oriented reasoning.

The Architecture of Reasoning

At the core of Gemini 4 Argon is a refined transformer-based architecture that prioritizes "test-time compute" or "inference-time reasoning." Unlike standard autoregressive models that generate the next token based solely on statistical likelihood, Argon utilizes a hidden reasoning chain that allows it to explore multiple solution paths before committing to a final output. This process is particularly evident in its performance on complex benchmarks like the International Mathematical Olympiad (IMO) and advanced competitive programming challenges.

The shift toward test-time compute marks a significant evolution in model design. By allocating more FLOPS during the inference phase, Argon can effectively "double-check" its own logic. This is achieved through a mixture of verifiers and search-based exploration—essentially a reinforcement learning framework that has been baked into the model's weights. Developers familiar with Chain-of-Thought (CoT) prompting will find Argon’s behavior both familiar and significantly more robust. Instead of requiring the user to explicitly ask the model to "think step-by-step," the model’s internal weights have been trained on curated reasoning traces. This "native reasoning" reduces the frequency of logical "hallucinations"—where a model follows a correct premise to an incorrect conclusion—by validating intermediate steps against an internal world model.

Scaling Context Without Sacrificing Precision

One of the most significant technical achievements in Gemini 4 Argon is the maintenance of high retrieval accuracy across its 1-million token context window. In many large-context models, "needle-in-a-haystack" performance degrades as the context fills up, often leading to a loss of focus on the middle portions of the input. Argon addresses this via a new sparse attention mechanism that dynamically weights the relevance of distant tokens based on the current reasoning sub-task.

This sparse attention isn't just about efficiency; it's about semantic filtering. By identifying which segments of the context are most likely to contain relevant information for a specific logical sub-query, the model can maintain a sharper focus on high-entropy data. For developers, this means the ability to ingest entire codebases, multi-thousand-page technical specifications, or long-form legal documents without the need for complex Retrieval-Augmented Generation (RAG) pipelines. In recent tests, Argon demonstrated a near-perfect recall rate even when the target information was buried deep within 800,000 tokens of noise. This capability is not just about memory; it is about the model’s ability to synthesize disparate pieces of information located at opposite ends of the context window to solve a single problem. This synthesis is particularly useful when resolving dependencies in large microservice architectures where a single change in a utility library might have cascading effects across multiple services.

From Software Engineering to Robotics

DeepMind has also branched Argon into specialized deployments. "Gemini 4 Argon for Robotics" is a variant that unifies perception and action into a single reasoning loop. By treating visual inputs as tokens in the same semantic space as text and code, the model can reason about physical constraints in real-time. For example, when tasked with a complex manipulation chore in a lab environment, the model does not simply follow a hard-coded script; it reasons about the geometry of the objects it encounters and adjusts its plan if an initial attempt fails.

In the realm of software engineering, Argon serves as an autonomous agent capable of more than just boilerplate generation. It can perform repository-wide refactoring, identify subtle race conditions in concurrent code, and even suggest architectural improvements based on best practices. Because it can "see" the entire project at once, its suggestions are contextually aware in a way that smaller-context models cannot match.

Why This Matters for the ML Practitioner

The release of Gemini 4 Argon comes at a time when the industry is debating the "scaling laws" of pure data. Argon suggests that the next leap in performance will come not just from more parameters or more GPUs, but from more efficient ways of utilizing computation during the inference phase. By baking reasoning into the model itself, Google is lowering the barrier for building reliable autonomous agents.

For ML practitioners, the takeaway is clear: the focus is shifting from "how to prompt" to "how to orchestrate." As models become better at handling their own internal logic, the developer’s role becomes one of defining the goal, providing the necessary context, and auditing the reasoning traces to ensure alignment with business requirements.

Practical Takeaway: What to Watch

As we move into 2027, the integration of models like Gemini 4 Argon into developer workflows will likely center on "agentic IDEs" and "autonomous MLOps." The ability to run a model that can reason through a 1-million token codebase means that the time between identifying a bug and deploying a verified fix is shrinking.

ML engineers should begin experimenting with Argon’s reasoning transparency features. By inspecting the model’s intermediate steps, teams can gain a better understanding of where their data might be confusing the agent and where the model’s world view deviates from reality. This transparency is the key to moving from experimental "chatbots" to production-grade "thinking systems."

Sources and Further Reading

Primary Source: Google DeepMind: Introducing Gemini 4 Argon
Supporting Source: SpyingBee AI Updates: DeepMind's October 2026 Roadmap
Supporting Source: ExplainX: Analyzing the Flagship Reasoning Models of 2026

MemT-1.0: How Baking Memory Into the Model Itself Cuts KV Cache Storage by 66x

Prabhakar Chaudhary — Wed, 21 Oct 2026 16:04:41 +0000

MemT-1.0: How Baking Memory Into the Model Itself Cuts KV Cache Storage by 66x

Every time a large language model processes a long conversation, it pays a tax. The key-value (KV) cache — the data structure that stores intermediate attention states for every past token — grows linearly with context length. At 128,000 tokens, a single request from a model like Llama 3 70B can consume roughly 40 GB of GPU memory just for the cache, competing directly with model weights for scarce high-bandwidth memory. Scale that to a million tokens and the numbers become impractical for most deployments.

The standard responses to this problem — quantizing the cache, compressing it with grouped-query attention, or offloading it to CPU — are engineering patches on a structural issue. A new paper from October 2026, MemT-1.0: Towards Memory-Native Evolution, proposes a different answer: make memory an intrinsic part of the model's computation rather than an external appendage.

The Three-Way Memory Problem

Foundation models currently handle memory in one of three ways, each with a distinct cost profile.

Parametric memory bakes knowledge into model weights during training. It is fast at inference time but expensive to update — you cannot cheaply teach a model what happened last Tuesday without fine-tuning.

Retrieval-augmented generation (RAG) keeps knowledge outside the model in a vector store and fetches relevant chunks at query time. It is flexible and updatable, but it adds retrieval latency, requires a separate indexing pipeline, and the model must still fit the retrieved context into its attention window.

KV cache / long context simply extends the attention window and stores every past token's key-value pair. It is the most faithful representation of history but the most expensive: storage scales linearly, and re-transmitting the full context at every decoding step creates a bandwidth bottleneck that grows with sequence length.

MemT-1.0 targets a fourth option: a continuously updated internal state that accumulates information across interactions without growing unboundedly.

The Architecture: Gated DeltaNet-2 and Mixture-of-Storage

MemT-1.0 builds on Metis, a prior memory-native backbone, and introduces two core mechanisms.

Gated DeltaNet-2 performs stable token-level memory updates within each inference turn. Rather than appending new tokens to an ever-growing KV cache, the layer updates a fixed-size recurrent state using a gated delta rule. The key property is stability: the update mechanism is designed to avoid the state explosion or forgetting problems that plagued earlier linear-attention architectures. Because state updates live inside the layer mathematics, the model stops paying the bandwidth penalty of re-transmitting the full past at every decoding step.

Mixture-of-Storage scales native memory capacity beyond what a single recurrent state can hold. The mechanism allocates different types of storage — think of it as a learned routing between fast, compact states and slower, higher-capacity memory slots — allowing the model to retain information over longer interaction histories without reverting to full attention KV entries.

The training pipeline runs in two stages: continued pre-training followed by post-training, using approximately 34 billion tokens of training data. The team released two model variants:

MemT-1.0-Nano: 36B total parameters, 3B active, 10M dynamic memory. Backbone: Qwen3.6-35B-A3B.
MemT-1.0-Flash: 288B total parameters, 17B active, 336M dynamic memory. Backbone: DeepSeek-V4-Flash.

What the Numbers Show

The headline result is the storage comparison at 128K tokens of history. MemT-1.0-Nano achieves approximately 1.86x end-to-end inference speedup and a 66x reduction in persistent storage relative to a full-context KV cache. Time-to-first-token speedups reach 2.44x at 128K tokens and 4.19x at 256K tokens — gains that compound as context length grows.

On eleven memory benchmarks spanning MemBench, PersonaMem, LoCoMo, LongMemEval-S, MemoryAgentBench, and BEAM-1M, MemT-1.0-Flash averages 50.6 and Nano averages 47.0 — reported as 12.5 and 8.9 points above the strongest parametric baselines respectively. Both models remain competitive with retrieval-augmented methods while largely preserving general capabilities on MMLU-Pro, GPQA, IFEval, and LongBench (within one to two points of their teacher backbones).

The agentic results are notable: by internalizing retrieved procedural experience, MemT-1.0 gains 7.6 points on SWE-bench Verified and 8.3 points on SWE-bench Multilingual over the respective backbones without experience. For agents that need to accumulate task-specific knowledge across multi-step workflows, this is a meaningful signal.

Why This Matters for Practitioners

The practical implication is straightforward: if you are serving long-context workloads at scale, the KV cache is a cost center. Techniques like prefix caching and MLA compression have already moved the needle, but they are optimizations on top of the same structural design. MemT-1.0 changes the design itself.

For persistent assistants and personalized agents — systems that need to remember user preferences, past interactions, and accumulated procedural knowledge — native memory offers something RAG cannot easily provide: the memory is part of the model's computation, not a lookup that must be explicitly triggered and correctly retrieved. The model does not need to decide whether to consult an external store; the relevant history is already encoded in its internal state.

There are open questions worth tracking. The paper does not fully specify how the model handles memory isolation across users or tasks — a critical concern for multi-tenant deployments. The recurrent state can accumulate errors over long interaction chains, and the mechanisms for safe forgetting or state correction are not detailed in the abstract. The code is released under a PolyForm noncommercial license (the paper under CC-BY-NC-SA), so commercial deployment requires a separate license inquiry with the MemTensor team.

The Broader Trend

MemT-1.0 arrives alongside a cluster of related work: KronLSTM from JKU Linz, RPMem for cross-session parametric memory, and ARM (Attention with Routed-Memory). The pattern is consistent — the field is moving toward architectures where memory is a first-class citizen of the model rather than an afterthought managed by the serving infrastructure.

The 66x storage reduction is a structural cost advantage, not a tuning tweak. With public weights available on Hugging Face and SGLang session persistence support, every team running long-context inference now has a concrete benchmark target to evaluate against their current stack.

Whether MemT-1.0 becomes a production standard or a research stepping stone, it makes the case clearly: the KV cache is not the only way to give a model a past.

Primary source: MemT-1.0: Towards Memory-Native Evolution (arXiv:2610.26825, October 2026). Supporting coverage: AI Weekly, Hugging Face Papers, cctest.ai.

Reflection AI's Beam: How a 501B MoE Model Learns to Think in Fewer Tokens

Prabhakar Chaudhary — Mon, 19 Oct 2026 16:06:27 +0000

Reflection AI's Beam: How a 501B MoE Model Learns to Think in Fewer Tokens

When most open-weight model releases compete on raw benchmark scores, Reflection AI took a different angle with Beam: instead of asking "how capable can we make this model?", they asked "how much reasoning compute does a correct answer actually require?" The result is a 501-billion-parameter sparse Mixture-of-Experts model that achieves frontier-level coding and agentic performance while using roughly 3–4× fewer inference tokens than comparable open models.

Released on October 5, 2026, Beam is available under the Apache 2.0 license with weights on Hugging Face, making it one of the most capable fully open models for software engineering workloads.

The Core Idea: Inference Compute as a Scaling Axis

Most post-training work focuses on making models smarter at a fixed inference budget. Reflection's approach treats inference compute itself as the thing to optimize. Their key insight is that a model trained to solve problems correctly and concisely is more useful in production than one that reaches the right answer after generating thousands of unnecessary tokens.

To operationalize this, they introduced a controllable length penalty into the reinforcement learning reward function. The penalty discourages unnecessary token generation while still rewarding successful task completion. The result is a model that learns to match its reasoning depth to the actual complexity of the task at hand.

Reflection estimates inference efficiency using a simple formula: generation compute ≈ 2 × active parameters × average tokens generated per attempt. By this measure, Beam achieves comparable solution quality to GLM-5.2 while using 3–4× less generation compute — not by being less capable, but by being less verbose.

Architecture: Stability First

Beam's architecture is a sparse MoE with 501.3 billion total parameters and 23.4 billion active parameters per token. That active-parameter count is what matters for inference cost: each forward pass only activates a small fraction of the full model.

Several design choices prioritize numerical stability, which matters especially for downstream RL training:

Interleaved local and global attention layers balance short-range and long-range context processing
Fine-grained routed experts with near-uniform utilization — the busiest expert carries only 1.04× the average load at the end of pretraining, which prevents the expert collapse that plagues many MoE models
Controlled residual stream to keep gradient flow stable across the full depth of the network

The model was pretrained on 23.8 trillion high-quality tokens, then put through one of the largest RL runs conducted by an open-weight lab: over 100 million rollouts generated across 10,500 NVIDIA GB300 GPUs over four weeks, using asynchronous policy gradients to handle the staleness that comes with training at that scale.

How the Length Penalty Actually Works

The RL training unfolded in two distinct phases, which Reflection describes as a moving Pareto frontier between capability and token usage:

Phase 1 — Contraction: Early in RL, the model prioritized efficiency. It learned to solve the same problems it could already solve, but with shorter completions. Performance held steady while token counts dropped.

Phase 2 — Expansion: As the model acquired more advanced agentic capabilities — multi-step tool use, web browsing, complex code navigation — completion lengths grew again. But this growth was purposeful: the additional tokens were doing real reasoning work, not padding.

The net effect is a model that has internalized when to think briefly and when to think deeply. Users can also control this explicitly via a reasoning_effort parameter with five settings from low to max, letting developers tune the compute-quality trade-off for their specific workload.

Speculative Decoding with DFlash2

To address wall-clock latency separately from token count, Reflection released a companion model: DFlash2, a 1.93-billion-parameter draft model designed for speculative decoding.

DFlash2 generates candidate token sequences ahead of the main Beam model, which then verifies them in batches using rejection sampling. This preserves the exact output distribution of the full model while accelerating generation by 1.72× to 3.03× depending on task and hardware configuration. The NVFP4 version of Beam with DFlash2 shows the largest speedups, making it practical to run on eight-GPU configurations.

Benchmark Performance

On software engineering benchmarks, Beam posts competitive numbers:

SWEBench Verified: 81.6% (BF16)
SWE Bench Pro v2-Hard: 76.5% (BF16)
Terminal Bench v2.1: 80.1% (BF16)
GPQA Diamond: 90.6% (BF16)
MCP Atlas (tool calling): 80.8% (BF16)

These numbers are notable — independent analysis from Marktechpost confirms the efficiency claims — because they come from a model that is simultaneously optimizing for fewer tokens. The usual assumption is that token efficiency and task performance trade off against each other; Beam's results suggest that with the right RL setup, you can improve both simultaneously — at least up to a point.

Deployment Options

Beam is available in BF16, FP8, and NVFP4 formats, with deployment support via Amazon SageMaker JumpStart and Kubernetes clusters running vLLM or SGLang. The Apache 2.0 license explicitly permits fine-tuning, quantization, distillation, and model merging, which is a meaningful contrast to some open-weight releases that restrict downstream use.

The 1-million-token context window makes it viable for codebase-wide tasks — full repository analysis, large-scale refactoring, or long-horizon agentic workflows that need to hold a lot of state in context. A detailed technical breakdown from Xenospectrum covers the serving infrastructure in more depth, including the fused kernels Reflection developed to improve throughput on multi-GPU setups.

What This Means for Practitioners

The practical implication of Beam's approach is cost. If a model uses 3–4× fewer tokens to reach the same answer, API costs drop proportionally for generation-heavy workloads. For teams running large-scale agentic pipelines — automated code review, continuous integration agents, research assistants — that multiplier compounds quickly.

The reasoning_effort parameter also gives operators a lever that most models don't expose cleanly. Rather than choosing between a fast cheap model and a slow expensive one, you can run the same model at different effort levels and tune the trade-off per task type.

The open licensing and multi-format release (BF16/FP8/NVFP4) lower the barrier to self-hosting, which matters for teams with data-residency requirements or those who want to fine-tune on proprietary codebases.

Conclusion

Beam represents a coherent thesis about where open-weight model development should go: rather than simply scaling parameters or context windows, optimize for how much compute a correct answer actually requires. The combination of a length-penalized RL objective, a stable MoE architecture, and a speculative decoding companion model gives practitioners a tool that is both capable and economical to run. Whether this approach generalizes beyond coding and agentic tasks remains to be seen, but as a proof of concept for inference-compute efficiency as a first-class training objective, it makes a strong case.

Muse Spark Open 816B: Inside Meta's Largest Open-Weight Model

Prabhakar Chaudhary — Fri, 16 Oct 2026 16:04:19 +0000

Muse Spark Open 816B: Inside Meta's Largest Open-Weight Model

Meta Superintelligence Labs released Muse Spark Open on October 15, 2026 — an 816-billion-parameter sparse Mixture-of-Experts model designed for agentic coding and long-horizon reasoning. It is the largest open-weight model Meta has released to date, and it arrives with a genuinely interesting set of architectural choices worth understanding in detail.

Architecture: 816B Total, 42B Active

The headline number — 816 billion parameters — is the total weight count across all experts. At inference time, only 42 billion parameters are active per token. The model uses 62 transformer layers, each containing a MoE block with 256 experts, of which 8 are selected per token via learned routing. The hidden dimension is 8192, and the model uses grouped-query attention throughout.

This sparse activation pattern is what makes the model practical to serve. A dense 816B model would require roughly 1.6 TB of memory just for weights in BF16; the MoE structure means the compute per token is closer to a 42B dense model, even though the full parameter space is much larger.

For visual inputs, the model includes a dedicated 1.8-billion-parameter Perception Encoder that processes images and video frames before they enter the main transformer. Audio is handled by a separate encoder. All modalities produce text as output — the model is natively multimodal on the input side but text-only on the output side.

The 1M-Token Context Window

Muse Spark Open supports a 1-million-token context window, enabled by a hybrid attention architecture that alternates between sliding-window attention and full attention across layers. This design keeps the KV cache manageable: full attention at every layer for a 1M-token sequence would require roughly 266 GB of KV cache in BF16, which is prohibitive. The sliding-window layers reduce this significantly for most tokens, while the full-attention layers preserve global context at key positions.

The official deployment guide notes that the KV cache consumes approximately 250 KB per token in BF16 at full context, so practitioners running long-context workloads need to plan memory budgets carefully.

Reasoning Effort Levels

One notable design choice is the model's adjustable reasoning effort. The API accepts a reasoning_effort parameter with four levels: low, medium, high, and xhigh. The default is xhigh, which instructs the model to spend more compute on chain-of-thought before producing a final answer. Lower effort levels reduce latency at the cost of reasoning depth.

This is similar to the effort controls Anthropic introduced in Claude Haiku 5.5, and it reflects a broader trend: rather than shipping a single fixed inference mode, labs are exposing the compute-quality tradeoff directly to developers.

Where It Performs Well — and Where It Doesn't

Meta's benchmarks position Muse Spark Open as a tool-calling and agentic workflow specialist rather than a general-purpose coding leader. On MCP-Atlas, a benchmark for multi-step tool orchestration, it scores 89.2 — ahead of Qwen3.8 (84.5) and GLM 5.3 (84.2). On JobBench, an agent evaluation suite, it scores 61.8, well ahead of alternatives like Inkling (31.1).

The picture is different on raw coding benchmarks. Terminal-Bench 2.1 gives it 76.4, while Kimi K3 leads at 85.0. On DeepSWE 1.1, it scores 60 versus Kimi K3's 68. On the Artificial Analysis Intelligence Index, it sits at 39, behind GLM 5.3 (45) and Kimi K3 (44).

The pattern is consistent: Muse Spark Open is optimized for the orchestration layer — selecting the right tool, managing multi-step state, handling long contexts — rather than for the kind of single-shot code generation that Terminal-Bench measures. Whether that tradeoff is the right one depends on what you're building.

Hardware Requirements and Deployment Options

Running Muse Spark Open requires serious hardware. The BF16 checkpoint needs approximately 2 TB of VRAM, which means a cluster of 32 H200 GPUs or equivalent. The NVFP4 (4-bit) checkpoint brings this down to roughly 530 GB, validated on four NVIDIA Blackwell Ultra GPUs or eight H200s.

For teams without large GPU clusters, Meta has demonstrated the model running on two Apple M5 Ultra Mac Studios (512 GB of unified memory each) connected via Thunderbolt 5 with RDMA, using the MLX framework. AMD Instinct MI355X and MI350X are supported via FP8 and MXFP4 quantization. Unsloth provides 2-bit GGUF quantizations that fit into approximately 300 GB of combined system memory.

Supported inference runtimes at launch include SGLang, vLLM, TensorRT-LLM, Hugging Face Transformers, and llama.cpp.

Speculative Decoding: DFlash V2 and DSpark

Meta ships two dedicated draft models for speculative decoding, one per inference framework. DFlash V2 is designed for SGLang and proposes blocks of tokens in a single CUDA graph pass; on AMD MI355X hardware, it reportedly achieves 3.7× throughput improvement with an average of 6.4 accepted draft tokens per step. DSpark is designed for vLLM and proposes up to seven tokens per pass, with architecture specifically tuned to remain efficient at 1M-token context lengths by leveraging the model's sliding-window attention layers.

Speculative decoding at this scale is non-trivial — the draft model needs to be fast enough that the overhead of running it doesn't eat the gains from fewer full-model forward passes. Shipping framework-specific draft models rather than a single generic one suggests Meta has tuned these carefully for each runtime's batching and scheduling behavior.

Licensing: Open-Weight, Not Open-Source

Muse Spark Open is released under the Muse Model License, which is worth reading carefully before building on it. The key points:

Any product using the model must display "Built with Muse Spark Open" prominently.
Companies offering the model as an API service (Model-as-a-Service) with gross revenue exceeding $20 million in a 12-month period must enter a separate written agreement with Meta.
Fine-tuning and redistribution are permitted within these constraints.

This is the same general structure Meta used for earlier Llama releases — open enough for most research and product use cases, but with guardrails for large commercial API providers.

Practitioner Takeaways

Muse Spark Open is a serious model for teams building agentic systems that need strong tool orchestration and long-context handling. Its MCP-Atlas lead suggests it has been specifically optimized for the kind of multi-step, tool-heavy workflows that are increasingly common in production AI systems.

The hardware bar is high. Unless you have access to a multi-GPU cluster or a pair of M5 Ultra Mac Studios, you'll be working with quantized versions, which may shift the benchmark picture. The NVFP4 checkpoint is the practical starting point for most teams.

The benchmark gaps on Terminal-Bench and DeepSWE are real and worth factoring in if your use case is primarily single-shot code generation rather than agentic orchestration. For that workload, Kimi K3 or GLM 5.3 may still be the better choice.

What's notable about this release is less the raw parameter count and more the deliberate architectural choices: the hybrid attention for 1M-context efficiency, the framework-specific speculative decoding draft models, and the adjustable reasoning effort levels. These are the kinds of engineering decisions that reflect a model built for deployment, not just for benchmark headlines.