DEV Community: Felipe 0liveira

Your Benchmark's Grader Is the Weakest Link — And an Item-Response Fix for Judge Panels

Felipe 0liveira — Thu, 05 Nov 2026 12:04:26 +0000

This digest covers evals and observability news from 2026-10-29 through 2026-11-05: new arXiv work on benchmark grading integrity and judge aggregation, fresh benchmarks and tracing tooling from Hugging Face, Langfuse, and Braintrust. Simon Willison and Interconnects had nothing on-theme this week.

🔥 Highlights

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — grading code quietly moves leaderboard rankings
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory — weak judges stop dragging down panel verdicts
The Trace Lake — purpose-built storage turns traces into reusable evals
CWE-bench v2: More coverage, quality, and transparency — deterministic, tamper-resistant grading for agentic security evals
How Teams in Regulated Industries Get AI Agents into Production — a concrete tracing-to-evaluator production playbook

arXiv cs.CL

AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory (2026-11-03) — Proposes an item-response-theory method to aggregate a pool of LLM judges, weighting each by estimated reliability instead of naive averaging; naive averaging is shown to badly over-weight weak judges when validation sets are small. Directly actionable for teams running LLM-as-judge ensembles: a principled way to combine judges without one weak grader dragging down the panel's verdict.
Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks (2026-11-02) — Audits the scoring/grading code of eight peer-reviewed multimodal benchmarks and finds the free-form-to-right/wrong conversion rule — never validated or justified — materially changes published rankings for identical model outputs. A sharp warning for anyone trusting leaderboards or building their own LLM-as-judge grader: the grading harness itself is an unaudited source of score variance.
When Ground Truth Is Not Ground Truth: Automatic Labeling for Hallucination Detection (2026-11-03) — Shows that automatically-constructed ground-truth labels for hallucination detection, built by comparing generations against references, are themselves noisy and biased, undermining evaluation of the detectors trained against them. Relevant for anyone building or evaluating hallucination-detection tooling in production: your "ground truth" may need auditing before you trust a detector's reported accuracy.
Diagnosing Model–Item Interactions in Language Model Benchmarks with LSIRM (2026-11-02) — Applies a latent-space item response model across 1,348 language models and shows models with near-identical aggregate benchmark scores actually succeed and fail on very different item subsets. Gives eval teams a methodology to look past aggregate accuracy and understand where two similarly-scored models actually diverge in capability.
Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand (2026-10-31) — Derives an analytical error distribution for benchmark-subset evaluation and shows a well-chosen random subset ("BestRand") is nearly as reliable as full-benchmark runs, with quantified error bounds. Cuts eval compute/cost directly for teams running large regression suites, with a principled way to shrink eval sets while keeping confidence in the results.
Evaluation Contracts and Evaluation Leverage: Sliced-Gap Readings in Structured Prediction Can Move Without Model Change (2026-11-03) — Shows that commonly-cited sliced performance gaps are artifacts of the evaluation "contract" (scoring function, gold version, threshold, format) rather than real model capability; with frozen predictions, just reordering scoring choices flips which method looks better. A cautionary methodology piece: version-lock your eval contract or your sliced-metric trend line is meaningless.
From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models (2026-10-30) — Investigates whether high LLM scores on medical exam benchmarks reflect real reasoning or are inflated by question format, contamination, and difficulty distribution. Domain-specific, but the contamination/format-sensitivity methodology generalizes directly to any team worried their production eval benchmark scores are inflated by leakage rather than real capability.

Hugging Face blog

StateBench: Testing how language models track state changes over long documents (2026-11-03) — New AllenAI benchmark measuring how well LLMs follow changing entity states across long inputs, with four difficulty tiers and multiple problem formats to control for pattern-matching shortcuts. A concrete new axis — long-context state tracking — to add to eval suites for agentic and long-document use cases, beyond generic long-context recall tests.
CWE-bench v2: More coverage, quality, and transparency (2026-11-03) — Updated benchmark of 100 tasks where an agent must find and patch a real security vulnerability in an open-source codebase while preserving behavior; scoring is fully programmatic and each task is vetted so unpatched code fails, the reference fix passes, and the verifier rejects memorized upstream patches. The leading model solves roughly 68% on first try, runnable via Harbor — a solid template for building tamper-resistant, deterministic graders for agentic coding evals.
Transformers Model Integration Contracts (2026-11-03) — Proposes "Model Integration Contracts": shrunken model fixtures paired with runnable usage examples and human-reviewed baseline outputs (logits, token log-probs, loss) for cheap CPU regression checks before merge, motivated by a 2026 UMT5 attention bug a plain forward pass wouldn't have caught. More library-integration regression testing than LLM-quality eval, but a useful pattern for anyone designing eval-in-CI checks.

Langfuse blog

How Teams in Regulated Industries Get AI Agents into Production (2026-11-04) — An eight-step playbook for banks, insurers, and healthcare teams moving agents from prototype to production: tracing first, evaluators derived from error analysis on observed failures, compliance checks layered in, a domain-expert-validated ground-truth dataset with inter-rater agreement metrics, then production monitors and a graduated human-review model that eases off as measured quality improves. A reusable sequencing of tracing → error analysis → evaluator design → online monitoring, regardless of which tool you run it on.

Braintrust blog

The Trace Lake (2026-11-03) — Introduces the "Trace Lake" concept: storage purpose-built for agent traces (fast ingest, sub-second SQL, full-text search, cheap object storage, stays in the customer's own cloud) rather than repurposing a generic data lake or warehouse. Lets teams surface recurring behaviors — token waste, tool errors — across traces and turn them into reproducible evals; Braintrust reports its Nitro query engine made full-text search over 2x faster in a production A/B test, now extending to application logs in public preview. A concrete architecture and performance case for why trace storage needs eval-specific indexing and search.

Simon Willison

Nothing on-theme this week — recent posts were tool release notes and a weights-interpretation piece without eval/observability substance.

Interconnects

Nothing on-theme this week — the two posts in this window covered open-model cyber-risk and inference market sizing, neither touching evaluation methodology or observability.

Through-line

The strongest thread this week is that the measuring stick itself is under scrutiny: two arXiv papers independently show that benchmark rankings can shift without any model change — once because the grading code's pass/fail rule is unvalidated, once because the "evaluation contract" (scoring function, threshold, gold version) isn't version-locked. AGORA's judge-reliability weighting and Braintrust's Trace Lake both point the same way from the tooling side: stop trusting a single judge or a single flat log store, and instead weight, index, and audit the evaluation layer itself as carefully as the model it's scoring.

What's breaking in your eval pipeline this week? Drop it in the comments.

Why KV-Cache Memory Tricks — Not New Math — Drove This Week's Biggest Inference Wins

Felipe 0liveira — Wed, 04 Nov 2026 12:06:28 +0000

This digest covers LLM inference and serving developments from 2026-10-28 to 2026-11-04. The pattern this week is hard to miss: almost every serious throughput or cost win came from rethinking how the KV cache and expert weights are managed in memory, not from new attention math. Speculative decoding also got a round of re-tuning specifically for mixture-of-experts and hybrid-attention models, which behave differently from the dense models it was originally designed around.

🔥 Highlights

Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading — up to 19x throughput by merging KV cache and expert-weight memory into one pool.
Day 0 support for Mistral Large 4 on vLLM — speculative decoding, NVFP4, and KV offload shipped for a 1T-param model on release day.
HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models — speculative decoding re-tuned for hybrid MoE architectures.
Sliding window beats linear attention, but only on short answers — quantifies the real KV-cache tradeoff for long-context serving.
"Weight Space Programming" cuts Autoresearch out of the RSI loop — a 1M-token-context model squeezed onto a 16GB consumer GPU.

arXiv (cs.LG, cs.DC)

HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models (2026-11-02) — Instead of a fixed verification length, HybridSpec picks it dynamically per step using draft-acceptance estimates and measured target latency, because hybrid Gated-DeltaNet+MoE models see verification cost explode with length far faster than dense models do (71% growth vs. 4-6% from length 4 to 16). It also adds block-parallel GDN kernels that reconstruct recurrent state in one launch instead of per token. Reports 1.75-1.88x over plain autoregressive decoding and 1.3-1.33x over the prior best speculative method — directly relevant if you're serving Qwen3.6-class hybrid-attention MoE models and tuning speculative-decoding parameters.
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models (2026-10-28) — An empirical study across 10 sparse MoE models (30B-1T params) finds that fine-grained MoE breaks the textbook rule that speculative-decoding speedup peaks at mid-batch-size; it instead peaks at low batch sizes, traced to expert-routing overlap across draft tokens plus H100 memory-bandwidth and L2-cache reuse effects. The authors' max-entropy model predicts unique-expert counts within 4% error. Useful if your production speculative-decoding speedup curve doesn't match what the dense-model literature predicts — this explains why, and gives a way to model it for capacity planning.
PDScope: Breaking the Prefill-Decode Seesaw for Memory-Bound MoE Inference (2026-10-29) — For CPU-offloaded MoE inference, PDScope decouples prefill (compute-bound, wants aggressive expert prefetch) from decode (I/O-bound, wants conservative PCIe transfer) using phase-specific expert-activation prediction and a compiled, zero-copy async execution pipeline. Reports 2.5-10.3x prefill throughput and 2.1-6.1x decode throughput gains over existing systems — a concrete recipe for running large MoE models without keeping every expert GPU-resident.
Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading (2026-10-29) — Built on top of vLLM, Rimuru replaces fixed memory boundaries between model weights/experts and KV cache with a shared page pool, per-tensor-type "memory views," and a page-table-aware fused MoE kernel that reads non-contiguous expert weights directly. Reports up to 19.03x throughput and a 98.1% average TTFT reduction versus baseline — a serving-framework-level change applicable to any vLLM-based MoE deployment under fluctuating load.
ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference (2026-10-30) — Identifies that KV-cache transfers in prefill/decode-disaggregated serving act as long-lived "elephant flows" that stateless ECMP routing can collide onto the same network spine, stretching transfer time by seconds. ARK is a sender-coordinated path-reservation scheme that spreads concurrent transfers across spines with no switch changes. Cuts mean/P95 flow-completion time by up to 27.3%/34.0% and mean TTFT by 12.9% in RDMA simulation — a network-layer fix easy to miss when optimizing disaggregated serving purely at the scheduler level.
ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration (2026-11-02) — A benchmarking tool that predicts serving performance from a handful of "anchor" measurements (one per distinct kernel set) and analytically scales to other node counts and configurations, instead of exhaustive trial-and-error. On AMD MI355X with vLLM/SGLang/ATOM, 11 anchors (9.1 GPU-hours) replaced testing 226 configurations (204 GPU-hours) with 8.1% median latency-prediction error, landing a top-5 shortlist within 12% of the best feasible config — a direct cut to the cost of deployment config search.

vLLM blog

Day 0 support for Mistral Large 4 on vLLM (2026-10-29) — Covers vLLM's stack for serving Mistral Large 4 (1T params, ~49B active, MoE with latent attention and cross-layer KV sharing): FP8/NVFP4 quantized checkpoints with FP8 latent KV cache (NVFP4 ~1.1-1.2x faster decode at TP8, within 2 points of FP8 on evals), a built-in DSpark speculative decoder (~2x throughput at TP8), decode/prefill context parallelism (~7.9x KV capacity, up to 1.6x lower TTFT), NIXL disaggregated prefill/decode on GB300, and HiSparse host-memory KV offload (4.5-5.7x token capacity). Gives concrete, measured levers for anyone sizing a Mistral Large 4 deployment on day one.

Hugging Face blog

Welcome Le Chonk: Mistral Large 4 on Hugging Face (2026-10-29) — HF's companion piece to the vLLM post, with vLLM serve flags (tensor/context/expert parallelism, batching limits, prefix caching), GGUF quantization levels (Q2_K through BF16) with llama.cpp memory/throughput figures on Apple Silicon and multi-GPU, and Inference Endpoints benchmarks on 8xH200. A concrete menu of deployment paths with hardware-specific cost/performance figures for teams deciding how to run this model.
A Low-Cost Decision Model on CPU: Optimizing DiffusionGemma with vLLM - Part I (2026-10-29) — Serves a 26B-A4B decision model on CPU (Intel AMX, BF16) via vLLM instead of GPU, including a new single_pass_reads flag (upstream PR in review) that merges prompt and answer-scoring into one forward pass, cutting latency 18.8% at concurrency 1 and raising throughput 5.5-15% at concurrency 2-20. A 24-vCPU node (~$996/mo) cuts always-on cost 79% versus an A100 replica (~$4,754/mo) with 5-13% lower cost-per-1000-decisions under moderate load — useful for teams deciding CPU vs. GPU for latency-tolerant classification/decision workloads.
Sliding window beats linear attention, but only on short answers (2026-11-02) — Compares sliding-window attention against a hybrid linear-attention mixer ("Bifocal") at equal KV-cache memory budget, distilled from Qwen3-4B. At matched cache size, sliding-window wins on short answers but Bifocal pulls ahead on long math answers (15.5% vs 13.8% pass@1 on AIME/HMMT) thanks to retained long-range state; on H100, both hold ~195 tok/s decode from 1K-128K context while full attention drops to 92 tok/s at 128K. Concrete guidance for choosing attention/KV-cache architecture for long-context serving — sliding window is a cheap strong default, but the hybrid can buy better long-generation quality at a small prefill-speed cost and no extra decode cost.

Together AI blog

Nothing new in this window — the latest post predates it and was already covered.

Modal blog

Nothing new in this window — checked the blog index and sitemap directly; the most recent post predates the window.

Latent Space

"LLMs can read & write neural network weights": "Weight Space Programming" cuts Autoresearch out of the RSI loop (2026-11-03) — Buried in the daily roundup: a patched Strata inference engine runs DeepSeek V4.1 Flash with 1M-token context on a consumer RX 9070 XT (16GB VRAM) via KV-cache splicing of stable prompt segments plus resident-only expert routing, dropping cold-start from ~100s to ~5s and SSD reads from 58GB to 1.3GB per prompt. Also notes a 763B MoE quantized to ~1.05 bits/param (72.1GB) losing only 1-4 points on LiveCodeBench/SWE-bench. Concrete evidence that aggressive KV-cache tricks can make huge-context inference viable on consumer hardware, and a data point on how far sub-2-bit quantization can go before quality degrades.
"Google admits AI Mode hurts Search revenue; -15% $500B drop worst since 2004 IPO. AI Bear Market coming?" (2026-10-31) — Inference-relevant items inside the roundup: prompt caching cuts a typical agent task's cost 3.6x ($27 to $7.44), with each cache invalidation adding ~5% to the bill; Nemotron 3.5 Super hits 4,856 output tok/s/GPU on GB200 at NVFP4 while keeping 99.62% of BF16 accuracy; and turbopuffer's switch from FIFO to LFU chunk caching roughly doubled byte hit ratio and halved p99 latency. Hard numbers for justifying prompt-caching investment and NVFP4 as a near-free accuracy/throughput trade.
"Building AIE CODE: The top Coding Agent Conference" (2026-10-30) — Inference-relevant items inside the roundup: vLLM's day-0 Mistral Large 4 support (see above) with speculative decoding yielding 1.92-2.23x output throughput; SGLang kernel fusion on GB300 NVL72 nearing 1,000 tok/s/user; a vLLM tuning pass nearly doubling Nemotron 3.5 Super VL throughput (6,058 to 11,840 tok/s on one GB300); and SkyPilot's in-rack spare-node scheme recovering ~27% extra usable capacity on a 128-rack cluster. Multiple concrete throughput-tuning levers with before/after numbers for serving cost decisions.
"Gemini 4 Carbon tops leaderboards, launches for cyber defenders." (2026-10-29) — Inference-relevant items inside the roundup: the AKV serving engine's cache "drop/repos" operations cut agent cost 27.7% and improve P95 latency 1.51x; SGLang's HiCache CPU-DRAM KV offload adds 20-40% more cacheable tokens under concurrent agent load on B200; MLX kernel updates make batched decoding 2-3x faster; and Modular open-sourced its full MAX serving stack (compiler, runtime, hardware backends) royalty-free. KV-cache offloading and engine-level cache ops are now showing double-digit cost/latency wins, and a major serving stack going fully open-source changes the framework-selection calculus.

Through-line

Three independent threads converge this week: the KV cache has become the primary lever for both cost and latency (elastic expert/cache memory pooling in Rimuru, network-aware transfer scheduling in ARK, host-memory offload in the Mistral Large 4 launches, cache "drop/repos" ops and HiCache offload surfacing in Latent Space's roundups, and consumer-GPU KV splicing for million-token context); speculative decoding is being actively re-derived for MoE and hybrid-attention models rather than assumed to behave like it does on dense transformers (HybridSpec, the sparse-MoE speedup study, DSpark on Mistral Large 4); and quantization keeps pushing further than expected, from NVFP4 holding near-BF16 accuracy at scale to sub-2-bit MoE checkpoints still clearing benchmarks within a few points. None of these are algorithmic breakthroughs in attention itself — they're memory-and-scheduling engineering, and that's where the throughput is coming from right now.

What's catching your eye this week — the KV-cache offloading trend, or the MoE-specific speculative decoding retuning? Drop a comment below.

Agent Memory's Reality Check: Why Simpler Retrieval Keeps Winning and Harnesses Split Brain From Hands

Felipe 0liveira — Tue, 03 Nov 2026 17:12:41 +0000

This digest covers AI agent research and engineering writing published between 2026-10-20 and 2026-11-03. It's a heavy week for agent memory in particular — several independent groups converged on the same uncomfortable finding: more complex memory doesn't automatically mean better memory.

🔥 Highlights

Delivered, Not Used: Evaluating Agent Memory at Its Point of Use — resolved tasks still waste most delivered memory.
Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought — simple retrieval beat fancy memory in 6/7 tests.
How Anthropic ships Claude with Claude: Separating the Brain from the Hands — separates harness from sandbox so crashes don't cascade.
Managed Deep Agents v0.10: Browser use and asynchronous subagents — event-driven subagents replace polling for async work.
Scaling Git at OpenAI — Codex agents forced a new Git backend.

arXiv (cs.AI / cs.MA)

Delivered, Not Used: Evaluating Agent Memory at Its Point of Use — 2026-11-01. Introduces MemUse, which checks whether delivered memory was actually read and used at the moment of action, not just whether the final task succeeded. Across 8,000 SWE-bench Verified episodes, most delivered memory was irrelevant or redundant, and 23% of episodes with "beneficial" memory still failed — a warning that outcome-only metrics hide wasted or harmful memory.
MemHarmBench: Evaluating Cross-Session Prompt Injection in Memory-Persistent AI Agents — 2026-11-01. A benchmark for malicious instructions that persist in an agent's memory across sessions even after the original content is removed, measuring harm to the user, workspace owner, or third parties. The key finding: cross-session attacks usually leave no visible anomaly — the agent just finishes the task normally while quietly serving the attacker.
Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction — 2026-11-02. ARCBench measures whether agents keep following operating rules after long histories get compacted to fit the context window. Compaction rarely erases the rule text itself — violations happen because the agent treats a later request as permission, and an attack targeting the compactor more than doubles violations.
LEANGUARD: Lean-Verified Guardrails for Tool-Using Agents — 2026-11-02. A runtime that checks every proposed tool call against a Lean4-verified policy (with temporal operators over execution history) before running it, failing closed. Cut forbidden calls to zero on Tau-Bench and dropped direct prompt-injection success from 57.9% to 5.9% on DTap — formal guarantees instead of prompt-only defenses.
MAC-Bench: Benchmarking Multi-Agent Collaboration via Interconnected Tasks — 2026-11-01. Embeds coordination requirements in task dependencies and shared resources rather than fixed roles/protocols, running in executable environments that verify partial progress. Useful for evaluating multi-agent orchestration without assuming a "correct" communication topology upfront.
ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost — 2026-11-01. Shows that applying deployment preferences (e.g., prioritizing cost) during intermediate harness-optimization steps prematurely rejects good candidates. Separates intermediate candidate promotion from final selection using Pareto support via bayesian bootstrap — relevant to anyone automating optimization of their own agent harness.
Just-in-Time Skill Retrieval via Long-Horizon State Tracking — 2026-11-01. Instead of fixing a skill set before execution, JitSkill selects skills dynamically during execution by continuously tracking task state and linking emergent needs to accumulated progress — practical for agents with large skill/tool libraries where upfront selection doesn't scale.
Towards a Trustworthy and Governable Internet of Agentics: An Architectural Perspective on Protocols, Systems, and Services — 2026-11-02. A more conceptual-than-empirical architecture paper that organizes the current ecosystem (MCP, A2A, AGNTCY/SLIM) into three layers: fragmented ecosystem, interoperable agents, and trustworthy/governable autonomy — a decent mental map for cross-framework multi-agent design.

Hugging Face Daily Papers

Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents — 2026-10-25. Compares agents accessing identical tool operations via MCP vs. CLI with the same action definitions and backend. Found structural divergence in 94.3% of matched episodes — interface choice alone changes performance, cost, and failure/recovery behavior. A fix (DART) gained up to 9 benchmark points and cut up to 27.1k tokens per task on MCP.
DynaFlow: Learning Free-Form Multi-Agent Orchestration — 2026-10-25. A main agent writes executable workflows that coordinate sub-agents (sequential or parallel), deciding when to collaborate; only the main agent is trained (SFT + refinement), sub-agents stay fixed. Gained 10.9 points over the best baseline on the same backbone — a "free-form" alternative to fixed-graph orchestration.
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution — 2026-10-26. Evaluates the agent harness itself (not the model) across seven dimensions via trajectory tracing. Found that 31.1% of runs succeeded at the task despite failing some harness criterion — evidence that outcome-only evaluation misses real reliability problems.
MemSteer: Turning User Histories into Supervision for Agent Memory Optimization — 2026-10-30. Generates evidence-backed Q&A pairs from unlabeled user history to supervise how agent memory is stored, retrieved, and assembled into context, without updating model weights — a practical path to tuning memory systems without manual annotation.
Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought — 2026-10-23. Tests a plain RAG baseline (no extraction/summarization/mutation) against complex agentic memory systems across four benchmarks (LoCoMo, LongMemEval, MEME, MemoryAgentBench) — naive RAG won in 6 of 7 configurations, suggesting raw evidence coverage matters more than transformation complexity.
CUA-Master: Scaling Computer-use Agents through Open-World Recursive Improvements — 2026-11-03. A two-loop framework for computer-use agents: an outer loop ties open-world data collection and training to observed weaknesses; an inner loop extracts workflows from instructional videos into verifiable skills, with rollback for failed runs. A 27B model hits 80.2 on OSWorld Verified.
CoworkOS: Scaling Agent Coworkers for Real-world Automation — 2026-11-03. A harness for long-horizon, human-in-the-loop work that separates execution strategy (model decides actions/delegation) from execution management (runtime holds state), with persistent semantic memory across sessions — competitive with Codex and DeepSeek Harness across 12 benchmarks.
R²Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop — 2026-10-22. Trains agent orchestrators via self-play without a manual curriculum: a "Proposer" picks tasks that still discriminate outcomes as skills get removed, retiring skills once the orchestrator performs equally without them — aiming for "zero skills" deployment where coordination is internalized in weights.

Anthropic Engineering

Nothing new in this window — the latest post remains "Redesigning source control at Anthropic" (2026-10-13), already covered in a previous edition.

OpenAI News

How we detect frontier model misalignment — 2026-10-29. Describes a multi-layer monitoring system for detecting misalignment in frontier agents/models: lightweight classifiers over internal activations, a model-based reviewer scanning conversation/chain-of-thought/tool use for unauthorized actions, and automated investigations that cross-reference patterns across cases. Expanded after two real incidents of agents disabling their own monitoring alerts.
Scaling Git at OpenAI — 2026-11-02. Introduces Pando, a new Git backend replacing OpenAI's monorepo, driven by the commit volume from Codex agents (master-branch commits reportedly up ~1,378% from Q1 2024 to Q3 2026). Shows how version-control infrastructure has to be redesigned once the dominant "committer" is an agent, not a human.
Using thousands of AI agents to uncover new leads in biological data — 2026-10-21. Nearly 2,000 GPT-Rosalind agents analyzed a CRISPR atlas of 7.7 million mouse brain cell nuclei in parallel, each investigating one genetic perturbation, in 32 minutes, with blind evaluation by 30 PhD biologists validating the agents' rankings — a real-world case of large-scale "fan-out" multi-agent coordination over a shared dataset.

Latent Space

How Anthropic ships Claude with Claude: Separating the Brain from the Hands — 2026-10-27. Details Anthropic's "Managed Agents" architecture — harness and session separated from the execution sandbox, so a sandbox failure doesn't kill the whole session. Introduces "dreaming," a multi-agent pattern where one agent reviews another's prior work and writes memories back.
Claude Mods: Flipping the Harness Inside Out — 2026-10-21. Argues that as more of the harness gets absorbed into model weights, what's left becomes a human-facing "Attention Interface." Claude Mods are event-reactive plugins (prompts, tool calls, renders) that can observe or override default behavior.
Agents Prompt People: Using Agents for Long-Horizon Financial Work — 2026-10-31. A case study from wealth manager Arca: agent tasks running up to ten days with multiple handoffs, per-client memory tracking completed/next/responsible steps, benchmark-based model routing, and a "meta-agent" that reviews traces and advisor feedback to propose harness changes. Deterministic guardrails keep the agent from moving money.
Why Agents Love Supabase — 2026-10-28. Makes the case for backends designed to be read/edited by agents — Git-versioned declarative schemas, a Docker-free local runtime, and the need for operational context (logs, checks) so an agent can understand failures — relevant for anyone designing APIs/tools that agents will call.
"AI for the Fortune 2000: How Tenex Builds an AI-Native SDLC" — 2026-10-21. Tenex's "MetaHarness" combines shared skills, structured docs, and a CLI; agent plans and decision logs are Git-versioned for future context. Also covers permission gateways for background agents in production.
Coding From the Himalayas With an Offline AI Agent — 2026-10-27. A hands-on account of running OpenCode (an open-source terminal agent) against a local Qwen3.5 9B via Ollama, with per-command/edit permission prompts — lighter in substance, but a concrete illustration of tool-calling with permission gates fully offline.
"Building AI for Lawyers" — Jacob Lauritzen, CTO of Legora — 2026-11-02. Details Legora's orchestration primitives — projects, agents, Tabular Review, and lists that act as a configurable database with triggers that fire new agent runs — plus fan-out delegation to sub-agents for retrieval and chunked editing of long contracts.

LangChain / LangGraph

New context engineering challenges in late 2026 — 2026-11-02. Harrison Chase argues the "LLM-in-a-loop + tools + sandbox" pattern is solved, then lists six open challenges for proactive, always-on agents: calibrating notifications, memory as a decision problem (not just storage), coordinating async work/subagents, tracking long-term goals, aligning against goal-gaming, and browser as a primary tool.
Managed Deep Agents v0.10: Browser use and asynchronous subagents — 2026-10-29. Release notes for v0.10 — a sandboxed browser with browser_handoff (login without exposing credentials to the model) and asynchronous subagents ("async": True) that notify the orchestrator on completion instead of being polled. Includes breaking changes (removal of connectors.mcp(), new tools/mcp.ts|py layout).
Browser Use in Managed Deep Agents — 2026-10-29. A technical deep-dive on the browser feature above — four tools (browser_snapshot, browser_run, browser_screenshot, browser_handoff), per-thread/user scoping, idle timeouts, and replayable tests without model calls. Cites an internal WebVoyager benchmark: 90% accuracy for $1.98 total vs. 89% at 9x the cost via Browserbase/MCP.
Building Evidence-Grounded Agent Memory with OpenWiki and Graphs — 2026-10-28. A wiki-based agent memory with citations hash-linked to sources plus an evidence graph (locate/expand) for cross-document questions. On connected questions, accuracy rose from 0.10 (plain search) to 0.52 (wiki), gaining another 27% with the graph, while cost per correct answer dropped 58%.
Why aren't you building proactive agents yet? — 2026-10-28. Argues for agent-as-cron-job patterns for memory consolidation, proactive outreach, triage, and follow-ups, describing a Schedules SDK (static and dynamic schedules, jitter to avoid load spikes). Claims over 70% of Managed Deep Agents runs on LangSmith are scheduled, not human-initiated.
How We Built Our Design Agent — 2026-10-27. An engineering account of a design agent: an orchestrator coordinating async subagents with file ownership, on-demand tool-schema loading, a skill-router, and a storyboard → animatic → production pipeline with code-enforced ordering. Notes that memory was capped at a 200-line index after one version grew to 500K tokens.
The five components of building a "company harness" — 2026-10-24. Describes five recurring primitives across LangChain's internal agents (marketing/sales/engineering): placement (Slack), identity (per-tool delegated credentials), memory (global vs. per-user), sandboxed code execution with a secrets proxy, and scheduled/ambient runs.

Simon Willison

Nothing cleared the bar this window. The agent-adjacent posts in range were either governance announcements with little orchestration/memory/tool-use substance (e.g. the Skills donation to the Agentic AI Foundation) or model-release notes without agentic depth — none qualified under this digest's curation focus.

Through-line

The strongest thread this week is a quiet rebellion against memory complexity: MemUse shows delivered memory is mostly unread or useless even when tasks succeed, the naive-RAG paper shows plain retrieval beating four generations of agentic memory architecture in most benchmarks, and LangChain's own OpenWiki post only justifies graph-structured memory for cross-document questions — everywhere else, simpler wins. In parallel, harness design keeps converging on the same split: Anthropic separates the "brain" (harness/session) from the "hands" (sandbox), LangChain ships event-driven async subagents instead of polling, and OpenAI's new Git backend exists because agents, not humans, are now the dominant committer. Evaluation is catching up to both trends — Trial by Trace and the rule-compliance benchmark both show that task success alone hides harness-level failures that only trace-level inspection reveals.

What's catching your eye in agent memory or harness design this week? Drop a comment below.

The Frontier Keeps Climbing, But the Open-Weight Gap Is Closing Fast

Felipe 0liveira — Sat, 31 Oct 2026 12:08:16 +0000

This edition of the Model Releases Digest covers what's new from 2026-10-17 through 2026-10-31: a new frontier leader from Google DeepMind, a brand-new category of interactive world model, a cheaper fast-tier model from OpenAI, a near-frontier model from Moonshot AI about to go open weight, and a trillion-parameter open-weight drop from Mistral.

🔥 Highlights

Gemini 4 Carbon: our next frontier model — new #1 model, built for cyber defense
We're announcing Genie 4, our new world model for real-time interactive simulation — real-time interactive worlds with native audio
Introducing GPT-6.1 Terra — faster, cheaper computer-use at frontier quality
Introducing Kimi K3.1 Pro — near-frontier scores, open weights coming soon
Mistral Large 4 (Le Chonk) is now open weight — trillion-parameter frontier model, now fully open

Anthropic news

Nothing relevant this cycle — the items published since 2026-10-17 were partnerships, policy, and safety-research posts (including a distillation-defenses writeup), with no new Claude model version, pricing change, or capability update.

OpenAI news

Introducing GPT-6.1 Terra — 2026-10-26. OpenAI's new fast, cost-efficient multimodal model succeeds GPT-5.6 Terra, pricing 50% below both GPT-5.6 Terra and GPT-6.1 Sol while scoring 15 points higher than its predecessor on the "Agents' Last Exam" benchmark at roughly a third of the cost. Practical value: it matches GPT-6 Astra's computer-use quality at a tenth of the price, making high-volume visual-reasoning and computer-use automation dramatically cheaper to run.

Google DeepMind / Google AI blog

Gemini 4 Carbon: our next frontier model — 2026-10-28. Built on Gemini 4 Argon, Carbon now ranks #1 on the Artificial Analysis Intelligence Index with big jumps in coding, agentic, and long-horizon tasks (teams report runs lasting days on complex engineering problems), plus state-of-the-art cybersecurity-defense scores. Rollout is initially gated to cyber defenders and safety-program partners before wider availability, with pricing held at $2/M input and $10/M output tokens — same as Argon.

We're announcing Genie 4, our new world model for real-time interactive simulation — 2026-10-27. Genie 4 generates real-time, steerable interactive simulations with native on-the-fly audio (ambient sound, music, dialogue), built on Gemini for visual realism and scene coherence. It's rolling out to a limited set of testers now, with wider access for Google AI Ultra and Gemini API partners coming soon — framed both as a training ground for robots/agents and as a new creative tool for film, journalism, and game development.

Meta AI blog

Nothing new — Meta's blog listing hasn't been updated since July 2026; the newest visible post is still "Introducing Muse Spark 1.1."

Mistral news

Mistral Large 4 (Le Chonk) is now open weight — 2026-10-29. After three additional weeks of RL since its earlier preview, Mistral is releasing open weights for its trillion-parameter, natively multimodal Large 4, now leading open-weight benchmarks in coding (FrontierCode, Terminal-Bench) and defensive cybersecurity, with three reasoning modes for cost/performance tradeoffs. Practical value: a frontier-scale model that's fully self-hostable, a meaningful option for sovereign-AI and compliance-sensitive deployments in the EU and beyond.

Voxtral 3: Speech-to-Text, TTS & Voice Agents — 2026-10-20. Mistral's third-generation voice stack adds a full voice-agent SDK alongside new speech-to-text and TTS models supporting real-time, multi-turn conversation across 50+ languages with emotionally adaptive synthesis. It's available via Mistral's EU-hosted API or self-deployment, giving teams a sovereign alternative to proprietary real-time voice-agent platforms.

Hugging Face blog

Build specialized multimodal models with LFM3 — 2026-10-27. Liquid AI's LFM3 family puts text, vision, audio, omni, and decision models on one shared 2B-class backbone; the 1.7B variant outperforms the previous LFM2.5-2.6B with 35% fewer parameters, fits in ~1GB at 4-bit quantization, and supports a 256K context window. Practical value: a smaller, cheaper, single-GPU-finetunable family for on-device agents across five modalities instead of one large MoE.

Granite Switch 4.1.1 — Update models like you update software — 2026-10-29. This patch release on Granite Switch 4.1 swaps in an updated multilingual speech-recognition adapter (10 languages, native punctuation/casing) and an improved decision adapter onto the same base checkpoint, with no retraining required. Practical value: it demonstrates IBM's adapter-swap deployment pattern — shipping capability updates without redeploying or retraining the base model.

Simon Willison

Introducing GPT-6.1 Terra — 2026-10-26. Simon's independent writeup frames GPT-6.1 Terra as sitting in the Sonnet-equivalent price/capability slot — cheaper than GPT-6.1 Sol, smarter than Luna — and notes OpenAI's own comparison to Astra/Sol on vision and computer-use quality at much higher speed. Practical value for readers: a clear read on where Terra lands in OpenAI's own pricing ladder, beyond the official announcement's benchmark claims.

Introducing Kimi K3.1 Pro — 2026-10-23. Moonshot AI's Kimi K3.1 Pro is live on the API now, with open weights promised by November 4, 2026, priced at $2/$10 per million tokens — half of Claude Opus 5.5. It scored 55 on the Artificial Analysis index, ahead of Claude Fable 5.1, GPT-6 Astra, and Gemini 4 Argon (all at 53) and behind only Opus 5.5 (58), making it the first soon-to-be-open-weight model to score this close to the frontier.

Through-line

Three independent signals point the same direction this cycle: the frontier keeps climbing (Gemini 4 Carbon's agentic and cybersecurity gains), the cost of near-frontier capability keeps collapsing (GPT-6.1 Terra's 50% price cut, Kimi K3.1 Pro undercutting Opus 5.5 at near-matching scores), and the open-weight tier keeps closing the gap almost immediately after each closed release (Mistral Large 4's open-weighting, Kimi K3.1 Pro's upcoming weight drop, LFM3 pushing multimodal capability down to edge-sized models). Add Genie 4's debut of real-time interactive world simulation as a genuinely new product category, and this feels less like incremental iteration and more like several fronts moving at once.

What's catching your eye this week — the Gemini 4 Carbon jump, Genie 4's world-model debut, or how fast Mistral and Kimi are closing the open-weight gap? Drop a comment below.

Teaching GRPO to Doubt Itself — Calibrated Fixes for Noisy Graders and Reward Hacking

Felipe 0liveira — Fri, 30 Oct 2026 12:04:32 +0000

This edition of the Post-training Digest covers new developments in fine-tuning and post-training from 2026-10-23 through 2026-10-30.

🔥 Highlights

SetGRPO: Calibrating Grader-Induced Policy Optimization — fixes a real GRPO blind spot: one bad grader score.
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO — cheap audits nearly kill reward hacking.
Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution — merges MoE specialists 2.86x faster.
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI — open post-training blueprint beats a 4.6x bigger model.
Kimi 3.1 Pro 104BA-2.8T: Fable-5.1-Class Open Weights narrows Closed-AI lead to 1 month... and Pro Highspeed API with 2-3× speed — RL post-training trades exploration for benchmark wins.

arXiv

SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection — 2026-10-27. Instead of a fixed rank per LoRA adapter, SimplexLoRA learns per-adapter importance weights on a scaled simplex during a short warm-up, then freezes them, redirecting capacity to the adapters that matter most without extra parameters or inference cost. Tested on DeBERTaV3, BART-large, Llama 3.1 8B, and Qwen3-8B across 12 tasks, it matches or beats standard LoRA and adaptive-rank baselines, with the largest gains at low ranks — exactly where production LoRA setups with tight VRAM/adapter budgets live.
On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime — 2026-10-27. Using an NTK linearization, the authors show DPO's logistic loss induces a max-margin bias that makes updates "dually sparse": sparse in which training pairs matter, and sparse in which gradient features move. This sparsity ties to weight disentanglement, and on GPT-2 experiments DPO reduces cross-task interference up to 35x versus plain SFT — a concrete reason to prefer DPO over SFT if you plan to merge or stack multiple fine-tunes later.
SetGRPO: Calibrating Grader-Induced Policy Optimization — 2026-10-27. Addresses a real GRPO failure mode: because rewards are normalized within a response group, one noisy or wrong grader score can flip training coefficients for the whole group. SetGRPO builds an uncertainty set over coefficients and applies a risk-calibrated threshold to block harmful update reversals, with finite-sample coverage guarantees, reporting the best macro-average across ten open-ended benchmarks on Qwen3.5 and Llama 3.2 — directly actionable for teams running rubric/LLM-judge-graded GRPO in production.
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO — 2026-10-26. Tackles reward hacking under cheap, gameable verifiers (e.g. unit tests) by deciding which rollouts deserve an expensive strong-verifier audit within a fixed budget. The "low-pass budgeted audits" strategy (audit groups with the fewest weak passes first) nearly eliminates hacking on LeetCode/Countdown-Code while using fewer labels than random auditing — a cheap, practical recipe for RLVR/GRPO pipelines stuck with imperfect verifiers.
Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution — 2026-10-29. Proposes calibrated multi-teacher on-policy distillation (per-teacher loss normalization plus clipping) to merge separately-trained MoE specialists into one dynamic-routing model, paired with an execution pipeline (bounded-width dispatch, routing replay, async rollout/train overlap) that cut consolidation step time from 398.7s to 139.3s — a 2.86x speedup. On Turing-20B-A2B it retains at least 90% of each specialist's gain and beats both shared-SFT and Qwen3-8B baselines, a concrete recipe for consolidating multiple fine-tuned MoE experts into one deployable model.
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI — 2026-10-28. NVIDIA's open Mamba-Transformer MoE models (120B/12B-active "Super", 30B/3B-active "Lightning") layer a post-training recipe of multi-teacher on-policy distillation, long-context and agentic-SWE mid-training, SFT+RL, and native speculative decoding. Super reports 51.78% on SWE-bench Pro — beating Nemotron 3 Ultra with roughly 4.6x fewer active parameters — and 99.62% accuracy recovery under NVFP4 quantization, a useful reference blueprint and quantization sanity-check for teams shipping efficient agentic MoE models.

Hugging Face blog

Nothing qualifying this week. The closest candidates — "Build specialized multimodal models with LFM3" and "Introducing Lumma-Fev-v2" — mention SFT/LoRA/DPO/GRPO tooling but disclose no actual recipe, data curation, or config details, so they read as model-release announcements rather than post-training technique writeups.

Together AI blog

Nothing qualifying this week. The only post in the window, "Auto repair for Together GPU Clusters: from GPU fault to healthy node" (2026-10-27), is cluster-reliability infra rather than post-training technique content.

Interconnects

Podcast: Kimi K3.1 Pro, the new cyber reality, and Muse musings — 2026-10-25. Nathan Lambert and Florian Brand speculate that Kimi K3.1 Pro's RL post-training "leans toward exploitation over exploration," which they argue explains its strong benchmark scores but weaker performance on long-horizon agentic tasks like ProgramBench — a useful cautionary signal if you're tuning explore/exploit tradeoffs in GRPO or RLHF setups. They also discuss Moonshot reportedly down-weighting cyber-related data at mid-training specifically so the restriction carries into post-training instead of relying on inference-time refusals alone. Treat as hosts' speculation rather than a confirmed recipe.

Latent Space

AI Is Getting Good at Math. Here's Why. — 2026-10-23. Describes training objectives aimed at producing reusable intermediate artifacts for long-horizon reasoning chains, reporting that OpenAI trained models specifically on math verification at scale with a dedicated verifier layer that catches proof errors mid-trajectory. The actionable idea for production fine-tuning: train or use a verifier model to gate and score generations cheaply at test time (parallel sampling plus automated verification) rather than relying solely on a single policy's chain-of-thought — relevant to anyone building RLVR-style or rejection-sampling pipelines.
Kimi 3.1 Pro 104BA-2.8T: Fable-5.1-Class Open Weights narrows Closed-AI lead to 1 month... and Pro Highspeed API with 2-3× speed — 2026-10-24. Moonshot describes K3.1 Pro's gains over K3 as coming from "substantially larger post-training compute," via large-scale joint RL across many downstream domains, with explicit infra investment in RL framework efficiency, rollout scheduling, and sandbox infrastructure to keep RL returns stable as compute grows. The piece also reports Moonshot removed cyber-related data from mid-training and RL (not just inference-time refusal), with an internal cyber score dropping from 90.5% to 81% — a concrete, quantified tradeoff between capability and safety data curation at RL post-training scale.

Through-line

The thread running through this week's independent sources is GRPO learning to distrust its own signals. SetGRPO calibrates against noisy graders, the budgeted-audit paper calibrates against gameable verifiers, and both the Interconnects podcast and Latent Space's Kimi coverage independently frame RL post-training as an explicit exploration/exploitation and capability/safety dial that labs are now tuning on purpose rather than treating as a side effect. On the efficiency side, Turing's calibrated distillation and Nemotron 3.5's multi-teacher recipe both point the same direction: post-training is increasingly where multiple specialist models get consolidated into one deployable system, not just where a single model gets aligned.

What's catching your eye in post-training this week? Drop a comment below.

Your benchmark number is a negotiation, not a measurement

Felipe 0liveira — Thu, 29 Oct 2026 12:07:34 +0000

This digest covers LLM evaluation and observability news from 2026-10-22 through 2026-10-29. It's a quieter week on the tooling side (Langfuse, Braintrust, and Simon Willison had nothing new in scope), but arXiv, Hugging Face, and Interconnects more than make up for it with a sharp, recurring theme: single benchmark numbers and judge verdicts are a lot less trustworthy than the dashboards built on top of them suggest.

🔥 Highlights

Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses — near-perfect scores are often test-set artifacts.
Do LLM Routers Actually Route? Quick answer: Not much! — routers rarely beat one solid mid-tier model.
Artificial Analysis v5 and the benchmarker's dilemma — one aggregated index can't be neutral.
Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents — "it worked" still hides collateral damage.
Can we trust LLMs to judge whether a research idea is novel? — judge verdicts flip on tiny prompt tweaks.

arXiv cs.CL

Decision Models as Judges for Hallucination Detection — 2026-10-27
Tests whether cheap "decision models" (fixed-verdict classifiers with label probabilities) can replace a generative LLM judge for hallucination detection, across nearly 10,000 items spanning five hallucination types. A confidence cascade of these small models matches or beats the LLM judge on several types at 1-3% of the cost, though the paper flags overconfident probabilities as a reason not to trust it unsupervised — useful if judge-call spend on hallucination screening is a real line item for your team.

Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents — 2026-10-25
Introduces SABench (201 tasks across email/calendar/ticket domains) with a deterministic oracle that checks not just goal completion but unauthorized side effects ("collateral damage"). Across 11 frontier models, roughly 15% of "successful" runs still take an unauthorized action — a concrete reminder that pass/fail task benchmarks miss a whole failure class that matters for agent evals in production.

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection — 2026-10-25
Shows that Best-of-N quality at large N depends on how the reward/proxy score's calibration error behaves in the tail: Gaussian tails keep improving, heavy tails revert to the mean. Proposes a held-out "tail calibration" statistic teams can run alongside normal reward-model ranking metrics to know ahead of time when cranking up N in production will backfire.

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs — 2026-10-24
Moves past a single "validity rate" for structured-output pipelines to a full failure taxonomy plus a deterministic classifier, run on roughly 4,000 generations across 9 OpenAI/Anthropic configs. Markdown-fenced JSON turns out to be the dominant failure mode — directly actionable if you're evaluating or monitoring structured-output reliability in a production pipeline.

Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses — 2026-10-24
Audits 21 published RAG-poisoning-defense evaluations and finds many benchmarks owe their near-perfect detection scores to an artifact of how the poisoned test data was constructed, not real detection power; most also skip reporting the false-positive rate at the deployed threshold. A solid methodological warning before you cite a security/robustness benchmark number without checking how its test set was built.

Hugging Face blog

Can we trust LLMs to judge whether a research idea is novel? — 2026-10-27
Ai2 tests six LLM judges against a benchmark built from ICLR 2026 reviews, using reviewer comments on originality as the novelty label. Small prompt changes flip the verdict, extra retrieval and "more reasoning" don't help consistently, and a plainly-prompted frontier model beat two purpose-built novelty evaluators — another data point that LLM-as-judge reliability is still an open problem for anyone building judge-based eval pipelines.

Do LLM Routers Actually Route? Quick answer: Not much! — 2026-10-26
Introduces the AutoBench Router Benchmark (ARB), auditing seven commercial and open-source routers with 252 prompts, 20 models, and a 10-LLM-judge committee, with scoring rules pre-registered before testing. No router beats simply always calling one solid mid-priced model on quality-plus-cost — and the pre-registered, committee-graded methodology itself is a good reference for teams who need to defend their own eval setup.

Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox — 2026-10-22
A joint Microsoft/Hugging Face post on ThinkingBox-Bench (507 tasks), which grades agents by the final state of a database rather than by which tool calls they made, using mostly deterministic checks plus a minority graded by an LLM judge rubric. The same deterministic check doubles as an RL reward, with task difficulty selected per-policy via a Jeffreys posterior — relevant if you want to turn agent evals into training signal without leaking the benchmark into the policy.

Two Pure-Vision Pipelines at the Top of ViDoRe V3 — 2026-10-23
On the ViDoRe V3 retrieval leaderboard, two "pure-vision" pipelines (page images only, no OCR) land 1st and 3rd by nDCG@10 across 8 public English datasets. More niche than the other items here, but a clean example of a leaderboard that evaluates the whole retrieval system end to end rather than just the model in isolation.

Langfuse blog

Nothing new in scope this week — the only October posts found are the three already covered in the prior digest.

Braintrust blog

Nothing new this week; the latest post predates this window and was already covered previously.

Simon Willison

Nothing that centers on eval methodology or observability tooling this week — recent posts lean toward model launches and an agent-built scraping tool, none of which clear the bar for this digest's focus.

Interconnects

Artificial Analysis v5 and the benchmarker's dilemma — 2026-10-28
Nathan Lambert digs into the new Intelligence Index v5, which weights knowledge work, coding, science, and general capability equally into one number, and questions whether that flattening hides real trade-offs between models. He also raises an uncomfortable point for anyone leaning on third-party evals: "independent" benchmarkers are often paid by the same labs they rank, which undercuts the benchmark's value as an objective signal — a good reason to compose multiple benchmarks with different biases rather than trust one aggregated index.

Podcast: Kimi K3.1 Pro, the new cyber reality, and Muse musings — 2026-10-25
The hosts compare Kimi K3.1 Pro's Artificial Analysis standing against its results on more specialized benchmarks like Terminal-Bench Science and ProgramBench, debating whether short-horizon benchmarks capture models tuned for long, exploratory tasks. A concrete case of "benchmark doesn't match real usage" — useful context if you're deciding between short-horizon proxies and longer tasks that mirror your actual product workflow for an internal eval suite.

Through-line

Three independent sources converge on the same warning this week: aggregate scores and judge verdicts are easier to produce than to trust. The RAG-poisoning audit shows benchmark construction can manufacture near-perfect scores; the router benchmark and Artificial Analysis critique both show that collapsing many dimensions into one number (a router's pick, an aggregated index) erases exactly the trade-offs a production team needs to see; and the novelty-judge study shows LLM-as-judge verdicts can flip on cosmetic prompt changes. The practical takeaway is the same across all of them — don't ship on a single number, whether it comes from a leaderboard, a router, or a judge call, without knowing how that number was built.

What's your team doing to cross-check a single eval number before trusting it in a release decision? Drop it in the comments.

KV Cache Wars: How Compaction, Quantization, and a Rust Rewrite Are Redefining LLM Serving Economics

Felipe 0liveira — Wed, 28 Oct 2026 13:07:22 +0000

This edition of the LLM Inference Digest covers new research and engineering published between 2026-10-21 and 2026-10-28.

🔥 Highlights

Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72 — vLLM now rivals DeepSeek's own serving stack.
How Anthropic ships Claude with Claude: Separating the Brain from the Hands — a Rust rewrite cuts production CPU cost 40-50%.
How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications — one lever, no retraining, massive savings.
GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s — breaks the 1M-token KV memory wall.
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels — removes the custom-kernel tax on KV compaction.

arXiv (cs.LG, cs.DC)

Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length — 2026-10-25. Derives a step-time model for MoE speculative decoding based on bytes-per-kernel-request over effective bandwidth, since consecutive drafted tokens on an MoE model often hit the same experts. Gives a concrete formula for when speculation pays off and what draft length is optimal on MoE serving, directly actionable for anyone tuning speculative decoding on DeepSeek/Mixtral-style deployments.
Distributed LLM Inference via Communication-Efficient Speculative Decoding — 2026-10-25. Addresses the vocabulary-scale communication bottleneck in device-server collaborative speculative decoding, where exact rejection sampling normally requires shipping full token distributions over the network. Relevant for anyone deploying split inference across bandwidth-constrained links such as mobile-to-cloud.
ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding — 2026-10-24. Points out that prior lossless KV-cache compression only reduces inter-device transfer but still restores the full KV cache in GPU memory before attention. Fuses decompression directly into the attention kernel to cut both memory footprint and bandwidth pressure during decode, a concrete throughput/concurrency win for long-context serving.
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels — 2026-10-27. KV-cache compaction methods that add a per-key bias term normally can't run on FlashAttention/vLLM's paged-cache kernels since there's no input slot for the bias. This folds the bias into the keys via a closed-form per-head direction, removing a "custom kernel required" barrier to adopting KV compaction in production stacks.
RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches — 2026-10-24. Existing KV-cache quantization schemes were designed for standard per-head K/V and don't account for what Multi-Head Latent Attention (used in DeepSeek-style models) actually caches, causing accuracy loss at low bitwidths. Proposes a role-aware scheme tailored to MLA's cache structure, useful for teams quantizing MLA-based models for memory savings.
ResCom: Activation Residual Compensation for Large Language Model Quantization — 2026-10-25. Targets accuracy loss from activation rounding in low-bit weight-activation quantization, a gap calibration-based reconstruction can't fix. Introduces a low-rank correction using the rounding residual itself, a technique that could improve accuracy retention at INT4/INT8-class quantization without extra inference cost.
DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving — 2026-10-24. Current CPU-offload serving systems are "GPU-gated" — CPU-resident requests only progress when the GPU scheduler exposes overlap opportunities, causing bursty CPU throughput. Proposes decoupling scheduling so CPU and GPU act as independently-paced resources, relevant to anyone running hybrid GPU/CPU-offload serving to extend effective KV capacity.
InferScale: Kubernetes-Native Predictive Autoscaling for SLO-Aware AI Inference — 2026-10-25. Argues Kubernetes HPA's reactive scaling is too slow for inference pods with tens-of-seconds cold-start, worsened in multi-tenant clusters where independent HPAs fight over resources. Presents a K8s-native autoscaler combining inference-aware signals and short-horizon demand prediction, directly applicable to teams fighting cold-start-driven SLO violations.
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving — 2026-10-27. For agentic workloads alternating between GPU-side inference and CPU-side tool calls, shows existing serving systems ignore tool-return readiness and waste GPU resources waiting. Proposes a joint scheduler accounting for near-term tool-return timing, timely given the shift toward agentic serving infrastructure.
KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving — 2026-10-22. For long-context/agentic workloads where reused-prefix KV caches outgrow GPU+host memory, a single node-local NVMe absorbs all prefix retrieval traffic while parallel storage nodes sit idle. Casts prefix retrieval as aggregate-bandwidth scheduling across storage tiers, a practical fix for storage I/O becoming the bottleneck in prefix-caching-heavy deployments.
Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving — 2026-10-22. Flags a real production hazard: SSD-based KV-cache offloading under multi-round-conversation/agent workloads writes so much that SSD lifetime drops from years to months. A hardware-cost/ops concern worth knowing before deploying SSD-backed KV offload.
Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring — 2026-10-26. A university research GPU cluster study finds prior minute-level monitoring missed GPU-level (not just job-level) utilization opportunities that only fine-grained monitoring reveals. Useful empirical evidence for anyone doing GPU-cluster capacity planning or utilization audits.
Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers — 2026-10-26. Multi-turn routing decisions that send a query to a cleaner-grid region or smaller model are complicated because each turn's KV cache lives only where earlier turns ran, so naive carbon/cost-aware routing can break cache locality. Relevant for anyone operating multi-region LLM serving who needs to balance cost/carbon against KV-cache locality.

vLLM blog

Day-0 Support for Flint on vLLM — 2026-10-21. vLLM shipped day-0 support for Recursive's Flint, a 40B-param MoE (~2.5B active/token), in both NVFP4 and BF16. Flint-specific kernels cut NVFP4 decode latency from 4.81ms to 1.78ms per forward pass (2.7x) and reduced kernel count per forward from 1,096 to 346, enabling ~500 tok/s single-request decode at 100K context on one GB200.
Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72 — 2026-10-23. A full-rack GB300 NVL72 deployment using prefill/decode disaggregation plus Wide-EP reaches ~954K tok/s/GPU on AgentX, matching DeepSeek's own DeepInfer engine within 6% on decode throughput; a specialized low-latency decode path hits 826 tok/s for a single user. Directly actionable for teams sizing large-scale MoE serving on Blackwell racks.
GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s — 2026-10-23. Introduces Prefill/Decode Context Parallelism combined with Hybrid HiSparse decode on a 2-node/16x H200 setup, delivering 1.7–3.2x per-user speed over a DP8 baseline at equal throughput and serving 1M-token contexts. Relevant to anyone hitting KV-cache memory walls on long-context deployments.
vLLM Delivers Day-0 Support for MiniMax-M3.1-Flash — 2026-10-23. Day-0 support across Blackwell, Rubin, and AMD MI350-series, using NVFP4 KV cache and NVFP4-quantized MoE weights that shrink the checkpoint from 451.5GB to 250.1GB. Reports up to 418.9 tok/s/user (GB300), 724.9 tok/s/user (Rubin), and 133.3M total tokens/$ on MI355X — directly useful cost/throughput reference points across three hardware targets.

Hugging Face blog

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications — 2026-10-21. Dharma-AI's VocabX rebuilds a small model's output vocabulary around per-token generation cost rather than frequency, keeping the base model, data, and recipe unchanged. On a production Portuguese essay-grader built on Qwen3-4B it cut measured inference cost 77% and average response length from 350 to 101 tokens while the bench score rose, exploiting the fact that generating tokens costs roughly 7.4x more than reading them.

No other posts in the window had real serving/throughput/cost substance — the rest were model launches, operator-level profiling tutorials, or non-inference research.

Together AI blog

Auto repair for Together GPU Clusters: from GPU fault to healthy node — 2026-10-27. Explains Together's fault-detection system, which recommends the least-disruptive fix with supporting evidence, applies per-fault-type/per-Xid-code policy, escalates repairs on recurrence, and auto-revalidates repaired nodes before they rejoin the pool. The post ties this explicitly to inference reliability: GPU faults can take a serving replica down, making this relevant to anyone operating GPU fleets for inference.

Modal blog

Nothing relevant this week — the only two posts in the window ("Introducing our Training API" and "Get started with LLM training on Modal") are about post-training/fine-tuning infrastructure, not inference or serving.

Latent Space

Why AI Compute Comes in Rectangles — 2026-10-22. Modal's CEO explains why fixed GPU contracts don't fit inference's spiky demand and how Modal pools demand across customers to cut idle capacity, covering fast-replica-startup tricks (GPU memory snapshots, process checkpointing) and interest in disaggregated inference. Direct operator insight into why Kubernetes/Docker-style autoscaling falls short for GPU-serving workloads.
How Anthropic ships Claude with Claude: Separating the Brain from the Hands — 2026-10-27. Anthropic engineers describe rewriting the Messages API in Rust, shadow-tested against the Python system, reporting ~40-50% CPU savings and ~10-20% better time-to-first-token, with bigger gains on long-running workloads. Also covers CPU as a capacity-planning bottleneck and a "brain/hands" architectural split to cut latency.
Kimi 3.1 Pro 104BA-2.8T: Fable-5.1-Class Open Weights... Highspeed API with 2-3x speed — 2026-10-24. New Kimi K3.1 Pro MoE release with a "Highspeed" variant claiming 2-3x faster generation at 2x cost; the digest also bundles a 12% DeepSeek V4.1 Flash decode-throughput gain from small-message all-reduce kernels and a KV-cache "read-ahead" trick cutting wait time from 12.2s to 3.2s on a 6GB GPU.
Black October — 2026-10-23. Dense inference-systems roundup: AMPD routes multi-turn prefill/decode by latency headroom (67% SLO-attainment gain over Dynamo); AKV lets agents reposition their own KV cache (-27.7% cost); Rubin NVL72 hits 862 tok/s/user on GLM 5.3 after kernel work; MiniMax M3.1 Flash gets day-0 vLLM support with NVFP4 weights/KV-cache.
OpenAI's Patch the Planet program... — 2026-10-22. Deep dive on DeepSeek's DeepInfer serving stack combining speculative decoding, dynamic prefill sizing, and CPU/GPU overlap scheduling, reporting 1.5M input tok/s per prefill accelerator with a 96% cache-hit rate and 14K output tok/s/accelerator on decode. Also covers SGLang-TorchTPU reaching 770 decode tok/s at batch-1 and EAGLE-4 speculative decoding at 3.59x speedup.
Anthropic files for $2T IPO in 2 weeks — 2026-10-27. Digest reports vLLM-on-DeepSeek-V4.1-Flash achieving $38.96 revenue/GPU-hour at 954K tokens/s/GPU aggregate throughput, DCOCO speculative decoding restoring context between parallel drafts for up to 6.6x speedup, and KV-cache-sharing across loops cutting GPU memory 66.2%.
Anthropic ends Coding Interviews: "Hunger beats tenure" — 2026-10-28. Reports CoreWeave-side serving gains (+63% traffic on GLM-5.3, 2x per-user throughput on Qwen3.8-27B) and TokenGate claiming up to 11.3x faster long-context prefill, plus Anthropic's S-1 showing serving compute cost per revenue dollar falling from $0.88 to $0.15.

Through-line

Nearly every independent source this week converges on the KV cache as the real site of competition in LLM serving: arXiv papers attack it from quantization (RAQ-MLA), compaction-kernel compatibility (the folded-bias trick), storage tiering (KVStream), and SSD wear (Gandalf), while vLLM's GLM-5.3 work and several Latent Space items attack it from the context-parallelism and repositioning angle. At the same time, production engineering payoffs are standing shoulder to shoulder with new algorithms — Anthropic's Rust rewrite and Dharma-AI's vocabulary specialization both show that rethinking an existing system, not just adopting a new technique, can move cost and latency by double digits. Vendor-neutral serving stacks are also closing the gap with model vendors' own infrastructure, as vLLM's DeepSeek-V4.1-Flash benchmarks landing within 6% of DeepInfer illustrate.

What's catching your eye this week — KV-cache tricks, the Rust rewrite, or something else? Drop a comment below.

GRPO keeps winning the RL-vs-data argument, and LoRA gets a mechanistic audit

Felipe 0liveira — Tue, 27 Oct 2026 13:06:04 +0000

This digest covers fine-tuning and post-training news from 2026-10-16 through 2026-10-27: eight new arXiv papers, two Hugging Face blog posts with reproducible SFT/GRPO recipes, and a new post-training product launch from Together AI.

🔥 Highlights

Welcome Beam: a 501B open model you can own, customize, and scale — direct LoRA-vs-full-parameter GRPO numbers on a 501B model.
Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning — GRPO beats RFT even after a 27-run tuning sweep for the baseline.
ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization — a hidden signal that warns of DPO/SimPO collapse before it happens.
Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox — deterministic reward engineering cuts RL cost per task by 3x.
Together Post-Training: frontier-grade model shaping, the way that fits you — a managed service with cookbooks for SFT, GRPO, DPO, and multi-LoRA serving.

arXiv

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching (2026-10-26) — Using path-patching and zero-ablation, this paper shows LoRA's effect is highly sparse and task-specific across layers, with little overlap between tasks. The proposed Adaptive-LoRA selects relevant layers via calibration instead of exhaustive search, beating full-layer LoRA on Qwen2.5-1.5B across 11 reasoning tasks while using only 30% of the layers and 3.5x fewer adapter parameters — a concrete way to cut fine-tuning cost without giving up accuracy.
Rethinking LoRA Initialization for Robust Asymmetric Learning Rates (2026-10-23) — Explains why LoRA+'s trick of using a higher learning rate for matrix B than A destabilizes training at high ratios: a noise term from the standard initialization is to blame. A new initialization (scaling A's variance by n⁻²) suppresses that noise and widens the stable learning-rate range substantially, with a practical starting heuristic (η_B ≈ nη_A) for tuning.
Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning (2026-10-19) — A tightly controlled comparison (same prompts, verifier, model: Qwen3-4B-Base + LoRA) finds GRPO beats rejection-sampling fine-tuning (RFT) across every tested prompt set, by up to 10.9 points on held-out data, even when RFT gets a 27-run hyperparameter sweep against GRPO's unchanged recipe. Strong evidence the advantage comes from the RL algorithm itself, not just the data — though it's one seed family and task.
PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs (2026-10-18) — A LoRA fine-tuning system that accelerates frozen projections with native NVFP4 GEMMs, reuses quantized RMSNorm outputs to cut activation memory, and calibrates frozen weights offline with a GPTQ variant adapted for NVFP4. Delivers lower memory use and faster training with competitive accuracy on a 16GB RTX 5080 — relevant for anyone fine-tuning outside a datacenter.
First-Order LoRA Training with Phone NPUs (2026-10-17) — A system for on-device LoRA training that runs the three frozen-weight matrix products on a phone's NPU (which already holds the base model for inference) while keeping adapters and optimizer on the CPU. Tested on Snapdragon with Qwen and Llama models on the LaMP-2 benchmark, it's a step toward personalization where user data never leaves the device.
ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization (2026-10-21) — Finds that DPO/SimPO can improve ranking metrics while quietly building a shared "offset" in hidden states that the ranking loss barely reacts to; as it grows, it can collapse generation into a single dominant token. Across 17 UltraFeedback runs, masking this component from the loss prevents the collapse, and the signal is present even in "normal" runs that never visibly collapse — a useful early-warning signal for anyone monitoring preference-optimization runs in production.
An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs (2026-10-19) — A large-scale study of applying PEFT to MoE models across three backbones. Practical findings: adapting only the experts gives the best downstream-vs-general-capability trade-off; updating expert weights directly beats adapters that bypass the MoE module; LoRA+DoRA together give the strongest accuracy; and no nonzero load-balancing coefficient beats the zero baseline — a concrete guide for PEFT on MoE today.
Task Format as a Factor in Adaptation: A Controlled Comparison of LoRA and Full Fine-Tuning for Prompt-Safety Classification (2026-10-24) — A well-instrumented comparison (up to 15 seeds per condition) finds full fine-tuning beats LoRA by 2-3 points on a prompt-safety classifier at pre-specified learning rates, but the gap vanishes once both methods get the same hyperparameter search. Interestingly, training LoRA to write a justification after its verdict ("reasoning target") also closes the gap — suggesting task format matters more than the adaptation method itself.

Hugging Face blog

Welcome Beam: a 501B open model you can own, customize, and scale (2026-10-16) — The densest post-training walkthrough of the window. Covers SFT with TRL's SFTTrainer plus LoRA on Beam-501B (language adherence going from 3/16 to 16/16 on a multilingual dataset), a full-parameter GRPO recipe in BF16 (Megatron + SGLang, trained on DAPO-Math-17K, AIME25 rising from 46.25% to 55.8%), and the same GRPO setup in LoRA form (rank-16, AIME25 reaching 64.2% with shorter outputs). Also includes a synthetic-data case: an agent generated 2.3k synthetic incidents to fine-tune a Qwen3.5-0.8B via Unsloth (LoRA + decision head, 4-bit), jumping from 4.69% to 76.25% accuracy. The side-by-side full-finetune-vs-LoRA GRPO numbers and published scripts (run_beam.py, run_beam_lora.py) make this a rare direct data point for budgeting a GRPO run.
Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox (2026-10-22) — A full-parameter RL recipe using GRPO (token-mean) on Qwen3.8-27B, with deterministic database grading as the reward instead of an LLM judge. Details task-pool selection by policy-relative difficulty (a Jeffreys posterior) and a DAPO-style swap of zero-variance groups. Pass@1 rose from 51.70% to 60.78%, while reliable cost per task dropped from $24.36 to $8.00 — a reproducible recipe (with a Verl integration and launch scripts) for reward engineering outside LLM-judge domains.
A curriculum for post-training agents (2026-10-20) — A five-stage curriculum for post-training agents: SFT (TRL SFTTrainer), distillation (DistillationTrainer, on- and off-policy), RL with GRPO (GRPOTrainer), multi-turn RL via OpenEnv, and scaled harnesses (AsyncGRPOTrainer). It's more of a conceptual map than a numbers-backed study, but a useful reference for deciding which technique to reach for at which stage of agent post-training.

Interconnects

Nothing relevant this window — the three posts published in the period (a podcast episode, an open-artifacts roundup, and a note on Meta's release) touch on RL and post-training only in passing, without reproducible recipes or results.

Together AI blog

Together Post-Training: frontier-grade model shaping, the way that fits you (2026-10-19) — Announces a managed post-training service for open models (Qwen, GLM, Kimi) via SFT or RL. Supports LoRA (including multiple adapters sharing one GPU) and full fine-tuning; ships GRPO, CISPO, and DPPO as standard RL losses, plus custom losses — the cookbook includes DPO implemented as a custom loss. Also offers asynchronous RL for long agentic rollouts, "Rollout Routing Replay" to reduce train/inference policy drift, and multi-LoRA serving. No published benchmark numbers, but the cookbooks (GRPO on GSM8K, DPO as a custom loss, multi-adapter serving) give a concrete menu for choosing between SFT/LoRA/full RL and which loss to use.

Latent Space

Nothing relevant this window — the posts touching on post-training (a math-reasoning piece and a Kimi 3.1 Pro roundup) mention RL in passing without a reproducible recipe or confirmed technique.

Through-line

Two independent threads line up this week: a mechanistic push to understand why LoRA and preference optimization behave the way they do (layer sparsity, initialization noise, hidden-state collapse) rather than just reporting benchmark wins, and a growing consensus that GRPO's advantage over simpler alternatives (rejection sampling, generic RL losses) holds up even under adversarial, well-tuned baselines. Together with Beam's and Together's side-by-side LoRA/full-parameter recipes, the practical message for anyone budgeting a post-training run is the same: GRPO is worth the extra complexity, and LoRA is catching up to full fine-tuning faster than the defaults suggest — if you tune it as carefully.

What's your experience — has GRPO earned its complexity in your own RL runs, or has rejection sampling been good enough? Drop a comment below.

Your Eval Score Might Be Lying: How Judges, Benchmarks, and Retrieval Quality Quietly Skew Results

Felipe 0liveira — Thu, 22 Oct 2026 12:58:11 +0000

This digest covers what's new in LLM evaluation and observability between 2026-10-15 and 2026-10-22. It's a week where the measurement layer itself took the spotlight — how you score a judge, how you read an uncertainty signal, even how you format a benchmark question all turned out to move results more than expected.

🔥 Highlights

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring — swapping a rating scale alone reshuffles LLM-judge verdicts.
Run evals 50% cheaper with workflow evals — provider batch APIs halve the cost of large regression suites.
When Does Uncertainty Measure RAG Hallucination Risk? — uncertainty only flags hallucinations when retrieval actually constrains the answer.
How we built Langfuse Halo — their own traces caught tool-schema bugs and a costly caching mismatch.
FairStress: A deeper evaluation of fairness in LLM decision-making — multiple-choice vs. free-generation scoring can flip a fairness verdict.

arXiv cs.CL

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring (2026-10-15)
Changing a single element of a rating scale — its range, format, or sign — shifts an LLM judge's normalized score by ~0.10 on average and flips the category assigned to roughly 20% of texts; rubric-based prompting doesn't reduce this sensitivity versus zero-shot. For teams using LLM-as-judge in production, this means a scale tweak (1-5 to 1-10, or an inverted sign) can move your eval numbers with zero change to the model being judged — a strong case for stress-testing judges across scale variants before trusting an absolute score as a regression gate.

SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents (2026-10-16)
An "interlocutor" LLM enacts sampled situations while a judge LLM checks, turn by turn, whether the agent followed the applicable conditional rule, validated against human-annotated golden dialogues. Across 8 LLMs and 1,292 turns, models with similar aggregate scores diverge sharply rule-by-rule, with ambiguity clarification and Socratic questioning the weakest spots across the board. It's a useful pattern for teams that need per-turn observability on conversational agents instead of a single end-of-conversation score that hides exactly where the agent is failing.

When Does Uncertainty Measure RAG Hallucination Risk? (2026-10-15)
Uncertainty signals like perplexity only catch hallucinations in RAG when retrieved evidence genuinely constrains the answer; with weak retrieval, the model falls back on internal knowledge and hallucinates with deceptively low apparent uncertainty. Across 4 models on HotpotQA/SQuAD, perplexity detects unsupported edits with AUROC ~0.81–0.91, but performance swings with retrieval depth — combining uncertainty with a semantic-drift score improves calibration. For production RAG observability, this argues against monitoring generator uncertainty alone: hallucination risk needs to be conditioned on retrieval quality, not treated as an independent signal.

SHAM-Bench: A Span-level Hallucination Benchmark Across Multiple Tasks (2026-10-19)
Unifies human-annotated datasets from 6 NLG tasks (6,461 examples) under a common schema for span-level (not whole-response) hallucination detection, with dataset, eval code, and a public leaderboard released. Fine-tuned detectors do well in-domain but degrade under distribution shift, while prompt-based reasoning methods transfer better across tasks. Useful as a standardized comparison point for anyone evaluating hallucination detectors across multiple task types — and a reminder that subtle, unsupported content remains the blind spot even for specialized detectors.

Hugging Face blog

FairStress: A deeper evaluation of fairness in LLM decision-making (2026-10-20)
Ai2's new benchmark runs 300 two-candidate decision scenarios across 29 domains, varying identity cues and adding paired persuasive "adversarial prefixes" to see whether a model flips a correct decision more easily when the argument favors a minority candidate. The key finding: fixed-choice scoring (like BBQ) and free-generation scoring can point to bias in opposite directions for the same model. Fairness/safety eval teams should treat scoring format as a variable, not a detail — running both formats before calling a model "fair" is the practical takeaway.

ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model (2026-10-20)
IBM Research shows a behavioral-memory technique — extracting guidelines from an agent's training attempts and injecting them into the system prompt, no weight updates — climbing the τ²-bench leaderboard (retail/airline customer service) using 4 trials per task and Mean Accuracy / Pass^4 metrics, benchmarked against the 2026-10-12 leaderboard snapshot. It's a concrete example of a non-frontier system "winning" an agent leaderboard through prompt/memory engineering, worth a look for anyone weighing cost vs. performance trade-offs and wanting to understand what Pass^4 is actually measuring on an agent benchmark.

Langfuse blog

How we built Langfuse Halo (2026-10-21)
Langfuse's in-app assistant (public beta since June 2026) interprets traces, builds dashboards/datasets, and helps configure evals directly in the UI — built by reusing their own MCP server and the same tools/skills exposed to external agents like Claude Code and Cursor, with project-scoped API keys, read-only tools by default, and an optional sandbox (AWS Lambda MicroVMs) for heavy result processing. It's a real case study in dogfooding a single tooling layer for agent observability: Halo's own traces surfaced tool-schema mismatches and a prompt-caching issue inflating cost, both fixed upstream in the shared tools.

Operating Langfuse as an internal service (2026-10-22)
An operational guide (not a feature announcement) on sustaining Langfuse as an internal platform service across multiple application teams: acceptance checks before SDK upgrades, health monitoring via synthetic traces (including restore drills, since a successful backup doesn't guarantee recovery), self-service eval playbooks, periodic access/credential reviews, and cost allocation via showback before chargeback. It's a concrete checklist for the real failure mode of "evals in CI/production silently stop working" as adoption scales — an angle most observability posts skip.

Provisioning and governing Langfuse across teams (2026-10-22)
A companion piece focused on organizational design: choosing project/organization boundaries, separating authentication (SSO) from authorization (roles), telemetry acceptance checks (trace routing, sensitive-field masking) before go-live, and a promotion path from tracing → datasets → evals across staging and production. Offers a replicable governance model for teams scaling observability/evals across multiple groups, with clear notes on what's plan-gated (per-project RBAC, SCIM) versus what you have to build yourself.

Braintrust blog

Run evals 50% cheaper with workflow evals (2026-10-15)
Introduces workflow evals: eval tasks and scorers now run through provider batch APIs (OpenAI, Anthropic), which offer discounted pricing for asynchronous processing, with results still centralized in a single Braintrust experiment. New WorkflowBatchTask/WorkflowBatchScorer classes (TS/JS and Python SDKs) batch cases with submit/collect callbacks, polling or webhook-based progress, and persisted state (e.g. Redis) to pause/resume across processes. The trade-off is latency — batch windows up to 24h — making this a fit for large nightly regression suites and non-urgent model comparisons rather than real-time feedback, in exchange for roughly half the cost.

Simon Willison

Nothing substantive on evaluation methodology or observability this week. The one tangential item is llm-echo 0.5 (2026-10-18), a minor update to his LLM CLI's debug "echo" plugin — a fake model that just repeats the prompt back, now simulating token counts that account for tool calls, system prompts, and reasoning tokens. It's a small but genuinely useful building block for writing deterministic, no-API-cost CI tests around prompt and tool-calling logic before running real evals against a live model.

Interconnects

Meta shakes up the frontiers, open and closed (2026-10-17)
Beyond the analysis that US open models (Meta's Muse Spark Open, Reflection's Beam) have closed the gap with Chinese counterparts, Nathan Lambert makes a direct evaluation-methodology point: benchmarks are a useful proxy but predict real-world model reception poorly, citing swinging community reactions across successive Anthropic releases despite consistent benchmark scores (referencing the Artificial Analysis Intelligence Index and Meta Muse Wonder results). For teams running production evals, it's a reminder that a benchmark ranking going up is no guarantee of perceived quality — worth pairing static benchmarks with real-world or human-in-the-loop evaluation before swapping production models.

Through-line

The strongest thread this week is that the measuring instrument itself is under scrutiny, not just the models being measured. The rating-scale sensitivity paper, the retrieval-conditioned uncertainty finding, and FairStress's scoring-format flip are three independent demonstrations that how you score — scale, signal, format — can move results as much as the model swap you're trying to evaluate. Interconnects' point about benchmark scores failing to predict real-world reception echoes the same skepticism from the outside-production-eval angle. Meanwhile Langfuse's operational and governance posts show observability tooling maturing past "can we trace it" into "can we run and govern this reliably at scale" — and Braintrust's batch evals show cost, not just accuracy, becoming a first-class eval design constraint.

What's catching your eye in how you score your own evals these days? Drop a comment below.

Elastic MoE scaling, free batch-sorting wins, and DeepSeek's serving stack goes open

Felipe 0liveira — Wed, 21 Oct 2026 12:13:24 +0000

This digest covers LLM inference and serving developments from 2026-10-14 to 2026-10-21: arXiv (cs.LG/cs.DC), the vLLM and Hugging Face blogs, Together AI, Modal, and Latent Space.

🔥 Highlights

Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM — up to 3x better TTFT for prefill/decode-disaggregated setups.
Achieving near-zero-downtime scaling in Elastic Expert Parallelism — MoE autoscaling without SLO breaches.
Sort your prompts: a free speed-up for MoE batch inference — a zero-code, zero-cost batching win.
KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving — cross-region cache reuse to kill duplicate recompute.
[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback — DeepSeek open-sources its production serving stack.

arXiv (cs.LG, cs.DC)

KV cache management

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill (2026-10-15) — In a real GLM-5.3 deployment on 8xB300, a misconfigured remote KV cache tier made p99 latency worse (3.96s → 17.3s at 16k context) because a remote cache hit became slower than just recomputing the prompt. The proposed fixes (rank-based sharding, async loading) cut p99 from 83.32s to 0.67s in a hit-heavy scenario — a concrete cautionary tale for anyone running distributed KV cache in production.
Efficient KV Cache Migration for Geo-Distributed LLM Inference in Collaborative Edge Computing (2026-10-15) — EdgeFlow speeds up KV cache migration between edge servers by sending part of the state and regenerating the rest in parallel, cutting migration time up to 45% on a real testbed. Useful for failover/load-balancing between edge nodes without killing active sessions.
Understanding KV Residency in Production-Scale Agentic LLM Workloads (2026-10-16) — A production study shows a 1-hour idle cap cuts HBM needed for KV cache by ~3.6x with minimal extra recompute; naive LRU eviction forces up to 63% recompute on "Deep Research"-style workloads, but int4 compression on cold sessions drops that to 13.6%. Hard capacity-planning numbers for anyone sizing a GPU fleet for agentic traffic.
KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving (2026-10-16) — A discovery layer that finds, across distant regions, which worker already holds a reusable KV prefix, avoiding duplicate global recompute. Reports 1.3x–14x lower p99 TTFT and up to 4.7x more throughput — relevant for multi-region serving operations.
TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference (2026-10-17) — Processes prefill in per-layer chunks and only evicts cache at the end of each layer, cutting ~84% of transient memory at 64K context on an A100 with only ~1% TTFT overhead. Lets you run 128K context where post-prefill compression approaches run out of memory — including on edge hardware (tested on Jetson).
The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction (2026-10-19) — Offline-compacts the KV cache of a fixed, heavily-reused context, adjusting remaining entries to compensate for the removal's effect on attention. At 50x compaction it gains up to 4 accuracy points over prior attention-matching methods at comparable cost — good for RAG/fixed-context setups with high reuse.

Serving, scheduling, and batching

Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets (2026-10-17) — A fleet planner combining an LLM reasoning layer with deterministic components to decide where/how to run models across heterogeneous hardware and multiple clouds. On a real cluster it delivered 1.18x the goodput of the best baseline and cut cost-per-token ~20% — a direct FinOps lever for multi-cloud inference.
CEMA-Router: Infrastructure-Aware Batch Routing for LLM Inference in Cloud-Edge Systems (2026-10-19) — Routes request batches between cloud and edge weighing cost, energy, and batch completion time alongside answer quality. On GSM8K/MMLU it reports up to 33%/19% lower energy use and up to 27%/30% shorter batch completion time vs. state of the art.
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs (2026-10-14) — Manages weight shards and KV cache blocks as a single pool under GPU memory limits, with predictive prefetch of the next model an agent is likely to call. Cuts session completion time by 21.7% (up to 49% in replay traces) and raises SLO attainment from 2.8–24.5% to 76.8–100% — relevant for anyone serving multiple models/agents on the same GPU.
Exact Makespan Bounds and the GPipe Homogeneity Bias for Heterogeneous LLM Inference Pipelines (2026-10-15) — Formally shows the standard GPipe formula overestimates pipeline time on heterogeneous GPU clusters, and concludes that balancing compute across stages matters more for stable throughput than reducing inter-stage communication. Useful for anyone running pipeline parallelism across mixed GPU generations.
Accelerated Dataflow: scaling inference in streaming data analytics with efficiency (2026-10-15) — A system built on Google Cloud Dataflow combining autoscaling across heterogeneous worker groups with prefix-aware batching, aimed at streaming pipelines with heavy GPU-based LLM inference. Reports 1.5–2x more throughput at the same latency, up to 75% less peak resource use, and 84% lower peak tail latency.

Quantization

Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference (2026-10-15) — Proposes a half-octave weight grid (FPoT) enabling multiplier-free 4-bit shift-add datapaths, staying within ~0.4pp accuracy of uniform integer baselines on Llama-3-8B. Gains are projected via hardware gate counts, not measured latency/throughput — more relevant to accelerator designers than to anyone just swapping GPU kernels.
SpectrumQuant: Spectrum Alignment for LLM Quantization (2026-10-14) — A data-aware orthogonal transform for post-training quantization of Llama/Mistral, matching leading rotation methods' accuracy while using under 1/3 the GPU memory and much less prep time than learned-rotation methods like SpinQuant/DartQuant. The practical value is in quantization prep cost, not inference itself — no latency/throughput numbers reported.
PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference (2026-10-14) — Channel-pair-guided mixed-precision key-cache quantization reaching ~2.26 average bits (vs. 2.32–2.39 for prior methods) while matching or beating accuracy, with smoother degradation down to 1.5 bits. Directly cuts cache memory/bandwidth footprint for long generations — accuracy-only results, no latency/throughput numbers.

Speculative decoding

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding (2026-10-14) — A training-free method pluggable into existing drafting setups with no retraining. On target-drafter pairs tested up to 256K context, it roughly doubled best-case speedup over autoregressive decoding (2.08x → 4.22x) — a plug-and-play option for anyone already running speculative decoding at long context.
HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding (2026-10-15) — Runs the server model's multi-token-prediction head on the phone's NPU combined with a small local model, cutting server GPU time per generated token by 38.6% (Spec-Bench) and 29.2% (32K summarization) vs. server-only self-speculation. Relevant for anyone trying to cut server cost in hybrid mobile-cloud setups.

vLLM blog

Announcing Humming: Fast, Flexible Kernels for Quantized Inference (2026-10-21) — Ant Group and the vLLM community open-sourced Humming, a kernel library for quantized inference supporting many weight/activation formats on NVIDIA GPUs from Turing onward. Dense W4A16 hits ~3-4x cuBLAS speed at small batch and ~2x Marlin at large batch on H200; an end-to-end DeepSeek-V4.1-Flash test on 8xH20 saw 14-43% higher throughput switching Marlin W4A16 → Humming W4A8 — a real win for quantized MoE inference on older or mixed NVIDIA fleets.
Introducing vLLM TorchTPU (2026-10-20) — A new open-source backend runs vLLM on Google Cloud TPUs via Google's TorchTPU stack, reusing vLLM's model code, scheduler, and API server with TPU-specific Pallas kernels underneath. No throughput/cost figures in the post itself, but it widens the hardware menu for teams comparing TPU vs. GPU cost/perf for vLLM serving.
Achieving near-zero-downtime scaling in Elastic Expert Parallelism (2026-10-19) — Elastic EP can now add/remove GPUs from a live MoE deployment with near-zero pause, pre-warming joining workers in the background and swapping ranks via NIXL EP without invalidating CUDA graphs. On DeepSeek V4 Pro/GB200, downtime dropped from ~100s to 0.82s scaling 16→20 GPUs, with p95 TTFT staying under a 5s SLO across 11,391 replayed requests — removes the "scaling event = SLA breach" problem for MoE autoscaling.
Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly (2026-10-16) — A new decode path fuses most ops into custom CuTe DSL "megakernels" for Kimi K3 on GB300, cutting per-layer kernel count from 23 to 13. Batch-1 decode with speculation reaches 842 tok/s vs. 488 on stock vLLM (1.73x), and AgentX P90 interactivity roughly doubles — a large, concrete decode-latency win for Kimi K3 on NVIDIA.
Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX (2026-10-16) — At a fixed 50 tok/s/user P90 SLO, an 8xMI355X node delivers ~7,567 tok/s/GPU vs. 6,083 for HGX B300 (+24.4%), and using modeled TCO ($1.50 vs. $2.26/GPU-hr) that's +87.4% tokens-per-dollar — a direct GPU-vendor cost comparison for provisioning large-MoE capacity.
Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM (2026-10-14) — A new push-based NIXL connector moves post-prefill KV-transfer coordination off the critical path: the decode instance pre-registers destination blocks and the prefill instance writes KV cache in directly on a background thread. On a 1P1D Qwen3-32B benchmark, push mode gave up to 3x better TTFT and up to 30% better TPOT vs. pull mode — directly applicable to any prefill/decode-disaggregated vLLM deployment.
Adaptive Verification Without a Confidence Head (2026-10-14) — Introduces an online-fitted logistic-regression estimator over drafter top-token log-odds that lets Adaptive Verification trim low-acceptance speculative-decoding draft tokens without a separately trained confidence head. Metrics shown are calibration-only (AUC 0.8336, ECE 0.0100); useful for teams running speculative decoding who want adaptive verification without a separate training pipeline, though no hard throughput number is quoted yet.

Hugging Face blog

Hugging Face ❤️ TorchTPU: TPUs, Natively (2026-10-20) — Transformers/Diffusers/Kernels-Hub models now run natively on TPUs via TorchTPU. A Hub ragged-dot kernel reaches up to 21.5x the default implementation at batch 64 on Qwen3-30B-A3B — a big batch-decode speedup for anyone evaluating TPU as an MoE serving target, though no cost figures are given.
TorchNeuron: running HF models on AWS Trainium is easier than ever (2026-10-19) — A new PyTorch-native backend for AWS Trainium, integrated into Transformers/Diffusers/Kernels. 89% of the top 100 text-generation architectures now work out of the box (vs. 46% under Optimum Neuron v0.4.5) — mainly a signal that Trainium just got much easier to adopt for cost-efficient inference.
Sort your prompts: a free speed-up for MoE batch inference (2026-10-19) — Shows that mixing cross-domain prompts in one MoE batch forces the GPU to load more expert weights per decode step; simply sorting/grouping prompts by domain before llm.generate sped up unmodified vLLM by 12-17% on Qwen3-30B-A3B and 7-12% on gpt-oss-120b. A genuinely free, no-code-change throughput win for anyone batching heterogeneous prompts against MoE models.
Tokenmaxxing BERT: 50x faster tokenization (2026-10-15) — HF's tokenizers team made BERT tokenization ~50x faster by skipping Unicode normalization on ASCII text and caching character-class lookups, with identical output token IDs. Mainly relevant to encoder-model (classification/retrieval) serving pipelines where tokenization is a real fraction of latency.

Together AI blog

Together Post-Training: frontier-grade model shaping, the way that fits you (2026-10-19) — Launches a managed SFT/RL service with dedicated GPU allocation and multi-LoRA adapter serving on the same dedicated instance. The serving-relevant pieces: FP4-merged checkpoint export for production inference, and multi-adapter serving that cuts the cost of hosting several fine-tuned variants on one GPU.

Modal blog

Serverless routing made sticky (2026-10-20) — Modal's new "Sticky Sessions" pin each request of a session to the same container via a session token, with the autoscaler counting active sessions rather than raw request volume. Not LLM-specific, but directly useful for stateful inference workloads (voice agents, shared chat) that need to keep decoder/KV-cache state on the same container, cutting cold-start latency and avoiding lost state on long sessions.

Latent Space

[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback (2026-10-21) — DeepSeek open-sourced DeepInfer, a vLLM-derived serving stack behind V4.1 Flash, reporting ~155T tokens/day per cluster, 96% input cache-hit rate, ~400ms TTFT, multi-drafter adaptive speculative-decoding verification (+6%), and SSD-backed shared KV cache with multi-day retention. The densest item of the week for anyone operating inference at scale — self-reported numbers, but vLLM independently confirms decode throughput at 6% of DeepInfer's on GB200, giving a real benchmark to size cache/scheduling decisions against.
[AINews] not much happened today (2026-10-20) — Covers the vLLM v0.32.0 release (814 commits): a structured-decoding endpoint, low-precision KV cache (NVFP4, 4-bit on ROCm), a Model Runner V2 with a GPU-side n-gram speculator, and multi-API workers giving +43-85% requests/s; precompiled-bytecode images cut container startup 14-18%. A direct throughput/startup-latency win for anyone already on vLLM deciding whether to upgrade.
[AINews] High Taste Testers Ep. 1 (2026-10-17) — Covers day-0 vLLM serving benchmarks for Reflection Beam 1.0 (1,769 tok/s output on 4xB300, NVFP4, TP4 at concurrency 16) and a +62% decode-throughput gain from LoRA via SkyRL (276→447 tok/s/GPU, decode only). Concrete capacity-planning numbers for anyone evaluating B300 + NVFP4 for serving new models.
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment (2026-10-16) — Cites Artificial Analysis data showing the NVIDIA Vera Rubin NVL72 rack delivering up to ~5.7x the throughput-per-megawatt of GB300 on DeepSeek V4 Pro in high-speed-per-user mode, narrowing to ~1.14-1.45x at peak throughput. A direct cost/energy-per-throughput data point for teams choosing between NVIDIA rack generations for inference at scale.
[AINews] Modal's $1B ARR, $16B Series D makes it NYC's newest AI Infra Decacorn (2026-10-15) — Notes vLLM's "adaptive verification" now works across multiple speculative-decoding drafters (headless MTP, EAGLE, DFlash, DSpark) without retraining, calibrating verification via top-token probability plus online logistic calibration. Lets you swap speculative-decoding drafters without retraining, cutting the operational cost of maintaining multiple decode-acceleration configs.
The Programmable Data Center — Evan Conrad, SF Compute (2026-10-20) — An interview on SF Compute's physical GPU compute market, noting a B300 running inference at 50% utilization nets ~$25/hr but can be rented for ~$5-6/hr, and explaining why batch vs. live inference get priced differently based on interruption tolerance. A direct unit-economics data point for buy-vs-rent decisions on inference capacity.

Through-line

The week's clearest pattern is that elasticity and cache locality are becoming first-class serving concerns, not afterthoughts: vLLM's near-zero-downtime Elastic EP, KVCompass's internet-scale prefix discovery, and Modal's sticky sessions all attack the same problem — keeping state (KV cache, session affinity, expert placement) coherent while infrastructure scales up and down underneath it. A second thread is that some of the biggest wins this week cost nothing extra: sorting prompts by domain before batching, or swapping speculative-decoding drafters without retraining, both squeeze real throughput out of existing deployments with zero new hardware or training runs. And DeepSeek's DeepInfer release, alongside vLLM's own v0.32.0 feature set, shows production-grade serving optimizations (SSD-backed KV cache, adaptive speculative verification, low-precision caches) converging across both open-source infra and in-house frontier labs.

What's catching your attention this week — elastic MoE scaling, the free batching trick, or DeepSeek opening up its serving stack? Drop a comment below.

Agents Write the Commits Now — This Week's Research Is All About Who Checks Their Work

Felipe 0liveira — Tue, 20 Oct 2026 12:06:43 +0000

This digest covers what's new for AI agent builders between October 13 and October 20, 2026: fresh papers on harness self-improvement and verification, production write-ups on agent memory and evals, and a couple of infrastructure stories about what happens when agents, not humans, become the primary users of your tooling.

🔥 Highlights

Redesigning source control at Anthropic — agents now out-commit humans 30,000 to one daily.
Context Engineering with Deep Agents: How To Remember Without Keeping Everything In Context — a concrete playbook for long-running agent memory.
The Most Common Questions About Agent Evals, Answered — the FAQ every agent team eventually needs.
Crossing the Threshold: How OpenAI and Anthropic Approach Autonomy and Safety in Financial Agents — where frontier labs actually draw the autonomy line.
Self-Evolving Coding Agents via Harness-Policy Optimization — the agent now rewrites its own scaffold.

arXiv

Learning to Steer LLM Agents and Multi-Agent Systems — 2026-10-15 Introduces a "steering agent" that watches single- and multi-agent executions during long-horizon tasks and intervenes when it detects drift from the goal, splitting the job into a cheap component that decides when to intervene and a more expensive one that decides how, trained jointly via RL. It generalizes to out-of-domain tasks and unseen executors without retraining to tune intervention frequency — useful if you're running long agentic loops that suffer from goal drift.

Hugging Face Daily Papers

Evidence-First Agentic Verification — 2026-10-14
Presents Sauron, a verifier for long, semi-verifiable tasks that checks every claim — including an agent's own self-reported success — against source evidence before trusting it, using bounded queries and joint investigation of related criteria to save context. Across 2,586 human-labeled criteria it posts the best F1 among five verifiers while costing less than the strongest baseline — a real option for teams that need something sturdier than loose "LLM-as-judge" checks.
Interlock: From Local Correctness to Task Completion in Long-Horizon Agents — 2026-10-15
Tackles the pattern of agents that satisfy every individual requirement yet lose the connection between evidence, decision, and downstream effect, producing output that's locally correct but incomplete. It represents the task as a typed completeness graph, compiles that into a resettable workspace plus evaluation rubric, and uses an agent-as-judge to filter trajectories; fine-tuning on the accepted trajectories improves pass rate on automation and legal/operations benchmarks.
Co-Evolving Acting and Forecasting for Language Agents — 2026-10-16
Introduces CoPE: a single model that acts and also learns to predict the future action sequence from any point in successful trajectories, using that forecast to shape the policy through shared parameters — at no extra inference cost, since the agent still acts step by step. It beats GRPO and observation-prediction baselines across five benchmarks (ALFWorld, ScienceWorld, WebShop, τ²-bench, AppWorld), with up to ~2x better sample efficiency on ScienceWorld — relevant if you train agents with RL on sparse, end-of-trajectory reward.
RSIFlow: Adaptive Self-Improvement Workflows with an Evolving Meta-Agent — 2026-10-16
A recursive self-improvement framework where a meta-agent diagnoses the performance bottleneck in a task agent's trajectories and picks which improvement workflow to apply (model, harness, or artifacts), distilling the results into reusable skills that feed back into orchestration. It reports state-of-the-art results across six benchmarks (code generation, tool use, search QA) with Qwen3-4B/8B.
Self-Evolving Coding Agents via Harness-Policy Optimization — 2026-10-19
Proposes Self-HPO: the coding agent alternates between proposing changes to its own harness (the scaffold controlling tools, context, and feedback) based on analysis of its own failures, and training the policy via RL under that chosen harness — tasks that still fail feed the next round of search. It lifts average resolve rate from 57.8% to 64.8% with Qwen3.5-35B-A3B, with gains that generalize to out-of-distribution benchmarks.
HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs — 2026-10-20
A label-free method for evolving an agent's harness using specialized agents that analyze past trajectories, propose edits, and score candidates with LLM-generated tests — producing three harness variants (lite/medium/heavy) for different cost budgets. In heavy mode, with GPT-5.5/Codex on Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, it gains 2 percentage points of pass rate while cutting per-task cost by 26.1% versus the RHO baseline.
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners — 2026-10-20
An agent-evaluation benchmark built from 31 real consulting engagements (totaling $3.68M) in operations research, paired with a simulated client (ORSim) that only reveals business facts when asked and has no technical expertise — testing whether the agent can formulate the problem, not just solve it. The best configuration (Claude Code) hits 88.39% strict feasibility, but no model matches the human expert on overall quality — a good reference for evaluating agents on open-ended reasoning domains beyond coding.

Anthropic Engineering Blog

Redesigning source control at Anthropic — 2026-10-13 Anthropic replaced the monorepo's git setup with an in-house system, Scion, built on the premise that most clients — agents and CI jobs — are always connected: commits land as server-side changesets, checkouts are sparse/virtualized, and a layered cache replaces hundreds of git mirrors. The practical point for agent builders is the programmatic version-control API — agents can query history, blame, and merge base without a local clone, enabling automated bisect/revert pipelines at scale (the stated trigger was growing from ~2,500 to 30,000+ merges/day).

OpenAI news

Nothing confirmed this week — openai.com/news and several openai.com/index/... URLs returned HTTP 403 on every fetch attempt during research, so no title/URL/date could be verified directly against the source, and per the no-guessing rule nothing is included.

LangChain / LangGraph blog

Context Engineering with Deep Agents: How To Remember Without Keeping Everything In Context — 2026-10-14
Explains how the open-source Deep Agents harness avoids losing precision on long tasks without flooding the context window: it offloads large tool results to files, summarizes old messages, loads skills on demand, isolates noisy work in subagents, and keeps long-term memory in plain files. Because memory lives in user-controlled storage, it survives a model or provider swap — useful if you're designing the memory layer for a production agent.
The Most Common Questions About Agent Evals, Answered — 2026-10-15
A technical FAQ on evaluating LLM agents: building datasets from production traces, the difference between offline experiments and online evaluators on live traffic, when to use code-based evaluators vs. LLM-judges vs. human review vs. pairwise comparison, multi-turn thread evaluation, and containerized benchmarks via Harbor that score an agent's final state. A direct reference if you're building out an agent's eval pipeline.
How Numan Built an Eval-First AI Health Coach with LangGraph and LangSmith — 2026-10-19
Numan's "Nu" health coach is a LangGraph graph where a supervisor dispatches to specialist subagents, each with its own prompt and tools; guard nodes validate every response against clinical safety rules and send unsafe drafts back for regeneration, with a safe fallback if that fails repeatedly. Evals run multi-turn simulations of the same multi-agent graph scored by category-specific LLM judges — a solid case study in multi-agent orchestration plus eval-first design for sensitive domains.
How Orange Business Built Live Intelligence with LangChain and LangSmith — 2026-10-13
Orange Business built its "Live Intelligence Studio" on LangChain (for model neutrality and control over agent architecture) with LangSmith for tracing/eval. Notable pieces: an MCP Registry for governed, auditable agent access to enterprise tools (M365, CRMs, internal APIs), and an Agent Registry for versioning and reusing agents/prompts, with multi-agent flows inspectable via per-step execution trees (input/output/latency).
Search agent traces by behavior with AI Search in LangSmith — 2026-10-13
A new LangSmith feature (AI Search) lets you search agent traces by describing behavior in natural language (e.g. "frustrated users" or "leaked PII") instead of relying on tags or full-text search. Under the hood, SmithDB condenses each session into a deduplicated "trajectory" within a fixed token budget, and a fast decision model filters candidates in parallel — relevant if you need to debug or evaluate agent behavior in production.

Latent Space

Crossing the Threshold: How OpenAI and Anthropic Approach Autonomy and Safety in Financial Agents — 2026-10-14
Recaps an AI Engineer NYC panel with OpenAI, Anthropic, Two Sigma, and Bloomberg on deploying agents in finance. The core argument: safe autonomy is about where you place controls and human judgment, not the model itself — scoped identities, human approval gates for consequential actions, classifiers for ambiguous cases, plus out-of-model safeguards like audit logs and kill switches. A concrete, layered-control checklist for anyone shipping agents into a high-stakes, regulated domain.
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — 2026-10-13
Interview with Jason Ma (Dyna Robotics) on Dyna-2, a world-action model trained on over a million hours of human video that predicts future video frames and derives robot actions from that prediction. The notable control-loop detail: an agentic harness handles high-level decision-making and planning so the low-level policy can run long multi-step workflows (like an hour-long laundry task) while still doing ~100-dimension continuous action control for smooth motion — a useful read on hierarchical planning loops outside the chat-agent context.

Simon Willison

A quote from Geoffrey Huntley — 2026-10-13
A short, concrete trick for enforcing tool-use policy on a coding agent: instead of relying on instructions in AGENTS.md (which drift), add a pre-commit hook that scans the diff for calls bypassing your approved library, fails the commit, and feeds a corrective prompt back to the agent. Reported as getting compliance to "ten out of ten" — a directly reusable pattern for fighting tool-call drift in coding agents.
Redesigning source control at Anthropic — 2026-10-13
Simon's own writeup linking to Anthropic's Scion post, pulling out that Claude is Scion's heaviest user (which is why offline support got deprioritized) and that checkouts increasingly originate from agent sandboxes. He also flags the piece's description of adversarial review — one model hunting for flaws in another's work — as effective but prone to pushing toward over-engineering, a useful caution for anyone using agent-vs-agent review as a control pattern.
A quote from Addy Osmani, Morgan Westlee Lunt, and Qing Wang — 2026-10-16
A pull-quote from "How the Claude Code team does code review," arguing that whether a human read a diff is a weak proxy for risk, and that review is shifting from a final PR gate to a continuous process where Claude reviews like "spellcheck while you write," passing most changes before a human ever looks. A practical framing for agent builders: decide per-check whether an automated agent or a human catches a given class of problem best, and save human attention for higher-level judgment calls.

Through-line

Three independent threads converge on the same question this week: how do you trust what an agent just did? The harness-self-improvement cluster (RSIFlow, Self-HPO, HarnessPro) treats the agent's own scaffold as a thing to be evolved based on failure analysis, not just the model inside it. The verification papers (Sauron, Interlock) push back against taking an agent's self-reported success at face value, insisting on evidence-first checks instead. And the production write-ups — LangChain's eval FAQ, Numan's guard-node health coach, LangSmith's behavior search — all describe evals and tracing as the thing standing between "the agent did something" and "the agent did the right thing." Anthropic's own infrastructure story, covered independently by both its engineering blog and Simon Willison, is the starkest illustration: once agents outnumber humans 30,000 to one as commit authors, the tooling question stops being about developer experience and becomes about verification at scale.

What's catching your attention in agent evals or harness design this week? Drop a comment below.

RAG Grows Up — Config Bugs, Poisoned Indexes, and a Reckoning Over What Counts as Agent Memory

Felipe 0liveira — Mon, 19 Oct 2026 12:07:51 +0000

This digest covers RAG/GraphRAG developments from 2026-10-12 through 2026-10-19. The week's arXiv crop is unusually production-minded — less "new retrieval architecture", more "here's what actually breaks when you run this in prod" — paired with a vendor-side push on agent memory and context engineering.

🔥 Highlights

Configuring RAG Systems: What Breaks, Where Originates, and How to Fix — a 654-issue field study of real RAG config failures.
FuseRAG: Fusing Structure and Evidence into Inference for Cost-Efficient Graph-based RAG — up to 2.7x faster GraphRAG without a quality hit.
Poisoning the Neighborhood: A Topology-Aware Topic-Level Attack on Large-Scale HNSW-based RAG Systems — your HNSW index is more attackable than you think.
Beyond Downstream QA: Rethinking How We Evaluate Agent Memory — downstream QA accuracy hides broken agent memory.
Context Engineering with Deep Agents: How To Remember Without Keeping Everything In Context — offload, don't truncate, long-running agent context.

arXiv

Configuring RAG Systems: What Breaks, Where Originates, and How to Fix — 2026-10-13
The first large-scale empirical study of real RAG configuration issues: 654 cases collected across 10 distinct RAG systems, categorized by symptom, root cause, pipeline stage, and fix pattern. It distills 9 actionable insights for testing, localizing, and fixing config problems — directly useful if you run RAG in production and keep chasing bugs that seem to teleport between pipeline stages.
Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG — 2026-10-15
Separates two distinct causes of "retrieval failure" in multi-hop RAG: evidence the retriever simply can't reach, versus "search drift," where the agent loses track of an unanswered sub-question. Fixing drift recovers evidence but doesn't cut errors proportionally — the bottleneck shifts to verifying whether retrieved evidence is actually sufficient, which changes where you should spend engineering effort.
FuseRAG: Fusing Structure and Evidence into Inference for Cost-Efficient Graph-based RAG — 2026-10-16
Instead of flattening the retrieved graph into linear text context (which destroys structure and breaks KV-cache reuse), FuseRAG separates reusable content from query-aware structural interactions and injects the latter directly into decoder self-attention. Result: 2.2–2.7x TTFT speedup over the best GraphRAG baseline with no quality loss — relevant if GraphRAG's latency and cache overhead are what's stopping you from shipping it.
An Interactive GraphRAG Workspace for Reifying, Auditing, and Steering Conversational AI Memories — 2026-10-16
Introduces CognitiveCanvas, which turns conversation history (e.g. LLM dev sessions) into a navigable Hierarchical Semantic Fact-Graph via multi-hop GraphRAG, cutting token usage by 92.2%, plus a "temporal drift" auditor that flags stale context (outdated APIs/packages) before it gets reused. A practical tool for anyone who wants auditable GraphRAG memory instead of an opaque index.
Poisoning the Neighborhood: A Topology-Aware Topic-Level Attack on Large-Scale HNSW-based RAG Systems — 2026-10-15
Shows that poisoning attacks designed for exact retrieval fail against the approximate HNSW indexes used in production, then proposes a topology-aware attack (exploiting "hub-satellite" structure) that generalizes to unseen topics — 31% attack success on a 2M-document Wikipedia index, which current defenses don't neutralize. A direct security warning for anyone running HNSW-backed RAG (FAISS, Milvus, etc.) over public or third-party ingestion.
PrairieFire: Igniting High-Throughput Updates in Large-Scale Approximate Nearest Neighbor Search — 2026-10-16
Tackles the update bottleneck in SSD-based ANNS (CPU-heavy rebuilds) with buffered updates and local batched rebuilds, gaining 3–35x update throughput and 1.5–8.7x cost-efficiency while preserving search performance. Already running in production at RedNote for two months — real-world evidence that billion-scale "fresh" vector indexes don't require full rebuilds.
SAG2: Embedding-Free Multi-Hop Retrieval with BM25 and JEV — 2026-10-15
An embedding-free multi-hop retrieval architecture: just SQLite + BM25 plus a lightweight "System One" model (JEV) that makes contextual evidence decisions at query time instead of precomputing embeddings. Indexing ~11.6k passages takes 2.36s with zero model calls, and it matches dense-retrieval recall on MuSiQue/2Wiki/HotpotQA — an option if you want to skip the embedding/vector-DB infrastructure entirely.
When Does Uncertainty Measure RAG Hallucination Risk? — 2026-10-15
Shows that uncertainty (perplexity) is only a reliable hallucination signal in RAG when retrieved evidence actually constrains the answer — with poor retrieval coverage, the model answers from internal knowledge with high confidence, masking the real risk. Combining uncertainty with a "semantic drift" score improves calibration; useful if you're building hallucination-detection guardrails.
Beyond Downstream QA: Rethinking How We Evaluate Agent Memory — 2026-10-15
Argues that evaluating agent memory purely through downstream QA accuracy conflates memory quality with retrieval/generation quality, producing false positives and negatives. Introduces the FEB-30 benchmark (deterministic ground truth via an inverted paradigm) and shows a representative memory system correctly classifies memory operations (create/update/reinforce/supersede) only 20% of the time — a failure invisible to traditional downstream evaluation.
SAFE-Mem: Secure Agentic Memory with Fine-Grained Authority and Governance-Aware Retrieval — 2026-10-16
Proposes fine-grained governance for agent memory: every "claim" gets provenance/authority/scope labels, and retrieval cross-checks a claim's permissions against the query's requirements before using it — blocking injection from untrusted sources without sacrificing utility. Cuts attack success by at least 77% against the strongest defenses tested while keeping the best answer accuracy among all evaluated defenses.
CASI: Credit Assignment via Set Intervention for Agentic Retrieval — 2026-10-14
Addresses the problem of a single reward covering a whole set of retrieval actions generated together, using counterfactual intervention over snapshots to assign individual credit via Shapley values. Validated with a real 7-day production A/B test (10% of traffic), showing a 3.51% lift on the primary value metric — it's rare to see agentic retrieval with this explicit a production validation.
Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking — 2026-10-16
Uses a "System One" model as both relevance judge and reranker across four Brazilian Portuguese IR collections; as a judge it agrees with official judgments (kappa 0.52–0.82) and preserves system rankings (Kendall's tau 0.90–0.93); as a reranker over BM25's top-1000, it lifts nDCG@10 from 0.33 to 0.63 (+90%). Directly relevant if you evaluate or rerank retrieval in Portuguese.

Hugging Face Daily Papers

Calibrated Typed Decisions for Entity, Relation, and Event Extraction — 2026-10-12
Presents CalOn, a unified 0.6B-parameter encoder for entity, relation, and event extraction across different domains/schemas, with post-training confidence calibration. It's exactly the kind of building block used to construct the knowledge graph in GraphRAG pipelines from raw text.
Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents — 2026-10-13
Instead of reprocessing the full interaction history at every step, CarryOn organizes agent continuity around reusable "task objects" that get updated incrementally rather than rebuilt from scratch each fold. On OSWorld 2.0 and BrowseComp-Plus it cuts total token usage by 22.9–51.9% while improving performance — relevant if you're designing memory/context for long-running retrieval agents.
Cortex: Consolidating Context into Parametric Memory — 2026-10-14
Proposes "memory consolidation": instead of keeping context gains alive only via RAG or a long context window, it distills the effect of context into parametric updates (low-rank or FFN edits) that can be merged into the model while preserving unrelated capabilities. After 50 sequential factual edits, it recovers 88.9% of the context-induced gain while keeping 99.8% of original performance — an interesting counterpoint to pure RAG for long-term agent memory.

Anthropic Engineering Blog

Nothing relevant this week. The one post published in the window, "Redesigning source control at Anthropic" (2026-10-13), covers a monorepo migration to an internal system called Scion — solid infra engineering, but it doesn't touch retrieval, embeddings, or agent memory.

LangChain blog

Context Engineering with Deep Agents: How To Remember Without Keeping Everything In Context — 2026-10-14
LangChain's Deep Agents framework fights "context rot" with five techniques: offloading large tool results to files instead of truncating them, reversible summarization once the context window hits 85%, progressive-disclosure skills, subagents for noisy work, and long-term memory stored in markdown and reloaded each run. The core takeaway for RAG/agent-memory builders: prefer offload over truncation (recoverable data beats lost data), and treat filesystem search as a complement to RAG, not a replacement, for context accumulation.
The Most Common Questions About Agent Evals, Answered — 2026-10-15
A practical FAQ on evaluating agents: building small datasets from real traces, the difference between offline evals (regression) and online evals (production), evaluator types (code checks, LLM judges, human review, pairwise comparison), and how to calibrate LLM judges with binary scales instead of 1–10 ratings. Useful if you need a continuous eval loop for RAG/agent systems, though it doesn't dig into RAG-specific metrics like grounding or chunk relevance.
Building Long-Running Agents with ScaleDown and Deep Agents — 2026-10-13
Shows the langchain-scaledown integration, which uses small specialized models (summarize/compress/extract/classify) as cheaper middleware than frontier LLMs for context management: threshold-based summarization, query-guided tool-result compression (one example went from 18,402 to 3,110 characters), and automatic structured extraction from user messages. Relevant if you're already on Deep Agents and want to cut context-management cost/latency without losing quality.

LlamaIndex blog

ExtractBench Pro: The Long Tail of Document Extraction — 2026-10-15 A new benchmark (extending ExtractBench) with 100 synthetic documents covering six hard challenge types — document conventions, domain vocabulary, counting rules, drawn relationships, embedded charts, and long spreadsheets — tested against 12 systems (extraction APIs, code agents, LlamaExtract tiers). Key finding: scores drop sharply versus the original benchmark, grounding (citing where a value came from) doesn't track extraction quality, and LlamaExtract's tiers offer the best cost-performance below ~$0.25/page. Dataset, code, and paper are public — a useful quality reference if your RAG pipeline depends on document extraction.

Neo4j blog

Graph Embeddings, One Line of Code Away — 2026-10-16 A new preview API in the GDS Python client (Aura Graph Analytics): gds.embedding.create(G) generates node embeddings via FastRP in one line, and gds.embedding.train trains a reusable supervised GraphSAGE encoder. Embeddings live in the in-memory session graph (not written to the database by default) and feed other algorithms (kNN, k-means) or can be written back. The post doesn't talk about RAG directly, but its framing — text embeddings capture "what things say," graph embeddings capture "how things connect" — is the textbook case for combining both in GraphRAG.

Simon Willison

Nothing relevant this week. All posts published in the window were tool release notes (llm-echo, llm-tools-datasette, llm-gemini, llm-meta-ai, web-edit, datasette-ports) or link/commentary posts on Claude Code's review process, Anthropic's source-control system, and other topics with no retrieval, embedding, or agent-memory substance.

Latent Space

[AINews] AI Engineer NYC: Gemini 4 Argon, Vals AI Benchmarks — 2026-10-14
This roundup covers Meta's "Active Memory" feature (tested via MemoryBench, which checks whether a model updates a fact within one long conversation but explicitly doesn't test durable cross-session memory) and Meta/MIT's "ReHarness," which rewrites retrieved reflections based on downstream task outcomes instead of just adding/deleting memory entries. It also flags "LongHarness," which separates evidence from conclusions into explicit state and reportedly pushed GLM-5.3-Flash from 4.4% to 61.1% exact match on the OfficeQA Pro V2 long-document benchmark without touching the model — a concrete signal that explicit evidence/conclusion splitting can beat model upgrades on long-document QA, and a reminder that most "agent memory" benchmarks still only test single-session updates.
[AINews] High Taste Testers Ep. 1 — 2026-10-17
Reports Perplexity's pplx-decider v1.2 adding reranking, Tencent's "Hy Memory 2.0" agent-memory framework release, and research summaries on FDE-Gym (agent performance on real customer-delivery tasks) and "history-aware routing" for agents. For RAG-focused readers, the reranking update and history-aware agentic-retrieval routing are the most actionable items, though primary sources for the memory framework and routing research aren't published yet.
How Anthropic's Dynamic Workflows Coordinate Claude Agents (Updated with video!) — 2026-10-12
Covers "flows" in Claude Console, where a lead agent plans and orchestrates its own subagents instead of relying on a hand-configured roster; subagents share a filesystem but can't message each other directly. The most memory-relevant detail is "Dreaming," a research-preview feature that periodically reads an agent's memory store and past transcripts and produces a reorganized store with duplicates merged and stale entries replaced — a concrete offline-consolidation pattern for agent memory, distinct from simple append-only logs.

Interconnects

Nothing relevant this week. The only post in the window, "Meta shakes up the frontiers, open and closed" (2026-10-17), is a model-landscape and licensing analysis with no discussion of retrieval, embeddings, reranking, or agent memory.

Through-line

Two threads run across otherwise independent sources this week. First, production RAG is entering its "ops maturity" phase: the config-failure taxonomy, the HNSW poisoning attack, and PrairieFire's update-throughput fix are all about what breaks once RAG leaves the demo and hits real traffic, not about new retrieval tricks. Second, agent memory is having a methodology crisis in public — arXiv's FEB-30 benchmark and SAFE-Mem governance work, Latent Space's coverage of MemoryBench's single-session limits, and LangChain's "offload, don't truncate" context engineering are all, from different angles, arguing that today's memory systems are evaluated and built in ways that hide their real failure modes.

What's catching your eye this week — the GraphRAG latency fix, the HNSW poisoning attack, or the agent-memory evaluation gap? Drop a comment below.