DEV Community: pueding

Agent-Harness Scaling Law: Feedback Quality Predicts Success, Not Raw Compute: Effective Feedback Compute (EFC)

pueding — Wed, 10 Jun 2026 11:15:58 +0000

What: A new agent-harness scaling-law paper introduces Effective Feedback Compute (EFC) — a single quantity that predicts whether an agent finishes a task from the quality of the feedback its harness returns each step, scored on four axes and normalized by how hard the task is.

Why: It reframes agent reliability as a feedback-quality problem, not a token-budget problem — plotted against EFC, harness-run success follows a clean law (R²≈0.94–0.99), while against raw compute the same runs barely fit (R²≈0.33–0.42).

vs prior: Prior reliability work leaned on raw-compute scaling — more tokens, more tool calls, bigger reasoning budgets — but EFC shows that axis is nearly flat, since lifting only feedback quality moved success from 0.27 to 0.90 with cost and tool-call counts held fixed.

Think of it as

a student with a sharp tutor instead of just re-reading the textbook

                  SAME EXAM, SAME HOURS LOGGED
                             │
               ┌─────────────┴──────────────┐
               ▼                            ▼
       ┌───────────────┐          ┌───────────────┐
       │  RE-READ THE  │          │  SHARP TUTOR  │
       │    TEXTBOOK   │          │  per problem  │
       │ (raw compute) │          │ (feedback Q)  │
       └───────┬───────┘          └───────┬───────┘
               │                          │
      pages logged, but          points at the exact
      no correction lands        mistake — and it sticks
               │                          │
               ▼                          ▼
         ✗ grade ~0.27              ✓ grade ~0.90
         effort, no signal         signal absorbed

agent harness = the study setup that feeds you a correction each round
raw compute = hours logged and pages re-read
feedback quality = how useful the tutor's correction is each time
informativeness = the tutor points at the exact mistake, not "study harder"
validity = the correction is actually right, not misleading
non-redundancy = the tutor doesn't repeat a note you already wrote down
retention = you keep the correction in your notes for the next problem
EFC = total useful correction absorbed, divided by how hard the exam is

Quick glossary

EFC — Effective Feedback Compute — the paper's core metric. It measures how much useful feedback signal a harness feeds back into the agent loop, scored on four axes (informativeness, validity, non-redundancy, retention) and normalized by task demand. It is the x-axis of the proposed scaling law, replacing "tokens and tool calls spent."

Agent harness — The scaffolding around the model — the loop that runs tool calls, observes results, and feeds the next observation back to the model. The harness is what delivers feedback, so it is where EFC is won or lost. Covered in Agent Engineering → Production Harness Architecture.

Scaling law — An empirical curve that predicts an outcome (here, task success rate) from one quantity (here, EFC). A tight scaling law means the curve explains most of the variation; a loose one means the quantity is a poor predictor.

R² (fit quality) — The fraction of variation in success the curve explains, from 0 (the x-axis predicts nothing) to 1 (it predicts everything). EFC reaches R²≈0.94–0.99; the raw-compute baseline only 0.33–0.42. Higher R² = a better predictor.

The four feedback axes — Informativeness (does the message localize the error?), validity (is the correction actually right?), non-redundancy (is it new, or a repeat?), and retention (does the agent still have it later?). EFC is built from all four, so a harness can fail on any one of them.

Task demand — How much corrective signal a task actually needs to be solved. EFC divides feedback quality by task demand so harnesses can be compared fairly across easy and hard tasks — the same crisp feedback is worth more on a demanding task than a trivial one.

The news. On May 28, 2026, researchers posted an agent-harness scaling-law paper to arXiv introducing Effective Feedback Compute (EFC) — a metric that predicts agent success from the quality of feedback the harness returns, not the compute it spends. Plotted against EFC, harness-run success rates fit a clean scaling law (reported R²≈0.94–0.99 across datasets); plotted against raw compute, the same runs barely fit (R²≈0.33–0.42, rising to ~0.88 only with a hand-built multivariate baseline). In one controlled comparison, lifting feedback quality moved success from 0.27 to 0.90 with token cost and tool calls held fixed.

Picture two students prepping for the same exam. The first logs ten hours re-reading the textbook cover to cover — enormous effort, page after page. The second spends one hour with a sharp tutor who, after each practice problem, points at the exact line where the reasoning went wrong, confirms the fix is correct, never repeats a note already written down, and makes sure it lands in the margin for next time. On exam day the second student wins, and it is not close. The hours-logged number — the raw compute — told you almost nothing. The number that predicted the grade was how much useful correction actually got absorbed. That second number is what this paper names Effective Feedback Compute, and the claim is that agent harnesses behave the same way.

The mechanism is a re-definition of the x-axis. Instead of counting tokens or tool invocations, EFC measures the useful signal the harness feeds back each step — scored on four axes (informativeness, validity, non-redundancy, retention) — and then normalizes by task demand so a crisp correction counts for more on a hard task than an easy one. That normalized quantity becomes the horizontal axis of a scaling law that fits success rates across the paper's datasets. The practical reading for anyone building agents: the lever is not your reasoning budget but what your harness chooses to log and return after every tool call.

This is why the raw-compute axis goes flat. A harness can burn an enormous budget returning low-quality feedback — a terse exit code 1 with no stack trace (low informativeness), a linter warning that is actually a false positive (low validity), the same "tests failed" string ten turns in a row (high redundancy), or an error the agent has already forgotten by the time it matters (low retention). All of that is real compute and real tool calls, and on the EFC axis it is worth almost nothing. The tutor who just says "study harder" for an hour spent the hour; the student learned nothing. Worse, in a long rollout the low-signal steps let compounding errors accumulate unchecked, so the spend actively buys you a longer path to the same failure.

Where the feedback gap actually comes from

Hold three variables fixed. One agent. One task. Two runs at the same budget — 40 tool calls, ~120K tokens each. The only difference is the harness's feedback quality. In Run A, every step returns a terse pass/fail string; say each step carries about 0.1 units of useful, valid, non-redundant, retained signal, so over 40 steps the agent accumulates 40 × 0.1 = 4 units. The task demands roughly 30 units to solve, so EFC = 4 / 30 ≈ 0.13 — low on the law's curve, landing near the 0.27 success rate the paper reports at the bottom of its range. In Run B, the harness returns the failing assertion, the offending input, and a one-line diff each step — call it 0.8 units per step, 40 × 0.8 = 32 units, EFC = 32 / 30 ≈ 1.07, high on the curve and up near 0.90 success. Same cost, same tool count, ~8× the effective feedback (illustrative decomposition calibrated to the paper's 0.27→0.90 and R² headline figures — the per-step unit values and task-demand figure are stand-ins, not measured constants). The success jump is the headline; the per-call yield jump is the deeper story.

Scaling-law x-axis	What it counts	Fit to success (R²)
Raw compute	tokens + tool calls spent	~0.33–0.42 — poor (paper)
Multivariate compute baseline	several spend features combined	~0.88 — better, hand-built (paper)
Effective Feedback Compute (EFC)	4-axis feedback quality ÷ task demand	~0.94–0.99 — tight (paper)

A caveat worth stating plainly: this is a scaling-law fit on the paper's own datasets, and a tight fit is a strong correlation, not a guaranteed control knob. EFC is also harder to move than a token budget — "return better feedback" is a design problem, not a slider, and scoring the four axes reliably is itself non-trivial. The honest framing is that EFC gives you a yardstick and a direction: instrument the feedback your harness returns, A/B candidate changes in shadow, and treat feedback quality as a first-class number alongside latency and cost. Whether the exact coefficients transfer to your stack is exactly the kind of thing you should measure, not assume.

Goes deeper in: AI Agents → Evals & Diagnostics → Error analysis first

Related explainers

PushBench — Quantitative Goal Persistence (QGP) — another harness-level number for long-horizon agent reliability
FutureSim — harness-level agent eval — why evaluating the harness, not the model alone, is the trend
Cursor Composer 2.5 — targeted textual feedback RL — the training-time analogue: a sharp, targeted correction beats a blunt end-of-rollout reward

FAQ

What is Effective Feedback Compute (EFC)?

EFC is a metric that predicts agent-harness success from the quality of the feedback the harness returns each step, rather than from the raw compute it spends. It scores feedback on four axes — informativeness, validity, non-redundancy, and retention — and normalizes by task demand so harnesses can be compared fairly across easy and hard tasks. Plotted against EFC, the paper reports success rates fitting a scaling law at R²≈0.94–0.99, far tighter than the ~0.33–0.42 fit against raw compute.

Why does feedback quality predict success better than raw compute?

A harness can spend an enormous budget returning low-quality feedback — terse pass/fail strings, false-positive warnings, repeated messages, or errors the agent has already forgotten. That is real compute that carries almost no useful signal, so the raw-compute axis goes nearly flat. EFC captures the signal that actually reaches the agent, which is why it fits success so much more tightly. In one controlled comparison, lifting only feedback quality moved success from 0.27 to 0.90 with token cost and tool-call counts held fixed.

How do I improve a harness's EFC in practice?

Treat the feedback your harness returns as a first-class design surface: make tool-call results localize the error (informativeness), verify the signal is correct before returning it (validity), suppress repeated or stale messages (non-redundancy), and persist corrections so they survive later in the rollout (retention). Because EFC is a measurable yardstick rather than a slider, the practical loop is to instrument the feedback you return, A/B candidate changes in shadow mode, and track feedback quality alongside latency and cost.

Originally posted on Learn AI Visually.

AutoLab Benchmarks Frontier Agents on Long-Horizon R&D Tasks: Iterative Experiment-Loop Evaluation

pueding — Tue, 09 Jun 2026 11:25:04 +0000

What: The AutoLab benchmark scores agents with iterative experiment-loop evaluation — 36 realistic R&D tasks (optimize a system, tune a CUDA kernel, build a model) where the agent has to propose a change, run an experiment, measure the result, and refine, over and over.

Why: Across 17 frontier models, the strongest predictor of success was sustained iteration that incorporates empirical feedback plus time-awareness — knowing when to keep going — rather than the quality of the first answer.

vs prior: Most LLM benchmarks grade a single answer once; AutoLab grades the whole propose → run → measure → refine loop under a budget, exposing two failure modes a one-shot score is blind to: stopping too early and burning the budget with no measured progress.

Think of it as

tuning a race car in the pit, reading lap times until qualifying closes

         SAME CAR, SAME LAP BUDGET (12 laps)
                          │
        ┌─────────────────┬─────────────────┐
        ▼                 ▼                 ▼
   ┌─────────┐       ┌─────────┐       ┌─────────┐
   │ PARK    │       │ RE-TUNE │       │ TIME +  │
   │ EARLY   │       │ NEVER   │       │ TUNE    │
   │         │       │ TIME    │       │ EVERY   │
   │ 4 laps, │       │ 12 laps,│       │ LAP     │
   │ then    │       │ no clock│       │ 8 timed │
   │ quit    │       │ reading │       │ laps    │
   └────┬────┘       └────┬────┘       └────┬────┘
        ▼                 ▼                 ▼
   stops at          random-walks       compounds to
   ~0.46             ~0.27              ~0.76
   ✗ budget          ✗ no measured      ✓ best lap
     left unused       progress           wins slot

task = set the fastest lap before qualifying closes
experiment loop = adjust the setup → run a lap → read the lap time → adjust again
empirical feedback = the lap time on the stopwatch, not a guess from the spec sheet
budget = the laps you have before the qualifying flag drops
stopping early = parking after two laps with time still on the clock
burning the budget = re-tuning every lap but never reading the timer
persistence = keep timing and tuning until the very last lap

Quick glossary

Long-horizon task — A task that takes many steps and a real budget to finish — not one question with one answer, but a goal you reach by doing work, checking it, and adjusting. AutoLab's tasks run for many tool-using steps.

Experiment loop — The repeating cycle at the heart of R&D work: propose a change → run an experiment or benchmark → measure the outcome → refine. AutoLab scores whether an agent actually keeps this loop turning, not just whether its first attempt looked good.

Empirical feedback — A result you measured by running something — a benchmark number, a test pass/fail, a latency reading — as opposed to a guess. The key move is conditioning the next edit on a number the agent ran itself.

Time-awareness — The agent's sense of how much budget is left and whether more iteration is worth it. Failing it shows up two ways: quitting with budget unspent, or thrashing until the budget runs out with nothing to show.

Agent harness — The runtime that wraps a model into an agent — it schedules tool calls, runs the experiments, and feeds results back into the loop. The same model in a better harness can score very differently.

CUDA-kernel optimization — One of AutoLab's four domains: rewrite a GPU kernel to run faster, then benchmark it to see if it actually did. It is a textbook measure-and-refine loop — and it ties this agent benchmark to the GPU & CUDA track.

The news. Posted to arXiv on June 3, 2026, AutoLab is a benchmark of 36 long-horizon R&D tasks across four domains — system optimization, puzzle & challenge, model development, and CUDA-kernel optimization — that ask an agent to propose changes, run experiments, measure outcomes, and iterate. Evaluating 17 state-of-the-art models, the dominant predictor of success was persistence in repeatedly benchmarking, editing, and incorporating empirical feedback — not the quality of the initial response. Most frontier models either stopped prematurely or burned their budget with minimal progress; Claude-opus-4.6 showed the strongest long-horizon optimization behavior. Read the paper →

Picture a pit crew with a fixed number of laps before qualifying closes. The car that wins the slot isn't the one that posted the best first lap — it's the one whose crew keeps reading the lap time, adjusting the setup, and sending it back out until the flag drops. AutoLab is built on exactly this insight for agents: it hands an agent a real engineering goal and a budget, then watches not the first attempt but whether the agent keeps the experiment loop — propose → run → measure → refine — turning all the way to the deadline.

That loop is the whole concept of iterative experiment-loop evaluation. A classic LLM benchmark asks one question and grades one answer; the agent never gets to run anything. AutoLab instead scores the agent on tasks where it must execute its own experiments and read its own results — the errors compound across a long trajectory, so the only way to climb is to measure, learn, and correct. Crucially, the useful signal here is empirical feedback the agent generates itself (it benchmarks its own kernel and reads the number), which is a different lever from feedback a harness hands back step-by-step.

The benchmark's headline finding is that frontier models fail this in two distinct ways, and both are about knowing when to stop. Some agents stop too early — they post a decent second attempt and quit with most of the budget unspent. Others burn the whole budget but skip the measure step: they keep editing without conditioning each change on a result, so the score random-walks and never compounds. The agents that did well — led by Claude-opus-4.6 — spent their reasoning budget on a disciplined measure-then-refine cadence, which is exactly the time-awareness a one-shot eval can never see.

Why does this matter beyond a leaderboard? Because it relocates the bottleneck for long-horizon agents from raw capability to behavior under a budget. The same skill that tops AutoLab — sustained, measured iteration — is what production teams care about when an agent tunes a config, optimizes a kernel, or chases a flaky test over an afternoon. That makes AutoLab a production-eval signal, not just an academic one: it predicts whether an agent will actually grind a real task to a good result instead of giving up or spinning.

AutoLab domain	What the agent iterates on	What it measures each loop
System optimization	Configs, flags, resource allocation	Throughput / latency of a benchmark run
CUDA-kernel optimization	A GPU kernel's implementation	Wall-clock kernel time vs a baseline
Model development	Training / architecture choices	A validation metric on a held-out set
Puzzle & challenge	Candidate solutions to a hard problem	Pass / fail against the checker

Four domains, 36 tasks total across them; the exact per-task scores are reported in the paper, and the row examples above are illustrative of the loop structure (AutoLab, arXiv 2606.05080).

Where the budget actually goes (numbers illustrative — AutoLab reports the model ranking and the persistence finding, not these per-task point values). Hold three things fixed: a budget of 12 experiment runs, a starting score of ~0.23 (the first answer — roughly the same for all three agents), and a per-loop gain that only lands when the agent measures. Agent A makes 4 measured runs at about +0.06 each, reaches ~0.46, then stops with 8 runs unused. Agent B spends all 12 runs but skips the measure step, so its edits aren't conditioned on a read result — its score random-walks around ~0.27 and never compounds. Agent C makes 8 measured runs, each conditioned on the last result, compounding to ~0.76. Same start, same budget; the entire gap comes from how the loop was spent, not from the first try.

Goes deeper in: AI Agents → Evals & Diagnostics → Compounding errors

FAQ

What is iterative experiment-loop evaluation?

It is scoring an agent on whether it keeps a propose → run → measure → refine loop turning, rather than grading a single answer. AutoLab gives the agent a real R&D task and a budget, then rewards measured iteration toward a better result instead of a good-looking first attempt.

Why does sustained iteration beat initial answer quality?

On long-horizon tasks the first attempt is rarely the best one, and errors compound. The agents that win are the ones that read an empirical result, correct, and repeat — using their whole budget. AutoLab found this disposition, not first-shot quality, was the dominant predictor across 17 models.

How does AutoLab relate to benchmarks like EFC and QGP?

They are complementary lenses on long-horizon agent reliability. EFC isolates the quality of the feedback signal a harness returns; QGP measures whether an agent finishes a fixed count of work without spinning; AutoLab measures whether the agent sustains its own measure-and-refine loop under a budget on realistic R&D tasks.

Originally posted on Learn AI Visually.

MCP SEP-2106: Full JSON Schema 2020-12 in Tool I/O

pueding — Mon, 08 Jun 2026 11:18:18 +0000

What: MCP SEP-2106 — merged into the protocol on May 18, 2026 — lets an MCP tool describe its inputs and outputs with the full JSON Schema 2020-12 keyword set in inputSchema and outputSchema, and widens structuredContent from object-only to any JSON value.

Why: Composition (oneOf / anyOf / allOf), conditionals (if / then / else), and references ($ref / $defs) let a tool author push contract rules out of free-form description prose and into the schema, where runtimes and SDKs can validate them before the call ever reaches the tool.

vs prior: The previous MCP spec accepted only a narrow JSON Schema subset (object root with a basic type / properties / required vocabulary); composition, conditionals, refs, and non-object output shapes were not part of the wire vocabulary and had to live in tool description prose.

Think of it as

It's like a job application form with conditional sections, alternatives, and refs.

           THE TOOL'S inputSchema (a form)
                        │
        ┌───────────────┴────────────────┐
 ┌──────▼───────┐                 ┌───────▼──────┐
 │ BEFORE 2106  │                 │ AFTER 2106   │
 │ plain fields │                 │ fields PLUS  │
 │ + prose note │                 │ oneOf / if / │
 │ at the bottom│                 │ then / $ref  │
 └──────┬───────┘                 └───────┬──────┘
        │                                 │
  rules live in                    rules live in
  English prose                    the schema
        │                                 │
        ▼                                 ▼
 ✗ runtime CANNOT                 ✓ runtime REJECTS
   check them                       bad calls early

MCP tool inputSchema = the application form a tool requires from the agent
basic keywords (type / properties / required) = plain text fields and required checkboxes
oneOf / anyOf / allOf = pick exactly one / any combination / all of these alternatives
if / then / else = if you marked 'married', also fill spouse details
$ref / $defs = see the 'Company Address' subform on page 4

Quick glossary

MCP — The Model Context Protocol — a JSON-RPC wire protocol that lets LLM clients (Claude, ChatGPT, IDEs) discover and call tools served by external processes. See the MCP step in the Tool Use module.

SEP — A Specification Enhancement Proposal — the MCP equivalent of a Python PEP or a TC39 proposal. Each SEP is a numbered RFC merged into the spec only after review.

inputSchema / outputSchema — The two JSON Schema documents an MCP server attaches to a tool definition — one for the arguments the agent must send, one for the structured value the tool returns. The runtime validates traffic against them before either side sees a malformed payload.

structuredContent — The field inside a tool result that carries a typed value alongside the human-readable content blocks. Pre-SEP-2106 the TypeScript type was { [key: string]: unknown } — objects only; after SEP-2106 it is plain unknown, so arrays and primitives are wire-legal too.

JSON Schema 2020-12 — The 2020-12 draft of the JSON Schema spec — the most recent stable version. Adds composition (oneOf / anyOf / allOf / not), conditionals (if / then / else), references ($ref / $defs), and tighter $dynamicRef semantics over the older draft-07 vocabulary MCP previously implied.

oneOf / anyOf / allOf — JSON Schema composition keywords. oneOf = match exactly one of N subschemas; anyOf = match at least one; allOf = match every subschema (intersection).

if / then / else — JSON Schema conditional keywords. If a value matches the if subschema, it must also match then; otherwise it must match else. Lets a single schema express "if roundTrip is true, return_date is required."

$ref / $defs — JSON Schema reference keywords. $defs declares reusable named subschemas; $ref points at one of them by JSON Pointer. Lets a long schema avoid copy-pasting the same address or money sub-shape three times.

The news. On May 18, 2026, SEP-2106 merged into the MCP specification. The change widens the schema vocabulary that tools may use to describe their input and output: inputSchema now allows the full JSON Schema 2020-12 keyword set inside its required type: "object" root, outputSchema drops the object-root constraint entirely and accepts any 2020-12 schema, and structuredContent is retyped from object-only to plain unknown. Loosening on paper — but the SEP is explicit that compatibility is asymmetric: a newer server emitting a non-object structuredContent or a composition-rich schema may be rejected by an older client that hasn't been updated, so the SEP recommends servers also emit a serialized TextContent fallback for non-object results during the transition.

Picture a job application that, until last week, only let you fill in plain text fields and checkboxes — name, address, "married?" yes/no. If the form needed something conditional ("if married, also provide spouse name") or alternative ("attach exactly ONE of passport, driver's license, or state ID"), the only way to express it was a paragraph of free-text instructions at the bottom of the page. SEP-2106 hands the form designer a richer template language: now the conditional, the alternatives, and the cross-references to other subforms are spelled out on the form itself, in a way the form's automated validator can actually check before the application gets routed.

The technical reason mirrors the metaphor. Before SEP-2106, the MCP wire spec implied a narrow JSON Schema subset — basically the keywords a 2014-era schema validator would understand: type, properties, required, items, enum, additionalProperties. If a tool needed to express "either a one-way booking (no return date) or a round-trip booking (return date required)," the schema author had two bad options: split it into two separate tools (now the model has to pick), or leave it as one tool with a permissive schema and a paragraph of natural-language instructions in description. The first option inflates the agent's tool registry; the second relies on the model honoring prose constraints that the runtime can't enforce.

Three surfaces, three changes

SEP-2106 touches three places on the wire, with slightly different shapes of change.

Surface	Before SEP-2106	After SEP-2106
`inputSchema` root	must be `type: "object"` (SEP-2106 commit)	must be `type: "object"` (unchanged)
`inputSchema` keywords inside the object root	restricted vocabulary the spec named — `type` / `properties` / `required` (SDKs typically also accepted `items`, `enum`, `additionalProperties`)	full JSON Schema 2020-12 — adds `oneOf` / `anyOf` / `allOf` / `not`, `if` / `then` / `else`, `$ref` / `$defs`, and the rest of the 2020-12 keyword set (SEP-2106 commit)
`outputSchema`	basic, object-rooted (mirrored `inputSchema`) (SEP-2106 commit)	fully flexible — any 2020-12 schema, including array roots, primitive roots, and composition (SEP-2106 commit)
`structuredContent` TypeScript type	`{ [key: string]: unknown }` — object only (SEP-2106 commit)	`unknown` — array, primitive, union, object all wire-legal (SEP-2106 commit)

The root constraint on inputSchema is preserved because every tool call still ships a JSON-RPC arguments object — the call is arguments: { ... }, not arguments: 7. What changed is everything inside that object, plus the symmetric story for what a tool can return.

A worked example

Picture a book_flight tool. Before SEP-2106, its inputSchema could declare four fields — from, to, departure, optional return — using the restricted vocabulary the spec named (type, properties, required). To express "round-trip flights require return, one-way flights forbid it," the author had three options: split into two tools (book_one_way, book_round_trip), leave a permissive schema and write a paragraph of description prose, or both. After SEP-2106, the same tool fits in one schema using composition:

{
  "type": "object",
  "properties": {
    "from": {...}, "to": {...},
    "departure": {...}, "return": {...},
    "roundTrip": { "type": "boolean" }
  },
  "required": ["from", "to", "departure", "roundTrip"],
  "oneOf": [
    {
      "properties": { "roundTrip": { "const": true } },
      "required": ["return"]
    },
    {
      "properties": { "roundTrip": { "const": false } },
      "not": { "required": ["return"] }
    }
  ]
}

The new schema reaches for oneOf, two branch subschemas with their own properties and required, a not, and two const guards — every one of those keywords lived in the JSON Schema 2020-12 standard already, but none were in the wire vocabulary MCP would accept before this SEP. The runtime can now reject a malformed call before it ever reaches the tool, instead of relying on the LLM to read and honor a paragraph of English in the description field.

Why this lands now

Two pressures converged. First, tool authors kept hitting the prose-vs-schema boundary: every nontrivial real-world tool grew a description paragraph explaining what its schema couldn't say, and that paragraph then needed to be re-explained to every model that called the tool. Second, the structured tool I/O step of the agent stack — where output validation lives — assumed an object-rooted structuredContent shape that forced tools returning a list (list_files) or a scalar (count_rows) to wrap their result in { "value": ... }. Both pressures land at the schema vocabulary, so SEP-2106 widens both at once.

The rollout story is more nuanced than "strictly loosening." Existing tools that already used only the previously-allowed keywords keep working unchanged, and the wire protocol stays backward-compatible at the schema vocabulary level — composition keywords like oneOf are legal JSON either way, so an older client that doesn't validate them will simply skip the extra checks (the schema still parses, just with weaker validation). The friction is asymmetric: a newer server emitting a non-object structuredContent or a primitive-rooted outputSchema may be rejected by an older client whose type checks still expect an object, which is why the SEP recommends servers also emit a serialized TextContent fallback for non-object results during the transition. SDK consumers also see one TypeScript source break — the narrower { [k]: unknown } type loses to plain unknown, and any code that depended on the narrower type needs to widen its own annotations to match.

Goes deeper in: AI Agents → Tool Use → Structured tool I/O and AI Agents → Tool Use → MCP

FAQ

What changed in MCP SEP-2106 in one sentence?

SEP-2106 lets MCP tool authors describe their inputs and outputs with the full JSON Schema 2020-12 keyword set — composition (oneOf / anyOf / allOf / not), conditionals (if / then / else), and references ($ref / $defs) — and widens structuredContent from an object-only TypeScript type to plain unknown, while keeping inputSchema's root type: "object" constraint unchanged.

Why does richer tool-schema vocabulary matter for agents?

The wire vocabulary is the only contract the runtime can validate before traffic reaches the tool. Anything that lives in the tool's free-form description prose has to be re-explained to every model that calls the tool, and the runtime can't reject a malformed call until the tool itself errors out. Pushing rules like "if roundTrip is true then return is required" into the schema means the SDK can reject the call before invocation and the model gets a structured error it can react to, instead of a tool-side stack trace.

Does SEP-2106 break existing MCP tools?

Existing tool definitions remain valid because the change only adds allowed keywords and widens types — nothing is removed. Compatibility is asymmetric, though: a newer server emitting a non-object structuredContent or a primitive-rooted outputSchema may be rejected by an older client whose type checks still expect an object. The SEP recommends servers also emit a serialized TextContent fallback for non-object results during the transition. There is also one source-level TypeScript break — consumers whose generic types narrowed structuredContent from unknown to { [key: string]: unknown } see a type error when they upgrade SDK versions, fixed by widening the consumer's type to match.

Originally posted on Learn AI Visually.

MarginGate: Margin-Gated Verification for Batch-Invariant Decoding

pueding — Sun, 07 Jun 2026 11:16:59 +0000

What: The MarginGate paper (arXiv) targets a subtle serving bug with margin-gated verification for batch-invariant decoding: temperature-0 BF16 decoding is treated as reproducible, yet the same prompt can emit different tokens decoded alone versus inside a larger batch.

Why: Reproducibility is load-bearing for debugging, evals, caching, and audits — yet in BF16 greedy serving, the batch a request lands in can silently change which token it emits from one run to the next.

vs prior: Always-on FP32 verification also restores determinism, but MarginGate re-checks only the sparse low-margin steps to reach it at roughly 2× less verification overhead in the paper.

Think of it as

An airport security line with a fast lane and a secondary-screening booth.

                      DECODE STEP
                          │
                 how wide is the margin?
                          │
           ┌──────────────┴──────────────┐
           │                             │
    ┌──────▼───────┐             ┌───────▼──────┐
    │  clean scan  │             │   near-tie   │
    │ wide margin  │             │  tiny margin │
    └──────┬───────┘             └───────┬──────┘
           │                             │
     FAST LANE (BF16)            SECONDARY (FP32)
     wave through                re-check the step
           │                             │
           ▼                             ▼
    ✓ same token, every         ✓ flip caught; K/V
      batch (no jitter)           column repaired

decode step = a traveler reaching the security checkpoint
logit margin = how clearly their boarding pass scans
high-margin step = a clean scan → waved through the fast lane (BF16)
low-margin step = a borderline scan → pulled into secondary screening (FP32)
K/V cache column repair = fixing the one mis-tagged bag before boarding

Quick glossary

BF16 (bfloat16) — A 16-bit floating-point format used for fast inference. It keeps FP32's exponent range but drops mantissa bits, so rounding errors are larger — enough that the order of a sum can change the result.

FP32 — 32-bit floating point — slower but far more precise. MarginGate uses it as the trusted reference to re-check only the steps that might be wrong.

logit margin — The gap between the top-1 and top-2 token scores at a decode step. A large margin means the winner is unambiguous; a tiny margin means a small numerical nudge can flip it.

greedy decoding (temperature 0) — Always emit the single highest-scoring token. People assume this is deterministic — the catch is that "highest-scoring" can change when the arithmetic changes.

floating-point reduction order — Summing numbers in a different order gives slightly different results in finite precision (addition isn't perfectly associative). GPU kernels pick their reduction order based on batch size — so the logits shift.

batch-invariance — The property MarginGate restores: a request produces the same tokens no matter how many other requests share its batch.

K/V cache — The cached keys and values from earlier tokens. When a step is repaired, MarginGate swaps the offending column of this cache so the rest of the sequence stays consistent. See the KV Cache module.

continuous batching — A serving technique where requests join and leave the running batch every step — which is exactly why a request's batch size (and its results) can vary run to run. See Batching.

The news. On May 28, 2026, a paper introduced MarginGate (arXiv 2605.30218), starting from an uncomfortable fact: temperature-0, greedy BF16 decoding is usually assumed to be reproducible, yet the same request can return different tokens depending on how many other requests happen to share its batch. MarginGate measures that batch-induced token flips are rare, then verifies only the steps at risk. Read the paper →

Picture an airport security line. Almost every traveler has a boarding pass that scans cleanly, so the agent waves them straight through the fast lane — that's a decode step with a wide logit margin, where the top token wins by a mile and no amount of numerical jitter would change it. The trouble is the occasional borderline pass: a near-tie between the top two tokens. For those travelers, a tiny nudge decides which way they go — and at temperature 0, that nudge can come from something as invisible as the batch they were standing in.

Why would the batch matter? Because the GPU sums each token's scores in a reduction order that depends on batch size, and in BF16 addition isn't perfectly associative — re-order the sum and the last bit can change. For a confident step that is harmless. For a near-tie it can flip the winner, so the very same prompt emits one token when decoded alone and another when it rides inside a larger batch. The root cause lives one level down, in how BF16 trades mantissa bits for speed versus FP32.

MarginGate's move is to gate on the margin. High-margin steps keep the cheap BF16 fast lane untouched. Only the sparse low-margin steps are sent to secondary screening — a re-computation in FP32, the same verify-then-correct shape that speculative decoding uses. If the trusted FP32 result disagrees with what BF16 produced, MarginGate repairs the step by swapping the offending column of the K/V cache so the rest of the sequence stays consistent. The expensive check fires on a handful of travelers, not the whole terminal.

How much does that save? Take a 1,000-token completion (illustrative). MarginGate flags the low-margin steps — about 18%, or ~180 steps — for an FP32 re-check, while the other ~820 keep the fast path. Of those 180, only a few are genuine flips: the paper measures flip rates of 0.3–1.3% of all steps (just 0.48% for Llama-3.1-8B on MATH500), so on the order of 3–13 tokens would actually have changed. In the paper's tested settings, MarginGate catches and repairs each one. Always-on verification would instead re-run all 1,000 steps in FP32 for the identical result — which is why margin-gating reports ~2× lower overhead (2.23× and 1.99× in the paper) while still restoring 100% sequence-level determinism on the models the paper tested (Llama-3.1-8B and Qwen2.5-14B).

Strategy	Steps re-checked	Determinism	Relative overhead
Trust BF16 (no verify)	none	✗ batch-dependent	1× (baseline)
Always-on FP32 verify	every step	✓ 100%	~2× the gate, varies by model (paper)
MarginGate (margin-gated)	~15–18% (paper)	✓ 100%	~2× lower than always-on (2.23× / 1.99×, paper)

The deeper lesson is that temperature 0 was never a determinism guarantee — it only fixes the sampling rule, not the arithmetic underneath it. MarginGate is cheap precisely because the failure is rare and predictable from the margin: you don't have to distrust every token, just the few that are genuinely on the fence.

Goes deeper in: LLM Internals → Batching → Continuous Batching, and LLM Serving → Serving Metrics & SLOs.

FAQ

What is batch-invariant decoding?

Batch-invariant decoding means a request produces the exact same tokens regardless of how many other requests share its GPU batch. It is the property most people assume temperature-0 greedy decoding already has — and MarginGate is a method for restoring it cheaply when it has quietly broken.

Why does temperature-0 BF16 inference give different tokens in a batch?

Because the GPU sums each step's scores in a reduction order that depends on batch size, and BF16 addition isn't perfectly associative, the logits shift by a tiny amount. On a near-tie between the top two tokens (a low logit margin), that tiny shift can flip which token wins, so the same prompt can emit a different token alone versus inside a larger batch. The paper measures these flips at roughly 0.3–1.3% of steps on the models it tested.

How is MarginGate different from always-on FP32 verification?

Always-on verification re-checks every decode step in FP32; it restores determinism but carries roughly 2× the verification overhead MarginGate does in the paper. MarginGate verifies only the sparse low-margin steps — about 15–18% in the paper — and repairs a true flip by swapping the offending K/V cache column, reaching the same determinism the paper reports (100% sequence-level on Llama-3.1-8B and Qwen2.5-14B).

Originally posted on Learn AI Visually.

MCP 2026-07-28 RC: Stateless Transport

pueding — Sat, 06 Jun 2026 11:16:43 +0000

What: The MCP 2026-07-28 release candidate reworks transport so the tools/call request itself carries every field a server needs to handle it — protocol version, capabilities, auth context, routing keys. The headline framing is stateless transport: any server in a fleet can serve any request, with no per-session pin to a specific instance.

Why: The previous design forced sticky routing: a session was bound to a single server for its lifetime, so load balancers had to either pin connections by session ID or replicate session state out-of-band. Horizontal scaling, blue/green deploys, and crash-recovery all suffered. The 2026-07-28 RC is the headline change of the next stable MCP spec — and it touches every harness that talks to MCP.

vs prior: Earlier MCP transports treated the first request as a handshake that established server-local state; subsequent requests had to land on the same instance. The new design drops the in-process session: each request is self-contained, and when long-lived cross-request state is genuinely needed (subscriptions, sampling sessions, auth tokens) it lives in a shared store any server can read — not in one server's memory.

Think of it as

A self-addressed envelope at a post office with many windows.

                    tools/call  (one letter)
                              │
                ┌─────────────┴─────────────┐
                │                           │
        ┌───────▼───────┐           ┌───────▼───────┐
        │ sticky clerk  │           │ self-addressed│
        │ (one window)  │           │   envelope    │
        └───────┬───────┘           └───────┬───────┘
                │                           │
       state in HER drawer          address + tracking
       notebook, hers alone         ID on the envelope
                │                           │
                ▼                           ▼
       ✗ wait at her window         ✓ any open window
         again; if she's              serves it — open
         out, trail is lost           ten more, all equal

tools/call = handing a letter to a clerk
sticky routing = a clerk who only remembers your shipment from a notebook on her desk — come back to HER for status
self-addressed request = a letter with the destination, sender, and tracking ID printed on the envelope — any window reads it
shared session store (when needed) = the post office's central tracking database — any clerk queries it
horizontal scaling = open ten more windows in the same office; any one serves you

Quick glossary

MCP — The Model Context Protocol — an open protocol for connecting LLM hosts to external tool servers. The host runs the model and the agent's tool loop; servers expose tools, resources, and prompts over JSON-RPC.

SEP — Specification Enhancement Proposal — MCP's RFC-style change document. The 2026-07-28 RC bundles twenty-two scoped SEPs covering the transport rework, the new Extensions framework, MCP Apps, Tasks, and authorization fixes.

Sticky routing — A load-balancing pattern where a session ID is pinned to a single backend instance for its lifetime. The load balancer hashes the session ID and always routes to the same server. Works fine until that one server is overloaded, restarted, or replaced.

Self-contained request — A request shape where every field the server needs to handle it — protocol version, declared client capabilities, routing keys, auth context — travels with the request itself. The server does not assume any prior state from earlier messages on the same socket.

Shared session store — An out-of-process store (Redis-equivalent, a database, an object store) that any server in the fleet can read and write. Used for the small subset of MCP interactions that genuinely need cross-request state — long-lived subscriptions, sampling sessions, OAuth tokens. The transport itself is still stateless; the store is an implementation pattern for state that has to survive across requests.

Tasks extension (SEP-2663) — The async-handle model for long-running tools: a server returns a Task handle the client drives with tasks/get, tasks/update, tasks/cancel. It composes naturally with stateless transport because the task handle is the only cross-request key the client needs.

The news. On May 22, 2026, the MCP project landed PR #2750 — the blog announcement for the 2026-07-28 specification release candidate. The post leads with the stateless transport rework as the headline change, with a before/after HTTP example showing a self-contained tools/call request. Extensions, MCP Apps, and Tasks follow as the new capability story; the authorization changes are summarized by the failure modes they fix rather than enumerated SEP-by-SEP. All twenty-two scoped SEPs are linked from the announcement.

Picture the post office with many windows. The slow path is the sticky clerk: you hand your letter to clerk #3, and clerk #3 jots the details in a notebook only she keeps in her drawer. If you come back to check on your shipment, you have to wait at her window — none of the other clerks can tell you anything. If clerk #3 is busy, or goes on break, or quits, the trail of your shipment goes with her. The line at her window grows; the other windows are quiet. That is exactly what sticky-routed MCP looks like today. The agent's tool-use loop opens a session, the load balancer pins that session to one server, and every follow-up call has to land on that same server. One server gets the traffic; the others sit idle.

The fast path is the self-addressed envelope. You write the destination, the sender, and a tracking ID on the front of every letter, and the post office stops needing any one clerk to remember anything about your shipment. Any open window will do. That is the 2026-07-28 framing: each tools/call carries the protocol version it expects, the client capabilities it declared, any routing keys the server fleet needs, and the auth context — all in the request itself. The server reads the envelope and acts. No drawer notebook. No "come back to me." A second request half a second later can land on a different server entirely and produce identical behavior.

There is a real subtlety worth saying out loud. A few MCP interactions genuinely do need cross-request memory — long-lived subscriptions, sampling sessions, OAuth tokens that have to outlive a single call. The new design does not pretend those don't exist. It externalizes them: the central tracking database the metaphor mentions is a shared store (a Redis-equivalent, a database, an object store) that any server queries when it needs to hydrate that bit of cross-request state. The transport is still stateless — the request itself is self-contained — and the implementation pattern of a shared store is what makes the small slice of stateful behavior work across a fleet. Mixing those two ideas up is easy and worth keeping straight: the protocol's change is at the transport layer; the shared store is one way servers can choose to persist what little state has to outlive a request.

The capacity argument writes itself. Consider 300 concurrent agent sessions, each holding open MCP traffic at ~2 calls per second, hitting a fleet of 3 servers. Sticky routing assigns each session to one server at session open. Distribution is rarely uniform — three or four "power user" sessions can pin one server's load near saturation while the others sit at 10-20%. Numerically: a typical sticky-imbalance run might leave S1 at ~92% utilization while S2 and S3 sit at ~8% and ~41% (illustrative). Under stateless transport with the same workload, the load balancer can spray every call independently. The same 600 calls/sec land on three servers at ~49% each (illustrative) — a ~1.9× improvement in usable fleet headroom before any vertical scaling.

Where the rework earns its keep

Sticky routing's failure modes are well-known in the agent harness world: one hot server, blue/green deploys that have to drain sessions for minutes, crash recovery that can't transparently re-route. The 2026-07-28 RC closes all three at the transport level. Self-contained requests do not pin to anything, so a deploy that rolls a server out of rotation finishes in seconds — pending requests just hit the next server. A server that crashes drops its in-flight requests, and the client retries against the fleet — the next call lands somewhere else and proceeds. The only state that needs to survive the crash is whatever the workload chose to put in the shared store, which is the small minority of interactions.

The shape of what the RC actually changes is concrete. The table below contrasts the legacy and new transport.

Aspect	Sticky-routed transport (legacy)	Stateless transport (2026-07-28 RC)
Session lifetime	Bound to one server for the session's life	No per-session server binding
Routing key	Session ID hashed to a specific instance	None — any instance, any request
First request	Handshake that creates server-local state	Self-contained, no implicit setup
Cross-request state	In server memory	In a shared store, only when needed (subscriptions, sampling, auth)
Horizontal scale-out	Awkward — uneven load by session hash	Native — load balancer sprays calls
Server restart	Drops the session; client must rebuild	Drops in-flight; retry hits any other server

A related design point is worth knowing. The Tasks extension (SEP-2663) ships a complementary idea one layer up: it gives the client a long-lived taskId it can poll across reconnects. SEP-2663 needed the transport rework to be fully useful — a taskId polled across reconnects only works if the next tasks/get doesn't have to land on the same server that issued the handle. Stateless transport is what makes that work: the taskId is the only cross-request key the client carries, the server fleet hydrates the task's state from the shared store, and the polling call goes to whichever server is least busy.

The boundary of what the RC changes is the transport itself, not the protocol semantics. Tools still return tool results; resources still return resource contents; the wire format of a method call is the same JSON-RPC envelope. What changes is what a server is allowed to assume: nothing about prior calls on the same connection. That single discipline is enough to make every harness operator's life easier and to make the parallel-tool-call patterns the Cost & Latency module recommends actually achievable in a fleet.

FAQ

What does stateless transport mean in the MCP 2026-07-28 RC?

It means the tools/call request itself carries every field a server needs to handle it — protocol version, declared client capabilities, routing keys, auth context. The server is not allowed to assume any state from prior calls on the same connection. A consequence is that any server in a fleet can serve any request, so no sticky session binding is needed at the load balancer.

What replaces sticky routing for state that genuinely has to live across requests?

A shared store. The small subset of MCP interactions that need cross-request memory — long-lived subscriptions, sampling sessions, OAuth tokens — moves out of any one server's process and into a Redis-equivalent (or database, or object store) the entire fleet reads. The transport itself is still stateless; the shared store is an implementation pattern for the slice of state that must survive across requests.

How does the transport rework relate to the Tasks extension (SEP-2663)?

They compose. SEP-2663 lets a server return a long-lived taskId the client polls later. Stateless transport is what makes that poll robust across a fleet: the next tasks/get does not need to land on the same server that issued the handle. Together they let an agent harness survive server restarts, blue/green deploys, and load-balancer reshuffles without any session affinity.

What needs to change in existing MCP server code to support stateless transport?

Concretely: stop reading state from the connection. Any field the server used to learn once at session-establish and remember for the lifetime of the connection — declared client capabilities, protocol version, auth identity, routing tenant — must now be read from each tools/call request instead. Servers that already drove every decision off the incoming request payload need minimal changes. Servers that built up per-connection caches (negotiated capabilities, OAuth introspection results, tenant routing decisions) need to externalize those caches into a shared store the whole fleet reads, or push them to the client to re-send. Most production MCP servers will land in the middle: a few small migrations rather than a rewrite.

How does stateless transport affect MCP authentication and authorization?

Auth context becomes a per-request field rather than a per-session attribute. The 2026-07-28 RC expects every tools/call to carry whatever proof the server needs — a bearer token, a signed capability, a tenant identifier — so any server in the fleet can verify the call without consulting prior connection state. The net effect on a production stack is that a load-balancer reshuffle, a server restart, or a blue/green deploy mid-flight no longer drops the agent's authorization, because no server held it in process memory in the first place. Token introspection caches still live somewhere, but in a shared store the entire fleet shares (Redis-equivalent), not in any single server's per-connection state.

Originally posted on Learn AI Visually.

Token Budgets Paper: Affine-Typed Budget Ownership

pueding — Fri, 05 Jun 2026 11:16:05 +0000

What: The Token Budgets paper catalogs 63 real LLM-agent cost-overrun incidents and ships a Rust crate that models a token/cost budget as an affine-typed (use-at-most-once) resource the compiler tracks.

Why: Cost is a production failure mode, and the paper finds it's multi-agent delegation — not single agents — that drives the overruns: fan out work to parallel sub-agents and each one quietly reserves budget against a cap nobody is decrementing.

vs prior: Versus a runtime budget guard — an assert that fires at spend time, after the tokens are already committed — affine typing makes an overrun a compile-time error, so the unsafe code path can't ship in the first place.

Think of it as

One prepaid gift card a group splits at dinner.

                  ONE $1,000 GIFT CARD
                          │
          ┌───────────────┴───────────────┐
          │                               │
  ┌───────▼───────┐               ┌───────▼───────┐
  │  PHOTOCOPY IT │               │   SPLIT IT    │
  │ (static copy) │               │ (affine move) │
  └───────┬───────┘               └───────┬───────┘
          │                               │
 4 copies x $350 each         $300+$220+$260+$220
 nobody debits the card       money moves out, no copy
          │                               │
          ▼                               ▼
   ✗ bill = $1,400               ✓ total = $1,000
     over a $1,000 cap             bounded to the cap

token budget = the card's balance ($1,000)
sub-agent = a friend who wants to spend
static reservation = everyone photocopies the card and assumes the full balance
overshoot = four copies each spend $350 — the bill hits $1,400 on a $1,000 card
affine ownership = split the card into prepaid sub-cards — money moves out, can't be photocopied

Quick glossary

Token / cost budget — A hard cap on how many tokens (and therefore dollars) one agent task is allowed to spend. Where those tokens go is the first thing a production agent has to account for.

Affine type — A type that may be used at most once. The compiler tracks the value's ownership, so you can move or split it but never copy it — exactly the property a budget needs.

Delegation fan-out — When an orchestrator hands a task to several sub-agents running in parallel. Each child needs some budget, and the question is who keeps the shared total honest.

Static vs adaptive reservation — Static reservation grabs a fixed slice up front and over-provisions 4–6×; adaptive reservation re-estimates per call and over-provisions 2.11× — fewer wasted tokens, but still a runtime accounting trick.

Compile-time vs runtime check — A runtime check tests the budget while the agent runs (too late to un-spend); a compile-time check rejects the unsafe program before it ever runs. Affine typing moves the cap into the second category.

Cohen's kappa — An inter-rater agreement score (1.0 = perfect). The paper's 8-category failure taxonomy reaches 0.837, i.e. two independent reviewers classified the incidents almost identically.

The news. On June 2, 2026, the Token Budgets paper landed: an empirical catalog of 63 production cost-overrun incidents in LLM-agent systems, pulled from a review of 21 orchestration frameworks spanning 2023–2026 and clustered into an 8-category failure taxonomy (inter-rater Cohen's kappa 0.837). As a mitigation, the authors ship a 1,180-line Rust crate that uses affine-type ownership to turn budget violations into compile-time errors. In controlled tests, single-agent runs never overshot (0/30) while multi-agent asyncio delegation overshot every time (30/30); the mitigated runs then logged 0 cap violations across 160 live-API tests. Read the paper →

Picture the group dinner. There's one prepaid gift card with $1,000 on it, and four friends who all want to order. The cheap, lazy move is for everyone to photocopy the card and assume they each have the full balance — four copies, four people each cheerfully spending $350, and a $1,400 bill arrives against a card that only ever held $1,000. The card was never debited as people spent, so nothing stopped the overshoot until the bill came. Affine-typed budget ownership is the opposite rule: there is exactly one card, and the only legal operation is to split it into prepaid sub-cards — the money physically moves out of the original, and a photocopy simply isn't allowed.

In an agent system the "photocopy" bug is a delegation fan-out: an orchestrator spawns parallel sub-agents, and each one reserves a chunk of the token budget against a cap that no single owner is decrementing. The paper's headline number is that this pattern overshot 30 out of 30 runs, while a single agent — which spends against one running total — overshot 0 of 30. The fix is to make the budget an affine value: the Rust compiler tracks it as use-at-most-once, so a code path where two sub-agents could both hold the same budget fails to type-check. The cap is enforced by construction rather than by an assert that fires after the tokens are already gone — the same shift from runtime to compile-time that separates a retry loop that quietly re-bills you from one that can't.

Where the budget actually goes

A back-of-envelope walk-through (illustrative cap and slice sizes; the overshoot and over-reservation counts are the paper's). Say the shared cap is 1,000 tokens and the orchestrator fans out to four sub-agents. Under static reservation each child grabs a fixed 350, and because the reservations are effectively copies, the total claimed is 4 × 350 = 1,400 — a 400-token (40%) overshoot that nothing rejects until the spend lands. Make the budget affine and the same 1,000 is split into owned slices — say 300 + 220 + 260 + 220 = 1,000 — where the fourth claim can only take what the first three left behind. The sum is bounded to the cap by construction, which is the property the paper's Rust crate enforces: across 160 live-API tests it logged 0 cap violations, where unbounded multi-agent delegation had overshot all 30 runs. Static reservation's habit of grabbing 4–6× the budget it needs (adaptive trims that to 2.11×) is the same waste, viewed from the other side.

Approach	When the cap is checked	Multi-agent overshoot	Over-reservation
Runtime budget guard	at spend time — after tokens commit	possible (the default failure)	—
Static reservation	up front, no shared cap	30/30 runs (Token Budgets paper)	~4–6× (paper)
Adaptive reservation	re-estimated per call	not reported (paper)	~2.11× (paper)
Affine-typed ownership	compile time — won't type-check	0 violations / 160 tests (paper)	bounded to the cap

The catch is that this only buys you safety where you can express ownership in the type system — a Rust crate gets it for free, a Python orchestrator built on asyncio.gather does not, which is exactly where the paper's 30/30 overshoots came from. But the lesson generalizes past the language: in a multi-agent team the budget is a shared resource, and who is allowed to hold it, and whether they can copy it, is a design decision — not something to discover when the bill arrives.

Goes deeper in: Agent Engineering → Cost & Latency Engineering → Where the tokens go

Related explainers

StreamMA — Streaming inter-agent reasoning — a different multi-agent cost: wall-clock latency from serial handoffs, cut by pipelining rather than by bounding tokens
Maestro — RL orchestrator over frozen experts — the orchestrator-over-sub-agents topology where this fan-out budget problem lives
EFC — feedback-quality scaling law — what actually predicts agent-harness success, the other half of "spend the budget well"

FAQ

What is affine-typed budget ownership?

It models an agent's token or cost budget as an affine-typed value — one the compiler allows you to use at most once. You can split the budget into smaller owned slices or move it to a sub-agent, but you can't copy it, so two parts of the system can never both spend against the same cap. The Token Budgets paper implements this in a Rust crate and reports 0 cap violations across 160 live-API tests.

Why do multi-agent systems overshoot their token budget?

Because delegation fans the work out to parallel sub-agents that each reserve budget against a cap no single owner is decrementing. The reservations behave like copies, so their sum can exceed the real limit. In the paper's controlled tests, multi-agent asyncio delegation overshot 30 of 30 runs while a single agent — spending against one running total — overshot 0 of 30.

How is a compile-time budget check different from a runtime guard?

A runtime guard (an assert or limiter) checks the budget while the agent runs, which is too late to un-spend tokens already committed. A compile-time check rejects the unsafe program before it runs: with affine typing, a code path where two sub-agents could hold the same budget simply fails to type-check, so the cap is enforced by construction rather than by hoping the guard fires in time.

Originally posted on Learn AI Visually.

Microsoft MAI-Code-1-Flash: Adaptive Solution-Length Control

pueding — Thu, 04 Jun 2026 11:16:35 +0000

What: Microsoft's first in-house coding model, MAI-Code-1-Flash (launched at Build 2026 alongside the MAI-Thinking-1 reasoner), ships adaptive solution-length control — the model decides how many reasoning tokens to spend based on how hard the task is.

Why: Reasoning tokens are the dominant cost of a thinking model: every token is a decode step you pay for in latency and dollars. Spending the same long chain on a one-line fix as on a cross-file refactor wastes most of that budget — adaptive length spends it only where it buys accuracy.

vs prior: A fixed-budget reasoner thinks to roughly the same length on every prompt; adaptive control stops at the point the answer is reached, so Microsoft reports the model hitting its scores with up to 60% fewer tokens than a flat budget would burn.

Think of it as

A test-taker who budgets minutes by how hard each question looks.

              SAME EXAM: easy · medium · hard
                            │
            FIXED BUDGET    │    ADAPTIVE CONTROL
            same slot/Q     │    slot sized to Q
                            │
      easy   ██████████ 10m │ easy   █ 0.5m
      medium ██████████ 10m │ medium █████ 6m
      hard   ██████████ 10m │ hard   ██████████ 10m
                            │
       ✗ burns minutes on   │  ✓ same score,
         the easy ones      │    far fewer minutes

task = one exam question
reasoning tokens = minutes spent working it out
fixed budget = giving every question the same long time slot
adaptive control = sizing the minutes to each question's difficulty
answer reached = the moment you have it, so you move on

Quick glossary

Reasoning tokens — The intermediate "thinking" tokens a model generates one at a time before its final answer (the chain-of-thought). More reasoning tokens = more decode cost.

Solution length — How long that reasoning chain runs before the model commits to an answer. Adaptive solution-length control lets the model choose this length per task instead of using a fixed cap.

Test-time compute — Compute spent at inference (not training) — chiefly by generating more reasoning tokens. Spending more usually helps hard problems and is wasted on easy ones.

SWE-Bench Pro / Verified — Benchmarks of real GitHub issues a model must resolve with working code. Microsoft reports MAI-Code-1-Flash at 51.2% on SWE-Bench Pro vs Claude Haiku 4.5's 35.2%, using up to 60% fewer tokens on SWE-Bench Verified.

Sparse MoE — Mixture-of-Experts: each token is routed through a small subset of "expert" sub-networks. MAI-Thinking-1, the reasoner alongside the coding model, is a sparse MoE with 35B active parameters and a 256K context.

Underthinking — Stopping the chain too early and committing to a wrong answer — the failure mode a fixed-minimum budget risks, and the reason a good stop signal is the hard part of adaptive length.

The news. On June 2, 2026, at Build 2026, Microsoft introduced its first in-house frontier models — MAI-Thinking-1 (a 35B-active sparse MoE reasoner with a 256K context, which Microsoft says was trained from scratch on licensed data with no distillation from third-party models) and MAI-Code-1-Flash, a small, inference-efficient coding model built end-to-end by Microsoft and rolling out to GitHub Copilot users in VS Code. MAI-Code-1-Flash reportedly leads Claude Haiku 4.5 by 16 points on SWE-Bench Pro (51.2% vs 35.2%) while using up to 60% fewer tokens, which it credits to adaptive solution-length control. Read the announcement →

Picture the test-taker for a second. Two students sit the same exam. The first was told to spend exactly ten minutes per question — so she burns the full ten on "2 + 2," sits there second-guessing a settled answer, and runs short on the proof at the end. The second reads each question, sizes up the effort, and moves on the moment she's sure — thirty seconds on the arithmetic, the full ten on the proof. Same paper, same score, far less time. Adaptive solution-length control is the second student: the model spends its reasoning where difficulty actually demands it, instead of paying a flat tax on every task.

Under the hood, the "minutes" are reasoning tokens. A thinking model generates its chain-of-thought one token at a time before answering, and every one of those tokens is a decode step you pay for in latency and dollars. A fixed budget sets one length for all prompts; adaptive control instead decides how long to keep thinking and, crucially, when to stop. Microsoft hasn't disclosed the exact controller — whether the length is learned, predicted up front, or a learned stop signal mid-chain — so treat the mechanism as undisclosed; what's reported is the outcome: the same benchmark scores at a fraction of the tokens.

Where the tokens actually go

A back-of-envelope walk-through (illustrative numbers; the 60% figure is Microsoft's). Take three Copilot tasks: an easy one-line fix, a medium multi-step bug, and a hard cross-file refactor. A fixed budget of ~2,000 reasoning tokens spends all three the same way → ~6,000 tokens total, even though the easy fix had its answer after ~200. Adaptive control stops each chain at its answer — roughly ~200 + ~650 + ~1,650 ≈ ~2,500 tokens — for the same result. That's ~58% fewer tokens in this toy mix, right in line with the up to 60% fewer Microsoft reports. The hard task barely changes; the savings come almost entirely from not over-thinking the easy and medium ones.

Three ways to set the reasoning length

Strategy	Easy task	Hard task	Main risk
Fixed-max budget	thinks far past the answer	fits — has room	over-thinking: burns tokens it doesn't need
Fixed-min budget	fits — short is fine	cut off too early	underthinking: commits to wrong answers
Adaptive control	short chain	long chain	needs a reliable stop signal

The catch lives in that last cell. A fixed budget is dumb but safe; adaptive length is only as good as its sense of when it's done. Stop one token too early on a hard task and you get underthinking — a confident wrong answer that's worse than a slow right one. That's why the headline number is a coding model's: in software, a test or verifier can often tell the model whether it's actually done, giving the stop signal something concrete to lean on. The win is real and specific — fewer reasoning tokens for the same accuracy — and it rides entirely on getting that stop right.

Goes deeper in: AI Agents → Planning & Reflection → Reasoning budget

Related explainers

Compute Where It Counts — Per-token compute controller — the other axis of adaptive compute: how much work each token gets, vs how many tokens the chain runs
LongTraceRL — Rubric reward (process supervision) — how reasoning chains get trained, where a good stop signal would come from
Gemini 3.5 Flash — Agent-first model design — a related angle: building a model for the agent loop rather than retrofitting chat

FAQ

What is adaptive solution-length control?

It's a model's ability to scale the length of its reasoning chain to the difficulty of the task. Instead of a fixed cap on reasoning tokens for every prompt, the model spends a short chain on easy tasks and a long one only on hard tasks, stopping when it has reached an answer. Microsoft's MAI-Code-1-Flash uses it to hit its benchmark scores with up to 60% fewer tokens than a flat budget would use.

Why does it save so much without losing accuracy?

Because the savings come from tasks that were over-thought, not under-thought. On an easy fix, a long reasoning chain reaches the answer early and then generates tokens past it — those extra tokens cost latency and money but don't change the result. Trimming the chain to the point the answer was reached removes pure waste. Hard tasks, which genuinely need a long chain, are barely affected.

How is it different from a per-token compute controller?

They tune different dials. A per-token compute controller (as in the "Compute Where It Counts" paper) changes how much compute each individual token gets — attention sparsity, layer pruning, bit-width. Adaptive solution-length control changes how many reasoning tokens the chain runs in total. One sizes the work per token; the other sizes the number of tokens. They're complementary.

Originally posted on Learn AI Visually.

Harness-1: State-Externalizing Search Harness

pueding — Wed, 03 Jun 2026 11:16:02 +0000

What: The Harness-1 paper introduces a 20B RL-trained search agent that externalizes its working memory into a structured harness — candidate pools, evidence links, and verification records — instead of an ever-growing transcript.

Why: A deep search agent that replays its whole history every step runs the context window dry. Harness-1 makes context cost stay flat as the search deepens, which is the harness-as-state idea the agent-engineering world preaches, made concrete and RL-trained.

vs prior: Earlier search agents train over a growing transcript, so every candidate, observation, and verification lands back in context. Harness-1 trains over an external workspace and renders only a budget-bounded slice — the policy decides what to search and verify; the harness owns the memory.

Think of it as

A detective's case-board on the wall, briefed by index card.

                  THE GROWING CASE
                         │
              ┌──────────┴──────────┐
              │                     │
      ┌───────▼────────┐    ┌───────▼────────┐
      │  HARNESS-1     │    │ GROWING        │
      │  case-board    │    │ TRANSCRIPT     │
      │  on the wall   │    │ lug whole file │
      └───────┬────────┘    └───────┬────────┘
              │                     │
      carry one index card  haul the entire box
      into each interview    into every interview
              │                     │
              ▼                     ▼
      ✓ desk stays clear    ✗ desk overflows
        context stays flat     window overruns

case-board on the wall = the durable harness workspace (every lead, evidence link, verified fact)
index-card briefing = the budget-bounded slice rendered into the model's context each step
lugging the whole case file into every interview = replaying the entire growing transcript
running out of desk space = overflowing the context window as the search deepens

Quick glossary

Harness — The scaffolding around the model that owns tools, state, and exactly what gets shown to the model each step. The model is the brain; the harness is the desk, filing cabinet, and notepad.

Context window — The fixed token budget the model can read on any single step. Anything outside it is invisible to the model — and tokens are not free, so a full window is both a cost and a hard ceiling.

Growing transcript — The naïve agent-memory design: concatenate the full action-and-observation history and feed it back every step. It grows without bound, so a long search eventually overruns the context window.

State externalization — Keeping durable working memory outside the model's context — in the harness — so accumulated evidence does not spend context budget. The model reads a rendered view, not the raw store.

Budget-bounded rendering — Each step, the harness selects only a token-budgeted slice of the workspace to render into context, so context size is constant regardless of search depth.

Curated set — The agent's running shortlist of importance-tagged, verified evidence — distinct from the raw candidate pool. Harness-1's headline metric is curated recall: how much of the gold evidence lands in this set.

Curated recall — The fraction of the gold (correct) evidence that ends up in the curated set, averaged across 8 retrieval benchmarks. Harness-1 reports 0.730, +11.4 points over the next-best open search agent.

The news. On June 1, 2026, Harness-1 (arXiv:2606.02373) introduced a 20B-parameter search agent that separates semantic decision-making from state management. The policy decides what to search, inspect, curate, verify, and when to stop; a state-externalizing harness holds the working memory — candidate pools, importance-tagged curated sets, evidence links, verification records, and compressed observations — and renders only a budget-bounded slice into the model's context each step. Rather than training over an ever-growing transcript, the agent is trained with reinforcement learning over a structured external workspace. It reports 0.730 average curated recall across 8 retrieval benchmarks (web, finance, patents, multi-hop QA), +11.4 points over the next-strongest open search sub-agent. Read the paper →

Picture a detective working a long case. Every lead, photo, and verified alibi gets pinned to the case-board on the wall and connected with red string — the board is the durable record, and it only ever grows. When the detective walks into an interview, they don't wheel the entire case file into the room; they carry a single index-card briefing with just what this conversation needs. The board stays on the wall; only a briefing walks in. A rookie who instead lugs the whole growing file box into every interview eventually runs out of desk space — that is exactly what happens when a search agent replays its entire transcript into a finite context window.

That is the move Harness-1 makes concrete. The naïve design treats the agent's memory as a growing transcript: every observation, every candidate document, every verification step is concatenated and fed back to the model on the next step. It works for a few steps, then the transcript balloons and the search has to stop — not because the agent ran out of leads, but because it ran out of room. Harness-1 instead keeps that durable state in the harness — the case-board — and lets the policy decide where the agent's working state lives. Each step, the harness performs budget-bounded rendering: it selects a token-bounded slice of the workspace — the briefing — and shows only that to the model. The board can grow to hundreds of items while the briefing stays the same size, so context cost stays flat no matter how deep the search goes. Crucially, the agent is trained with reinforcement learning over this workspace, not over transcripts, so the policy learns the harness skills — curate, importance-tag, verify, compress, stop — as first-class actions.

Growing transcript vs state-externalizing harness

Design	What lives in context	Context cost as search deepens	Failure mode
Growing transcript	The full action + observation history, replayed every step	Grows with every step	Overflows the window; the search stalls on length, not leads
State-externalizing harness	A budget-bounded slice rendered from the workspace	~Flat, set by a render budget	A poorly-chosen slice can omit a needed item (mitigated by importance tags + curated recall)

The two rows describe the contrast Harness-1 draws between transcript-style memory and its externalized workspace; the "budget-bounded slice" claim is from the paper. Token figures in the hero animation are illustrative.

Walk the budget with some round numbers (illustrative). Say each search step adds about 2,000 tokens of fresh observations. Under the growing-transcript design, those tokens never leave: after 8 steps the model is reading roughly 16,000 tokens of history, after 20 steps about 40,000, and a genuinely deep multi-hop search marches straight past a typical working window. Under the state-externalizing harness, those 2,000-token observations land in the workspace, but the model is only ever shown a fixed ~6,000-token render — step 8 and step 20 cost the same 6,000 tokens in context. The accumulated evidence still exists; it just lives on the case-board instead of in the briefing. That is why Harness-1 can keep curating to 0.730 recall across deep benchmarks where a transcript agent would have run out of room — and it's the same lever the agent-engineering track frames as durable state the harness owns, rather than state smeared across a prompt.

It lands as a sharp companion to the recent push on how search agents act — GrepSeek learns a better action space (shell commands over a corpus), while Harness-1 learns a better state substrate (an externalized workspace). Same RL-trained-search-agent family, orthogonal levers. As the work frames it, the model should make the semantic calls and the harness should own the memory — a clean division that the standard fixes for an overflowing context have been circling, now learned end-to-end.

Goes deeper in: AI Agents → The Agent Loop & State → The Anatomy of a Harness

Related explainers

GrepSeek — training a shell-command search agent — the other lever: learning the search action space instead of the state substrate.
Is Grep All You Need? — grep vs vector retrieval — empirical evidence that harness design dominates the retrieval algorithm.
RecMem — subconscious + recurrence-triggered memory — another way to keep durable agent memory off the live context.

FAQ

What is Harness-1?

Harness-1 is a 20B-parameter, RL-trained search agent that separates the model's semantic decisions (what to search, inspect, curate, verify, and when to stop) from state management. A state-externalizing harness holds the durable working memory — candidate pools, importance-tagged curated sets, evidence links, verification records, and compressed observations — and renders only a budget-bounded slice into the model's context each step. It reports 0.730 average curated recall across 8 retrieval benchmarks, +11.4 points over the next-strongest open search sub-agent.

Why does externalizing state matter?

A search agent that replays its full transcript into context each step grows that context with every observation, so a deep search eventually overruns the context window and stops on length rather than on evidence. Externalizing state keeps the accumulated evidence in the harness and renders only a fixed-size slice, so context cost stays flat regardless of search depth — letting the agent keep curating across deep, multi-hop benchmarks.

How is this different from just a growing transcript?

A growing transcript concatenates the entire action-and-observation history and feeds it back every step, so its size scales with the number of steps. Harness-1 instead stores that history in a structured external workspace and trains the policy with reinforcement learning over that workspace — so the model learns to curate, verify, and compress as explicit actions, and the context the model reads is a budget-bounded rendering of the workspace rather than the raw, unbounded log.

Originally posted on Learn AI Visually.

GrepSeek Trains a Search Agent to Use Shell Commands: GRPO-Trained Shell-Command Search

pueding — Tue, 02 Jun 2026 11:17:41 +0000

What: GrepSeek (Salemi, Zamani et al.) is a recipe for training an agent to search a raw text corpus by writing shell commands — grep, pipes, and the like — instead of querying a pre-built vector index.

Why: Agentic search usually leans on an embedding model, a vector store, and an ANN index. GrepSeek shows you can instead learn a policy that searches the raw files directly, and it reports the strongest F1 / Exact Match across 7 open-domain QA benchmarks while staying index-free.

vs prior: The earlier "Is Grep All You Need?" study just wired an untrained grep tool into agents and measured it; GrepSeek instead trains the search behaviour — a two-stage Tutor/Planner distillation followed by GRPO — so the agent learns which commands to run rather than grepping by hand-written heuristics.

Think of it as

A rookie detective learning to search a case archive.

   CASE ARCHIVE: folders of files, no index to query
                         │
                         ▼
        ┌─────────────────────────────────┐
        │  ANSWER-AWARE TUTOR             │
        │  knows the answer; demonstrates │
        │  the drawer-pulls that crack it │
        └────────────────┬────────────────┘
                         │  rookie copies the moves
                         ▼
        ┌─────────────────────────────────┐
        │  ANSWER-BLIND PLANNER           │
        │  practises blind; keep only     │
        │  searches that solved the case  │
        └────────────────┬────────────────┘
                         │  GRPO: solved case = reward
                         ▼
   ✓ trained agent runs ONE targeted search,
     not a random rummage through the files

raw corpus = a detective's case archive — folders of files, no index
shell command = pulling a specific drawer or running Ctrl-F on a document
answer-aware Tutor = a mentor who already knows the answer and demonstrates the efficient search
answer-blind Planner = the rookie practising the moves without seeing the answer, keeping only searches that crack the case
GRPO reward = the case getting solved, which reinforces the search habits that worked

Quick glossary

Agentic search — A retrieval pattern where the LLM iteratively calls a search tool and decides what to look for next, instead of being handed chunks in one shot.

Vector index (ANN) — The default for "RAG": embed every chunk, embed the query, return the top-k nearest by an Approximate Nearest Neighbour index. GrepSeek skips this entirely.

GRPO — Group Relative Policy Optimization — an RL method that scores a group of sampled answers against each other instead of training a separate value model, then pushes the policy toward the above-average ones.

Tutor / Planner — GrepSeek's two trajectory generators: an answer-aware Tutor demonstrates effective shell-search sequences, and an answer-blind Planner mimics them under realistic uncertainty.

Verified trajectory — A recorded sequence of shell commands that is kept for training only if it actually reached the correct answer — the filter that stops the agent learning impressive-looking but useless searches.

Byte-exact parallel engine — A sharded execution engine that runs the agent's shell commands concurrently yet returns output identical to a sequential run — up to 7.6× faster with no change in results.

The news. On May 28, 2026, GrepSeek: Training Search Agents for Direct Corpus Interaction (arXiv:2605.29307, Salemi, Zeng, Diaz, Zamani et al.) trained LLM agents to interact with a text corpus through executable shell commands rather than a pre-built dense index. Training is two-stage: an answer-aware Tutor and answer-blind Planner generate verified search trajectories, then the policy is refined with GRPO. The paper reports the strongest token-level F1 and Exact Match across seven open-domain QA benchmarks, with a byte-exact parallel execution engine that speeds shell retrieval up to 7.6×. Read the paper →

Picture the rookie detective again. On day one she rummages through the case archive more or less at random — yanks open a drawer labelled "office," dumps a hundred folders on the desk, and can't say which line answers the question. That's an untrained agent firing a broad grep "office" at the corpus: lots of hits, almost all noise, wrong answer. Crucially, there is no card catalogue — no embeddings, no vector index to ask. The only way through the archive is to run a literal search and read what comes back.

GrepSeek's move is to coach the search itself. First a mentor who already knows each case's answer — the answer-aware Tutor — demonstrates the efficient sequence of drawer-pulls. Then the rookie, the answer-blind Planner, practises those moves without peeking at the answer, and the team keeps only the trajectories that actually cracked the case. That distilled set seeds a second stage of reinforcement learning: GRPO samples several command sequences per question, compares them against each other, and nudges the policy toward the ones that landed the right answer. Over training, the same agent stops grepping by habit and starts emitting a targeted pipeline — grep -i paris *.md | grep Q3 — that returns the handful of lines that matter.

Because the tool the agent calls is a literal shell, the win is two-sided. The retrieval is index-free, so there is no embedding pass or ANN store to build and keep fresh — the agent's tool is just a command line over the raw files. And the searches are learned end-to-end against the answer, so the policy adapts to this corpus and this question style rather than trusting a fixed similarity metric. This is the line worth drawing under the older agentic-retrieval results: where "Is Grep All You Need?" showed an untrained grep tool already competitive, GrepSeek shows what happens when you train the grep.

Where it sits among the options

Approach	Retrieval backend	Training	Adapts to the corpus?
Classic RAG	embeddings + ANN vector index	none (frozen retriever)	only via re-embedding
"Is Grep All You Need?" (explainer)	literal grep tool, untrained	none (hand-wired tool)	no — fixed heuristics
GrepSeek	shell commands, no index	Tutor/Planner distill → GRPO	yes — learned against the answer

Why the parallel engine matters for training

The byte-exact engine sounds like a systems footnote until you remember where RL spends its time. Each GRPO update needs many rollouts per question, and every rollout actually runs the agent's shell commands against the corpus. Say a single sequential grep sweep over the shards takes 760 ms (illustrative) and a training run does 100,000 rollouts: that is roughly 21 hours of pure retrieval before you count a single gradient step. The sharded-parallel engine runs those shards concurrently for a byte-exact identical result, collapsing 760 ms → ~100 ms — the same 100,000 rollouts now cost about 2.8 hours. The speedup is the real, reported 7.6×; the win compounds precisely because, in RL, you pay the retrieval cost on every rollout, not once.

Goes deeper in: AI Agents → Retrieval & RAG → RAG failure modes

Related explainers

Is Grep All You Need? — grep vs vector retrieval for agentic search — the untrained-grep study GrepSeek builds on.
VPO — vector reward vs GRPO — a closer look at the GRPO objective GrepSeek refines with.

FAQ

What is GrepSeek?

GrepSeek is a method for training an LLM agent to retrieve from a raw text corpus by writing executable shell commands — grep, pipes, and the like — instead of querying a pre-built vector index. It distills verified search trajectories from an answer-aware Tutor and answer-blind Planner, then refines the policy with GRPO.

Why does it matter?

It shows agentic search can be a learned skill rather than a fixed retrieval stack. By skipping the embedding model, vector store, and ANN index and learning shell-command search end-to-end against the answer, GrepSeek reports the strongest F1 and Exact Match across seven open-domain QA benchmarks while staying index-free.

How is it different from the "Is Grep All You Need?" study?

That study wired an untrained grep tool into agents and measured it against vector retrieval; it does no learning. GrepSeek instead trains the search behaviour — a two-stage Tutor/Planner distillation followed by GRPO — so the agent learns which commands to run rather than relying on hand-written heuristics.

Originally posted on Learn AI Visually.

AgentDoG 1.5: Small Inline Guard Models for Agent Actions

pueding — Mon, 01 Jun 2026 11:16:41 +0000

What: AgentDoG 1.5, an arXiv preprint posted in May 2026, is a family of small inline guard models (0.8B–8B parameters) that sit beside an agent and screen each action — a tool call, a shell command, a code-execution request — as safe or risky before it runs.

Why: Every production agent needs something watching its actions, and the lethal trifecta means an agent with private data, untrusted input, and a way to act can be steered into harm; a guard model is the screen that catches it.

vs prior: The usual screen is a large closed safety model (GPT-class) or a heavyweight sandboxed checker run per action; AgentDoG reports matching that catch rate with a model trained on only ~1,000 purified samples at roughly 100× less deployment overhead.

Think of it as

A rookie door guard who studied a veteran's casebook.

              AN AGENT ACTION REACHES THE DOOR
             (tool call / shell / code execution)
                            │
              ┌─────────────┴──────────────┐
              │                            │
     ┌────────▼────────┐          ┌────────▼────────┐
     │   ROOKIE GUARD  │          │  VETERAN CHIEF  │
     │  0.8B–8B model  │          │   closed model  │
     │     (inline)    │          │  (own service)  │
     └────────┬────────┘          └────────┬────────┘
              │                            │
     ~1k-case casebook;          same catch rate, but
     cheap on every action       ~100× deploy overhead
              │                            │
              ▼                            ▼
       ✓ screen them all          ✗ too costly to screen
         affordably                 every single action

agent action = a visitor at the door asking to come in (a tool call or shell command)
guard model = the rookie who clears safe visitors and stops risky ones, in real time
closed safety model = the veteran chief — just as sharp, but expensive to keep on the payroll
~1,000 training samples = a thin casebook holding only the most instructive incidents
influence-function purification = throwing out the case files that taught the rookie nothing

Quick glossary

Guard model — A separate, dedicated classifier that screens an agent's inputs and actions for risk — distinct from the agent LLM doing the task. It lives inline in the loop and returns an allow / block decision per action. See Agent Engineering → Output Filters.

Lethal trifecta — The three ingredients that make an agent dangerous together: private data access, exposure to untrusted content, and an exfiltration channel. A guard model is one way to break the chain. See AI Agents → The Lethal Trifecta.

Influence functions — A method that estimates how much each training example actually helps the model. AgentDoG uses it to purge low-value samples so that roughly 1,000 high-signal cases remain — the "purification" step.

Taxonomy-guided data engine — A pipeline that synthesizes training examples from a structured catalogue of risk categories. AgentDoG's taxonomy is updated to explicitly cover code-execution risks, not just text-level prompt attacks.

SFT + RL — Supervised fine-tuning (learn from labeled examples) followed by reinforcement learning (learn from reward in an environment). AgentDoG trains its guards in an "agentic safety" SFT+RL setup so they see realistic action traces, not isolated prompts.

Defense-in-depth — Layering independent filters so no single failure is fatal — input filters, output filters, policy checks, fail-safe defaults. A cheap guard model makes it affordable to add more layers. See Agent Engineering → Defense-in-Depth.

The news. On May 29, 2026, researchers posted AgentDoG 1.5, a lightweight alignment framework for agent safety. It trains guard models at 0.8B, 2B, 4B, and 8B parameters on roughly 1,000 samples, using a taxonomy-guided data engine (now covering code-execution risk) with influence-function purification. The paper reports performance comparable to leading closed models such as GPT-5.4 on agent-risk screening, while cutting Docker-level deployment overhead by about two orders of magnitude. Read the preprint →

Picture a doorway with a guard. Every visitor — a delivery, a contractor, someone who insists they were invited — stops at the door and the guard makes one call: come in, or turn around. That is exactly the job of a guard model in an agent system. The agent emits an action — call this tool, run this shell command, execute this code — and before the action reaches the real world, a small dedicated model screens it. The rookie at the door is not the chief of security; it is a 0.8B-to-8B model whose only skill is clearing safe actions and stopping risky ones, fast. The expensive alternative is to put the veteran chief on the door: a large closed safety model that is just as sharp but costs far more to keep standing by for every action.

What makes the rookie good is the casebook, not the size of its brain. AgentDoG's guards are not trained on millions of examples; they are trained on roughly 1,000. A taxonomy-guided data engine synthesizes candidate cases from a structured list of agent risks — and crucially the taxonomy is extended to cover code execution, the place where a steered agent does the most damage. Then influence functions estimate which of those synthesized cases actually move the model and throw the rest away, the way you would keep the three case files that taught a real lesson and bin the hundred that were routine. The cleaned set trains the guard in an SFT + RL loop that shows it realistic action traces rather than isolated prompts.

In the layered-defense picture, a guard model is an input and output filter standing in the agent loop. It is one concrete way to cut a leg off the lethal trifecta: even when an agent has private data and reads untrusted content, the guard can refuse the action that would exfiltrate it through a tool call. Because the guard is cheap, you can afford to run it on every action and still add other layers — which is the whole point of defense-in-depth. The design choice that remains yours is what the guard does when it is unsure: fail-safe (block and ask) is the conservative default, fail-open (allow) trades safety for uptime.

How the guard sizes stack up

Guard variant	Params	~4-bit footprint	Where it fits
AgentDoG-0.8B	~0.8B	~0.4 GB (derived: 0.5 byte/param)	Sidecar on the same GPU, or even CPU
AgentDoG-2B	~2B	~1 GB (derived)	Sidecar on the agent's GPU
AgentDoG-4B	~4B	~2 GB (derived)	Sidecar on the agent's GPU
AgentDoG-8B	~8B	~4 GB (derived)	Shares one GPU with the agent
Closed safety model	tens of billions (setup-dependent, illustrative)	~100+ GB (illustrative)	Its own Docker-sandboxed service

Where the ~100× actually comes from

Hold the catch rate fixed — the paper's claim is that the small guard matches the closed model there — and the win is in Docker-level deployment overhead: the standing service and sandbox each screen needs. Walk a back-of-envelope version (illustrative footprints; the preprint reports the overhead ratio, not these absolutes). A frontier-scale closed safety model runs to tens of billions of parameters — on the order of ~100 GB in FP16 — and is typically deployed as its own sandboxed service. AgentDoG-8B in 4-bit is about 8B × 0.5 byte ≈ 4 GB, and the 0.8B variant is under 0.5 GB — small enough to ride as an in-process sidecar next to the agent rather than as a separate service. That difference — a separate Docker-sandboxed service versus an in-process sidecar — is the roughly two-orders-of-magnitude (~100×) deployment-overhead cut the paper reports, and it is what makes screening every action affordable instead of sampling a few.

The catch, and the reason a small guard is not a free win: a model trained on ~1k cases only knows the risks in its taxonomy, so coverage gaps are real, and a determined attacker can probe for the action it does not recognize — the same evasion pressure that the camouflage-injection detection gap explainer describes. The parity-with-GPT-5.4 number is the paper's reported result, not an independent reproduction, and "comparable on a benchmark taxonomy" is narrower than "as safe in the wild." Treat AgentDoG as a cheap, always-on layer — not a replacement for capability scoping and a real data-flow review.

Goes deeper in: Agent Engineering → Layered Guardrails → Output Filters

Related explainers

Camouflage-injection detection gap — the attack a guard model has to catch: prompt injection hidden where a screener does not look
Copilot/Cowork image-URL exfiltration — a concrete exfiltration channel an output filter is meant to block
Glasswing — detection in a saturated pipeline — what changes when the screener itself becomes the bottleneck

FAQ

What is a guard model for agent actions?

A guard model is a small, dedicated classifier that sits inline in an agent's loop and screens each action — a tool call, a shell command, a code-execution request — as safe or risky before it runs. It is separate from the agent LLM doing the work; its only job is the allow/block decision. AgentDoG 1.5 trains such guards at 0.8B, 2B, 4B, and 8B parameters so the screen can run as a cheap sidecar rather than a heavyweight closed safety model.

Why does AgentDoG only need ~1,000 training samples?

It uses a taxonomy-guided data engine to synthesize candidate cases from a structured catalogue of agent risks (including code execution), then applies influence functions to keep only the examples that measurably improve the model and discard the rest. The result is a small, high-signal "casebook" of roughly 1,000 samples instead of millions, trained in an SFT-then-RL agentic-safety setup. Quality and coverage of the taxonomy matter more than raw sample count.

How does a guard model relate to the lethal trifecta?

The lethal trifecta is private-data access plus untrusted content plus an exfiltration channel — dangerous only when all three combine. A guard model is one way to cut a leg off that triangle: even when an agent holds private data and reads untrusted input, the guard can refuse the specific action that would leak it. Because a small guard is cheap to run on every action, it slots into a defense-in-depth stack alongside capability scoping and data-flow review rather than replacing them.

Originally posted on Learn AI Visually.

Claude Opus 4.8: Parallel-Subagent Dynamic Workflows

pueding — Sun, 31 May 2026 11:17:47 +0000

What: The Claude Opus 4.8 release adds "dynamic workflows" in Claude Code: a lead agent can fan out parallel subagents instead of running subtasks one after another.

Why: Independent subtasks — search the docs, read the code, run the tests — don't need each other's output, so running them concurrently and merging the results finishes in the time of the slowest one and, in the usual subagent design, keeps each subtask's context isolated.

vs prior: The usual single-agent loop does one tool call at a time, so wall-clock grows with the sum of the subtasks and a single context window has to hold everything at once.

Think of it as

A pit crew changing four tires at once instead of one mechanic doing all four.

                    CAR NEEDS 4 NEW TIRES
                            │
            ┌───────────────┴───────────────┐
            │                               │
   ┌────────▼────────┐             ┌────────▼────────┐
   │   ONE MECHANIC  │             │     PIT CREW    │
   │     (serial)    │             │   (parallel)    │
   └────────┬────────┘             └────────┬────────┘
            │                               │
   tires done one-by-one          4 crew, one tire each
      each in turn                     all at once
            │                               │
            ▼                               ▼
   ✗ time = sum of all 4         ✓ time = the slowest one

orchestrator = the crew chief who sends everyone in at once
subagent = one crew member on one tire, working independently
parallel run = all four tires changed in the time of the slowest one
merge = the chief waves the car out once all four are done

Quick glossary

Orchestrator (lead agent) — The agent that owns the task, decides how to split it, and dispatches the pieces. It does not do the subtask work itself — it coordinates. See Agent Engineering → Agent Teams → Supervisor/worker.

Subagent — A spawned worker agent that handles one subtask and reports a result back; in the usual design it runs in its own context window. Background: AI Agents → Context Engineering → Subagents.

Fan-out / fan-in — The shape of the workflow: fan-out is the orchestrator launching many subagents at once; fan-in is collecting their results back into one place to merge.

Context isolation — In the usual subagent design, each subagent gets a fresh, narrow context window with only what its subtask needs, so the orchestrator's window doesn't fill with every subtask's raw output. A context-engineering win as much as a speed one.

Wall-clock latency — Real elapsed time the user waits, as opposed to total compute. Parallelism trades more concurrent compute for less wall-clock. See Agent Engineering → Cost & Latency → Parallel tools.

Orchestrator-workers pattern — The classic agent design this productizes: a central agent splits a task, hands pieces to worker agents, and synthesizes their outputs. Walkthrough: AI Agents → Workflow Patterns → Orchestrator-workers.

The news. On May 28, 2026, Anthropic released Claude Opus 4.8. Among the agentic upgrades, the announcement calls out "dynamic workflows" that let Claude Code run parallel subagents. Anthropic framed it as a harness capability rather than a model-internals change — the model got better at the judgment of when to split work, and the harness got the machinery to run the pieces at once. Read the release →

Stay with the pit crew for a second. A single mechanic changing all four tires does them in series: jack, loosen, swap, torque, repeat — four times. A pit crew sends one person to each wheel at the same time, so the stop lasts as long as the slowest corner, not the sum of all four. The crew chief doesn't turn a single wheel; their whole job is to send everyone in together and wave the car out once all four are done. That division of labor is the entire idea behind a parallel-subagent workflow.

In an agent, the crew chief is the orchestrator. Faced with a task whose parts don't depend on each other — search the docs, read the code, run the tests, draft a summary — it can fan out a subagent per part instead of doing them back-to-back. In the usual subagent design each one works in its own context window, so the orchestrator's window doesn't bloat with four subtasks' raw output, and the results fan back in to be merged into one answer. The payoff is wall-clock: with independent subtasks, elapsed time tracks the slowest subagent, not the running total. (Anthropic disclosed that Claude Code can run parallel subagents; the fan-out/isolate/merge shape here is the established orchestrator-workers pattern the release productizes, not a newly published harness internal.)

The sharp edge is the word independent. The pit crew only works because the four corners don't wait on each other; if torquing the front-left depended on first finishing the rear-right, you'd be back to serial. The same holds for agents — parallelization is a win for subtasks that can run blind to each other, and a trap for a dependency chain where step two needs step one's output. The orchestrator's real skill is telling those apart, which is why the release pairs the harness machinery with better agentic judgment about when to split.

Serial vs. parallel, and when each wins

Approach	Wall-clock for N subtasks	Best when
Single agent, serial	≈ sum of all subtasks	subtasks form a dependency chain (step 2 needs step 1)
Parallel subagents	≈ slowest subtask + merge (coordination overhead, illustrative)	subtasks are independent and read-mostly

Walk the numbers with four illustrative subtasks taking 5.0s, 6.2s, 6.8s, and 4.4s. Run serially, wall-clock is the sum: 5.0 + 6.2 + 6.8 + 4.4 = 22.4s (illustrative). Fan them out as parallel subagents and wall-clock collapses to the slowest corner — 6.8s — for a 22.4 / 6.8 ≈ 3.3× speedup (illustrative). Coordination isn't free: the orchestrator spends a little time dispatching and a little merging, so a realistic number is a touch below 3.3×. And the ceiling is fixed by that slowest subtask — adding a fifth fast subagent doesn't help once "run the tests" is already the long pole.

Goes deeper in: Agent Engineering → Agent Teams → Supervisor/worker

Related explainers

Claude Opus 4.8 — Cache-preserving mid-task system messages — the other harness-level feature from the same release, on the caching side rather than the orchestration side
MSR delegation study — Cascading fidelity loss over 20 iterations — the cost of delegation: handing work to subagents is fast, but each handoff can lose fidelity if you aren't careful

FAQ

What are parallel-subagent dynamic workflows?

They are a Claude Code capability in Opus 4.8 where a lead agent (the orchestrator) splits a task into independent subtasks and launches a separate subagent for each one to run at the same time, then merges their results. "Dynamic" means the orchestrator decides the split at run time based on the task, rather than following a fixed, pre-wired script. It is the orchestrator-workers pattern made native to the harness.

Why does running subagents in parallel cut wall-clock time?

Because independent subtasks don't have to wait for each other. Run serially, elapsed time is the sum of every subtask; run in parallel, elapsed time is just the slowest one plus a little coordination overhead. For four subtasks of roughly equal size that is close to a 4× reduction in wall-clock (illustrative) — though the real ceiling is fixed by the single longest subtask, so an uneven split benefits less.

When does parallelizing subagents NOT help?

When the subtasks form a dependency chain — if step two needs step one's output, you can't start it early, so parallelism buys nothing and just adds coordination cost. It also adds little when one subtask dominates the others (the slowest one sets the floor), or when the subtasks share so much state that isolating their context windows loses important information. The orchestrator's job is to recognize these cases and keep them serial.

Originally posted on Learn AI Visually.

OmniRetrieval: Source-Native Query Dispatch

pueding — Sat, 30 May 2026 11:26:26 +0000

What: The OmniRetrieval paper introduces source-native query dispatch: a router sends a natural-language query to whichever knowledge source fits — text, tables, or graphs — and runs each source's own query engine instead of embedding everything into one vector store.

Why: Many vector-first RAG stacks flatten every source into a single embed-then-ANN pipeline, which throws away the structure that makes tables and graphs useful. Keeping each source native lets JOINs and graph edges survive retrieval, so the generator sees answers a flat similarity search can't assemble.

vs prior: The previous default is a unified vector index — chunk, embed, and nearest-neighbour everything together. Its failure mode is structural collapse: a table's column relationships and a graph's edges are averaged into a single vector and can no longer be queried as structure.

Think of it as

A reference desk that routes each question to the right specialist clerk.

                THE QUERY
                    │
             router (clerk)
                    │
        ┌───────────┴───────────┐
        │                       │
 ┌──────▼────────┐     ┌────────▼──────┐
 │    FLATTEN    │     │   DISPATCH    │
 │  one vector   │     │  text, SQL,   │
 │   index bin   │     │  graph query  │
 └──────┬────────┘     └────────┬──────┘
        │                       │
  shred docs into        ask each clerk
  one big bin            in its own tongue
        │                       │
        ▼                       ▼
 ✗ JOINs & edges        ✓ JOINs compose,
   averaged away          edges traverse

query = a question handed to the reference desk
router = the clerk who decides which specialist to ask
source-native query = asking each specialist in their own language — ledgers, archives, family trees
unified vector index = shredding every document into one bin of identical index cards
preserved structure = the ledger keeps its columns and the family tree keeps its lines

Quick glossary

RAG — Retrieval-Augmented Generation — fetch relevant context from a knowledge store, then condition the model on it.

Vector index — The default RAG store: every chunk is embedded into a fixed-dimension vector, and retrieval returns the top-k nearest by cosine or dot-product similarity. Structure inside a chunk is not preserved — only its position in embedding space.

ANN — Approximate Nearest Neighbour — the index family (HNSW, IVF, FAISS) that makes vector search fast by trading exactness for speed.

Source-native query — Running a source in its own engine: full-text search over passages, a SQL-style query (with JOINs) over tables, a traversal over a graph — rather than one similarity lookup over a shared embedding space.

Heterogeneous-source retrieval — Retrieval across sources of different kinds — unstructured text, relational tables, and graphs — kept in their native form instead of homogenised into one representation.

Knowledge base (KB) — A single corpus the system retrieves from. OmniRetrieval reports evaluating across 309 distinct KBs spanning 13 datasets.

The news. On May 29, 2026, the OmniRetrieval paper (arXiv:2605.29250) proposed a retrieval framework that accepts a natural-language query and routes it to the appropriate knowledge source — unstructured text, relational tables, or graphs — dispatching each source's native query to its own execution engine rather than flattening everything into a single embedding index. The authors report evaluating across 13 datasets and 309 distinct knowledge bases, and exceeding single-source retrieval baselines.

Picture the reference desk again. A question comes in — "which suppliers shipped to the Berlin warehouse in Q3, and who introduced them?" The clerk doesn't translate the question into one bland house dialect and shout it at the whole building. They split it: the archivist searches the prose contracts, the accountant runs the numbers in their ledgers, and the genealogist walks the introductions graph. Each specialist answers in their own language, keeping the structure that makes their corner useful — the ledger's columns, the family tree's lines.

The animation above is that desk. In the first beat the query flows query → router → one flat vector index: every source is shredded into the same uniform grid of vectors, and the table's JOIN arc and the graph's edges go dashed and grey — flattened into embedding space where structure can't be queried. Then the router flips from flatten to dispatch: the same three sources light up in their native forms, and three typed queries fan out — text search, table query · JOIN, graph traversal — with the JOIN arc and graph edges restored in green.

The mechanism is a router plus per-source engines. Instead of an embed-then-ANN lookup over one homogenised store, OmniRetrieval generates a query in each source's own language and runs it on that source's engine, then unifies the heterogeneous results for the generator. Because the table never left its relational form, a JOIN still composes rows by key; because the graph never left its node-edge form, a traversal still follows edges. Those are exactly the structural affordances a single similarity vector erases.

Why flattening loses the answer

Hold the Berlin question fixed and walk the two paths. Say the answer needs a JOIN of a 5,000-row suppliers table with a 40,000-row shipments table on supplier_id. The relational path evaluates the key match exactly — every shipment resolves to its supplier, and the introductions graph then traverses two hops to the people who introduced them. The flat-index path instead embeds each row into, say, a 768-dimension vector and returns, say, the top-k = 20 nearest chunks to the query. Two hundred million potential supplier–shipment pairings (5,000 × 40,000 = 200,000,000, illustrative) collapse into 20 fuzzy neighbours chosen by cosine distance — and "who introduced them" is an edge that was never embedded as an edge. The JOIN and the traversal are not slow in the flat index; they are absent. That is the RAG failure mode source-native dispatch is built to remove.

Flat vector index vs. source-native dispatch

Property	Unified vector index	Source-native dispatch
Index built upfront	one embedding pass + ANN index over all sources	each source keeps its native engine (full-text, SQL, graph)
Query form	one nearest-neighbour lookup over shared space	a source-native query per source, chosen by the router
Structure preserved	no — JOINs and edges averaged into a vector	yes — JOINs compose, edges traverse
Best when	fuzzy topical match over homogeneous prose	answer spans tables / graphs, needs exact relations
Reported scope	—	~13 datasets · 309 KBs, beats single-source baselines (OmniRetrieval)

The table isn't an argument that embeddings are obsolete — fuzzy topical recall over prose is exactly what a vector index is good at. It's that a single representation can't be the right one for text, tables, and graphs at once. Routing to a source-native query lets each source answer with the structure it actually has, and only then unifies — so the generator receives composed relations, not a bag of nearest neighbours.

Goes deeper in: AI Agents → Retrieval & RAG → RAG Failure Modes

Related explainers

Is Grep All You Need? — Grep vs vector retrieval for agentic search — another result that the vector index is one option among many, not the default
CDD — Context-Driven Decomposition for RAG knowledge conflict — what to do once retrieval returns sources that disagree

FAQ

What is source-native query dispatch?

It's a retrieval design where a router sends a natural-language query to whichever knowledge source fits — unstructured text, a relational table, or a graph — and runs that source's own query engine (full-text search, a SQL-style query with JOINs, or a graph traversal) instead of embedding everything into one shared vector store. OmniRetrieval reports doing this across 13 datasets and 309 knowledge bases and exceeding single-source baselines.

Why not just embed tables and graphs into the same vector index?

Because embedding collapses structure. A table's columns and a graph's edges become a single vector positioned by similarity, so a JOIN can no longer compose rows by key and a traversal can no longer follow edges — those relations are averaged away rather than slowed down. Keeping each source native preserves the structural affordances that answer relational and multi-hop questions, which a top-k nearest-neighbour lookup over one space cannot reconstruct.

Does this mean vector retrieval is obsolete?

No. A unified vector index is still the right tool for fuzzy, topical recall over homogeneous prose, where semantic similarity is doing the real work. Source-native dispatch matters when an answer spans different kinds of sources or needs exact relations — tables to JOIN, graphs to traverse. The shift is in the default: route to the source's native engine first, and treat the shared embedding space as one source among several rather than the only one.

Originally posted on Learn AI Visually.