DEV Community: Shridhar Shah

Stop Paying the Worst Case on Every Query: Adaptive Test-Time Compute

Shridhar Shah — Wed, 05 Aug 2026 16:21:45 +0000

Best-of-N spends the same reasoning budget on trivial and brutal questions alike. A difficulty gate plus a verifier concentrates compute on the queries that need it — same accuracy, a fraction of the cost.

TL;DR: The standard way to buy accuracy at inference — best-of-N, sample N reasoning attempts and pick the best — is flat-rate. It burns the full budget on a trivial query and still under-serves a brutal one. Adaptive test-time compute allocates instead: a cheap gate estimates difficulty and a verifier checks samples as they arrive, so easy queries stop after one accepted answer and hard queries keep exploring up to a cap. In a runnable Python simulation, adaptive matched best-of-8's 91% accuracy using 77% fewer samples (3,200 → 732; 1.8 samples/query vs 8). 2026 research reports the same effect on real benchmarks (adaptive verifier-guided allocation, budgeted metacognitive control).

Mental model: an exam where you're told to spend exactly 8 minutes on every question. You waste 7 minutes on the ones you knew instantly and run out of time on the killers. Any real test-taker budgets by difficulty — glance, answer the easy ones, and pour the saved time into the hard tail.

The problem: uniform compute is the wrong default

Test-time scaling is one of the biggest levers of the last two years: sample multiple reasoning trajectories, verify or vote, and accuracy climbs. But the common implementations spend uniformly — best-of-N, self-consistency with a fixed sample count, fixed-width beams. Every query gets the same budget regardless of whether it's a lookup or a competition math problem.

That's doubly wasteful. On the easy majority you pay for samples you never needed — the first answer was already right. On the hard tail, a fixed N may not be enough, and the budget you could have spent there was squandered on the easy ones. As the 2026 work puts it, useful scaling should be selective: driven by intermediate quality and uncertainty, not spread evenly across trajectories.

The pattern: gate, verify, stop early

Two cheap components turn a flat budget into an allocated one:

A difficulty gate — a fast, approximate estimate of how hard the query is (a small classifier, a confidence probe, a router). It sets the cap: how many samples this query is even allowed.
A verifier — a check on each sample (a process reward model, an NLI/consistency check, unit tests for code). It sets the stop: as soon as a sample is accepted, stop spending.

Easy query → gate says "cap 1", first sample is accepted, done. Hard query → gate raises the cap and the loop keeps exploring until the verifier is satisfied or the cap is hit. The verifier is the same signal both policies could use — adaptive just also uses it to stop:

def adaptive(rng, queries):
    for d in queries:
        d_hat = clamp(d + noise())            # cheap, noisy difficulty gate
        budget = round(1 + (N_MAX - 1) * d_hat)  # easy -> 1, hard -> up to N_MAX
        for i in range(budget):
            ok = sample_is_correct(rng, d)
            if verifier_accepts(rng, ok):
                break                          # accepted: stop spending compute here

The result

Adaptive Test-Time Compute — spend reasoning where it pays, not uniformly

  before → after:  3,200 samples for 91% accuracy (uniform best-of-8)  →  732 samples for 91% (77% less compute)

  400 queries, 113 hard / 287 easy; verifier 90% true-accept, 6% false-accept.

   policy                       samples   accuracy
   uniform best-of-8              3,200        91%
   adaptive (verifier-gated)        732        91%

   early stops (solved in 1 sample)    243   (61% of queries)
   avg samples / query                 1.8   (uniform always spends 8.0)

Identical accuracy, less than a quarter of the compute. 61% of queries were resolved in a single sample — the easy majority that best-of-N was over-serving — and the saved budget stays available for the hard tail. The lever is the difficulty distribution: the more skewed toward easy your traffic, the bigger the win.

Reality check: the 77% is from the simulation above — directional, not a benchmark. The 2026 papers report the same effect on real benchmarks — matching or beating uniform best-of-N at a fraction of the compute, with several-fold gains on hard math per unit compute — and your savings scale with how skewed-easy your traffic is and, above all, how good your verifier is (a weak verifier silently caps the whole approach).

Why this is where 2026 is heading

Uniform test-time scaling is now understood as leaving compute on the table. What If We Allocate Test-Time Compute Adaptively? (arXiv 2602.01070) replaces fixed sampling with a process-reward-model signal that guides pruning and expansion within a query and selection across iterations, concentrating computation on high-utility reasoning paths and reporting several-fold gains on hard benchmarks per unit of compute. CoT2-Meta (arXiv 2603.28135) frames it as metacognitive control: a controller decides whether to expand, prune, repair, stop, or abstain on each partial trajectory, separating object-level reasoning from meta-level budget decisions. The same instinct shows up in Adaptive RAG, which routes a query to a direct answer, a single retrieval, or a full agentic loop based on estimated complexity.

The transferable engineering idea needs no new model: put a verifier and a difficulty estimate in front of your sampling loop, and make the budget a function of both. Even a crude gate (prompt length, a cheap classifier, first-sample confidence) plus a domain verifier (tests, a schema check, self-consistency) captures most of the savings.

How faithful is this demo?

It models allocation, not reasoning: a "sample" is a difficulty-weighted coin flip and the verifier is a fixed true/false-accept rate, so it deliberately skips why a sample is right. Two caveats it makes visible: the verifier isn't perfect (a 6% false-accept rate means adaptive can stop early on a wrong answer — verifier quality is the ceiling on this whole approach), and the difficulty gate is noisy, so some hard queries get under-budgeted. Both are exactly the failure modes the 2026 papers work to control with process reward models and calibrated confidence. Real gains depend on your verifier being better than your generator.

When not to use this

You don't have a decent verifier. The verifier is the ceiling: a weak one stops early on wrong answers and quietly caps accuracy. No trustworthy check → don't gate on it.
Traffic is uniformly hard. With no easy majority to stop early on, adaptive collapses back to best-of-N — you save nothing.
A single pass already suffices. If one sample is reliably right, you don't need test-time scaling at all; adaptive only pays once you're already sampling multiple times.

Try it

python3 demo.py   # standard library only

Sources & further reading

Papers (2026)

What If We Allocate Test-Time Compute Adaptively? (arXiv 2602.01070) — a verifier-guided framework where a process reward model drives per-query pruning, expansion, and selection instead of uniform sampling; large gains on hard math benchmarks per unit compute.
CoT2-Meta: Budgeted Metacognitive Control for Test-Time Reasoning (arXiv 2603.28135) — separates object-level reasoning from a meta-level controller that decides expand / prune / repair / stop / abstain under a budget.
Self-Correcting RAG (ACL Findings 2026) — casts context selection as a knapsack problem and uses NLI-guided search to spend test-time compute on faithfulness.

Background

Snell et al., Scaling LLM Test-Time Compute Optimally (2024), and self-consistency / best-of-N — the uniform baselines this pattern improves on.
Adaptive RAG — route each query to the cheapest path (direct answer, single retrieval, or agentic loop) that will solve it.

One Long Prompt Shouldn't Freeze Everyone's Tokens: Prefill/Decode Disaggregation

Shridhar Shah — Wed, 05 Aug 2026 16:21:43 +0000

An LLM request is two workloads in a trench coat — a heavy, bursty prefill and a stream of tiny latency-sensitive decodes. Running them on the same engines lets one big prompt stall everyone. Splitting them fixes it.

TL;DR: Every LLM request is two very different jobs. Prefill reads the whole prompt — heavy, bursty, and slow for long contexts. Decode then emits tokens one at a time — tiny, but latency-sensitive. Run them on the same engines and a big prefill jumps ahead of everyone's decodes: head-of-line blocking, and the token stream stutters. Prefill/decode disaggregation puts prefill and decode on separate pools so decodes never queue behind a prefill. In a runnable Go simulation, splitting the pools cut p99 inter-token latency by 66% (88ms → 30ms) — trading a little time-to-first-token for a far smoother stream. This is now standard practice in frontier serving stacks (DistServe, Splitwise, vLLM × Mooncake).

Mental model: a coffee shop with one worker who both grinds beans and pours espresso. A customer orders a giant batch grind and everyone waiting for a simple pour is stuck behind it. Split the shop into a grinder station and a pour station and the pours keep flowing no matter how big the grind.

The problem: two workloads, one queue

Serving an LLM token is not one kind of work — it's two:

Prefill processes the entire prompt to build the KV cache. It's a big, compute-bound burst, and it scales with prompt length: a 100k-token context is a genuinely slow operation.
Decode generates the response one token at a time, each step cheap but on the critical path of what the user feels — the inter-token latency (ITL) is the smoothness of the stream.

Colocate them — the default — and they fight for the same compute. Continuous batching helps throughput but doesn't remove the conflict: when the engine runs a long prefill, the decodes it's also hosting have to wait. And they can't flee to a less-busy engine, because this engine holds their KV cache. So one long-context prompt lands and every active token stream on that engine stutters. Your p99 ITL is hostage to your longest prompt.

The pattern: split the pools

Disaggregation (DistServe, Splitwise) separates the two phases onto different hardware:

A prefill pool does nothing but build KV caches — bursty, compute-heavy work, isolated.
A decode pool does nothing but stream tokens — steady, latency-sensitive work, isolated.
The KV cache is handed off from prefill to decode (the expensive-to-move part — this is exactly what fast KV transfer layers like Mooncake exist to make cheap).

Now a giant prefill can't block anyone's decodes, because it physically runs on a different pool. Each pool can also be tuned and scaled independently for its own SLO (TTFT for prefill, ITL for decode) instead of compromising on one knob for both.

The simulation pins each request's decodes to the engine that ran its prefill (KV-cache locality) — the constraint that makes colocated blocking unavoidable:

if disaggregated {
    if j.prefill {
        server, dur = 0, w.prefill[j.req] // prefill pool
    } else {
        server, dur = 1, decodeDur        // decode pool — never behind a prefill
    }
} else {
    server = j.req % 2                    // pinned engine holds this request's KV cache
    // ...its decodes are stuck behind whatever prefill lands here
}

The result

Prefill/Decode Disaggregation — keep long prefills from stalling the token stream
  before → after:  p99 inter-token latency 88ms (colocated)  →  30ms (disaggregated)   (66% lower)
  240 requests, 2 servers, 20% long-context bursts (700–1800ms prefill), 20 decode steps × 5ms.

   layout                  p99 token lat mean token lat    mean TTFT
   colocated (shared)              88 ms          15 ms       426 ms
   disaggregated (split)           30 ms          11 ms      1032 ms

Same workload, same total hardware (2 engines each). Colocated lets bursty prefills jump ahead of tiny decodes, so the p99 token stutters to 88ms. Disaggregated isolates decodes and holds p99 to 30ms — a 66% smoother stream. The honest cost is TTFT: with only one engine dedicated to prefill, first tokens arrive later (426ms → 1032ms). That's the real knob disaggregation gives you — pool sizing lets you buy back TTFT by provisioning prefill and decode independently.

Reality check: these are simulated queue latencies, not GPU numbers — directional only. The direction is well-established: DistServe and Splitwise report multiples-higher goodput under latency SLOs from PD disaggregation, and the real-world win hinges on how cheap your KV-cache transfer is and how much prefill actually contends with decode.

Why this is where 2026 is heading

Disaggregation went from research idea to default in two years. DistServe showed that separating prefill and decode and sizing each for its own SLO can serve multiples more requests under latency constraints; Splitwise made the same case for splitting the phases across different hardware. By 2026 it's productized: vLLM ships PD disaggregation, and the vLLM × Mooncake work pairs it with a distributed KV cache pool so prefill and decode engines — even on different machines — share caches over fast transport. The load-bearing enabler is exactly the KV-cache handoff this demo hand-waves.

The transferable idea generalizes past LLMs: when one queue mixes bursty heavy work with steady latency-sensitive work, isolate them. It's the same instinct as separating batch from interactive traffic, or OLAP from OLTP — applied at the token level.

How faithful is this demo?

It models queueing and head-of-line blocking, not GPUs: "prefill" and "decode" are service times, and it ignores continuous batching, the real (non-zero) cost of KV-cache transfer, and memory pressure — all of which real systems must handle, and which is why cheap KV transport matters so much. Two honest caveats: disaggregation isn't free (you pay a transfer and a TTFT hop, visible above), and it only pays off when prefill bursts actually contend with latency-sensitive decodes. Short, uniform prompts under light load won't show the gap — measure your ITL tail before splitting.

When not to use this

Small scale / a single GPU. With nothing to disaggregate across, the KV-transfer hop and the TTFT cost dominate any benefit.
Short, uniform prompts. No bursty long prefills means no head-of-line blocking to fix.
TTFT is your primary SLO. Disaggregation trades first-token latency for a smoother stream; if users care most about time-to-first-token, it can be a net loss.

Try it

go run .   # standard library only

Sources & further reading

Papers

Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024, arXiv 2401.09670) — the case for splitting the phases and sizing each pool for its own latency target.
Patel et al. — Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024, arXiv 2311.18677) — splits prefill and decode across distinct machines to raise throughput per dollar and per watt.
Qin et al. — Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST 2025, arXiv 2407.00079) — trades more storage for less compute; the KV-cache pool that makes cross-engine handoff practical.

Engineering

Serving Agentic Workloads at Scale with vLLM × Mooncake (2026) — production PD disaggregation plus a distributed KV cache pool for multi-turn, agentic serving.
KV Cache Offloading: LMCache vs Mooncake vs Dynamo — how the KV-transfer tier underneath disaggregation actually works.

Your Agent Has a Bug You Can't Reproduce. Here's How to Catch It.

Shridhar Shah — Wed, 05 Aug 2026 16:21:08 +0000

Deterministic simulation testing drives every fault, clock, and random choice from one seed — so a flaky, once-in-production agent bug becomes a reproducible artifact you can shrink to one line.

TL;DR: The worst agent bugs only appear under a specific interleaving of faults — a tool fails right after a side effect, a retry fires, and money moves twice. Happy-path tests miss it, and when it hits production you can't reproduce it. Deterministic simulation testing (DST) — the technique behind FoundationDB, TigerBeetle, and Antithesis — makes faults, timing, and randomness a pure function of one seed, so any failure replays exactly and can be shrunk to its minimal cause. In a runnable Python demo, the happy path passes, seeded fuzzing catches a double-charge, replays it identically, and shrinks a 4-fault schedule down to the single fault that matters.

Mental model: a flight simulator with a record button. Instead of waiting for a storm to hit a real plane, you conjure storms on demand — and when one crashes the plane, you can replay that exact storm frame-by-frame until you understand it, then strip it down to the one gust that did the damage.

The problem: the bugs that matter are the ones you can't reproduce

Agents run in a hostile world. Tools time out, APIs return errors, retries fire, and steps race. Most of the time everything is fine. But somewhere in the space of when exactly does the fault land hides a bug — a retry that isn't idempotent, a state update that assumes a call succeeded, a compensation that runs twice. It shows up once, in production, moves real money, and then vanishes: you re-run the same input and it works, because the timing was different this time.

Traditional tests can't help. Example-based tests exercise the happy path. Even randomized tests, if they do trip the bug, can't tell you how — the randomness that triggered it is gone.

The pattern: make nondeterminism a function of a seed

DST flips the model. Every source of nondeterminism — fault injection, the clock, thread scheduling, RNG — is routed through a single seed. That buys three things:

Reproducibility. The same seed always produces the same run. A failure is a permanent artifact.
Exploration. Sweep thousands of seeds (fast, in simulated time) to explore fault interleavings a human would never think to write by hand.
Shrinking. Once you have a failing scenario, mechanically remove pieces until only the minimal trigger remains — turning a chaotic failure into a one-line repro.

Here a "fault schedule" is just the set of steps that will fail on their first attempt — the entire source of nondeterminism, made explicit and seedable. The workflow has a real bug: confirm's retry re-runs charge with no idempotency guard.

for step in PLAN:
    if step == "confirm":
        for t in range(2):
            try:
                attempt("confirm", t); break
            except Fault:
                attempt("charge", 1)   # <-- the bug: a second, unguarded charge on retry
    else:
        for t in range(2):            # every other step has a safe, idempotent retry
            try:
                attempt(step, t); break
            except Fault:
                continue

Because the run is a pure function of its schedule, fuzzing, replay, and shrinking are trivial:

def fuzz(seeds):
    for seed in range(seeds):
        schedule = random_schedule(random.Random(seed))
        if not invariant_holds(run_agent(schedule)):   # property: charges ≤ 1
            return seed, schedule

def shrink(schedule):
    minimal = set(schedule)
    for step in list(minimal):
        if not invariant_holds(run_agent(minimal - {step})):
            minimal -= {step}       # drop faults that aren't needed to trigger the failure
    return minimal

The result

1. happy-path test (no faults):      PASS  <- the bug is invisible here

2. fuzzing seeded fault schedules:   FAIL on seed 1
      schedule = ['analyze', 'confirm', 'notify', 'plan']  ->  charged the customer 2x

3. replay same schedule twice:       2x and 2x charges  ->  identical (reproducible)

4. shrink to the minimal cause:      ['confirm']
      one fault at 'confirm' is all it takes to double-charge.

The happy path passed — that's why this bug would ship. Fuzzing found a failing schedule with four faults, three of them irrelevant noise. Replay proved it reproduces exactly. Shrinking stripped the noise to a one-line repro: a single fault at confirm double-charges. That's a bug report a developer can fix in minutes, not a haunted "works on my machine."

Reality check: the state space here is tiny, but the technique isn't a toy — this exact method (FoundationDB, TigerBeetle's VOPR, Antithesis) catches real, money-moving bugs in production databases. At scale the hard part is exploration — which fault interleavings to try — not the seed-and-replay mechanism shown here.

Why this is where 2026 is heading

DST is having a moment. FoundationDB pioneered it; TigerBeetle's VOPR "speeds up time arbitrarily — one minute of simulation equals days of real testing"; Antithesis raised a large round selling a deterministic hypervisor that replays every instruction to reproduce failures. Nearly every serious database company is now using or evaluating it.

Agents are the next obvious target, and arguably a better fit: an agent's think→act→observe loop is already a sequence of discrete, mockable steps with well-defined failure points (tool errors, timeouts, partial writes). The engineering shift is to write the agent so its nondeterminism is injectable — clock, retries, and tool faults behind seams you control — and then let a simulator manufacture the chaos. The property you assert ("charges ≤ 1", "no orphaned reservations", "the plan never regresses") becomes the specification of correct behavior under failure.

How faithful is this demo?

It's a minimal model — nondeterminism is just "which steps fault first" and the state space is tiny. Real DST is harder in two ways: determinism is a discipline (every clock read and interleaving must route through the seed, which is why Antithesis built a hypervisor to enforce it), and exploration is the real problem (huge state spaces mean DST pairs with property-based testing and guided fuzzing). Start small: make one agent's faults and clock injectable, then assert one invariant across many seeds.

When not to use this

Stateless, single-shot prompts. With no faults, retries, or ordering to interleave, there's no nondeterminism to seed — plain example tests are enough.
The bug is model quality, not orchestration. DST catches control-flow and fault-handling bugs; it won't tell you the model gave a worse answer.
You can't make nondeterminism injectable. If the clock, retries, and tool faults can't be routed through a seam you control, you can't get determinism — retrofit the seams first.

Try it

python3 demo.py   # standard library only

Sources & further reading

Platforms & write-ups

Antithesis — Deterministic simulation testing: how it works and when to use it — seeds, fault injection, time-travel debugging, and why DST pairs with property-based testing.
TigerBeetle — the VOPR simulator — stubs out clock/network/disk, injects partitions and corruption, and replays any failure from (seed, commit).
FoundationDB — simulation testing — the origin of the approach.
Phil Eaton — What's the big deal about Deterministic Simulation Testing? — an approachable tour of controlling clocks and randomness.

Stop Making Your Agent Wait: Branch Prediction for Tool Calls

Shridhar Shah — Wed, 05 Aug 2026 16:21:05 +0000

Speculative tool execution guesses the next tool while the model is still reasoning, runs it in parallel, and hides the latency — discarding the guess when it's wrong.

TL;DR: An agent's think→act→observe loop is serial, so it burns a huge share of wall-clock sitting idle while a tool runs. Speculative tool execution borrows branch prediction from CPUs: a cheap predictor guesses the next tool call and runs it during the model's reasoning. If the guess matches, the result is already there and the wait disappears; if not, it's discarded and the real call runs — never a wrong answer. A confidence gate avoids wasting compute on low-odds guesses. In a runnable Go demo, a 58% hit rate cut wall-clock by 1.3× with zero correctness impact. Four 2026 papers (Speculative Actions, SPORK, PASTE, Speculate While You Reason) report 20–48% real speedups.

Mental model: a chef who starts searing the steak they're pretty sure you'll order while you're still reading the menu. If you order it, dinner is early. If you don't, they toss it and cook what you asked — you never get served the wrong dish, they just did some work in the background.

The problem: the agent loop wastes its own time

Agents run a strictly serial loop: reason, emit a tool call, wait for the result, reason again. That wait — a web request, a database query, a code execution — is dead time. The GPU (or your API budget) idles while the tool runs. Prior work measures this idle at 16–37% of wall-clock in typical workloads and up to 61% in tool-heavy ones. For a chatty, multi-step agent, that's the single biggest lever on latency, and no amount of prompt tuning touches it.

The catch is that the loop is only logically serial. The model doesn't strictly need to finish reasoning before the tool runs — it just needs the tool's result to be there when it's done.

The pattern: predict, pre-execute, verify

CPUs solved this with branch prediction: guess which way a branch goes and execute ahead speculatively, rolling back if wrong. LLM inference reuses the idea in speculative decoding. Applied to agents it becomes speculative tool execution:

Predict the next tool call from the trajectory so far (a cheap draft model, a probe on the agent's own logits, or — as here — a learned pattern of recurring tool sequences).
Pre-execute the guessed call in parallel while the model keeps reasoning.
Verify when the model emits its real call. Exact match → reuse the ready result. Mismatch → discard and run the real one.

The correctness guarantee is the whole point: the agent's actual output is always the ground truth. Speculation can only ever remove waiting, never change the answer.

A learned predictor over recurring tool patterns:

func (p *predictor) predict(last string) (guess string, confidence float64) {
    row := p.counts[last]
    best, bestN, total := "", 0, 0
    for tool, n := range row {
        total += n
        if n > bestN { best, bestN = tool, n }
    }
    return best, float64(bestN) / float64(total)
}

And the speculate-while-thinking core — the guessed tool runs in a goroutine, overlapping the model's reasoning; a confidence gate keeps us from speculating when the odds are poor:

guess, conf := pred.predict(last)
if guess != "" && conf >= gate {
    specCh = make(chan string, 1)
    go func(t string) { specCh <- runTool(t) }(guess) // runs in parallel with think
}
time.Sleep(thinkMS * time.Millisecond) // the model reasons

if specCh != nil && guess == actual {
    <-specCh          // correct: result already ready — latency hidden
    hits++
} else {
    runTool(actual)   // miss: discard the speculation, run the real tool
}

The result

Speculative Tool Execution — guess the next tool and run it while the model thinks
  24-step agent loop; think=40ms, tool=60ms per step.

   serial agent loop        2.443s   (think, then wait for the tool, every step)
   speculative loop         1.869s   (14 hits / 4 misses / 1 gated out)

   next-tool hit rate          58%
   wall-clock speedup        1.31x  (tool latency hidden behind reasoning)
   wasted speculations          4   (discarded on a miss — never a wrong answer)

The four misses cost some throwaway background work; the one gated-out step shows the confidence gate declining to gamble. The 14 hits each hid a tool call behind reasoning that was going to happen anyway. Correctness is untouched — every actual tool call still ran (or was proven already run).

Reality check: the wall-clock here comes from fixed sleeps (and drifts a little run-to-run), so read the hit rate and the mechanism, not the exact milliseconds. The direction is real: the 2026 papers report 20–48% speedups on live agents — SPORK predicts the next tool at 74–99% accuracy, PASTE cuts task-completion time ~48% — and your gain scales with how predictable your tool sequences are and how much of each step is actually tool latency.

Why this is where 2026 is heading

This jumped from theory to a small research wave in 2026, and the designs converge on the same shape. SPORK forks a probe off the agent's own KV cache to predict the next tool with 74–99% accuracy and gates dispatch on confidence — exactly the gate-and-discard structure above. PASTE mines recurring tool-call patterns (the assumption this demo's predictor encodes) and reports up to 48% lower task completion time. Speculative Actions frames it generally: a fast model stages likely actions so that validation, not waiting, is the critical path. And Speculate While You Reason shows the agent is its own best predictor.

The transferable engineering idea needs no training: any time your agent has predictable structure and idle wait, you can overlap them. Recurring tool sequences, known data-fetch fan-outs, and retry-then-reopen patterns are all speculatable today with a controller sitting over a normal completion API.

How faithful is this demo?

It models control flow and timing, not reasoning: "thinking" and "tool latency" are fixed sleeps and the predictor is a first-order Markov count over tool names — real speculators also predict tool arguments (much harder) and pay for a draft model. Two caveats the papers stress: speculation only helps when tool latency is a real slice of step time, and mispredicted side effects must be safe to discard (a read-only call is free to speculate; a charge_customer is not). Start by speculating idempotent, read-only tools on your most predictable loops.

When not to use this

Unpredictable next tools. A low hit rate means you mostly discard speculative work — you burn extra compute for little latency win.
Side effects that aren't safe to discard. Speculate read-only calls freely; never speculate a charge_customer or any write you can't cleanly roll back.
Tool latency is a small slice of step time. If reasoning dominates, there's little wait to hide, and compute is better spent elsewhere.

Try it

go run .   # standard library only

Sources & further reading

Papers (2026)

Zhou et al. — Speculative Actions: A Lossless Framework for Faster Agentic Systems (ICLR 2026) — fast models predict and stage actions; up to 55% next-action accuracy → up to 20% latency reduction, with a cost–latency analysis for tuning speculative breadth.
SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference (arXiv 2607.03333) — training-free controller; a prefix-cache fork predicts the next tool at 74.6–99.6% accuracy, a confidence gate filters mispredictions, serial fallback on rejection.
Act While Thinking: Pattern-Aware Speculative Tool Execution (PASTE) (arXiv 2603.18897) — exploits recurring tool-call sequences and parameter dependencies; 48.5% lower task completion time, 1.8× tool throughput.
Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent–Speculator RL (arXiv 2607.25816) — the agent itself is the speculator; Hit@1 improves from ~44% to ~61% while preserving task success.

Background

Speculative decoding for LLMs and branch prediction in CPUs — the same predict-execute-verify lineage this pattern descends from.

The Agent That Answers Before You Ask

Shridhar Shah — Wed, 05 Aug 2026 16:20:31 +0000

Sleep-time compute: split the budget so a background worker does the predictable thinking while idle — and the user waits far less when they finally ask.

TL;DR: Most agents only think when a request arrives — the user waits and you pay full latency every time. But between sessions there's idle capacity, and many queries are predictable variants of past ones over context that barely changes. Sleep-time compute (Letta; Lin et al., 2025) splits the budget: a background worker pre-answers likely queries while idle, so the foreground serves warm answers instantly and only falls back to a live call on a miss. A freshness check makes sure a changed document never yields a stale answer. In a tiny demo, foreground latency dropped 57% — with novel queries still handled live and stale pre-answers correctly rejected.

Mental model: a prep cook who chops the vegetables before the dinner rush. When orders come in, plating is fast because the prep is already done — and anything that spoiled gets thrown out and re-prepped fresh, never served stale.

The problem: you pay full price at the worst possible moment

Interactive inference spends compute at test time — the exact moment the user is waiting. For one-shot questions over fresh context that's unavoidable. But a huge share of real traffic isn't one-shot: users query the same codebase, the same document set, the same dashboard repeatedly, and the underlying corpus doesn't change between most of those queries. You're re-deriving the same expensive answers on the critical path, over and over.

Meanwhile the infrastructure sits idle between sessions, with nobody waiting.

The pattern: move predictable work to idle time

Sleep-time compute separates compute while serving a query from compute between queries. A background worker runs during idle periods and does two things: distill the standing context into dense summaries, and speculatively pre-answer the queries most likely to be asked next.

Prediction can be simple — the queries you've seen most are the ones you'll likely see again — and each pre-answer is tagged with the source version it was computed against, so freshness can be checked later:

def prepare(self, history, top_n=8):
    for query, _ in Counter(history).most_common(top_n):     # predict likely next queries
        ver = self.corpus.version[query_topic(query)]
        if self.cache.get(query, (None, None))[1] != ver:    # (re)compute if missing or stale
            self.cache[query] = (f"answer[{query} @v{ver}]", ver)
            self.bg_cost += LIVE_COST                        # paid in the background

At request time the foreground checks the cache — and serves a pre-answer only if it's still fresh. A miss or a stale entry falls back to a live call:

def serve(self, query):
    hit = self.cache.get(query)
    if hit and hit[1] == self.corpus.version[query_topic(query)]:  # present AND fresh
        return hit[0], WARM_MS, WARM_COST, "warm"
    return None   # miss or stale -> go live

The result

Sleep-Time Compute — precompute while idle so the user waits far less
  400 queries, 70% predictable / 30% novel, one source update mid-stream.

                           foreground latency   foreground cost
   answer on demand                  320.0s             8.00$
   with sleep-time                   137.2s             3.40$

   warm hits served instantly : 230/400  (58%)
   stale pre-answers rejected  : 1  (freshness check forced a live call — no wrong answer)
   background cost (amortized) : $0.26  spent while idle, off the critical path

The predictable 70% mostly got answered before the user asked. The novel 30% still went live — as they should. And when a source document changed mid-stream, the stale pre-answers were rejected and recomputed rather than served wrong.

Reality check: the 57% is from the toy model above — directional, not a benchmark. The paper behind it reports ~5× less test-time compute for equal accuracy and ~2.5× lower cost per query when a context is shared across queries — and, crucially, the win tracks how predictable your traffic is (it's pure overhead for genuinely one-shot asks).

Why this is where 2026 is heading

The proven part: the associated research (Letta's write-up; Lin et al.) shows that shifting reasoning to before a query arrives measurably cuts test-time latency and cost when the work is reused. The transferable engineering idea is the split budget: reserve interactive compute for latency-sensitive work, and spend cheap background compute where it can be amortized.

Where it's heading: as agents accumulate persistent memory, idle time stops being wasted. Expect a foreground agent and a cheaper background agent to share memory — the background one consolidating summaries, reflecting over past sessions, and pre-answering likely follow-ups. The same design shows up as "context distillation" and memory consolidation across 2026 agent stacks.

How faithful is this demo?

It's a minimal model of the control flow, not the reasoning: "inference" is a fixed-latency stub, prediction is a frequency count, and freshness is an integer version bump. Real systems predict from richer signals and must guard hard against two failure modes this only gestures at — serving stale answers (freshness must gate every hit) and privacy leakage during background consolidation. And the honest caveat stands: sleep-time compute pays off only when future queries reuse the work — it's pure overhead for one-shot asks.

When not to use this

One-shot or unpredictable traffic. If future queries rarely reuse the work, precompute is pure wasted spend.
Fast-changing context. If the underlying data churns constantly, pre-answers go stale faster than you can serve them — you'll pay to recompute anyway.
No cheap idle capacity. The whole trick assumes background compute is cheaper than blocking a user; if idle time costs the same, there's no arbitrage.

Try it

python3 demo.py   # standard library only

Sources & further reading

Paper

Lin, Snell, Wang, Packer, Wooders, Stoica, Gonzalez — Sleep-time Compute: Beyond Inference Scaling at Test-time (arXiv 2504.13171) — ~5× less test-time compute for equal accuracy; 2.5× lower cost/query when a context is shared across queries; efficacy correlates with query predictability. Code & data.

Engineering blogs

Letta — Sleep-time Compute (foreground + background agents sharing memory)
Sleep-time compute pattern (agent patterns catalog) — distillation + speculative pre-answering, freshness gating
Sleep-Time Compute for AI Agents (the split-budget framing and rollout advice)

Your Agent's Context Window Is RAM. Start Paging It.

Shridhar Shah — Wed, 05 Aug 2026 16:20:28 +0000

Treat the context window as a scarce cache: keep the working set resident, evict cold items to a store, and page them back on a fault. Locality keeps the fault rate low and nothing is ever lost.

TL;DR: A long-running agent keeps piling tool results and notes into its context window, but attention is a fixed budget — dilute it and both accuracy and cost degrade. Hard-capping the window and dropping the oldest items loses things the agent needs later. Operating systems solved this with demand paging: keep only the working set resident, evict cold items to a backing store, and page them back on a fault when referenced again. Agent sessions have strong locality, so the fault rate stays low and nothing is lost. In a runnable Go demo, "keep everything" balloons to 7.6× the window budget, while an LRU-paged window stays within budget at a 15% fault rate and zero information lost.

Mental model: your laptop runs apps that need far more memory than it has RAM, and you never notice, because the OS keeps the pages you're actively using in RAM and quietly parks the rest on disk — fetching them back the instant you touch them. An agent's context window is that RAM.

The problem: the window is small, the session is long

The context window is not memory — it's a cache, and a small one. Everything an agent does adds to it: a tool returns 30k tokens, a document gets pasted in, each step leaves notes. Two bad things happen as it fills. First, the attention tax: transformer attention is a fixed budget, and published measurements show accuracy sliding from ~95% to 60–70% as that budget is spread across more tokens. Second, cost and latency scale with tokens in the window, every single step.

The naive fix — cap the window and drop the oldest content — trades one failure for another. The moment a later step references something you dropped, it's gone: the agent silently recomputes it (paying for the tool call again) or, worse, hallucinates it.

The pattern: demand paging for the context window

Operating systems have run programs whose address space dwarfs physical RAM since the 1960s, using demand paging and Denning's working-set model. The mapping onto agents is direct:

OS	Agent
RAM	the resident context window (fixed budget)
Disk	a backing store (a file, a vector DB)
Page	a tool result / document / note
Page fault	a reference to something evicted → fetch it back

Keep only the working set resident. When the window is full and something new comes in, evict the least-recently-used items to the store (not the void). When a later step references an evicted item, that's a page fault — page it back in. Nothing is destroyed; it's just moved to a cheaper tier.

The eviction-to-fit and fault-on-miss logic is the whole pattern:

for _, item := range stream {
    if el, resident := pos[item]; resident {
        lru.MoveToFront(el)          // hit: already in the window
        continue
    }
    faults++                          // miss: page it back in from the store
    evictToFit(item)                  // make room by evicting LRU residents (to the store)
    tokens += size[item]
    pos[item] = lru.PushFront(item)
}

func evictToFit(incoming int) {
    for tokens+size[incoming] > budgetTokens && lru.Len() > 0 {
        victim := lru.Back().Value.(int)
        lru.Remove(lru.Back())
        delete(pos, victim)
        tokens -= size[victim]        // evicted to the backing store, not destroyed
    }
}

The result

Context Demand Paging — the context window is a cache, not infinite memory
  1500-step session, 60 distinct items, 8000-token window budget.

   keep everything resident   60,728 tokens   (7.6× the window — attention tax + cost blowup)
   demand paging (LRU)        7,997 tokens   (stays within budget)

   page faults                  226   (15% of references — the rest hit the working set)
   information lost               0   (faults page back in from the store — nothing destroyed)

Keeping everything resident hits 7.6× the window budget — the exact attention tax the budget was meant to prevent. Demand paging holds the window at budget, and because the agent keeps touching the same working set, only 15% of references miss — each a cheap re-read from the store, never a loss.

Reality check: the 7.6× and the 15% fault rate come from the locality model above — directional, not a benchmark, and your fault rate depends entirely on your workload's locality. What's independently real is the attention tax this targets (published long-context accuracy slides from ~95% to 60–70% as the window fills), and the direction of travel: 2026's Neural Paging formalizes the problem and learns the eviction policy instead of using the plain LRU shown here.

Why this is where 2026 is heading

As agents move from single answers to long-horizon work — hours of coding, multi-document research, sessions that outlive many context resets — context management stops being an afterthought and becomes the systems problem. The field is converging on the OS analogy fast: 2026's Neural Paging formalizes the "Context Paging Problem" and a learned page controller acting as a neural MMU, while practitioner write-ups describe context offloading, just-in-time context, and demand-paged working sets with measured fault rates. The transferable engineering idea: stop treating the window as memory you fill, and start treating it as a cache you manage — resident working set, cheap backing store, fault-driven recovery, eviction by utility.

How faithful is this demo?

It's the paging mechanism, not a real agent: "items" are integers with token sizes, the access stream is a locality model, and the store is assumed cheap and lossless. Real systems earn the two hard parts — what to evict (pure LRU by recency is weak for agents; pin by reference / working set, since a file read during planning stays relevant all session) and what the resident handle says (a pointer the model can't act on is worse than the payload). Start by offloading big tool results behind a summary+handle and paging them back only when a step needs them.

When not to use this

Short sessions. If the whole session fits the window with room to spare, paging is machinery you don't need.
No locality. If the agent touches items near-randomly, the fault rate climbs and you're just paying store round-trips — plain truncation or a summary may beat it.
A good summary beats the raw item. When a distilled note serves later steps as well as the original, consolidate instead of paging the full payload back and forth.

Try it

go run .   # standard library only

Sources & further reading

Papers & foundations

Neural Paging: Learning Context Management Policies for Turing-Complete Agents (arXiv 2603.02228) — formalizes the Context Paging Problem and a differentiable page controller acting as a "neural MMU."
Peter J. Denning — The Working Set Model for Program Behavior (1968) — why pinning the active working set beats evicting by recency.

Engineering write-ups

Context offloading: 3 patterns for AI agents — note-taking, sub-agent delegation, just-in-time retrieval; the attention-budget framing.
Demand paging for the AI context window — the OS mapping, fault rates, and "pin by reference, not recency."
Just-in-Time Context for AI Agents — keep the window full of pointers, load heavy content only when needed.
Context Offloading for AI Agents: writing tool results to disk — deferring the retrieval decision from write-time to read-time.

Your Load Balancer Is Throwing Away the KV Cache

Shridhar Shah — Wed, 05 Aug 2026 16:19:53 +0000

Round-robin is prefix-blind. KV-cache-aware routing sends requests to the replica that already prefilled their prefix — cutting time-to-first-token without hotspotting.

TL;DR: Before an LLM emits a token it must prefill the whole prompt into a KV cache — the expensive part. In 2026, requests share massive prefixes (system prompts, retrieved documents, the conversation so far). A round-robin load balancer is prefix-blind: it scatters those requests so every replica recomputes the same prefix. KV-cache-aware routing sends a request to the replica that already has its prefix warm. In a tiny Go demo it lifted cache hits from 47% to 85%, recomputed 66% fewer prefill tokens, and halved p50 TTFT — while keeping load far more even than naïve affinity. No GPU: latencies are simulated.

Mental model: a coffee shop that routes you to the barista who already knows your usual. Same regular → same barista → they start immediately. A round-robin queue sends you to a random barista who takes your whole order from scratch every time.

The problem: prefill is expensive and prefixes repeat

An LLM request has two phases: prefill (read the whole prompt into the KV cache) and decode (generate tokens). Prefill dominates time-to-first-token, and it scales with prompt length.

Here's the thing about agent traffic: prompts are mostly shared prefix. The same 2,000-token system prompt. The same retrieved document. The same conversation history, turn after turn. If a replica already prefilled that prefix, its KV cache can be reused and the prefix is essentially free. A round-robin balancer throws that away — it spreads requests evenly and blindly, so the hot prefix gets recomputed on all four replicas instead of once.

The pattern: route by prefix affinity, but balance load

Three routers, same request stream:

Round-robin — perfect load spread, zero cache awareness:

func roundRobin(_ []*replica, _ request, seq int) int { return seq % replicas }

Prefix-affinity — same prefix always lands on the same replica, so caches stay warm. But a hot prefix now hammers one replica:

func prefixAffinity(_ []*replica, req request, _ int) int {
    h := fnv.New32a()
    fmt.Fprintf(h, "%d", req.prefix)
    return int(h.Sum32()) % replicas
}

Cache-aware + balanced — the 2026 endpoint-picker idea (llm-d / Gateway API Inference Extension): prefer a replica that already has the prefix warm, but spill to the least-loaded replica if the warm one is slammed. Locality first, then balance:

// warm replica exists and isn't overloaded -> reuse its cache
if best != -1 && float64(reps[best].load) <= avg*overloadFactor {
    return best
}
// otherwise send it to the least-loaded replica
return leastLoaded(reps)

Time-to-first-token in the sim rewards a warm hit by skipping the prefix prefill:

prefill := req.suffixTokens          // the unique tail always needs prefill
if !r.has(req.prefix) {
    prefill += req.prefixTokens      // cold: pay to prefill the whole prefix
}
ttft := decodeBase + float64(prefill)*prefillPerTok

The result

KV-cache-aware routing — reuse warm prefixes instead of recomputing them
  2000 requests, 4 replicas, 40 distinct prefixes (traffic skewed to the hot few)

   router                    hit rate   prefill work    p50 TTFT   load skew
   round-robin                  46.6%   2,809,330 tok       61ms        1.0x
   prefix-affinity              86.0%     893,365 tok       30ms        1.8x
   cache-aware + balanced       84.8%     957,709 tok       30ms        1.2x

Round-robin has perfect balance (1.0x) but the worst cache behaviour. Blind affinity gets the cache hits but a 1.8x load skew — one replica does nearly twice the work of another. Cache-aware + balanced keeps ~85% of the hits and a gentle 1.2x skew. That's the sweet spot.

Reality check: these are simulated latencies, not a GPU benchmark — directional only. The tradeoff is real: production KV-aware routers (SGLang's RadixAttention, llm-d / Gateway API's endpoint picker) report large TTFT and throughput gains from exactly this locality-vs-load balance, and how big the win is depends entirely on how much prefix your traffic actually shares.

Why this is where 2026 is heading

The proven part: prefix / prompt caching is already GA across major providers and inference engines (vLLM, SGLang, TensorRT-LLM), and it's a real discount. The open question was always routing — caching only helps if the request lands where the cache lives.

Where it's heading: 2026 inference stacks make the router KV-cache-aware. The Gateway API Inference Extension's endpoint picker (EPP) and llm-d schedule replicas by KV-cache locality and load together — exactly the third router here. As context windows grow and agents replay long histories, where you route becomes as important as what model you route to.

How faithful is this demo?

It's a simulation of serving behaviour, not a GPU benchmark: latencies are a linear tokens × cost model and the cache is a per-replica LRU of whole prefixes. Real systems match on prefix blocks (shared leading tokens), evict at block granularity, and factor in queue depth and memory pressure — but the qualitative result (affinity wins big, blind affinity hotspots, locality-plus-balance is the sweet spot) is exactly what production routers target. The sharp edge to watch: a single very hot prefix can still overwhelm the one replica that caches it, so the router must be willing to trade cache reuse for load — spill to a colder replica under pressure — or you simply swap recompute cost for queueing latency.

When not to use this

Your prompts share little prefix. Short, unique requests get almost no cache reuse, so affinity routing buys nothing and just adds complexity.
You run a single replica. There's no routing decision to make.
Autoscaling is aggressive. Sticky, cache-aware routing fights elastic scale-in/out; you'll need to weigh cache locality against how often replicas churn.

Try it

go run .   # standard library only

Sources & further reading

Papers

Zheng et al. — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention: automatic KV-cache reuse via a radix tree + cache-aware scheduling.
Srivatsa et al. — Preble: Efficient Distributed Prompt Scheduling for LLM Serving (2024) — prefix-aware scheduling that places requests near their cached KV state.
GORGO: Maximizing KV-Cache Reuse While Minimizing Network Latency in Cross-Region LLM Load Balancing (arXiv 2602.11688) (2026) — treats prefix locality as one input to a latency cost model.

Engineering blogs & docs

Stop Streaming Tools Through Your LLM

Shridhar Shah — Wed, 05 Aug 2026 16:19:51 +0000

The 2026 shift from tool-calling to Code Mode: let the agent write one script instead of narrating fifty tool calls — and watch context tokens drop ~99%.

TL;DR: The classic agent loop loads every tool definition into the context window and pipes every intermediate result back through the model. Connect a few dozen tools and the context is full before the user even speaks. The 2026 move — Code Mode — is to let the agent write one short script that calls tools directly; bulk data stays in a sandbox and only the final answer comes back. In a tiny runnable demo it cut context from 36,781 tokens to 222 — a 99.4% reduction. Same answer, no API key.

Mental model: instead of reading a worker every page of 50 manuals and every row of a spreadsheet out loud, you hand them the 3 manuals the job needs and let them do the math at their own desk. You only get back the final answer — not the raw data.

The problem: tool-calling doesn't scale with tool count

Function/tool-calling is how most agents act today. It works beautifully with five tools. But the whole tool surface gets serialized into the context window on every request, and every intermediate result the model asks for is streamed back through the context too.

So a task like "count open support tickets per plan tier and save the report" looks like:

Load all 50 connected tool schemas into context.
Call list_tickets → 2,000 rows come back through the model.
Call get_customers → 400 rows come back through the model.
The model joins and counts these blobs in its head.
Call save_report.

The model paid for 50 schemas it mostly didn't need, and for 2,400 rows of raw data it only needed to aggregate, not read. Anthropic measured a real Google-Drive-to-Salesforce task at 150,000 tokens; Cloudflare hit ~1.17M tokens of tool definitions on a 2,500-endpoint API.

The pattern: write code that calls tools

Code Mode (Anthropic's "code execution with MCP", Cloudflare's "Code Mode") flips the loop:

Tools become a code API — a filesystem of typed functions, one per tool.
Progressive disclosure — the agent reads only the few signatures the task needs.
The model emits one script, not a stream of tool calls.
A sandbox runs it. Intermediate data lives in the sandbox; only the final result returns.

Here's the entire script the "model" writes in the demo — the 2,000 tickets and 400 customers are joined inside the sandbox and never touch the context:

tickets = list_tickets("open")            # 2000 rows: fetched and joined entirely in the sandbox
plan = {c["id"]: c["plan"] for c in get_customers()}
counts = {t: 0 for t in ("free", "pro", "enterprise")}
for tk in tickets:
    counts[plan[tk["customer_id"]]] += 1
rows = [{"tier": t, "open_tickets": counts[t]} for t in counts]
result = save_report("open_by_tier", rows)

The sandbox exposes only the tool API — no builtins, no imports — so the script can compose tools but can't reach the rest of the process:

api = {"list_tickets": list_tickets, "get_customers": get_customers, "save_report": save_report}
sandbox = {"__builtins__": {}, **api}
exec(script, sandbox)      # only the script text + final answer ever cross the context window

The result

Code Mode — write code that calls tools, don't stream tools through the model

  Task: count open tickets per plan tier over 2000 tickets / 400 customers.
  Tools connected to the agent: 50 (this task needs 3).

   classic tool-calling    36,781 context tokens   (all schemas + raw data pass through)
   code mode                  222 context tokens   (3 signatures + one script + answer)
   ------------------------------------------------
   context reduction        99.4%

  Same result either way: [free: 720, pro: 596, enterprise: 684]

The gap isn't a constant — it compounds with tool count and data size.

Reality check: the exact figure is from the toy model above — treat it as directional, not a benchmark. The shape is real and measured in production: Anthropic's Drive→Salesforce task dropped from ~150k to ~2k tokens, and independent reproductions land anywhere from 78% to 99.9% depending on tool count and data size.

Why this is where 2026 is heading

The proven part: LLMs are extremely good at writing code, and a code API is a far denser way to express "do these five things and combine the results" than five separate tool-call round trips. Anthropic, Cloudflare, and an independent study of MCP design choices all converge on the same finding — token usage becomes roughly constant in tool count because the model only reads what it opens.

Where it's heading: as agents connect to hundreds of MCP servers, "load everything up front" simply stops being viable, and the tool-calling layer moves onto the computer. Expect the sandbox — not the tool-call — to become the default action primitive, with tool schemas synced to a filesystem and disclosed on demand.

How faithful is this demo?

It's a minimal model, not a benchmark: "tokens" are a chars/4 proxy, the "model" writes a fixed script, and the sandbox is a restricted exec. Real systems generate the script with an LLM and isolate it far more aggressively (gVisor, containers, V8 isolates) — which is the pattern's main cost: you now run untrusted, model-written code, so sandboxing and limits are mandatory. The token economics, though, are exactly what production reports show.

When not to use this

You have a handful of tools and small results. If the whole tool surface and its data already fit comfortably in context, code-gen plus a sandbox is pure overhead — just call the tools.
The task is a single tool call. There's nothing to compose, so there's nothing to save.
You can't sandbox safely. Code Mode means executing model-written code; without real isolation (gVisor/containers/isolates) and resource limits, don't ship it.

Try it

python3 demo.py   # standard library only

Sources & further reading

Papers

Wang et al. — From Tool Orchestration to Code Execution: A Study of MCP Design Choices (arXiv 2602.15945) — benchmarks the "context-decoupled execution model" and shows token use becomes roughly constant in tool count.

Engineering blogs

Anthropic — Code execution with MCP: building more efficient AI agents (the 150k → 2k token result)
Cloudflare — Code Mode: the better way to use MCP and give agents an entire API in 1,000 tokens (typed API from MCP schemas, executed in a V8 isolate)
Code Execution With MCP: Cut Tool Tokens up to 98% (independent reproductions across tool counts)

AI Agents That Live Inside a Dreamed-Up World

Shridhar Shah — Tue, 21 Jul 2026 05:51:55 +0000

An agent watches a game, learns to hallucinate the next frame, then plays inside its own dream — but only the model that knows players react to each other stays true.

TL;DR: The hottest idea in agents right now: don't feed them the real world — let them dream it. An agent watches some footage, learns to hallucinate the next frame, and practices inside its own head. I built a tiny one, and found the catch: a dream only stays true if it knows the players react to each other. Runs on a laptop.

The world

Two players in an 11-cell corridor: a predator steps toward the prey, the prey steps away. Every move is a reaction. An agent watches random games, then closes its eyes and dreams 15 frames ahead, feeding each prediction back in as the next input. I built two dreamers from the exact same footage:

single-player — predicts each player from its own position alone
multiplayer — predicts both positions together

Training an agent inside its own learned dream goes back to Ha & Schmidhuber's World Models (2018); the open frontier is making that dream multiplayer — modeling agents reacting to each other, not just physics.

The 10-second version

% of the dream still matching reality, this many frames ahead:

frames ahead	1	3	5	10	15
single-player dream	11	0	0	9	0
multiplayer dream	100	100	100	100	100

The single-player dream falls apart almost immediately; the multiplayer one stays locked to reality. That gap is the whole point — and, as we'll see, it comes down to what each model is even able to represent.

How it works

The real world is one tiny rule — predator steps toward prey, prey steps away:

def real_next(p, q):                     # p = predator cell, q = prey cell
    return clamp(p + sign(q - p)), clamp(q - sign(q - p))

The agent never sees that rule. It just watches random games and counts what follows what. The only design choice is the shape of the memory it keeps — and that's the whole story:

multi[(p, q)][(p2, q2)] += 1   # multiplayer: "when BOTH are here, both go there"
one_p[p][p2]           += 1    # single-player: "when the predator is here, it goes there"
one_q[q][q2]           += 1    # single-player: "when the prey is here, it goes there"

To dream, it rolls its own predictions forward — feeding each guessed frame back in as the next input — and we check how long the dream keeps matching reality:

real = dream = start
for _ in range(15):
    real  = real_next(*real)   # what actually happens
    dream = model(*dream)      # the model predicting on its OWN last frame
    match += (dream == real)

Same footage, same loop. The only difference is whether the model remembered the two players together or apart.

Why single-player collapses

The prey moves because the predator moved. A single-player model looks at each player in isolation, so it structurally can't represent that reaction — its errors compound each frame until the dream is fiction. The multiplayer model conditions on both, so it captures the coupling. To be fair, this isn't a surprising empirical result so much as a demonstration: the corridor is deterministic, so the outcome really follows from what each model is allowed to see. That's exactly why the framing matters.

A dream you can act in is a superpower — an agent can practice a thousand risky moves for free. But a dream that forgets everyone else reacts to you isn't practice. It's a delusion.

Why it's exciting

The proven part: training agents inside a learned dream works — from World Models (2018) to DreamerV3 (2023) mastering 150+ tasks, and Genie (2024) learning playable worlds from video alone. Dreamed worlds let agents rehearse infinitely, safely, at zero real-world cost.

Where it's heading: those dreams are mostly single-agent today. The 2026 push is multiplayer — worlds where agents model each other. The lesson from this demo is the whole ballgame: model the reactions or the dream drifts. Get it right and agents can plan against each other entirely in imagination.

How faithful is this?

A real world model learns from raw pixels with a neural net, in a noisy, stochastic world. This is that mechanism stripped to its core — a frequency table over a tiny, deterministic game. It's built to make the intuition concrete, not to reproduce the papers; those (linked below) do the heavy, learned version.

Try it

git clone https://github.com/Shridhar-2205/secret-lives-of-agents
cd secret-lives-of-agents/03-dreamed-world && python demo.py

The series — The Secret Lives of AI Agents

Agents invent their own language
Agents build a culture on a decaying notepad
Agents that live inside dreamed-up worlds (you're here)

Shridhar Shah — Senior Software Engineer on the AI team at Cisco. GitHub · LinkedIn

Sources & further reading: Ha & Schmidhuber, World Models (2018) — the "train inside a dream" idea · Hafner et al., Mastering Diverse Domains through World Models (DreamerV3, 2023) · Bruce et al., Genie: Generative Interactive Environments (2024).

I Gave 3 AI Agents a Decaying Notepad and They Built a Culture

Shridhar Shah — Tue, 21 Jul 2026 05:51:51 +0000

One shared memory that keeps fading, three minimal agents, no boss — and a story that outlives every note that carried it.

TL;DR: Give a few bare-bones agents nothing but a shared notepad that constantly fades, and they spontaneously build a culture — a story they keep alive together, long after any single note has decayed to nothing. No coordinator, one tiny rule. Runs on a laptop.

The setup

Three agents share one whiteboard. Every tick, everything on it fades; a note left alone dies in ~4 ticks. Each tick an agent scribbles a noisy observation, and a swarm agent also reinforces whatever's currently strongest. That's the whole rule. It's stigmergy — how termites build cathedrals by reacting to each other's mud (Grassé, 1959) — and the same dynamic that lets populations of agents settle on shared conventions (Perez et al., 2024).

The 10-second version

	❌ Each on its own	✅ The swarm
A shared story exists	11% of ticks	100%
Same story tick-to-tick	3%	99%

How it works

No agent has a memory of its own. The only shared state is pad — a dict of note -> strength that fades every tick. Each tick, three agents scribble a noisy observation, and the "culture" rule is a single line: reinforce whatever the group is already backing.

pad = {}                                    # the shared, fading notepad

for tick in range(100):
    for note in list(pad):                  # 1. everything decays; faint notes are forgotten
        pad[note] *= 0.5
        if pad[note] < 0.1:
            del pad[note]

    for _ in range(3):                       # 2. three agents act this tick
        obs = random.choice(FACTS)           #    each jots a weak, noisy observation
        pad[obs] = pad.get(obs, 0) + 0.3
        if imitate:                          #    the ENTIRE culture rule:
            leader = max(pad, key=pad.get)   #      reinforce what the group already backs
            pad[leader] += 1.0

Flip imitate off and you get pure noise. Flip it on and a single story takes over — chosen by no one, kept alive by everyone.

The part that got me

Tick 0:   group rallies around  "water down"
Tick 100: group still holds     "water down"
   ...but that original note faded to ~8e-31 within a few ticks.
   It survived only because the agents rewrote it 300 times.

The story is 100 ticks old; the note that started it has been gone since tick ~4. What persists isn't any note — it's the meaning, re-inscribed by the group. It's the Ship of Theseus: no original plank left, yet the ship sails on. Reseed and a different story wins — the agreement is what's real.

Why it matters

The proven part: stigmergy — coordination through traces left in a shared medium — is old biology (Grassé, 1959), and populations of LLM agents have already been shown to form and transmit shared conventions (Perez et al., 2024). This demo distills it to a single rule on a decaying pad.

Where it's heading: we keep trying to fix agent memory with bigger storage. This points the other way — a group of forgetful agents in 2026 can hold knowledge no single one could, just by reminding each other. Persistence becomes a property of the society, not the context window.

These agents can't remember much alone. Together, on a whiteboard that won't stop erasing itself, they keep a story alive as long as they care to. That's culture.

How faithful is this?

This is stigmergy distilled to one reinforcement rule on a decaying dictionary — not LLM agents reasoning in language. It shows the dynamic (a shared story outliving the notes that carried it); the cited work studies it with real populations of generative agents.

Try it

git clone https://github.com/Shridhar-2205/secret-lives-of-agents
cd secret-lives-of-agents/02-emergent-culture && python demo.py

The series — The Secret Lives of AI Agents

Agents invent their own language
Agents build a culture on a decaying notepad (you're here)
Agents that live inside dreamed-up worlds

Shridhar Shah — Senior Software Engineer on the AI team at Cisco. GitHub · LinkedIn

Sources & further reading: Stigmergy (Grassé, 1959) — the biology of indirect coordination · Park et al., Generative Agents: Interactive Simulacra of Human Behavior (2023) · Perez et al., Cultural Evolution in Populations of Large Language Models (2024).

I Watched Two AI Agents Invent Their Own Language

Shridhar Shah — Tue, 21 Jul 2026 05:46:59 +0000

No shared words, no dictionary — just two agents that negotiate a private code from scratch and hit ~97%.

TL;DR: Give two AI agents a reason to coordinate and they'll make up their own language — one we never designed. I built the tiniest version: two agents, zero shared words, and from "did we understand each other?" alone they invent a private code and hit ~97%. Runs on a laptop, no API key.

The game

A sender sees a secret object (say 🍎) and holds up one of a few random shapes: ◇ △ ○ ☆ □. A receiver sees only the shape and guesses the object. Right guess → both remember that pairing. No dictionary, no translator. This is the classic Lewis signaling game — the cleanest way to watch language appear from nothing.

The 10-second version

	❌ No memory	✅ Remembers
After 2,000 rounds	~56% (chance)	~97%
A language formed?	no	yes

Blind guess = 20%. Watch it crystallize:

round    1:   0%
round  500:  94%
round 2000:  97%   apple=◇  banana=□  cherry=△  grape=☆  lemon=○

How it works

There's no neural network here — just two habit tables per agent and one rule: when a guess lands, both sides strengthen the exact link they just used. That's the entire learning algorithm.

class Agent:
    def __init__(self):
        # habit tallies, both directions, all starting at zero
        self.obj_to_sym = {o: {s: 0.0 for s in SYMBOLS} for o in OBJECTS}
        self.sym_to_obj = {s: {o: 0.0 for o in OBJECTS} for s in SYMBOLS}

    def pick(self, habits):
        # mostly reuse the strongest habit; occasionally explore something new
        if random.random() < self.explore:
            return random.choice(list(habits))
        return max(habits, key=habits.get)

    def say(self, obj):   return self.pick(self.obj_to_sym[obj])   # object -> symbol
    def guess(self, sym): return self.pick(self.sym_to_obj[sym])   # symbol -> object

    def reward(self, obj, sym):          # a win: reinforce the SAME link on both maps
        self.obj_to_sym[obj][sym] += 1
        self.sym_to_obj[sym][obj] += 1

And the whole game is just:

obj = random.choice(OBJECTS)          # sender sees a secret object
sym = sender.say(obj)                 # sender picks a symbol for it
if receiver.guess(sym) == obj:        # receiver decodes it — did they match?
    sender.reward(obj, sym)           # yes: both lock in that pairing
    receiver.reward(obj, sym)

That's it. Run it a few thousand times and a clean one-to-one code falls out. Reseed and they invent a different code (apple=☆ …) — arbitrary, but agreed. And memory is what makes it stick: agents that only recall the last few rounds never settle, exactly as this referential game — introduced by Lazaridou, Peysakhovich & Baroni (2017) — predicts.

Why it's exciting (and a little eerie)

The proven part: two neural agents reliably invent a working code from scratch — shown since Lazaridou et al. (2017) and surveyed in Lazaridou & Baroni (2020). This demo just strips the idea to 100 lines so you can watch it happen.

Where it's heading: the systems we're shipping in 2026 are LLM swarms that talk to each other nonstop. A private, compressed code lets them coordinate faster and cheaper than plain English — a real efficiency win. The flip side: if agents settle on a protocol we didn't design, we may not be able to read what they tell each other.

A language is just a bet that a symbol means the same thing on both ends. These agents make that bet round by round, with nobody refereeing.

How faithful is this?

This is the classic referential game in ~100 lines — reinforcement over simple habit tables, not a neural network. It captures the mechanism (a shared code emerging from feedback alone); the papers below scale the same idea to real networks and richer, compositional languages.

Try it

git clone https://github.com/Shridhar-2205/secret-lives-of-agents
cd secret-lives-of-agents/01-invented-language && python demo.py

The series — The Secret Lives of AI Agents

Agents invent their own language (you're here)
Agents build a culture on a decaying notepad
Agents that live inside dreamed-up worlds

Shridhar Shah — Senior Software Engineer on the AI team at Cisco. GitHub · LinkedIn

Sources & further reading: Lewis, Convention (1969) — the original signaling game · Lazaridou, Peysakhovich & Baroni, Multi-Agent Cooperation and the Emergence of (Natural) Language (ICLR 2017) · Havrylov & Titov, Emergence of Language with Multi-agent Games (NeurIPS 2017) · Lazaridou & Baroni, Emergent Multi-Agent Communication in the Deep Learning Era (2020, survey).

Agents Are Learning to Write Their Own SKILL.md Files

Shridhar Shah — Sat, 27 Jun 2026 21:53:22 +0000

The Agent Skills open standard today, and the 2026 research on agents that write their own skills.

TL;DR: I've been using skills in Claude Code daily, and one question stuck with me: what happens when the agent writes them itself? Quick background: "Agent Skills" (late 2025) are a dead-simple way to teach an agent a task — a folder with a SKILL.md file of Markdown instructions, now an open standard. The wild part is what's coming next: agents that write their own skills. I built a demo where an agent solves a task the hard way once, saves a real SKILL.md, and then reuses it — cutting its total effort almost in half. No API key.

First, what's a "skill"?

If you've used Claude Code or similar tools lately, you've probably seen SKILL.md files. The idea is refreshingly low-tech. A "skill" is just a folder with a Markdown file that says how to do something:

---
name: csv-to-markdown
description: Turn comma-separated text into a Markdown table. Use when the input looks
  like CSV and the user wants a table.
---

# CSV to Markdown

## Instructions
Split the text into rows on newlines and columns on commas. Make the first row the
header, add a `---` divider row, then format every row as `| a | b | c |`.

That's it. No SDK, no config. Anthropic introduced this in October 2025 and then published it as an open standard (agentskills.io) in December 2025, so the same skill folder now works across ~30+ different agent tools (Claude Code, Cursor, Copilot, and more).

The full rules are short (agentskills.io/specification): the only required fields are name (1–64 chars, lowercase-with-hyphens, and it must match the folder name) and description (≤1024 chars, saying what it does and when to use it). Everything else — license, metadata, compatibility, allowed-tools — is optional. That's the whole spec. The SKILL.md files my demo writes follow it to the letter, so they'd load unmodified in any compatible CLI.

The clever trick: progressive disclosure

Here's the smart part. If you just dumped 50 skills' worth of instructions into the agent's context, you'd fill it up and leave no room for actual work. So skills load in stages:

Always loaded: just the name and one-line description of every skill (tiny).
Loaded only when it matches: the full instructions, once a task actually needs them.
Loaded only if referenced: extra files or scripts the skill bundles.

So the agent can have hundreds of skills installed and barely pay for it — it only reads the short descriptions until one matches, then pulls in the details. My demo shows the math: to use 1 skill out of 3 installed, loading everything costs ~1500 "tokens"; the SKILL.md way costs ~560. That gap gets huge as your library grows.

This is also why people say skills and MCP are teammates, not rivals: MCP is how an agent connects to tools; a skill is how an agent knows the procedure for using them.

The frontier: agents that write their own skills

Today, humans write SKILL.md files. The 2026 research is about agents that write their own — and get better over time as their skill library grows. This goes back to Voyager (2023), an agent that played Minecraft and saved working code as reusable skills, getting dramatically faster at the game. The new wave makes it general:

MUSE-Autoskill (2026) treats a skill as a living asset with a full lifecycle — create it, give it its own memory file, manage it, test it, and refine it. Each skill even keeps a .memory.md of notes about itself.
Memento-Skills (2026) stores skills as Markdown files that double as the agent's evolving memory, and turns task failures into new skills automatically.
Skill-Pro (2026) defines a skill as "when to use it + how to do it + when to stop," and only keeps a new skill if it passes a quality gate — so the library improves instead of filling up with junk.

The common thread: solve it once, save the recipe, reuse it forever — and let the collection get smarter on its own.

📄 The "this is the future" link: Anthropic's own writeup, Equipping agents for the real world with Agent Skills, and the open standard at agentskills.io. For the research direction, MUSE-Autoskill (arXiv:2605.27366) and Skill-Pro (arXiv:2602.01869) are the clearest reads on agents that grow their own skill libraries.

You can do this today in the Claude Code CLI

This isn't theoretical — the exact pattern from my demo already ships in coding CLIs. In Claude Code, a skill is just a folder under .claude/skills/ in your repo:

# Anywhere in your project — drop a skill in and the CLI auto-discovers it
mkdir -p .claude/skills/csv-to-markdown
$EDITOR .claude/skills/csv-to-markdown/SKILL.md   # same SKILL.md format as my demo

Now the agent loads only that skill's one-line description until a task matches — then pulls in the full instructions (that's progressive disclosure doing its job). Type /skills inside the CLI to see what's loaded.

The best part: because it's an open standard, the same folder works unmodified across tools. You're not locked in:

Claude Code — Anthropic's CLI, where the format started.
opencode — a popular open-source terminal agent.
Goose — Block's open-source agent.
Plus Cursor, GitHub Copilot, and 30+ others.

Write the skill once, use it everywhere. The future bit my demo points at: instead of you hand-writing that file, the agent writes it for itself after solving the task the first time — and from then on, your repo quietly accumulates a library of skills your agent earned.

The 10-second version (my demo)

Same stream of 7 tasks. "Cost" is how much effort each one took.

	No-skills agent	Skill-writing agent
What it does	re-solves everything from scratch	learns a task once, saves a `SKILL.md`, reuses it
Total cost	35	19
Both correct?	7/7	7/7

[5] csv-to-markdown  learned it and wrote SKILL.md
[5] slugify          learned it and wrote SKILL.md
[1] csv-to-markdown  reused skill 'csv-to-markdown'   ← cheap now
[5] extract-emails   learned it and wrote SKILL.md
[1] slugify          reused skill 'slugify'
[1] csv-to-markdown  reused skill 'csv-to-markdown'
[1] extract-emails   reused skill 'extract-emails'

It writes real SKILL.md files into a ./skills folder you can open. The first time it sees a task it pays full price; after that, it finds its own saved skill and reuses it for cheap.

Why this matters

Two big reasons engineers should care:

Agents stop repeating themselves. Right now most agents re-derive the same thing over and over, paying for it every time. A skill library means "figure it out once, then it's free" — like a teammate who writes things down instead of relearning them daily.
A whole new ecosystem. There are already 65,000+ shared skills and a scramble to build "the npm of agent skills" — registries and marketplaces where you install a skill like a package. Skills are becoming a unit of shareable expertise: a senior engineer's know-how, packaged in a folder, that any agent can pick up.

Tools tell an agent what it can do. Skills tell it how to do things well — and soon, agents will write that part themselves, and trade it with each other.

Try it

git clone https://github.com/Shridhar-2205/living-software
cd living-software/06-agent-skills
python demo.py
cat skills/csv-to-markdown/SKILL.md   # a skill the agent wrote itself

Honest note: this is a POC. Real systems decide when a new skill is worth saving, test it, and refine it over time (that's exactly what the 2026 papers above tackle). Mine keeps that part simple so the core idea — learn once, save a SKILL.md, reuse it — is easy to see.

The rest of the series — Toward Living Software

I built an AI agent that rewrites its own code
Do AI agents need to sleep?
Can an AI agent pass the Sally-Anne test?
An AI agent that gets curious on its own
How do you trust an AI agent with your money?
Agents that write their own SKILL.md files (you're reading it)

Shridhar Shah — Senior Software Engineer on the AI team at Cisco. Part 6 of Toward Living Software.

GitHub · LinkedIn

Sources: Anthropic, "Equipping agents for the real world with Agent Skills" (2025) and the Agent Skills open standard (agentskills.io); Voyager (arXiv:2305.16291); MUSE-Autoskill (arXiv:2605.27366); Memento-Skills (arXiv:2603.18743); Skill-Pro (arXiv:2602.01869); MemSkill (arXiv:2602.02474).