DEV Community: Veera Ravindra Divi

How to Build an Evaluation Harness for Your AI Agent (So It Doesn't Break in Production)

Veera Ravindra Divi — Wed, 08 Jul 2026 01:21:27 +0000

The agent passed every test I threw at it by hand. Then a user asked it to "book the cheaper flight," it happily called book on the wrong flight ID, and nobody caught it for three days because the demo I kept running never once asked for the cheaper flight.

That is the trap. When you test an agent by running it yourself, the demo is the test — you type the happy-path prompt, watch it work, and ship. But your users don't type your happy path, and your model provider silently ships a new checkpoint next Tuesday. An AI agent evaluation harness is the thing that tells you your agent actually works, and — more importantly — that last week's prompt tweak didn't quietly break the flight-booking flow you're not looking at.

This tutorial builds one. By the end you'll have a small, runnable Python repo that grades an agent on real tasks, checks that it called the right tools with the right arguments, judges the fuzzy stuff with an LLM, and fails your CI build when a change makes the agent worse. No product pitch, no framework lock-in — the pattern works with LangChain, LangGraph, raw MCP, or whatever you wired together yourself.

What You'll Build: A Minimal AI Agent Evaluation Harness

We'll test a tiny travel-support agent with three tools — search, get_weather, and book — because tool calls are checkable in a way that free-form chat isn't. Along the way we'll build four graders. I call them the Four Graders, and they map cleanly onto what you're actually afraid of:

Grader	Question it answers	How it works
Truth	Did the agent call the right tool with the right arguments?	Deterministic code check
Path	Did it take a sane route, or loop and thrash?	Trajectory / step-efficiency check
Judge	Did it actually accomplish the goal when there's no single right string?	LLM-as-judge with a rubric
Gate	Did this change make the agent worse than last week?	Aggregate score + CI regression gate

Truth and Path are cheap, fast, and deterministic. Judge is flexible but needs calibration. Gate is what turns all of it into a safety net instead of a science project.

Prerequisites

Python 3.10+ and comfort with pip install.
pytest (pip install pytest). For the LLM-as-judge step, any provider SDK — we'll use openai as the example, but the call is one swappable function, so Anthropic, a local model, or anything with a chat endpoint works.
An agent you can call from Python that returns what tools it called and its final answer. If your framework doesn't expose that yet, capture it — most do via callbacks or a trace/span export.
No prior eval experience. This is the piece I wish existed when I shipped an agent and realized "it works on my machine" is not a test strategy.

Why Manual Testing Lies

Three things make agents uniquely dangerous to eyeball-test.

The demo is the test. You built the agent by running prompts and fixing what broke. So the prompts you run are, by construction, the ones it already handles. You are grading the exam you wrote the answer key for.

Non-determinism. Same prompt, same model, different output. Temperature, a re-ranked retrieval, a provider-side model update — any of these can flip a passing run into a failing one tomorrow with zero code change on your side. A single manual run tells you almost nothing about the distribution of behavior.

Silent regressions. This is the killer. You tighten a system prompt to fix formatting, and it turns out that phrasing also nudged the model to skip the get_weather check before booking. Nothing errors. Nothing logs red. The agent just gets 8% worse at a task you weren't watching, and you find out from a user. An eval harness is a smoke detector for exactly this.

The fix isn't "test more by hand." It's a small set of tasks, graded automatically, run on every change.

The Three Altitudes of an Agent Eval

Before we grade anything, know where you're grading. An agent run is a tree of spans — the top-level task, the reasoning steps and tool calls underneath, and the individual components (a retriever, a single tool, a sub-agent) at the leaves. You can evaluate at three altitudes, and good harnesses use all three (Confident AI lays these out well):

End-to-end — treat the whole agent as a black box: given this input, did the final outcome succeed? This is your Judge grader.
Trajectory — look at the plan: the tool calls, retries, and handoffs that produced the result. Did it take a sane path? This is your Truth and Path graders.
Component — zoom into one leaf: is this retriever or this tool the bottleneck? You reach for this when a higher altitude tells you something's wrong and you need to find where.

Start end-to-end and trajectory. Drop to component-level only when you need to localize a failure. Here's the mapping onto a span tree:

[end-to-end]     Task: "Book AA123 Fri unless it's raining in Denver"
                 │
[trajectory]     ├─ get_weather(city="Denver")     ← Path + Truth grade here
                 ├─ book(flight_id="AA123", ...)    ← Truth grades args here
                 └─ final_answer: "Booked AA123..."  ← Judge grades here
                     │
[component]          └─ weather_tool latency / retriever recall  ← only when localizing

Step 1: Build a Golden Dataset From Real Failures

Everything downstream depends on this. Your dataset is a small set of tasks with known-good expectations. Two rules, both from Anthropic's eval guide:

Start small and source from real failures. "20-50 simple tasks drawn from real failures is a great start." Don't wait for a thousand-row dataset. Every bug report and every manual check you've done is a test case — turn them into rows.
Make pass/fail unambiguous. Design each task so "two domain experts would independently reach the same pass/fail verdict." If you can't decide whether a run passed, neither can a grader.

Here's a starter dataset for our travel agent. This is a plain Python list — put it in evals/dataset.py:

# evals/dataset.py — the golden set. Each row is one checkable task.
GOLDEN = [
    {
        "task_id": "weather-gate-book",
        "input": "Book flight AA123 for Friday, but only if it isn't raining in Denver.",
        "expect_tools": [
            {"name": "get_weather", "args": {"city": "Denver"}},
            {"name": "book", "args": {"flight_id": "AA123", "date": "Friday"}},
        ],
        "success_contains": "booked",
    },
    {
        "task_id": "search-before-book",
        "input": "Find the cheapest flight to Seattle next Monday and book it.",
        "expect_tools": [
            {"name": "search", "args": {"destination": "Seattle"}},
            {"name": "book", "args": {}},  # empty = don't care about specific args
        ],
        "success_contains": "booked",
    },
    {
        "task_id": "no-book-when-raining",
        "input": "Book AA200 Friday only if Denver is clear. (It is raining.)",
        "expect_tools": [{"name": "get_weather", "args": {"city": "Denver"}}],
        "forbid_tools": ["book"],  # booking here is a hard failure
        "success_contains": "did not book",
    },
    {
        "task_id": "weather-only",
        "input": "What's the weather in Denver?",
        "expect_tools": [{"name": "get_weather", "args": {"city": "Denver"}}],
        "forbid_tools": ["book", "search"],
        "success_contains": "denver",
    },
    {
        "task_id": "ambiguous-needs-search",
        "input": "Book me something cheap to the coast.",
        "expect_tools": [{"name": "search", "args": {}}],
        "success_contains": "options",  # should clarify/search, not blind-book
    },
]

What you should see: five tasks, each with expected tool calls, some with forbid_tools (calls that must not happen) and a success_contains string for the final answer. Notice no-book-when-raining and ambiguous-needs-search — those are the sneaky-failure cases a happy-path demo never exercises. That's the whole point.

We also need a uniform shape for what the agent did. Put this in evals/types.py:

# evals/types.py — the trace shape every grader reads.
from dataclasses import dataclass

@dataclass
class ToolCall:
    name: str
    args: dict

@dataclass
class AgentRun:
    task_id: str
    tool_calls: list        # list[ToolCall], in order
    final_answer: str

Your job when integrating: make your real agent emit an AgentRun. LangChain/LangGraph expose this through callbacks or the run tree; MCP servers log tool invocations; a hand-rolled loop already has the list. For the tutorial, we'll grade sample runs directly so every block below is runnable without an API key.

Step 2: Grader 1 — Truth (Tool-Call Correctness Check)

Truth answers the cheapest, highest-value question: did the agent call the right tools with the right arguments? This is a code-based grader — fast, free, objective, reproducible. No LLM involved. Put it in evals/graders.py:

# evals/graders.py — Truth: are the required tool calls present & correct?
def grade_truth(run, task):
    actual = run.tool_calls

    # 1. Forbidden calls are a hard fail.
    for c in actual:
        if c.name in task.get("forbid_tools", []):
            return 0.0, f"called forbidden tool: {c.name}"

    # 2. Every expected call must appear with matching args.
    for exp in task["expect_tools"]:
        hit = next(
            (c for c in actual
             if c.name == exp["name"]
             and all(c.args.get(k) == v for k, v in exp["args"].items())),
            None,
        )
        if hit is None:
            return 0.0, f"missing/incorrect: {exp['name']}({exp['args']})"

    return 1.0, "all expected tool calls present with correct args"

Two distinct checks are hiding in there, and it's worth naming them because the eval literature does: tool correctness (was the right tool called at all?) and argument correctness (were the parameters right?). An agent that calls book — correct tool — with the wrong flight_id — wrong argument — is a bug that costs real money. grade_truth catches both. An empty args dict means "I only care that the tool was called, not how," which is how you keep tests from being brittle.

Let's run it on a good run and a buggy one:

# try_truth.py — runnable: grade a good run and a broken one.
from evals.types import AgentRun, ToolCall
from evals.dataset import GOLDEN
from evals.graders import grade_truth

task = GOLDEN[0]  # weather-gate-book

good = AgentRun("weather-gate-book",
    [ToolCall("get_weather", {"city": "Denver"}),
     ToolCall("book", {"flight_id": "AA123", "date": "Friday"})],
    "Booked AA123 for Friday.")

buggy = AgentRun("weather-gate-book",
    [ToolCall("get_weather", {"city": "Denver"}),
     ToolCall("book", {"flight_id": "AA999", "date": "Friday"})],  # wrong ID
    "Booked AA999 for Friday.")

print(grade_truth(good, task))
print(grade_truth(buggy, task))

What you should see:

(1.0, 'all expected tool calls present with correct args')
(0.0, "missing/incorrect: book({'flight_id': 'AA123', 'date': 'Friday'})")

The buggy run booked the wrong flight and the grader caught it deterministically, in milliseconds, for zero cost. This one grader would have saved me those three days.

Step 3: Grader 2 — Path (Trajectory / Step Efficiency)

Truth tells you the agent eventually did the right things. Path tells you it didn't flail to get there. An agent that calls search six times, loops on get_weather, and burns 40 seconds and $0.30 to book one flight is a production incident waiting to happen — slow, expensive, and one retry-loop away from a runaway bill. The metric here is step efficiency: did it avoid unnecessary steps, retries, and loops?

Add to evals/graders.py:

# evals/graders.py (cont.) — Path: efficient trajectory, no thrashing.
def grade_path(run, task):
    expected = len(task["expect_tools"])
    actual = len(run.tool_calls)

    # Duplicate identical calls = looping/thrashing.
    seen, duplicates = set(), 0
    for c in run.tool_calls:
        key = (c.name, tuple(sorted(c.args.items())))
        if key in seen:
            duplicates += 1
        seen.add(key)

    if duplicates:
        return 0.0, f"{duplicates} duplicate tool call(s) — looping"
    if actual > expected * 2:  # generous budget: 2x the minimal path
        return 0.0, f"inefficient: {actual} calls for a {expected}-call task"
    return 1.0, f"efficient: {actual} calls, no duplicates"

The expected * 2 budget is deliberately generous — real agents legitimately take an extra step to clarify or recover. You're not demanding the minimal path, you're catching pathological paths. Tune the multiplier to your agent; the duplicate-call check is the part that catches infinite-loop regressions before your bill does.

Run it against a thrashing agent:

# grade a run that loops on the same weather call twice
from evals.types import AgentRun, ToolCall
from evals.dataset import GOLDEN
from evals.graders import grade_path

looper = AgentRun("weather-gate-book",
    [ToolCall("get_weather", {"city": "Denver"}),
     ToolCall("get_weather", {"city": "Denver"}),   # duplicate
     ToolCall("book", {"flight_id": "AA123", "date": "Friday"})],
    "Booked AA123.")

print(grade_path(looper, GOLDEN[0]))

What you should see:

(0.0, '1 duplicate tool call(s) — looping')

Step 4: Grader 3 — Judge (LLM-as-Judge With a Rubric)

Some things you can't check with an ==. "Did the agent's final answer actually accomplish the user's goal?" often has no single correct string. That's where a model-based grader earns its keep: flexible, handles open-ended tasks, scores against a rubric. The cost is that it's non-deterministic and — read this twice — must be calibrated against human labels before you trust it. More on that caveat below.

The key design move: keep the LLM call in one swappable function so the grader isn't married to any provider. Add to evals/graders.py:

# evals/graders.py (cont.) — Judge: LLM scores goal completion via a rubric.
import json

RUBRIC = """You are a strict grader for an AI agent's answer.
Score 1 only if the answer fully accomplishes the user's goal; else 0.

User task: {task}
Agent's final answer: {answer}

Respond with ONLY a JSON object: {{"score": 0 or 1, "reason": "<one sentence>"}}"""

def grade_judge(run, task, chat):
    prompt = RUBRIC.format(task=task["input"], answer=run.final_answer)
    data = json.loads(chat(prompt))         # chat() is provider-agnostic
    return float(data["score"]), data["reason"]

chat is any function that takes a prompt string and returns the model's text. Here's the OpenAI version — swap the body for Anthropic, Gemini, or a local model and nothing else changes:

# evals/judge_client.py — one swappable function. Works with any provider.
def openai_chat(prompt: str) -> str:
    from openai import OpenAI
    client = OpenAI()  # reads OPENAI_API_KEY from env
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,                      # determinism where you can get it
        response_format={"type": "json_object"},
    )
    return resp.choices[0].message.content

What you should see when you call grade_judge(run, task, openai_chat): a tuple like (1.0, 'The agent booked the requested flight after checking weather.'). Set temperature=0 and force JSON output — you want the judge as reproducible as a stochastic thing can be.

Now the caveat I promised, because skipping it is how teams fool themselves: an unvalidated LLM judge is a vibe with a number attached. Before you trust it in CI, hand-label 20–30 runs yourself, run the judge on the same runs, and check agreement. If the judge disagrees with you often, fix the rubric — don't ship the judge. Anthropic is blunt about this: model-based graders "require calibration against human judgment," and human labels are the gold standard you calibrate against. Use the judge for the fuzzy end-to-end verdict; lean on Truth and Path — which need no calibration — for everything you can check deterministically.

Step 5: Grader 4 — Gate (Wrap It in pytest + CI)

Here's where it stops being a script and becomes a safety net. The Gate does two jobs: run every grader over every task, and fail the build when the aggregate score drops below a baseline. That last part is the entire reason to do any of this — it's what catches the silent regression.

One more principle before the code: isolate your grading. Anthropic's rule is that "each trial should be 'isolated' by starting from a clean environment" — no shared state bleeding between runs and inflating (or tanking) your scores. In practice: fresh fixtures per task, no reused sessions, no test depending on the one before it. pytest gives you this for free if you don't fight it.

Put this in evals/test_agent.py:

# evals/test_agent.py — the gate. Runs in CI on every PR.
import pytest
from evals.dataset import GOLDEN
from evals.graders import grade_truth, grade_path
from evals.runner import run_agent   # your agent → AgentRun

DETERMINISTIC = {"truth": grade_truth, "path": grade_path}
BASELINE = 0.90   # ship bar: aggregate score must stay >= this

@pytest.mark.parametrize("task", GOLDEN, ids=lambda t: t["task_id"])
def test_each_task_passes_deterministic_graders(task):
    run = run_agent(task["input"])          # clean run per task = isolation
    for name, grader in DETERMINISTIC.items():
        score, reason = grader(run, task)
        assert score == 1.0, f"[{name}] {task['task_id']}: {reason}"

def test_aggregate_no_regression():
    scores = []
    for task in GOLDEN:
        run = run_agent(task["input"])
        per_task = [g(run, task)[0] for g in DETERMINISTIC.values()]
        scores.append(sum(per_task) / len(per_task))
    avg = sum(scores) / len(scores)
    assert avg >= BASELINE, f"REGRESSION: {avg:.2f} < baseline {BASELINE}"

What you should see: run pytest evals/ -q. Each golden task shows up as a named test (weather-gate-book, no-book-when-raining, …), and a failing task prints exactly which grader failed and why — [truth] weather-gate-book: missing/incorrect: book({'flight_id': 'AA123'...}). Break your prompt on purpose and watch the aggregate test go red with the score. That red X is the thing you were missing.

I kept the Judge grader out of the hard assert gate on purpose: because it's non-deterministic, treat its score as a monitored metric (log it, alert on drops) rather than a pass/fail that flakes your build. Deterministic graders gate; the judge informs.

Now wire it into CI so it runs on every pull request. GitHub Actions, .github/workflows/agent-evals.yml:

# .github/workflows/agent-evals.yml — block the merge on a regression.
name: agent-evals
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest openai
      - run: pytest evals/ -q
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

What you should see: open a PR that changes the prompt or bumps the model, and the agent-evals check runs. If the agent got worse, the check is red and the merge is blocked — same as a failing unit test. Your agent is now gated in CI like normal software, which is the whole point.

Conclusion: Stop Shipping Agents on Vibes

You now have the four pieces that separate "it worked in the demo" from "I know it works": a golden dataset drawn from real failures, a Truth grader that catches wrong tools and wrong arguments deterministically, a Path grader that catches looping and thrash, a Judge for the fuzzy goal-completion verdict, and a Gate that fails your build the moment a change makes the agent worse.

None of it is exotic. It's the same instinct that made you write unit tests for the rest of your code — you just didn't have a shape for it when the thing under test was a non-deterministic agent calling tools. Now you do.

Where to go next:

Grow the dataset. Every production incident becomes a new golden row. Aim for the 20–50 real-failure tasks Anthropic recommends before you trust the aggregate score.
Calibrate the judge. Hand-label a batch, measure agreement, fix the rubric until the judge matches you. Then — and only then — start trusting its number.
Add component-level evals when a failure needs localizing: grade the retriever's recall or a single tool in isolation.
Track the score over time. A dashboard of aggregate score per commit turns your harness from a gate into an early-warning system.

Your agent looks great when you run it by hand. That was never the question. The question is whether it still works after the next prompt tweak, the next model update, and the next user who does something you didn't demo. Build the harness, and you'll actually know.

MCP just deleted the session: what the July 28 spec breaks in your server

Veera Ravindra Divi — Mon, 29 Jun 2026 02:32:20 +0000

The July 28 MCP rewrite goes stateless. It fixes your load balancer and does nothing for the bill you actually pay: tool-schema context bloat.

If you shipped an MCP server in the last year, you built it around a lie the spec told you: that a connection is a thing you can hold onto. On July 28, 2026, that assumption gets deleted.

The 2026-07-28 release candidate locked on May 21, and it goes stateless. No more initialize/initialized handshake. No more Mcp-Session-Id header pinning a client to one process. The whole "open a session, keep it warm, route everything back to the same box" model that every tutorial taught you is now legacy. (It's still an RC until July 28, so treat the wire details below as the candidate, not the carved-in-stone final — but the SDK teams are already migrating against it.)

I want to make two arguments at once. First: going stateless is the right call, and it's overdue. Second: it fixes the operational pain everyone complained about while doing nothing for the thing that's actually expensive about MCP. Those aren't in tension. You can ship a beautiful round-robin deployment and still set your token budget on fire.

What actually changed on the wire

Here's the old request path. A client connects, does the handshake, gets a session ID, and from then on every request carries that ID and has to land on the same server instance that minted it.

POST /mcp HTTP/1.1
Content-Type: application/json
Mcp-Session-Id: 4e9c1a7f-2b3d-44a8-9f10-0c2d6a1b88ef

{"jsonrpc":"2.0","id":1,"method":"tools/call",
 "params":{"name":"search_repo","arguments":{"q":"retry"}}}

That Mcp-Session-Id is the load-balancer tax. It forces sticky sessions or a shared session store, because instance B has no idea what instance A negotiated during initialize.

The new path drops the session entirely. Client metadata that used to be negotiated once at connection setup now rides per-request in a _meta field, and two new required headers — Mcp-Method and Mcp-Name — let gateways and rate-limiters route without parsing the JSON body.

POST /mcp HTTP/1.1
Content-Type: application/json
Mcp-Method: tools/call
Mcp-Name: search_repo

{"jsonrpc":"2.0","id":1,"method":"tools/call",
 "params":{"name":"search_repo","arguments":{"q":"retry"},
   "_meta":{"client":{"name":"my-agent","version":"3.1.0"}}}}

Capability exchange that used to happen in initialize now happens through a server/discover call any instance can answer. List and resource responses also pick up ttlMs and cacheScope so a gateway can cache them like HTTP. That last part matters more than it looks — I'll come back to it.

Deleting the session store

The migration most people will underestimate is server state. If your server looks like this, every line of it is now dead weight:

# BEFORE — stateful, 2025-11-25 style
sessions = {}  # in-memory, or worse, Redis you now have to operate

def on_initialize(req):
    sid = uuid4()
    sessions[sid] = {"caps": req.params["capabilities"],
                     "client": req.params["clientInfo"]}
    return {"sessionId": sid, "capabilities": SERVER_CAPS}

def on_tools_call(req, headers):
    s = sessions[headers["Mcp-Session-Id"]]  # KeyError on the wrong box
    ...

After, there is no session map and no on_initialize. Each handler is pure with respect to the request — it reads what it needs from _meta and answers:

# AFTER — stateless, 2026-07-28 RC style
def on_discover(req):
    return {"capabilities": SERVER_CAPS}   # any instance can answer

def on_tools_call(req):
    client = req.params["_meta"]["client"]  # travels with the request
    return run_tool(req.params["name"], req.params["arguments"])

The payoff is exactly what the spec authors advertised: you can put this behind "a plain round-robin load balancer" with no sticky routing and no shared store. Server-to-client prompts that used to need a persistent SSE stream now return an InputRequiredResult carrying echoed state, so any instance can resume the round trip. Kill your Redis session cache. Kill your sticky-session annotations. This is a genuinely smaller system to operate.

Tasks got demoted, and the lifecycle flipped

Tasks were an experimental core feature. In the RC they're reclassified as an opt-in extension, and tasks/list is gone — the spec is blunt that it "can't be scoped safely without sessions." If you can't pin a client to a box, you can't safely enumerate "its" tasks.

The lifecycle inverts. Instead of the server tracking long-running work against a session, tools/call returns a task handle and the client becomes the driver, polling and steering with tasks/get, tasks/update, and tasks/cancel.

// 1. client calls a long-running tool
{"jsonrpc":"2.0","id":7,"method":"tools/call",
 "params":{"name":"reindex","arguments":{"scope":"all"}}}

// 2. server hands back a task handle (illustrative shape)
{"jsonrpc":"2.0","id":7,
 "result":{"task":{"id":"tsk_91af","status":"working"}}}

// 3. client drives it from any instance
{"jsonrpc":"2.0","id":8,"method":"tasks/get",
 "params":{"id":"tsk_91af"}}

If you leaned on tasks/list to rebuild a dashboard of in-flight work, that pattern is over. You now own task identity — persist the handles client-side (or in a real datastore your tools write to), because the protocol won't remember them for you.

The error code and auth changes that'll trip your tests

Two smaller changes will fail your integration suite quietly if you don't grep for them.

The resource-not-found code moves from the MCP-specific -32002 to the standard JSON-RPC -32602. If you have assertions or client branches matching on -32002, they'll silently stop firing.

- if (err.code === -32002) handleMissingResource();
+ if (err.code === -32602) handleMissingResource();

On auth, six SEPs pull MCP toward plain OAuth/OIDC. The one that bites first is mandatory iss validation per RFC 9207: the authorization response now has to carry the issuer identifier and your client has to check it matches the server you started the flow with. If you hand-rolled an OAuth client that ignored iss, it was technically exploitable (mix-up attacks) and is now non-conformant. Roots, Sampling, and Logging also enter deprecation on 12-month removal clocks — not gone in July, but don't build anything new on them.

Why none of this fixes the actual disease

Here's my contrarian read. Everything above is plumbing. It makes MCP cheaper to operate and does almost nothing about what makes MCP expensive to run: the model pays for your tool schemas on every single turn.

MCP co-creator David Soria Parra put it plainly in April 2026 — "a significant portion of the context window is consumed before the model does any actual reasoning." Statelessness doesn't touch that. Arguably it makes the framing worse: by deleting the session you delete the one place a server could have remembered "this client already saw these 40 tool definitions," so the obvious answer becomes re-sending schemas and re-sending context on more requests, not fewer.

I pulled traces on one of my own agents after reading the RC. The tool-schema block plus re-sent prior context regularly ate well over half the prompt before a single user token showed up. That's not a spec bug — it's the cost model. And the stateless rewrite's headline win is operational, not token-economic.

The one lever the RC actually hands you here is the new ttlMs and cacheScope on list/resource responses. That's the part to obsess over. A gateway that caches tool listings instead of re-fetching them per call is the closest thing in this release to addressing context cost — and it's buried under the routing-headers announcement. Use it. Cache aggressively at the edge, trim your tool surface to what the agent actually calls, and stop shipping 40 tools when the agent uses 6.

The takeaway

You have roughly ten weeks. The work is mechanical: delete initialize, delete the session store, move client info into _meta, add Mcp-Method/Mcp-Name, rewrite Tasks as client-driven, fix the -32002 → -32602 assertions, and validate iss. Do it, and your deployment gets simpler and cheaper to operate.

But don't confuse a simpler deployment with a cheaper agent. Statelessness solved the load balancer's problem. The model's problem — paying rent on your whole tool catalog every turn — is still sitting there, and the spec just made it slightly harder to cache your way out of. Migrate for the operations win. Then go fight the real bill.

Are you migrating before July 28 or waiting for the final spec to land? And has anyone actually measured what their tool schemas cost per turn — what's your number? Drop it in the comments.