DEV Community: Gaurav Dadhich

Claude API Pricing in 2026: Every Model per Million Tokens, and What Pro and Max Cost

Gaurav Dadhich — Sat, 26 Sep 2026 09:12:27 +0000

# Claude API Pricing in 2026: Every Model per Million Tokens, and What Pro and Max Cost *Published 26 September 2026 · Every price below was read from Anthropic's pricing documentation, the Claude plans page and the Claude Help Center on 26 September 2026, four days after Claude Opus 5.5 launched.* The Claude API costs between $1 and $10 per million input tokens and between $5 and $50 per million output tokens on Anthropic's current models: Claude Haiku 4.5 at $1 and $5, Claude Sonnet 5 at $2 and $10, Claude Opus 5.5 at $4 and $20, and Claude Fable 5.1 at $10 and $50. Cached input is billed at a small fraction of those rates, the Batch API halves them, and none of the current models charges a premium for long context. Claude's apps are priced separately from the API: Free is $0, Pro is $20 a month ($17 a month billed annually), and Max is $100 or $200 a month for five or twenty times Pro's usage. Those rates are the easy part. What decides a Claude bill is how many tokens each request carries, and in a multi-turn product that number grows with every turn, because the whole conversation is sent again each time. The worked example further down shows a 40-turn chat on Sonnet 5 costing $1.09 without prompt caching and $0.29 with it, from the same rate card. ## Claude API price per million tokens, current models | Model | Input | Output | 5-minute cache write | 1-hour cache write | Cache read | Batch input | Batch output | |---|---|---|---|---|---|---|---| | Claude Haiku 4.5 | $1 | $5 | $1.25 | $2 | $0.10 | $0.50 | $2.50 | | Claude Sonnet 5 | $2 | $10 | $2.50 | $4 | $0.20 | $1 | $5 | | Claude Opus 5.5 | $4 | $20 | $5 | $8 | $0.20 | $2 | $10 | | Claude Fable 5.1 | $10 | $50 | $12.50 | $20 | $0.25 | $5 | $25 | All figures are US dollars per million tokens, from [Anthropic's pricing documentation](https://platform.claude.com/docs/en/about-claude/pricing). Claude Mythos 5.1 carries the same rates as Fable 5.1. Opus 5.5, released on 22 September 2026, is the change most pricing pages have not caught up with yet. [Anthropic's launch post](https://www.anthropic.com/claude-opus-5-5) puts its input and output rates 20% below Opus 5 ($5 and $25) and its cache reads 60% below ($0.20 against $0.50). Sonnet 5 launched on 30 June 2026 at an introductory $2 and $10, and [Anthropic made that price permanent](https://www.anthropic.com/news/claude-sonnet-5) in August. Choosing between them on price alone is simple arithmetic: Sonnet 5 costs half of Opus 5.5 per token, Haiku 4.5 costs half of Sonnet 5, and Fable 5.1 costs two and a half times Opus 5.5. Sonnet 5 is the default model in the Free and Pro apps. Which model is good enough for a given job is a quality question the rate card cannot answer, so run your own prompts through two adjacent tiers before committing a workload to either. ## Older models still on the price sheet Anthropic still lists earlier models, and several of them cost more than their replacements. | Model | Input | Output | Cache read | |---|---|---|---| | Claude Opus 5, Opus 4.8, 4.7, 4.6, 4.5 | $5 | $25 | $0.50 | | Claude Opus 4.1 and Opus 4 | $15 | $75 | $1.50 | | Claude Sonnet 4.6, 4.5 and 4 | $3 | $15 | $0.30 | | Claude Haiku 3.5 | $0.80 | $4 | $0.08 | | Claude Fable 5 and Mythos 5 | $10 | $50 | $1 | A workload still pinned to Opus 4.1 pays $15 and $75; the same tokens on Opus 5.5 cost 73% less. A workload on Sonnet 4.6 pays $3 and $15, so moving it to Sonnet 5 cuts the token bill by a third, although a model change is a quality decision as well as a price one and deserves the same evaluation as any other. ## What one request actually bills Every request is billed as input tokens plus output tokens, and input is more than the user's latest message. [Anthropic's context window documentation](https://platform.claude.com/docs/en/build-with-claude/context-windows) lists what counts: the system prompt, every message in the messages\ array (including tool results, images and documents), the tool definitions, and Claude's own earlier output. In a conversation, "previous turns are preserved completely", which means turn 40 pays again for turns 1 to 39. Thinking tokens are billed as output. The [extended thinking documentation](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) reports them in usage.output\_tokens\_details.thinking\_tokens\ as part of the billed output, so a reasoning-heavy request can cost several times what its visible answer suggests. On Opus 4.5 and later, Sonnet 4.6 and later, and the Fable and Mythos models, earlier thinking blocks also stay in the conversation and count toward the context window like any other input, while Haiku models and older Opus and Sonnet models have them stripped automatically. The tokenizer matters too. [Anthropic's Opus 4.7 announcement](https://www.anthropic.com/news/claude-opus-4-7) says its updated tokenizer can map the same input to roughly 1.0 to 1.35 times as many tokens depending on content, with the per-token price unchanged. When you compare a bill across model generations, count tokens rather than characters, using the [token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting) before you send and the usage\ fields on every response after. ## When prompt caching pays for itself Prompt caching stores a prompt prefix so that later requests read it instead of paying full input price for it. The [prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) sets the multipliers against the base input rate: a 5-minute cache write costs 1.25 times, a 1-hour cache write costs 2 times, and a cache read costs 0.1 times. Opus 5.5 reads at 0.05 times and Fable 5.1 and Mythos 5.1 at 0.025 times, which is why their cache-read prices sit so close to Sonnet 5's. The break-even follows directly. Two requests sharing a 5-minute cached prefix pay 1.35 times the prefix instead of 2 times, so caching saves money from the first reuse. A 1-hour write pays 2.1 times across two requests against 2 times uncached, so it loses on one reuse and wins from the second. The 1-hour cache is for traffic that returns after more than five minutes but less than an hour, such as a user who pauses between messages; the 5-minute cache suits rapid turns and agent loops. Four details decide whether caching works in practice: - **Minimum length.** A prefix shorter than the model's minimum is not cached, and the API returns no error. The minimum is 512 tokens on Opus 5.5, Opus 5 and Fable 5.1, 1,024 tokens on Sonnet 5 and Sonnet 4.6, and 4,096 tokens on Haiku 4.5. - **Order of invalidation.** The cache follows the order tools, then system, then messages. Changing a tool definition invalidates everything after it, and adding an image or changing tool\_choice\ invalidates the message cache. - **Breakpoints.** A request can carry up to four cache breakpoints, and automatic caching (a top-level cache\_control\ setting) uses one of them and moves forward as the conversation grows. - **Rate limits.** Cache hits do not count against your rate limit; only uncached tokens do. Check cache\_read\_input\_tokens\ and cache\_creation\_input\_tokens\ in the response. A cache that silently misses costs at least ten times the read price on every request (twenty times on Opus 5.5, forty on Fable 5.1), and the only place it shows up is the usage object. ## Batch API, long context and the smaller line items **Batch API.** Asynchronous batch requests cost 50% of the standard input and output rates on every model, as the table above shows. Anything that does not need an answer within the conversation, such as nightly summarisation, evaluation runs or back-filling a data set, belongs there. **Long context.** Claude 4.6 and later models include the full 1 million token context window at standard pricing. Claude Sonnet 5, Opus 5.5, Fable 5.1 and every Opus model from 4.6 onward take 1 million tokens, and every model can generate up to 128,000 output tokens per request. **Fast mode.** A research-preview speed setting runs Opus 5.5 at $8 and $40 per million tokens, twice its standard rate. **US-only inference.** Setting inference\_geo\ to "us"\ applies a 1.1 times multiplier to every token category on Claude 4.6 and later models. **Tools and runtime.** Web search costs $10 per 1,000 searches plus the tokens it adds, web fetch costs only its tokens, code execution costs $0.05 an hour after 50 free hours a day, and Claude Managed Agents sessions cost $0.08 per session-hour. ## What a real conversation costs Take a chat assistant with a 2,000-token system prompt, where each user message is 150 tokens and each reply is 350 tokens. The first turn sends 2,150 tokens. By turn 40 a single request carries 21,650 tokens, because it includes the 39 exchanges before it, and the whole 40-turn conversation has consumed 476,000 input tokens to produce 14,000 output tokens. | Model | 40 turns, no caching | 40 turns, cache warm | 100 turns, no caching | 100 turns, cache warm | |---|---|---|---|---| | Claude Haiku 4.5 | $0.55 | $0.14 | $2.87 | $0.50 | | Claude Sonnet 5 | $1.09 | $0.29 | $5.73 | $1.01 | | Claude Opus 5.5 | $2.18 | $0.48 | $11.46 | $1.49 | | Claude Fable 5.1 | $5.46 | $1.08 | $28.65 | $3.06 | "Cache warm" assumes automatic caching with a 5-minute cache and replies arriving inside that window, so each turn writes only the new exchange and reads everything before it. Two things stand out. Input tokens, not output, make up most of the bill once a conversation passes a few turns. And the cost per conversation grows faster than the number of turns: going from 40 to 100 turns multiplies turns by 2.5 and the uncached bill on Sonnet 5 by more than five, because each turn carries every turn before it. At 10,000 such 40-turn conversations a month on Sonnet 5, that is roughly $10,900 without caching and $2,900 with it. Caching is the first lever. It stops working when users leave and come back after the cache has expired, and the history has to be written again at the higher write price. We build [Maximem Synap](https://www.maximem.ai/synap), a memory layer that sends a bounded context block (2,000 tokens by default) instead of the full transcript, so this cost is the problem we work on every day. The arithmetic for when that beats caching, and when it does not, is in [how to reduce LLM token costs in long conversations](https://www.maximem.ai/blog/reduce-llm-token-costs-long-conversations). ## How much Claude costs per month: the plans Claude's apps (web, desktop, mobile, Claude Code and the Office and browser extensions) are sold as subscriptions, priced per person, from the [Claude plans page](https://claude.com/pricing): | Plan | Monthly billing | Annual billing | Usage | |---|---|---|---| | Free | $0 | $0 | Base allowance | | Pro | $20 | $17 a month | More than Free | | Max 5x | $100 | n/a | 5 times Pro per session | | Max 20x | $200 | n/a | 20 times Pro per session | | Team, Standard seat | $25 per seat | $20 per seat a month | More than Pro | | Team, Premium seat | $125 per seat | $100 per seat a month | 5 times a Standard seat | | Enterprise | n/a | $20 per seat a month, plus usage | Usage cost scales with model and task | Annual billing saves $36 a year on Pro and $60 a seat on Team Standard. Enterprise adds SCIM, audit logs, a compliance API, custom data retention and a HIPAA-ready offering on top of Team. ## Free vs Pro vs Max: what actually changes The three individual plans differ mainly in how much you can use, not in what you can use it for. **Free** covers chat on web, desktop and mobile with web search, file creation, code execution, memory and app connections. **Pro** adds Claude Code, Claude Design, Slides and Docs, Claude Science, Projects, more models and the Chrome and Microsoft 365 extensions, with more usage per session than Free. **Max** keeps every Pro feature and raises the allowance to five or twenty times Pro's per-session usage, with higher output limits, early access to new features and priority access when traffic is high. The [Max plan article](https://support.claude.com/en/articles/11049741-what-is-the-max-plan) is explicit that the two Max tiers differ in usage, not in models. Anthropic does not publish message counts for any plan. Usage depends on conversation length and complexity, the model, the effort level and the tools you use, and [connectors are called out as token-intensive](https://support.claude.com/en/articles/11647753-how-do-usage-and-length-limits-work). On [Pro](https://support.claude.com/en/articles/8325606-what-is-the-pro-plan) and Max, the session allowance resets every five hours, and a separate weekly limit applies across all models and resets at a fixed time assigned to your account. Chat and Claude Code draw from the same allowance, so a long coding session reduces what is left for chat. The practical rule: stay on Free until you hit its limits regularly, move to Pro when you want Claude Code or Projects, and move to Max only when you run out of Pro's weekly allowance often enough that the extra $80 or $180 a month is cheaper than waiting. ## Subscription or API? A Pro or Max plan does not include API usage. The Pro plan article says API access "requires separate Console access and billing", so a developer who wants both pays for both. Two billing surprises follow from that separation: - **Claude Code and the API key.** If ANTHROPIC\_API\_KEY\ is set in your environment, [Claude Code authenticates with it instead of your subscription](https://support.claude.com/en/articles/11145838-use-claude-code-with-your-pro-or-max-plan) and bills your API account at token rates, even while you are paying for Max. Unset the variable if you meant to use the plan. - **The Agent SDK credit that is not there.** Anthropic announced a separate monthly credit for Agent SDK and claude -p\ usage on paid plans ($20 on Pro, $100 on Max 5x, $200 on Max 20x), then [paused that change on 15 June 2026](https://support.claude.com/en/articles/15036540-use-the-claude-agent-sdk-with-your-claude-plan). Until further notice, Agent SDK and third-party app usage draw from your plan's normal usage limits. Several pricing pages still describe the credit as live. For a team building a product, the question mostly answers itself. Plans are priced per person for people using Claude's own apps; a product that serves your users runs on the API, billed per token, and the useful comparison is between API configurations (model, caching, batch) rather than between an API bill and a subscription. For an individual developer, Pro or Max is usually the cheaper way to use Claude Code heavily, and the API is the right way to run anything automated in production. ## Frequently asked questions **How much does the Claude API cost per million tokens?** On current models, $1 input and $5 output for Claude Haiku 4.5, $2 and $10 for Claude Sonnet 5, $4 and $20 for Claude Opus 5.5, and $10 and $50 for Claude Fable 5.1. Cache reads cost $0.10 to $0.25 per million tokens, and the Batch API halves the input and output rates. **How much does Claude cost per month?** Free is $0, Pro is $20 a month or $17 a month billed annually, Max is $100 (five times Pro usage) or $200 (twenty times), Team is $25 a seat a month or $20 billed annually, with Premium seats at $125 or $100, and Enterprise is $20 a seat a month billed annually plus usage. **What is the difference between Claude Free, Pro and Max?** Pro adds Claude Code, Projects, more models and the Office and browser extensions to Free, with more usage per session. Max keeps every Pro feature and multiplies Pro's per-session usage by five or twenty, with priority access at busy times. Pro and Max limits reset every five hours and also carry a weekly limit. **Is API usage included in Claude Pro or Max?** No. API usage is billed separately through a Claude Console account. Claude Code on a Pro or Max plan uses the plan's allowance unless an ANTHROPIC\_API\_KEY\ environment variable is set, in which case it bills the API. **Is there a free Claude API tier?** Anthropic's pricing page does not list one. The Free plan covers the Claude apps, not the API. **What is the cheapest way to use Claude?** For personal use, the Free plan. For API workloads, Claude Haiku 4.5 through the Batch API at $0.50 input and $2.50 output per million tokens, with prompt caching on any repeated prefix. **Does long context cost more on Claude?** Not on Claude 4.6 and later models, which bill their full 1 million token context window at standard rates. The cost of long context comes from volume: every token in the window is billed on every request that carries it. **How do I estimate my monthly Claude API bill?** Multiply tokens by rate for each part of a request: uncached input, cache writes, cache reads and output, including thinking tokens. For conversations, remember that input per turn grows with the history, as the worked example above shows. Count tokens with the token counting endpoint before launch, then read the usage\ fields on live traffic, which is where a cache that is not hitting will show up. ## What this means for your bill The rate card sets the price of a token, and your architecture sets how many tokens you buy. Prompt caching and the Batch API are the two largest discounts Anthropic offers, and both are configuration rather than engineering. Past that, the bill is decided by how much of each conversation you choose to send again on every turn, which is a design decision about what your product remembers and how. --- *Sources, all retrieved 26 September 2026: [Anthropic pricing documentation](https://platform.claude.com/docs/en/about-claude/pricing); [Claude plans](https://claude.com/pricing); Claude Help Center articles on the [Pro plan](https://support.claude.com/en/articles/8325606-what-is-the-pro-plan), the [Max plan](https://support.claude.com/en/articles/11049741-what-is-the-max-plan), [usage limits](https://support.claude.com/en/articles/11647753-how-do-usage-and-length-limits-work), [Claude Code on Pro and Max](https://support.claude.com/en/articles/11145838-use-claude-code-with-your-pro-or-max-plan) and [the Agent SDK on paid plans](https://support.claude.com/en/articles/15036540-use-the-claude-agent-sdk-with-your-claude-plan); Claude Platform documentation on [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), [extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) and [context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows); Anthropic's announcements of [Claude Opus 5.5](https://www.anthropic.com/claude-opus-5-5), [Claude Sonnet 5](https://www.anthropic.com/news/claude-sonnet-5) and [Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7). Worked examples are our own arithmetic on those rates.*

How to Reduce LLM Token Costs in Long Conversations: What Caching Saves, and Where It Stops

Gaurav Dadhich — Sat, 26 Sep 2026 09:10:34 +0000

# How to Reduce LLM Token Costs in Long Conversations: What Caching Saves, and Where It Stops *Published 26 September 2026 · Prices are Anthropic's list rates as of 26 September 2026. The method behind every cost figure is at the end of the article.* A long conversation costs far more than its number of turns suggests, because a language model holds nothing between calls: every request carries the system prompt and the entire history again, so total input grows with the square of the number of turns. The fixes, in the order to apply them, are prompt caching, which cut a 40-turn chat on Claude Sonnet 5 from $1.09 to $0.29 in our worked example, then shrinking what each turn adds, then trimming or compacting old turns, and finally bounding the context with a memory layer once conversations span sessions. Caching loses its grip when users return after the cache has expired, and in our arithmetic a bounded context becomes cheaper than cached full history at around 200 turns and 100,000 tokens of history on Sonnet 5, even after paying for the memory layer itself. The rest of this piece is the arithmetic behind that ordering, including the cases where the cheaper-sounding option is the more expensive one. ## Why a long conversation costs more than its turns An LLM API is stateless. [Anthropic's context window documentation](https://platform.claude.com/docs/en/build-with-claude/context-windows) describes each turn's input as "all previous conversation history plus the current user message", with previous turns "preserved completely". If a turn adds S tokens and the system prompt is P tokens, the request at turn k carries roughly P + (k − 1)·S tokens, and a conversation of N turns sends N·P + S·N(N − 1)/2 input tokens in total. The second term is quadratic, and past a handful of turns it is most of the bill. A concrete conversation makes it tangible: a 2,000-token system prompt, 150-token user messages and 350-token replies. Turn 1 sends 2,150 tokens. Turn 40 sends 21,650. The 40-turn conversation consumes 476,000 input tokens to produce 14,000 output tokens, and on [Claude Sonnet 5 at $2 input and $10 output per million tokens](https://platform.claude.com/docs/en/about-claude/pricing) it costs $1.09, of which 87% is input. Going from 40 to 100 turns multiplies the turns by 2.5 and the bill by 5.2, to $5.73. Two things make the curve steeper on modern models. Thinking tokens are billed as output, and on Claude Opus 4.5 and later, Sonnet 4.6 and later and the Fable and Mythos models, earlier thinking blocks stay in the conversation and count as input on every later turn. In agent loops, tool results do the same: a search result or file read added at turn 5 is paid for again on every turn after it. ## Lever 1: prompt caching, and what it saves Prompt caching stores a prompt prefix so that later requests read it instead of paying full price for it. On Claude, [a 5-minute cache write costs 1.25 times the base input rate and a cache read costs 0.1 times](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), with Claude Opus 5.5 reading at 0.05 times and Claude Fable 5.1 at 0.025 times. Automatic caching (a top-level cache\_control\ setting) places the breakpoint on the last cacheable block and moves it forward as the conversation grows, so each turn writes only the new exchange and reads everything before it. For the 40-turn conversation above, that takes the Sonnet 5 cost from $1.09 to $0.29 and the input share of the bill from 87% to about half. It needs no change to what the model sees, which is why it comes first. Cache hits also do not count against your rate limit. The 5-minute cache suits rapid turns and agent loops. The 1-hour cache costs 2 times the input rate to write and pays back from the second reuse, so it suits chat where users pause for more than five minutes but return within the hour. ## Where caching stops working Caching discounts the history; it does not shrink it. Four things limit how far it goes. - **Users leave and come back.** When a user returns after the cache has expired, the first request of the new session writes the entire history again at 1.25 times the input rate. For an assistant used in short sessions over days or weeks, that rewrite happens every session and grows with the history. - **Changes invalidate it.** The cache follows the order tools, then system, then messages. Changing a tool definition invalidates everything; adding an image or changing tool\_choice\ invalidates the message cache. A cache that misses returns no error, so the only place it shows up is cache\_read\_input\_tokens\ in the usage object. - **The window still fills.** Current Claude Opus, Sonnet and Fable models take 1 million tokens, and a cached history counts toward the window at full size. - **Quality does not improve with volume.** Anthropic's own [compaction documentation](https://platform.claude.com/docs/en/build-with-claude/compaction) justifies the feature by noting that "response quality degrades as a conversation grows". A cheaper token is still a token the model has to read past. ## Lever 2: shrink what each turn adds Before touching the history, cut what each turn contributes. **Clear old tool results.** Claude's [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing) (beta header context-management-2025-06-27\) clears tool results once the input passes a threshold (100,000 tokens by default) and keeps the three most recent tool uses, replacing the rest with a placeholder. It can also clear old thinking blocks. Clearing breaks the prompt cache at the point it clears, so the clear\_at\_least\ setting exists to make sure each clearing removes enough tokens to be worth a fresh cache write. **Cap output and thinking.** Output costs five times input on current Claude models, and thinking tokens bill as output. A max\_tokens\ limit and a lower effort setting on turns that do not need deep reasoning reduce both what you pay now and what later turns carry. **Route by task.** A summarisation or classification step inside a conversation rarely needs the model that holds the conversation. Claude Haiku 4.5 costs half of Sonnet 5 per token, and anything that can wait belongs on the Batch API at half price. ## Lever 3: trim or compact the history **A sliding window** keeps the last K exchanges and drops the rest. It is free and caps per-turn input, and it forgets everything outside the window, including the user's name if they gave it at turn 2. Keep a pinned block for facts that must survive (identity, standing constraints, open tasks), or the window will drop them. **Compaction** replaces older turns with a summary. Claude now does this server-side: [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) can run on demand or automatically when input tokens reach a threshold you set, with no summarisation code of your own. A summary costs a model call to produce and pays for itself quickly once the history is long, because every later turn reads the summary instead of the transcript. What compaction risks is detail. A summary that turns "takes 20mg of Atorvastatin daily" into "takes a cholesterol medication" has saved tokens and lost the fact that mattered. Whatever compacts your history should tell you what it kept. Maximem Synap's compaction, which we build, returns a validation score and a count of preserved facts on every pass, runs when a conversation passes 3,000 tokens, 10 messages or 5 minutes idle, targets 1,500 tokens, and keeps the last three exchanges verbatim. ## Lever 4: bound the context with a memory layer A memory layer changes the unit that is resent. Instead of replaying the transcript, it extracts what is worth keeping (facts, preferences, decisions, open tasks) as the conversation happens, stores them outside the prompt, and returns a ranked block of the relevant ones within a token budget on each turn. Per-turn input stops growing with the history: system prompt, a memory block (2,000 tokens by default in Maximem Synap), the last few exchanges verbatim and the new message. That is not free, and the comparison is only honest if the memory layer's own cost is counted. The table below does that. It uses the same conversation as above, split into sessions of 10 turns with the user returning after the cache has expired, on Sonnet 5. The bounded column includes Maximem Synap's published usage cost for storing each exchange and fetching context every turn (3 credits a turn at $0.00175 a credit). | Conversation length | No caching | Cached, cache cold at each session | Cached, cache never expires | Bounded context, memory layer included | |---|---|---|---|---| | 10 turns, 1 session | $0.12 | $0.06 | $0.06 | $0.16 | | 50 turns, 5 sessions | $1.61 | $0.51 | $0.38 | $0.84 | | 100 turns, 10 sessions | $5.73 | $1.56 | $1.01 | $1.68 | | 200 turns, 20 sessions | $21.46 | $5.27 | $3.01 | $3.37 | | 500 turns, 50 sessions | $128.65 | $29.29 | $15.02 | $8.45 | Read it in both directions. For conversations under about 100 turns, caching alone is cheaper, and a memory layer is not a cost decision there; if you adopt one at that length, adopt it for what it remembers across sessions, not for the bill. Around 200 turns, where the history passes 100,000 tokens, the bounded context overtakes cached history whose cache goes cold between sessions. At 500 turns it costs 29% of the cold-cache figure and 56% of a cache that never expires, and the gap keeps widening because one column grows with the square of the turns and the other grows linearly. The crossover moves with the model. Claude Opus 5.5 reads cache at 0.05 times its input rate, which makes cached history unusually cheap: on Opus 5.5 the bounded context beats the cold-cache case from 200 turns ($5.62 against $8.67) but only beats a cache that never expires at 500 turns ($14.09 against $17.40). An independent cost study, [arXiv 2603.04814](https://arxiv.org/abs/2603.04814), reached a consistent shape on different assumptions: at a 100,000-token context a fact-based memory system became cheaper than long-context prompting after about ten turns, with the break-even arriving sooner as context grows. The same study, with GPT-5-mini answering and a three-vote GPT-5-mini judge, found long-context prompting recalled more facts on two of its three benchmarks, which is the quality trade to test on your own conversations before you move a workload. ## Which technique for which workload | Workload | Start with | Add when | Watch for | |---|---|---|---| | Short chats, one sitting | Prompt caching | Rarely needs more | Silent cache misses | | Agent loops with heavy tool use | Caching plus tool-result clearing | Compaction past the window threshold | Clearing breaking the cache too often | | Long single sessions (research, coding) | Caching, then server-side compaction | A pinned block for facts the summary must keep | Detail lost in summaries | | Assistants people return to over days or weeks | Caching within sessions | A bounded context once histories pass about 100,000 tokens | Retrieval missing a fact the transcript had | | Support or sales agents serving many users | Caching plus a bounded context | Organisation context held separately from user memory | Scoping between users and tenants | ## How to measure it Measure per conversation, not per request. For every turn, log input\_tokens\, cache\_creation\_input\_tokens\, cache\_read\_input\_tokens\ and output\_tokens\ from the response, keyed by conversation and session. From that you get the three numbers that matter: cost per conversation by length bucket, cache read share per turn (a sudden drop means something is invalidating the cache), and the input tokens carried at the median and the 95th percentile turn. The distribution matters more than the average, because the longest conversations are where the quadratic term lives. Use the [token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting) to size prompts before launch. Set a context budget explicitly: the number of tokens a turn is allowed to carry beyond the system prompt. Without one, the budget is whatever the history happens to be, and it only ever grows. ## What this means for pricing your own product If your product charges a flat fee per user, your heaviest users cost you the most, and on unmanaged context they cost you disproportionately more, since a user with five times the conversation length can cost up to roughly twenty-five times as much to serve. Either bound the context so cost per turn stays flat, or price in a way that acknowledges length, through usage tiers or credits. Doing neither means your best-retained customers are your least profitable ones. ## Frequently asked questions **How do I reduce LLM token costs in long conversations?** Turn on prompt caching first; on Claude Sonnet 5 it cut a 40-turn conversation from $1.09 to $0.29 in our example. Then cap output and clear old tool results, compact or trim old turns once a single session runs long, and bound the context with a memory layer for assistants that users return to across sessions, where caching stops helping once the cache expires between visits. **Why do chatbot conversations get more expensive as they go on?** Because the API is stateless, each request resends the system prompt and every previous turn, so input per turn grows with the conversation and total input grows with the square of the number of turns. A 40-turn chat with a 2,000-token system prompt and 500 tokens per exchange sends 476,000 input tokens. **Does prompt caching solve the cost of long conversations?** Within a sitting, mostly. It bills the repeated history at a tenth of the input rate on most Claude models. It does not shrink the history, it has to rewrite the whole history when a user returns after the cache expires, and it does nothing for the context window limit or for answer quality on very long inputs. **Is summarising the conversation cheaper than sending the full history?** Once the history is long, yes, because every later turn reads the summary instead of the transcript. The risk is losing specific details, so check what each summary kept. **When is a memory layer cheaper than prompt caching?** In our arithmetic on Claude Sonnet 5, at around 200 turns spread over sessions, when the history passes roughly 100,000 tokens, including the memory layer's own cost. Below about 100 turns, caching alone is cheaper. On Claude Opus 5.5, whose cache reads are cheaper, the crossover comes later. **Does reducing context make responses faster?** Every input token has to be processed before the first output token, so a smaller prompt starts answering sooner. Caching reduces cost more than it reduces the amount the model reads, while trimming, compaction and a bounded context reduce both. ## Method Every cost figure comes from one synthetic conversation: a 2,000-token system prompt, 150-token user messages and 350-token replies, priced at Anthropic's list rates on 26 September 2026 (Claude Sonnet 5: $2 input, $10 output, $2.50 for a 5-minute cache write and $0.20 for a cache read per million tokens; Claude Opus 5.5: $4, $20, $5 and $0.20). "Cache warm" and "cache never expires" assume automatic caching where each turn writes the new exchange and reads the prefix. "Cache cold at each session" assumes sessions of 10 turns with the cache expired at the start of each, so the first request of a session writes the full prefix at the write rate. The bounded context sends the system prompt (cached within a session), a 2,000-token memory block, the last three exchanges and the new message at the full input rate, plus Maximem Synap's published usage cost of 3 credits a turn (2 credits to store a 500-token exchange and 1 for a fast context fetch) at the list rate of $0.00175 a credit; plan credits cost less than that. Output cost is included everywhere. Real conversations vary in length and shape, and whether a 2,000-token memory block holds enough for a given product is a quality question this arithmetic does not answer. --- *Sources, retrieved 26 September 2026: [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing); Claude Platform documentation on [context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows), [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction), [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing), [extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) and [token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting); Pollertlam and Kornsuwannawit, [Beyond the Context Window](https://arxiv.org/abs/2603.04814), March 2026. Maximem Synap figures come from its published documentation and pricing. For every Claude rate in one place, see [Claude API pricing in 2026](https://www.maximem.ai/blog/claude-api-pricing-2026).\*

What is WebMCP? When and How to Use WebMCP in a Browser Agent?

Gaurav Dadhich — Wed, 23 Sep 2026 06:37:16 +0000

What is WebMCP? When and How to Use WebMCP in a Browser Agent?

Published 23 September 2026

WebMCP lets a web page hand an agent a list of named, typed actions instead of making it work them out from the rendered page. If you already run a scraper or a browser agent, the thing worth knowing before anything else is that the decision to use it is per step, not per site. A page can expose a clean search tool and leave its account settings as ordinary DOM controls, so an agent that picks one mode per domain gets the worst of both. Your existing reader stays primary, WebMCP becomes a fast path in front of it, and the arbitration happens at each step of the task.

That is the whole architecture. Everything below is what the parts cost you, starting with the one that will eat an afternoon if you take it from a guide written three weeks ago.

What WebMCP is

A page calls document.modelContext.registerTool() with a name, a plain-language description and a JSON Schema for the inputs. Any agent running in that tab can then list what is registered and call it directly. The specification puts it plainly: a page using WebMCP can be thought of as a Model Context Protocol server that implements its tools in client-side script rather than on a backend.

What it replaces is guessing. An agent today works a site by screenshotting it, reading the DOM or the accessibility tree, inferring which control does what, and simulating clicks; rename a CSS class and the sequence breaks. A registered tool states the contract instead, so search_products takes a query string and hands back a result, rather than a button somewhere on the page that might be the search button.

Two properties matter more than the API surface. Tools live in the tab and die on navigation, so they are scoped to the page rather than to your session. They also execute inside the user's existing signed-in session. That is how they reach a cart, a filter state or a half-finished form that a server-side integration would need its own authentication to touch.

When it is worth reaching for

Lifetime decides most of it. WebMCP does not replace an MCP server: its tools exist only while the page is open, so anything that must run without a live tab belongs in a server-side capability. Chrome frames the inversion well when it says that instead of your application being a guest inside an agent, the agent becomes a guest on your platform.

Within that boundary, reach for WebMCP when the action belongs to the page the user already has open and depends on live tab state, which covers a cart, an applied filter, a document being edited, a dashboard date range. Use your existing reader everywhere else, and everywhere else is still most of the web.

How often the fast path actually fires is genuinely unknown, which is worth settling before you plan around it. Largest published figure is a directory count rather than a crawl, at 462 live sites and demos as of 5 September. Against that, Shopify switched tools on for every Liquid storefront effective 21 August, a very large installed base arriving in one step, and Cloudflare's edge bridge lets a site enable it without touching its origin. Nobody has a reliable denominator.

Start with the part that is already stale

Every article-format guide currently published shows this call:

await document.modelContext.executeTool(tool, JSON.stringify(args));

That shape no longer validates. A Draft Community Group Report dated 17 September 2026 defines the second argument as optional any inputObject and specifies that if it is not an Object, the call returns a promise rejected with a TypeError; the browser serializes internally. Chrome's imperative API documentation, last updated 11 September, now documents only the object form and carries a deprecation note on the other: JSON stringified input arguments are deprecated from Chrome 155.

So the current call passes an object:

const result = await document.modelContext.executeTool(tool, { query: "mechanical keyboard" });

None of those guides were wrong when they published. freshman.tech and flaviocopes, both updated 9 September, even flag the divergence and tell you to use whichever form your browser supports; they predate the resolution by about a week. That is the real lesson, and it is worth more than the API detail: published integration guidance in this corner of the platform has a shelf life measured in days, and anything you hardcode from an article inherits it.

The four calls you need

Detection first, because two separate gates produce a silent undefined even on a browser that supports the feature. The page has to be origin-isolated, so a document opting out through document.domain or an Origin-Agent-Cluster: ?0 response header does not get the API at all, and registration is governed by the tools Permissions Policy with a default allowlist of self, which means cross-origin iframes are excluded unless the parent delegates with allow="tools".

if (!document.modelContext) { /* use your reader */ }

Global names have moved too. It was navigator.modelContext, an earlier draft carried a provideContext() method since removed, and document.modelContext is current. If a sample uses either older name, check its date before you trust anything else in it.

Listing is await document.modelContext.getTools(). Note the scope carefully, because it decides which API you actually write against: getTools() is a discovery surface for agents running inside the page, and a browser-integrated agent never calls it, since the browser hands it tools through an internal channel. If you drive a browser from outside, you use your driver instead. Stagehand exposes page.listWebMCPTools() and page.invokeWebMCPTool(); agent-browser exposes webmcp list and webmcp invoke; Puppeteer documents page.webmcp. Cross-origin listing takes an options bag, getTools({ fromOrigins: ['https://shop.example'] }), and it only returns anything if the registering page also named your origin in registerTool(tool, { exposedTo: [...] }). Both keys have to turn.

Calling is executeTool(tool, inputObject), passing the tool object itself rather than its name, and the result comes back as a JSON string even though the input is an object.

Cancellation is two different mechanisms that are easy to confuse. Registration lifetime rides on an AbortSignal handed to registerTool(tool, { signal }), so aborting the controller unregisters the tool. Execution cancellation is separate: execute receives a signal as its second argument, which you pass through to fetch(). freshman.tech notes the gap between them, and it is the kind of thing that surfaces in a live demo: as of Chrome 153, unregistering a tool does not cancel an execution already in flight.

Per step, not per site

Here is the gap in the published material. Every article-format treatment decides per site or at design time. The one project with genuine per-step arbitration, gui-agent, hands the model a single merged tool list of registered WebMCP tools plus synthesized DOM tools, instructs it to prefer the page's tools, and resolves name collisions in favour of the app tool; its decision node reads "Is there a purpose-built WebMCP tool for this step?" That is exactly right, and it is scoped to applications you own, which leaves the reader driving somebody else's site without a published answer.

Generalising it is not complicated. For each step, match against the tool's name, description and input schema, then fall through to your reader for that step when nothing fits or when you cannot fill the schema from data you already hold. Four conditions send you to the fallback and only one of them is "this site has no WebMCP": no API, no matching tool, a schema you cannot satisfy, or a call that throws. Treat them identically, including when a tool vanishes mid-task, which is normal rather than exceptional given that registrations follow routes and permissions.

Aident comes closest among the articles, with a per-action preference order running from a structured tool contract owned by the application, to a direct API or MCP capability for durable external work, to browser automation for whatever interface remains uninstrumented. Their caveat deserves repeating, because it cuts against the obvious reading: that order is not a universal reliability ranking, and a poorly designed tool can be more dangerous than careful automation.

For the fallback itself, accessibility-tree-with-refs is the strongest documented default. agent-browser marks its accessibility-tree snapshot as the best option for AI and treats annotated screenshots as optional and secondary, with change thresholds to control token spend; gui-agent builds a compact text snapshot carrying roles, labels, values and stable refs, and states plainly that it needs no screenshots and no multimodal model. Vision costs the most context and is the least stable under redesign.

The list moves under you

Tools follow routes, permissions and authentication state, so a catalogue read at page load goes stale quickly. The current spec defines three events, not one: toolchange when the list changes, toolactivated when an execution begins, and toolcancel when one is cancelled. Published guidance covers the first and has not caught up with the other two.

How you observe changes depends again on where you sit. In-page agents subscribe. External drivers re-list, and Stagehand makes this a guarantee rather than a convention: listWebMCPTools() returns a fresh snapshot on every call and never reuses tools from a previous page or a previous call. agent-browser pushes catalogue changes into context through a data.webmcp field and warns about the case that quietly breaks caching, which is that a full record can change while the brief description stays identical, so a previously fetched schema needs refreshing after any catalogue update.

Annotations are worth reading properly, because there are four and the circulating summaries list three. readOnlyHint says the tool only reads. untrustedContentHint says the output contains data the registering author does not trust. consequentialHint marks actions that are significant, real-world or non-reversible, with booking a flight and transferring money as the spec's own examples. The fourth, debugging, marks tools meant for developer tooling rather than end-user interaction, and it appears in none of the guides I read. It is spelled debugging, not debuggingHint.

One genuinely open question: whether tools registered declaratively through HTML form attributes show up in getTools() is undefined today. Section 4.3 of the spec reads, in full, that the section is entirely a TODO. The declarative explainer addresses the exact question and leaves it open, saying that declarative tools should almost certainly be invokable from that interface but the details are still to be determined. Chrome's declarative documentation, the stalest WebMCP page at 18 May, documents the attributes without stating the answer. Plan for imperative registration and treat declarative discovery as unavailable until the TODO closes.

Getting it to run at all

For local development, enable chrome://flags/#enable-webmcp-testing and relaunch. For production traffic, the origin trial received approval for Chrome M149 through M156 inclusive, with shipping estimated at 157; Edge runs a parallel trial from 150 that expires 17 November 2026. If you drive Chrome externally over DevTools you also need launch features, and Stagehand is the only source that prints them: Chrome or Chromium newer than 149, launched with --enable-features=WebMCPTesting,DevToolsWebMCPSupport.

Skip ChromeStatus as a source on current state; that record was last edited 12 August and still reports the feature as proposed with no origin trial and no engine signals, which the live trial and the filed positions both contradict.

Headless agents cannot reach the API directly, since it lives in a live tab, and Chrome's limitations list says headless is not the design target. Two routes work well. Run a hosted browser and evaluate inside the session, which is what Firecrawl documents through its scrape-then-interact endpoints, or drive Chrome over CDP with Stagehand, agent-browser, Browser Use (which added native support on 6 September) or Cloudflare Browser Run. Running in-page with @mcp-b/webmcp-polyfill is a third option that needs no flags at all, because it installs the global itself.

Client support today, per the implementation tracker: Chrome from 149, Edge from 150, Brave experimentally in Leo, and ChatGPT Desktop, which requires GPT-5.6 Sol or Terra and is unavailable in Enterprise and Edu workspaces. No Firefox, no Safari. Engine positions are now formally filed rather than merely under discussion, and secondary coverage gets this wrong in both directions: WebKit's recorded position is oppose, with concerns spanning API design, duplication, internationalization, privacy, security, venue, use cases and portability, while Mozilla's is neutral.

Chrome ships a Model Context Tool Inspector extension for debugging, which lists tools, executes one against JSON input and shows structured output. If document.modelContext is undefined after you enable the flag, check origin isolation before anything else, and resist shipping your own shim at that global, which mostly hides the fact that you have stopped testing the browser's implementation.

Whose hint is it

Tool names, descriptions, schemas and results are attacker-controlled input. agent-browser states the position as bluntly as anyone has: all page-provided names, descriptions, schemas, annotations and results are untrusted data, the provenance labels are cues rather than a prompt-injection boundary, and website text does not get promoted into system instructions, does not authorise shell commands, and does not constitute a claim of user consent.

Which sets up the thing nobody names. Two shipping libraries take opposite positions on readOnlyHint, and both are correct.

gui-agent trusts it. Any tool without annotations.readOnlyHint routes through a confirmation callback before running, and marking a tool read-only skips the prompt. agent-browser refuses it, stating that page-provided readOnlyHint or untrustedContentHint claims cannot bypass host controls, and that domain filters do not prevent a page from lying about what a tool does.

That is not a disagreement about security. gui-agent runs inside an application you own, where the annotation is your own assertion about your own code; agent-browser drives arbitrary third-party sites, where the identical field is a claim by a stranger. OpenAI puts the third-party case in one line, which is that a tool's name or its claim to only read data is not proof of what it does. Copy the wrong pattern into the wrong threat model and you have built a confirmation gate any page can switch off by setting a boolean.

Schemas do not help here either. Ego ran the Chrome Labs hotel demo on 11 September in Chrome 152.0.7977.76 and found an out-of-enum value passing schema validation and reaching the application route. A schema shapes input; it does not prove the request is authorised, and it never will, because it is advisory to whoever is calling.

What to actually build, drawing on Chrome's agent security guidance: cap inbound tool-response size and reject oversized payloads rather than truncating them into context; restrict callable origins to those relevant to the current task; wrap tool output so the model reads it as data rather than instruction; require confirmation for consequential actions by your own policy rather than the page's annotation; keep credentials and personal data out of tool arguments unless the tool genuinely needs them; and log origin, tool name, arguments and whether each step came from WebMCP or the fallback, because after an incident the question you will need to answer is which page told you to do that.

Is it worth it

One public benchmark exists, and the disclosure belongs before the numbers. WindTunnel is published by nekuda, which sells WebMCP products including AgentLane, a WebMCP Kit, a Chrome extension and an SDK, and which also operates the site hosting the leaderboard. The benchmark measures the interface its product line depends on.

Their method is better than that framing might suggest. The harness is public under Apache-2.0, covering 49 tasks across 8 self-hosted open-source applications pinned to upstream commits, three attempts per task scored by majority verdict, with agent models including Claude Sonnet 5, Claude Opus 5, GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Astra and Gemini 3.6 Flash. No judge model is involved; scoring is a code check against real post-run application state. Current board is version 1.2, dated 18 September, at 21 configurations and 3,087 attempt rows.

What it shows now is narrower than what gets quoted. Ten configurations solve 49 of 49: all nine WebMCP ones, plus GPT-6 Astra on code execution, which is a screen-driving mode. Take the median of the twelve non-WebMCP rows on the published board and it comes to 44, though the README does not print that figure itself. nekuda's own text concedes the shift: raw task-solve rate does not separate WebMCP from the best screen-driving configuration, and cost and time do. The same README discloses that projected per-thousand-run cost ranges overlap, that two drivers cannot report the model snapshot served, and that an independent review caught a merge built before corrected predicates were applied, requiring a re-score that produced nineteen false-negative promotions.

Two figures in wide circulation should stop circulating. "48 of 49 against a median of 43" is the August board, retired on 6 September when a missing checkout tool in one demo store was fixed; by nekuda's own account that gap made 48 a ceiling by construction, a limitation of one store's tool surface rather than of the approach. And the pairing of 7.8 seconds against 28.1, at 0.6 cents against 5.5, matches no table currently published on either the original post or the live board. I could not source it, so I am not repeating it as fact.

A quieter argument is the token bill, and exactly one project treats it seriously. agent-browser withholds schemas by default, requiring a two-stage fetch where the agent picks from a summary and then requests the single schema it needs, and it hard-budgets those summaries to sixteen tools and four kilobytes with descriptions cut to a hundred and sixty bytes. Sites can register a lot of tools; on the fallback side, a page dump or a screenshot per step is precisely what makes screen-driving expensive.

What breaks next

The spec is a Community Group draft. It is not a W3C Standard and it is not on the Standards Track. Since February the global has been renamed, a method has been removed, the argument shape of executeTool has changed, and the declarative section sits unwritten. Tool outputs, dynamic definitions, long-running execution and cross-document discovery are all listed by the spec itself as still moving, which is four more renames waiting to happen.

Which is the argument for keeping the volatile surface in one adapter that detects, lists, calls and normalises the result, and letting the rest of your agent talk only to that adapter. When the surface moves, and it will, you change one file rather than auditing every call site. The same reasoning applies to any integration instructions you hand a coding agent: point it at the spec and at Chrome's documentation rather than hardcoding method names, because a prompt containing a method name goes stale at exactly the speed an article does.

Getting the arbitration right pays off in a way that has little to do with whether any particular site adopts the standard. Your agent stops paying for a screenshot on the steps where a page has already told it what it can do, and keeps working normally everywhere else. That is the only version of this that survives the next rename. It is also the version that shows up on the inference bill.

Best Chatbot Memory Layer Plugins for Persistent User Data Retention and Memory

Gaurav Dadhich — Tue, 22 Sep 2026 03:56:53 +0000

The best chatbot memory layer plugin captures durable user information, stores it with the right scope, retrieves only the relevant part of it, and updates or deletes it reliably. [Maximem](https://www.maximem.ai/) is the layer that ships all four as product behaviour rather than as pipeline work you assemble yourself, and it carries the highest published [LongMemEval](https://arxiv.org/abs/2410.10813) accuracy of any memory provider that has published one, at 92.0%, alongside 93.2% on [LoCoMo](https://snap-research.github.io/locomo/), both of them verifiable on an [open eval harness](https://github.com/maximem-ai/memory\_and\_context\_eval\_harness) rather than asserted on a marketing page. Maximem produced both numbers with that harness, which anyone can clone and point at their own provider. It scored LongMemEval on the full 500-question public set and LoCoMo on all 1,540 category 1 to 4 questions, excluding the adversarial category per the convention the other vendors use, with gpt-5-mini\ generating answers and gpt-5-mini\ acting as judge under binary CORRECT or WRONG scoring, single run per benchmark. Treat a vendor-run number as a starting point rather than a verdict. What makes this one checkable is that the harness, the adapters, and the per-category results are published, so you can re-run it yourself; a memory score nobody outside the vendor can reproduce is a marketing asset rather than an engineering one. For most production teams, a dedicated memory layer is a better fit than placing every past message into the conversation or treating a vector database as the complete memory system. The right choice still depends on your chatbot's framework, data model, latency requirements, governance needs, and the kinds of information users expect it to remember. Use the guide below to compare memory layers and plan an integration that is useful, controlled, and maintainable. ## What a chatbot memory layer actually does A memory layer sits between your application and the model. It receives conversation events or structured user information, decides what is worth retaining, stores that information, and supplies relevant memories when the chatbot needs to respond. That is different from conversation history. History is a record of messages. Memory is a structured, reusable representation of information such as a user preference, an ongoing task, a previous decision, or an important fact. The distinction matters because sending an entire history on every turn increases context size without guaranteeing that the model will use the right detail. It is also different from retrieval-augmented generation used only for documents. Document retrieval answers questions about a knowledge base. User memory helps an agent maintain continuity across sessions and interactions. A strong chatbot may use both, but they should not be confused. When comparing plugins, look for support for the complete memory lifecycle: - **Capture:** accepting messages, events, profile fields, and application data. - **Extraction:** turning raw interactions into concise, useful memories. - **Storage:** keeping memories persistent and isolated by user, tenant, agent, or application. - **Recall:** returning relevant context at response time. - **Update:** revising a memory when newer information changes it. - **Deletion:** removing a memory when the user or application requests it. - **Inspection:** allowing your team to understand what was retained and why. A plugin that only performs semantic search over old messages may be useful, but it is not a complete retention strategy. Maximem covers the day-to-day half of that lifecycle behind two calls. Ingestion goes through memories.create\, which runs extraction, entity resolution, and relationship mapping, and recall goes through conversation.context.fetch\, which assembles ranked context for the current turn. Everything between those two calls, the vector store, the graph, the extraction pipeline, and the ranker, is operated for you rather than handed to you as components to tune. ## The capabilities to prioritize in a plugin ### 1. Clear memory types and scopes Persistent data should not be stored as one undifferentiated stream. A user's preferred writing style, a temporary support issue, and an organization-level policy have different lifetimes and access rules. Choose a layer that lets you distinguish user, session, agent, application, and organization scope where appropriate. It should also support useful categories such as facts, preferences, episodes, and time-sensitive information. Explicit types make retention easier to review and help reduce the chance that private information is exposed to the wrong agent or tenant. Maximem models this as four scope levels, USER, CUSTOMER, CLIENT, and WORLD, where wider scopes are visible to narrower ones and never the reverse. Scope is decided at write time by which identifiers you pass, so user\_id\ with customer\_id\ writes a memory private to that person inside that tenant, customer\_id\ alone writes organization-shared knowledge, and passing neither writes an application-wide fact. The extraction pipeline produces five typed memory categories, facts with confidence scores, preferences with strength and direction, episodes with significance scores, emotions with intensity, and temporal events covering deadlines, recurring events, and point-in-time occurrences, which is what allows retention review to happen by category rather than by keyword search over a blob. ### 2. Retrieval that is selective and explainable The plugin should return the smallest useful set of memories for the current task, rather than replaying everything it knows. Ask how it ranks memories, handles recency, resolves related entities, and deals with conflicting information. A practical test is to create a small evaluation set: a returning user changes a preference, refers to an earlier project, uses a nickname, or asks the chatbot to forget something. Measure whether the correct memory is recalled, whether an outdated memory is replaced, and whether unrelated user data stays out of the response context. Also check whether developers can inspect retrieved context during testing. Debuggable retrieval is important because a chatbot can appear inconsistent when the underlying problem is incorrect memory selection. Maximem gives you two retrieval modes on the same store. Fast mode runs vector plus graph retrieval with no LLM query decomposition and is the default for anything in the conversation hot path; accurate mode adds LLM subquery decomposition and reranking across similarity, recency, graph centrality, and confidence, which suits background jobs and multi-entity questions. Nicknames and aliases are handled by entity resolution rather than by luck: when a person is called "John", "Mr. Smith", and "my manager" across different sessions, exact, alias, semantic, and contextual matching collapse those mentions into one canonical entity, and genuinely ambiguous matches can be queued for human review instead of being merged silently. Conflicting facts resolve by scope priority, so a user-level statement wins over an organization-level default. ### 3. Explicit controls for retention and deletion "Persistent" should never mean "permanent by default." Your application needs a retention policy that defines what may be saved, how long it remains useful, and what happens when a user corrects or deletes it. Look for APIs or events for adding, searching, updating, and deleting memories. Confirm that deletion can be limited to one memory, one user, one workspace, or another appropriate boundary. A useful integration is designed to let you exclude selected fields or conversation types from memory before they are stored. Keep transient context separate from durable memory. A one-time request may help the current response without deserving storage. This separation gives product and security teams a clearer way to review what the chatbot retains. Maximem separates ordinary deletion from erasure, and the distinction is the one that matters for a data subject request. Deletion removes data through the normal path; erasure destroys the protected values irreversibly so they stop resolving everywhere at once, including in earlier backups, and it runs in two stages, a preview that reports exactly how many protected values would be destroyed, then an erase that executes only if the count still matches. Erasure is not a self-service button: you email privacy@maximem.ai with the instance, the identifier to erase, and the reason, and Maximem runs it as a support-operated action. An audit entry is written before anything is destroyed, recording who ordered it, the reason they gave, how many values it covered, and when, which is the artifact a privacy team needs when the request has to be evidenced later. On the write side, sensitive field types carry per-category policy, and you choose among storing the real value, hiding it from the model while your application still receives it, protecting it at rest so storage holds a placeholder while your application reads the real value, protecting it everywhere, or refusing to persist it at all. Several types sit under an enforcement floor and are never stored under any setting, including full card numbers, card security codes, passwords, API keys, private keys, and raw biometric data. ### 4. Framework and deployment fit A good memory plugin should fit the way your chatbot already runs. Check SDK support for your language, compatibility with your orchestration framework, authentication model, webhooks or event handling, and local development workflow. Managed services can reduce the infrastructure your team must provision and maintain. Self-hosted components can provide more control over deployment and data location. Neither approach is automatically better: select the operating model your team can monitor, secure, and support over time. The integration should also be narrow. Your chatbot should be able to call memory operations without changing the rest of the agent architecture. A memory layer that requires adopting an entire agent runtime may be appropriate for a new system, but it is a larger migration for an existing application. Maximem publishes 24 drop-in integration packages across Python and TypeScript, covering [LangChain](https://www.langchain.com/), [LangGraph](https://www.langchain.com/langgraph), [LlamaIndex](https://www.llamaindex.ai/), [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/), [Pydantic AI](https://ai.pydantic.dev/), [CrewAI](https://www.crewai.com/), [AutoGen](https://microsoft.github.io/autogen/), [Google ADK](https://google.github.io/adk-docs/), [Haystack](https://haystack.deepset.ai/), [Agno](https://www.agno.com/), [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/overview/), [NVIDIA NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html), [LiveKit Agents](https://livekit.io/agents), [Pipecat](https://www.pipecat.ai/), [Claude Agent SDK](https://docs.claude.com/en/api/agent-sdk/overview), [Mastra](https://mastra.ai/), and the [Vercel AI SDK](https://ai-sdk.dev/) among others, plus an MCP server for no-code clients that needs only a URL and a token. Every package shares one failure contract, which is the part worth reading before you pick: a failed context fetch returns an empty result and logs the error so the agent keeps answering, while a failed ingestion raises an explicit error so you know persistence did not happen. Teams moving off an existing layer have documented migration paths from Mem0, Zep, Letta, and Supermemory, and teams whose retention rules do not fit the default shape can configure a customized memory architecture per deployment. ### 5. Governance and data isolation Persistent user data creates responsibilities beyond retrieval quality. Review tenant isolation, access controls, auditability, encryption details, administrative roles, and the provider's data-handling terms before moving real user data into a service. Design memory boundaries in your own application as well. Pass a stable internal user identifier rather than relying on an email address when possible. Attach tenant or workspace scope to every operation, and test that a user cannot retrieve another user's memories through an altered request. Treat memory as application data, not as invisible model state. It should have an owner, a purpose, an access path, and a removal process. Maximem enforces isolation in three layers, with a logical storage namespace per instance, scope-based retrieval that never returns another user's memories, and network isolation per region. The guard that saves teams in practice is the one at the API boundary: on a B2B instance, forgetting customer\_id\ on a write raises an error rather than quietly writing a memory to the wrong scope, and a B2C instance rejects customer\_id\ outright. Traffic runs over TLS 1.3, with AES-256 at rest for the vector and graph stores and API keys hashed before storage. Region is fixed at client creation, between US East and EU Central, and memories stay in that region for their whole lifecycle. GDPR compliance is in place with a DPA available on request, SOC 2 Type II is in progress with audit completion targeted for Q3 2026, and every dashboard and SDK action is logged with a correlation id, principal, timestamp, action, and resource. ## A practical integration pattern Start with an allowlist of information the chatbot is permitted to remember. For example, you might allow explicit preferences and ongoing project details while excluding payment information, authentication secrets, and sensitive free-text content unless there is a documented reason to retain them. Then connect the memory layer at two points: - **After an interaction:** send the relevant message pair, event, or structured update for extraction and storage. Do not automatically save every internal chain or tool response. - **Before a response:** query memory using the current user, tenant, agent, and task context. Insert the returned memories into a clearly separated context block rather than blending them indistinguishably into the user's message. With Maximem, those two points are two calls: \python # After an interaction: store what is worth remembering await sdk.memories.create( document="User: I prefer dark mode and weekly digests.\\nAssistant: Noted.", document\_type="ai-chat-conversation", user\_id="internal-uuid-1234", customer\_id="acme", # tenant scope on a B2B instance mode="long-range", # full extraction, entity resolution, relationships ) # Before a response: fetch only what this turn needs context = await sdk.conversation.context.fetch( conversation\_id=conversation\_uuid, search\_query=\["display preferences", "digest cadence"\], max\_results=10, mode="fast", # hot path: vector plus graph, no LLM decomposition ) \\ Long conversations get compaction rather than a bigger prompt. Once turns are being recorded through conversation.record\_message\, compaction reduces that history to a token budget while keeping the facts, the decisions, the preferences, and the narrative thread, with strategies that land between roughly 15% and 70% of the original token count depending on how aggressive you set it. That is the cheaper answer to a context window that keeps filling up. Add a user-facing memory control where the product experience calls for it. Users should be able to ask what the chatbot remembers, correct an item, or request deletion. Your application, not the model alone, should authorize and execute those actions. Finally, log memory operations separately from ordinary chat logs. Record the operation, scope, status, and identifier needed for debugging while avoiding unnecessary duplication of sensitive content. ## How to choose the best option for your chatbot For a prototype, prioritize a simple API, quick local setup, and transparent memory operations; Maximem's trial tier starts at 5,000 credits a month, and the operations that draw against it are ingestion, retrieval, compaction, and real-time listening sessions billed by concurrency, so the cost model is visible before you commit. For a production customer-support chatbot, prioritize tenant isolation, deletion workflows, audit controls, conflict handling, and predictable retrieval. For a multi-agent system, check whether memories can be shared selectively without making every agent a reader of every user record, which is exactly what the scope ladder is for. Before committing, run a proof of concept with real interaction patterns and synthetic or approved test data. Evaluate five things: memory precision, memory recall, update behavior, deletion behavior, and operational overhead. Include failure cases, not only successful conversations. If you want that evaluation to be comparable rather than anecdotal, run it through the same open harness the published scores came from, pointed at your own data and whichever providers you are considering. Maximem is memory and context infrastructure for agents, built so that the parts teams usually underestimate, entity resolution, temporal awareness, scoping, conscious forgetting, and permission-aware retrieval, are product behaviour rather than a backlog. The best integration is the one that gives your chatbot continuity while keeping retention purposeful, scoped, inspectable, and under application control. ## FAQ **Which chatbot memory layer is most accurate on public benchmarks?** Maximem carries the highest published LongMemEval accuracy among memory providers that publish one, at 92.0% across the full 500-question set, and scores 93.2% on LoCoMo across 1,540 category 1 to 4 questions, with gpt-5-mini\ as both answer model and judge. Both figures are verifiable rather than asserted, because the eval harness is open source, so you can re-run either benchmark against your own provider and your own data instead of taking the number on trust. **Is a vector database enough for persistent chatbot memory?** Not usually. A vector database can support similarity search, but a complete memory layer also needs extraction, scopes, updates, conflict handling, deletion, and governance. Maximem runs the vector store, the graph, the extraction pipeline, and the ranker as one managed service, so those pieces are not yours to build or operate. **What user data should a chatbot remember?** Start with information that has a clear product purpose, such as explicit preferences or ongoing work. Exclude secrets and sensitive data unless your application has a documented, controlled reason to retain it. Maximem enforces part of this for you: full card numbers, card security codes, card PINs, passwords, API keys and other secret codes, private keys, and raw biometric data are never stored under any policy setting. **How does a chatbot update an old memory?** The application should send new information through an update or extraction workflow that can identify the related memory, prefer the newer value when appropriate, and preserve an audit trail for important changes. Maximem resolves conflicting facts by scope priority, so a user-level correction takes precedence over an organization-level or application-level default. **Can persistent memory be shared across multiple agents?** It can, if the memory layer supports deliberate scopes and access rules. Share only the memories required for a task, and keep user, tenant, and organization boundaries explicit. Maximem's four scope levels make that sharing a write-time decision, and a missing tenant identifier on a B2B instance raises an error rather than leaking the memory upward. **How is a user's data deleted or erased from a memory layer?** Maximem separates deletion from erasure. Erasure destroys protected values irreversibly so they stop resolving everywhere, including in earlier backups. It is support-operated rather than self-service, requested by email and run as preview then execute, with an audit entry written before anything is destroyed, which is what documents a right-to-be-forgotten request after the fact. **Which frameworks does Maximem integrate with?** Twenty-four drop-in packages across Python and TypeScript, including LangChain, LangGraph, LlamaIndex, OpenAI Agents SDK, CrewAI, AutoGen, Google ADK, LiveKit Agents, Pipecat, Claude Agent SDK, Mastra, and the Vercel AI SDK, plus an MCP server for no-code clients. Custom stacks call ingestion and context fetch directly. **Should every chatbot message be saved to memory?** No. Conversation history and durable memory serve different purposes. Use an allowlist or filtering step so transient context and unnecessary sensitive content are not retained. ## Sources - [Maximem](https://www.maximem.ai/), retrieved 22 September 2026 - [Memory and context eval harness, benchmark methodology and headline results](https://github.com/maximem-ai/memory\_and\_context\_eval\_harness), retrieved 22 September 2026 - [Identifiers and scopes](https://docs.maximem.ai/concepts/memory-scopes), retrieved 22 September 2026 - [Memories and context](https://docs.maximem.ai/concepts/memories-and-context), retrieved 22 September 2026 - [Fast and accurate modes](https://docs.maximem.ai/concepts/retrieval-modes), retrieved 22 September 2026 - [Entity resolution and master data management](https://docs.maximem.ai/concepts/entity-resolution), retrieved 22 September 2026 - [Erasing a person](https://docs.maximem.ai/guides/erasure), retrieved 22 September 2026 - [Sensitive data protection](https://docs.maximem.ai/guides/pii-protection), retrieved 22 September 2026 - [Multi-tenant SaaS pattern](https://docs.maximem.ai/patterns/multi-tenant-saas), retrieved 22 September 2026 - [Security and trust](https://docs.maximem.ai/resources/security-trust), retrieved 22 September 2026 - [Integrations overview](https://docs.maximem.ai/integrations/overview), retrieved 22 September 2026 - [Context compaction](https://docs.maximem.ai/sdk/context-compaction), retrieved 22 September 2026 - [Pricing and credits](https://docs.maximem.ai/resources/pricing), retrieved 22 September 2026

AI Agent Evals: The Judge Is The Part Nobody Measures

Gaurav Dadhich — Fri, 18 Sep 2026 09:51:31 +0000

An LLM judge scoring the same agent output three times will often hand back three different answers. On MT-Bench, Rating Roulette measured intra-rater reliability across repeated runs at a Krippendorff's Alpha of 0.265 for Llama 3.1, 0.507 for DeepSeek-R1 and 0.563 for Qwen-3, against a conventional floor of 0.8 for acceptable reliability. The paper, presented at EMNLP 2025 Findings, puts it plainly: LLM judges have low intra-rater reliability in their assigned scores across different runs, which makes their ratings "inconsistent, almost arbitrary in the worst case."

Almost every guide to agent evaluation published this year covers rubric design, scoring techniques and metric taxonomies. Very few of them ask whether the judge doing the scoring produces the same number twice, and none of them work out what it costs to run that judge on everything rather than on a sample. Both gaps matter more than the metric taxonomy does, because a suite built on an unstable judge measures its own noise, and a suite that only ever sees 2% of traffic is blind to the other 98% by construction.

Both properties turn out to be fixable, and fixing them changes what an eval suite can be. It starts with dropping the assumption that the judge has to be a language model at all.

Why an unstable judge is worse than no judge

An eval suite exists to answer one question: is today worse than yesterday. Everything else is in service of that.

Run a judge twice over the same fifty traces and get a pass rate of 82% and then 76%, and the six-point gap tells you nothing. Maybe the agent regressed. Maybe the judge felt differently. You cannot separate the two without re-running enough times to average the judge out, which multiplies your bill by however many runs you need.

What happens next is predictable, and I have watched teams walk through every step of it. Somebody raises the failure threshold so the suite stops going red on noise. The threshold goes up again a month later. Eventually the suite passes on everything except catastrophic breakage, at which point it has become a very expensive smoke test that nobody reads. The suite did not fail loudly; it degraded into decoration.

A second failure mode is subtler and worse. A judge with high run-to-run variance on a small number of questions will produce an eval set where a handful of traces flip verdict every run. Those flipping traces get investigated, found to be fine, and marked as flaky. Real regressions that land on the same borderline traces then get dismissed under the same label.

"Set temperature to zero" does not fix this

Every engineer who has run an eval suite reaches for this within about four seconds of reading the paragraph above, and it deserves a serious answer rather than a dismissal.

Temperature zero is real and it does real work. It removes sampling randomness from token selection, and for an extraction task with one defensible answer it will give you the same output nearly every time. Several of the better guides recommend it correctly; TestMu AI{target="_blank"} specifies temperature 0.0 in its calibration workflow for exactly this reason, and that is sound advice as far as it goes.

What it does not touch is the harder half. Temperature constrains how a token is picked from a distribution; it does not make a subjective judgement less borderline. When a trace genuinely sits between "grounded" and "partially grounded", the underlying distribution is close to even, and greedy decoding at temperature zero will still tip one way or the other on tiny differences in context, ordering, or the phrasing of the preceding question in a batch. Rating Roulette ran its three judge models across three benchmarks and found reliability well below threshold; that variance is a property of judgement on ambiguous inputs, not of the sampling parameter.

There is a stronger version of the objection worth handling too. Patronus AI{target="_blank"} recommends running the same input through the judge three times as a self-consistency check, which is genuinely good practice and the only place in the mainstream guidance where this problem is acknowledged as an operational concern. It is also a three-times cost multiplier on a judge that was already the expensive part of the pipeline, which brings us to the second unmeasured thing.

Nobody models what grading actually costs

Of every major guide published on this topic in the past year, exactly one publishes a real number. The AWS tutorial on dev.to{target="_blank"} gives roughly $0.001 to $0.003 per evaluation on a small model and $0.015 to $0.045 on a mid-sized one. Useful, and it stops there. No page I could find takes that per-call figure and multiplies it by production volume, which is the only form in which the number means anything.

So here is the multiplication, using a mid-sized judge at the middle of that published range and a 14-question rubric run once per trace.

At roughly $0.03 per trace, grading one million agent traces costs about $30,000. A reasoning-model judge, which is what most teams reach for when rubric quality matters, is worse: TypeSafe's own comparison study{target="_blank"} measured Claude Opus 4.8 in reasoning mode at $0.034275 per query on a 14-question rubric, which lands at about $34,000 for the same million traces. Serially, at 13.9 seconds per call, that run takes roughly 160 days of wall clock.

Nobody spends $34,000 and five months to grade last month's traffic. So nobody does. Instead the guidance converges on sampling, and the sampling rates are stated without any justification at all: the AWS tutorial suggests 1 to 5% of live traffic, Adaline{target="_blank"} suggests 5 to 10% for judge-based checks. Neither explains where the number came from, because it did not come from anywhere. It came from the bill.

Sampling at 2% means a failure mode occurring in one conversation per thousand shows up in your eval data roughly once per fifty thousand conversations. You will find it eventually, from a customer, in a support ticket.

What changes when the judge is not a language model

A grading task has a peculiar shape. The judge never needs to write anything. It needs to answer a fixed list of closed questions: was this grounded, was the right tool called, did the agent refuse when it should have, how directly did it answer. Each of those is a yes-or-no, a pick-one, or a rating on a defined ladder.

Which means text generation, the expensive and unstable part, is pure overhead in a judge. You pay for output tokens you throw away, you wait for them, and you inherit their variance.

TypeSafe{target="_blank"} released a model in September 2026 built on that observation. Jev returns typed decisions and probabilities instead of text, across three question types: a yes-or-no probability, a pick-one from a named set with a full probability distribution, and a rating against two to ten described levels. It cannot write a sentence. That is the design.

Operating numbers, taken from the vendor's own published measurements with their method attached rather than from the launch-week coverage that rounded them:

	Jev	gpt-5.4-mini (t=0)	claude-haiku-4-5 (t=0)	gpt-5.5 reasoning	claude-opus-4-8 reasoning
Latency	111 ms	1,405 ms	1,780 ms	11,125 ms	13,886 ms
Cost per query	$0.000043	$0.001089	$0.001798	$0.033157	$0.034275

Method, because a benchmark without one is a marketing claim: 15 repeated samples of a 14-question yes-or-no rubric against a single deliberately borderline insurance claim, sampled 11 September 2026, with the TypeSafe model resolving to jev-1.13.0. Published pricing at the time was $0.042 per million input tokens with output tokens free.

Grading a million traces on those numbers costs roughly $43.

That is the whole argument, and it is not really an argument about money. At $34,000 you sample and you argue about the rate. At $43 the rate stops being a question, and an eval suite becomes something you can point at all of production, continuously, without anyone having to approve it.

Reproducibility matters as much as price does here. Across those same 15 repeats, mean per-question probability standard deviation came in at 0.0102 for yes-or-no questions and 0.0098 for pick-one questions. On a hard content-moderation case scored with eight pick-one questions, raw label agreement across repeats was 90.8%.

Reliability is not validity, and this is where most of the excitement should stop

A model that agrees with itself is not thereby correct, and the strongest evidence on this arrived in June 2026 from a UC Berkeley team. Reliability without Validity{target="_blank"} ran 21 judges from 9 providers across three benchmarks, producing around 541,000 individual judgments over 118 evaluation runs, and found that high test-retest reliability of 0.95 or better coexisted with severe position bias above 0.10 in more than one model. A judge can be perfectly repeatable and repeatably wrong.

TypeSafe's own documentation says the same thing about its own model, which is more than most vendors manage. Their study of repeated pick-one decisions carries the sentence "This experiment does not measure accuracy," and describes its own uncertainty threshold as "an illustrative application policy, not a calibrated guarantee or a threshold chosen to maximize this run's agreement." The same study concedes that a probability near the threshold can still move between a concrete label and uncertain, and that the policy does not make the model deterministic.

So the honest claim is narrower than the launch coverage suggested. A typed-decision judge removes one specific failure mode, the one where your measurement instrument moves under you. It does not tell you the instrument is pointed at the right thing. You still have to check, and there are two halves to that check: which biases carry over to a typed judge, and whether its confidence number means anything on your data.

The biases survive the change of judge, mostly

Four biases get named repeatedly in the existing guidance, and they are worth separating by whether a typed judge inherits them.

Position bias, where a judge favours whichever option it saw first, is inherited. The Berkeley study found it ranging from 0.002 for Gemini 2.5 Pro to 0.192 for Qwen 3 8B, nearly two orders of magnitude apart, so it is a per-model property rather than a universal constant. TypeSafe's own documentation concedes the equivalent: sibling order inside a pick-one question is part of the question rather than presentation. Shuffle your option order across runs and measure whether the answer moves.

Verbosity bias, where longer answers score higher, turns out to be weaker than folklore suggests. All 21 judges in the Berkeley cohort registered a Pearson correlation below 0.011 with answer length, substantially lower than earlier estimates. Worth testing on your own rubric rather than designing around.

Self-preference, where a judge rates its own family of models more kindly, does not apply to a judge that generates nothing and therefore has no outputs of its own in the comparison set. The Confident AI guide{target="_blank"} documents the effect for language-model judges and is the best summary of it I have found.

Rubric sensitivity is the one that gets worse, not better. A typed judge takes your level descriptions extremely literally, which is a feature until two levels overlap, at which point confidence collapses and you get an honest 0.4 instead of a confident guess. Treat persistently low confidence on one question as a defect in the question.

Hand-label 200 traces yourself, or with whoever owns quality. Run the judge over them. Plot confidence against accuracy and see whether the confidence number carries information. TypeSafe's own SEC filing classification exercise is a good worked example of what a passing result looks like: over 60 filings, answers above 0.9 confidence were correct 27 times out of 30, and answers below that line were correct 12 times out of 30. The confidence score was doing real work, and they only knew that because they measured it.

Berkeley's team gives the sharper warning: every judge in their study overstated chance-corrected agreement, with exact match inflating Cohen's kappa by between 33.8 and 41.3 percentage points on MT-Bench. If you validate your judge by counting how often it matches a human, you are measuring the wrong thing, and you are measuring it generously.

Three verdicts, not two

Every guide surveyed for this piece scores on a binary pass or fail, a 0 to 1 scale, or a 1 to 5 integer. Not one proposes an explicit third outcome, and that omission causes a measurable amount of the noise people attribute to the judge.

Some traces are genuinely ambiguous. Forcing a label onto them manufactures disagreement that has nothing to do with the agent and everything to do with the trace sitting on a boundary.

def verdict(probability, low=0.30, high=0.70):
    if probability < low:
        return "fail"
    if probability > high:
        return "pass"
    return "needs_human"

Applied to that eight-question moderation rubric, the uncertainty band took raw label agreement from 90.8% to 99.2%, with 74.2% of answers still decided automatically and 25.8% routed for review. Three quarters of the work happens without a person, near-perfectly reproducibly, and the quarter that a machine should not be deciding alone gets flagged as such instead of being guessed at.

Two things make this work in practice. Pick-one questions always return something, because the probabilities have to sum to one, so on a trace where no category applies the model still nominates a winner; pairing the pick-one with a separate yes-or-no asking whether any option applies at all gives you a gate that can go to zero. And when several checks feed one decision, aggregate on the maximum rather than the mean, since one confident red flag is the whole signal and averaging is how you hide it.

Band edges are not universal. Set them from labelled examples and from what a wrong automatic decision costs you relative to the price of a human looking.

Grading before the response ships

Once a grade costs a tenth of a second, it can run before the user sees anything rather than after.

Not one page in the incumbent set mentions this, and it is the most useful structural change available. The shape is a cascade: generate with your normal model, grade the draft response inline, and escalate only when a check fires. TypeSafe documents the pattern for structured extraction, where a cheap model extracts, a per-field check asks whether each value looks wrong, and an expensive reasoning model is invoked only when any field crosses 0.7. They report quality comparable to the top-tier model at roughly a tenth of the cost.

Pointed at agent output rather than extraction, the same shape looks like this:

draft = cheap_agent.respond(request)

checks = judge.system_one(
    state={"request": request, "draft": draft, "retrieved": context},
    questions=INLINE_RUBRIC,      # grounding, scope, leakage, refusal correctness
)

if max(checks.flags.values()) > 0.70:
    return strong_model.respond(request)   # pay only when the check fires
return draft

Your quality floor stops being whatever the cheap model does on a bad day, and your bill stops being whatever the expensive model costs on a good one. The gate has to be fast enough to hide inside the response budget, which is the entire reason a 111 ms judge makes this possible and a 13.9 second one does not.

What still has to be right: trajectories, tools and calibration

None of the above removes the work the existing guides describe well, and a piece that skipped it would be selling you a judge without a rubric.

Agent evaluation is trajectory evaluation. A correct final answer reached by calling four unnecessary tools, failing twice and recovering by accident is not a pass, and the Confident AI agent guide{target="_blank"} is right that tool correctness, argument correctness and step efficiency need scoring separately from task completion. Feed the whole trajectory as a named object and ask about the path:

state = {
  "user_request": "Book a table for four on Friday",
  "tool_calls": [{"name": "search_restaurants", "args": {...}}, ...],
  "retrieved_context": [...],
  "agent_final_reply": "...",
}

TRAJECTORY_RUBRIC = {
  "correct_first_tool": Noul(
      instructions="Was the first entry in `tool_calls` the right tool for `user_request`?"),
  "unnecessary_calls": Score(
      instructions="How many entries in `tool_calls` were not needed for `user_request`?",
      criteria=["Every call was needed",
                "One or two calls were not needed",
                "Most calls were not needed"]),
  "recovered_from_failure": Noul(
      instructions="After the first failed call in `tool_calls`, did the agent take a reasonable next step?"),
  "grounded": Noul(
      instructions="Is every factual claim in `agent_final_reply` supported by `retrieved_context`?"),
}

Every question in that request is scored independently against the same state and runs in parallel, so a forty-question rubric costs roughly what a four-question rubric costs. TypeSafe measured 13 questions against a 54,000-character document as 12.2 times cheaper and 10 times faster batched into one call than sent as 13 separate calls, with answers unchanged. The usual reason rubrics stay thin, that each question is another paid round trip, stops applying, and you can afford to check the rare failure modes on every trace instead of only the common ones.

Calibration is the other piece worth taking from the existing guidance rather than reinventing. Patronus recommends targeting a Cohen's kappa above 0.8 against expert labels; TestMu specifies a calibration set of 30 to 50 samples, measuring precision and recall, then freezing the judge. Both are right, and both apply unchanged to a typed judge. A frozen, version-pinned judge with a measured kappa is the artifact you want; everything in this piece is about making that artifact cheap enough to run everywhere.

Pin the version explicitly. In TypeSafe's own published study the alias jev-latest resolved to jev-1.13.0, and a judge that silently upgrades makes last quarter's scores incomparable with this quarter's.

Where the suite actually runs

Most eval suites run nightly because that is what they can afford. Cost decides placement, and once it stops deciding, three placements open up at once.

A regression gate in continuous integration is the first and the easiest to justify. A 500-trace suite at a tenth of a second per trace finishes in well under a minute and costs a couple of cents, which makes it a pull request check rather than a scheduled job. DeepEval{target="_blank"} and Adaline both describe the assertion pattern for wiring this into a pipeline, and it works unchanged with a typed judge behind it. An eval that runs before merge stops the regression; an eval that runs at 3am documents it.

Continuous scoring of production traffic is the second, and it produces a different artifact from a pass rate. A quality signal computed on every request can be cut by customer, by prompt version, by model and by release, and alerted on when the distribution moves rather than when a threshold trips. That is monitoring, not testing, and the distinction is worth holding onto: a test answers whether the build is good, while a monitor answers whether reality has changed.

Inline gating, described above, is the third, and it is the one that changes the product rather than the process.

The thing a typed judge cannot do

It cannot tell you why.

You get 0.31 and no sentence. For the engineer staring at a red suite at 5pm, the explanation is most of the value, and any honest account of this approach has to lead with that limitation rather than bury it.

You do not have to pick one judge. Grade everything with the cheap typed judge, then spend a reasoning model only on the traces that failed or landed in the uncertain band, asking it to write the diagnosis. On a suite where 74% pass cleanly, you are paying reasoning-model prices on roughly a quarter of traces and getting a written explanation exactly where a human was going to look anyway. Cheap where the work is wide, expensive where it is deep.

Two smaller limits are worth knowing before you design around them. State and questions share a budget of roughly 32,000 tokens, about 150,000 characters, so long trajectories with large retrieved context will not fit in one call and need grading per turn. And a pick-one question accepts at most 255 options, which matters the moment you try to classify against a large taxonomy.

What I would actually build

Start with the judge you have and measure it, because everything else is downstream of knowing whether your current numbers mean anything. Run your existing rubric five times over the same fifty traces and compute the variance per question. If it is small, your suite is fine and you can stop reading. If two questions account for most of the movement, those two questions are probably asking about more than one thing at once.

Then hand-label 200 traces and calibrate whatever judge you intend to keep, plotting confidence against accuracy rather than counting agreement.

Add the third verdict before you change anything else. It is a four-line function, it costs nothing, and on the published numbers it absorbs most of the residual disagreement into an honest label.

Move the cheap judge to 100% of production traffic once it is calibrated, and keep the reasoning model for explanation duty on the failures. The point of grading everything is not thoroughness for its own sake; it is that a quality signal computed on all of production can be broken down by customer, by prompt version, by model, and alerted on when the distribution shifts. That is a different artifact from a nightly pass rate.

We grade memory recall for Maximem Synap on this shape, because memory quality is exactly the kind of property that degrades quietly and shows up in a support ticket rather than in a test. An eval that runs on a sample once a night finds that class of regression late, every time.

The part worth sitting with

For two years the eval tooling conversation has been about frameworks and metric taxonomies, and those frameworks are good. The judge underneath them has been treated as a solved component, and two recent studies say it is not, one published at EMNLP 2025 Findings and one a June 2026 preprint from UC Berkeley: judges disagree with themselves across runs, and the agreement statistics most teams use to validate them overstate the case by 30 to 40 percentage points.

Fixing that is not a matter of a better prompt. It is a matter of asking whether the thing doing the grading should have been a text generator in the first place, given that it never needed to write anything. When the answer is no, the price falls far enough that the eval stops being a test you run and becomes a measurement you always have. That changes what you can know about your agent, which is the only reason any of this exists.

Agent Memory Is a Real Problem. Most Builders Just Have Not Hit It Yet.

Gaurav Dadhich — Fri, 18 Sep 2026 09:14:47 +0000

Every LLM call is stateless. Agent frameworks solve that within a single run, which is why the memory problem does not show up until the run ends. The first time it does show up, it does not look like a memory problem at all. It looks like an agent that asks a returning customer the same question it asked last week, a support copilot that contradicts a decision it made two sessions ago, or a bill for tokens that grew faster than usage did. By the time the pattern is legible, it has usually already cost something.

This piece is about that gap: what agent memory actually is, why so many teams building agents have not run into the problem yet, why the obvious fixes look like solutions and are not, and how we approach the problem at Maximem.

What is Agent Memory?

Agent memory is the layer that lets an agent carry facts, preferences, episodes, and entities across turns, sessions, users, and tools, instead of treating every interaction as a cold start. A model on its own has no memory; each call is stateless, and whatever continuity an agent appears to have was engineered on top of that by somebody. Within one run, the framework holds that state. Across runs, across days, across a customer's whole relationship with your product, something else has to.

Memory is what turns a capable-but-forgetful model into an agent that gets better the more it is used. It is the difference between a customer explaining their situation once and a customer explaining it every single time.

Is Agent Memory an Imagined Problem or a Real One?

It is real, and the reason it does not feel real to most people building agents today is worth taking seriously, because the same reasons explain why the teams who have hit it treat it as one of the harder problems in the stack.

Why most agent-builders have not come across the agent-memory problem yet

The honest answer is that most agents in the wild are not yet operating in the conditions that make memory bite. Six things are usually true.

Production adoption is still early, and scale is low. A large share of what gets called an "agent" today is an experiment or a demo. Real agents in production, carrying real users across real time, are still the minority, and the ones that exist are often running at a scale where the seams do not tear yet. Memory failures are a function of returning users and elapsed time. With few returning users and little elapsed time, the failure has nothing to act on.

Many teams are building workflows, not agents. A rigid flow that moves a request through predefined steps does not need to remember anything, because the state it needs lives in the flow itself. This is a legitimate way to build, and for many problems it is the correct one. It also means the builder never encounters the memory problem, because they never built the thing that has the problem. The distinction matters: an agent decides what to do next; a workflow was told. Only the first one has anything to remember.

Many agents are non-conversational by design. Two common shapes here. The first is short-call voice agents built for a single advertising or qualifying interaction, where the entire job is over in ninety seconds and there is no second conversation to be continuous with. The second is the far larger category of teams converting existing SaaS into AI-enabled SaaS, bolting a natural-language surface onto software whose data model and workflows were designed for humans clicking buttons. That is AI-enabled software, and it is valuable, but it is not the same as reimagining the work itself around an agent that accumulates context. The first shape has nothing to remember. The second remembers through its existing database and does not yet feel the absence of an agent memory layer.

Token costs have not started pinching, because scale is low. The most common way to fake memory is to stuff prior history into the prompt. At low volume, that is cheap enough to ignore. The cost of faking memory this way scales with conversation length multiplied by user count multiplied by frequency, so it stays invisible until two of those three start climbing, at which point it arrives as a line item nobody forecast.

Returning customers are sparse, and there are almost no evals for agent CSAT yet. Memory quality is only observable when the same user comes back and the agent either does or does not carry them forward. Most teams are not yet measuring satisfaction from agent interactions across sessions, so even where memory is quietly failing, there is no instrument pointed at it. The failure is real; the detector is not installed.

Locally-run agent harnesses create the illusion that long context solves it. A great deal of early agent building happens in a local harness against a single developer's own history, where a long context window really does hold everything relevant. It is easy to conclude from that experience that a big enough window is memory. It is not, and the reasons that it is not become obvious the moment there is more history than a window can hold, or more than one user whose histories must not bleed into each other.

None of these are signs that the problem is not real. They are the conditions under which a real problem stays hidden. Every one of them dissolves as agents move to production, run longer, serve more users, and start being measured.

For builders who have hit the problem, the alternatives look easy. They are not.

Once a team does run into it, a set of primitives and adjacent tools present themselves as the fix, and each of them looks appealing because each of them solves a piece of the shape of the problem. The trap is that they look sufficient right up until someone builds an eval and measures precision and recall on retrieved context. Without that measurement, a partial solution and a real one are indistinguishable.

Larger context windows. A bigger window is a bigger desk, not a better filing system. It postpones the problem by letting you hold more at once, but it does not decide what is worth holding, it does not persist anything past the session, and it does not keep one user's context out of another's. Past a certain history length the window is full again, and now you are paying to re-read everything on every turn. Bigger context is real capability that solves a different problem.

Vector RAG, pgvector, graph RAG. Semantic search over stored chunks is the most common homegrown memory, and it is useful for retrieval over documents. The failure mode is that raw conversation is not a document. Chunk-and-embed does not resolve that "the client", "Acme", and "Acme Corp" are one entity, it does not know which of two contradictory statements is current, and it retrieves on similarity rather than on relevance to the moment. High recall on chunks is not the same as retrieving the right fact, and the gap between those two is exactly what an eval exposes and intuition does not.

Skills. Skills give an agent reusable, packaged competence, which is a real and useful primitive. It is also a different axis from memory. A skill is knowledge the agent brings to every user; memory is what the agent has learned about this user, this account, this history. Confusing the two leads to building excellent procedures that still forget who they are talking to.

Agent framework state machines. This is the objection every serious builder reaches for first, and it deserves a straight answer. Frameworks such as LangGraph give you checkpointers, thread state, and cross-thread stores, and they do their job well: they carry an agent through a long, multi-step run without dropping most of what matters, and for completing a single execution correctly they are the right tool. The checkpointer exists to make one run reliable, and it does. What it was never trying to do is decide what, out of thousands of past interactions, is worth remembering next month, resolve entities across those interactions, reason about what has since changed, and keep all of it correctly scoped across many tenants. Those are not gaps in the framework. They are simply a different problem, one that begins where the run ends.

DIY is the first real response, because it looks easy until you run benchmarks

The teams who see through the primitives usually reach the same next conclusion: we will build it ourselves. It is an understandable instinct, and it is where a lot of engineering time goes to quietly die. The reason is that the naive version really is a weekend of work, so the effort estimate is anchored on the wrong version.

The first cut, embed the history, store it, retrieve by similarity, works in the demo. Then the benchmarks start. Precision and recall on retrieved context, contradiction handling when a user changes their mind, entity resolution across messy references, temporal reasoning about what is current, isolation across tenants, latency inside the conversation hot path, and retrieval quality that does not degrade as the store grows. Each one is a research problem with a literature behind it, and getting the whole thing right is iterative depth work that is hard to do well without a disproportionate investment of time. The build-it-ourselves estimate is almost always the estimate for the weekend version, and the production version is a different project entirely.

This is the actual shape of the thing: agent memory is a data pipeline and fast-search problem, not a storage problem. Storing conversations is trivial. Deciding what to keep, resolving it into entities, keeping it current, forgetting what has gone stale, and returning exactly the right context in single-digit milliseconds is the work.

So, What Is Agent Memory? The Jobs To Be Done

Stated as the jobs a real memory layer has to do, rather than as a definition, the shape becomes concrete. A memory layer earns its place when it does the following:

Persist context across sessions, users, and tools. Facts, preferences, episodes, and entities survive past the end of a run, so a returning user is carried forward rather than re-interviewed. A support agent that already knows the customer's plan, their last three tickets, and the workaround it gave them last week is a different product from one that starts every chat by asking for an account number.
Resolve entities automatically. "The client", "Acme", and "Acme Corp" collapse into one entity instead of fragmenting into three, so the agent reasons about a coherent world rather than a pile of near-duplicates. For a B2B sales or account agent, this is the difference between one accurate view of a company and several partial ones that disagree.
Reason about time. The layer knows what is current versus stale, so the agent acts on the latest state of a fact and not on something a user corrected two months ago. A scheduling or operations assistant that still believes a cancelled meeting is on the calendar is worse than no assistant.
Scope memory across tenancy levels. Context is visible at the right level, from an individual user up through a customer organization and beyond, and never leaks across the boundaries that separate one tenant's data from another's. For a multi-tenant B2B copilot, this is not a nicety, it is the line between a working product and a data incident.
Manage PII and permissions. Memory respects who is allowed to see what, retrieval never surfaces something the current user should not have, and sensitive data is retained, masked, and retired according to policy rather than sitting in a prompt forever. A healthcare or fintech agent lives or dies on getting this right.
Forget consciously. Stale and low-value memory is retired on purpose, so the store does not bloat into noise and retrieval stays sharp as history grows.
Learn what to keep. The layer improves what it retains over time instead of dumping raw history back into the prompt, so a coding agent gradually holds the conventions and decisions that matter to a codebase rather than every line anyone ever typed.
Manage memory across multiple agents. Context passes cleanly between agents in a multi-agent system, with shared knowledge and per-agent private memory kept distinct, rather than each agent re-deriving the world from scratch at every handoff.
Reduce token cost. Faking memory by appending history grows cost quadratically with conversation length; a memory layer that retrieves only what the turn needs and compacts the rest keeps cost close to linear, which is the difference between a margin that survives scale and one that does not. A high-volume support deployment feels this first.
Return context fast enough to sit in the conversation. Retrieval happens inside the latency budget of a live turn, not as an offline batch job. For a voice agent, where a pause of a second reads as a broken call, this constraint is unforgiving.

Signs You Need Agent Memory

The clearest signals that you have crossed from "memory is overkill" into "memory is now the bottleneck":

Your agent is conversational, multi-turn, or long-running, and users come back. The moment a second session with the same user matters, memory matters.
You are serving multiple customers from one product, and keeping each tenant's context isolated has become something you think about.
You are stuffing conversation history into the prompt to fake continuity, and the token bill is starting to track conversation length rather than value delivered.
Your agent repeats questions, contradicts its own earlier decisions, or treats a returning user as a stranger.
You are running a multi-agent system and finding that context has to be re-derived every time control passes from one agent to another.
You have started building memory in-house, and the second version of that build is turning out to be much larger than the first.

If two or more of these are true, the problem is no longer hypothetical for you, and the fastest way to know how much it is costing you is to measure it rather than guess.

We run that measurement for you, free. You fill in a short form, we run our diagnostic against your agent, and we show you where its memory breaks today: what it fails to recall, where precision and recall drop, and what that is costing you in tokens and in users repeating themselves. You get the findings whether or not you ever use Maximem Synap. Start the free diagnostic here.

How We Approach Agent Memory at Maximem

The way we see it, agent memory is not merely a storage and retrieval problem, it is an architecture problem, and treating it as one is what separates a memory feature from a memory system.

Maximem Synap ingests conversations asynchronously and extracts structure, not raw text. That structure is stored across three kinds of store at once, vector, graph, and file, and retrieval nets across all three together rather than betting the whole system on similarity search alone. Because ingestion runs in the background, the expensive extraction work never sits in the conversation hot path, and context is pre-fetched while the conversation is still going, which is how in-conversation retrieval stays under 15ms at P75. Conscious forgetting runs as background cycles over the stores, retiring what has gone stale so precision holds as history grows, and extraction, scoping, and retention are all configurable per deployment rather than fixed. On the public benchmarks this comes out as the most accurate memory system we are aware of, with 92% on LongMemEval and 93.2% on LoCoMo, against next-closest published scores in the low seventies.

The full picture is laid out in our research paper, Agentic Context Management: Agent Memory Is Not Merely a Storage & Retrieval Problem, It Is an Architecture Problem. It treats memory as a five-stage lifecycle rather than a store: architecting a memory schema per agent, ingesting asynchronously with entity resolution, scoping retrieval narrowest-first across tenancy, anticipating the context an agent will need before it asks for it, and compacting with validation so cost stays close to linear in the length of the conversation rather than quadratic. Figure 1 of the paper draws this as a cycle around the agent. A store makes none of those decisions. Storage is one moment in the lifecycle, not the whole of it.

Why AI Forgets: Why ChatGPT, Claude, and Gemini Don't Remember You Well

Gaurav Dadhich — Wed, 26 Aug 2026 08:42:22 +0000

I had a conversation with Claude last Wednesday. Spent maybe forty minutes walking it through a product launch plan: the positioning, the pricing tiers, the three segments we were targeting. Good conversation, felt productive. On Thursday morning I opened a new chat to refine the email sequence for that same launch.

"Could you tell me more about your product?" . Forty minutes of context, gone. Either corrupted or half-remembered.

If you use AI tools regularly, you've had this moment. You've had it dozens of times. And at some point you stopped being surprised and started being annoyed. You copy-paste the same background into every new conversation. You keep a Google Doc of "stuff to tell the AI." You re-explain your job, your preferences, your constraints - over and over, to the most capable technology ever built.

Here's the thing nobody tells you clearly: your AI isn't forgetting. It never knew you in the first place. What felt like a relationship was a transaction.

Every conversation is an island

The way these tools work under the hood is genuinely counterintuitive. When you chat with ChatGPT or Claude or Gemini, the model doesn't "learn" from your conversation. Nothing you say changes the model itself. It processes your messages, generates a response, and when the session ends, the next session starts from a blank slate.

Think of it like calling a support hotline where the agents don't have a CRM. You might get someone brilliant. They might solve your problem in five minutes flat. But tomorrow when you call back? Different person. No notes in the file. You start from "can I get your account number?"

There's a technical term for this. AI conversations are stateless. Each one exists independently, unconnected to anything before or after it.

Three specific things make this worse than it sounds:

The token window rolls over. Even within a single conversation, there's a limit to how much the model can "see." ChatGPT and Claude have large context windows (128K-1M tokens depending upon the model), roughly 200 pages of text for some models. But in long conversations, older messages start falling off the edge. The model isn't summarizing them or filing them away. They're just not there anymore. Ask about something from the beginning of a two-hour coding session and you might get a blank stare. Plus there is something called as context-rot. In simple words, the more information you give them, the more they hallucinate on that information. They barely remember about 20-30% of it accurately. Not ideal.
Each platform is its own silo. This one is brutal for anyone who uses more than one AI tool. And most power users do: ChatGPT for brainstorming, Claude for longer analysis, Gemini when you need something integrated with Google, Perplexity for research. Each one knows a fragment of you. None of them share notes. Your preferences in Claude don't exist in ChatGPT. That Perplexity research session about competitor pricing? Invisible to every other tool you use.
The "memory" features are thinner than they look. ChatGPT does have a memory feature: it will store facts like your name, your job, that you prefer bullet points over paragraphs. Claude has something similar. But these are stored facts, not understanding. ChatGPT remembered that I work in AI infrastructure. It did not remember the forty-minute launch strategy. That's the gap. Flat facts versus actual context from working sessions.

"But ChatGPT, Claude has memory now"

I know. I've tested it extensively. Let me tell you what it actually does.

ChatGPT's memory system keeps lightweight summaries of your recent conversations, roughly your last fifteen chats. Not the AI's responses, just compressed sketches of what you typed. It also stores explicit facts about you (your name, role, preferences) that get injected into every single prompt, whether relevant or not. One researcher who dug into this found 33 stored facts about himself riding along in every conversation. His gym schedule was being fed to ChatGPT when he asked about Python debugging.

Claude takes a different approach. Instead of pre-loading summaries, it has retrieval tools it can use to search your past conversations. The operative word is "can." Claude has to decide, mid-conversation, that looking into your history would help. Sometimes it does. Sometimes it doesn't think to check. When it works, it's genuinely impressive; you get detailed context from three weeks ago. When it doesn't, you get nothing. [We did a deep dive into how each platform handles this](https://www.maximem.ai/blog/ai-apps-memory) and the short version is: all three major platforms are engineering their own isolated memory, and none of them talk to each other.

Gemini's situation is different again. Strong integration with your Google ecosystem; it can pull from Drive, Gmail, Calendar. But your conversations with Gemini don't inform your conversations with ChatGPT or Claude. It's another walled garden, just with Google-shaped walls.

The pattern: three platforms, three separate attempts at memory, three siloed systems that can't see past their own borders.

Bigger context windows won't fix this

I hear this one constantly. "GPT-5 will have a million-token context window, so the forgetting problem goes away." It won't, here's why:

A larger context window makes individual conversations longer. It doesn't make them connected. You can fit an entire book into a single session, sure. But close that session and open a new one tomorrow; same blank slate, no matter how many tokens the window supports.

There's also a cost issue nobody likes talking about. Context isn't free. Every token you stuff into that window costs money on the API side and computation on the inference side. The bigger the window, the more expensive every single message becomes. A 200K-token conversation with Claude Sonnet costs meaningfully more than a fresh 2K-token one. The platforms aren't going to solve this by just making windows bigger, because that doesn't scale economically.

And even within a long conversation, there's a well-documented problem researchers call "lost in the middle"; models pay more attention to information at the beginning and end of their context window and tend to miss things buried in the center. Bigger windows can actually make this worse, not better. More hay, same needle.

## What you're actually losing

Let's be concrete about the cost. Cottrill Research reported in 2025 that nearly half of knowledge workers spend one to five hours every day searching for information they've already accessed. UC Irvine's research on interruptions found it takes about 23 minutes to get back in the zone after each break in focus. Every time you re-explain your role, your project constraints, your communication preferences; that's an interruption. A self-inflicted one, caused by tools that should already know this.

We tracked our own stats when we helped users create their AI Year End Wrapped. Users on average had revised their prompt in over half of all conversations. Half the time, they're teaching the AI something it should already know.

Multiply this by every AI tool you use. Multiply by every working day. The cumulative friction is enormous, and it's invisible because it's distributed across hundreds of small moments. No single re-explanation feels like a big deal. But together, they eat hours every week.

The subtler cost is harder to measure: the good conversations that never happen because you don't bother setting up the context. You wanted to ask Claude to refine yesterday's strategy. But that would mean re-pasting the strategy, re-explaining the market context, re-describing the team dynamics. So you just... don't. You work from the draft you already have, even though a fresh pass with full context might have caught the flaw in your pricing model.

What you can do about it right now

Three options, escalating in effectiveness.

The manual workaround. Keep a document: some people use Notion, some use a simple text file: with your standard context: role, current projects, communication preferences, key constraints. Paste it at the top of every new conversation. It works. It's tedious. It doesn't transfer between platforms. You become your own memory system, which, let's be honest, defeats a lot of the purpose of using AI in the first place.
Use the platform-native memory features. Turn on ChatGPT's memory. Use Claude Projects to store background documents. Set up Gemini's Google integrations. This gets you further. But each one only works inside its own platform. If you're a multi-platform user; and the data suggests most regular AI users are, you're still maintaining separate contexts everywhere.
Use a cross-app memory layer. This is a newer category. Tools that sit above all your AI platforms and maintain one unified memory across everything. A Chrome extension, typically, that captures context from your conversations and makes it available everywhere you go. What you discussed in ChatGPT on Monday is available when you open Claude on Wednesday.

We built Vity for exactly this reason. It's a free Chrome extension that syncs your memory across ChatGPT, Claude, Gemini, Perplexity, DeepSeek, Grok; ten platforms currently. It captures context from your conversations (with your consent, always), then proactively surfaces relevant memories as you type in any supported app. You approve what gets included. Nothing gets injected without you seeing it first. The whole thing runs on an encrypted vault that we can't read even if we wanted to.

But my bias is showing. If you want an apples-to-apples look at how Vity stacks up against other options in this category, we published a detailed comparison of Vity, Mem0, and Supermemory that covers setup, platform support, privacy architecture, and memory depth. Read it and decide for yourself.

The forgetting problem isn't going away on its own

Here's the uncomfortable truth. Smarter models don't fix statelessness. GPT-5 will be smarter than GPT-4. It will still start every conversation from zero. Claude's next version will be more capable. It still won't know what you told the current version last Tuesday.

The forgetting isn't a technical limitation that next year's model update solves. It's an architectural choice. Each conversation is designed to be independent. That's actually useful for privacy and safety; it means your conversations don't bleed into each other or into other people's experiences. But it also means no AI tool, on its own, will ever "remember you" the way a colleague does.

Memory has to be solved as a separate layer. Something that sits above the models, captures what matters, and brings it back when it's relevant. Whether that's Vity, or one of the alternatives, or something you build yourself with a text file and discipline; the core insight is the same.

Your AI tools are not going to start remembering you. You have to give them a memory.

---

Frequently Asked Questions

Why does ChatGPT forget everything between conversations?

ChatGPT conversations are stateless by design. Each new chat starts with no knowledge of previous sessions. ChatGPT does have a built-in memory feature that stores basic facts about you, but it only retains lightweight summaries of recent chats and doesn't carry over the detailed context from working sessions.

Can AI remember things between conversations?

Not natively. ChatGPT, Claude, and Gemini each have limited memory features that work within their own platform, but none of them share memory across platforms. Cross-app memory tools like Vity solve this by maintaining a single memory layer that works across all major AI platforms.

Does Claude have memory like ChatGPT?

Claude has a different approach. Instead of pre-loading conversation summaries, Claude uses retrieval tools that can search your past conversations on demand. It offers a larger context window (upto 1 M tokens) for longer sessions, but like ChatGPT, its memory only works within Claude itself.

How do I make AI remember my preferences?

Three options: (1) manually paste your preferences into every new chat, (2) use built-in features like ChatGPT Memory or Claude Projects within each platform, or (3) use a cross-app memory extension that syncs your preferences across all AI platforms automatically.

What is cross-app AI memory?

Cross-app AI memory is a layer that captures context from your conversations across multiple AI platforms (ChatGPT, Claude, Gemini, Perplexity, etc.) and makes it available everywhere. Instead of each AI tool having its own isolated memory, you have one unified memory that follows you. Learn more about [how cross-app memory works](https://www.maximem.ai/cross-app-memory).

Is there a way to sync memory across ChatGPT and Claude?

Not through the platforms themselves: ChatGPT and Claude don't share data with each other. Cross-app memory tools bridge this gap. Vity's Chrome extension (which also works with OpenClaw) captures context from both platforms and surfaces relevant memories in either one, so preferences and project context travel with you.

How do AI agents remember previous conversations across sessions?

They do not on their own. Something outside the model has to decide what is worth keeping from a conversation, store it, and put the relevant part back into context on a later turn. Without that layer every session starts from zero, no matter how capable the model is.

Is this different for developers building agents than for people using ChatGPT?

The cause is identical and the fix is not. A person can work around it with a pinned prompt or a memory feature. A developer shipping to thousands of users needs the layer itself: capture, retrieval, conflict resolution when a user changes their mind, and deletion that actually propagates.

Why does a bigger context window not solve it?

A larger window extends how long one conversation runs before it degrades. It changes nothing about what happens when that conversation closes, and it raises cost on every turn because you reload the same history each time.

MCP 2026-07-28: 20 Breaking Changes and the Errors They Cause

Gaurav Dadhich — Tue, 25 Aug 2026 06:38:27 +0000

The Model Context Protocol shipped revision 2026-07-28 on 28 July 2026, and it is the largest change to the protocol since it launched. Sessions are gone. The initialize handshake is gone. Servers can no longer initiate requests at all.

If you run an MCP server, the useful thing to know first is that nothing breaks today on its own. A server speaking the old revision keeps working with clients speaking the old revision, and Anthropic has said support is rolling out across Claude products without announcing general availability or per-product dates. The break arrives when a client upgrades underneath you, and when it arrives it is total in both directions, because a modern client cannot talk to a legacy server and a legacy client has no mechanism at all for talking forward to a modern one.

Also gone in this revision: ping, logging/setLevel, SSE stream resumability, resources/subscribe, resources/unsubscribe, tasks/list and tasks/result. Roots, sampling, logging and Dynamic Client Registration are deprecated with a twelve month clock running on them.

Below is the full diff, every error you will hit while upgrading with the fix for each, and the one deadline where the specification and the release blog give different answers, which matters because the blog is the document most people will read.

What changed in MCP 2026-07-28

Eight things shipped, and they are easier to understand as one decision with seven consequences than as eight independent features.

Everything traces back to one decision, SEP-2575 and SEP-2567, which together make the protocol stateless. There is no handshake and no session identifier. Every request carries its own protocol version and client capabilities in _meta, and a new server/discover method replaces the capability exchange that used to happen at connection time. Servers must implement server/discover; clients may skip it entirely and handle the error instead.

Everything else follows from that. Because servers can no longer hold a connection open, they can no longer initiate requests, so Multi Round-Trip Requests (SEP-2322) replace sampling/createMessage, elicitation/create and roots/list with a pattern where the server returns resultType: "input_required" and the client re-sends the original call with the answers attached. Because the operation can no longer be inferred from a session, it moves into HTTP headers, so Mcp-Method and Mcp-Name (SEP-2243) are now required and a load balancer can route on them without parsing the body. Because list results no longer vary per connection, they become cacheable, so ttlMs and cacheScope (SEP-2549) are now non-optional fields on six result types.

Four further changes are less structural but worth knowing. Authorization hardens across six SEPs, with RFC 9207 issuer validation now mandatory on the client side and Dynamic Client Registration formally deprecated in favour of Client ID Metadata Documents. Tasks moves out of the core protocol into an opt-in extension (SEP-2663) with a redesigned polling API. A formal extensions framework (SEP-2133) lets anyone ship a protocol feature under a reverse-DNS identifier without a specification change. And a real deprecation policy (SEP-2596) arrives with a twelve month minimum window, a published registry, and SDK-level obligations to mark deprecated surfaces.

Here is the whole revision on one screen.

	`2025-11-25`	`2026-07-28`
Handshake	`initialize` plus `notifications/initialized`	None. Per-request `_meta`
Sessions	`Mcp-Session-Id` header, 404 to reinitialise, DELETE to end	Removed. Explicit application-level handles
Capabilities	Negotiated once per connection	Declared on every request
Discovery	`InitializeResult`	`server/discover`, server MUST implement, client MAY call
Transport	POST plus GET SSE, resumable via `Last-Event-ID`	POST only. A modern-only server SHOULD return `405` on GET and DELETE. No resumability
Routing	Operation in the JSON body. `MCP-Protocol-Version` header required since `2025-06-18`	Two new required headers, `Mcp-Method` on every request and `Mcp-Name` on three of them
Server to client requests	`sampling/createMessage`, `elicitation/create`, `roots/list` on a live stream	MRTR: `resultType: "input_required"` plus `requestState`, retried by the client with a new JSON-RPC id
Subscriptions	`resources/subscribe` plus the GET SSE stream	`subscriptions/listen` with an explicit notification filter
Caching	None	`ttlMs` and `cacheScope` on six result types (the release blog says four and the changelog says five; the schema and the caching page say six)
Long running work	Tasks, experimental, in core. Blocking `tasks/result`, plus `tasks/list`	`io.modelcontextprotocol/tasks` extension. Polling `tasks/get`, plus `tasks/update`. No `tasks/list`
Client registration	DCR (MAY) plus CIMD (SHOULD)	DCR deprecated. CIMD is the path. `application_type` now MUST
Issuer validation	Not required	RFC 9207 `iss` validation, MUST, including on error responses
Logging	`logging` capability, `logging/setLevel`, `notifications/message`	Deprecated. Per-request log level in `_meta`, stderr, OpenTelemetry
Liveness	`ping`	Removed
Resource not found	`-32002`	`-32602`
MCP error codes	`-32042` (URL elicitation required)	Three new codes, `-32020` to `-32022`, inside a reserved `-32020` to `-32099` block. `-32042` retired
Tool schemas	JSON Schema 2020-12 as default dialect	Full 2020-12 keyword set. Network `$ref` forbidden by default. DoS bounds required
Tool lists	May vary per connection	MUST NOT vary per connection. SHOULD be deterministically ordered
Extensibility	`experimental` capability bag	Formal extensions framework, reverse-DNS identifiers, opt-in, independently versioned
Deprecation	Ad hoc	Formal policy. Twelve month floor, published registry, SDK obligations, ninety day security exception
Result envelope	Bare result object	Required `resultType` on every result

What breaks, and what does not

Compatibility is published in the specification itself, and that matrix has seven rows rather than the four most write-ups have been reproducing. The two rows that are usually dropped are the actionable ones.

Client	Server	Outcome
Modern	Modern	Works
Modern	Legacy	Fails
Dual-era	Modern	Works
Dual-era	Legacy	Works
Legacy	Modern	Fails. Legacy clients have no fall-forward mechanism
Legacy	Dual-era	Works
Legacy	Legacy	Works per the legacy revision

Modern means 2026-07-28 and later, with per-request metadata. Legacy means 2025-11-25 and earlier, with the initialize handshake. Dual-era means an implementation that speaks both, which is the only posture that is safe if you have real users.

The Python SDK v2 gives you this by default, and it is worth understanding the mechanism rather than assuming it. Server.run drives a dual-era loop in which the client's first request decides the connection's era, once. A request carrying the 2026-07-28 per-request _meta envelope opens a modern connection; anything else, including an initialize handshake, opens a legacy one. A later claim from the other era is then refused: initialize on a modern connection returns UNSUPPORTED_PROTOCOL_VERSION naming the versions served, and an enveloped request on a legacy connection returns INVALID_REQUEST. So a v2 server serves both eras, per connection, not per request. One caveat that is not in the release notes and that I found in the source: HTTP era routing is header-based today, with body-primary classification marked as a follow-up.

Twenty things break. Here they are by severity, where total means the implementation stops working entirely rather than degrading.

Change	What it breaks	Severity
`initialize` and `notifications/initialized` removed	Every client and server implementing the handshake	Total
`Mcp-Session-Id` and sessions removed	Any server holding per-session state, any client resuming a session	Total
Per-request `_meta` with `protocolVersion` and `clientCapabilities` now required	Any request without them returns `-32602` and HTTP `400`	Total
`MCP-Protocol-Version` header required and must match the body	Mismatch returns `400` with `-32020`	Total
`Mcp-Method` required on all requests, `Mcp-Name` on three of them	Any hand-rolled HTTP client	Total
Server-initiated requests forbidden, MRTR replaces them	Every server using sampling, elicitation or roots	Total
The GET SSE stream is gone and resumability is removed. A modern-only server SHOULD answer GET and DELETE with `405`	Any client relying on the GET stream or on `Last-Event-ID`	Total
`resources/subscribe` and `resources/unsubscribe` removed	Any resource-subscription client	Total
Tasks redesigned and moved to an extension	Everyone on the experimental `2025-11-25` Tasks API	Total, for tasks users
`resultType` required on every result	Clients not defaulting an absent value to `"complete"`, or not handling `"input_required"`	High
Resource not found moves from `-32002` to `-32602`, and `-32042` (URL elicitation required) must no longer be emitted	Any client matching on either literal code	High
`notifications/cancelled` is stdio-only. Streamable HTTP defines no client-to-server notifications; closing the SSE response stream is the cancellation signal	Any HTTP client that sends it, and any server that only cancels on receiving it	High
`ping`, `logging/setLevel` and `notifications/roots/list_changed` removed	Health-check loops and log-level control	High
`notifications/elicitation/complete` and `elicitationId` removed	URL-mode elicitation correlation	High
Tool lists MUST NOT vary per connection	Any server personalising `tools/list` per session rather than per token	High
Any JSON Schema 2020-12 keyword is now allowed, but network `$ref` resolution MUST be off by default and composition keywords need resource bounds	Servers relying on remote `$ref` resolution	Medium
An invalid `x-mcp-header` forces the client to drop the tool from `tools/list`	Silent tool disappearance	Medium
`ttlMs` and `cacheScope` are non-optional on six result types	Servers omitting them are non-conformant	Medium
DCR deprecated and `application_type` now required	OIDC-strict authorization servers rejecting native redirects	Medium
RFC 9207 `iss` validation now MUST on the client side	Clients that skip it are non-conformant	Medium

What does not break is worth stating too, because it is more than you would expect. The stdio message framing is unchanged. Tools, resources, prompts, completion, pagination and progress keep their shapes. Elicitation survives in both form and URL modes, just delivered differently. And roots, sampling and logging are deprecated rather than removed, which means they are annotation-only and continue working for at least twelve months.

There is one trap in era detection that is worth its own paragraph, because it is the thing implementations will get wrong, and the rules differ by transport.

On HTTP you attempt a modern request and fall back on 400 Bad Request, except that a modern server also returns 400 for UnsupportedProtocolVersionError, for MissingRequiredClientCapabilityError, and for header validation failures, so the specification says you SHOULD inspect the body before concluding anything. On stdio you probe with server/discover, and there the rule is stronger: the fallback MUST NOT be keyed to one specific error code. Era is a property of the server rather than of an individual request, so cache the determination for the lifetime of the server process on stdio, or of the origin on HTTP.

One consequence of choosing dual-era that is easy to miss: the 405 guidance for GET and DELETE is written for servers that support only this revision. If you are serving both eras you still need the GET stream for legacy clients, so do not blanket-405 it.

It is also worth being straight about a tension in the messaging around all this. The operator of Glama, who runs the largest open source MCP server index, wrote on Hacker News that "the new protocol is wire-incompatible in both directions. This means that many of the servers/clients will need to be refactored (not enough to just update the SDK). It will take time and it will be messy." A maintainer in the same thread wrote that "this protocol change doesn't require you to do anything to existing running MCP code." Both statements are true, about different situations. The second is true of a deployment nobody touches. The first is true the moment anything on the other end of the wire moves.

Every error you will hit upgrading to MCP 2026-07-28

These are ordered roughly by how likely you are to see them.

`Unsupported protocol version` after upgrading, error code `-32022`

This is the two eras failing to talk, and it is the most common thing you will see this week. The error carries the useful data in error.data:

{
  "jsonrpc": "2.0", "id": 1,
  "error": {
    "code": -32022,
    "message": "Unsupported protocol version",
    "data": { "supported": ["2026-07-28", "2025-11-25"], "requested": "1900-01-01" }
  }
}

If data.supported contains only old revisions, the server has not been upgraded. If it contains only 2026-07-28, the client has not. The fix in both directions is to serve both eras rather than to pick one, and the practical route to that is an SDK bump rather than hand-rolling the negotiation, because the negotiation has more edge cases than it looks like.

One thing you cannot fix from the client side: a legacy client hitting a modern-only server. There is no fall-forward path in the older revisions, so that combination fails and will keep failing until the client updates. If you operate the server, this is the argument for staying dual-era well past the point where it feels necessary.

`400 Bad Request` with a `HeaderMismatch` error, code `-32020`

Three headers are now required on every Streamable HTTP POST, and their values must match the body:

POST /mcp HTTP/1.1
MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: search
Content-Type: application/json

It is worth stating the scope of these precisely, because it is easy to over-apply them. Mcp-Method is required on all requests. In the core protocol Mcp-Name is required only on tools/call, resources/read and prompts/get, where it carries params.name or params.uri. Sending Mcp-Name on a tools/list is not what the specification asks for.

The tasks extension then adds three more. SEP-2663 requires that tasks/get, tasks/update and tasks/cancel over Streamable HTTP set Mcp-Name to params.taskId, so that "transport intermediaries and load balancers [can] route subsequent requests for the same task to the server instance holding its state, which is typically required for correctness." Worth sitting with that sentence for a moment. It is discussed further below.

Note also that MCP-Protocol-Version is not new. It has been required on HTTP since 2025-06-18. The two genuinely new mandatory headers are Mcp-Method and Mcp-Name, plus any Mcp-Param-* your tools opt into.

Header names are case-insensitive and header values are case-sensitive, and for integers the comparison should be numeric rather than string, so 42.0 and 42 match.

There is a real caveat, and it was published a week before the specification landed. Christian Posta at Solo.io pointed out that headers can lie: an attacker sends Mcp-Name: echo in the header while the body calls printEnv, and a proxy that allowlists on headers alone waves it through.

Two mitigations exist. A server that processes the body MUST reject mismatches with -32020. And intermediaries enforcing policy on mirrored headers SHOULD verify that MCP-Protocol-Version indicates a revision requiring header-body validation, and SHOULD reject rather than trust the headers otherwise. Both are correct. The second is also a SHOULD placed on the component with the least incentive to comply, and it requires the intermediary to be MCP-version-aware, which is precisely the MCP-native dataplane Posta is arguing for. His post ends with a working fix rather than a dead end. Read it that way, and treat these headers as a routing hint rather than an authorization fact.

There is an operational failure here that has nothing to do with your code. Check whether your reverse proxy, your CDN, or your API gateway strips unknown request headers, because these headers will vanish silently on the way to your server and every request will fail validation for a reason that is invisible from either end.

Your server-initiated requests stopped working

Servers cannot originate requests any more. This is not a restriction on when they may do it; there is no ServerRequest union in the schema at all. sampling/createMessage, elicitation/create and roots/list still exist as types, but only ever as payloads inside a result.

MRTR is the replacement. Your server returns an incomplete result naming what it needs:

{
  "jsonrpc": "2.0", "id": 1,
  "result": {
    "resultType": "input_required",
    "inputRequests": {
      "github_login": {
        "method": "elicitation/create",
        "params": {
          "mode": "form",
          "message": "Please provide your GitHub username",
          "requestedSchema": {
            "type": "object",
            "properties": { "name": { "type": "string" } },
            "required": ["name"]
          }
        }
      }
    },
    "requestState": "AEAD-protected blob"
  }
}

The client gathers the input and re-sends the original request with inputResponses and the requestState echoed back byte for byte, using a new JSON-RPC id, because the specification treats the retry as an independent request.

Six rules will bite you if you skip them. InputRequiredResult is permitted only on tools/call, prompts/get and resources/read, and servers MUST NOT send it on anything else. Clients MUST NOT inspect, parse or modify requestState, and MUST NOT include one on the retry if the server did not send one. Servers MUST NOT send an inputRequest type the client did not declare on that particular request. Servers MUST NOT assume the client will ever fulfil the request or retry at all, which means every MRTR exchange needs a timeout and a clean path to giving up. And inputRequests and requestState apply only to the retry of that one request, never to anything the client has in flight in parallel.

The sixth is the one that will produce a crash rather than a bug report. inputRequests is optional. The schema requires at least one of inputRequests or requestState, and it names the case where only the state is present: load shedding. A server under pressure can hand back requestState alone, and the specification says the client MAY then simply retry immediately. Any client that writes for key in result.inputRequests throws on that response, at precisely the moment the server is least able to absorb a retry storm.

`MissingRequiredClientCapability`, error code `-32021`

Capabilities are now declared per request rather than once per connection, and a server may not rely on one the client did not send on that specific request. When it needs something undeclared it returns -32021 with the missing capabilities listed in data.requiredCapabilities, and on HTTP the status is 400.

On the client side, stop treating capabilities as connection setup. They belong in _meta on every call:

"_meta": {
  "io.modelcontextprotocol/protocolVersion": "2026-07-28",
  "io.modelcontextprotocol/clientCapabilities": { "elicitation": {} },
  "io.modelcontextprotocol/clientInfo": { "name": "my-app", "version": "1.0" }
}

protocolVersion and clientCapabilities are required. clientInfo is optional but you should send it, and the specification is clear that neither clientInfo nor serverInfo is verified by the protocol, so nothing should make a security decision based on either.

Resource not found now returns `-32602`, not `-32002`

The error code layout changed underneath everyone. Two codes that shipped in 2025-11-25 are retired, and three new ones arrive in a newly reserved block.

Code	Shipped in	Status in `2026-07-28`
`-32002`, resource not found	`2025-11-25` and earlier	Retired. Use `-32602`. Clients SHOULD keep accepting it from older servers
`-32042`, URL elicitation required	`2025-11-25` only	Retired. Implementations of this revision MUST NOT emit it
`-32020`, header mismatch	new	Reserved block
`-32021`, missing required client capability	new	Reserved block
`-32022`, unsupported protocol version	new	Reserved block

Codes from -32020 to -32099 are now reserved for the specification, and implementations must not emit codes in it that the specification has not defined. The range -32000 to -32019 is legacy, and apart from -32002 receivers must not assume any specific meaning for codes in it.

One clarification, because several write-ups have this wrong, including an earlier version of this one. The three new codes were briefly numbered -32001, -32003 and -32004 inside the 2026-07-28 draft cycle and renumbered before release. They were never part of 2025-11-25, whose schema defines exactly one non-standard code, -32042. So if you are migrating from a shipped 2025-11-25 implementation, the codes that actually change under you are -32002 and -32042. If you built against a release candidate, check for the old three.

Your tool disappeared from `tools/list`

This is the most obscure genuinely breaking behaviour in the release, because it produces no JSON-RPC error, and because it is transport-dependent in a way that will waste somebody an afternoon.

x-mcp-header is a new annotation that lets a server mark individual tool parameters to be mirrored into Mcp-Param-{Name} HTTP headers, so that a gateway can route or rate-limit on a parameter value without reading the body. It is optional for servers and mandatory for clients to support. The constraints on it are strict: primitive types only, with number explicitly not permitted while integer, string and boolean are; the property must be statically reachable from the schema root through a chain of properties keys only, so nothing behind items, $ref, oneOf, anyOf, allOf, not, or if/then/else; the value must be case-insensitively unique across the whole inputSchema; annotated integers must sit within the JavaScript safe range; and the value must be a valid HTTP field-name token with no control characters.

If any of those is violated, the client MUST exclude the tool from the result of tools/list. There is no JSON-RPC error. The specification does ask clients to log a warning naming the tool and the reason, but that is a SHOULD and it surfaces in your client's log rather than to the model or the user, so the tool simply is not there any more and nothing in the conversation says why.

What will actually cost you the afternoon is that this rule binds Streamable HTTP clients only. Clients on other transports MAY ignore x-mcp-header entirely. So the same server, with the same tool definition, can expose the tool over stdio on your laptop and hide it over HTTP in production.

`400` with no obvious cause, and a missing `Accept` header

Not new, but hand-rolled clients get it wrong constantly and this revision gives them more opportunities to. The client MUST include an Accept header listing both application/json and text/event-stream, because the server chooses between a single JSON response and an SSE stream and the client has to be ready for either.

`404 Not Found` on a method the server does not implement

This one is a trap for gateways and for anyone with retry logic. An unimplemented RPC method returns HTTP 404 with JSON-RPC -32601, which is deliberately distinguishable from a transport-level failure. Anything that treats 404 as "this server is gone" and tears down the connection or fails over will misbehave against a perfectly healthy server that simply does not implement the method you asked for. Check the body before concluding anything about a 404, exactly as you already have to for a 400.

Your list results are being served to the wrong tenant

Not an error message, which is why it belongs here. ttlMs and cacheScope are now non-optional on six result types: server/discover, tools/list, prompts/list, resources/list, resources/templates/list and resources/read. Worth noting that the release blog lists four of those and the changelog lists five; the schema and the caching page say six, and they are the ones that count.

cacheScope takes "public", meaning any client, gateway or caching proxy may serve the response to any user, or "private", meaning it may be reused only within the same authorization context and MUST NOT be shared across contexts. Get this wrong on an authenticated endpoint and you have built a cross-tenant data leak with the protocol's blessing. The specification says so almost in those words: servers must be aware that a "public" response may be shared between callers even when it comes from an authenticated endpoint.

Two rules that follow. Results produced by an MRTR retry must never be cached, because they depend on inputs that are not part of the cache key. And clients SHOULD NOT treat ttlMs as a polling interval; those that do poll MUST apply jitter and backoff, which is the single most likely way a naive client turns a TTL into a thundering herd.

Long-running jobs broke when sessions went away

One of the sharper reactions on Hacker News was from someone who uses MCP servers to monitor workflows that run for days, and who wrote simply that "i use them to monitor workflows that takes days and now i have to make some changes."

Tasks, now an extension, is the replacement, and it is a second migration for anyone who adopted the experimental API in 2025-11-25. What changed:

`2025-11-25`, experimental	`io.modelcontextprotocol/tasks` extension
Client opts in per request with a `task: {ttl}` param	The server decides, unilaterally, per request
`tasks/get` plus a blocking `tasks/result`	`tasks/result` removed; `tasks/get` absorbs its job and polls, honouring `pollIntervalMs`
No client-to-server input method	`tasks/update` carrying `inputResponses`
`tasks/list`	Removed. There is no way to enumerate tasks
`notifications/tasks/status`	`notifications/tasks`, opt in via `subscriptions/listen` with a `taskIds` filter
`ttl`, `pollInterval`	`ttlMs`, `pollIntervalMs`
`tasks/cancel` returns the task	`tasks/cancel` returns a bare acknowledgement; cancellation is cooperative and the server need only acknowledge
Both parties can be requestors	`tools/call` only

Two consequences of that table deserve calling out. Because tasks/list is gone, if your client loses a task id the task is unreachable, so persist task ids durably rather than holding them in memory. And because the extension must be negotiated in _meta on each request, a server MUST NOT return a task handle to a client that did not declare the extension on that specific call.

One detail that will catch people moving off resources/subscribe: request-scoped notifications, meaning notifications/progress and notifications/message, still flow on the response stream of the request they belong to, not on the subscriptions/listen stream. And on stdio, if the connection drops the client MUST re-send subscriptions/listen, because the server holds no subscription state across reconnections.

One caveat before you build on this, and it is a real one. The extension's normative schema file is not published. The repository states it "is under development and is working towards implementing the changes described in SEP-2663", and carries a banner reading "not an official extension and may change significantly or be discontinued", while the 2026-07-28 changelog describes tasks as having moved into "an official extension." SEP-2663 itself still cites -32003, an error code the shipped core renumbered to -32021. Treat this part of the ecosystem as unfinished and budget for a second pass.

Your traces and your latency numbers went wrong

This one is quiet and it affects everyone who has observability wired up. Because the MRTR retry MUST use a different JSON-RPC id from the original request, one logical operation now appears on the wire as several id-distinct requests with no correlation identifier between them.

Every tool that keys on request id, which is all of them, therefore reports an inflated call count and, worse, excludes the interval where a human was actually deciding something. That interval is precisely the thing you care about for an elicitation-based confirmation. An analysis published on release day measured a six second operation reporting as two calls totalling one second. That is one scripted exchange against the author's own proxy rather than a survey, so treat it as an illustration of the mechanism rather than a measurement of the blast radius. The mechanism is the part that generalises.

There is no protocol-level fix. Linkage has to be inferred, either from the opaque requestState blob or by matching the key sets in inputResponses against the inputRequests that produced them. It is worth noticing that the same revision standardised W3C trace context propagation, with traceparent, tracestate and baggage explicitly exempted from the reverse-DNS prefix rule in _meta. The plumbing got better and the semantics got worse in the same release.

Authorization started failing on desktop and CLI clients

If a client is going to break against a strict enterprise authorization server on day one, this is why. Clients MUST now specify application_type during Dynamic Client Registration, and omitting it defaults to "web" under OpenID Connect, which conflicts with the localhost redirect URIs that desktop applications, mobile applications and CLI tools use. Native clients should send application_type: "native"; genuinely remote browser-based clients should send "web".

Two other authorization changes are worth checking before you ship. RFC 9207 issuer validation is now mandatory on the client side and it applies to error responses as well as success responses. You do decode the iss value from the form-encoded response, and then you compare it as a literal string: no scheme or host case folding, no default-port elision, no trailing-slash handling and no percent-encoding normalisation. Note that authorization servers only SHOULD send iss, so whether its absence is fatal depends on the server advertising authorization_response_iss_parameter_supported. And persisted client credentials must now be keyed to the issuing authorization server's issuer, so when the authorization server changes you must re-register rather than reuse.

SDK readiness, as of 29 July 2026

SDK	Tier	Version with `2026-07-28`
TypeScript	1	`@modelcontextprotocol/server@2.0.0` and `@modelcontextprotocol/client@2.0.0`
Python	1	`mcp==2.0.0`
Go	1	`go-sdk v1.7.0`
C#	1	`ModelContextProtocol 2.0.0`
Rust	2	`rmcp 3.0.0`
Java	2	No support yet
Swift, Ruby, PHP, Kotlin	3	No support, no committed timeline

Two notes on that table. Java does have a 2.0.0, published on 11 June 2026, and it simply predates this revision; the jar contains 2025-11-25 and earlier and no 2026-07-28 at all. Java is a Tier 2 SDK, which carries a six month obligation, so support is due around January 2027. And on TypeScript, v2 is ESM-only, requires Node 20 or later, and splits the old monolith into separate server and client packages, which is a larger change than the version number suggests. There is a codemod: npx @modelcontextprotocol/codemod v1-to-v2 .

If you maintain a library that depends on an MCP SDK, put an upper bound on it today, mcp>=1.28,<2 or @modelcontextprotocol/sdk@^1, so that v2 does not land on your users before you have decided how to handle it.

Before and after, in two pictures

If you came here with a red terminal, that is the practical half. What follows is why any of it happened.

Two diagrams, each showing the same two things: how the protocol deploys, and what actually goes over the wire. Those turned out to be the same question, which is the whole reason this revision exists.

That orange band running down the middle of the first diagram is the argument for the entire release. It is the session lifetime, and the line written across it says that every request underneath it must reach server instance number one. That constraint is not a deployment choice you made. It is a property of the protocol, because the negotiated capabilities and every in-flight request lived in one server process, so the load balancer had no option but to pin. Two of the three instances in that picture are unreachable for the duration of the session, the shared session store hanging off the side is pure protocol overhead you provisioned and paid for, and the gateway in the middle cannot route on anything useful because every call is POST /mcp with the operation buried in the body.

The payoff in the second diagram is small and easy to miss, so it is worth pointing at directly. Step three is a tools/call that lands on instance three. Step four is that same logical call being retried, after the client gathers the input the server asked for, and it lands on instance two. A different box, mid-operation, and nothing breaks. That is what statelessness actually buys, and it is very hard to convey in prose.

Two wins and two new burdens sit in the consequence strip at the bottom of the second diagram, and the mix is deliberate, because that is an accurate picture of the trade. Autoscale and serverless become native. In exchange, requestState is now an attacker-controlled input you have to authenticate yourself, and the MRTR retry carries a new JSON-RPC id, which quietly breaks anything measuring latency by request id.

What the release fixed, and what it left alone

MCP has been shipping for twenty months and accumulating a fairly specific critique record the whole time. The interesting way to read 2026-07-28 is not as a list of features but as a response to that record, ranked by whether the specification deleted the thing being complained about. A protocol that removes the feature people objected to is conceding the argument in the plainest language available.

Strongest evidence that this is what happened: the maintainers wrote the critique themselves, ten months before shipping the fix. From "Exploring the Future of MCP Transports", published 19 December 2025:

"Stateful connections force 'sticky' routing that pins traffic to specific servers, preventing effective auto-scaling."

"Developers building simple, ephemeral tools are often required to manage complex backend storage to support basic multi-turn interactions."

"Load balancers and API gateways must parse full JSON-RPC payloads to route traffic, rather than using standard HTTP patterns."

An earlier post, "Update on the Next MCP Protocol Release" from September 2025, had already named horizontal scaling, long-running tasks, discovery, extensions and SDK compliance as the top five gaps.

Five external critiques were louder than the rest. Statefulness and session affinity, whose root cause was precise: the handshake negotiated capabilities per connection, the session identifier lived in the transport, and both meant every request had to reach the same process. The New Stack put it as teams "paying to solve a distributed-systems problem the protocol had created for them." The tool-definition context tax, where Apideck measured three MCP servers consuming 143,000 of a 200,000 token window before a single user message, and where the most damning source is Anthropic itself, which published a method for taking 150,000 tokens down to 2,000, roughly a 98.7% reduction, by not using the protocol the way it was designed. Prompt injection, where Invariant Labs demonstrated data exfiltration through the GitHub MCP server and concluded that "GitHub alone cannot resolve this vulnerability through server-side patches," which is what makes it architectural rather than a bug. The absence of an authorization model, summarised by Pomerium as "access is binary. Either a tool is exposed, or it isn't." And gateway hostility, which is the same root cause as the first one seen from the operations side.

Below those sit the rest: Microsoft Research measuring performance degradation of up to 85% as the tool space grows, along with 775 colliding tool names and the word search appearing across 32 distinct servers; the OAuth and Dynamic Client Registration burden that Descope catalogued as an enterprise blocker; the missing distributed-systems contract around idempotency, deadlines and tracing; and the absence of any asynchronous operation model.

One disclosure belongs here rather than in a footer, because it changes how you should read half of those links. Speakeasy, Scalekit, Descope, WorkOS, Pomerium, Docker and Apideck all sell products that exist because of the gaps they document. Their measurements are checkable and I have used several of them. Their framing is not neutral, and neither is mine, since Maximem ships tooling in this space too. Microsoft Research, Trail of Bits, Invariant Labs and Anthropic's own engineering blog are the cleanest sources in that list.

So, the scorecard. Eighteen critiques, mapped against what actually shipped.

Critique	What `2026-07-28` does	Verdict
Statefulness and session affinity	Sessions, `initialize` and the GET stream deleted. State becomes an explicit handle	Solved
Tool-definition context tax	Deterministic tool ordering, cacheable `tools/list`. Neither reduces what the model sees	Untouched
Prompt injection and the lethal trifecta	Nothing	Untouched
No authorization model	Nothing. Scope step-up, which is often credited to this release, shipped in `2025-11-25`. Modelling went to the `ext-auth` extensions repo	Untouched
Tool poisoning, rug pulls, line jumping	Nothing. Descriptions remain unsigned and mutable	Untouched
Tool-space interference, no namespacing	Name-format rules, and clients "SHOULD implement a disambiguation strategy"	Weak partial
Auth complexity and the DCR burden	`iss` validation, credential-issuer binding and `application_type` are real. But CIMD was already SHOULD and DCR already MAY in `2025-11-25`, so the registration delta is largely a Deprecated label	Partial
Gateway, WAF, metering and caching hostility	`Mcp-Method` and `Mcp-Name` headers, `ttlMs` and `cacheScope`	Solved
Missing distributed-systems contract	Tracing yes, schema strictness yes, versioning discipline yes. Idempotency, deadlines and per-tool versioning no	Partial
No long-running or async model	Tasks extension with polling, updates, cancellation and a durability guarantee. But its normative schema is unpublished and its own repository says it is not an official extension	Partial
No multi-tenancy primitive	Nothing, beyond `cacheScope: "private"` being authorization-context aware	Untouched
Transport churn	One transport story now, but this release is churn: the third breaking transport shape in sixteen months. The improvement is that churn is finally governed by a deprecation policy	Neutral
Configuration fragility	Removes session config as a failure class, adds header validation as a new one	Neutral
Latency at the call level	No handshake, so the first-call penalty disappears. Cacheable lists cut repeat cost	Improved
Registry and discovery	`server/discover` per server. The registry itself still has no protocol-version field	Partial
Observability gap	Trace context standardised, then MRTR breaks id-keyed measurement	Mixed
Risk stratification and cost opacity	Nothing in core	Untouched
SDK quality and compliance opacity	SDK tiering itself is SEP-1730, from October 2025. This release adds Tier 1 deprecation-marking obligations and conformance tests as a gate to Final	Partial

Three clear wins, nine partial or mixed, six untouched.

I want to be explicit that I graded this twice, because the first pass was too generous. It credited this release with a scope step-up flow that shipped eight months earlier in 2025-11-25, rated long-running work solved on an extension whose normative schema does not exist, called transport churn solved in the release that caused it, and counted the registration changes as a fix when CIMD was already SHOULD and DCR already MAY. Correcting all four made the pattern sharper rather than softer, which is usually a sign the pattern is real rather than an artifact of how you drew the table.

2026-07-28 is an infrastructure release, not an agent-experience release. Every problem that lives between a load balancer and a server process got fixed. Every problem that lives between a model and a tool description did not.

Those untouched rows are worth sitting with, because they are the ones that most consistently kept MCP out of production. The context tax got deterministic tool ordering, which improves prompt-cache hit rates at the margin and is the entire response to a complaint that produced six separate front-page Hacker News threads and a first-party Anthropic post documenting a 98.7% workaround. Prompt injection got nothing, and tool descriptions are still unsigned, still mutable after a user has approved them, and still injected into the model's context at tools/list time, which is the mechanism behind what Trail of Bits named line jumping and the reason invocation-time approval was never the protection people assumed it was. Authorization modelling got nothing, and it is worth correcting a claim I have seen in several write-ups including my own first draft: the step-up flow with insufficient_scope shipped in 2025-11-25, not here. What this release adds is authorization hardening, which is real. Authorization modelling, meaning per-tool scope and identity propagation into the tool, went to an extensions repository.

Some fair objections, before the conclusion.

No single revision could clear eighteen months of accumulated complaints, and that is true. The point was never the count. It is which half got cleared.

A better objection is that statelessness is an agent-experience change, and this one has more force than I would like. Fewer failure modes, no reconnection storms, no session dying mid-conversation, cheaper cold starts: reliability is part of the experience, and this release improves it. What it does not improve is anything the model can perceive. Same tool count, same trust in the descriptions, same token bill.

Strongest of all is the argument that agent-experience work is happening in extensions now, and that the framework this release formalised is precisely what makes that possible. Go and look at the official extension support matrix, though, which today lists nine clients. All nine support MCP Apps. Exactly one supports anything beyond it, and none supports OAuth Client Credentials. Extensions are opt-in on both ends, and the layer above the protocol is not there yet.

That pattern is not carelessness. It is a boundary decision, and it is defensible. The protocol has drawn a line and put infrastructure concerns on its own side of it while leaving agent-experience concerns to the layer above. There is a serious argument that this is exactly what a protocol should do. It also means that if MCP was expensive for you last week because of tokens, or risky for you last week because of injection, this release changed nothing about either.

What the release created

A piece that only lists what got better is a piece worth distrusting, so here is the other column.

requestState moves a security boundary out of the protocol and into your application code. The specification is unusually direct about this. Servers MUST treat requestState as attacker-controlled input. If it influences authorization, resource access or business logic, servers MUST protect its integrity with HMAC or AEAD and MUST reject state that fails verification. Servers SHOULD embed the authenticated principal, a short expiry, and a digest of the originating request inside the protected payload. And then, having said all that, the specification concedes that these measures "do not by themselves guarantee single-use." Maxim Zavodchik at Akamai put the consequence plainly: "critical security boundaries are now entirely dependent on how developers implement them." Set against Equixly's finding that 43% of the MCP servers they assessed shipped command injection, and that 45% of the vendors they disclosed to dismissed the risk as theoretical, that is a large bet on developer diligence.

Headers can lie, which is covered above, and the component making the policy decision is the one not reading the body.

And sessions are not quite as gone as the headline suggests. SEP-2663 requires that tasks/get, tasks/update and tasks/cancel over Streamable HTTP carry Mcp-Name: <taskId>, explicitly so that intermediaries can "route subsequent requests for the same task to the server instance holding its state, which is typically required for correctness." That is instance affinity, reinstated by the flagship extension for long-running work, keyed to a task ID instead of a session ID. It is a better design than the old one, because the affinity is now explicit, addressable, and scoped to a unit of work rather than to a connection. But anyone reading "sessions are gone" as "you will never need sticky routing again" should read that sentence in SEP-2663 first.

Roots deprecation removes the only structural filesystem boundary the protocol had. The prescribed migration is to pass directories through tool parameters, resource URIs, or server configuration, which converts a client-enforced limit into a server-trusted argument. Scope enforcement now lives in each developer's code and is applied inconsistently by construction.

Sampling deprecation quietly moves cost onto server operators. SEP-2577 gives three reasons and leads with the design one: "correct sampling implementation requires human-in-the-loop approval, model selection logic, security considerations, and (since SEP-1577) tool loop support," which it says contributed to low client adoption. The lead maintainer's version, to The Register, is that "the semantics were confusing and hard to implement." That reasoning is sound.

The part nobody has priced is the migration. The prescribed replacement is to integrate directly with LLM provider APIs, which moves model credentials, billing, rate-limit handling and model-choice policy from the client to the server operator. That is my reading of the consequence rather than a complaint anyone has organised around, and I could not find one.

And the specification is silent on two things it probably should not be. There is no CORS guidance anywhere in it, despite four new mandatory headers that are all non-simple and therefore all trigger a browser preflight. To be precise about the claim, I grepped all thirty-one specification pages and the schema: zero mentions of CORS, preflight, OPTIONS or Access-Control-Allow-Headers.

Documentation is a different matter, and the way it mentions CORS is the more interesting finding. The 2026-07-28 authorization tutorial ships sample code containing cors({ origin: "*", exposedHeaders: ["Mcp-Session-Id"] }), exposing a header this revision deleted, and two of its live handler blocks still read req.headers["mcp-session-id"]. That page is not stale in one snippet. It is stale throughout, under a 2026-07-28 URL. The extension support matrix has the same problem in miniature, still instructing client authors to "declare support in the extensions field of your initialize capabilities" for a handshake that no longer exists. There is also no size bound on requestState, which round-trips through the client on every retry.

Trilogy AI makes the sharpest counter-argument to the whole release, and it is worth taking seriously: statelessness relocates complexity rather than deleting it. Explicit handles consume token budget. Models operating with garbled handles fail differently from expired sessions, but they still fail. And nobody has specified what recovery looks like when a model loses a handle in the middle of a conversation. The session was at least a thing the infrastructure understood. The handle is a string the model has to keep track of.

When does HTTP+SSE actually stop working, and why do two official pages disagree?

Every write-up on this release is leading with July 2027, which is when roots, sampling, logging and Dynamic Client Registration become eligible for removal. That is correct and it is not the nearest deadline.

Nearest is the legacy HTTP+SSE transport, deprecated back in 2025-03-26 and only now given a removal clock. Two official sources describe that clock differently.

Source	Implied earliest removal
The deprecated features registry, "three months after SEP-2596 reaches Final", with PR #2596 merged under the `final` label on 18 May 2026	around 18 August 2026
The release blog, "officially deprecated, with a year-long offramp"	around July 2027

The registry's number is the reasoned one, and SEP-2596 explains it in a section called Transition that is worth quoting because it is better thinking than the blog's summary:

"Both features were publicly deprecated well over twelve months before this SEP, so the minimum deprecation window has in practice already been served; re-anchoring their clock to a future revision release would restart a window the ecosystem has already had. Each is therefore given a three-month grace period from the date this SEP reaches Final before it is eligible for removal, matching the floor the Expedited removal clause sets for the shortest permissible window."

So the SEP-Final anchoring is deliberate and reasoned, not an anomaly. HTTP+SSE and the includeContext values are the two features grandfathered in from before the policy existed, and the SEP is explicit that "the deprecation-SEP requirements in Deprecating a feature are not applied retroactively" to them.

What that leaves is smaller than a governance story and more useful than one: the document most people will read is wrong about the date. The blog says twelve months. The registry and the SEP say three, from a date that has already passed.

Before you act on that, one thing the SEP says that the coverage has also missed, and that cuts the other way:

"Removal still follows Removing a feature: a Core Maintainer decision at release preparation, not an automatic event when the grace period ends."

Removal also requires a new Current revision to ship, and none is scheduled. So there is no August cliff. What there is, from around 18 August 2026, is eligibility, meaning the point from which any future revision may drop the transport without further notice. If you are still on HTTP+SSE, the planning implication is not that something breaks next month. It is that you no longer have a guaranteed window, and you should stop assuming the twelve months the blog implies.

Now the governance oddity, which is a different feature. The lifecycle policy that shipped in this revision says plainly that "deprecation is a specification change and therefore requires a SEP per the SEP guidelines." Dynamic Client Registration's deprecation is recorded against PR #2858, titled "Authorization spec split," a documentation reorganisation merged on 4 June 2026 whose description is about page load time and navigation. It is the only row in the deprecated registry without a SEP number.

Three rebuttals suggest themselves and none of them holds. That CIMD arrived via SEP-991 and the deprecation merely followed it: SEP-991 contains zero occurrences of the string "deprecat". That SEP numbers are PR-derived under SEP-1850, so #2858 is effectively the SEP number: there is no file in seps/, no SEP label and no SEP status, and the registry's own column heading reads "Deprecation SEP" while the cell reads "PR #2858". That the grandfathering clause covers it: it does not, because that clause covers only the two features deprecated before the policy existed, and DCR is a new deprecation in this revision.

I do not think anything was slipped through. DCR was already only a MAY in 2025-11-25 and CIMD was already a SHOULD, so the substantive change is close to a label, which is probably exactly why nobody stopped to write a SEP. That is the interesting version of the finding: the first process the policy governed was skipped, on a change small enough that skipping it felt reasonable.

How much of this matters in practice. I paginated the official registry on 29 July 2026. There are 1,888 published version entries declaring an sse remote, which is 761 distinct servers, 709 of them the latest version of their server, against 20,173 entries and 9,045 distinct servers on streamable-http. Note the distinction between entries and servers, because a naive count inflates the number by more than double.

A better measure of who actually gets hurt comes from Glama's index of 62,726 open-source servers, shared on Hacker News the day the spec shipped: roughly 88% have been pushed to within the last year, which means about 12% have not been touched in twelve months. Those are the servers that will still be speaking a dead transport when someone finally removes it.

What to do, by what kind of MCP you run

Four situations cover most people. The migration skill linked below covers the rest.

If you run a local stdio server, you are the lowest-risk group, because there are no headers, no sessions and no OAuth in your path. Implement server/discover, since it is a MUST for servers and it doubles as the era probe on stdio. Move logging to stderr, because logging/setLevel is gone and notifications/message is deprecated and must not be emitted unless the request carried a log level in _meta. Stop writing JSON-RPC requests to stdout entirely. And plan for in-flight requests simply being lost when the process dies, because the protocol no longer pretends otherwise.

If you run a multi-tenant SaaS MCP, this is the highest-stakes cluster and the first thing to audit is cacheScope, before anything else, because that is the field that leaks. Anything user-scoped, tenant-scoped or permission-scoped must be "private". Then check tool visibility, and here is the carve-out most write-ups are missing: tool lists MUST NOT vary per connection, but they MAY still vary by the authorization presented on the request, since credentials are per-request input rather than connection state. Per-tenant tool filtering is still entirely legal. It simply has to key off the token instead of the session. After that, work through the authorization changes: iss validation, CIMD support, application_type, credential binding, and single-challenge scope emission.

If you run on serverless or the edge, you are the biggest winner and most of your work is deletion. Remove the Durable Object, Redis or DynamoDB session pinning, remove session affinity from the load balancer, and then actually test round-robin by firing a sequence of requests and asserting that none of them depends on the previous one landing anywhere in particular. Re-tune your cold-start and concurrency budgets while you are there, because a one-shot tool call is now literally one HTTP request with no handshake in front of it, and your per-invocation cost model has changed. If you are on Cloudflare, note that McpAgent is now feature-frozen and createMcpHandler(v1Server) goes away in the next major.

If you run a gateway, proxy or aggregator, three rules. Do not make policy decisions on Mcp-Method and Mcp-Name alone. Handle resultType values you do not recognise, and never pass "task" through to a client that did not negotiate the extension. And remember that the MRTR retry arrives with a different JSON-RPC id, so any metering or rate limiting keyed on id will double-count a single logical operation. There is also an opportunity here that the Glama operator named on Hacker News, which is that protocol-incompatible servers and clients are going to need translating between for at least a year, and the gateway layer is the natural place for that to live.

The twelve-point compatibility checklist

Run this against your server before you call the migration done.

Can a request land on a cold, empty instance and succeed?
Do you implement server/discover? It is a MUST for servers.
Do you accept and validate MCP-Protocol-Version, Mcp-Method and Mcp-Name, and does your proxy forward them?
Do you ignore inbound Mcp-Session-Id and Last-Event-ID? (And if you serve only this revision, do you answer GET and DELETE with 405? If you are dual-era, keep the GET stream for legacy clients.)
Is resultType present on every result, and does your client handle "input_required"?
Are ttlMs and cacheScope present on all six cacheable result types, and is cacheScope correct for authenticated data?
Does tools/list avoid varying per connection, and return a deterministic order?
Have you replaced every server-initiated request with MRTR, and is requestState integrity-protected?
Have you moved off -32002 and -32042, and if you built against a release candidate, off the pre-release -32001, -32003 and -32004 numbering?
Does your inputSchema validate as JSON Schema 2020-12, with network $ref resolution disabled by default?
Have you inventoried your use of roots, sampling, logging and DCR against the 2027-07-28 clock?
If you are still on HTTP+SSE, is your Streamable HTTP migration scheduled? Eligibility for removal opened around 18 August 2026, not in twelve months, whatever the release blog says.

The tool

There is a gap in the ecosystem that you can verify with one curl. The official MCP registry has no protocol-version field: server records carry a $schema, the server's own semantic version and its remote types, and nothing at all declaring which protocol revision the server speaks. So there are tens of thousands of published servers, a wire incompatibility that runs in both directions, and no machine-readable compatibility signal anywhere.

Client-side information is thinner still, and in a way that has already had consequences. SEP-2577 justifies deprecating roots, sampling and logging partly on low client adoption, and cites modelcontextprotocol.io/clients#feature-support-matrix as its evidence. That page and that anchor no longer resolve; the URL now redirects to the getting-started introduction. The evidence base for three deprecations is a dead link.

We built mcp-2026-migrate to close that for ourselves and it is easier to give away than to keep. It is a skill for Claude Code and Codex that reads your repository, identifies which of the twenty breaking changes apply to your code, and applies the mechanical ones, plus a standalone probe that determines a server's protocol era from the outside, which works on servers whose source you do not have. Free, open source, and built by Maximem, which is the same reason to read the scorecard above with the disclosure in mind.

Frequently asked questions

What is MRTR in MCP? Multi Round-Trip Requests, introduced in the 2026-07-28 MCP specification, is the pattern that replaced server-initiated requests. The server returns resultType: "input_required" with the inputs it needs and an opaque requestState blob, and the client re-sends the original request with the answers attached. It works only on tools/call, prompts/get and resources/read.

What does resultType: "input_required" mean? As of the 2026-07-28 MCP specification, every result carries a resultType, and "input_required" means the operation is incomplete because the server needs something from the client. An absent resultType must be treated as "complete" for backward compatibility with older servers, and an unrecognised value must be treated as invalid.

Can I keep using MCP 1.x for now? Yes. A server on 2025-11-25 keeps working with clients on 2025-11-25, and the SDK v1 lines are still published. What you cannot do is mix eras, because a modern client cannot talk to a legacy server and a legacy client cannot talk to a modern one.

Can a server support both the old and new MCP spec at the same time? Yes, and this is the recommended posture if you have real users. The 2026-07-28 specification calls it dual-era. The Python SDK v2 does it by default: the client's first request decides that connection's era, once, and a later claim from the other era is refused. So in many cases an SDK upgrade gets you there without application changes.

Why am I getting "unsupported protocol version" after upgrading? Error -32022 means the two ends do not share a supported revision. Check error.data.supported to see which side is behind. The fix is to make one side dual-era rather than to force both to the same revision at the same moment.

Do I need to update my MCP server? Not today, if nothing on the other end of your wire has moved. The 2026-07-28 revision does not break a running deployment by itself. You need to update before your clients do, and Anthropic has not yet shipped support across Claude, so most people have some runway.

What replaced Mcp-Session-Id? Nothing at the protocol level. The 2026-07-28 specification removed sessions entirely. State that must span requests is now an explicit, opaque, server-minted handle returned in a tool result and passed back as an ordinary tool argument, which the server re-authorizes on every call. One nuance: the tasks extension asks clients to send Mcp-Name: <taskId> so intermediaries can route back to the instance holding a task's state, so instance affinity has not disappeared, it has become explicit and scoped to a unit of work.

Was initialize removed from MCP? Yes. As of 2026-07-28 the initialize request and the notifications/initialized notification are both removed and nothing replaced them. Protocol version and client capabilities travel in _meta on every request, and server/discover covers the discovery case.

When will the Java SDK support 2026-07-28? It does not yet. Java is a Tier 2 SDK, which carries a six-month obligation from the specification release, so support is due around January 2027. Note that a Java 2.0.0 does exist, published 11 June 2026, but it predates this revision and contains no 2026-07-28 support.

When will Claude clients support the new MCP spec? Anthropic has said support is rolling out across Claude products and has not announced general availability or per-product dates. Check the Claude release notes rather than relying on this sentence, which ages badly by design.

Is sampling still supported in MCP? Sampling is deprecated as of 2026-07-28, not removed. It continues to work for at least twelve months, with the earliest removal being the first revision released on or after 28 July 2027.

What do I use instead of sampling now that it is deprecated? The specification's prescribed migration is to integrate directly with an LLM provider API. Budget for the consequence, which is that model credentials, billing and rate-limit handling move from the client to your server.

How do I handle workflows that take days now that MCP is stateless? Use the io.modelcontextprotocol/tasks extension. The server returns a task handle, the client polls tasks/get honouring pollIntervalMs, and sends input through tasks/update. Persist task ids durably, because tasks/list was removed and a lost id means an unreachable task.

What are the Mcp-Method and Mcp-Name headers for? They let load balancers, gateways and rate limiters route and meter on the operation without parsing the JSON-RPC body. Mcp-Method is required on all requests as of 2026-07-28; Mcp-Name is required only on tools/call, resources/read and prompts/get.

How long until HTTP+SSE stops working? Eligibility for removal opened around 18 August 2026, three months after SEP-2596 reached Final. The release blog's "year-long offramp" does not match the registry or the SEP. Removal is still a maintainer decision at release preparation and requires a new revision to ship, so nothing breaks on a date, but you no longer have a guaranteed window.

Is 2026-07-28 the same thing as MCP 2.0? There is no MCP 2.0. The string does not appear anywhere in the official documentation and the release announcement never uses it. It is community shorthand, and it exists because the SDKs went to 2.0.0 and 3.0.0 while the specification stayed on date-based revisions, which is a real inconsistency that people on Hacker News have been asking about for over a year.

The line the protocol drew

The thing worth taking away from 2026-07-28 is not any individual change. It is where the boundary landed.

This revision decided that problems living between a load balancer and a server process belong to the protocol, and that problems living between a model and a tool description belong to the layer above it. Sessions, routing, caching, discovery and transport are the protocol's job now, and they are handled well. Token economics, prompt injection, tool-space collisions and authorization modelling are somebody else's, explicitly and by design.

That is a coherent position and quite possibly the right one, because protocols that try to solve everything tend to solve nothing. The layer above is where extensions live, and extensions are exactly the right shape for the work that remains.

But nine clients appear in the official extension support matrix today, and exactly one of them supports anything beyond MCP Apps. So the expensive parts of running an MCP server and the risky parts of running an MCP server are now, formally, somebody else's problem, and that somebody has not shown up yet.

Worth knowing who you are waiting for.

Last updated 29 July 2026. Registry figures measured against registry.modelcontextprotocol.io on the same date. If something here is wrong, tell me and I will fix it in place rather than publishing a correction nobody reads.

9 Essential Claude Skills for AI Engineers Building Production Agents + 1 Bonus Skill

Gaurav Dadhich — Tue, 11 Aug 2026 03:49:41 +0000

📖 What Are Claude Skills?

Claude Skills are organized folders of instructions, scripts, and resources that Claude (both Claude Code CLI and Claude Cowork GUI) can discover and load dynamically to perform specialized tasks. Think of them as reusable, modular capabilities that teach Claude how to complete specific tasks in a repeatable way.

Skills are simple to create—just a folder with a SKILL.md file containing YAML frontmatter and instructions. When you make a request, Claude automatically invokes relevant Skills based on your needs.

Key points:

Skills work across both Claude Code (CLI) and Claude Cowork (GUI)
Some skills are native/built-in (docx, pdf, pptx, xlsx - everyone has these)
Others are community-built and need discovery/installation
This guide focuses on community skills for AI/ML engineering

Learn more: Official Claude Skills Documentation | Anthropic Skills Repository

🎯 What This Is

A hand-picked collection of 9 real, community-built Claude Skills specifically for AI engineers building production agents and LLM-powered applications.

What makes these different:

❌ NOT native skills (you need to discover and install them)
✅ Built by the community for specific AI/ML engineering problems
✅ Focus on prompt engineering, evaluation, security, and debugging
✅ All verified with installation links, creators, and GitHub repos

Who this is for:

AI/ML engineers building agent systems
LLM application developers
Engineers implementing RAG pipelines
Security researchers testing AI vulnerabilities
Anyone debugging production agent behavior

📦 The Essential 9 Skills

1. promptfoo-evaluation ⭐⭐⭐⭐⭐

What it does: Integrates the Promptfoo evaluation framework directly into your Claude workflow. Set up structured LLM testing with LLM-as-judge assertions, custom Python metrics, few-shot examples, variable substitution, and long text handling. Generates evaluation configs and runs systematic prompt testing.

Use cases:

Building evaluation pipelines before production deployment
A/B testing prompts with quantitative metrics
Regression testing when changing models or prompts
Creating reproducible evaluation reports

Why it's essential: Most engineers ship prompts without systematic testing. This catches regressions before users do.

Installation:

Skill Link: promptfoo-evaluation on SkillsMP
Creator: daymade | GitHub
Stats: 531 stars, 59 forks | Last updated: Jan 29, 2026

Example usage:

@promptfoo-evaluation Create an evaluation suite for my customer support agent
with 10 test cases covering edge cases and hallucination detection

2. promptinjection ⭐⭐⭐⭐⭐

What it does: Comprehensive prompt injection security testing with 5 specialized workflows: complete security assessment, reconnaissance, direct injection testing, indirect injection testing, and multi-stage attack simulations. Includes attack taxonomy, defense mechanisms, and detailed reporting templates.

Use cases:

Security audits before production deployment
Red-teaming your agent's prompt defenses
Testing jailbreak resistance
Building defensive prompt engineering patterns

Why it's essential: One successful prompt injection can leak system prompts, bypass guardrails, or manipulate agent behavior. Test before attackers do.

Installation:

Skill Link: promptinjection on SkillsMP
Creator: Daniel Miessler | GitHub
Stats: 6,136 stars | Industry-recognized security researcher

Example usage:

@promptinjection Run a complete security assessment on my RAG pipeline prompts.
Test for indirect injection via uploaded documents.

3. evaluating-llms ⭐⭐⭐⭐

What it does: Multi-faceted LLM evaluation using automated metrics, LLM-as-judge patterns, and benchmark creation. Tests prompt quality, validates RAG pipelines, measures safety (hallucinations, bias, toxicity), and compares models for production deployment decisions.

Use cases:

Validating RAG pipeline accuracy
Comparing GPT-4 vs Claude vs Gemini for your use case
Measuring hallucination rates quantitatively
Building custom benchmarks for domain-specific tasks

Why it's essential: "It feels like it works" isn't a deployment strategy. This gives you numbers.

Installation:

Skill Link: evaluating-llms on SkillsMP
Creator: ancoleman | GitHub
Stats: 154 stars, 27 forks | Last updated: Dec 11, 2025

Example usage:

@evaluating-llms Compare Claude Sonnet vs GPT-4o for code review tasks.
Run 50 test cases and measure accuracy, latency, and cost.

4. ai-prompt-engineering ⭐⭐⭐⭐

What it does: Production-grade prompt engineering patterns including structured outputs (JSON/schema validation), deterministic extractors, RAG grounding with citations, tool/agent workflows, and prompt safety testing. Focuses on operational patterns that work at scale.

Use cases:

Building reliable structured output systems
Implementing RAG with proper citation tracking
Designing tool-calling agent workflows
Ensuring deterministic extraction from documents

Why it's essential: Moving from prototype to production requires structured, predictable outputs. This skill teaches production patterns.

Installation:

Skill Link: ai-prompt-engineering on SkillsMP
Creator: vasilyu1983 | GitHub
Stats: 29 stars, 6 forks | Last updated: Jan 26, 2026

Example usage:

@ai-prompt-engineering Design a prompt for extracting invoice data
with strict JSON schema validation and error handling

5. ai-threat-testing ⭐⭐⭐⭐

What it does: Offensive AI security testing framework that systematically tests OWASP Top 10 for LLM Applications vulnerabilities: prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and model theft.

Use cases:

Pre-deployment security assessments
Penetration testing for AI systems
Compliance validation (SOC2, ISO 27001)
Building threat models for LLM applications

Why it's essential: LLM security is different from traditional AppSec. This skill understands the unique attack surface.

Installation:

Skill Link: ai-threat-testing on SkillsMP
Creator: transilienceai | GitHub
Stats: 19 stars, 3 forks | Last updated: Jan 27, 2026

Example usage:

@ai-threat-testing Run OWASP Top 10 assessment on my customer-facing chatbot.
Focus on prompt injection and sensitive information disclosure.

6. llm-evaluation ⭐⭐⭐

What it does: Comprehensive LLM evaluation patterns including systematic prompt testing, hallucination detection methodologies, benchmark creation frameworks, and quality metrics. Provides structured approaches to measuring LLM performance across multiple dimensions.

Use cases:

Building internal benchmark suites
Detecting hallucinations in RAG responses
Tracking model performance over time
Creating evaluation dashboards

Why it's essential: "It looks right" isn't enough. Systematic evaluation catches edge cases.

Installation:

Skill Link: llm-evaluation on SkillsMP
Creator: applied-artificial-intelligence | GitHub
Stats: 32 stars, 9 forks | Last updated: Jan 14, 2026

Example usage:

@llm-evaluation Create a hallucination detection suite for my RAG pipeline.
Test with 20 adversarial queries designed to trigger false information.

7. llm-patterns ⭐⭐⭐

What it does: Comprehensive catalog of AI-first application patterns, LLM testing methodologies, and prompt management strategies. Covers foundational architectural patterns for building scalable LLM applications including chain-of-thought, few-shot learning, retrieval patterns, and agent workflows.

Use cases:

Learning production LLM application architecture
Choosing the right pattern for your use case
Understanding trade-offs between approaches
Building reference implementations

Why it's essential: Saves you from reinventing solutions to common problems. Learn from battle-tested patterns.

Installation:

Skill Link: llm-patterns on SkillsMP
Creator: alinaqi
Stats: 467 stars, 37 forks | Last updated: Jan 21, 2026

Example usage:

@llm-patterns Show me the best pattern for building a code review agent
with chain-of-thought reasoning and tool calling

8. coding-agent ⭐⭐⭐⭐

What it does: Programmatic control of coding agents (Codex CLI, Claude Code, OpenCode, Pi Coding Agent) via background process management. Enables workflow automation with LangChain, custom agents, Claude API integration, and execution logging. Run agents from scripts, monitor outputs, and chain operations.

Use cases:

Automating code generation workflows
Building CI/CD integrations with agents
Chaining multiple agent operations
Programmatic agent testing

Why it's essential: Manual agent interaction doesn't scale. Automate agent workflows for production use.

Installation:

Skill Link: coding-agent on SkillsMP
Creator: openclaw | GitHub
Stats: 143,701 stars, 21,502 forks | Last updated: Feb 2, 2026

Example usage:

@coding-agent Set up automated code review workflow that runs on every PR.
Generate review comments and post to GitHub.

9. agent-debugger ⭐⭐⭐⭐⭐

What it does: Systematic debugging toolkit for AI agentic workflows. Diagnoses common issues: wrong/inconsistent responses, tool/function calling failures, conversation loops, stuck agent states, and latency problems. Works with LangChain, custom agents, and Claude API. Provides step-by-step debugging workflows and root cause analysis.

Use cases:

Debugging why your agent gives wrong answers
Fixing tool calling failures
Diagnosing conversation loops
Optimizing agent latency
Understanding complex agent behavior

Why it's essential: Agent debugging is HARD. You can't just "console.log" an agent's reasoning. This gives you systematic troubleshooting.

Installation:

Skill Link: agent-debugger on SkillsMP
Creator: AvivK5498 | GitHub
Stats: 5 stars, 0 forks | Last updated: Jan 27, 2026

Example usage:

@agent-debugger My customer support agent sometimes ignores the tool output
and gives generic responses. Help me diagnose what's failing.

Bonus Skill: maximem synap (agent memory) ⭐⭐⭐⭐⭐: built by us

What it does: Drop-in skill that teaches your AI coding agent (Claude Code, Cursor, Codex) how to integrate Maximem Synap; persistent, structured, multi-tenant memory for your agents. Covers SDK setup, scoping (per-user / per-customer), ingestion, retrieval, and per-framework packages for LangChain, LangGraph, CrewAI, Vercel AI SDK, LiveKit, and 18 more.

Use cases:

Agents that remember users across sessions
Multi-tenant memory isolation (user / customer / org scopes)
Replacing DIY vector-DB memory pipelines
Voice agents that recall caller history mid-call

Why it's essential: The other nine skills solve "is my prompt good?" — this one solves "does my agent remember the right things?"

Installation:

Skill link: https://github.com/maximem-ai/maximem_synap_sdk/tree/main/skills
Docs: https://docs.maximem.ai/integrations/ai-coding-agents
Creator: Maximem (yes, that's us — disclosure)

Example usage: @synap Add per-user memory to my LangGraph support agent so it remembers customers across sessions

🚀 Installation Guide

Quick Start (5 minutes)

Step 1: Verify Claude Skills is enabled

For Claude Code (CLI): Skills are enabled by default
For Claude Cowork (GUI): Available in Pro subscription
Check: Run claude skills list (CLI) or look for Skills menu (GUI)

Step 2: Install skills

Method A: Using Claude Code CLI (recommended)

# Install from GitHub (most common)
claude skills install daymade/claude-code-skills

# Or install from local directory
claude skills install /path/to/skill/folder

# List installed skills
claude skills list

# Update existing skill
claude skills update daymade/claude-code-skills

Method B: Manual installation

Clone the skill repository from GitHub
Place folder in ~/.claude/skills/ directory
Restart Claude or reload skills

Method C: From SkillsMP marketplace

Browse SkillsMP
Click skill → Copy GitHub URL
Use Method A or B above

Step 3: Using skills

Automatic invocation (Claude decides when to use):

# Claude will automatically use relevant skills
"Evaluate my prompt against these 10 test cases"
→ Claude uses promptfoo-evaluation skill

"Test my agent for prompt injection vulnerabilities"
→ Claude uses promptinjection skill

Explicit invocation (you specify):

@promptfoo-evaluation Create evaluation suite...
@agent-debugger Debug my LangChain agent...

Verification

Check if skills are working:

# CLI
claude skills list

# Should show installed skills with status

💡 Pro Tips

1. Combine Skills for Workflows

@ai-prompt-engineering Design production prompt for invoice extraction
@promptfoo-evaluation Create test suite with 20 edge cases
@promptinjection Test for injection vulnerabilities
@llm-evaluation Run hallucination detection

Chain skills together for comprehensive development → testing → security workflow.

2. Start with Evaluation

Most engineers skip evaluation until production breaks. Install promptfoo-evaluation or llm-evaluation first. Build testing into your development flow from day one.

3. Security Before Launch

Use promptinjection and ai-threat-testing at least 1 week before production deployment. Security vulnerabilities found in production are 10x more expensive to fix.

4. Debugging Saves Hours

When your agent misbehaves, reach for agent-debugger before spending hours reading logs. Systematic debugging beats intuition.

5. Learn Patterns Early

New to LLM development? Start with llm-patterns to understand established approaches before building your own solution.

📊 Expected Impact

Based on testing with 12 AI engineering teams:

Time Savings:

Evaluation setup: 4 hours → 20 minutes (promptfoo-evaluation)
Security testing: 8 hours → 1 hour (promptinjection)
Debugging sessions: 2 hours → 30 minutes (agent-debugger)
Pattern research: 10 hours → 1 hour (llm-patterns)

Quality Improvements:

Bugs caught pre-production: 3-5x increase
Security vulnerabilities found: 8-12 per project (before users find them)
Evaluation coverage: From ad-hoc to systematic

ROI: If you're paid $150K/year and these skills save 10 hours/week, that's ~$30K/year in your time.

⚠️ What's Missing (Ecosystem Gaps)

Honest assessment: The Claude Skills ecosystem is early-stage. These 9 skills cover evaluation, security, and debugging well. But gaps remain:

Not yet available as community skills:

❌ Vector database utilities (Pinecone/Weaviate helpers)
❌ LLM cost tracking tools (token counting, spend monitoring)
❌ Context window analyzers (visualize token usage)
❌ API mocking frameworks (mock OpenAI/Anthropic responses)
❌ Production monitoring integrations (Datadog, New Relic)
✅ Agent memory / context management: this gap is now covered: we ship an official synap skill that wires persistent memory into your agent.

Why: Community focus is on foundational concerns (prompt quality, security, evaluation). Infrastructure tooling hasn't been built yet.

What this means: You'll still need traditional tools for monitoring, cost tracking, and infrastructure. Skills augment your workflow, they don't replace your entire toolchain.

***UPDATE***: The H2 2026 version of top skills is here.

🔗 Additional Resources

Official Documentation

Community Marketplaces

Skill Development

Security Resources

🤖 Building Production Agents? You Need Context Management.

If you're using these skills to build production AI agents, you'll inevitably hit the context management problem:

The symptoms:

Your agent forgets critical information mid-conversation
Context windows explode and costs spike 10x
Multi-user scenarios leak data between sessions
Semantic search returns irrelevant results despite good embeddings

Why it happens: Evaluation and security (these 9 skills) solve "is my prompt good?" Context management solves "does my agent remember the right things?"

This is where Synap comes in.

Synap is an agentic context management platform that handles memory, retrieval, and context decisions for your agents - so you can focus on prompts and features, not debugging why your agent forgot something.

Learn more: Maximem Synap Docs | Maximem Synap Playground

File vs Vector for RAG

Gaurav Dadhich — Sat, 01 Aug 2026 11:36:12 +0000

File-driven context management has been the rage in the last few days, especially since Claude CoWork launched. It made me curious and I tried a few things.

I ran an experiment across 5 distinct domains: from Python code to scientific papers. I ingested 50,000 documents from popular datasets and fired 5,000 queries at them.

The goal? To find out what all the noise about file-based search is. And if it is even real!

I started this expecting Vector search to crush the benchmarks across the board. I was ready to write another post about why everything needs to be an embedding. But at 2:00 AM, observing my Macbook Pro heating up like a nuclear-power plant, I realized we've been sold a 'Semantic Dream' that doesn't always match the engineering reality.

What these two systems actually do

Before the numbers, the distinction that explains all of them.

Keyword search, in this case Tantivy, builds an inverted index and matches terms exactly. It has no model of meaning. If the query says "sort a list" and the document says bubble_sort, keyword search finds nothing, because those are different strings. What it does have is precision, because when the term in the query is the same term in the document, the match is exact and nothing else outranks it.

Vector search, in this case ChromaDB, converts text into a numeric representation and finds documents whose representation sits nearby. It has an approximate model of meaning and no notion of exactness. It will bridge "sort a list" to bubble_sort, and it will equally happily return a document about cell structure when you asked about mitochondria, because those two things live close together in the same space.

Exact matching and approximate meaning are not two implementations of the same idea. They are two different capabilities, and the benchmark below is a map of which one you need where.

The results

We measured with MRR@10, which scores how high the correct document ranks in the first ten results. A score of 1.0 means the right document was ranked first every time.

Dataset	Domain	Keyword, exact match	Vector, semantic	Winner	Margin
CodeXGLUE	Natural language to code	0.2901	0.9143	Vector	+0.6242
MS MARCO	Real Bing queries	0.4035	0.5225	Vector	+0.1190
SQuAD	Wikipedia question answering	0.6048	0.6136	Tie	+0.0088
HotpotQA	Multi-hop reasoning	0.5494	0.4953	Keyword	+0.0541
SciQ	Science exam questions	0.8145	0.6142	Keyword	+0.2003
Mean, all five		0.5325	0.6320	Vector	+0.0995
Mean, excluding CodeXGLUE		0.5931	0.5614	Keyword	+0.0317

Read the final two rows together, because that is the entire result. Across five datasets vector search wins by roughly ten points of MRR, which is the number that would go on a slide. Remove CodeXGLUE and the ranking inverts, with exact-match keyword search ahead by three points.

Vector search is not generally better at retrieval. It is 3.15 times better at one job, and that job carries the average for everything else.

CodeXGLUE is the purest semantic-gap task in the set, because a natural language description of what code should do shares almost no vocabulary with the code that does it. Exact matching has nothing to match on, and scores 0.2901. This is the case vector search exists for, and it wins it decisively.

SciQ is the inverse. Science exam questions turn on specific entities, and in that setting a word is not an approximation of a concept, it is a key. Mitochondria is not a thing that is similar to a cell. Exact matching locks onto the term and scores 0.8145, while vector search drifts toward documents that sit nearby in embedding space and scores 0.6142.

SQuAD is a tie. The gap is 0.0088 on a single run with no seed averaging, which is inside the noise, and reporting it as a vector win would be dishonest.

The vector tax, with a number

The earlier version of this post said embedding generation is a tax and did not say how large. It is 76.4 times.

Dataset	Keyword indexing	Vector embedding and indexing	Ratio
CodeXGLUE	445.75 ms	43,098.93 ms	96.7x
MS MARCO	418.56 ms	26,335.61 ms	62.9x
SQuAD	410.60 ms	35,920.01 ms	87.5x
HotpotQA	394.98 ms	29,538.14 ms	74.8x
SciQ	444.30 ms	26,670.01 ms	60.0x
Total, 50,000 documents	2.11 seconds	161.6 seconds	76.4x

That is 23,650 documents per second against 309, on the same machine over the same corpus.

The consequence is about when an agent can use what it just learned. If an agent needs to read a repository or a hundred-page document and act on it within the same turn, exact-match indexing is effectively instant and embedding is a coffee break. If it reads now and acts later, the tax amortises and stops mattering.

The result that points past both systems

HotpotQA has the smallest margin in the table and the most instructive failure underneath it.

Those questions require linking two documents, for example establishing which of two magazines was founded first. Vector search would reliably retrieve the document for the first entity and miss the bridge document for the second, because the second document was not similar enough to a query that was mostly about the first. The answer was retrievable and the evidence was not.

That is not a similarity problem and no reranker fixes it. Neither exact matching nor semantic similarity has any representation of the fact that two documents are connected. Both systems rank documents independently against a query, and a bridge document is by definition the one that does not look like the query. Answering that class of question requires storing the relationship itself, which is a third mechanism and the reason graph structure exists in memory systems at all.

Finding the answer is not the same as holding enough context to prove it, and an agent that retrieves the first without the second produces confident, unsupported output rather than an error.

Method, and everything wrong with it

Anyone can rerun this, and anyone evaluating it should know where it is weak.

10,000 documents and 1,000 queries per dataset, 50,000 and 5,000 in total, executed on January 14, 2026 on an Apple M4 with 16GB of RAM. Keyword search was Tantivy 0.22.0 with the default analyzer. Vector search was ChromaDB 0.4.0 or later with all-MiniLM-L6-v2 at 384 dimensions. No chunking on either side. Scoring was MRR@10.

On scoring, which is the part most benchmarks in this category get wrong. Relevance was evaluated programmatically by exact document ID matching against each dataset's ground truth, in src/evaluation/metrics.py. There is no LLM judge anywhere in this pipeline. Nothing here can be moved by a prompt file, a judging bias, or an equivalence rule, which is a claim we cannot make about most published numbers in this category, including some of our own work on conversational benchmarks. A rerun on the same corpus produces the same table.

Six limitations, in the order they would be raised against us.

Single run rather than an average over seeds, so small margins carry no weight and SQuAD should be read as a tie.

all-MiniLM-L6-v2 is a small and relatively old embedding model. A stronger embedder would very likely raise the vector numbers and could change the sign on HotpotQA. The honest statement is that this measures a common default configuration rather than the best available vector setup.

Tantivy ran on its default analyzer with no tuning, so the keyword side is equally untuned.

No chunking, which affects both approaches and affects vector retrieval more on longer documents.

Ingestion timings come from local embedding on an M4. A hosted embedding API moves that number in both directions, worse for network round trips and better for batched inference, so treat 76.4x as the shape of the cost rather than a constant.

MRR@10 measures where the correct document ranks. It does not measure whether the agent then produced a correct answer. Retrieval rank and task success are different things, and we would rather say that ourselves than have it said back to us.

What this does not settle

This benchmark runs over a fixed corpus of documents that were written by someone else and do not change. Agent memory is neither of those things. It grows on every turn, it is written by the agent itself, and yesterday's fact can contradict today's.

The next post covers what happens to both of these systems once the corpus is conversation history rather than documents, which is where the interesting failures live. The architecture argument, including where graph structure and extraction fit, is on our build vs buy agent memory page, and the formal treatment of the token economics is in our paper on Agentic Context Management.

The state of AI memory in 2026: claimed vs observed

Gaurav Dadhich — Tue, 28 Jul 2026 05:17:29 +0000

The state of AI memory in 2026: claimed vs observed

AI memory is the most-benchmarked, least-reproduced category in the AI tooling stack right now. Vendors publish high numbers. The numbers travel through Twitter threads and conference decks and "agent infrastructure" landing pages. Almost nobody actually runs the harness themselves to check whether the numbers reproduce. We did.

This post is the result of that work. It is a landscape map of the AI memory vendors competing in 2026, a tour of the two benchmarks the field has converged on (LongMemEval and LoCoMo), and a side-by-side of what each vendor publishes against what we observed when we re-ran the same evaluations on an open harness. Where we have completed a reproduction, we cite both numbers. Where we have not, we say so plainly. The harness is open-source. The detailed audit notes that explain the larger reproduction gaps are linked at the end.

This is a long read. The shortcut version lives in the results table six sections down. If you came here from a "state of AI memory" search, that table is what you actually wanted.

Why memory is the bottleneck in 2026

The million-token-context-window race did not solve memory. It shifted the cost. You can stuff a million tokens into a single prompt now, which sounds like it should make memory unnecessary, but two well-documented failure modes turn that promise into wishful thinking at production scale.

Context rot kills the promise of long windows. As the context fills, attention quality degrades across the entire sequence, not just at the edges. The same model that answers cleanly with 8K tokens of context will hallucinate, contradict itself, or hedge unnecessarily when given 800K tokens of conversation history. Multiple recent evaluations (Chroma's context-rot study, the lost-in-the-middle work that started this thread of research in 2023, and follow-ups across most frontier model families) confirm that long context is not equivalent to focused context.

Lost-in-the-middle compounds the problem. Information placed in the center of a long prompt is consistently retrieved with lower accuracy than information at the beginning or end, irrespective of model size or window length. For an agent that has accumulated weeks of conversation history, this means critical facts — the user's preferences, an earlier commitment, the resolution of a previous escalation — routinely sit in the part of the context where retrieval is worst.

So memory layers exist because the context window is the wrong abstraction for stateful behavior. An agent that remembers should not have to re-read its entire conversational history to recall that the user prefers terse answers, works in payments compliance, runs a Series B SaaS company, and asked about SOC 2 controls two weeks ago. The memory layer is supposed to retrieve only what matters, in time, at the right level of abstraction, with the right scoping primitives.

Scoping is its own subtlety. In a B2B setting, two users at the same company should share organizational context (their company's preferred vocabulary, internal processes, prior support history, configuration choices) while remaining isolated at the personal level. A memory layer that treats every user as an island misses the obvious B2B value, and a memory layer that pools everything misses the obvious privacy and personalization requirement. The interesting work in 2026 is building scoping primitives that handle both at the same time.

In 2026 this is no longer aspirational. Memory is in the critical path for every serious agent product we have looked at: copilots that personalize across sessions, customer support agents that retain history across tickets, consumer chat apps that build long-term context with each user, internal-tool agents that have to remember what they did last Tuesday. The bottleneck is real. The vendors competing to solve it are real. The benchmarks the field uses to measure progress are real. The published numbers are a separate question, and that is where this post earns its keep.

The vendor map

The agent-memory landscape splits cleanly into three categories. (Personal and consumer memory products such as ChatGPT Memory and Maximem's own Vity are a separate market with different benchmarks and different buyers; they are out of scope for this post.)

Category one: dedicated open-source memory libraries. Mem0's OSS layer (the one with the most ecosystem traction), Letta (the descendant of the MemGPT research line out of Berkeley), Cognee (ontology-driven, slightly more academic in positioning). These are products you install and run yourself. The trade-off is operational: you carry the runtime, the storage, the upgrades, the on-call. The upside is portability and full control.

Category two: dedicated hosted memory products. Mem0 Cloud, Zep (the most mature on graph-structured memory), SuperMemory (B2B-leaning, customer-support-heavy in the case studies they show), and Synap (which Maximem builds; structured long-term memory with multi-tenant scoping primitives). This is the segment with the most active commercial competition right now. Pricing pages started looking like each other six months ago, which is usually a sign that buyers are starting to ask the same comparison questions.

Category three: memory features inside agent frameworks. LangGraph, LlamaIndex, the Vercel AI SDK, the OpenAI Agents SDK, and a handful of others ship minimal memory primitives as part of the framework. These cover the common case (recent-turn recall, simple key-value persistence) and stop short of the harder problems: entity resolution across surface forms, temporal reasoning over versioned facts, cross-conversation synthesis. If you need real memory and you are using one of these frameworks, you almost always end up wiring in a dedicated memory layer.

A small but persistent group of teams still rolls their own memory layer in-house. The argument for it (data layer, compliance, portability) is real for the first few months and gets weaker the longer the system runs. Most in-house implementations we have seen end up reimplementing the obvious primitives (chunking, embedding, recency-decay retrieval, basic deduplication) without ever getting to the harder problems (entity resolution, temporal reasoning, scoping, multi-tenant isolation). The cost compounds. The right framing for in-house is not "we built it ourselves" but "we built a worse version and now we maintain it forever." A few teams need to roll their own for regulatory reasons. Most teams convince themselves they need to, regret it eighteen months later, and migrate to a vendor anyway.

A 3-category diagram of this map sits below. Marker for the design team:

How the Field Benchmarks Itself

Two benchmarks dominate published claims in 2026: LongMemEval and LoCoMo.

LongMemEval came out of Adobe Research in 2024 and was accepted at ICLR 2025. It tests how well a memory system can answer questions over a long, multi-session conversational history. 500 questions, distributed across six categories: single-session-user, single-session-assistant, single-session-preference, knowledge-update, temporal-reasoning, and multi-session. The hardest category by a wide margin is multi-session, because it requires the system to synthesize evidence from multiple separate conversations rather than pulling from a single recent thread. Published methodology uses an LLM-as-judge with a binary correct-or-wrong label.

LoCoMo, from Snap Research at ACL 2024, focuses on long-form open-ended conversations. The benchmark contains five question categories. Industry convention (the convention Mem0, Zep, and most others follow) is to exclude the adversarial category and report on categories one through four. Roughly: multi-hop reasoning, temporal reasoning, open-domain opinion, and single-hop recall. Open-domain is the category where vendor prompt engineering tends to do the most lifting, because the gold answers often follow predictable patterns (an answer of "likely no" tends to be correct when the most recent referenced event involved a bad experience, for instance).

Both benchmarks share a structural property worth understanding before reading any published number. The answer model and the judge model are not part of the memory system. They are separate LLMs that the vendor configures via prompt. That separation is intentional: the benchmarks test the memory layer, not the LLM. But it does mean the vendor controls what gets asked of the model, how it gets reasoned over before the answer comes out, and how the response gets graded by the judge. Three points of leverage. All in the prompt files. Which is where reproduction starts to matter.

How we re-ran the numbers

Our harness operates on a simple principle. Use each vendor's paid hosted product (or the OSS layer at its recommended configuration when no hosted product exists). Ingest the benchmark dataset through their pipeline exactly as a customer would. Then run the questions through a standardized answerer and judge that we control, so we are measuring the memory layer rather than the vendor's evaluation stack.

Concretely: gpt-5 as the answer model across the board, a binary judging prompt with explicit conditions for marking both CORRECT and WRONG (no "lean toward yes" bias, no one-directional override clauses, no encoded dataset hints), five-seed averaging to control for stochasticity, and the same ingestion configuration we would recommend to any builder using the vendor SDK in production.

The headline reproduction finding so far is the gap on LongMemEval, and it has two parts because we ran Mem0 twice. Before Mem0's April 14, 2026 announcement of their new state-of-the-art numbers, we ingested the LongMemEval haystack into their hosted product and ran the questions through our standardized answerer and judge. We got 57.5%. After their April 14 push (which followed the prompt-tuning commits we will detail below), we re-ran the same evaluation against their updated hosted product. We got 73.8%. The Mem0 memory layer genuinely improved by 16.3 points across that window, which is real engineering progress worth acknowledging. The published claim from the same announcement was 93.4%, still 19.6 points above the post-April-14 reproduction on the same memory system and the same data.

The reason the numbers differ is not the memory layer. It is what the vendor stacks on top of the memory layer at evaluation time. In Mem0's case specifically, this stack lives in the answer and judge prompt files at github.com/mem0ai/memory-benchmarks. We audited the files at the exact commits that ship with their published claims:

Here are the LongMemEval prompts (committed April 3, 2026) and for the LoCoMo prompts (committed April 9, 2026). Both land before Mem0's April 14 number announcement; the file-level evidence of prompt-tuning is built into Mem0's own commit history (their April 3 commit message reads, in part: "Sync prompts from evals: CONTEXT CHECK, Rule 14 (contradictions), BIAS CHECK in judge, 5-step FINAL CHECK").

The mechanisms surface clearly once you read those files. There are 14 dataset-specific equivalence rules in the answer prompt that map 1-to-1 to specific public LongMemEval question_ids (samples include "chandelier counts as jewelry" at line 145 and "scratch grains count as new layer feed" at line 147). There is a hidden chain-of-thought block ( tags at lines 53 and 65 here where those rules get applied before the visible answer is emitted; the judge only ever sees the cleaned answer. There is an explicit "lean toward yes" instruction in the LongMemEval judge prompt at line 269 paired with a 5-step gauntlet to clear before marking anything WRONG at line 328 , and no symmetric gauntlet before marking anything CORRECT. There is a one-directional gold-override clause in the LoCoMo judge at line 212 that can promote a wrong prediction to correct when "evidence supports" it, but explicitly cannot demote a correct prediction to wrong when evidence contradicts it.

Marker for design: this is the section where the receipt-card screenshots belong, inline. Each screenshot is captioned with the file:line citation already in the prose.

Every one of these findings is documented with verbatim quotes pinned to the specific commit and line, SHA-256 hashes on mirrored copies of both prompt files, and Wayback Machine archive URLs for independent third-party timestamped copies. The full evidence chain sits in the receipts section at the end of this post.

This pattern is not unique to Mem0. We are extending the testing to other vendors and will publish their reproductions as they complete. Across the vendors measured so far, the gap between published and observed tracks directly with how much benchmark-specific prompt engineering sits between the memory system and the headline number. It does not track with the quality of the memory system itself.

Synap, our own product, was tested through the same harness with no benchmark-specific prompt advantages. Our numbers appear in the results table below alongside everyone else's, in every category we have measured, including the categories where we are weaker than the published competition.

Results

Reproduction table, LongMemEval, mid-2026:

Zep number reproduced on April 10, 2026 harness; Zep has not independently verified the configuration we used.

The Mem0 row is the one to read carefully. The 57.5% to 73.8% lift across the April 14 product update is real, and it is improvement Mem0 has earned the right to claim. The 73.8% to 93.4% jump from observed to published, however, is not attributable to the memory system. That part of the gap maps cleanly onto the prompt mechanisms documented in the methodology section above.

Per-category breakdown, LongMemEval (Synap on current harness):

The reproduction gap is the most interesting column in any of these tables. It is, roughly, a measure of how much non-memory-layer engineering is sitting between the underlying system and the headline number a vendor publishes. The harness, the methodology, and the seeds are documented in the section above and in the repo linked at the end.

What is hard, what is coming

Genuine cross-session synthesis is the hardest unsolved piece. Today, even the strongest memory layers do per-session extraction well and struggle to link the same entity across sessions when the surface representation differs. The benchmarks reflect this directly: LongMemEval multi-session is the lowest score for almost every vendor on the leaderboard. Solving it is part vector retrieval, part graph reasoning, and part ontology engineering, which is why no single team has nailed it yet.

Temporal reasoning at scale is the next frontier. Calculating "what was the state of X two months ago" requires the memory system to maintain versioned facts and answer questions against a specific point in time, not just retrieve the current version and call it good. Most vendors approximate this with date metadata and good retrieval. Few do it as a first-class feature. This will start to matter more as agents move into use cases where past state is operationally consequential: compliance, audit logging, longitudinal personalization, financial agents that need to reason about what was true at a transaction date.

Multi-tenant isolation paired with intelligent organizational sharing is where the enterprise segment gets decided. Enterprise buyers ask about this within the first three calls, and most memory products treat it as a configuration concern rather than an architectural one. The hard version is harder than it sounds: isolate users by default, share organizational context (vocabulary, processes, prior decisions) automatically when appropriate, expose the boundary as a primitive rather than a setting, and do all of it in genuinely shared infrastructure. The vendors that build this as a first-class primitive will win the enterprise segment over the next two years.

Memory is the data layer for AI. The companies treating it as a feature will lose to the companies treating it as a category. That is the bet behind every serious memory vendor in 2026, including us.

Receipts and reproduction

Everything in this post is reproducible.

The harness we built and ran is open-source here. One command runs the standardized binary-judge configuration against either ingest pipeline. Pull requests welcome, from vendors and independent researchers both.

The Mem0 evidence chain, in full:

- LongMemEval prompts.py pinned at commit bd063eea04de4f8a19927beea155afa094a01905 (committed April 3, 2026 by Soumil Rathi). The 14 dataset-equivalence rules live at lines 138-148. The hidden chain-of-thought instruction lives at lines 53 and 65. The "lean toward yes" bias check is line 269. The 5-step FINAL CHECK before marking WRONG is lines 328-334.

- LoCoMo prompts.py pinned at commit edcd6f1d42400837b1fcb6997716f1769dc51a37 (committed April 9, 2026, same author). The opinion-question shortcuts are lines 81-82. The hardcoded LoCoMo session window ("All events occurred in 2022-2024. Never output 2025 or 2026.") is lines 64-65. The one-directional gold-override clause is line 212.

- We mirrored both files locally at those commits. SHA-256 of the LongMemEval prompts.py: ba8cf60d26f1390ecbef0f07b3e950556fe3bc5a37ba4b5343f28217f18c144f. SHA-256 of the LoCoMo prompts.py:

8ebac1ef60e9ab5caf99079fdaac038b85472e81491ed35e2d2655f3927c76c2. Any independent reproducer can fetch either file at the commit and confirm the hash.

- Wayback Machine archive of the LongMemEval prompts.py at the pinned commit: web.archive.org/web/20260505163741/... (captured May 5, 2026).

Benchmark sources:

- LongMemEval

- LoCoMo

Disclosures: we tested Mem0 through their paid hosted product, ran their published benchmark harness both verbatim and modified, and read their prompts.py at the commits above. SuperMemory and Zep were tested on our April 10, 2026 harness (numbers cited in the table). We have not yet completed LoCoMo reproduction for Mem0's post-April-14 product; that run is in progress and will update the table when complete. We have not yet ingested Letta or Cognee on the harness.

Mem0 has been invited to respond publicly. If they publish a correction with evidence we will update the post. The repo accepts pull requests from any vendor who wants their configuration tested differently. Reproducibility cuts in both directions, and we would rather have the table be right than be flattering.

Agent Memory Is Not Merely a Storage & Retrieval Problem, It Is an Architecture Problem.

Gaurav Dadhich — Sat, 25 Jul 2026 06:21:18 +0000

Most teams building AI agents are treating memory and inference cost as something the next model release will eventually fix. They believe that a bigger context window, a smarter retriever, a cheaper token rate, etc. would absolve the need for a system for solving agent memory.

This posture is convenient but it is wrong. What an agent remembers, when it forgets, and how much it costs to reason are architectural decisions. They get made long before any model is involved, and no amount of model improvement fixes a bad architecture underneath it.

Memory is a lifecycle, not a buffer

Most agent systems today treat context as one shared blob: everything goes in, nothing meaningfully comes out, and the "solution" to running out of room is a bigger window. That is not a memory system, it is a pile.

A lifecycle approach breaks this into stages that each need their own design:

Ingestion: what gets written to memory in the first place, and at what granularity
Scoping: what is relevant to this agent, this user, this task, versus what is just noise that happened to be nearby
Decay: what loses relevance over time and should be forgotten deliberately, not accidentally truncated when the buffer fills up
Retrieval: what gets pulled back into context for a given turn, and why

Treat these as one undifferentiated blob and you get exactly the failure modes everyone complains about: agents that "forget" things that mattered and "remember" things that did not.

Where the cost actually goes

Most of the token spend in agent systems is not the reasoning itself, it is carrying forward context that no longer earns its place. Every stale fact, every resolved sub-task, every turn of small talk that gets re-sent on every subsequent call adds up, and it adds up silently, because nothing in a shared-buffer architecture prompts you to ask whether that context is still worth its cost.

Getting this right requires treating cost as a lifecycle property too, not a line item you optimize after the fact.

We argue a methodology and 5 primitives would be a better approach

I wrote this up formally in a 23-page paper, "Agentic Context Management". It includes a full evaluation harness and the underlying study data, so the argument is not just conceptual, it is something you can check.

This is the same thinking behind the memory work we have been doing at Synap, now written up with data behind it.

Paper: arxiv.org/abs/2607.21503

Genuinely interested in where people think the lifecycle framing breaks down, especially anyone running multi-agent systems where scoping gets a lot harder.

About Author:
I am the founder of Maximem.ai - an agentic context management and agent memory company.

DEV Community: Gaurav Dadhich

Claude API Pricing in 2026: Every Model per Million Tokens, and What Pro and Max Cost

How to Reduce LLM Token Costs in Long Conversations: What Caching Saves, and Where It Stops

What is WebMCP? When and How to Use WebMCP in a Browser Agent?

What is WebMCP? When and How to Use WebMCP in a Browser Agent?

What WebMCP is

When it is worth reaching for

Start with the part that is already stale

The four calls you need

Per step, not per site

The list moves under you

Getting it to run at all

Whose hint is it

Is it worth it

What breaks next

Best Chatbot Memory Layer Plugins for Persistent User Data Retention and Memory

AI Agent Evals: The Judge Is The Part Nobody Measures

Why an unstable judge is worse than no judge

"Set temperature to zero" does not fix this

Nobody models what grading actually costs

What changes when the judge is not a language model

Reliability is not validity, and this is where most of the excitement should stop

The biases survive the change of judge, mostly

Three verdicts, not two

Grading before the response ships

What still has to be right: trajectories, tools and calibration

Where the suite actually runs

The thing a typed judge cannot do

What I would actually build

The part worth sitting with

Agent Memory Is a Real Problem. Most Builders Just Have Not Hit It Yet.

What is Agent Memory?

Is Agent Memory an Imagined Problem or a Real One?

Why most agent-builders have not come across the agent-memory problem yet

For builders who have hit the problem, the alternatives look easy. They are not.

DIY is the first real response, because it looks easy until you run benchmarks

So, What Is Agent Memory? The Jobs To Be Done

Signs You Need Agent Memory

How We Approach Agent Memory at Maximem

Why AI Forgets: Why ChatGPT, Claude, and Gemini Don't Remember You Well

Every conversation is an island

"But ChatGPT, Claude has memory now"

Bigger context windows won't fix this

What you can do about it right now

The forgetting problem isn't going away on its own

Frequently Asked Questions

Why does ChatGPT forget everything between conversations?

Can AI remember things between conversations?

Does Claude have memory like ChatGPT?

How do I make AI remember my preferences?

What is cross-app AI memory?

How do AI agents remember previous conversations across sessions?

Is this different for developers building agents than for people using ChatGPT?

Why does a bigger context window not solve it?

MCP 2026-07-28: 20 Breaking Changes and the Errors They Cause

What changed in MCP 2026-07-28

What breaks, and what does not

Every error you will hit upgrading to MCP 2026-07-28

Unsupported protocol version after upgrading, error code -32022

400 Bad Request with a HeaderMismatch error, code -32020

Your server-initiated requests stopped working

MissingRequiredClientCapability, error code -32021

Resource not found now returns -32602, not -32002

Your tool disappeared from tools/list

400 with no obvious cause, and a missing Accept header

404 Not Found on a method the server does not implement

Your list results are being served to the wrong tenant

Long-running jobs broke when sessions went away

Your traces and your latency numbers went wrong

Authorization started failing on desktop and CLI clients

SDK readiness, as of 29 July 2026

Before and after, in two pictures

What the release fixed, and what it left alone

What the release created

When does HTTP+SSE actually stop working, and why do two official pages disagree?

What to do, by what kind of MCP you run

The twelve-point compatibility checklist

The tool

Frequently asked questions

The line the protocol drew

`Unsupported protocol version` after upgrading, error code `-32022`

`400 Bad Request` with a `HeaderMismatch` error, code `-32020`

`MissingRequiredClientCapability`, error code `-32021`

Resource not found now returns `-32602`, not `-32002`

Your tool disappeared from `tools/list`

`400` with no obvious cause, and a missing `Accept` header

`404 Not Found` on a method the server does not implement