DEV Community: Steef-Jan Wiggers

The Model Was Never the Problem

Steef-Jan Wiggers — Thu, 24 Sep 2026 08:07:57 +0000

The shortlist post ended with a promise: stage two of the eval, selection within the shortlist, measured with a live model, scored so that a recall miss can never masquerade as a selection miss. This post pays that debt. The design came straight from the comment threads: same tool pools, same task pairs, shortlists from the same retriever, and every result row tagged with the description-set hash so the numbers stay attributable across changes. The runner is measure/eval_selection.py in the companion repo, stdlib Python against any Azure OpenAI deployment.

The method

Stage one showed recall failing at 5 percent on paraphrased requests, so scoring stage two on retrieved shortlists would mostly measure retrieval again. Instead, every shortlist here is forced: the correct tool plus its four strongest BM25 distractors, deterministically shuffled. That construction has a pleasant side effect. Because the distractors are whatever retrieval scores highest for the request, they are the correct tool's same-noun siblings. When the expected tool is approve_claim, the shortlist also contains cancel_claim, update_claim, and search_claim. The model does not get to coast on topic matching; it has to read verb semantics.

The model is gpt-5.4-mini, deliberately the small and cheap one, at its default temperature, sixty tasks per condition, half vocabulary phrasing and half paraphrase, across the four description conditions from the recall post. Scored on two things: did the model pick the expected tool, and did the task's reference value (the INV-2005 in the request) land in the arguments of a correct pick. The argument check is deliberately lenient; exact schema validation would punish harmless formatting.

One anecdote from setup that belongs in this series: the eval refused to run with an API key, because the Azure AI resource had disableLocalAuth set. The harness that measures keyless MCP tooling had to authenticate with an Entra bearer token itself. The 401 post's argument, arriving in person.

The numbers

Selection accuracy, forced shortlist of five, sixty tasks per condition:

Descriptions	Selection, vocabulary	Selection, paraphrase
Terse	97%	97%
Realistic	97%	93%
Verbose	90%	93%
Realistic plus aliases	97%	97%

Read that table as flat, and I mean that as a finding, not a shrug. At thirty tasks per cell, every difference in it is one or two tasks, and one run at default temperature is a sample, not a truth. The honest summary is a single sentence: selection accuracy is 90 to 97 percent everywhere. A small model picks the right tool from a five-tool shortlist about nineteen times in twenty, whether the descriptions are terse or verbose, whether the user echoed the tool's vocabulary or paraphrased it into synonyms, and with the tool's own siblings crowding the list. Argument filling lands slightly lower, 87 to 97 percent, under the lenient scoring.

Two null results hide in there, and both are useful. Description verbosity, which did nothing for recall, also does nothing for selection; terse held its own against descriptions three times its token price. And the paraphrase gap, which was catastrophic for recall, nearly vanishes at selection time: once the right tool is in front of it, the model bridges "sign off on the damage report" to approve_claim without difficulty. The vocabulary gap is fatal to retrieval and almost irrelevant to judgment.

Multiply the stages

The two-stage arithmetic is the real headline. End to end, task success is roughly recall times selection. For a paraphrased request against one hundred realistic tools:

Setup	Recall	Selection	End to end
Realistic descriptions	5%	93%	~5%
Realistic plus aliases	100%	97%	~97%

The entire end-to-end difference lives in stage one. Report one blended number, as most tool-calling benchmarks do, and you would conclude the model picks tools badly, when the model almost never saw the right tool at all. Measured separately, the model turns out to be nearly blameless, and the 13-token alias sentence from the recall post is carrying the whole system.

What this changes about the budget

The reflex when an agent picks wrong tools is to reach for a bigger model. At this shortlist size, that money is misspent, because selection was never the failing stage. The failing stage was vocabulary coverage in the descriptions, which is fixed with words that cost thirteen tokens per tool, not with a model that costs ten times more per call. There will be scales where that stops being true; a shortlist of twenty near-duplicates or genuinely ambiguous intents will stress judgment in a way five siblings do not, and that is a measurable follow-up. But the burden of proof has moved. Before spending on model size for tool calling, measure your recall, because the odds are the model never saw the tool you are blaming it for not picking.

The caveats, as always

One model, one run, sixty tasks per condition, differences within the table smaller than its noise floor. The forced shortlist isolates selection but flatters no condition, since all four saw identical lists. And the ceiling on the alias numbers inherits the recall post's caveat: my aliases match my paraphrases by construction. What survives all of that comfortably is the shape: selection high and flat, recall variable and decisive.

The runner, the results with their description hashes, and everything needed to reproduce this against your own deployment are in the repo under measure/; the whole run costs a few cents. Argue with the numbers directly, it is what the comment section of this series is for.

The Shortlist Decides First

Steef-Jan Wiggers — Tue, 22 Sep 2026 06:21:29 +0000

The eval this post runs was designed by the readers. A commenter on What the Agent Pays for Discovery reframed tool selection as a two-stage system: first recall, whether the correct tool makes it into the shortlist that deferred loading or retrieval produces, then selection, whether the model chooses it from that shortlist. He added the design constraint that stuck with me: a cheap context with the right tool absent is the worst outcome, because it fails silently. The model does not error; it improvises with what it has.

That splits the eval into two experiments with very different price tags. Stage two needs a live model, many runs, and scripted tasks. Stage one needs no model at all. Deferred tool loading in practice means retrieving candidate tools by matching the user's request against tool names and descriptions, and retrieval quality is a property of the descriptions. That is measurable deterministically, so this post measures it. The harness lives in the companion repo under measure/, next to the token counter from the discovery post.

The setup

The tool pool is the same deterministic set of one hundred synthetic enterprise tools from the discovery post (search_invoice, approve_claim, and so on), at the same three description levels: terse, realistic, verbose. Retrieval is BM25 over each tool's serialized definition, name, description, and parameter descriptions, with a shortlist of five, which is the curated-manifest size from the break-even discussion.

The tasks are the interesting part. Two hundred of them, one pair per tool, in two kinds. Vocabulary tasks echo the tool's own domain words: "Please approve the claim CLM-2041." Paraphrase tasks say the same thing the way a person who never read your API docs would: "Can you sign off on that damage report, number CLM-2041?" Same intent, same identifier, not one shared content word.

The numbers

Recall at five, one hundred tools, two hundred tasks:

Descriptions	Vocabulary	Paraphrase
Terse	100%	5%
Realistic	97%	5%
Verbose	97%	4%

Two findings, one comfortable and one not. When the user speaks the tool's vocabulary, retrieval is essentially solved; even terse descriptions score perfectly, because the tool name alone carries the match. When the user paraphrases, recall collapses to five percent, and here is the uncomfortable part: verbosity does nothing. The verbose tier costs 208 tokens per tool, roughly double realistic, and its recall is statistically identical. All those extra words are the same words repeated. More prose is not more retrieval surface if it keeps drawing from the same vocabulary.

Which exposes the assumption hiding inside "just defer tool loading and retrieve on demand." Retrieval only sees what the description says. Your users say "bill", "delivery", "sign off". Your descriptions say "invoice", "shipment", "approve". Between those two vocabularies sits a 95 percent silent failure rate, and the failure mode is exactly the one the commenter flagged: the shortlist arrives cheap, plausible, and wrong, and the model does its best with it.

Can a longer shortlist buy it back?

The obvious counter is to widen the shortlist, so I swept it. The shortlist size is also a context budget, priced with the tokenizer from the discovery post: five realistic tools cost about 542 tokens per model call, ten cost 1,040.

Shortlist	Tokens per call	Paraphrase recall (realistic)
3	~362	3%
5	~542	5%
10	~1,040	10%

Doubling the price buys recall from five percent to ten. That is not a fix, that is a linear crawl toward a target the vocabulary gap keeps out of reach; extrapolate it and you are back to shipping all hundred tools, which is the situation deferred loading exists to avoid. Meanwhile the alias condition below sits at 100 percent already at a shortlist of three. You cannot buy your way out of a vocabulary gap with a longer list; you can only close the gap with words.

The thirteen-token fix

The fix follows directly from the diagnosis. If recall is vocabulary coverage, add vocabulary. I appended one sentence to each realistic description, naming its synonyms: "Users may also say 'sign off on' or 'damage report'." Then reran:

Descriptions	Vocabulary	Paraphrase	Tokens per tool
Realistic	97%	5%	~111
Realistic plus aliases	97%	100%	~124
Verbose	97%	4%	~208

The alias line costs 13.4 tokens per tool. The verbose tier costs 97 extra tokens per tool and buys nothing. Thirteen tokens of the right words beat ninety-seven tokens of ceremony, which sharpens the budget rule from the contract post: spend description tokens on disambiguation, on provenance, and now on the user's own vocabulary. An alias sentence is the cheapest line in the whole contract.

One honest caveat, because this series has a rule about that. My alias sentences contain the same synonyms my paraphrase tasks use, so 100 percent is the ceiling case; it demonstrates the mechanism, not a production guarantee. In a real system, the aliases come from somewhere messier and better: the confusion telemetry from the contract post. Every recovery string an agent triggers, and every phrasing that produced a retrieval miss, is a user-vocabulary sample you did not have at design time. The telemetry loop and the alias line are the same feedback cycle, observed and then closed.

A second caveat, aimed at the comment I expect first: the retriever here is BM25, which is lexical, and an embedding retriever handles synonyms better out of the box. It softens the cliff, but it does not repeal the mechanism, because any retriever can only match against the surface the description exposes, and embedding similarity between "approve claim" and "sign off on the damage report" is still weaker than a description that contains both phrasings. The alias line enriches the surface for either retriever. The harness's retriever is one pluggable class, so an embedding comparison is a fair follow-up experiment, and I would genuinely like to see the numbers.

What stage one changes about stage two

Stage two, selection within the shortlist, still needs the live model, and it is coming. But stage one has already changed what stage two must control for. If the correct tool misses the shortlist 95 percent of the time on paraphrased requests, then any end-to-end accuracy number that does not separate the stages is mostly measuring retrieval, not the model's judgment. Report one blended number, and you will conclude the model picks tools badly, when the model never saw the right tool at all. Recall misses get their own headline number, exactly as the comment demanded, or the eval lies to you.

The stage-two design, for the record it will be measured against: the same tool pools and task pairs, shortlists produced by the same retriever, the model asked to choose and call a tool, scored on tool choice and argument correctness, per description variant, with every run tagged by the description hash from the telemetry thread. Selection accuracy conditional on successful recall, so the two failure modes stay separated.

The takeaway

Before the model chooses, the shortlist decides, and the shortlist is only as good as the vocabulary in your descriptions. Verbosity is not coverage, and neither is shortlist size: doubling the description bill moved recall not at all, doubling the shortlist bought five points, and a thirteen-token alias sentence took paraphrase recall from five percent to perfect at the smallest shortlist tested. So treat user vocabulary as part of the tool contract: put aliases in the descriptions, feed them from telemetry, and lint for them in CI next to provenance and when-to-call clauses. The silent failure the commenter warned about is real, cheap to measure, and cheap to fix.

The harness, the task set, and the results are in the repo under measure/; run python eval_recall.py and argue with the numbers directly.

What a 401 Means to an MCP Client

Steef-Jan Wiggers — Fri, 18 Sep 2026 11:06:43 +0000

Twice in this series, a client refused to connect to a working MCP server, and both refusals looked like bugs until they turned out to be the specification doing its job.

The first was in MCP vs API Is the Wrong Question. The Functions-hosted MCP server was protected by its mcp_extension system key, and Claude Code, VS Code, and MCP Inspector all connected happily with the key in an x-functions-key header. Then I tried adding the same server as a connector in the Claude desktop app, and it failed with "Couldn't register with the sign-in service." The second was in the APIM setup in the repo: MCP Inspector, pointed at the gateway without its subscription key attached, reported "Dynamic Client Registration rejected (HTTP 404)."

Different clients, different servers, the same wall. This post explains what that wall is and how to take it down properly with Microsoft Entra ID.

A 401 is not a request for a header

To a developer, a 401 means "add credentials and retry." To a client that implements the MCP authorization specification, a 401 means something more specific: start OAuth. The client fetches the server's Protected Resource Metadata to learn which authorization server protects it, registers itself with that authorization server (dynamic client registration), and walks the user through a sign-in and consent flow. No step in that sequence involves asking a human to paste a key into a header field.

That is why both failures happened. My key-protected servers returned 401 without any Protected Resource Metadata behind them, the clients went looking for an OAuth story that did not exist, and the attempt died at registration with a 404. The clients were not broken. They were refusing to work around a server that spoke half a protocol.

Which splits the MCP client world cleanly in two. Developer tools like Claude Code, VS Code, and MCP Inspector let you attach arbitrary headers, so a shared key works, the way it always has for developers who read documentation. End-user clients like the Claude desktop app deliberately offer no header escape hatch: the spec flow is the only flow. Keys are for tools; OAuth is for people.

Turning on the real thing

The Functions MCP extension supports built-in authentication with Entra ID as the identity provider, and the setup has become almost anticlimactic. In the Azure portal, the function app now has an AI (preview) tab with an Authentication section and a "Turn on MCP authentication" button. Give the Entra app registration a name, save, and it creates the registration, wires up App Service authentication, adds the required settings, and disables key-based access to the MCP endpoint in one step.

The manual route shows what that button actually does, and it is worth seeing once. Key access is turned off by setting the webhook authorization level to anonymous, because the platform's authentication layer now stands in front of it:

az functionapp config appsettings set --name <app> --resource-group <rg> --settings "AzureFunctionsJobHost__extensions__mcp__system__webhookAuthorizationLevel=Anonymous"

Then App Service authentication (the feature long known as Easy Auth) is configured with Microsoft as the identity provider, a new app registration, unauthenticated requests answered with 401, and the token store enabled. Finally, the server advertises its Protected Resource Metadata so that spec-compliant clients can discover the authorization server:

az functionapp config appsettings set --name <app> --resource-group <rg> --settings "WEBSITE_AUTH_PRM_DEFAULT_WITH_SCOPES=<scope from Expose an API>"

That last setting is the difference between a dead end and a doorway. With it in place, the 401 comes with directions.

The second wall: invalid_target

Almost anticlimactic, I said. Then I connected VS Code and hit a second wall the tutorial does not mention. Discovery worked, the client found the metadata and the authorization server, and Entra refused the token request with AADSTS9010010 invalid_target: "The resource parameter provided in the request doesn't match with the requested scopes."

The cause is visible in the metadata itself. A spec-compliant client sends the RFC 8707 resource parameter, taken straight from the Protected Resource Metadata, and Entra now enforces that this resource matches the audience of the requested scope. The button-made setup fails that check twice over. The metadata advertised resource: https://.azurewebsites.net but a scope of api:///user_impersonation, two different names for the same app. And the document clients actually read is the path-specific one at /.well-known/oauth-protected-resource/runtime/webhooks/mcp, whose resource includes the full endpoint path.

The fix is three alignments, so that resource and scope audience are character-for-character identical:

On the app registration, under Expose an API, change the Application ID URI from the default api:// to the endpoint URL: https://.azurewebsites.net/runtime/webhooks/mcp.
Update the advertised scope to match: set WEBSITE_AUTH_PRM_DEFAULT_WITH_SCOPES to https://.azurewebsites.net/runtime/webhooks/mcp/user_impersonation.
In App Service authentication, add the same URL to the allowed token audiences.

Then fetch the path-specific metadata document and check that resource and the prefix of scopes_supported are the same string. When they were, VS Code connected on the next start: 401, discovery, sign-in, token, and "Discovered 3 tools" in the output, with no key anywhere in the configuration.

This is not a quirk of one sample. The same error is currently open against the Azure DevOps MCP server and VS Code, and there is a growing catalog of Entra and MCP authorization mismatches. The pattern is the post's thesis repeating one level down: a strict client, a server speaking half a dialect, and the client refusing to guess.

The client connection, before and after

Before, the client configuration carried a secret:

{
  "servers": {
    "restaurant-directory": {
      "type": "http",
      "url": "https://<app>.azurewebsites.net/runtime/webhooks/mcp",
      "headers": { "x-functions-key": "<the system key>" }
    }
  }
}

After, it carries nothing:

{
  "servers": {
    "restaurant-directory": {
      "type": "http",
      "url": "https://<app>.azurewebsites.net/runtime/webhooks/mcp"
    }
  }
}

On first connect, the client receives the 401 plus metadata, discovers Entra ID, and opens a browser window. You sign in as yourself, consent once, and the client holds a token that refreshes automatically. The credential in the picture is now an identity, revocable per person, subject to Conditional Access, visible in sign-in logs. Everything the system key was not.

The Entra wrinkle: known clients only

There is one honest caveat, and it is where the story gets interesting for enterprise architects. Entra ID does not do open dynamic client registration; it wants to know its clients. The tutorial flow handles this by preauthorizing specific client applications on the app registration: VS Code, for example, is client id aebc6443-996d-45c2-90f0-388ff96faa56, added under "Expose an API" so users are not prompted for admin consent.

For clients outside the preauthorized list, you register a client application yourself and hand its id to the client. The Claude desktop connector anticipates exactly this: the error message from the first post already offered a field to "add an OAuth Client ID in the connector settings." What read as a workaround then is the designed path now: create the client registration in your tenant, preauthorize it against the server's app registration, give the connector the client id, and the sign-in flow completes inside your own identity perimeter.

Seen from a governance angle, this is a feature wearing a limitation's clothes. Anonymous dynamic registration means any client anywhere can start a flow against your server. A preauthorized client list means your identity team decides which MCP clients exist in your estate. For a regulated organization, that is not friction; that is the control you would have had to build anyway.

What the token does and does not do

The token asserts who the caller is, which tenant they belong to, and that your authorization rules let them reach the endpoint. It does not decide what they may do once inside. Tool-level authorization, the difference between a user who may search restaurants and one who may place orders, remains your code's job, and the claims in the token are the input to it. Authentication moved to the platform; authorization stayed in the kitchen, exactly where the first post's caveats said it lives.

Verdict

Keys for demos and developer tooling, where a header field is available and rotation is cheap. Entra ID with built-in MCP authentication for anything a human signs into, and for any server that should exist inside an organization's identity perimeter rather than beside it. The clients already enforce this split; the only choice you get is whether to be surprised by it.

The sample repo has both doors and the APIM patterns this builds on; the Functions MCP tutorial has the full authentication walkthrough.

Tool Descriptions Are the Contract

Steef-Jan Wiggers — Thu, 17 Sep 2026 07:37:43 +0000

This series started with a claim: MCP and REST are two doors into the same kitchen. Three posts later, the comments have pushed the argument down to its foundation. One reader on dev.to said the useful test is whether both doors preserve the same auth, error, and idempotency behavior. Another warned about granularity drift and asked how anyone validates tool descriptions. Both are pointing at the same uncomfortable fact: in an MCP server, prose is load-bearing. The description decides whether the agent picks your tool. The parameter text decides whether it calls the tool correctly. The error message decides whether it recovers. That prose is the API contract now, and almost nobody treats it like one.

The contract already lives in three places

Here is what I did not expect to find in my own sample repo. The restaurant backend describes its behavior in prose three times. The Functions MCP door carries descriptions in C# attributes:

[McpToolTrigger("search_restaurants",
    "Searches the restaurant directory. Both filters are optional; call it without arguments to list every restaurant.")]

The APIM REST door carries operation descriptions in Bicep. And the third pattern from the APIM post, where the gateway manufactures an MCP server from the REST API, carries its own tool descriptions in Bicep too:

description: 'Searches the restaurant directory. Both cuisine and city filters are optional; call without arguments to list every restaurant.'

Read those two closely. They are not the same sentence. I wrote them two days apart, and they drifted: "Both filters" became "Both cuisine and city filters", "call it without" became "call without". Harmless here. But this is a three-tool sample maintained by one person for one week, and the copies already disagree. Scale that to fifty tools, three teams, and a year of changes, and the gateway's MCP door describes a backend that no longer behaves the way its C# door says it does. That is the granularity drift the commenter meant, in miniature: the contract forks the moment it exists in two files, and nothing in the toolchain notices.

The REST door gets away with prose drift because its real contract is elsewhere: routes, status codes, an OpenAPI document that tooling can diff. The MCP doors have no elsewhere. The description is the interface.

What a good description buys, at a known price

The measurement post put numbers on the cost side: a realistic description runs about 111 tokens per tool, and a verbose one 208, paid on every model call. So the budget question is what each sentence buys. The sample's descriptions follow three rules, and each rule earns its tokens differently.

Say what the tool does, in one sentence, in terms that distinguish it from its neighbors. Say when to call it, which is mostly about optionality: "both filters are optional; call it without arguments to list every restaurant" prevents the agent from inventing a filter just to have one. And say where values come from: "the restaurant id, as returned by search_restaurants (for example 'r1')". That last rule is the one a JSON schema cannot express. A schema says restaurantId is a required string. Only prose says the string must come from a previous search_restaurants call and must not be guessed. Provenance is the highest-value sentence in the whole contract, and it is the difference between an agent that chains tools and one that hallucinates identifiers.

The verbose tier from the measurement post doubles the bill mostly with ceremony the model would infer anyway. Spend on disambiguation and provenance, cut the rest.

Door parity is about decisions, not formats

The commenter's test, applied to the sample: auth and idempotency live in the kitchen, in the shared service that both doors call, so neither door can drift on what is allowed or what happens on a retry. Error rendering is the one thing the doors deliberately do differently. The REST door returns machine-shaped errors, a 4xx status with a small JSON body, because its caller is code and code branches on status. The MCP door returns instructions:

return menu is null
    ? $"Restaurant '{restaurantId}' was not found. Use search_restaurants first to get a valid id."
    : JsonSerializer.Serialize(menu, JsonOptions);

Same fact, two renderings. That passes the parity test rather than failing it: parity means the doors agree on the decision, not that they speak the same dialect. The kitchen decides the order is invalid; each door reports it in the shape its caller can act on.

Your error strings are confusion telemetry

Those recovery sentences do a second job that I have not seen anyone talk about. They are constants, so every time one is returned, you know exactly which misunderstanding just happened. "Use search_restaurants first to get a valid id" fires when an agent invented a restaurant id, which means the provenance sentence in get_menu's description did not land. Add one log line where each recovery string is returned and Application Insights turns into a description quality dashboard: a KQL query grouping by sentinel string, per tool, per client. A rising count on one string is not an outage. It is a failing sentence in your contract, located precisely. I know of no cheaper contract test than instrumenting the errors you already wrote.

Validating the contract

Which is the honest answer to the commenter's validation question: start by treating descriptions as artifacts that CI can check without any LLM. Lint that every identifier parameter states its provenance. Lint that every tool description contains a when-to-call clause. And budget-gate the whole array: the token harness from the measurement post lives in the repo under measure/, and pointing it at your tools array in CI turns "our context bill crept up" into a failing build with a number in it. Drift between copies is checkable the same way; if the gateway's Bicep descriptions and the code's attribute descriptions are both machine-readable, a test can diff them.

What CI cannot check is whether the model picks the right tool, and I will not pretend otherwise. That needs the deferred experiment: scripted tasks with known correct tool sequences, run repeatedly against description variants. A commenter on the measurement post sharpened how to score it: as a two-stage system. Stage one is recall, whether the correct tool made it into the shortlist that deferred loading or retrieval produced. Stage two is selection, whether the model chose it from that shortlist and called it correctly. The stages fail differently: a recall miss is a retrieval or granularity problem, a selection miss is a description problem. And recall misses need their own headline number, because they fail silently; a cheap context with the right tool absent is the worst outcome, and the model will improvise something plausible instead of erroring. The measurement harness was structured so that this eval slots in on top, and it is the next piece of this series with numbers in it.

Verdict

The description is the interface. So give it the treatment interfaces get: one authoritative copy where you can manage it, review on every change, a token budget enforced in CI, provenance sentences on every identifier, and telemetry on the error strings that reveal where the contract fails. The doors were never the hard part. The words were.

The sample, the APIM patterns, and the token harness are all in the repo.

What the Agent Pays for Discovery

Steef-Jan Wiggers — Wed, 16 Sep 2026 11:15:02 +0000

A reader of MCP vs API Is the Wrong Question asked the question I could not answer at the time: did you measure the token cost of the live MCP discovery round trip against handing the model a curated tool manifest up front, and where does the break-even sit? I answered from first principles then. This post answers with numbers, from a small measurement harness that now lives in the companion repo under measure/.

Where the tokens actually go

First, the framing correction that makes the question answerable. The live discovery round trip is nearly free: an MCP client calls tools/list once per session, not per turn, and the payload that comes back is essentially the same manifest you would curate by hand. The cost that matters is different and larger: whatever tools the client ends up with, their definitions are injected into the model's context on every single call. A curated manifest pays that too. So "discovery versus manifest" is not the axis. The axis is how many tool definitions sit in context, how wordy they are, and how many model calls the task takes.

That is measurable without an LLM in the loop, exactly, with a tokenizer.

The method

The harness serializes a tools array the way it reaches a model (JSON: name, description, input schema with per-parameter descriptions) and counts cl100k tokens. The anchor is real: the three tools of the restaurant sample, exactly as the Functions MCP extension serves them. Around that anchor it generates synthetic enterprise tools (search_invoice, approve_claim, and so on, two to four parameters each) at three description levels: terse (a sentence fragment), realistic (purpose plus parameter guidance, modeled on the sample's actual descriptions), and verbose (usage guidance, examples, and edge-case notes on everything). One honest caveat: harnesses reformat definitions slightly differently, so treat the absolute numbers as close and the relative differences as solid.

The numbers

The real three-tool restaurant server costs 277 tokens per model call. That is the whole standing bill, and it is why break-even at this scale is immediate: there is nothing meaningful to save.

Scale changes the picture:

Tools	Terse	Realistic	Verbose
3	209	362	668
10	596	1,040	1,972
30	1,856	3,233	6,089
100	6,370	11,132	20,814

Two things jump out. The growth is linear in tool count, roughly 64 tokens per tool terse, 111 realistic, 208 verbose; there is no cliff, just a slope. And description style is its own multiplier: at every size, verbose costs about 1.9 times realistic and 3.3 times terse.

The slope compounds per task, not per session. A hundred realistic tools cost 11,132 tokens on every model call, so a 20-call agent task pays roughly 223,000 tokens for definitions alone, before a word of conversation or a byte of tool output. The curated comparison the commenter asked about: five relevant tools picked out of that hundred cost 542 tokens per call, a 95 percent saving.

So where is the break-even?

For a server like the sample, there is none to find; 277 tokens is noise. My reading of the curve: below roughly ten tools, do nothing. Around thirty realistic tools you are paying 3,200 tokens per call and about 65,000 per task, which is real money at fleet scale but rarely worth architectural surgery. At a hundred tools you must do something, and the interesting part is that a curated manifest is only one of four options, and the crudest one.

A curated manifest is static tool selection: you save 95 percent of the context by deciding at design time which five tools matter, which is precisely the design-time knowledge MCP exists to avoid assuming. The alternatives keep discovery and cut the bill differently. Split the server by domain, so an agent connects to the invoice server or the shipment server rather than the everything server; this is the maître d' post's fleet argument wearing a cost hat. Use a client that defers tool loading and searches definitions on demand, which more agent runtimes now do. Or design coarser, intent-sized tools so a hundred resource-level operations become fifteen task-level ones, which a commenter on the first post predicted and the numbers now justify.

The tension with post one

The first post argued that tool descriptions are load-bearing: vague text makes vague agents. This post seems to argue that words cost money. Both are true, and the numbers say where the balance sits. Realistic descriptions, around 25 words of purpose and parameter guidance per tool, cost 111 tokens each; that is the price of an agent that picks the right tool and recovers from errors, and it is worth paying. The verbose tier doubles the bill mostly with prose the model would infer anyway. So the budget rule I take away: spend description tokens on disambiguation between tools and on where identifiers come from, never on ceremony.

What this does not measure

Selection accuracy. The other half of the commenter's question is whether the model picks the right tool less often at a hundred tools than at ten, and that needs a live model, many runs, and scripted tasks with known correct tool sequences. The harness is structured so that eval slots in next; that experiment, together with the description validation ideas from the comment thread, is its own post. What is measured here is the bill; whether the agent's judgment also degrades with the menu size is the follow-up.

The takeaway

Discovery is not what costs you; standing context is. The bill is linear in tool count, multiplied by description style, and paid on every model call. Below ten tools, ignore it. Above that, the fix is not abandoning discovery for a hand-curated list; it is scoping servers, deferring definitions, and sizing tools to intents, with descriptions that earn their tokens.

The harness, the data, and the chart are in the repo under measure/; run python token_cost.py and argue with the numbers directly.

Same Kitchen, Two Doors, One Maître d'

Steef-Jan Wiggers — Sun, 13 Sep 2026 14:10:21 +0000

In MCP vs API Is the Wrong Question I deployed one Azure Functions app that exposes the same backend twice: a REST door for callers who know the contract, and an MCP door for agents that discover it. The caveats section admitted, out loud, what the demo left open: the REST endpoints were anonymous, and the MCP endpoint was guarded by a system key that every client had to carry.

A reader summarized the gap better than I did: the useful test is whether both doors preserve the same auth, error, and idempotency behavior, because otherwise the integration surface splits even when the backend does not. Auth parity is exactly what the demo lacked. This post closes that gap with Azure API Management, and it turns out APIM can play three different roles in front of this architecture. Sticking with the restaurant analogy: the kitchen cooks, the doors let people in, and APIM is the maître d'. It checks reservations, limits party size, and keeps the guest book, without touching a single pan.

Why the doors need a maître d'

Both doors in the sample share one property: they authenticate the caller mechanically, not organizationally. A function key or an mcp_extension system key says "this caller has the string" and nothing else. There is no identity, no revocation short of rotating the key for everyone, no per-caller limits, and no answer to the question every platform team eventually asks: who called what, how often, and should they be allowed to?

That question is gateway work. And the more actions we let agents perform, the more urgent it becomes, because an agent retries, chains calls, and occasionally misreads a tool description. You want the blast radius of that behavior governed somewhere the agent cannot negotiate with.

Pattern one: APIM in front of the REST door

The familiar pattern, and I will keep it short because it is the subject of my API Management for AI Workloads series. APIM fronts the /api endpoints: validate-jwt against Entra ID or subscription keys for consumers, rate limits and quotas per product, IP filtering where it applies, and the function app locked down so only the gateway reaches it. The REST door's contract does not change; it just stops being naked.

Pattern two: APIM in front of the MCP door

APIM can also expose an existing MCP server as a passthrough, and a Functions-hosted MCP endpoint is an explicitly supported backend. The MCP traffic (streamable HTTP) flows through the gateway, and policies apply to it like any other API.

This changes the security posture in one important way: the agent no longer holds the backend credential. The client authenticates to APIM properly, with OAuth against Entra ID or a subscription key, and a policy injects the x-functions-key toward the backend. Rotating the system key becomes an operation nobody outside the platform team notices. On top of that you get the gateway's usual gifts, now applied to tool calls: throttling per agent identity, request tracing that shows which tools are called at which rate, and since Build 2026 even MCP-aware content safety policies.

In the first post I noted that developer tools happily send a key in a header while end-user MCP clients follow the authorization spec and demand a real OAuth flow. Pattern two is one of the two clean answers to that boundary; the Functions extension's built-in Entra auth is the other. With a gateway in front, the OAuth conversation happens at the maître d', where it belongs.

Pattern three: APIM as the MCP door

The third option surprised me when I first tried it: APIM can expose a REST API as an MCP server by itself. You pick operations from an API already managed in the gateway, and APIM manufactures MCP tools from them. In that design the function app implements only the REST door, the MCP extension disappears from the codebase, and the second door is generated at the gateway.

This is genuinely attractive for one class of backend: the one you cannot change. An enterprise has hundreds of APIs already registered in APIM, owned by teams with no bandwidth for an agent project. Pattern three gives every one of them an MCP door without a single code change, and the same gateway policies govern it from day one.

But the first post argued that tool descriptions are load-bearing: the agent chooses tools based on that text, and vague descriptions produce vague agents. Pattern three moves those descriptions out of the code and into gateway configuration. The team that owns the business logic no longer owns the words the agent reads. In pattern two, the descriptions live in McpToolTrigger attributes, versioned in the same pull request as the logic they describe. That difference sounds small and is not; it decides who fixes the contract when the agent starts picking the wrong tool.

Where each pattern is the wrong answer

Pattern one is never wrong so much as incomplete; it governs only the door that agents do not use.

Pattern two is the wrong answer when there is no code to own. If the backend team cannot or will not adopt the MCP extension, insisting on in-code tools blocks the whole effort. It also costs you nothing to skip if no agent ever calls the system; an MCP door with no visitors is just surface area.

Pattern three has sharper edges. APIM's MCP support is tools-only today: no MCP resources, no prompts. The Functions extension gained resource and prompt triggers at Build 2026, so if your server needs more than tools, the gateway cannot generate it. Auto-generated tools also inherit the granularity of the REST operations, and a fifty-operation API becomes a fifty-tool server, which is more choices than an agent should have to reason about. And every tool call pays the extra hop, which is fine for governance but worth knowing.

The honest default for new systems: pattern two, with descriptions in code and the gateway doing identity, limits, and observability. Pattern three for the estate you already have. Pattern one always, underneath both.

What deploying it taught me

Three things the documentation did not. The mcpProperties.endpoints contract wants an object keyed by endpoint name, while both the published examples and the Bicep type definitions say array; the deployment error is the only place the truth lives, which is what a preview API version means in practice. The client-facing URL is the API path plus an /mcp suffix, a small trap everyone will hit exactly once. And when a spec-compliant MCP client receives a 401 from the gateway, it does not ask for a header; it starts OAuth dynamic client registration, which APIM answers with a 404. Two different clients have now walked into that same wall from different directions in this series, and it is the strongest argument yet for doing the OAuth setup properly.

The question that remains

The first post ended by replacing "MCP or API?" with a better question: where do I want discovery, and where do I want a hard contract? APIM adds the operational half: wherever you put discovery, put a gateway in front of it that knows who is discovering. The kitchen stays the kitchen. The doors stay honest. The maître d' remembers every guest.

The companion repo will gain an infra/apim variant with the Bicep and policies for patterns two and three. Until then, the original sample is the substrate all of this builds on.

MCP vs API Is the Wrong Question

Steef-Jan Wiggers — Fri, 11 Sep 2026 16:25:21 +0000

Recently, I saw a LinkedIn post discussing "MCP vs API" and felt like we needed to choose one. But I don't think we do, as each option solves a different problem, and the fastest way to see that is to run the same backend behind both. So I built one Azure Functions app that exposes identical business logic twice: once as a classic REST API and once as a remote MCP server. The code is in the companion repo, and this post walks through it.

Two different problems

The simplest way I think about it is this. An API tells software how to talk to another piece of software. MCP, the Model Context Protocol, helps an AI system understand what tools and resources are available, and how to use them.

Consider a normal application that needs a file from Google Drive. The developer already knows what is needed: the app, the Google Drive API, and the file. They read the API documentation, call the right endpoint, and process the response. The knowledge lives in the developer and gets compiled into the application.

Now ask an AI assistant: "Find our latest sales presentation, compare the numbers with our customer database, check whether the related GitHub project has changed, and summarise everything." The assistant may need to work across Google Drive, a database, GitHub, and internal documents. Nobody hardcoded that sequence. The AI has to discover what tools exist, what each one does, which parameters are required, and which tool to use next. That discovery problem is what MCP solves.

A good analogy: an API is calling a restaurant directly. You already know the restaurant, its number, and what you want. MCP is giving your assistant a standardized directory of restaurants, menus, and available actions. The assistant discovers what is available and chooses the right capability. And underneath, the restaurant still uses the same kitchen. That kitchen is often the API.

Same kitchen, two doors

I took that analogy literally. The sample is a small restaurant directory with exactly one implementation of its business logic, an IRestaurantDirectory service that can search restaurants, return menus, and place orders. Two doors sit in front of it.

Door one is the REST API, three HTTP-triggered functions:

[Function(nameof(GetRestaurants))]
public IActionResult GetRestaurants(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "restaurants")] HttpRequest req)
{
    string? cuisine = req.Query["cuisine"];
    string? city = req.Query["city"];
    return new OkObjectResult(directory.Search(cuisine, city));
}

Fixed endpoints, documented parameters, request, and response. If you know the contract, this is the most direct path.

Door two is the MCP server, built with the Azure Functions MCP extension. I compared the hosting options for remote MCP servers on Functions in an earlier post; this sample uses the binding extension, which has since reached a stable 1.x release with typed tool property attributes. The same operations become tools, and the trigger attributes carry something the REST door never needed: descriptions written for a machine that has to figure out what to call.

[Function(nameof(SearchRestaurantsTool))]
public string SearchRestaurantsTool(
    [McpToolTrigger("search_restaurants",
        "Searches the restaurant directory. Both filters are optional; call it without arguments to list every restaurant.")]
        ToolInvocationContext context,
    [McpToolProperty("cuisine", "Cuisine to filter by, for example 'Italian' or 'Japanese'.", isRequired: false)]
        string? cuisine,
    [McpToolProperty("city", "City to filter by, for example 'Nijmegen' or 'Utrecht'.", isRequired: false)]
        string? city)
    => ...

Neither door contains business logic. Both call the same service. The
Functions runtime hosts the MCP endpoint at /runtime/webhooks/mcp, protected in Azure by a system key, and the whole thing deploys to a Flex Consumption plan with azd up.

Here is the deployed architecture, exactly as the Bicep in the repo provisions it:

One deployment lesson worth passing on: function names must be unique across both doors. My first deployment had a PlaceOrder HTTP trigger and a PlaceOrder MCP tool trigger, and the host refused to start with a cryptic "Sequence contains more than one matching element". The MCP-facing tool names stay whatever you declare in the attribute; the .NET function names behind them need a suffix.

What changes at door two

Run the app locally, point an MCP client at it (the repo ships a .vscode/mcp.json for VS Code, and Claude works too), and ask: "Find me an Italian restaurant in Nijmegen, show me the menu, and order two Margheritas."

The agent lists the available tools, reads their descriptions, and chains search_restaurants, get_menu, and place_order on its own. It passes the restaurant ID from the first call into the second, and the exact item name from the second into the third. Nobody wrote that orchestration. Against the REST door, that same flow is three documented calls a developer wires together at design time.

That is the whole difference in one demo. The REST door serves callers who know. The MCP door serves callers who discover. Instead of teaching an AI system separately how to interact with twenty different tools, MCP gives those tools a consistent way to expose capabilities, which makes AI systems easier to extend, orchestrate, and maintain.

One consequence surprised me in a useful way: tool descriptions become load-bearing. The agent chooses tools based on the text in those attributes. Vague descriptions produce vague agents. Treat tool descriptions like API contracts, because for an agent, they are.

Where MCP is the wrong answer

MCP is not the right door for everything, and reaching for it by default is how we get the next round of architecture astronautics.

Skip MCP when the caller is deterministic software. A backend service that needs a file from Drive should call the Drive API. Adding an MCP layer between two pieces of conventional software adds latency and a dependency, and discovers nothing, because no model is doing the discovering.

Skip it when there is exactly one integration, and leave it that way. The discovery machinery pays off across many tools; for a single well-known endpoint it is overhead.

And be honest about what MCP does not do. It doesn't make an agent intelligent, and it doesn't do authorization for you. We still need everything we already need for APIs, authentication, permissions, observability, and governance. In fact, the more actions we allow AI to perform, the more important these become.

In the sample, the demo endpoints are deliberately anonymous, and the README's caveats section says so out loud; in production, the REST door belongs behind API Management and the MCP door behind Entra ID, which the Functions extension now supports as built-in MCP auth. I ran into that boundary while testing: developer tools like Claude Code, VS Code, and MCP Inspector happily send a system key in a header, but end-user clients that follow the MCP authorization spec expect a real OAuth flow and refuse anything less.

The real opportunity

So when someone asks me which is better, MCP or API, my answer is: wrong comparison. The architecture that keeps showing up in practice is AI assistant, MCP, existing API, business system. APIs connect software. MCP helps AI understand how to interact with that software. And AI agents turn those connections into actions and workflows.

The sample repo has the full code, local run instructions, and Bicep to deploy both doors to Azure Functions. Clone it, open both doors, and the debate settles itself.

Azure Functions Hosted Skills: I Built a Scam-Checking Agent, Then Removed the Agency

Steef-Jan Wiggers — Fri, 11 Sep 2026 10:21:05 +0000

Is that webshop legit? Most of us have pasted a URL into a reputation checker at least once. The signals those sites use are not magic: domain age, HTTPS, policy pages, archive history, reviews. That looked like a perfect weekend project for Azure Functions hosted skills, the feature Microsoft previewed at Build as the serverless agents runtime and has since renamed, and the same one I used for my weather agents sample. Define a hosted skill with a set of verification tools, let it gather evidence, and let the model form an opinion.

I built exactly that. It fell over in an instructive way. The fix was not a bugfix. It was an architecture decision: take the orchestration away from the model. This post walks through both versions, the failure in between, and why "evidence in code, judgment in the model" is the pattern I will reach for first from now on.

The full sample is on GitHub: legit-check-agent. Deploy it with azd up.

Designing the signals for a hosted skill

I made one decision early that shaped everything: no scraping and no per-provider review APIs. The Trustpilot content API needs a paid business account. Scraping review sites violates their terms. Instead, the hosted skill uses two tiers of signals.

The first tier is keyless and verifiable. RDAP gives the domain registration date, registrar, and age. A direct HTTPS fetch confirms the site is reachable and checks whether it links the pages a real shop has: contact, about, terms, privacy, returns. The Internet Archive CDX API shows when the site first appeared and whether it has a continuous history. A shop that claims ten years of trading but has a three-month-old domain and no archive footprint tells you something no review score can.

The second tier is reputation through web search grounding. A Tavily search runs targeted queries: site:trustpilot.com , the business name plus "reviews", and the business name plus "scam OR fraud OR oplichting" for Dutch shops. The model sees result titles, URLs, and snippets. It never invents a rating. If no key is configured, the tier reports itself as unavailable and the agent says so in its verdict. Graceful degradation was a design goal, and it mattered more than I expected.

Here is the RDAP helper. Nothing clever, which is the point:

def _domain_info(domain: str) -> dict[str, Any]:
    response = httpx.get(f"https://rdap.org/domain/{domain}",
                         headers={"User-Agent": USER_AGENT},
                         follow_redirects=True, timeout=15.0)
    if response.status_code == 404:
        return {"registered": False, "note": "No RDAP record found."}
    response.raise_for_status()
    data = response.json()
    events = {e.get("eventAction"): e.get("eventDate") for e in data.get("events", [])}
    ...
    return {"registered": True, "registration_date": registered_at,
            "age_days": age_days, "registrar": registrar}

Version 1: an agentic loop in Azure Functions hosted skills

The first version followed the textbook. Four @tool functions: check_domain, check_website, check_archive_history, web_search. The instructions in the .agent.md file told the model to run them in order and issue several search queries. The model orchestrated freely, the way agent demos do.

It worked in the sense that it compiled and deployed. Then the errors started.

First: "Model deployment rate limit exceeded" on gpt-4.1. I switched models, since gpt-4.1 is marked legacy in the catalog anyway, and moved to gpt-5.6-terra. Same error. I bumped the deployment capacity to 500K TPM. Then a new error: "Your input exceeds the context window of this model." A single question about a single webshop exceeded the context window of a frontier model.

Application Insights had the smoking gun. Azure Functions hosted skills log an agent_token_usage event per turn, and one line told the whole story:

"input_tokens": 47004, "output_tokens": 280

Forty-seven thousand tokens in, to produce a 280-token answer.

Here is the mechanism, and it is worth internalizing. Every tool call in an agentic loop is a full model round trip that carries the entire conversation: system prompt, tool schemas, chat history, and every previous tool result. The built-in chat UI stores the thread server-side, so failed turns and retries stay in it forever. And when a tool fails, the model retries. My web_search returned an error when no Tavily key was set, so the model helpfully tried again with a different query. Each retry compounded the history. The loop turned a missing API key into a quota problem, and the quota problem into a context overflow. Not one line of my tool code was wrong.

Version 2: evidence in code, judgment in the model

The redesign fits in one sentence. One deterministic tool gathers every signal in plain Python; the model calls it exactly once and writes the verdict. Two model turns per check, and the evidence pack is hard-capped at 8 KB.

@tool
def gather_business_check(args: dict[str, Any]) -> dict[str, Any]:
    business = str(args["business"])
    domain = _domain_of(business)
    evidence = {
        "domain": domain,
        "domain_registration": _safe(_domain_info, domain),
        "website": _safe(_website_info, domain),
        "archive_history": _safe(_archive_info, domain),
        "reputation_search": _safe(_review_search, domain, business),
    }
    text = json.dumps(evidence, default=str)
    if len(text) > 8000:
        evidence = {"truncated": True, "data": text[:8000]}
    return evidence

The _safe wrapper is the other half of the fix. A failed signal becomes a short error string inside the evidence, never an exception and never a retry:

def _safe(fn, *args):
    try:
        return fn(*args)
    except Exception as exc:
        return {"error": f"{type(exc).__name__}: {str(exc)[:200]}"}

The .agent.md instructions shrank to match: call the tool once, never retry it, treat missing signals as reported facts, then write a verdict with a 0 to 100 score and cite the sources. The model does the one thing it is uniquely good at here, which is weighing mixed evidence and writing a calibrated judgment. Everything else is boring code, and boring code does not blow up context windows.

The validation moment

I tested with thenewsound.nl, a Dutch audio webshop. Without a search key the agent said "Mixed signals, be careful" at 68/100 on technical evidence alone. With Tavily grounding enabled it firmed up to "Looks legitimate" at 78/100: it found a 4.6/5 Trustpilot rating from 31 reviews, and it explicitly discounted the shop's own claim of 4.8 from 890 reviews as self-published rather than independent. It also hit a Wayback timeout on this run and simply reported the archive signal as unknown, exactly as designed. A commercial reputation checker scored the same site 76/100, "Trusted but Verify". A weekend-size sample landed within two points of a commercial product, because the signals do the work and the LLM only weighs and explains them.

Where this is the wrong answer

Be careful before generalizing this into "agent loops are bad". When the task genuinely needs dynamic tool selection, open-ended research for example, the loop earns its cost. My task was a fixed checklist, and for a fixed checklist the loop was pure overhead with failure modes attached.

The verdicts themselves also deserve a caveat, and the agent states it in every answer: this is a signal-based assessment, not a guarantee. Sophisticated scams fake signals. Legitimate young shops look thin. The agent recommends buyer-protected payment methods whenever it is unsure, and its instructions explicitly refuse the reverse use case of making a site look more legitimate than it is.

Takeaways

Five things I would tell my past self from two days ago. Give the model judgment, not orchestration, when the workflow is a fixed checklist. Bound every tool result, because the model will see it on every subsequent turn, not once. Watch agent_token_usage in Application Insights from day one; input tokens per turn is the health metric of any agent. Model quota is per subscription, region, and model, and legacy models sit in crowded pools. And make every failing tool say "do not retry me", because otherwise the loop will.

Azure Functions hosted skills made all of this pleasantly small. The whole sample is a main.agent.md, one tools file, and trimmed Bicep. The lesson was never about the platform. It was about deciding where the agency belongs.

Note: Microsoft previewed this feature at Build 2026 as the "serverless agents runtime" and renamed it to Azure Functions hosted skills in September 2026. This post uses the current name; my earlier weather agents post predates the rename.

The Microsoft AI Stack in 2026 and the Certification Trail That Runs Through It

Steef-Jan Wiggers — Thu, 03 Sep 2026 13:35:30 +0000

I spent some time observing what's inside the Microsoft AI stack. Foundry, Agent Framework, Logic Apps, AI Search, Purview, Entra. After a while, I had a picture of how the pieces fit and decided to draw one myself to share.

Then recently Microsoft published AI-500, an expert certification for multi-agent systems. That made me curious whether Microsoft's view of the platform matches my own. So I plotted the certification trail against my diagram. This post shows the result.

The Microsoft AI stack as I see it

Six layers. Five of them stack vertically. The sixth runs down the side, through all the others.

Models sit at the bottom: GPT-5, Claude, Mistral, Grok, Microsoft’s own MAI and Phi, Llama, DeepSeek, and the open catalog. This is the least differentiated layer. In Foundry, swapping one model for another is a configuration change. It gets the most attention and deserves the least.

Infrastructure comes next. Foundry is the hub, alongside Azure OpenAI, Azure ML, AKS, Container Apps, App Service, and Foundry Local for the edge. This is hosting, serving, and compute. Solid and well understood.

Data and context: the real moat

I split the data layer in two, because it hides the most important part of the platform. Layer 3a holds the sources of truth: Microsoft Graph, SharePoint, Exchange, Fabric and OneLake, Dataverse, and the vector stores in Cosmos DB, Azure SQL, and PostgreSQL.

Layer 3b is context. Work IQ, Fabric IQ, Foundry IQ, and Azure AI Search turn enterprise data into grounding that respects who is asking. Permission-trimmed retrieval must honor Entra ACLs at query time, not filter after ranking. AI Search supports this through document-level access control. Get it wrong and answers leak, or recall collapses.

This is where the hard engineering hours go. Swapping a model is a config change. Graph-grounded context is months of work.

Agents, Copilots, and the layer vendors leave out

The agentic platform sits above the data. Microsoft Agent Framework merged Semantic Kernel and AutoGen. Next to it sit Foundry Agent Service, Copilot Studio, Logic Apps, Azure Functions, Service Bus, and Event Grid. MCP and A2A handle tools and agent-to-agent communication.

The Copilot layer is on top: Microsoft 365, GitHub, Security, Dynamics 365, Power Platform, and Teams. This is distribution. These are the surfaces people already live in. As a result, enterprise AI adoption is easier when data, permissions, infrastructure, and applications already exist in one ecosystem.

Then comes the sixth layer of the Microsoft AI stack, the one vendor diagrams leave out: governance and evidence. Entra ID and Entra Agent ID handle identity. Purview covers labels, DLP, and audit. Content Safety and API Management provide guardrails and the AI gateway. Defender, Sentinel, Azure Monitor, Log Analytics, and Azure Policy deliver traces, evaluations, and control evidence.

I work for a regulated organization. Before anything goes live, Legal and Internal Audit ask four questions. Who approved the agent? What data did it access? Which controls applied? What happened when it made a wrong decision? Distribution makes deployment easier. However, without traceable evidence it does not make the system production ready. That is why this layer runs vertically through my drawing.

The Microsoft AI certification trail

I knew AI-900. I had not followed what replaced it. This week I learned that Microsoft now offers a full set of AI certifications, from fundamentals to an expert exam. The expert exam, AI-500, is in beta. I read the study guide. Its scope says a lot: orchestration patterns, agent-to-agent protocols, observability, guardrails, and cost control. Architecture work, end to end.

What made it click was laying the whole trail side by side. Each step has its own verb.

AI-901, Azure AI Fundamentals. Understand the concepts and services.
AI-103, AI Apps and Agents Developer. Build applications and agentic solutions on Foundry.
AI-200, Azure AI Cloud Developer. Engineer cloud-native AI properly.
AI-300, ML Operations Engineer. Operate models in production.
GH-300 and GH-600, Copilot and Agentic AI Developer. Ship with agents working beside you.
AI-500, Multi-Agent AI Solutions Expert. Orchestrate systems of agents at scale.

Plotting the trail on the Microsoft AI stack

Six exams, six layers of the Microsoft AI stack. I mapped each exam to its layers, in its study guide exercises, and its center of gravity.

Microsoft AI certification trail from AI-901 to AI-500 plotted against the six layers of the Microsoft AI stack
The fundamentals exam touches every layer at concept depth. AI-103 lives in models, infrastructure, context, and the agentic platform. AI-200 moves down into infrastructure and data. AI-300 sits on infrastructure and the evidence plane, because operating models in production is mostly monitoring and evaluation. Meanwhile, the GitHub exams live at the top, where developers meet agents in the editor.

AI-500 is the interesting one. It spans context, the agentic platform, and governance. It does not test models at all. In fact, three of its five headline topics belong to the plane most stack diagrams leave out.

That was the moment the two pictures agreed. My diagram says the hard part of the Microsoft AI stack is context and evidence, not model choice. Microsoft’s expert exam tests context and evidence, not model choice. I did not expect a certification roadmap to confirm an architecture opinion, but here we are.

Where this is the wrong answer

Do not read the trail as a ladder you must climb in order. If you already run agents in production, AI-500 reflects your work, and AI-901 will teach you nothing. Platform engineers should look at AI-200 and AI-300 first. Developers should start with the GitHub exams.

Also, do not read my layer mapping as Microsoft’s. It is my reading of the study guides, and beta study guides move.

Finally, do not confuse the certificate with the evidence. Passing AI-500 shows you know what a control looks like. It does not produce the audit trail for your agent. That still takes Purview configured, Entra Agent ID issued, traces flowing to Log Analytics, and someone signing off. The exam is a map of the work. The work is still the work!

Original post from sjwiggers.com: Cloud Perspectives - The Microsoft AI Stack in 2026 and the Certification Trail That Runs Through It

When Agentic Workloads Break the PaaS Assumptions

Steef-Jan Wiggers — Thu, 03 Sep 2026 13:21:18 +0000

This series started with a map and grew into seven pieces. Five layers came first: compute, where load shape picks the service; messaging and orchestration, where two questions replace four product choices; data patterns, where idempotency and the outbox keep a platform correct; governance and identity, where policy and audit become the compliance posture; and observability and FinOps, where behaviour and cost become visible. Then came the lens: from design to demonstrable operation, the shift from "is it built?" to "can we operate it responsibly?"

Every one of those pieces rests on a shared assumption. The system does what you told it to do. You wrote the workflow, you defined the routes, you set the policies, and the platform executes them. That assumption has held for every integration platform I've built. Agentic workloads break it. So this capstone asks what changes when the thing making decisions inside your platform is a model, not your code, and why each layer, plus the readiness lens itself, deserves a second look because of it.

The assumption agentic workloads break

Conventional integration is deterministic. A message arrives, a workflow runs its defined steps, a router sends it where the rules say. You can read the code and know what will happen. You can test every path. When something fails, you trace it to a step you wrote.

Agentic workloads replace part of that determinism with a model that decides at runtime. The agent reads context, picks a tool, interprets the result, and chooses the next action. Moreover, it does so differently depending on inputs you didn't fully anticipate. That's the point of it: the flexibility is the feature. But it means you can no longer read the code and know what will happen. So the ground under every layer shifts: behavior is no longer exactly what you specified.

None of this argues against agentic workloads. It argues for revisiting each layer with the shift named explicitly. Let's do that.

Compute: the loop changes the shape of the work

The compute layer sorted workloads by load shape: steady request traffic to App Service, event-driven bursts to Functions or Container Apps. Agentic workloads add a shape that sorting didn't account for: the loop.

An agent doesn't process a request and return. It reasons, calls a tool, waits, observes, and reasons again, sometimes for many cycles, before it finishes. That's neither a clean request-response nor a discrete event. Instead, it's a long-running loop of unpredictable duration with external calls in the middle. So the compute question changes. You're no longer asking "steady or bursty" alone. You're asking how to host something that runs for seconds or minutes, holds state across tool calls, and scales on a dimension — concurrent reasoning loops — that CPU-and-memory autoscale captures poorly. Container Apps with event-driven scaling often fit better here than App Service, and the orchestration frequently belongs in a workflow engine rather than raw compute.

Messaging and orchestration: the agent is a non-deterministic router

The messaging layer drew a clean line. Deterministic routing rules sent messages where the logic dictated. An agent orchestrating tool calls is, in effect, a router too, but a non-deterministic one. It decides which tool to call from its reading of the context, not from a rule you wrote.

The reliability consequences are real. Delivery guarantees still matter; an agent that triggers a business action still needs that action to occur exactly once, so Service Bus and the idempotency store in the data layer remain as relevant as ever. What changes is predictability. You can't fully anticipate which actions the agent will trigger, or in what order. Therefore, the orchestration has to stay correct under sequences you didn't design for. One practical lesson from building these loops applies directly: agent outputs rarely arrive as the clean structures a deterministic step would emit, so you build explicit bridges between agent actions rather than assuming shape.

Data: state and correctness under non-determinism

The data patterns held a platform correct when systems it didn't control misbehaved. Agentic workloads make those patterns more necessary, not less, and they add one more.

Idempotency matters more because an agent may retry a tool call or repeat an action as it reasons, so the dedup store carries a heavier load. The outbox matters just as much, because an agent-triggered write still has to propagate reliably. Workflow state matters more too, since the reasoning loop is exactly the kind of long-running, restart-surviving process that needs durable state and a correlation ID. And then the new one: conversation and context state. An agent carries context across turns, and that context has to live somewhere durable and queryable, which explains why a flexible document store keeps showing up as the default for agentic conversation state. The access pattern points at the store. Same principle as the map, applied to a new kind of state.

Governance and identity: where the assumptions break hardest

This layer changes most, and I'd insist any integration architect think it through before shipping an agentic workload.

The governance layer secured a deterministic platform. Identity answered who the caller was; policy constrained what the platform could be. Both still matter. However, agentic workloads open a gap that neither fully closes. Identity secures who the agent is. It does not touch what a poisoned tool result or a manipulated retrieved document makes the agent do. Prompt injection rides in through the data the agent requested inside the reasoning loop, downstream of the perimeter check everyone assumes protects them.

So the governance layer needs additions a deterministic platform never required:

Authorization moves per-action: A validated identity at the edge isn't enough. Each tool call the agent makes needs its own check: is this specific action allowed for this tenant right now? The perimeter check happens once; the risk recurs on every call inside the loop.
Recovery means compensation, not retry: Agent actions have side effects across systems. A failed sequence three actions deep can't restart from the top; it needs compensating actions to undo what already happened. That's saga-style thinking, and you design it; it doesn't emerge.
Containment has to be possible: When an agent misbehaves, you stop it fast, and at more than one layer. Layered containment, from a single configuration flip-up to a full block, turns "contain the agent" from an incident-call debate into a seconds-long operation.
Evaluation becomes a first-class layer: Operational observability tells you the agent is running. It doesn't tell you the agent's outputs are quietly degrading. Under the EU AI Act's oversight and transparency duties, that stops being optional polish and becomes evidence you're meeting an obligation.

The readiness lens, asked again

The design-to-operation post posed the question that decides go-live: not "is it built?" but "can we operate it safely, recoverably, auditably, and predictably?" Agentic workloads sharpen every word of that sentence.

Safely now includes per-action authorization and containment, because the threat walks in as data. Recoverably now means compensation and sagas, because retry alone can't undo side effects. Auditably now covers what the agent accessed, which tool it called, why it acted, and what policy constrained it — evidence the EU AI Act increasingly expects. And predictably is precisely the property the agent gave up, which is why the surrounding architecture has to supply it instead. The production baseline, the demonstrable-versus-designed test, the three moments of readiness — all of it still applies. Each bar sits higher.

The revised framework

The map closes with five questions. For agentic workloads, they hold, and each gains a harder edge. Before an agentic workload goes near a real system, I'd add these:

Can I host a long-running reasoning loop, not just a request or an event? Can my orchestration stay correct when I can't predict the action sequence? Does my data layer hold conversation state as well as business state, with idempotency doing heavier duty? Is authorisation per-action, not just per-identity? Can I contain a misbehaving agent in seconds? And can I evidence what the agent did, why, and whether its quality held?

Those aren't different questions from the series. They're the same layers, asked again under non-determinism, and then held up against the readiness lens one more time.

The shape of it

Each layer in the series meets its agentic stressor: the loop, the non-deterministic router, conversation state, injection past the perimeter, the evaluation question, and the readiness lens. Spanning them all, the workload surrendered predictability, so the architecture supplies it.

Agentic workloads don't replace the Azure PaaS foundation an integration architect builds on. They stress it. Every layer in this series still includes compute, messaging, data, governance, and observability, but each one now supports a workload that decides for itself at runtime. The compute layer meets the loop. The messaging layer meets a non-deterministic router. The data layer meets conversation state and heavier idempotency. The governance layer meets a threat that walks in through the front door as data. And the readiness lens meets a workload that surrendered predictability, so the architecture has to supply it.

The through-line of the whole series holds here too. The model is the least differentiated part of a production agent. What separates a demo from something you can run against real systems in a regulated industry is the architecture around it: the same layers, asked harder, and proven in operation rather than promised in design. So the foundation was never wasted. It's exactly what agentic workloads need, applied with the assumptions made explicit.

That's the series. Start at the Azure PaaS map for the layer-by-layer foundation, take the design-to-operation lens with you as the test, and come back here for what changes when the workload thinks for itself.

Originally published on Cloud Perspectives.

Why Cosmos DB Ends Up as the Agent Memory Database

Steef-Jan Wiggers — Thu, 03 Sep 2026 12:57:42 +0000

The first post in a series on Cosmos DB agent memory for AI agents, starting nine years before "AI agent" was a category.

In 2017, I built a proof of concept for a customer: a knowledge base on Cosmos DB, using the Graph model and Search, running at roughly 1,000 euros a month. I presented it at CloudBrew. One attendee wasn't impressed:

The most uninteresting talk of the day came from Steef-Jan Wiggers, who, in my opinion, delivered an hour-long marketing pitch for CosmosDB. I think it's expensive for what it currently offers, and many developers could architect something with just as much performance without needing CosmosDB.

He wasn't wrong that 1,000 euros a month raises eyebrows as a line item. He was wrong about what the line item paid for: the knowledge base was the product a subscription business planned to sell. Compare the cost to the revenue it enabled, and it's negligible. Compare it to nothing, and of course it looks "expensive." I made the same point about Figma's AWS bill last year: $109 million a year sounds alarming until you check it against $821 million in revenue and a business model that requires sub-100ms real-time collaboration for 13 million users. In short, cost without context is just a number that sounds big.

So here's the same argument, nine years later, with a different workload. Agent memory — chat turns, tool call results, embeddings, user preferences — is expensive to store the wrong way and reasonably cheap to store the right way, and increasingly "the right way" means one database instead of three. I'll come back to the actual RU numbers in post 6; for now, this post is about why the architecture argument holds up before cost even enters the picture.

The Same Shape of Problem, Nine Years Apart

Strip away "AI agent" and look at what you're actually storing: short-lived, high-volume, time-ordered records that need fast writes and selective recall. That's chat turns and tool outputs today. It's also, structurally, what I modeled in a Cosmos DB Conf 2023 talk on end-to-end retail process monitoring — messages and batches flowing between an ERP, a WMS, and a PIM system, which I tracked so a retailer could tell where something broke.

Different domain, same shape, though: append-heavy writes, a need to reconstruct "what happened, in order," and a downstream system (an incident manager then, an LLM now) that needs the right slice of history on demand, not the whole history every time.

In general, agent memory falls into two categories:

Short-term (episodic/working) memory — the last 5–10 turns of a conversation, intermediate tool call results, partial task state. Useful for the current task, disposable afterward (Cosmos DB's time-to-live feature is a natural fit here — more on that in post 2).
Long-term memory — user preferences, summarized threads, facts the agent should persist and recall across sessions.

Both need somewhere to live, but the default answer for the last few years has been: somewhere different.

Why the Stitched Stack Breaks Down for Cosmos DB Agent Memory

The common pattern from 2022 through 2025 was to give each concern its own database — an in-memory store for caching and session state, a relational database for operational data and conversation logs, a purpose-built vector database for embeddings. A reasonable instinct, in theory: each tool for its own job.

In practice, though, it doesn't hold up once an agent is the thing reading and writing across all three, on every turn.

Three databases, three failure modes — versus one Cosmos DB agent memory layer.

Each piece has a real weakness once agents are the workload, not an afterthought:

Pure vector databases tend to offer no strong read/write guarantees, limited ingestion throughput, availability below 99.9%, a single (eventual) consistency level, and thin multitenancy support. Fine for an embeddings side-project. Shaky as the record of what an agent told a customer.
Relational databases fight the fluid, evolving schema of agent state — new fields, new memory types, nested tool outputs — without migrations and, often, downtime.
In-memory caches are fast and don't persist, which is exactly backwards from what long-term memory needs.

As a result, three systems also means three consistency models, three availability profiles, and three places a multi-agent system can silently desynchronize. And that complexity tax doesn't show up in any single service's bill, which is part of why it's easy to miss until something breaks in production.

The Unified Case — and How Much of It I'd Already Used

The pitch for Cosmos DB agent memory as a unified layer rests on a small set of properties: single-digit-millisecond latency, a 99.999% availability SLA on the NoSQL API, DiskANN-based vector indexing built into the same store as the operational data, multi-master writes, and five selectable consistency levels from strong to eventual. In plain terms, that's one system that's fast enough for the hot path, available enough for production, and flexible enough to hold embeddings next to the record they came from.

In fact, two of the pieces that make this work aren't new to me, or new to this blog.

Change feed and hierarchical partition keys weren't built for AI agents — they just turned out to be exactly what Cosmos DB agent memory needed.

Change feed, for example. In the retail monitoring solution, change feed was the mechanism that turned a write into a trigger — a new record landing in Cosmos DB fired a Function, which could raise an incident. That's the same primitive I'll use in post 4 to coordinate handoffs between agents in a multi-agent system: one agent's write becomes another agent's signal to act, without polling.

Hierarchical partition keys, likewise. I covered these in Azure Cosmos DB's Latest Performance Features back in 2023 — partitioning by tenant, then by item, to keep related data colocated while avoiding the 20 GB logical partition ceiling. The mechanism hasn't changed; what's changed is the workload. Post 2 uses the exact same [tenantId, threadId] pattern to isolate one customer's agent conversations from another's.

Even so, I didn't build either feature for AI agents. Both turned out to be exactly what agent memory needs — a decent sign that the underlying database was solid before the AI use case arrived, and nobody retrofitted it to fit.

Where This Series Is Headed

This post is the framing argument for Cosmos DB agent memory. From here, the rest of the series gets specific:

Post 2 — designing the agent memory schema itself: partition key choice, TTL, and the turn-based data model that works best in practice.
Post 3 — vector, full-text, and hybrid search for recalling the right memories, not just any memories.
Post 4 (TBD) — multi-agent state and coordination, including change feed as the handoff mechanism.
Post 5 (TBD) — wiring Cosmos DB into Microsoft Foundry Agent Service as bring-your-own thread storage.
Post 6 (TBD) — the cost conversation, properly this time: RU drivers, semantic caching, and what this actually costs to run at scale.

The 2017 knowledge base cost 1,000 euros a month and paid for itself many times over as a revenue-generating product. Ultimately, the question worth asking about agent memory infrastructure in 2026 isn't "is this expensive" — it's the same question it always was: expensive relative to what?

Sources

Microsoft Learn — AI agents in Azure Cosmos DB
Microsoft Learn — Time to live (TTL) in Azure Cosmos DB
Azure Cosmos DB's Latest Performance Features
Figma AWS Costs Explained: Beyond the Hype and Panic
Microsoft Learn Shows — Leveraging Azure Cosmos DB for End-to-End Monitoring of Retail Processes

Originally published on Cloud Perspectives.