DEV Community: Steef-Jan Wiggers

The Embedding Rematch

Steef-Jan Wiggers — Wed, 14 Oct 2026 02:15:28 +0000

The Shortlist Decides First ended with a caveat I promised to resolve: the retriever in that experiment was BM25, a lexical matcher, and its catastrophic 5 percent recall on paraphrased requests was exactly the failure lexical matching should produce. Would a semantic retriever change the verdict? That is a measurable question, so here is the rematch: the same 100 tools, the same 200 tasks, the same four description conditions and k-sweep, with one change. Instead of BM25, the retriever embeds every tool document and every request with text-embedding-3-small on Azure OpenAI and ranks by cosine similarity. The harness is measure/eval_recall_embed.py in the companion repo; nothing else moved.

The results

Recall of the correct tool into the shortlist, by description condition and task kind:

condition	vocab @3	vocab @5	vocab @10	para @3	para @5	para @10
terse	95%	99%	100%	59%	73%	87%
realistic	91%	98%	100%	59%	76%	94%
verbose	97%	99%	100%	65%	82%	98%
realistic + aliases	98%	99%	100%	99%	100%	100%

Three findings, one win, one surprise, one standing verdict.

The win: embeddings rescue paraphrase

BM25 scored 5 percent on paraphrased requests at k=5. Embeddings score 76 percent on the same tasks with the same descriptions. That is the rematch headline, and it lands exactly where the theory says it should: a user who says "I want to cancel my booking" shares almost no tokens with a tool described as "deletes a reservation", but the two sentences live close together in embedding space.

The k-sweep behaves differently now too, and this is worth noticing. With BM25, widening the shortlist from 3 to 10 bought paraphrase recall from 3 to 10 percent, because the correct tool usually was not even near the top of the ranking. With embeddings, the same widening buys 59 to 94 percent. Semantic misses are near misses; lexical misses were not misses at all, they were absences. When your retriever is at least semantically in the neighborhood, paying for a longer shortlist actually purchases something.

The surprise: verbosity finally pays, and exactness costs

The shortlist post concluded that verbosity is not coverage, because doubling description tokens moved BM25's paraphrase recall not at all. The rematch forces me to refine that: it is true lexically and less true semantically. Under embeddings, verbose descriptions beat terse ones on paraphrase at every k, 82 against 73 at k=5, because every extra sentence adds semantic surface for the query vector to land on. Verbosity is still a poor deal, you pay roughly triple the tokens per tool for nine points, but it is no longer a zero.

The reverse surprise sits in the vocabulary columns. On requests that echo the tool's own vocabulary, BM25 scored 97 to 100 percent at k=3; embeddings score 91 to 97. Embeddings blur exact matches. When the user says precisely the words in the description, nothing beats matching those words literally, and a semantic retriever occasionally ranks a similar-sounding sibling above the exact hit. The two retrievers fail on opposite task populations, which is the textbook argument for hybrid retrieval, and now the series has the numbers for it.

The standing verdict: the alias sentence still wins

Look at the last row. With the 13-token alias sentence per tool, recall is 99 to 100 percent everywhere, at every k, and it was also 100 percent under BM25. The cheapest intervention in the entire series makes the retriever choice irrelevant.

That reframes what an embedding index buys you. If you control the tool descriptions, the alias sentence gets you to perfect recall for about 13 tokens per tool and zero infrastructure. The embedding pipeline, an embedding model, an index, a lookup on every request, buys you from 5 to 76 percent on paraphrase when you cannot touch the descriptions, which is precisely the situation of a gateway or catalog fronting other teams' MCP servers. Both answers are right; they answer different ownership questions.

The end-to-end multiplication from The Model Was Never the Problem updates accordingly. Recall times selection on paraphrased requests with realistic descriptions: BM25 gave 5 percent times 93 percent, roughly 5 percent of tasks ending in the right tool call. Embeddings give 76 times 93, roughly 71 percent. Aliases give 100 times 97, roughly 97, with either retriever underneath.

Honest limits

One embedding model, text-embedding-3-small; a larger model or a hybrid lexical-plus-vector ranker would likely narrow both gaps, and the harness takes any Azure OpenAI embedding deployment if you want to try. The paraphrases are template-generated, which makes them systematic but tamer than real user phrasing. The catalog is 100 tools; at 1,000 the embedding index gets more attractive because the alias discipline gets harder to enforce across teams. And this measures retrieval in isolation; the selection stage was measured separately and multiplies on top.

The takeaway

The rematch confirms the retriever was the right suspect and the description was still the right fix. Embeddings turn paraphrase recall from a catastrophe into a tax, 5 to 76 percent, and they are the honest recommendation when the descriptions belong to someone else. But when the descriptions are yours, one alias sentence per tool beats the entire vector pipeline, at one ten-thousandth of the operational surface. Fix the words first. Add the index when the words stop being yours to fix.

The harness, both retrievers, and the full results are in the companion repo: measure/eval_recall.py for BM25, measure/eval_recall_embed.py for the rematch.

The Third Leg of the Parity Test

Steef-Jan Wiggers — Wed, 07 Oct 2026 12:31:29 +0000

Months ago, a commenter on this series proposed a test: if the same backend really stands behind both doors, then auth, error behavior, and idempotency must be preserved across them. Authorization Stays in the Kitchen settled the first leg. The confusion telemetry and recovery messages settled the second. This post settles the third, and it turns out to be the leg that matters most for agents, because agents retry.

Why agents make this urgent

Code retries when a developer writes a retry policy. Agents retry on their own judgment: a timeout, an ambiguous error, a truncated response, and the model decides the call probably failed and tries again. Nothing in the MCP protocol stops the second call from being a second order. The new driver in the companion repo proves the failure mode as a control case before proving the fix:

3. retry without clientOrderId    PASS: orderId 9714f45b then 99523aed (two orders exist, as feared)

Same restaurant, same item, same quantity, two calls, two orders. Every production incident involving an agent and a non-idempotent write looks like this line.

One decision, one place, again

The series rule holds: neither door contains business logic, and deduplication is business logic. The kitchen keeps a store keyed on the caller's idempotency key, remembers a fingerprint of the request alongside the original confirmation, and answers one of four ways: placed, replayed, conflict, or invalid.

if (idempotencyKey is not null
    && _byIdempotencyKey.TryGetValue(idempotencyKey, out var seen))
{
    return seen.Fingerprint == Fingerprint(request)
        ? new OrderResult(OrderOutcome.Replayed, seen.Confirmation)
        : new OrderResult(OrderOutcome.Conflict, null);
}

A replayed key with the same request returns the original confirmation and creates nothing. The same key with a different request is a conflict, which is always a caller bug and deserves to be refused loudly. The doors only translate those outcomes into their caller's dialect, exactly as they did for authorization.

The REST door speaks header

HTTP already has a convention for this: the Idempotency-Key request header, specified in an IETF httpapi draft that Stripe, Adyen, and most payment APIs shaped in practice. The draft says a completed operation replays its original result and a key reused with a different payload SHOULD get a 422, so that is what the door does.

Verified live, the same curl twice:

HTTP/1.1 200 OK
{"orderId":"97e92454","restaurantName":"Trattoria Valkhof","itemName":"Margherita","quantity":2,"total":23.00,"status":"confirmed"}

HTTP/1.1 200 OK
{"orderId":"97e92454","restaurantName":"Trattoria Valkhof","itemName":"Margherita","quantity":2,"total":23.00,"status":"confirmed"}

Same orderId, one order, and the client cannot tell the replay from the original, which is the whole point. Then the same key with quantity 5:

HTTP/1.1 422 Unprocessable Entity
{"error":"idempotency_conflict","detail":"This Idempotency-Key was already used for a different request. Send a new key for a new order."}
The MCP door has no headers

Here the two doors genuinely differ, and the difference is instructive. A tool call carries a name and arguments, nothing else, so the idempotency key cannot ride in a header. It has to be an argument, which means it has to be in the contract, which means a sentence has to teach the agent to use it:

[McpToolProperty("clientOrderId", "An order reference that comes from you, the caller: invent a unique value, and reuse the same value if you retry this order after an error or timeout.", isRequired: false)]
    string? clientOrderId

That sentence is the mechanism. On the REST door, idempotency works because a developer read the docs and wrote a header into their client. On the MCP door, it works only if the description convinces the model to invent a reference and hold onto it across a retry. There is no SDK to do it for the agent; the prose is the SDK.

The series tooling priced and policed the sentence immediately. The contract linter flagged clientOrderId as an identifier parameter the moment it existed, so the description had to say where the value comes from, and this parameter gave the provenance rule its first unusual answer: it comes from you. The token budget rule priced the addition at 42 tokens, taking the contract from 302 to 344 against a budget of 350, so the next contract sentence will have to pay for itself by shortening another. And because the deployed description hash is computed from these same attributes, the descHash changed, and the confusion telemetry can attribute any behavior shift to this exact wording.

The driver verified both halves end to end against the deployed server:

1. retry with same clientOrderId  PASS: orderId d120204a then d120204a (one order exists)
2. same key, different request    PASS: The clientOrderId 'order-2ccc9bd3' was already used for a different order.

The conflict answer is recovery prose, as this door's denials always are, and it goes out through the telemetry helper as a new order_conflict sentinel, so key misuse is observable next to every other contract failure. Replays log their own trace, which gives you a retry rate for free: how often agents actually double-fire is now a query, not a guess.

Honest limits

The store is in-memory, like everything in this sample, so the replay guarantee spans one instance and one process lifetime. A production version needs a durable store with a TTL, and on a plan that scales out, a shared one. The guarantee is also only as good as the agent's discipline: an agent that invents a fresh clientOrderId on every retry gets no protection, and nothing in the protocol forces it to comply. The description can only persuade. That makes idempotency on the MCP door a probabilistic defense in a way it never was for REST clients, and a good candidate for the eval harness to measure someday: given a simulated timeout, does the model actually reuse the key the description told it to reuse?

The takeaway

The parity test now passes on all three legs, and each leg landed the same way: one decision in the kitchen, one dialect per door. For idempotency the dialects are a standard header on one side and a contract sentence on the other, and the asymmetry is the lesson. REST made retries safe with infrastructure; MCP has to make them safe with language. If your tools write anything that costs money, the deduplication store is the easy half. The sentence that gets the agent to use it is the half this series keeps measuring.

The kitchen policy, both doors, and the double-fire driver are in the companion repo; run measure/drive_idempotency.py against your own server and see whether check 3 scares you as much as it should.

Authorization Stays in the Kitchen

Steef-Jan Wiggers — Fri, 02 Oct 2026 09:50:06 +0000

What a 401 Means to an MCP Client ended on a deliberately unfinished note: the token asserts who the caller is, it does not decide what they may do once inside. That sentence has been an IOU ever since. This post pays it, with claims-based authorization implemented in the sample, the same denial rendered as a 403 JSON body on one door and as recovery prose on the other, and every state verified live, including two detours that taught me more than the happy path did.

One decision, one place

The design rule from the first post was that neither door contains business logic, and authorization is business logic. So the policy lives in the kitchen:

public static class OrderPolicy
{
    public const string RequiredRole = "Orders.Place";

    public static bool MayPlaceOrders(CallerContext caller) =>
        caller.Roles.Any(r => string.Equals(r, RequiredRole, StringComparison.OrdinalIgnoreCase));
}

Anyone may search restaurants and read menus; placing an order requires the Orders.Place role. The doors do exactly two things: gather identity however their protocol allows, and render the kitchen's decision in their caller's dialect. The REST door answers code, so it returns a machine-shaped refusal:

{"error":"forbidden","detail":"Placing orders requires the Orders.Place role.","requiredRole":"Orders.Place"}

The MCP door answers an agent, so it returns an instruction it can act on: "This caller is not allowed to place orders; that requires the Orders.Place role. Searching restaurants and reading menus remain available." And because that sentence goes out through the telemetry helper from the confusion-telemetry work, every denial is also an attributable order_forbidden row in Application Insights.

The REST door: a sentence in a token

App Service authentication validates the bearer token and hands the function the caller's claims in the X-MS-CLIENT-PRINCIPAL header; the door parses out the roles and asks the kitchen. Verifying it took one Entra app role (value Orders.Place on the restaurant-mcp-auth registration), one role assignment to my own account, and produced the cleanest before-and-after in this series: the same curl, the same body, the same endpoint, a 403 with the role named in it, and then an order confirmation for 23 euros. The only thing that changed between the two was one sentence inside the token.

Getting that sentence took two detours worth telling. First, the role assignment needs an enterprise application to attach to, and the portal's one-click MCP auth button had created the app registration without a service principal; the "Managed application in local directory: Create Service Principal" link on the registration's overview was the missing step, and until it was clicked, the app simply did not exist in the Enterprise applications list to assign anyone to.

Second, and more operationally important: after the assignment, the token stubbornly kept returning no roles. Not propagation, not misconfiguration, but two layers of caching, the Azure CLI reusing its still-valid cached token, and, more fundamentally, the fact that claims are minted at token issuance, not looked up live. A role you grant right now reaches a caller only when they next acquire a token. For human users that is an inconvenience; for agents, which hold tokens and run long sessions, it is an architectural fact: revoking or granting a role changes agent behavior at token-expiry speed, not immediately. Decode the token when in doubt; the roles claim either is or is not in there, and no amount of retrying the API changes it.

The MCP door cannot see you

Here is the honest half. The Functions MCP extension authenticates the request, Entra and Easy Auth both did their jobs in the previous posts, and then hands the tool code a ToolInvocationContext containing the tool name, the arguments, a session id, and transport info. Nothing about the caller. The platform knows exactly who is calling; the tool does not get told. Per-user authorization on the MCP door is therefore not possible at this layer today, the same category of gap as the clientInfo one from the telemetry post, and the feature request writes itself: forward the authenticated principal to the tool invocation.

What remains honest in the meantime is door-level authorization: an MCP_DOOR_ROLES app setting grants roles to the door as a whole, an explicit operator decision about what the MCP channel may do, which is a real and common enterprise posture ("agents may read, agents may not transact") even though it is coarser than per-user claims.

And the telemetry proved the gate works, in the most satisfying way this series has produced. The traffic driver's deliberately wrong order calls hit the door before the grant and landed in Application Insights as order_forbidden. One app setting later, the same calls landed as order_rejected, because authorization now passed and the invalid arguments failed business validation instead. Same requests, different sentinel, and the shift between the two is the authorization gate becoming visible in a Log Analytics table. (One footnote for your own dashboards: the isolated worker emits each log line twice through the host, so absolute counts double; dedupe on operation id or read ratios, not totals.)

Door parity, completed

The commenter's test from months ago asked whether both doors preserve the same auth, error, and idempotency behavior. The answer the sample now gives in full: the decision is identical because it is taken once, in the kitchen, from the same policy. The rendering is deliberately different per door, a status code for code, a sentence for an agent, and the denial is observable in both channels, as a 4xx in the request logs and as a sentinel in the confusion telemetry. Parity of decisions, dialect per door.

The takeaway

Authentication moved to the platform over the course of this series; authorization never left your code, and this post is the argument that it should not. Put the policy in the kitchen where both doors share it, feed it claims where the platform forwards them and explicit door grants where it does not yet, render denials in each caller's dialect, and log every denial with the same attributable telemetry as any other contract failure. And remember the quiet operational lesson from the token detour: claims are minted, not looked up, so authorization changes reach your agents exactly as fast as their tokens expire, and no faster.

The policy, both doors, and the traffic driver are in the companion repo; the whole implementation is one small policy class and a dozen lines per door.

Linting Prose in CI

Steef-Jan Wiggers — Thu, 01 Oct 2026 06:25:36 +0000

A commenter on Watching the Contract Fail in Production put the conclusion of this whole series into one sentence: once agents consume tool descriptions as part of their decision-making, changing a sentence is no longer just documentation, it can be an interface change, so important wording changes should be treated like code changes. This post makes the build system agree. The companion repo now has a contract linter and a GitHub Action, and the first thing the linter did was fail its own repo.

Four rules, each one earned

The linter is about a hundred lines of dependency-free Python (measure/lint_contract.py), and every rule in it was paid for by a measurement earlier in the series:

Provenance. Every identifier parameter must say where its value comes from ("as returned by search_restaurants"). This is the sentence a JSON schema cannot express, and the difference between an agent that chains tools and one that invents ids; the confusion telemetry showed exactly that failure as the menu_unknown_restaurant sentinel.

When-to-call. Every tool description must contain a usage cue, not only a what-it-does sentence. Selection within a shortlist works because descriptions disambiguate between siblings, and when-to-call is where that disambiguation lives.

Token budget. The serialized tools array must stay under a budget, counted with the same cl100k tokenizer the discovery post used. The budget here is 350 tokens; every description edit now has a visible price.

Drift. The C# attribute descriptions and the APIM Bicep copies of the same tools must be character-for-character identical, because the contract post demonstrated that two copies of a contract fork within a week even with one author.

A fifth rule, the alias sentence that took paraphrase recall from 5 to 100 percent, is a warning rather than an error: proven valuable in the retrieval experiments, advisory at three tools.

The linter parses the contract straight out of the McpToolTrigger and McpToolProperty attributes, the authoritative copy, the same source the deployed description hash is computed from. One pleasant cross-check fell out of that: the parser's token count of the extracted contract was 277, exactly the number the discovery post measured from the live server's tools/list response. Two extraction paths, one contract, same count.

The first run: red, deservedly

I pushed the linter with the honest commit message "current contract fails its own lint," and the Action delivered:

Nothing here was staged. The drift errors are the exact fork the contract post documented by eye weeks ago, still sitting in the repo, now caught mechanically. The when-to-call errors are real too: get_menu's description said what the tool does and nothing about when to reach for it, and I wrote that description believing descriptions were the contract. The linter's first catch was its own author, which is the strongest argument I can offer for having one.

The fix: 25 tokens, priced

The fix commit added when-to-call clauses to two tools ("Call it after search_restaurants and before placing an order", "Use it only after get_menu has confirmed the exact item name") and aligned the three Bicep descriptions with the C#. The build went green, and the budget rule priced the edit: the contract grew from 277 to 302 tokens, 25 tokens for two sentences of disambiguation, well under budget and exactly the kind of spend the measurement posts said earns its keep.

The commit pair is the point of the whole exercise. One push failed CI over sentences; the next push fixed sentences and passed. Wording changes went through the same gate code goes through: a diff, a red build, a fix, a green build. And because the deployed description hash is computed from these same attributes, this fix also changed the descHash, so the confusion telemetry from the previous post can compare agent behavior before and after the new wording. The lint gate and the telemetry loop are two ends of one pipeline: CI checks what can be checked statically, the dashboard watches what only production can reveal.

What the gate can and cannot hold

Worth being precise about the boundary. These rules check the mechanical half of description quality: presence of provenance, presence of a usage cue, total cost, copy consistency. None of them can tell you whether the sentence is good, whether "Use it only after get_menu has confirmed the exact item name" actually improves selection, that is what the eval harness measures and what the telemetry observes. The lint is the cheap gate that runs in six seconds on every push; the eval is the expensive gate you run when the cheap one says the shape is right. Most teams have neither; the order to add them in is lint first, because it catches the failures that need no model at all.

The takeaway

The contract is part of the system behavior, so it gets the treatment behavior gets: versioned by hash, observed in telemetry, and now gated in CI. A one-word description edit can fail your build, and that is not overengineering, that is the build system catching up with where your interface actually lives. The linter, the workflow, and both commits, the red one and the green one, are in the companion repo; point measure/lint_contract.py at your own attributes and find out what your contract has been getting away with.

Foundry Model Ledger: Which Model, at What Price, Until When, and Which of Mine

Steef-Jan Wiggers — Tue, 29 Sep 2026 07:16:38 +0000

Every model conversation with a team ends with the same four questions. Is the model available in our region? What can it do? What does it cost per million tokens? And when does Microsoft retire it? Then the platform team asks a fifth: which of our own deployments are affected? Microsoft’s Foundry Model Explorer answers the first question well. However, it does not answer the other four. It cannot, because the answers live in your subscription and in a price list that was never meant to be read by software. So I built the Foundry Model Ledger to read them together. The interesting part is what the data looks like once you do.

What the Foundry Model Explorer already does

Microsoft’s explorer is a region availability reference. It shows one row per model and version, with lifecycle status, deployment SKUs, retirement date, and the list of regions that carry it, exportable to CSV. If your question is “where can I deploy gpt-5.6-terra”, it is the fastest answer there is. However, it is a periodic snapshot rather than a live read. It shows no prices in any currency. And it knows nothing about what you have deployed. Those three gaps are the Foundry Model Ledger.

Three sources, one Foundry Model Ledger

The model catalog lives in Azure Resource Manager. A single call, GET /subscriptions/{id}/providers/Microsoft.CognitiveServices/locations/{region}/models, returns every model and version a region carries. Each entry has its capabilities (chat completion, tool calling, embeddings, context window), its lifecycle status, its deployment SKUs, and its retirement date under deprecation.inference. This is the same data the Foundry portal shows. Moreover, it needs nothing more than Reader on the subscription.

Prices live in the Azure Retail Prices API. It is public, needs no authentication, and returns every meter Azure bills. Filter on serviceName eq 'Foundry Models' and a region, and you get the list prices for tokens, images, and hours, in any currency the API knows.

Your own deployments live under each Cognitive Services or Foundry account. At first I reached for Azure Resource Graph, which is the natural place to list resources across a subscription. It returned zero rows against a subscription with twelve accounts. Resource Graph does not index the accounts/deployments child type. So the Ledger lists the accounts through ARM and then calls each account’s /deployments endpoint, six at a time.

How the Foundry Model Ledger is built

The Foundry Model Ledger is a .NET 8 isolated Azure Function on Flex Consumption with a single-page UI. It deploys with azd up, in the same shape as the [INTERNAL LINK: RAG in 8 Steps on Azure] sample. A user-assigned managed identity with Reader on the subscription reads the first and third source. The second source needs no identity at all.

Pick a region and a currency, and the table shows model, version, lifecycle, capabilities, and retirement date with a days-left bar. Next to those sit the input and output prices per million tokens. Click a row for every SKU with its capacity range, every capability ARM reports, and every price meter the tool matched. The same panel has a button that checks every other region for the same model and version.

Foundry Model Ledger detail panel for gpt-5.6-terra with lifecycle, deployment SKUs, matched retail price meters and region availability
The detail panel for gpt-5.6-terra: lifecycle, retirement date, every SKU with its capacity range, and the retail meters the matcher tied to this model. The area: EUR line under capabilities is the catalog’s own geography tag, not the currency switch.

Finally, a second tab joins your deployments to the catalog of their own region and sorts them by soonest retirement.

Every model deployment in the subscription, listed through ARM per account and joined to the catalog of its own region, sorted by soonest retirement. Resource Graph returned zero rows for the same subscription.

What the data says

Sweden Central, on the day I took these screenshots, carried more than 300 catalog entries from 11 publishers. 216 were generally available, 66 in preview, and 16 marked as deprecating. 70 versions retire within 90 days. Furthermore, 30 are already past their retirement date but still listed. That last group matters. The catalog endpoint tells you what the region knows about, not what you can still deploy. For example, gpt-4o-mini has been closed to new deployments for a long time and still appears. The SKU list and the retirement date are the better signals.

The same model and version often appears twice, once for account kind OpenAI and once for AIServices, each with its own SKU list. The Foundry Model Ledger merges those into one row. If you script against the endpoint yourself, expect the duplicates.

The Retail Prices API returned 1,727 meters for Foundry Models in that one region. None of them carries a model identifier.

Prices are written for invoices

A meter name is a billing label, not a key. The same model, gpt-5.6-sol, is spread across meters such as 5.6 sol ShortCo Inp Std Gl 1M Tokens, 5.6 sol LongCo Cd Wr PP DZ 1M Tokens, and 56sol ShCo Cd Wr Fl Gl 1M Tokens. Older meters read gpt 4.1 nano cached Inp glbl Tokens and bill per 1K tokens; newer ones bill per 1M. Similarly, grok-4.6 appears as 4.6 Inp DZ Tokens under the product Azure Grok Models, with no “grok” in the meter name at all.

The abbreviations are their own dialect. Input is Inp, inpt, or in. Output is Outp, opt, outpt, or out. Cached input is Cd. Global, data zone, and regional are Gl, DZ, and regnl. Batch, priority, and flex tiers have their own tokens.

How the matcher works

So the Foundry Model Ledger has a matcher. First, it narrows meters to the model’s publisher. Then it tokenizes both sides the same way and requires every token of the model name (minus the family prefix) to appear in the meter, in order. It also rejects meters that carry a sibling variant such as mini or pro the model does not have. When a meter carries a date token, o3 0416 or chat-latest 08062026, it pins the version. Next, it classifies direction, deployment type, tier, and context length, and normalizes the unit to a price per million tokens. Finally, it picks a headline: global standard, short context, uncached. Every price in the table carries a confidence label, exact, name, or loose. In addition, the detail panel shows every matched meter, so the headline number is never the only evidence.

What the first live run got wrong

On the first live run it priced 204 of the entries in Sweden Central. The misses were instructive. FLUX image models came out at “40,000 per million” because their meters bill per 1K images, and I had treated every 1K unit as tokens. Qwen sits under product Qwen models, which the publisher map did not know. Moreover, most of its meters are fine-tuning meters that must never become a headline price. And the Anthropic models, plus Cohere rerank and parse, are in the catalog with no Foundry Models meter in the region at all. The model exists; the public price does not.

Foundry Model Ledger price meters tab showing raw Azure Retail Prices rows for gpt-5.6-sol in Sweden Central
The raw Azure Retail Prices rows for gpt-5.6-sol in Sweden Central. Thirty-odd meters, no model identifier, and the abbreviations the matcher has to decode: ShortCo, Cd, PP, Gl, DZ.

Two tabs that came from using it

The first version stopped at the tabs above. Two more followed within a day, because the first questions people asked were not the ones I had built for.

Where else is this model available

The region check in the detail panel gives a list of names. That is correct and hard to read. So the Foundry Model Ledger now has an availability map. Every Azure region is plotted from the subscription’s own location metadata, on a world outline from Natural Earth. Type a model name, pick a version, and the regions that carry it light up in the brand blue. Regions without Azure AI services show as a dashed ring, and regions that host the service but not the model stay grey. Zoom presets for Europe, North America, Asia Pacific, the Middle East and Africa, and South America keep the labels readable where regions cluster. In Europe alone there are twenty. The data behind the map is one lightweight catalog call per region, cached, so a check across every region takes a few seconds the first time and is instant after that.

Foundry Model Ledger availability map showing the Azure regions that carry a selected model version
Where claude-haiku-4-5:2 is available. Blue is carried, grey is hosted but absent, dashed is no Azure AI services. The Europe preset separates the twenty European regions.

What else does the same job for less

The second question is the one that decides budgets. If we use this model for that workload, what else in the region can do the same job, and what would it cost instead? The Alternatives tab answers it with the data the Ledger already has. Pick the model you use or consider. Its capability flags from the catalog, such as chat completion, tool calling, JSON schema output, and image input, become chips, and every chip is required by default. Click one off if you do not need it and the list widens. Every other model in the region that reports all remaining capabilities is listed with its list price, what your monthly workload would cost on it, and the delta against your model, cheapest first. Enter the workload as millions of input and output tokens per month. Filters restrict the list to the same publisher or to generally available models, and retired versions are excluded. Other versions of the same model are listed last, so they do not pose as alternatives.

The comparison is on capabilities, not on quality. A nano model will always look like a 99 percent saving next to a frontier model, and the table does not know whether it would pass your evaluation set. What it does give you is the shortlist and the price gap in one view, which is the part that used to take an afternoon with the pricing page open in three tabs.

Foundry Model Ledger alternatives tab listing models with the same capabilities and their monthly cost delta
Alternatives to claude-fable-5:1 with the same capability flags, priced for 10M input and 2M output tokens a month. Green is cheaper, orange is dearer. The match is on capabilities, not quality.

Where the Foundry Model Ledger is the wrong answer

Do not budget on it. These are list prices, matched heuristically, in whatever currency you pick. The matcher will be wrong somewhere the day Microsoft renames a meter. It also cannot price provisioned throughput, which is billed per hour per unit rather than per model. Use it to see the shape of a decision, then confirm on the pricing page.

Do not put it on the internet as is. The Function has no authentication, and it exposes your deployment list to anyone with the URL. Keep it internal, or put Easy Auth with Entra ID in front of it. If you already run model endpoints behind API Management, the same gateway can front the Ledger, with the policies from the APIM for AI Workloads series.

And do not read the catalog as an availability promise. Listed does not mean deployable, and a retirement date is a floor, not a schedule. For a plain “where is it available” question, Microsoft’s explorer remains the quicker tool.

What I would build next

The tab I did not plan is the one people ask about: my deployments, joined to the catalog, sorted by retirement. That is a governance question, not a browsing question. It sits next to the routing questions from Don’t Build Around Today’s Model. Build for the AI Control Plane. Consequently, the natural next steps are all on that side. All subscriptions in a tenant instead of one. An alert when a deployment sits on a version retiring within 90 days. A history, so you can see what appeared and what closed in a region since last month, which is a changelog Microsoft does not publish. And a cost delta for moving each deployment to its successor, which the Alternatives tab now does for one model at a time and should do for the whole deployment list.

The Foundry Model Ledger repo is at github.com/steefjan1/foundry-model-ledger. azd up deploys it; scripts/run-local.ps1 runs it against your az login; node tools/mock-server.mjs runs the UI on a sample snapshot without .NET or Azure. If the matcher misprices a model in your region, the detail panel shows why. An issue with that screenshot is the fastest way to get it fixed.

Watching the Contract Fail in Production

Steef-Jan Wiggers — Mon, 28 Sep 2026 06:00:40 +0000

Tool Descriptions Are the Contract made a claim I had not tested: your MCP server's recovery messages are free confusion telemetry, because every time one is returned, you know exactly which misunderstanding just happened. The comment thread then designed the implementation better than I had: track each string alongside the tool version and a description hash, add the schema version as its own dimension, and treat a missing value as a first-class result, otherwise the cleanest dashboard is the one hiding the deployment that broke observability. This post builds exactly that, drives real traffic at it through the Entra-protected MCP door, and checks the dashboard against ground truth.

The one-line change, and the hash that cannot drift

Each of the sample's three recovery returns now routes through one helper:

private string Confusion(string sentinel, string tool, string message)
{
    logger.LogWarning(
        "Contract confusion: {Sentinel} on {Tool} (descHash {DescHash}, schemaVersion {SchemaVersion})",
        sentinel, tool, ToolContract.DescriptionHash, ToolContract.SchemaVersion);
    return message;
}

The sentinels are stable slugs, one per recovery sentence: search_no_match, menu_unknown_restaurant, order_rejected. The structured parameters become customDimensions in Application Insights, so the log line is a queryable row, not prose.

The description hash is the part worth stealing. The contract post showed that prose copies drift, so a hash maintained by hand would just be one more copy to forget. Instead, ToolContract computes it at startup by reflecting over the same McpToolTrigger and McpToolProperty attributes the MCP extension serves to clients, hashing every name and description in the deployed binary. The hash cannot disagree with what agents actually see, because it is derived from it. Edit one word of one description, redeploy, and every subsequent confusion row carries a new hash: the before and after of a contract change, joinable in KQL.

Two doors before the traffic

Driving traffic at the server turned into its own small verification of the 401 post. The traffic driver is a Python script that speaks streamable-HTTP MCP directly, and since the server has built-in Entra authentication, the driver needs a bearer token. Getting one stopped at two separate gates.

First, Entra refused to mint the token at all: AADSTS65001, because Azure CLI, the tool requesting it, was not on the app registration's preauthorized client list. Then, with that fixed, the function app answered 403, because App Service authentication keeps its own allowed-client-applications list, and the CLI was not on that one either. The known-clients control from the 401 post exists twice, independently: Entra decides who may start the flow, Easy Auth decides whose tokens the app accepts, and a client must pass both. My own traffic generator had to be admitted as a known client, twice, before it was allowed to confuse the server. For a regulated organization, that is not friction; that is the audit trail existing before you asked for it.

The traffic, with ground truth attached

The driver then made 60 calls: a rotation of requests that are wrong the way agents are actually wrong. An invented restaurant id, a restaurant name where an id belongs, a menu item from memory instead of from get_menu, a quantity past the cap, filters that match nothing, and valid calls as control. Because the rotation is deterministic, the console output is ground truth: exactly 16 menu_unknown_restaurant, 16 order_rejected, 14 search_no_match, and 14 clean results.

Which makes the dashboard check a real test instead of a demo. In Application Insights:

traces
| where message startswith "Contract confusion"
| extend sentinel = tostring(customDimensions.Sentinel),
         tool = tostring(customDimensions.Tool),
         descHash = tostring(customDimensions.DescHash),
         schemaVersion = tostring(customDimensions.SchemaVersion)
| summarize confusions = count() by sentinel, tool, descHash, schemaVersion
| order by confusions desc

The result: order_rejected 16, menu_unknown_restaurant 16, search_no_match 14, every row carrying the same descHash and schemaVersion 1. The dashboard matches the driver's console exactly, so the pipeline is verified end to end, from a wrong argument in a JSON-RPC call to an attributable row in a Log Analytics table.

Reading the dashboard as a contract report

Each sentinel names a failing sentence. A rising menu_unknown_restaurant count says the provenance sentence in get_menu's description, "as returned by search_restaurants", is not landing: agents are inventing ids. A rising search_no_match says callers expect a wider catalog than the descriptions promise. And because every row carries the descHash, the response to a bad trend is an experiment, not an argument: rewrite the sentence, redeploy, and compare the same sentinel across the two hashes. The description edit gets the same before-and-after treatment a performance fix gets.

The commenters' missing-value rule matters here too. The healthy state of this dashboard is not empty, it is present-with-baseline: some confusion is normal while valid traffic flows. A sentinel that silently vanishes, or rows arriving without a descHash, means a deployment broke the telemetry, not that agents stopped being confused. The alert to write is on absence and on unknown dimension values, not only on spikes.

What is still missing, honestly

One dimension from the design is not in these rows: the client. The MCP initialize handshake carries clientInfo with a name and version, but the Functions MCP extension does not currently hand that to the tool method through ToolInvocationContext, so per-client confusion rates are not yet possible at this layer. That is a real gap; per-client breakdowns are how you distinguish one badly prompted agent from a contract problem affecting everyone. Until the extension exposes it, the client mix has to come from the platform's request telemetry instead of the tool's own rows.

The takeaway

This is the cheapest contract test I know of: one logging helper, three stable sentinels, a hash computed from the deployed attributes so it can never drift, and a KQL query. The recovery messages were already written, the dashboard was already there; connecting them turns every agent mistake into an attributable data point against a specific version of your words. The prose is the interface, and now the interface has monitoring.

The code, the traffic driver, and the queries are in the companion repo; the whole change is one class and one helper in src/RestaurantBackend/Mcp/RestaurantMcpTools.cs, and measure/drive_confusion.py reproduces the traffic against any MCP server of your own.

The Model Was Never the Problem

Steef-Jan Wiggers — Thu, 24 Sep 2026 08:07:57 +0000

The shortlist post ended with a promise: stage two of the eval, selection within the shortlist, measured with a live model, scored so that a recall miss can never masquerade as a selection miss. This post pays that debt. The design came straight from the comment threads: same tool pools, same task pairs, shortlists from the same retriever, and every result row tagged with the description-set hash so the numbers stay attributable across changes. The runner is measure/eval_selection.py in the companion repo, stdlib Python against any Azure OpenAI deployment.

The method

Stage one showed recall failing at 5 percent on paraphrased requests, so scoring stage two on retrieved shortlists would mostly measure retrieval again. Instead, every shortlist here is forced: the correct tool plus its four strongest BM25 distractors, deterministically shuffled. That construction has a pleasant side effect. Because the distractors are whatever retrieval scores highest for the request, they are the correct tool's same-noun siblings. When the expected tool is approve_claim, the shortlist also contains cancel_claim, update_claim, and search_claim. The model does not get to coast on topic matching; it has to read verb semantics.

The model is gpt-5.4-mini, deliberately the small and cheap one, at its default temperature, sixty tasks per condition, half vocabulary phrasing and half paraphrase, across the four description conditions from the recall post. Scored on two things: did the model pick the expected tool, and did the task's reference value (the INV-2005 in the request) land in the arguments of a correct pick. The argument check is deliberately lenient; exact schema validation would punish harmless formatting.

One anecdote from setup that belongs in this series: the eval refused to run with an API key, because the Azure AI resource had disableLocalAuth set. The harness that measures keyless MCP tooling had to authenticate with an Entra bearer token itself. The 401 post's argument, arriving in person.

The numbers

Selection accuracy, forced shortlist of five, sixty tasks per condition:

Descriptions	Selection, vocabulary	Selection, paraphrase
Terse	97%	97%
Realistic	97%	93%
Verbose	90%	93%
Realistic plus aliases	97%	97%

Read that table as flat, and I mean that as a finding, not a shrug. At thirty tasks per cell, every difference in it is one or two tasks, and one run at default temperature is a sample, not a truth. The honest summary is a single sentence: selection accuracy is 90 to 97 percent everywhere. A small model picks the right tool from a five-tool shortlist about nineteen times in twenty, whether the descriptions are terse or verbose, whether the user echoed the tool's vocabulary or paraphrased it into synonyms, and with the tool's own siblings crowding the list. Argument filling lands slightly lower, 87 to 97 percent, under the lenient scoring.

Two null results hide in there, and both are useful. Description verbosity, which did nothing for recall, also does nothing for selection; terse held its own against descriptions three times its token price. And the paraphrase gap, which was catastrophic for recall, nearly vanishes at selection time: once the right tool is in front of it, the model bridges "sign off on the damage report" to approve_claim without difficulty. The vocabulary gap is fatal to retrieval and almost irrelevant to judgment.

Multiply the stages

The two-stage arithmetic is the real headline. End to end, task success is roughly recall times selection. For a paraphrased request against one hundred realistic tools:

Setup	Recall	Selection	End to end
Realistic descriptions	5%	93%	~5%
Realistic plus aliases	100%	97%	~97%

The entire end-to-end difference lives in stage one. Report one blended number, as most tool-calling benchmarks do, and you would conclude the model picks tools badly, when the model almost never saw the right tool at all. Measured separately, the model turns out to be nearly blameless, and the 13-token alias sentence from the recall post is carrying the whole system.

What this changes about the budget

The reflex when an agent picks wrong tools is to reach for a bigger model. At this shortlist size, that money is misspent, because selection was never the failing stage. The failing stage was vocabulary coverage in the descriptions, which is fixed with words that cost thirteen tokens per tool, not with a model that costs ten times more per call. There will be scales where that stops being true; a shortlist of twenty near-duplicates or genuinely ambiguous intents will stress judgment in a way five siblings do not, and that is a measurable follow-up. But the burden of proof has moved. Before spending on model size for tool calling, measure your recall, because the odds are the model never saw the tool you are blaming it for not picking.

The caveats, as always

One model, one run, sixty tasks per condition, differences within the table smaller than its noise floor. The forced shortlist isolates selection but flatters no condition, since all four saw identical lists. And the ceiling on the alias numbers inherits the recall post's caveat: my aliases match my paraphrases by construction. What survives all of that comfortably is the shape: selection high and flat, recall variable and decisive.

The runner, the results with their description hashes, and everything needed to reproduce this against your own deployment are in the repo under measure/; the whole run costs a few cents. Argue with the numbers directly, it is what the comment section of this series is for.

The Shortlist Decides First

Steef-Jan Wiggers — Tue, 22 Sep 2026 06:21:29 +0000

The eval this post runs was designed by the readers. A commenter on What the Agent Pays for Discovery reframed tool selection as a two-stage system: first recall, whether the correct tool makes it into the shortlist that deferred loading or retrieval produces, then selection, whether the model chooses it from that shortlist. He added the design constraint that stuck with me: a cheap context with the right tool absent is the worst outcome, because it fails silently. The model does not error; it improvises with what it has.

That splits the eval into two experiments with very different price tags. Stage two needs a live model, many runs, and scripted tasks. Stage one needs no model at all. Deferred tool loading in practice means retrieving candidate tools by matching the user's request against tool names and descriptions, and retrieval quality is a property of the descriptions. That is measurable deterministically, so this post measures it. The harness lives in the companion repo under measure/, next to the token counter from the discovery post.

The setup

The tool pool is the same deterministic set of one hundred synthetic enterprise tools from the discovery post (search_invoice, approve_claim, and so on), at the same three description levels: terse, realistic, verbose. Retrieval is BM25 over each tool's serialized definition, name, description, and parameter descriptions, with a shortlist of five, which is the curated-manifest size from the break-even discussion.

The tasks are the interesting part. Two hundred of them, one pair per tool, in two kinds. Vocabulary tasks echo the tool's own domain words: "Please approve the claim CLM-2041." Paraphrase tasks say the same thing the way a person who never read your API docs would: "Can you sign off on that damage report, number CLM-2041?" Same intent, same identifier, not one shared content word.

The numbers

Recall at five, one hundred tools, two hundred tasks:

Descriptions	Vocabulary	Paraphrase
Terse	100%	5%
Realistic	97%	5%
Verbose	97%	4%

Two findings, one comfortable and one not. When the user speaks the tool's vocabulary, retrieval is essentially solved; even terse descriptions score perfectly, because the tool name alone carries the match. When the user paraphrases, recall collapses to five percent, and here is the uncomfortable part: verbosity does nothing. The verbose tier costs 208 tokens per tool, roughly double realistic, and its recall is statistically identical. All those extra words are the same words repeated. More prose is not more retrieval surface if it keeps drawing from the same vocabulary.

Which exposes the assumption hiding inside "just defer tool loading and retrieve on demand." Retrieval only sees what the description says. Your users say "bill", "delivery", "sign off". Your descriptions say "invoice", "shipment", "approve". Between those two vocabularies sits a 95 percent silent failure rate, and the failure mode is exactly the one the commenter flagged: the shortlist arrives cheap, plausible, and wrong, and the model does its best with it.

Can a longer shortlist buy it back?

The obvious counter is to widen the shortlist, so I swept it. The shortlist size is also a context budget, priced with the tokenizer from the discovery post: five realistic tools cost about 542 tokens per model call, ten cost 1,040.

Shortlist	Tokens per call	Paraphrase recall (realistic)
3	~362	3%
5	~542	5%
10	~1,040	10%

Doubling the price buys recall from five percent to ten. That is not a fix, that is a linear crawl toward a target the vocabulary gap keeps out of reach; extrapolate it and you are back to shipping all hundred tools, which is the situation deferred loading exists to avoid. Meanwhile the alias condition below sits at 100 percent already at a shortlist of three. You cannot buy your way out of a vocabulary gap with a longer list; you can only close the gap with words.

The thirteen-token fix

The fix follows directly from the diagnosis. If recall is vocabulary coverage, add vocabulary. I appended one sentence to each realistic description, naming its synonyms: "Users may also say 'sign off on' or 'damage report'." Then reran:

Descriptions	Vocabulary	Paraphrase	Tokens per tool
Realistic	97%	5%	~111
Realistic plus aliases	97%	100%	~124
Verbose	97%	4%	~208

The alias line costs 13.4 tokens per tool. The verbose tier costs 97 extra tokens per tool and buys nothing. Thirteen tokens of the right words beat ninety-seven tokens of ceremony, which sharpens the budget rule from the contract post: spend description tokens on disambiguation, on provenance, and now on the user's own vocabulary. An alias sentence is the cheapest line in the whole contract.

One honest caveat, because this series has a rule about that. My alias sentences contain the same synonyms my paraphrase tasks use, so 100 percent is the ceiling case; it demonstrates the mechanism, not a production guarantee. In a real system, the aliases come from somewhere messier and better: the confusion telemetry from the contract post. Every recovery string an agent triggers, and every phrasing that produced a retrieval miss, is a user-vocabulary sample you did not have at design time. The telemetry loop and the alias line are the same feedback cycle, observed and then closed.

A second caveat, aimed at the comment I expect first: the retriever here is BM25, which is lexical, and an embedding retriever handles synonyms better out of the box. It softens the cliff, but it does not repeal the mechanism, because any retriever can only match against the surface the description exposes, and embedding similarity between "approve claim" and "sign off on the damage report" is still weaker than a description that contains both phrasings. The alias line enriches the surface for either retriever. The harness's retriever is one pluggable class, so an embedding comparison is a fair follow-up experiment, and I would genuinely like to see the numbers.

What stage one changes about stage two

Stage two, selection within the shortlist, still needs the live model, and it is coming. But stage one has already changed what stage two must control for. If the correct tool misses the shortlist 95 percent of the time on paraphrased requests, then any end-to-end accuracy number that does not separate the stages is mostly measuring retrieval, not the model's judgment. Report one blended number, and you will conclude the model picks tools badly, when the model never saw the right tool at all. Recall misses get their own headline number, exactly as the comment demanded, or the eval lies to you.

The stage-two design, for the record it will be measured against: the same tool pools and task pairs, shortlists produced by the same retriever, the model asked to choose and call a tool, scored on tool choice and argument correctness, per description variant, with every run tagged by the description hash from the telemetry thread. Selection accuracy conditional on successful recall, so the two failure modes stay separated.

The takeaway

Before the model chooses, the shortlist decides, and the shortlist is only as good as the vocabulary in your descriptions. Verbosity is not coverage, and neither is shortlist size: doubling the description bill moved recall not at all, doubling the shortlist bought five points, and a thirteen-token alias sentence took paraphrase recall from five percent to perfect at the smallest shortlist tested. So treat user vocabulary as part of the tool contract: put aliases in the descriptions, feed them from telemetry, and lint for them in CI next to provenance and when-to-call clauses. The silent failure the commenter warned about is real, cheap to measure, and cheap to fix.

The harness, the task set, and the results are in the repo under measure/; run python eval_recall.py and argue with the numbers directly.

What a 401 Means to an MCP Client

Steef-Jan Wiggers — Fri, 18 Sep 2026 11:06:43 +0000

Twice in this series, a client refused to connect to a working MCP server, and both refusals looked like bugs until they turned out to be the specification doing its job.

The first was in MCP vs API Is the Wrong Question. The Functions-hosted MCP server was protected by its mcp_extension system key, and Claude Code, VS Code, and MCP Inspector all connected happily with the key in an x-functions-key header. Then I tried adding the same server as a connector in the Claude desktop app, and it failed with "Couldn't register with the sign-in service." The second was in the APIM setup in the repo: MCP Inspector, pointed at the gateway without its subscription key attached, reported "Dynamic Client Registration rejected (HTTP 404)."

Different clients, different servers, the same wall. This post explains what that wall is and how to take it down properly with Microsoft Entra ID.

A 401 is not a request for a header

To a developer, a 401 means "add credentials and retry." To a client that implements the MCP authorization specification, a 401 means something more specific: start OAuth. The client fetches the server's Protected Resource Metadata to learn which authorization server protects it, registers itself with that authorization server (dynamic client registration), and walks the user through a sign-in and consent flow. No step in that sequence involves asking a human to paste a key into a header field.

That is why both failures happened. My key-protected servers returned 401 without any Protected Resource Metadata behind them, the clients went looking for an OAuth story that did not exist, and the attempt died at registration with a 404. The clients were not broken. They were refusing to work around a server that spoke half a protocol.

Which splits the MCP client world cleanly in two. Developer tools like Claude Code, VS Code, and MCP Inspector let you attach arbitrary headers, so a shared key works, the way it always has for developers who read documentation. End-user clients like the Claude desktop app deliberately offer no header escape hatch: the spec flow is the only flow. Keys are for tools; OAuth is for people.

Turning on the real thing

The Functions MCP extension supports built-in authentication with Entra ID as the identity provider, and the setup has become almost anticlimactic. In the Azure portal, the function app now has an AI (preview) tab with an Authentication section and a "Turn on MCP authentication" button. Give the Entra app registration a name, save, and it creates the registration, wires up App Service authentication, adds the required settings, and disables key-based access to the MCP endpoint in one step.

The manual route shows what that button actually does, and it is worth seeing once. Key access is turned off by setting the webhook authorization level to anonymous, because the platform's authentication layer now stands in front of it:

az functionapp config appsettings set --name <app> --resource-group <rg> --settings "AzureFunctionsJobHost__extensions__mcp__system__webhookAuthorizationLevel=Anonymous"

Then App Service authentication (the feature long known as Easy Auth) is configured with Microsoft as the identity provider, a new app registration, unauthenticated requests answered with 401, and the token store enabled. Finally, the server advertises its Protected Resource Metadata so that spec-compliant clients can discover the authorization server:

az functionapp config appsettings set --name <app> --resource-group <rg> --settings "WEBSITE_AUTH_PRM_DEFAULT_WITH_SCOPES=<scope from Expose an API>"

That last setting is the difference between a dead end and a doorway. With it in place, the 401 comes with directions.

The second wall: invalid_target

Almost anticlimactic, I said. Then I connected VS Code and hit a second wall the tutorial does not mention. Discovery worked, the client found the metadata and the authorization server, and Entra refused the token request with AADSTS9010010 invalid_target: "The resource parameter provided in the request doesn't match with the requested scopes."

The cause is visible in the metadata itself. A spec-compliant client sends the RFC 8707 resource parameter, taken straight from the Protected Resource Metadata, and Entra now enforces that this resource matches the audience of the requested scope. The button-made setup fails that check twice over. The metadata advertised resource: https://.azurewebsites.net but a scope of api:///user_impersonation, two different names for the same app. And the document clients actually read is the path-specific one at /.well-known/oauth-protected-resource/runtime/webhooks/mcp, whose resource includes the full endpoint path.

The fix is three alignments, so that resource and scope audience are character-for-character identical:

On the app registration, under Expose an API, change the Application ID URI from the default api:// to the endpoint URL: https://.azurewebsites.net/runtime/webhooks/mcp.
Update the advertised scope to match: set WEBSITE_AUTH_PRM_DEFAULT_WITH_SCOPES to https://.azurewebsites.net/runtime/webhooks/mcp/user_impersonation.
In App Service authentication, add the same URL to the allowed token audiences.

Then fetch the path-specific metadata document and check that resource and the prefix of scopes_supported are the same string. When they were, VS Code connected on the next start: 401, discovery, sign-in, token, and "Discovered 3 tools" in the output, with no key anywhere in the configuration.

This is not a quirk of one sample. The same error is currently open against the Azure DevOps MCP server and VS Code, and there is a growing catalog of Entra and MCP authorization mismatches. The pattern is the post's thesis repeating one level down: a strict client, a server speaking half a dialect, and the client refusing to guess.

The client connection, before and after

Before, the client configuration carried a secret:

{
  "servers": {
    "restaurant-directory": {
      "type": "http",
      "url": "https://<app>.azurewebsites.net/runtime/webhooks/mcp",
      "headers": { "x-functions-key": "<the system key>" }
    }
  }
}

After, it carries nothing:

{
  "servers": {
    "restaurant-directory": {
      "type": "http",
      "url": "https://<app>.azurewebsites.net/runtime/webhooks/mcp"
    }
  }
}

On first connect, the client receives the 401 plus metadata, discovers Entra ID, and opens a browser window. You sign in as yourself, consent once, and the client holds a token that refreshes automatically. The credential in the picture is now an identity, revocable per person, subject to Conditional Access, visible in sign-in logs. Everything the system key was not.

The Entra wrinkle: known clients only

There is one honest caveat, and it is where the story gets interesting for enterprise architects. Entra ID does not do open dynamic client registration; it wants to know its clients. The tutorial flow handles this by preauthorizing specific client applications on the app registration: VS Code, for example, is client id aebc6443-996d-45c2-90f0-388ff96faa56, added under "Expose an API" so users are not prompted for admin consent.

For clients outside the preauthorized list, you register a client application yourself and hand its id to the client. The Claude desktop connector anticipates exactly this: the error message from the first post already offered a field to "add an OAuth Client ID in the connector settings." What read as a workaround then is the designed path now: create the client registration in your tenant, preauthorize it against the server's app registration, give the connector the client id, and the sign-in flow completes inside your own identity perimeter.

Seen from a governance angle, this is a feature wearing a limitation's clothes. Anonymous dynamic registration means any client anywhere can start a flow against your server. A preauthorized client list means your identity team decides which MCP clients exist in your estate. For a regulated organization, that is not friction; that is the control you would have had to build anyway.

What the token does and does not do

The token asserts who the caller is, which tenant they belong to, and that your authorization rules let them reach the endpoint. It does not decide what they may do once inside. Tool-level authorization, the difference between a user who may search restaurants and one who may place orders, remains your code's job, and the claims in the token are the input to it. Authentication moved to the platform; authorization stayed in the kitchen, exactly where the first post's caveats said it lives.

Verdict

Keys for demos and developer tooling, where a header field is available and rotation is cheap. Entra ID with built-in MCP authentication for anything a human signs into, and for any server that should exist inside an organization's identity perimeter rather than beside it. The clients already enforce this split; the only choice you get is whether to be surprised by it.

The sample repo has both doors and the APIM patterns this builds on; the Functions MCP tutorial has the full authentication walkthrough.

Tool Descriptions Are the Contract

Steef-Jan Wiggers — Thu, 17 Sep 2026 07:37:43 +0000

This series started with a claim: MCP and REST are two doors into the same kitchen. Three posts later, the comments have pushed the argument down to its foundation. One reader on dev.to said the useful test is whether both doors preserve the same auth, error, and idempotency behavior. Another warned about granularity drift and asked how anyone validates tool descriptions. Both are pointing at the same uncomfortable fact: in an MCP server, prose is load-bearing. The description decides whether the agent picks your tool. The parameter text decides whether it calls the tool correctly. The error message decides whether it recovers. That prose is the API contract now, and almost nobody treats it like one.

The contract already lives in three places

Here is what I did not expect to find in my own sample repo. The restaurant backend describes its behavior in prose three times. The Functions MCP door carries descriptions in C# attributes:

[McpToolTrigger("search_restaurants",
    "Searches the restaurant directory. Both filters are optional; call it without arguments to list every restaurant.")]

The APIM REST door carries operation descriptions in Bicep. And the third pattern from the APIM post, where the gateway manufactures an MCP server from the REST API, carries its own tool descriptions in Bicep too:

description: 'Searches the restaurant directory. Both cuisine and city filters are optional; call without arguments to list every restaurant.'

Read those two closely. They are not the same sentence. I wrote them two days apart, and they drifted: "Both filters" became "Both cuisine and city filters", "call it without" became "call without". Harmless here. But this is a three-tool sample maintained by one person for one week, and the copies already disagree. Scale that to fifty tools, three teams, and a year of changes, and the gateway's MCP door describes a backend that no longer behaves the way its C# door says it does. That is the granularity drift the commenter meant, in miniature: the contract forks the moment it exists in two files, and nothing in the toolchain notices.

The REST door gets away with prose drift because its real contract is elsewhere: routes, status codes, an OpenAPI document that tooling can diff. The MCP doors have no elsewhere. The description is the interface.

What a good description buys, at a known price

The measurement post put numbers on the cost side: a realistic description runs about 111 tokens per tool, and a verbose one 208, paid on every model call. So the budget question is what each sentence buys. The sample's descriptions follow three rules, and each rule earns its tokens differently.

Say what the tool does, in one sentence, in terms that distinguish it from its neighbors. Say when to call it, which is mostly about optionality: "both filters are optional; call it without arguments to list every restaurant" prevents the agent from inventing a filter just to have one. And say where values come from: "the restaurant id, as returned by search_restaurants (for example 'r1')". That last rule is the one a JSON schema cannot express. A schema says restaurantId is a required string. Only prose says the string must come from a previous search_restaurants call and must not be guessed. Provenance is the highest-value sentence in the whole contract, and it is the difference between an agent that chains tools and one that hallucinates identifiers.

The verbose tier from the measurement post doubles the bill mostly with ceremony the model would infer anyway. Spend on disambiguation and provenance, cut the rest.

Door parity is about decisions, not formats

The commenter's test, applied to the sample: auth and idempotency live in the kitchen, in the shared service that both doors call, so neither door can drift on what is allowed or what happens on a retry. Error rendering is the one thing the doors deliberately do differently. The REST door returns machine-shaped errors, a 4xx status with a small JSON body, because its caller is code and code branches on status. The MCP door returns instructions:

return menu is null
    ? $"Restaurant '{restaurantId}' was not found. Use search_restaurants first to get a valid id."
    : JsonSerializer.Serialize(menu, JsonOptions);

Same fact, two renderings. That passes the parity test rather than failing it: parity means the doors agree on the decision, not that they speak the same dialect. The kitchen decides the order is invalid; each door reports it in the shape its caller can act on.

Your error strings are confusion telemetry

Those recovery sentences do a second job that I have not seen anyone talk about. They are constants, so every time one is returned, you know exactly which misunderstanding just happened. "Use search_restaurants first to get a valid id" fires when an agent invented a restaurant id, which means the provenance sentence in get_menu's description did not land. Add one log line where each recovery string is returned and Application Insights turns into a description quality dashboard: a KQL query grouping by sentinel string, per tool, per client. A rising count on one string is not an outage. It is a failing sentence in your contract, located precisely. I know of no cheaper contract test than instrumenting the errors you already wrote.

Validating the contract

Which is the honest answer to the commenter's validation question: start by treating descriptions as artifacts that CI can check without any LLM. Lint that every identifier parameter states its provenance. Lint that every tool description contains a when-to-call clause. And budget-gate the whole array: the token harness from the measurement post lives in the repo under measure/, and pointing it at your tools array in CI turns "our context bill crept up" into a failing build with a number in it. Drift between copies is checkable the same way; if the gateway's Bicep descriptions and the code's attribute descriptions are both machine-readable, a test can diff them.

What CI cannot check is whether the model picks the right tool, and I will not pretend otherwise. That needs the deferred experiment: scripted tasks with known correct tool sequences, run repeatedly against description variants. A commenter on the measurement post sharpened how to score it: as a two-stage system. Stage one is recall, whether the correct tool made it into the shortlist that deferred loading or retrieval produced. Stage two is selection, whether the model chose it from that shortlist and called it correctly. The stages fail differently: a recall miss is a retrieval or granularity problem, a selection miss is a description problem. And recall misses need their own headline number, because they fail silently; a cheap context with the right tool absent is the worst outcome, and the model will improvise something plausible instead of erroring. The measurement harness was structured so that this eval slots in on top, and it is the next piece of this series with numbers in it.

Verdict

The description is the interface. So give it the treatment interfaces get: one authoritative copy where you can manage it, review on every change, a token budget enforced in CI, provenance sentences on every identifier, and telemetry on the error strings that reveal where the contract fails. The doors were never the hard part. The words were.

The sample, the APIM patterns, and the token harness are all in the repo.

What the Agent Pays for Discovery

Steef-Jan Wiggers — Wed, 16 Sep 2026 11:15:02 +0000

A reader of MCP vs API Is the Wrong Question asked the question I could not answer at the time: did you measure the token cost of the live MCP discovery round trip against handing the model a curated tool manifest up front, and where does the break-even sit? I answered from first principles then. This post answers with numbers, from a small measurement harness that now lives in the companion repo under measure/.

Where the tokens actually go

First, the framing correction that makes the question answerable. The live discovery round trip is nearly free: an MCP client calls tools/list once per session, not per turn, and the payload that comes back is essentially the same manifest you would curate by hand. The cost that matters is different and larger: whatever tools the client ends up with, their definitions are injected into the model's context on every single call. A curated manifest pays that too. So "discovery versus manifest" is not the axis. The axis is how many tool definitions sit in context, how wordy they are, and how many model calls the task takes.

That is measurable without an LLM in the loop, exactly, with a tokenizer.

The method

The harness serializes a tools array the way it reaches a model (JSON: name, description, input schema with per-parameter descriptions) and counts cl100k tokens. The anchor is real: the three tools of the restaurant sample, exactly as the Functions MCP extension serves them. Around that anchor it generates synthetic enterprise tools (search_invoice, approve_claim, and so on, two to four parameters each) at three description levels: terse (a sentence fragment), realistic (purpose plus parameter guidance, modeled on the sample's actual descriptions), and verbose (usage guidance, examples, and edge-case notes on everything). One honest caveat: harnesses reformat definitions slightly differently, so treat the absolute numbers as close and the relative differences as solid.

The numbers

The real three-tool restaurant server costs 277 tokens per model call. That is the whole standing bill, and it is why break-even at this scale is immediate: there is nothing meaningful to save.

Scale changes the picture:

Tools	Terse	Realistic	Verbose
3	209	362	668
10	596	1,040	1,972
30	1,856	3,233	6,089
100	6,370	11,132	20,814

Two things jump out. The growth is linear in tool count, roughly 64 tokens per tool terse, 111 realistic, 208 verbose; there is no cliff, just a slope. And description style is its own multiplier: at every size, verbose costs about 1.9 times realistic and 3.3 times terse.

The slope compounds per task, not per session. A hundred realistic tools cost 11,132 tokens on every model call, so a 20-call agent task pays roughly 223,000 tokens for definitions alone, before a word of conversation or a byte of tool output. The curated comparison the commenter asked about: five relevant tools picked out of that hundred cost 542 tokens per call, a 95 percent saving.

So where is the break-even?

For a server like the sample, there is none to find; 277 tokens is noise. My reading of the curve: below roughly ten tools, do nothing. Around thirty realistic tools you are paying 3,200 tokens per call and about 65,000 per task, which is real money at fleet scale but rarely worth architectural surgery. At a hundred tools you must do something, and the interesting part is that a curated manifest is only one of four options, and the crudest one.

A curated manifest is static tool selection: you save 95 percent of the context by deciding at design time which five tools matter, which is precisely the design-time knowledge MCP exists to avoid assuming. The alternatives keep discovery and cut the bill differently. Split the server by domain, so an agent connects to the invoice server or the shipment server rather than the everything server; this is the maître d' post's fleet argument wearing a cost hat. Use a client that defers tool loading and searches definitions on demand, which more agent runtimes now do. Or design coarser, intent-sized tools so a hundred resource-level operations become fifteen task-level ones, which a commenter on the first post predicted and the numbers now justify.

The tension with post one

The first post argued that tool descriptions are load-bearing: vague text makes vague agents. This post seems to argue that words cost money. Both are true, and the numbers say where the balance sits. Realistic descriptions, around 25 words of purpose and parameter guidance per tool, cost 111 tokens each; that is the price of an agent that picks the right tool and recovers from errors, and it is worth paying. The verbose tier doubles the bill mostly with prose the model would infer anyway. So the budget rule I take away: spend description tokens on disambiguation between tools and on where identifiers come from, never on ceremony.

What this does not measure

Selection accuracy. The other half of the commenter's question is whether the model picks the right tool less often at a hundred tools than at ten, and that needs a live model, many runs, and scripted tasks with known correct tool sequences. The harness is structured so that eval slots in next; that experiment, together with the description validation ideas from the comment thread, is its own post. What is measured here is the bill; whether the agent's judgment also degrades with the menu size is the follow-up.

The takeaway

Discovery is not what costs you; standing context is. The bill is linear in tool count, multiplied by description style, and paid on every model call. Below ten tools, ignore it. Above that, the fix is not abandoning discovery for a hand-curated list; it is scoping servers, deferring definitions, and sizing tools to intents, with descriptions that earn their tokens.

The harness, the data, and the chart are in the repo under measure/; run python token_cost.py and argue with the numbers directly.

Same Kitchen, Two Doors, One Maître d'

Steef-Jan Wiggers — Sun, 13 Sep 2026 14:10:21 +0000

In MCP vs API Is the Wrong Question I deployed one Azure Functions app that exposes the same backend twice: a REST door for callers who know the contract, and an MCP door for agents that discover it. The caveats section admitted, out loud, what the demo left open: the REST endpoints were anonymous, and the MCP endpoint was guarded by a system key that every client had to carry.

A reader summarized the gap better than I did: the useful test is whether both doors preserve the same auth, error, and idempotency behavior, because otherwise the integration surface splits even when the backend does not. Auth parity is exactly what the demo lacked. This post closes that gap with Azure API Management, and it turns out APIM can play three different roles in front of this architecture. Sticking with the restaurant analogy: the kitchen cooks, the doors let people in, and APIM is the maître d'. It checks reservations, limits party size, and keeps the guest book, without touching a single pan.

Why the doors need a maître d'

Both doors in the sample share one property: they authenticate the caller mechanically, not organizationally. A function key or an mcp_extension system key says "this caller has the string" and nothing else. There is no identity, no revocation short of rotating the key for everyone, no per-caller limits, and no answer to the question every platform team eventually asks: who called what, how often, and should they be allowed to?

That question is gateway work. And the more actions we let agents perform, the more urgent it becomes, because an agent retries, chains calls, and occasionally misreads a tool description. You want the blast radius of that behavior governed somewhere the agent cannot negotiate with.

Pattern one: APIM in front of the REST door

The familiar pattern, and I will keep it short because it is the subject of my API Management for AI Workloads series. APIM fronts the /api endpoints: validate-jwt against Entra ID or subscription keys for consumers, rate limits and quotas per product, IP filtering where it applies, and the function app locked down so only the gateway reaches it. The REST door's contract does not change; it just stops being naked.

Pattern two: APIM in front of the MCP door

APIM can also expose an existing MCP server as a passthrough, and a Functions-hosted MCP endpoint is an explicitly supported backend. The MCP traffic (streamable HTTP) flows through the gateway, and policies apply to it like any other API.

This changes the security posture in one important way: the agent no longer holds the backend credential. The client authenticates to APIM properly, with OAuth against Entra ID or a subscription key, and a policy injects the x-functions-key toward the backend. Rotating the system key becomes an operation nobody outside the platform team notices. On top of that you get the gateway's usual gifts, now applied to tool calls: throttling per agent identity, request tracing that shows which tools are called at which rate, and since Build 2026 even MCP-aware content safety policies.

In the first post I noted that developer tools happily send a key in a header while end-user MCP clients follow the authorization spec and demand a real OAuth flow. Pattern two is one of the two clean answers to that boundary; the Functions extension's built-in Entra auth is the other. With a gateway in front, the OAuth conversation happens at the maître d', where it belongs.

Pattern three: APIM as the MCP door

The third option surprised me when I first tried it: APIM can expose a REST API as an MCP server by itself. You pick operations from an API already managed in the gateway, and APIM manufactures MCP tools from them. In that design the function app implements only the REST door, the MCP extension disappears from the codebase, and the second door is generated at the gateway.

This is genuinely attractive for one class of backend: the one you cannot change. An enterprise has hundreds of APIs already registered in APIM, owned by teams with no bandwidth for an agent project. Pattern three gives every one of them an MCP door without a single code change, and the same gateway policies govern it from day one.

But the first post argued that tool descriptions are load-bearing: the agent chooses tools based on that text, and vague descriptions produce vague agents. Pattern three moves those descriptions out of the code and into gateway configuration. The team that owns the business logic no longer owns the words the agent reads. In pattern two, the descriptions live in McpToolTrigger attributes, versioned in the same pull request as the logic they describe. That difference sounds small and is not; it decides who fixes the contract when the agent starts picking the wrong tool.

Where each pattern is the wrong answer

Pattern one is never wrong so much as incomplete; it governs only the door that agents do not use.

Pattern two is the wrong answer when there is no code to own. If the backend team cannot or will not adopt the MCP extension, insisting on in-code tools blocks the whole effort. It also costs you nothing to skip if no agent ever calls the system; an MCP door with no visitors is just surface area.

Pattern three has sharper edges. APIM's MCP support is tools-only today: no MCP resources, no prompts. The Functions extension gained resource and prompt triggers at Build 2026, so if your server needs more than tools, the gateway cannot generate it. Auto-generated tools also inherit the granularity of the REST operations, and a fifty-operation API becomes a fifty-tool server, which is more choices than an agent should have to reason about. And every tool call pays the extra hop, which is fine for governance but worth knowing.

The honest default for new systems: pattern two, with descriptions in code and the gateway doing identity, limits, and observability. Pattern three for the estate you already have. Pattern one always, underneath both.

What deploying it taught me

Three things the documentation did not. The mcpProperties.endpoints contract wants an object keyed by endpoint name, while both the published examples and the Bicep type definitions say array; the deployment error is the only place the truth lives, which is what a preview API version means in practice. The client-facing URL is the API path plus an /mcp suffix, a small trap everyone will hit exactly once. And when a spec-compliant MCP client receives a 401 from the gateway, it does not ask for a header; it starts OAuth dynamic client registration, which APIM answers with a 404. Two different clients have now walked into that same wall from different directions in this series, and it is the strongest argument yet for doing the OAuth setup properly.

The question that remains

The first post ended by replacing "MCP or API?" with a better question: where do I want discovery, and where do I want a hard contract? APIM adds the operational half: wherever you put discovery, put a gateway in front of it that knows who is discovering. The kitchen stays the kitchen. The doors stay honest. The maître d' remembers every guest.

The companion repo will gain an infra/apim variant with the Bicep and policies for patterns two and three. Until then, the original sample is the substrate all of this builds on.