DEV Community: Amir Pournasserian

How to evaluate an AI router's 'none of these' branch

Amir Pournasserian — Fri, 06 Nov 2026 21:56:18 +0000

An AI router has to recognize requests outside its supported categories while still accepting useful work. Those are two different things to measure.

Between 23 and 28 October 2026, I compared four decision models and five small language models. This article focuses on one task from that comparison: choosing a domain for a request, including an explicit "out of scope" option.

The decision models received typed questions and returned probabilities over the options. The language models received the same questions and guidance as text, with their answers constrained to a JSON schema. No models were fine-tuned.

Give the router somewhere to decline

I used 1,000 sampled requests from the CLINC150 test split, mapped to ten domains, with an eleventh option for requests outside those domains.

Of those requests:

190 were genuinely out of scope.
810 belonged to a supported domain.

Every model received the same guidance for the out-of-scope option. The resulting behavior varied substantially.

Here are three examples from the nine-model comparison:

Model	Out-of-scope requests correctly declined, of 190	Out-of-scope requests missed, of 190	In-scope requests wrongly declined, of 810
Mammoth-1	152	38	100
Kev-4B	136	54	239
GPT-5.4 nano	46	144	18

The false-alarm counts come from subtracting correct rejections from total out-of-scope answers. Mammoth-1 selected that option 252 times: 152 correct rejections and 100 false alarms.

Kev-4B selected it 375 times, yet caught fewer genuinely out-of-scope requests. GPT-5.4 nano rejected only 18 valid requests, but missed 144 requests it should have declined.

A rejection count alone hides both kinds of error.

Score the branch separately

For a router with an out-of-scope option, I would report three measures alongside overall accuracy:

Out-of-scope recall: correctly declined out-of-scope requests divided by all truly out-of-scope requests.
False-rejection rate: in-scope requests wrongly declined divided by all truly in-scope requests.
In-scope routing accuracy: requests assigned to their correct domain divided by all truly in-scope requests, including those wrongly declined.

In this sample, the first denominator is 190 and the other two are 810.

For Mammoth-1, out-of-scope recall is 152/190, or 80%. Its false-rejection rate is 100/810. Its reported in-scope routing accuracy is 83.7%.

Keeping domain accuracy separate matters. A router can accept a valid request and still send it to the wrong destination.

I would also track malformed answers and request failures separately. In my benchmark, format failures counted as wrong in overall accuracy; they were not dropped to make the successful responses look better.

Apply the test to your own router

I would start by writing down exactly which requests each destination supports. Then I would build a labeled evaluation set containing both supported requests and plausible requests outside that scope.

For an initial comparison, keep the requests, category definitions and option guidance consistent across models. Record the model versions and settings so that a later run has a useful reference point.

After scoring, inspect examples from each error category. A false rejection may point to narrow option guidance. A missed out-of-scope request may expose an unclear boundary between supported and unsupported work. These are possibilities to investigate in the examples.

The choice depends on what those mistakes cost in the application. Sending unsupported work into a workflow and declining a valid customer request can have very different consequences.

If I added a confidence threshold, I would evaluate it on separate validation data and check calibration. The counts above describe the options the models selected; they do not establish an optimal threshold.

Inspect the experiment

The public repository contains the harness, task definitions, raw requests and responses, and reports.

Start with the saved results and reports to examine the existing experiment. The repository's README explains how to run it again.

These results describe the tested model versions on public data in October 2026. Model-specific prompting or fine-tuning could change the outcome. All nine models ran this CLINC task, and a production router still needs evaluation on its own traffic.

The full comparison on my site also covers calibration, format failures, latency, cost and repeatability across the other tasks.

Decision models against small LLMs: Jev, Kev, Laya and Mammoth-1 on 6,662 decisions

Amir Pournasserian — Thu, 05 Nov 2026 23:24:59 +0000

Four decision models and five small LLMs on 6,662 decisions: accuracy, calibration, format failures, latency and cost, every call published.

Decision models against small LLMs, measured | Amir Pournasserian

Four decision models and five small LLMs on the same 6,662 decisions: accuracy, calibration, format failures, latency and cost, every call published.

pournasserian.com

Validating a vector database built for RAG: four layers of checks, with the code to run them

Amir Pournasserian — Thu, 05 Nov 2026 23:23:01 +0000

How to check a RAG vector index on its own, in four layers from the chunks to the answer, with runnable Python and 25 evaluation tools compared.

Validating a vector database built for RAG: four layers of checks

How to check a RAG vector index on its own, in four layers from the chunks to the answer, with runnable Python and 25 evaluation tools compared.

pournasserian.com

I tested four decision models against five small LLMs on 6,662 decisions. The strongest model changed by task, and the full harness and raw calls are public. https://pournasserian.com/writing/decision-models-vs-small-llms-2026

Amir Pournasserian — Thu, 05 Nov 2026 23:20:31 +0000

Decision models against small LLMs, measured | Amir Pournasserian

Four decision models and five small LLMs on the same 6,662 decisions: accuracy, calibration, format failures, latency and cost, every call published.

pournasserian.com

Buying an MCP server? Ask these 12 questions first

Amir Pournasserian — Sat, 24 Oct 2026 12:49:20 +0000

A vendor demo can make an MCP server look great.

The tools work, the assistant gets useful answers, and everything feels effortless.

But what happens when a token expires, a user cancels a long-running request, or your company needs evidence that the server is actually secure?

That's the part worth investigating before signing a contract. 🙂

I published a practical guide to evaluating MCP server vendors, based on the 2026-07-28 Model Context Protocol specification.

What's in the checklist?

1. Protocol compliance

Does the server implement discovery correctly? Can any instance handle a request without relying on connection-specific session state? Are caching and transport requirements respected?

2. Security

How are tokens validated? Does the server check the intended audience? Are inputs validated, tool calls rate-limited, and outputs sanitized?

3. Production operations

What happens when a client disconnects or cancels a request? Can the vendor show useful metrics, tracing, and operational logs?

4. Evidence, not promises

Does the vendor run the official conformance suite? Does it test against both older and newer protocol implementations? Has it been tested across multiple AI hosts, including a text-only client?

The useful part: 12 questions you can actually send

The article ends with a one-page checklist designed for buyers, CTOs, security reviewers, and engineering teams.

It distinguishes requirements stated by the specification from my own recommendations, so you're not accidentally turning a good practice into a supposed protocol MUST.

The goal isn't to make procurement harder. It's to make the important questions easier to ask before a production incident asks them for you. 😅

👉 Read the full article: https://pournasserian.com/writing/mcp-buyer-vendor-requirements

This is part one of a series on MCP for buyers and CTOs. I'd love to hear what you'd add to the checklist, especially if you've already evaluated MCP vendors for production use.

MCP subscriptions/listen: change notifications without the surprises

Amir Pournasserian — Fri, 23 Oct 2026 16:05:26 +0000

Opening a stream is easy. Keeping it useful when connections drop and your server runs across multiple instances? That's where things get interesting. 😅

I just published a practical walkthrough of subscriptions/listen in the 2026 Model Context Protocol (MCP) specification.

What you'll learn

How subscriptions work: Open a long-lived stream and request the notification types you care about.

Why acknowledgements matter: The server acknowledges the subscription before sending notifications.

What notifications actually tell you: They're signals that something changed, not the updated resource itself.

How reconnects work: Missed events aren't replayed, so clients need to subscribe again and refresh their state.

What changes when you scale out: Multiple server instances need a shared pub/sub mechanism to keep their subscribers informed.

A little Python, too

The article includes examples using the MCP Python SDK (mcp 2.3.0), including how to notify clients when a resource changes and how to think about implementing a subscription bus across multiple server instances.

I also cover practical details like stream shutdowns, idle timeouts, authentication expiry, and the difference between a clean close and an unexpected disconnect.

The big takeaway

Don't treat a subscription as a durable event log.

If a connection drops, assume you may have missed changes. Reconnect, subscribe again, and refresh whatever state you cached.

And if you're running multiple instances, remember that a notification generated on one instance won't automatically reach a stream connected to another.

Distributed systems: finding new ways to remind us that "it works on my machine" is not an architecture. 🙂

👉 Read the full article: https://pournasserian.com/writing/mcp-2026-subscriptions-listen

If you're building MCP servers, I'd love to hear how you're handling subscription fan-out and reconnect recovery in your own setups.

MCP Elicitation: How AI Agents Ask for Human Input 🤖

Amir Pournasserian — Wed, 21 Oct 2026 15:30:18 +0000

Sometimes an AI agent can't finish a task without asking a person something.

Maybe it needs a deployment target. Maybe the user needs to sign in to another service.

MCP elicitation provides two ways to handle this:

📝 Form mode: collect structured input, such as choices or missing values.

🔗 URL mode: send the user to a browser for sign-in, consent, or other sensitive flows.

In my latest article, I explore how elicitation works under the 2026-07-28 MCP specification, including stateless retries, requestState, capability checks, and a working Python example using mcp 2.3.0.

One takeaway: design for the case where nobody is there to answer. Not every workflow can avoid human input, but unattended agents need a sensible fallback whenever possible.

👉 Read the full article

How are you handling human-in-the-loop steps in your MCP servers?

MCP #AI #Python #AIAgents

51 Open-Source AI Chat Apps Compared: Which One Fits Your Workflow?

Amir Pournasserian — Mon, 19 Oct 2026 16:12:04 +0000

Looking for a ChatGPT-style experience without being locked into one provider? The open-source ecosystem has plenty of options — maybe too many. 😅

I mapped 51 apps and compared 12 across 64 selection criteria, including:
🧠 Coding, research, and document chat
🔒 Privacy, telemetry, and offline use
🖥️ Desktop, mobile, and self-hosted deployment
🛠️ MCP support and AI agents
👥 Team features and administration
📜 Licensing and project activity

The results highlight some useful distinctions. Open WebUI is a strong all-rounder for self-hosted use, DeepChat stands out among installable desktop apps, and Jan and PocketPal AI are worth a look for private, offline workflows.

But stars aren't quality scores, and feature counts aren't universal verdicts. The article explains the methodology and its limitations, with downloadable spreadsheets so you can apply your own priorities.

👉 Read the full comparison

AI #OpenSource #SelfHosting #WebDev

Is Your Website Ready for AI Agents? Here's a Checklist 🤖

Amir Pournasserian — Sun, 18 Oct 2026 17:27:49 +0000

Making a website AI-friendly is more than adding an llms.txt file and hoping for the best.

I put together 8 practical steps, each with a test you can run yourself, covering:

Serving content directly in HTML
Configuring robots.txt intentionally
Keeping sitemaps and feeds fresh
Publishing Markdown versions of pages
Supporting coding agents with llms.txt
Making your site discoverable through MCP
Measuring verified AI crawler traffic

The checklist is based on real crawler logs from my site, along with current standards and documentation.

The goal: less guessing, more evidence.

👉 Read the full checklist

Question: What have you found actually helps AI crawlers discover and understand your website?

I made my website answer ChatGPT and Claude, in .NET

Amir Pournasserian — Sat, 17 Oct 2026 12:59:52 +0000

People now ask an assistant before they open a search box. When someone asks ChatGPT or Claude about something I wrote, the answer comes from whatever the model remembers, or from a web search that may or may not land on my pages. I wanted the assistant to be able to ask my site directly and quote the page it used.

The Model Context Protocol (MCP) is the standard way to do that. A site exposes a server with a few tools, and any MCP client can call them. So I built mcp-my-site: a command-line indexer that crawls a website, and a small ASP.NET Core app that serves the result as a remote MCP server. It is MIT licensed and aimed at small sites, under 1,000 pages.

You can try it before reading further. A live instance indexes my own site:

claude mcp add --transport http pournasserian https://ai.pournasserian.com/mcp

There is also a playground, which is an MCP client in the browser, so you can call the tools without installing anything.

The 60-second version

You need the .NET 10 SDK and, for this first run, nothing else. No API key.

dotnet tool install --global mcpmysite
mcpmysite init https://example.com --keyword-only   # writes appsettings.json
mcpmysite build                                     # crawls the site into ./index
mcpmysite search "how can I contact you"            # checks the index
docker run --rm -p 5000:8080 -v "$PWD/index:/index:ro" -e Index__Path=/index ghcr.io/pournasserian/mcp-my-site:latest

The server is then at http://localhost:5000. /mcp is the endpoint for agents, /playground lets you try it in a browser, /api/search?q=contact returns JSON, and /healthz shows the loaded index. Connect Claude Code with:

claude mcp add --transport http mysite http://localhost:5000/mcp

Without Docker you can run the server from a clone of the repository.

Design choices

Hybrid search, with a fallback. Keyword search matches words. Vector search matches meaning, so a question phrased differently from the page still finds it. The server runs BM25 and vector search over embeddings and fuses the two rankings per query. If the embedding provider is unavailable at query time, it falls back to keyword search and says so in the response. The live demo runs hybrid.

It works without a key. I did not want the first run to start with signing up for something. --keyword-only builds an index with no vectors, contacts no embeddings endpoint and reads no key. You give up matching by meaning: "how do I reach you" will not find a page that only says "contact". When you want that, change one setting, set AI_API_KEY, and rebuild. The embeddings endpoint can be OpenRouter, which is the default, or any OpenAI-compatible one: OpenAI, Azure OpenAI, Gemini, Ollama, a LiteLLM proxy. The key is read from the environment and never written to a file. With the default model, a small site embeds for a fraction of a cent.

Shaped for the ChatGPT connector contract. The server speaks MCP over Streamable HTTP and exposes two tools, search and fetch. They follow the shape ChatGPT connectors expect: every search result has an id, and fetch takes that id and returns the page as text. Other clients can pass a url to fetch instead. The server also sends instructions generated from the site itself, so the model knows what the site is about and when to use the tools. It works with Claude Code, Claude Desktop, VS Code, Cursor and the MCP Inspector.

One site per server. There is no multi-tenancy and no admin UI. Configuration is a file, appsettings.json. The indexer extracts the main content of each page automatically, so there are no CSS selectors to maintain. The server polls the index source and swaps in a new build without a restart, so updating means building in CI and publishing the index folder to a URL the server polls.

A search API for your own site. The same search code backs GET /api/search, with CORS, rate limiting and an OpenAPI document. The user guide has a drop-in snippet for a search box.

What it does not do

It indexes plain HTML pages reachable over HTTP, or a folder of built HTML. JavaScript-rendered sites, PDFs and pages behind a login are out of scope. A build is always a full rebuild, though unchanged chunks are not re-embedded.

Feedback I would like

Three things would help most. First, whether the quick start worked for you as written, and where it did not. Second, how it behaves on your site: pages that were extracted badly, or questions that returned the wrong page. The indexer writes a report.json that lists low-text pages, and attaching that to an issue helps. Third, which MCP clients you tried it with, especially ChatGPT connectors and anything I have not listed.

Questions and ideas go to Discussions, bugs to Issues.

DEV Community: Amir Pournasserian

How to evaluate an AI router's 'none of these' branch

Give the router somewhere to decline

Score the branch separately

Apply the test to your own router

Inspect the experiment

Decision models against small LLMs: Jev, Kev, Laya and Mammoth-1 on 6,662 decisions

Decision models against small LLMs, measured | Amir Pournasserian

Validating a vector database built for RAG: four layers of checks, with the code to run them

Validating a vector database built for RAG: four layers of checks

I tested four decision models against five small LLMs on 6,662 decisions. The strongest model changed by task, and the full harness and raw calls are public. https://pournasserian.com/writing/decision-models-vs-small-llms-2026

Decision models against small LLMs, measured | Amir Pournasserian

Buying an MCP server? Ask these 12 questions first

What's in the checklist?

1. Protocol compliance

2. Security

3. Production operations

4. Evidence, not promises

The useful part: 12 questions you can actually send

MCP subscriptions/listen: change notifications without the surprises

MCP Elicitation: How AI Agents Ask for Human Input 🤖

MCP #AI #Python #AIAgents

51 Open-Source AI Chat Apps Compared: Which One Fits Your Workflow?

AI #OpenSource #SelfHosting #WebDev

Is Your Website Ready for AI Agents? Here's a Checklist 🤖

I made my website answer ChatGPT and Claude, in .NET

The 60-second version

Design choices

What it does not do

Links

Feedback I would like