DEV Community: Suresh B

How to Build an MCP Server for Internal Tools: Architecture, Authentication and Error Handling

Suresh B — Fri, 25 Sep 2026 07:25:58 +0000

Most MCP tutorials stop at a local server that prints the weather. That is a fine way to see the protocol work, and a bad model for anything that touches your company's data.

An internal tools server is different. It runs somewhere other than your laptop, it answers to more than one user, and a mistake in it exposes real records. The protocol changed in 2026 to make that kind of server easier to build, and most of what you will find online still describes the old one.

This walks through the shape that holds up: what the server owns, how a model calls it, how you decide who is allowed to do what, and how errors should read to something that cannot ask a follow-up question.

Everything here runs on the MCP Python SDK 2.x against the 2026-07-28 protocol.

What the server owns, and what it does not

The most useful rule I know for MCP servers: the server is an adapter, not a place to put business logic.

Your internal API already knows how to look up an order, apply a discount or close a ticket. It already has the permission rules. The MCP server's job is to describe a few of those capabilities in a form a model can use, pass the call through, and translate the answer back.

That means:

One tool per business capability, not one tool per endpoint. search_orders is a capability. GET /v2/orders?filter= is an endpoint.
No new permission model. Decide authorization from the validated token on each call, using the rules your internal service already has.
No hidden state. If a piece of work spans two calls, the second call should carry everything it needs.

A server built this way stays boring, and boring is the goal. It also stays auditable: every tool maps to one capability, and every call carries an identity.

Architecture

Three roles, and they are easy to mix up:

The host is the application the user talks to, and the thing that owns the model.
The client lives inside the host and speaks MCP to exactly one server.
The server is what you are building.

For transport you have two realistic choices. stdio is for a server running as a local subprocess on the same machine as the host: no network, no authentication, ideal while developing. Streamable HTTP is what you want for anything shared, because it is an ordinary HTTP endpoint your platform team already knows how to run.

That last point is worth dwelling on, because it changed in 2026. Protocol sessions are gone from Streamable HTTP. There is no initialize handshake and no session id, which means every request is self-describing and any request can land on any instance. The full list of what the 2026-07-28 revision changed is worth reading before you port an older server. Your server sits behind a normal load balancer with no sticky routing, scales horizontally, and needs no shared session store.

If your server does need state across calls, make it explicit: return a handle from the first tool and accept it as an argument on the next. A handle is just a string your application understands, with an expiry and an owner. It survives a request landing on a different instance, which a session never did.

A server that does something real

Install the SDK:

pip install "mcp[cli]"

Here is a server with two tools over a pretend internal orders service. The structure is what matters; substitute your own API calls.

from typing import Annotated

from pydantic import BaseModel, Field
from mcp.server import MCPServer
from mcp.server.mcpserver.exceptions import ToolError

mcp = MCPServer("internal-tools")


class Order(BaseModel):
    order_id: str = Field(description="Internal order identifier.")
    status: str = Field(description="Current fulfilment status.")
    total_cents: int = Field(description="Order total in cents.")


@mcp.tool()
def get_order(
    order_id: Annotated[str, Field(description="Order id, for example A-1001.")],
) -> Order:
    """Look up a single order by its identifier."""
    record = internal_api.fetch_order(order_id)
    if record is None:
        raise ToolError(
            f"No order {order_id!r}. Order ids look like A-1001. "
            "Use search_orders if you only know the customer."
        )
    return Order(**record)


@mcp.tool()
def search_orders(
    customer: Annotated[str, Field(description="Customer account name.")],
    limit: Annotated[int, Field(description="Maximum rows to return.", ge=1, le=50)] = 10,
) -> list[Order]:
    """Find orders belonging to one customer account."""
    rows = internal_api.search(customer=customer, limit=limit)
    if not rows:
        raise ToolError(f"No orders for customer {customer!r}.")
    return [Order(**r) for r in rows]

Four things in that snippet do more work than they appear to.

The type hints are the input schema. The SDK builds it from your signature, so limit: int with ge=1, le=50 becomes a constraint the SDK enforces before your function body runs. A model asking for 999 rows gets a validation error back and never reaches your internal API.

The docstring is the description. This is the text the model reads when deciding whether to call your tool. It is documentation for a reader who cannot ask you what you meant, so write it for that reader.

The return type is the output schema. Declaring -> Order makes the SDK publish an output_schema alongside the input schema in tools/list, before the tool is ever called. The host gets machine-readable data rather than a sentence it has to parse.

Annotated[..., Field(description=...)] puts a description on individual arguments. Worth doing for anything whose meaning is not obvious from the name.

One detail to be aware of when you return a list: the structured result is wrapped in a result key, because a bare list is not a JSON object. search_orders above comes back as {"result": [{"order_id": "A-1001", ...}]}. Harmless once you know, confusing when you do not.

Authentication

For a shared server, this is the part worth slowing down for.

The model is not your user. The host is not your user. A token arrives with each request, and everything you decide flows from validating it yourself.

The SDK has no opinion about what a valid token looks like, which is correct: yours probably comes from your own identity provider. You implement one method.

from pydantic import AnyHttpUrl
from mcp.server import MCPServer
from mcp.server.auth.provider import AccessToken, TokenVerifier
from mcp.server.auth.settings import AuthSettings

RESOURCE = "https://tools.internal.example.com/mcp"


class MyTokenVerifier(TokenVerifier):
    async def verify_token(self, token: str) -> AccessToken | None:
        claims = await identity_provider.introspect(token)
        if claims is None:
            return None
        return AccessToken(
            token=token,
            client_id=claims["sub"],
            scopes=claims.get("scopes", []),
            resource=RESOURCE,
        )


mcp = MCPServer(
    "internal-tools",
    token_verifier=MyTokenVerifier(),
    auth=AuthSettings(
        issuer_url=AnyHttpUrl("https://auth.internal.example.com"),
        resource_server_url=AnyHttpUrl(RESOURCE),
        required_scopes=["orders:read"],
        validate_token_resource=True,
    ),
)

Returning None rejects the request. That is the whole contract.

AuthSettings describes your server as a resource server. issuer_url is the authorization server that issues tokens, resource_server_url is this endpoint's public URL, required_scopes must all be present on every token, and validate_token_resource=True refuses any token whose audience is not this server.

Turn that last one on. Without it, a token minted for a different service with the same signature is accepted by yours, which is the confused deputy problem wearing a hat. The SDK will make it the default for resource servers in 3.0; there is no reason to wait.

From those settings, the SDK serves an RFC 9728 Protected Resource Metadata document, which is how a client that has never seen your server discovers where to get a token:

{
  "resource": "https://tools.internal.example.com/mcp",
  "authorization_servers": ["https://auth.internal.example.com/"],
  "scopes_supported": ["orders:read"],
  "bearer_methods_supported": ["header"]
}

Scopes are not enough

Scopes answer "may this caller use this tool". They do not answer "may this caller see this row".

That second question has to be answered inside the tool, per call, from the validated identity:

from mcp.server.auth.middleware.auth_context import get_access_token


@mcp.tool()
def search_orders(customer: str, limit: int = 10) -> list[Order]:
    """Find orders belonging to one customer account."""
    token = get_access_token()
    rows = internal_api.search(
        customer=customer,
        limit=limit,
        acting_as=token.client_id,   # the internal API applies its own rules
    )
    return [Order(**r) for r in rows]

Push the identity down to the service that already owns the permission rules, rather than reimplementing them in the adapter. Filter before you fetch, not after: data the caller may not see should never be loaded into a variable in your process, let alone into a model's context. That is the same constraint that shapes retrieval design in production RAG systems, where the permission filter has to live inside the query rather than in a pass over the results afterwards. It is the kind of authorization aware retrieval work that is far cheaper to design in than to retrofit, and it is worth more than any amount of prompt-level guarding.

Error handling

Errors in an MCP server have an unusual audience: a model, which cannot ask you a clarifying question and will try something else based only on what you wrote.

There are three outcomes, and the difference matters.

Raise ToolError when the tool ran correctly and the answer is a problem the caller could fix. The request succeeds at the protocol level and returns a result with is_error set to true, carrying your message where the model reads it. The structured content is empty, because a failed call has no value to structure.

Raise MCPError when the protocol itself is the problem, not the tool.

Let anything else escape and it is a crash. The model learns only that the call failed; your log gets the traceback. That is the correct behaviour for an unexpected bug, and the wrong outcome for anything you could have anticipated.

The difference between a good and bad tool error is entirely in the message. Compare:

Error executing tool get_order: 404

Error executing tool get_order: No order 'NOPE'. Order ids look like
A-1001. Use search_orders if you only know the customer.

Both are one line. The second tells the model what went wrong, what the right shape looks like, and which tool to try instead. It will often recover on the next turn without the user seeing anything.

Write error messages as instructions to a competent colleague who cannot see your code.

Two more cases worth handling deliberately: an expired handle should say so explicitly rather than returning empty results, and a timeout against a slow internal service should say the work may still be running, because a model that assumes failure will retry an operation that already happened.

Running it

For local development against a host on the same machine:

if __name__ == "__main__":
    mcp.run()                       # stdio

For a shared deployment:

if __name__ == "__main__":
    mcp.run(transport="streamable-http")

The SDK also exposes a Streamable HTTP ASGI application, so you can mount the server inside an existing FastAPI or Starlette service rather than running a separate process. For internal tooling that is often the least disruptive option: the MCP endpoint lives beside the API it wraps, behind the same ingress and the same auth infrastructure.

Test with the Inspector before you point a model at it. mcp dev server.py starts your server with an interactive UI where you can read the generated schemas and call tools by hand. Most schema mistakes are obvious there and invisible once a model is in the loop.

Failure modes worth knowing about

Buffering proxies. Streamed responses depend on the proxy passing events through as they arrive. An nginx-style proxy that buffers will make a working server look dead.
Tool name collisions. A host aggregating several servers may see two search tools. Name yours for the domain, not the verb.
Schema drift. Your tool signature is a published contract. Changing an argument's meaning without changing its name breaks callers silently.
Trusting annotations. Tool annotations from another server are hints, not security. Never let one decide whether an action is safe.
Forwarded tokens. Do not pass the caller's token through to unrelated downstream services. Exchange it for one scoped to the specific thing you are calling.
Origin validation. For HTTP servers, validate the Origin header and bind local servers to localhost, or a web page can reach a server that was only meant to be local.

Readiness checklist

Area	Ready when
Scope	One tool per business capability; no business logic that belongs in the internal API
Schemas	Input constraints on every argument; a declared return type; descriptions written for a model
State	No dependence on a session; cross-call state is an explicit handle with an expiry and an owner
Authentication	Tokens verified on every call; audience validation on; scopes required
Authorization	Row-level decisions made per call from the validated identity, by the service that owns the rules
Errors	`ToolError` for anything the caller can fix, with a message naming the fix; crashes logged, not leaked
Transport	stdio locally, Streamable HTTP when shared; Origin validated; proxies pass events unbuffered
Operations	Rate limits; idempotency on writes; logs that record identity and tool, not payloads

Build the smallest server that exposes one capability, wire the token check, and call it from a real host before adding a second tool. The gap between a demo server and an internal one is almost entirely in the parts that have nothing to do with the model, which is the same gap the production readiness checklist for LLM applications covers for the system around it.

Drafted with AI assistance and reviewed, edited and approved by the author.

Production LLM Application Architecture in 2026: An Engineering Checklist Beyond the Demo

Suresh B — Wed, 23 Sep 2026 10:06:32 +0000

The demo worked. That is the problem.

A demo usually proves one narrow path under friendly conditions. Production exposes the same feature to unfamiliar inputs, real load, dependency failures and changes you did not schedule.

The gap is not only a model problem. Much of it is ordinary application, security and operations engineering.

This is a checklist for closing it. It is organised by how things fail, not by vendor feature, because the vendors keep renaming everything and the failure classes stay put.

Why the demo proves nothing

This checklist groups the main production failure modes into seven classes. A demo usually exercises only a small part of them.

Failure class	The question nobody asked during the demo
Input	What happens when the input is hostile, enormous, or in the wrong language?
Retrieval	What happens when the right document is not found, or the wrong user's document is?
Model	What happens when the provider is slow, down, or retires the model?
Output	What happens when the model returns something your code cannot parse?
Action	What happens when the model does something real and wrong?
Cost	What happens when usage grows ten times?
Operations	What happens when it breaks and nobody knows why?

Work through them and you have a useful production readiness baseline.

The shape that survives contact with users

A useful production architecture can be thought of as six core pieces.

A gateway. One place where requests enter, get authenticated, rate limited and tagged. Without this you cannot answer "who spent that money" or "which tenant caused that spike".

An orchestrator. The thing that decides what happens in what order: classify, retrieve, call, validate, respond. This is your application logic. It belongs in code you own.

Retrieval. Whatever finds the facts. Its most important property is not similarity scoring. It is knowing what the current user is allowed to see.

A model adapter. One internal interface that keeps provider specific API details in a single place. It should make model migrations testable, let you pin a production model or snapshot where the provider supports it, and give the team a documented migration or fallback path.

A second active provider is optional. It can be worth the cost when business continuity requirements justify it, but it also brings a second set of behaviours to evaluate and more surface to keep current. Decide it on requirements rather than by default.

A policy layer. Schema validation, content checks, permission checks, approval gates. Everything that decides whether the model's output gets to become an action.

Telemetry. Traces, costs and evaluation results. If you cannot see it, you cannot fix it.

Notice what is not on that list: a framework. You may use one. You will still need all six.

The flow through them is straightforward:

Client
  |
Gateway .......... authentication, rate limits, request ID
  |
Orchestrator
  |---- authorization scoped retrieval
  |---- model adapter
  |
Output validation and policy
  |
Approval when required
  |
Tools and side effecting actions
  |
Response

Telemetry, cost, security and evaluation metadata wrap the full path.

Read only work such as retrieval and lookups can happen earlier in that path. The ordering that matters is that a high impact action does not execute before the policy and approval gate it requires.

Controls that earn their place

Authorize before you retrieve

One serious class of RAG failure is applying authorization only after retrieval.

The rule worth holding onto: restricted content should not become model context unless the current principal is authorized to access it. Prefer authorization aware or pre retrieval filtering where your datastore and access model support it.

When the retrieval layer can apply the authorization filter efficiently, it can also reduce the candidate set and the amount of search work. Treat tenant isolation as a first class concern rather than a configuration detail, because a leak across tenants is the kind of incident that ends contracts.

Treat external content as data, not instructions

Anything the model reads can try to instruct it. That includes the obvious case of a user typing an attack into a chat box, and the less obvious case of instructions hidden inside a retrieved document, a scraped web page, an uploaded file or a tool response. The second kind is harder to spot, because the text arrives through a channel you trusted.

Design on the assumption that some of it gets through. Enforce authorization and tool permissions in application code rather than trusting a prompt to hold the line. Give each tool the narrowest scope that lets it work. Keep tenant boundaries enforced at the data layer.

Be deliberate about what leaves the system too. Secrets and sensitive data should not end up in prompts, logs or traces without a reason, and captured prompts, documents and outputs need retention and redaction rules that match the product.

Finally, bound consumption. Put limits on tokens, tool calls, agent steps and expensive operations, so a hostile or malformed input cannot turn into an unbounded bill or an infinite loop. OWASP's GenAI security material is the practical reference here, and its current edition is worth reading end to end.

Treat model output as untrusted input

Your code is going to parse this. Validate it against a schema before anything downstream touches it.

OpenAI and Gemini both support schema constrained structured outputs. When the feature is supported and the response completes normally, it can enforce the expected structure far more reliably than prompting for JSON alone. You still need to handle refusals, incomplete responses and application level validation.

Schema compliance is not semantic correctness. The provider can constrain the shape; your code still has to check the values, the permissions and the business rules. A perfectly valid JSON object can still contain a refund amount nobody is authorised to issue.

Put explicit approval in front of high impact actions

Not everything needs approval. Reading a document does not.

Explicit user confirmation or human approval earns its friction when an action is destructive, externally visible, financially significant, privilege changing, legally or compliance sensitive, or difficult to reverse. Reading and retrieval normally need far less.

Design can lower the bar. Staged operations, reversible changes and compensating actions sometimes reduce what needs approving in the first place, which is usually better than adding another confirmation dialog.

Whatever you gate, log it: who or what requested the action, what was approved, the resulting operation, and the final state.

Reliability is mostly boring

Timeouts on every external call. Bounded retries, not infinite ones. A fallback that degrades honestly rather than pretending.

Five failure modes catch people out repeatedly:

Partial streams. The connection drops halfway through a response. Your UI shows half an answer and your database saves it as complete.
Stale caches. You cached an answer that was true last week.
Schema drift. Your code, SDK or provider response evolves and an old parser assumption quietly becomes wrong.
Retry storms. The provider slows down, every client retries, and you complete the outage for them.
Unsafe retries. Retries around side effecting operations need idempotency or state reconciliation. A timeout does not prove the original operation failed, and repeating it can create a second charge, a second booking or a second email.

None of these are AI problems. They are the same distributed systems problems that have always existed, which is good news, because the fixes are well understood.

Cost, latency and the thing you cannot see

Attribute cost per request, per tenant and per feature from day one. Retrofitting this is miserable, and without it your only lever when the bill arrives is panic.

For latency, measure the median and the 95th percentile separately. Averages hide exactly the users who are having the worst time and telling other people about it.

One example from a client system we rebuilt. A document assistant was answering in roughly 10 to 30 seconds depending on the query, across a knowledge base of more than 120 GB and approximately 40,000 PDF documents. The rebuild put dynamic intent detection and routing in front of retrieval, made document retrieval authorization aware, and added caching and optimised context selection. First token response came down to under three seconds in typical interactions. This project illustrates the production AI engineering work JarvisBitz Engineering focuses on.

The transferable lesson is about method rather than numbers: measure latency by stage before blaming the model.

Evaluation is a release gate, not a research project

Start with a small representative set covering common requests, known failure cases, edge cases and adversarial inputs.

Not every task has one known good answer. Depending on the task, use reference answers, expected outcomes, or explicit acceptance criteria. A summarisation feature and a booking agent need different definitions of correct.

Add real production failures to the set as they appear. The bugs your users find are the most valuable test cases you will ever get for free.

Run the relevant evaluations before any change to prompts, retrieval, models, tools or workflow logic that could affect behaviour.

Make every change reversible

Version your prompts. Record the exact model or model snapshot where the provider exposes one. Record the retrieval or index version, and the application configuration that shaped the request.

For risky changes, use feature flags, a staged rollout or a canary deployment, and compare behaviour against your evaluation gates before going wide. Keep a rollback path you have actually tested, not one you assume works.

The goal is easy to state and easy to neglect: when a bad response turns up in a support ticket, you should be able to trace it to the exact model, prompt, retrieval and configuration versions that produced it. Without that, every investigation starts from guesswork.

Make traces answer questions

A useful production trace should be able to identify, where relevant: a request or trace ID, a tenant or principal reference that does not expose unnecessary personal data, the model and version, the prompt or configuration version, the retrieval or index version, tool calls with their results or status, retries, fallbacks, token usage, cost, per stage latency, and the final outcome or evaluation signal.

That is not a licence to log everything. Do not indiscriminately capture raw prompts, retrieved documents or secrets. Apply redaction and retention controls appropriate to the product and the data it handles.

The moving parts are genuinely moving

This year made the case better than any argument could.

OpenAI's Assistants API shut down on 26 August 2026, one year after notice, with the Responses and Conversations APIs as the replacement direction. Its transcription models were deprecated the same day, with removal set for 26 February 2027. A batch of legacy audio and realtime models goes on 20 January 2027. All of those dates are on OpenAI's deprecations page, which is worth a bookmark.

Google made its Interactions API generally available in June 2026 and recommends it for all new projects, with the original generateContent API now considered legacy but still fully supported. Separately, Vertex AI Platform became Gemini Enterprise Agent Platform, taking most of the surrounding product names with it, which Google documents in a name changes table.

OWASP published a new Top 10 for LLM Applications on 3 August 2026, with updated rankings and research drawn from real-world AI security incidents.

Two conclusions follow. Keep a deprecation calendar with real dates and a named owner, because these announcements do not arrive when it is convenient. And keep provider specific code behind your adapter, because "we will migrate later" ages badly when later has a published date attached.

The checklist

Area	Ready when
Input	Length limits, content checks, and a known answer for hostile input
Retrieval	Authorization applied before content becomes context; tenant isolation enforced; a defined behaviour for "nothing found"
Model	Provider specific code isolated; production model or snapshot pinned where possible; migration or fallback path tested; deprecation calendar owned
Output	Schema enforced by the provider and validated again by your code, including values and business rules
Action	Risk based approval for destructive, externally visible, financially significant, privilege changing or hard to reverse actions, with an audit trail
Reliability	Timeouts, bounded retries, honest degradation, idempotency or state reconciliation on writes
Security	Prompt injection has a threat model; tool scopes are least privilege; tenant and data boundaries enforced; sensitive data has retention and redaction rules; token, tool and step consumption bounded
Cost	Attributed per request, tenant and feature; alerts on the trend
Latency	Median and 95th percentile per stage, not just end to end
Evaluation	A representative set that gates changes to prompts, retrieval, models, tools and workflow logic
Release	Prompt, model and configuration versions recorded; evaluations gate changes; risky changes can be staged; rollback tested
Operations	Searchable traces, alerts and SLOs, version metadata, and an owner with a runbook who can diagnose failures

What you can safely leave until later

Not everything needs doing before launch, and pretending otherwise is how features never ship.

Fine tuning can often wait until evaluation evidence shows that prompting, retrieval, model choice or workflow changes are not closing the gap. A multi agent architecture can wait until a concrete need appears and one agent is demonstrably not enough. Self hosting a model can wait until you have a specific reason: cost at scale, data residency, a control requirement, or a capability no provider offers.

For a production feature handling real users, data or actions, each checklist area should have an explicit answer before launch, even if the first answer is deliberately simple. "We cap input length at 4,000 characters and reject the rest" is a real answer. "We have not thought about it" is not.

Take an existing feature and run it through the failure class table before you build anything new. The gaps you find there are usually worth more than the next feature on the roadmap.

Drafted with AI assistance and reviewed, edited and approved by the author.

Production Voice AI Architecture in 2026: How Realtime Conversational Agents Are Built

Suresh B — Tue, 22 Sep 2026 13:52:32 +0000

Every vendor documents its own path and calls it "how to build a voice agent". Read three of them and you have three incompatible mental models, none of which tells you which one you should be using. That is the actual first decision, and it is an architecture decision rather than a vendor one: where does reasoning live, and where do tools run?

A useful way to group current production voice-agent designs is into three shapes. A single realtime model that hears speech, decides what to do and speaks back. A full-duplex conversational model that handles the conversation and delegates the thinking to a backend you choose. And a chained pipeline where speech-to-text, an agent and text-to-speech are separate stages you own. Plenty of real systems are hybrids of these, so treat the grouping as a way to reason about the tradeoffs rather than a closed taxonomy. Each shape is a legitimate production choice, each fails in a different way, and the components you need around them are almost identical.

None of them wins outright. Which one is right depends on where your business logic already lives, which is why the decision belongs to you rather than to whichever vendor page you happened to read first.

Architecture 1: a single realtime model

Speech goes in, speech comes out, and the same model interprets audio, decides what to do, calls your tools and generates the reply. In OpenAI's Agents SDK a common setup pairs a RealtimeAgent with a RealtimeSession: the session connects over WebRTC in a browser or WebSocket from a server and handles audio turns, tools, interruptions and handoffs, while the agent carries tools, handoffs, guardrails and business logic. Underneath that, at the Realtime API itself, the central abstraction is the realtime session and the conversation events flowing through it. Google's Live API is the same shape over a stateful WebSocket connection, either server-to-server (your backend relays audio) or client-to-server (the browser connects directly, which Google notes generally offers better streaming performance because it skips the relay through your backend, but should use ephemeral tokens rather than API keys in production).

What you get for that is one model, one session, the fewest moving parts, and a model that hears tone and hesitation instead of a flattened transcript.

Where it breaks. Long tool calls are the first problem. The conversation is inside the session, so a tool that takes several seconds risks dead air unless the platform can say something while it waits. Both vendors have an answer, and they are different. OpenAI uses preambles, the short spoken updates that tell the caller work is happening, and its prompting guide notes gpt-realtime-2 generates them by default. Gemini handles it with non-blocking function declarations. A tool declared with behavior: NON_BLOCKING runs asynchronously, and the function response tells the model what to do with the result: surface it immediately, wait until the conversation is idle, or keep it silent for later use. On the current generation this is the default rather than an opt-in. Gemini 3.8 Live defaults to asynchronous execution, keeps synchronous BLOCKING available for backwards compatibility, and supports function scheduling. Gemini 3.8 Live Extended Thinking only accepts non-blocking declarations, returns a hard error for synchronous mode, and does not support scheduling configurations at all, which is worth knowing before you design around it. One practical note: Google's current documentation disagrees on the spelling of the immediate scheduling value, using INTERRUPT in the Live tools guide and INTERRUPTED on the 3.8 model and capabilities pages, so use whichever the SDK or schema you deploy actually accepts.

Policy is the next one. Burying complex business rules in the voice prompt makes those rules harder to test, version and audit, and it increases the amount of policy the conversational model has to carry alongside the instructions that keep its speech natural. The problem people notice last is auditability. Intermediate text exists, but what you are reading is the model's own transcription of a decision it has already made, not a text step you can inspect and gate before it becomes speech.

Architecture 2: full-duplex with a delegated backend

OpenAI's GPT-Live is the clearest implementation of this shape. The voice model does one job: hold the conversation, listen while speaking (that is the full-duplex part), and decide when to ask a backend for help. The backend does the reasoning and the tool work, and it is a component you choose. OpenAI calls sending work to the backend delegation and documents two modes:

Responses delegation: an OpenAI-hosted model runs the backend reasoning; GPT-Live supplies conversation context and manages the calls; your application still runs its own function tools.
Client delegation: you connect your own agent, workflow or service, with whatever model and harness you already run, and return results to GPT-Live.

The docs are explicit about the division of labour: keep speaking behaviour in the live model's prompt and business rules in the backend prompt. Your application checks permissions, obtains confirmations, runs the functions that touch your systems and saves task progress.

Two things follow from that, and both need designing rather than assuming. The caller can keep talking while backend work runs, which is the entire point, but it also means an interruption does not automatically cancel backend work. Your application decides whether to finish or cancel it, so a caller who says "no, cancel that" while the backend is mid-refund needs that path wired deliberately. Cancellation and interruption are not the same as undoing a side effect: the application still needs task state, retry reconciliation and explicit confirmation of completed actions. The other is more encouraging: this is a migration path rather than a rebuild. If you already have a text agent with tools and business logic, you can put a voice interface in front of it without moving any of that into a speech prompt.

Where it breaks. You now have two prompts, two failure surfaces and a delegation boundary to observe. The backend can become a major term in the latency budget and should be measured separately.

Architecture 3: a chained pipeline

Speech-to-text, then your agent as ordinary text, then text-to-speech. OpenAI documents it as the path to choose when you want to inspect or transform text between stages; examples of orchestration frameworks include LiveKit Agents and Pipecat, both actively released (livekit-agents 1.8.2 on 15 September 2026, pipecat-ai 1.11.0 on 18 September 2026).

Every stage is inspectable and replaceable: you can store the transcript, run policy checks before the agent answers, call internal systems, and only synthesise speech once the workflow reaches an approved answer. With a clean interface between stages, you can often swap the STT provider without changing the agent's core business logic. Much of an existing text agent, its tools and its policy logic can be reused, though voice still needs its own evaluation and its own handling for turn-taking, interruptions, audible latency and speech. For workflows that require inspectable intermediate text, explicit policy checks or approval before speech is generated, a chained pipeline offers a clearer control surface.

The cost is that you own turn-taking. Knowing when the user has finished speaking is the hardest unglamorous problem in voice, and the frameworks have specialised for it: LiveKit defaults to its turn-detector model for turn completion, which predicts the end of a turn from the meaning of the speech as well as its acoustics, with VAD-only, STT endpointing, realtime-model and manual modes available instead, plus explicit handling for false interruptions, where the framework detects speech and interrupts the agent but the transcription comes back empty. Pipecat splits the problem in two: a turn starts on VAD, or on an arriving transcription when VAD missed quiet speech, with an optional minimum-word strategy on top, and the turn end defaults to its Smart Turn model with a fixed speech timeout as the simpler alternative. You also pay latency at every stage boundary, and the agent reads a transcript rather than hearing the caller.

The components every voice agent needs

Whichever shape you choose, the same list applies. Many production failure modes come from missing one of these components rather than from choosing the wrong high-level architecture.

Component	What it must do	Notes that apply across vendors
Transport	Carry audio and events	WebRTC for browsers and mobile, WebSockets for server-side audio, SIP for telephony. OpenAI documents all three for its voice APIs; Gemini Live is WebSocket, with WebRTC available through partner integrations
Credentials	Never ship a key to a client	The mechanism is per platform. Realtime API: ephemeral client secrets minted server-side via `POST /v1/realtime/client_secrets`, or the unified interface where your server brokers the browser's connection. GPT-Live: the project API key stays on a trusted server that sets up the session. Gemini: ephemeral tokens
Turn detection	Decide when the caller has finished	Server VAD (silence-based) or semantic detection (model decides from the words). OpenAI exposes `server_vad`, tuned with `threshold`, `prefix_padding_ms` and `silence_duration_ms`, and `semantic_vad`, tuned with an optional `eagerness` value of `low`, `medium`, `high` or `auto`; frameworks add turn-detector models and endpointing
Interruption handling	Stop instantly and forget what was not heard	Who does this depends on the transport. On Realtime WebRTC and SIP connections the server owns the output buffer and truncates unplayed audio automatically. On a WebSocket connection the client owns playback, so it must stop it and send `conversation.item.truncate` with an `audio_end_ms`, or the model believes it said things the caller never heard. Note that this trims the audio: OpenAI's docs state the model cannot precisely align transcript to audio, so the transcript is not a reliable record of exactly what was heard
Tool execution	Run work, with permissions	Tools may be local functions, or remote MCP servers the platform calls for you. The Realtime API's MCP tool takes a `server_label` plus either `server_url` for a remote server or `tunnel_id` for a local one reached through Secure MCP Tunnel, with optional `allowed_tools` and `require_approval`. `connector_id` is deprecated for models released after 1 September 2026, though existing models keep connector support. Decide deliberately which tools need approval
Action state and idempotency	Prevent duplicate, stale or wrongly confirmed side effects	Track each side-effecting action with its own operation identifier, and each changed request with a task revision, so a later request supersedes an earlier one. Keep authoritative task state in your application. After a timeout or a lost response, reconcile against the backend before retrying, because the action may already have completed. Only tell the caller an action succeeded once the backend has confirmed the state change. Nothing in the protocol or the model gives you exactly-once execution; this is application work
Backend or agent boundary	Hold business rules	Whether it is a delegated backend or the agent stage of a pipeline, business rules belong here, not in the voice prompt
Session state and resumption	Survive a dropped connection	Calls drop. Decide what a reconnect restores and what it must not replay
Transcripts	Produce a record	Needed for evaluation, disputes and improvement. Check what each API exposes and what has to be enabled. GPT-Live exposes input and output transcript delta events in its event stream. On the Realtime API, assistant output audio has transcript events, while input audio transcription must be configured when you need it
Cost accounting	Attribute spend per call	Voice sessions can be billed by duration (GPT-Live is billed per second, and backend model and tool usage is billed separately), so per-call cost is a different calculation from text
Evaluation	Test the conversation and the task	See below

Testing the conversation and the task

OpenAI's evaluation guide makes a distinction worth borrowing: test both the conversation and the completed task. For a booking assistant, listen to the confirmation and check the correct appointment was saved. For GPT-Live specifically, their guide separates task and tool outcomes, conversational timing, speech and language, and session reliability. It also recommends adding complexity in stages: synthetic speech for repeatable single-turn tests, then replayed human recordings, then an independent simulated caller for multi-turn conversations with interruptions and corrections. If you measure latency, change one factor at a time and keep the caller, recording, backend model, prompt, transport, audio cadence and grader fixed when comparing frontend models. Measure acknowledgements like "I'm checking" separately from the answer the caller actually needs.

Choosing

If this is true	Lean toward	Because
You already have a text agent with tools and business rules	Full-duplex with delegation, or a chained pipeline	Both reuse the agent you have built with the least rework
Tool calls are fast and the domain is narrow	Single realtime model	Fewest moving parts in the voice path, and the model hears the caller directly. Tool implementations, permissions and guardrails still live in your code, but conversational policy, tool coordination and realtime-agent behaviour need careful design
Tool calls are slow, or work must continue while the caller talks	Full-duplex with delegation	It is the shape built for backend work overlapping speech; check your platform's cancel semantics
Every decision must be inspectable or approved before it is spoken	Chained pipeline	Text between stages is the audit surface
Telephony is the primary channel	Any architecture, provided the chosen stack has a SIP or telephony gateway	Native transport support varies by provider. OpenAI documents SIP for its voice APIs; Gemini Live itself is WebSocket and reaches telephony through partner integrations. Check this first, because it constrains the rest
You need to switch languages mid-conversation	Verify per model	Language support and whether the language can be set explicitly vary by model family
You need retrieval over private documents	Any, with retrieval in the backend or agent stage	Voice changes the interaction layer, not the retrieval architecture. Intent routing and authorization-aware retrieval still belong behind the voice layer. JarvisBitz Engineering works on this class of production AI systems

Production considerations

Never ship a long-lived API key in browser or mobile code. The replacement differs by vendor. OpenAI mints ephemeral client secrets server-side for untrusted clients (POST /v1/realtime/client_secrets). Google recommends ephemeral tokens rather than standard API keys for production client-to-server Live API applications, and its own documentation marks the ephemeral-token feature as Preview, so recheck its status on the day you publish or ship. Server-to-server architectures are a separate case: there the backend holds the credential and the client never sees it, which is why the server-to-server shape remains the simpler security story.

Reconnects. Design what happens when the connection drops mid-sentence: what the user hears, what the agent believes was said, whether a partially completed task resumes or restarts.

Observability at the boundaries. Record the stages your application can actually see. For a delegated architecture that means delegation receipt, backend request start, first useful result, tool start and end, result submission, audio arrival and client playback. For a pipeline it means the same discipline at each stage. Compare median and 95th percentile across similar calls rather than averages.

Keep the voice prompt about voice. In every architecture, the prompt that controls speaking should control speaking: pace, verbosity, when to acknowledge, when to hand off. Business rules, policy and tool workflows belong in the backend or agent layer where they can be tested as text.

Assume the model list will change. Model identifiers and feature availability have changed quickly in 2026. Pin versions in code, and re-read the model page before a release rather than trusting a blog post, including this one.

When not to build a voice agent

Voice is the right interface less often than the demos suggest.

The task is a form with four fields. A form is faster for the user and cheaper for you.
The answer has to be exact and remembered. An address, a reference number, a dosage: the caller will ask you to repeat it and then want it in writing anyway.
Your knowledge base cannot answer the questions in text. Voice will not fix that. Get retrieval right first, because a voice layer over a system that does not know the answers is just a faster way to disappoint people.
Nobody can staff the escape path. Define an escape path for cases the agent cannot safely or reliably complete. Depending on the product, that may be a human transfer, a callback, a support ticket or another channel. Without one, the agent becomes a wall.
Nobody will review the conversations. Voice systems fail in ways dashboards do not show. Review a representative sample of interactions, subject to the product's consent, privacy and retention requirements, and treat automated evaluation as a complement to that review rather than a replacement for it. Without either, the system drifts and nobody notices.

So the decision comes back to where reasoning and tools live, and all three answers are defensible. Pick the one that matches where your business logic already sits, then go through the component list, because that list is what separates a voice agent that survives contact with real callers from one that only demos well.

If you are building one now, take the component table and mark which components your current design has a real answer for. Whatever you cannot answer for is where to start.

Drafted with AI assistance and reviewed, edited and approved by the author.

MCP in 2026: What Changed in the 2026-07-28 Specification and How to Design Production Integrations

Suresh B — Mon, 21 Sep 2026 12:02:51 +0000

If you learned the Model Context Protocol from a tutorial written before August 2026, most of what it taught about transport is now wrong. There is no initialize handshake. There is no Mcp-Session-Id. There is no HTTP GET stream to listen on, and streams do not resume. The 2026-07-28 revision, released on 28 July 2026, removed all of them and replaced them with something closer to how the rest of the web works: every request is self-describing, so any request can land on any server instance. Protocol version, client identity and capabilities now travel per request in _meta; application state that must survive across tool calls can be represented by explicit handles passed as tool arguments.

This article is a migration-oriented reading of that revision. For each change it states what you can delete, what you must add, and what you have to keep if you still serve clients from the previous era. It ends with an integration decision table and a production checklist. Everything here is drawn from the specification, its changelog, the deprecated-features registry and the extensions documentation as they stand on 25 September 2026; where the SDKs lag the spec, that is stated.

MCP in September 2026

Three dates anchor the current state:

2025-11-25 is the last "legacy" revision: session-based, with an initialize handshake and server-initiated requests.
2026-07-28 is the current revision (release candidate 29 May 2026, final 28 July 2026). The specification's own versioning page calls implementations of this and later revisions modern, implementations of 2025-11-25 and earlier legacy, and anything that supports both dual-era. Those three words are used throughout this article.
28 July 2026 is also when Tier 1 SDKs shipped for the new revision. The TypeScript SDK was split into focused core, client, server and framework packages, launching at 2.0.0 alongside the protocol. Both SDK lines have moved on since. As of 25 September 2026, the main TypeScript core, client, server and node packages have 2.1.0 releases, while framework adapters can carry their own versions. The Python SDK's current stable v2 release is 2.2.0. Check the current release pages rather than pinning to the launch versions quoted in older write-ups, including this one.

The protocol is governed under the Agentic AI Foundation, with a formal SEP (specification enhancement proposal) process and, new in this revision, a feature lifecycle policy with a minimum twelve-month deprecation window. That last item matters for planning: nothing deprecated on 28 July 2026 can be removed before a revision released on or after 28 July 2027.

What 2026-07-28 removed

Sessions and `Mcp-Session-Id`

Protocol-level sessions are gone from the Streamable HTTP transport (SEP-2567). A modern server does not mint session IDs, does not echo them, and ignores an Mcp-Session-Id header if a legacy client sends one; a GET or DELETE to the MCP endpoint from an older client gets 405 Method Not Allowed. The consequence the changelog calls out explicitly: tools/list, resources/list and prompts/list no longer vary per connection. If your server previously returned a different tool set depending on session state, that behaviour has no home in the modern protocol; the tool set is a property of the server (and of the authenticated principal), not of a conversation.

The `initialize` handshake

There is no initialize request and no notifications/initialized (SEP-2575). Instead every request carries its own protocol version and client capabilities in _meta, and the server accepts or rejects each request independently. A version the server does not support gets an UnsupportedProtocolVersionError (code -32022) listing the versions it does support, and the client retries with one of them.

The GET stream, resumability and a few utilities

The standalone SSE stream a client used to open with HTTP GET is gone, replaced by subscriptions/listen (below). SSE resumability and message redelivery (Last-Event-ID, event IDs) are removed from the transport: a broken response stream loses the in-flight request and the client must re-issue it as a new request with a new ID. ping, logging/setLevel and notifications/roots/list_changed are removed too; log level is now requested per call via io.modelcontextprotocol/logLevel in _meta, and a server must not emit notifications/message for a request that did not ask for it.

Server-initiated requests

Servers no longer send their own JSON-RPC requests to clients on any stream. roots/list, sampling/createMessage and elicitation/create still exist as request shapes, but they are now carried inside results (see MRTR below). The notifications/elicitation/complete notification and the elicitationId field, both added only in 2025-11-25, are removed with them.

What replaced it

Per-request `_meta`

Every request now describes itself. The reserved keys are io.modelcontextprotocol/protocolVersion (required), io.modelcontextprotocol/clientCapabilities (required), and io.modelcontextprotocol/clientInfo (a client SHOULD send it). Servers SHOULD identify themselves with io.modelcontextprotocol/serverInfo in each result's _meta. On Streamable HTTP the version is mirrored into the MCP-Protocol-Version header, and the two must match or the server rejects the request with HeaderMismatch (-32020).

A modern tools/call over HTTP looks like this:

POST /mcp HTTP/1.1
Content-Type: application/json
Accept: application/json, text/event-stream
MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: get_weather

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": { "location": "Seattle, WA" },
    "_meta": {
      "io.modelcontextprotocol/protocolVersion": "2026-07-28",
      "io.modelcontextprotocol/clientInfo": { "name": "ExampleClient", "version": "1.0.0" },
      "io.modelcontextprotocol/clientCapabilities": {}
    }
  }
}

There is no prior message. That request is valid as the first thing a client ever sends.

`server/discover`

Servers MUST implement server/discover, which returns supported protocol versions, capabilities and identity. Clients MAY call it up front to choose a version, and on stdio it doubles as the backward-compatibility probe. It is optional for clients: a client is free to send any RPC inline and handle UnsupportedProtocolVersionError if it guessed wrong. The response is cacheable.

{
  "jsonrpc": "2.0",
  "id": "discover-1",
  "result": {
    "resultType": "complete",
    "supportedVersions": ["2026-07-28"],
    "capabilities": { "tools": {}, "resources": {} },
    "_meta": {
      "io.modelcontextprotocol/serverInfo": { "name": "ExampleServer", "version": "1.0.0" }
    }
  }
}

`subscriptions/listen`

A client that wants change notifications sends one subscriptions/listen request, opting in to specific types (toolsListChanged, promptsListChanged, resourcesListChanged, resourceSubscriptions). The response is a long-lived SSE stream that carries only those notifications, each tagged with a subscription ID. Request-scoped notifications (notifications/progress, notifications/message) do not travel on it; they stay on the response stream of the request they belong to. Servers are encouraged to send SSE comment lines as keep-alives on this stream.

Multi Round-Trip Requests and `InputRequiredResult`

This is the change that most affects tool design. When a server needs something from the client mid-call (user input via elicitation, an LLM completion via sampling, or the roots list) it no longer sends a request. It returns an interim result with resultType: "input_required" whose inputRequests map carries the requests it needs fulfilled, plus an opaque requestState blob (SEP-2322). The client gathers the input and retries the original request with a new ID, adding inputResponses and echoing requestState. The server reconstitutes whatever it needs from that blob; the two requests are fully independent and may be served by different instances.

Client                                   Server
  |  POST tools/call (id 1)                 |
  |---------------------------------------->|
  |                                         | needs user input
  |  InputRequiredResult                    |
  |  resultType: "input_required"           |
  |  inputRequests: { github_login: elicitation/create }
  |  requestState: "<opaque>"               |
  |<----------------------------------------|
  |  (client prompts the user)              |
  |  POST tools/call (id 2)                 |
  |  original params + inputResponses       |
  |  + requestState                         |
  |---------------------------------------->|
  |                                         | reconstitutes state, completes
  |  result, resultType: "complete"         |
  |<----------------------------------------|

Every result in the 2026-07-28 protocol carries a required resultType. Modern core results use "complete" or "input_required"; negotiated extensions may define additional values, and the Tasks extension uses resultType: "task" for its CreateTaskResult. Clients MUST treat results from earlier-protocol servers that omit the field as "complete".

Hosting consequences

The transport now mirrors selected body fields into HTTP headers so that intermediaries can route and inspect without parsing JSON. Mcp-Method is required on every POST; Mcp-Name is required on tools/call, resources/read and prompts/get (SEP-2243) (carrying the tool name or resource URI, Base64-wrapped when it is not header-safe). A server MAY annotate a tool parameter with x-mcp-header in its input schema, and conforming clients MUST then mirror that argument into an Mcp-Param-{Name} header; the canonical example is a region parameter that a gateway uses to route the call. Servers that process the body MUST validate that headers and body agree and reject mismatches with 400 and -32020.

Two more changes are aimed squarely at caches and prompt reuse. tools/list, prompts/list, resources/list, resources/read and resources/templates/list results now carry ttlMs (a freshness hint) and cacheScope ("public" or "private", controlling whether a shared intermediary may cache them) (SEP-2549). And servers SHOULD return tools from tools/list in a deterministic order, because a stable tool list is what lets the client's prompt cache hit.

Put together: normal modern MCP requests are stateless HTTP calls. A server can sit behind a round-robin load balancer, be scaled horizontally, and be fronted by a gateway that routes on Mcp-Name or Mcp-Param-Region. Clients and gateways can also cache eligible results using the protocol's own hints: ttlMs expresses freshness, and cacheScope controls whether a result may be shared across authorization contexts. These are MCP-level hints rather than HTTP Cache-Control headers, and they do not mean the server caches anything for you. That statelessness is the design goal the maintainers state in the release post, and it is the reason to migrate even if nothing you had was broken.

One rule to get right the first time: never mark principal-specific or tenant-specific data as public. A public result may be shared by an intermediary across authorization contexts, so anything whose content depends on who asked belongs in private scope. This matters most for resources/read and for any list already filtered by identity or tenant.

One caveat so the picture is complete: subscriptions/listen is still a long-lived SSE stream. If your deployment runs several instances and any of them can change a tool list or a resource, delivering those notifications to a client whose listen stream is held open by a different instance requires shared pub/sub or equivalent infrastructure. Request handling is stateless; notification fan-out is a deployment concern you still have to design.

Stateful work without sessions

Dropping protocol sessions does not force your application to be stateless. It moves state from the transport, where the model could not see it, into tool arguments, where it can. The pattern the maintainers recommend: mint an explicit, server-issued handle from one tool and have the model pass it back to later tools as an ordinary argument.

A worked example. A start_export tool creates a job and returns a handle:

{
  "resultType": "complete",
  "content": [{ "type": "text", "text": "Export started." }],
  "structuredContent": { "export_handle": "exp_9f31c2", "expires_at": "2026-09-21T13:00:00Z" }
}

A later get_export_status call takes export_handle as a parameter. For a horizontally scaled server, prefer keeping the state the handle refers to in shared, durable storage (a database, a job queue, an object store) so that any instance can serve any call. It is possible instead to mirror the handle into a header with x-mcp-header and have a gateway route every call for that export to the instance holding it in memory, but be clear about what that is: application-level affinity, reintroduced by your deployment, even though MCP itself remains sessionless. Handles should carry or imply an expiry, and a call with an expired or unknown handle should return an ordinary tool error the model can act on ("that export has expired, start a new one"), not a transport error.

For interactions that need user input mid-call, use MRTR rather than a handle. Two things about requestState are worth being precise about, because they are easy to get half right.

First, it round-trips through the client, so treat it as untrusted input when it comes back. Integrity protect it and verify it before use, and bind it to the authenticated principal, the originating method, the salient request parameters and an expiry. The official TypeScript helper uses HMAC for this.

Second, signing gives you integrity, not confidentiality. A client can still read a signed but unencrypted payload, so do not put secrets or sensitive internal state in there. If you need the contents hidden, use authenticated encryption such as AEAD, or keep the state server-side and round-trip an opaque reference to it.

One production warning while you are here. The retry carries a fresh JSON-RPC ID, and that does nothing for application side effects. If the first round can create a reservation, a charge or a booking before it returns input_required, the retry can create a second one. Keep the pre-input round free of side effects where you can. Where you cannot, mint an operation identifier, persist the operation state, bind the retry to the same operation, and reconcile instead of repeating. MCP gives you no exactly-once execution; that part is yours.

Compatibility matrix

The specification defines the expected outcome of every client and server pairing. Condensed:

Client	Server	Outcome
Modern	Modern	Works. `server/discover` optional; version mismatch surfaces as `UnsupportedProtocolVersionError` and the client retries with a supported version.
Modern	Legacy	Fails. The legacy server may reject, stay silent, or misinterpret. On stdio, send `server/discover` first so the failure is deterministic.
Dual-era	Modern	Works. The first modern request succeeds or returns a modern error; the client stays modern.
Dual-era	Legacy	Works. The modern request gets a 4xx without a recognised modern error body; the client falls back to `initialize`.
Legacy	Modern	Fails. Missing headers and `_meta` are rejected with `400`. Legacy clients have no fall-forward mechanism.
Legacy	Dual-era	Works. The server answers `initialize` and serves the legacy revision.

The detection rule for a dual-era client on HTTP: attempt a modern request; on 400, inspect the body. A recognised modern JSON-RPC error (UnsupportedProtocolVersion, MissingRequiredClientCapability, HeaderMismatch) means the server is modern, so correct the request or retry with an advertised version. An empty or unrecognised body means legacy, so fall back to initialize. Era is a property of the server, not of a request; cache the result per origin.

A dual-era server chooses its behaviour from how the client opens: a request carrying modern _meta is served statelessly, an initialize request selects legacy semantics scoped to that session. The Python SDK implements exactly this (MCPServer and mcp.Client with a server/discover probe and legacy fallback), and its release notes add a detail worth knowing: idle legacy Streamable HTTP sessions now expire, which does not affect stateless or modern connections.

Do not assume every SDK enables dual-era negotiation the same way. This one catches people. The Python client defaults to mode="auto": it sends a server/discover probe and falls back to initialize against a pre-2026 server, so you get era detection without asking for it. The TypeScript client is the opposite. Its documentation states that an absent mode, or mode: 'legacy', performs the 2025 initialize handshake byte for byte with no probe at all; you get modern negotiation only by passing versionNegotiation: { mode: 'auto' }, or pin a revision with mode: { pin: '2026-07-28' }. Same protocol, opposite defaults. Read your SDK's negotiation documentation before assuming a client is speaking the revision you think it is.

Deprecations and the lifecycle

The revision introduced a formal feature lifecycle (Active, Deprecated, Removed) with a minimum twelve-month deprecation window and a registry of deprecated features (SEP-2596); Roots, Sampling and Logging were deprecated under it (SEP-2577). As of 25 September 2026 the registry lists:

Feature	Deprecated in	Migration	Earliest removal
Roots	2026-07-28	Pass directories or files via tool parameters, resource URIs or server configuration	First revision on or after 2027-07-28
Sampling	2026-07-28	Integrate directly with LLM provider APIs	First revision on or after 2027-07-28
Logging	2026-07-28	`stderr` on stdio; OpenTelemetry for observability	First revision on or after 2027-07-28
Dynamic Client Registration (RFC 7591)	2026-07-28	Client ID Metadata Documents	First revision on or after 2027-07-28
`includeContext: "thisServer"` / `"allServers"`	2025-11-25	Omit or use `"none"`	Follows Sampling
HTTP+SSE transport (2024-11-05)	2025-03-26	Streamable HTTP	Registry wording: "Three months after SEP-2596 reaches Final"; check the deprecated-features registry before removal

Nothing has been removed under the policy yet. Deprecated features keep working during the window, but new implementations should not adopt them. The "earliest removal" column marks when a feature becomes eligible for removal; the registry states that actual removal is a maintainer decision taken during release preparation and may happen later, so treat the registry, not this table, as the source of truth. Note that Roots, Sampling and Elicitation are still delivered through MRTR in the current revision; Roots and Sampling are simply on their way out, while Elicitation is not deprecated.

Authorization hardening

The authorization section is optional for MCP implementations, but where a server implements it the requirements were tightened: OAuth 2.1 (draft 13) with the MCP server acting as a resource server; OAuth 2.0 Protected Resource Metadata (RFC 9728) is now mandatory for authorization server discovery; Client ID Metadata Documents are the preferred registration mechanism, with Dynamic Client Registration deprecated; resource indicators (RFC 8707) bind tokens to the server; authorization servers SHOULD return iss and clients MUST validate it (RFC 9207); client credentials are bound to the issuer that granted them; and a step-up flow lets a server challenge for additional scopes on a per-call basis. Two 2026-07-28 changelog items are easy to miss: the resource-not-found error code moved from -32002 to -32602, and a new error-code allocation policy reserves -32020 to -32099 for the specification (the earlier draft codes -32001, -32003 and -32004 were renumbered to -32020, -32021 and -32022).

Extensions, and an integration decision table

Capabilities now carry an extensions map. Official MCP extensions are maintained in ext- repositories under the Model Context Protocol GitHub organisation, each identified by an io.modelcontextprotocol/ prefix. A repository may contain more than one related extension: ext-auth holds both of the authorization extensions below. Official does not necessarily mean stable either, since extension maturity is tracked separately, so check the extension's current specification before depending on it. In ext-auth today, Enterprise-Managed Authorization is listed as Stable while OAuth Client Credentials is still Draft.

Tasks (io.modelcontextprotocol/tasks), for long-running asynchronous work with task state and polling. Moved out of the core in this revision and redesigned (SEP-2663): polling via tasks/get replaces the blocking tasks/result, tasks/update carries client-to-server input, tasks/list is gone, and a server may return a task handle unsolicited. The Python SDK 2.2.0 release notes state the Tasks extension is not yet implemented there; check your SDK before designing around it.
MCP Apps (io.modelcontextprotocol/ui), for interactive UI surfaces rendered by capable hosts.
OAuth Client Credentials (io.modelcontextprotocol/oauth-client-credentials), for machine-to-machine authentication where no interactive user authorization is available: background services, CI/CD jobs, server-to-server integrations, workers and daemons. It supplements the normal user authorization flow rather than replacing it.
Enterprise-Managed Authorization (io.modelcontextprotocol/enterprise-managed-authorization), for centralised enterprise identity and access policy through an organisation's IdP.
Skills (io.modelcontextprotocol/skills), which serves Agent Skills through MCP resources. SEP-2640 reached Final on 13 September 2026, and the extension adds skills/list and skills/get alongside an optional resources/directory/read.

Worth separating two things that are easy to conflate. Core authorization and authorization extensions are not the same layer: RFC 9728 discovery, resource indicators and issuer validation are core requirements wherever authorization is implemented, while OAuth Client Credentials and Enterprise-Managed Authorization are optional extensions beyond that flow. A server does not need an authorization extension merely to implement normal MCP OAuth authorization.

The extension ecosystem also moves independently of the core protocol. Extensions are versioned independently, their maturity differs, and the client support matrix on the documentation site is maintained by the community rather than generated from implementations. That page says as much itself, and points you at each extension's specification and repository for the latest status. So check the extension's own specification and repository before depending on it, and do not assume every client or SDK implements every official extension.

Which integration shape to choose:

Situation	Recommended shape	Why
Local developer tools, one user, one machine	stdio	No network, no auth, `server/discover` as the probe; stateless semantics still apply
Internal service used by several agents	Streamable HTTP, modern only, behind your normal load balancer	Stateless core; route on `Mcp-Method` / `Mcp-Name`; cache list results with `ttlMs`
Public or partner-facing server	Streamable HTTP, modern only, with the authorization section implemented	RFC 9728 discovery, resource-bound tokens, scope step-up; validate `Origin`
Many servers behind one entry point	Gateway that validates headers against bodies and routes on `Mcp-Name` and `Mcp-Param-*`	This is what the mirrored headers exist for; the gateway must reject header and body mismatches
Long-running work (minutes to hours)	Tasks extension where the SDK supports it; otherwise an explicit handle plus a polling tool	Avoid holding a response stream open across minutes; streams are not resumable
Clients you do not control and cannot upgrade	Dual-era server for the twelve-month window, with a dated plan to drop legacy	Legacy clients cannot fall forward; only the server can bridge

Failure modes

Legacy assumptions in new code. A common migration bug is code that still sends initialize first, or expects an Mcp-Session-Id in the response, and then treats a modern server's 400 as a connection failure. A modern 400 carries a JSON-RPC error body that tells you exactly what to do; read it before retrying.

Buffering proxies. Streamed responses depend on the proxy passing SSE events through as they arrive. The specification recommends servers send X-Accel-Buffering: no; nginx-style proxies otherwise hold events and the stream looks dead. Keep-alive comment lines on subscriptions/listen streams prevent idle timeouts.

Header and body disagreement. Anything that rewrites bodies (a middleware that renames a tool, a proxy that normalises JSON) will produce HeaderMismatch rejections. The mirrored headers are validated against the body on purpose, so intermediaries must rewrite both or neither. Intermediaries that route on headers should also check the MCP-Protocol-Version indicates a revision that requires validation, and reject older traffic rather than trust unvalidated headers.

Stale cached lists. With ttlMs on list results, a client may legitimately serve a cached tool list for the hint's duration. If you add or remove tools, emit notifications/tools/list_changed to subscribers and keep ttlMs short during rollouts. Treat change notifications as the signal to invalidate and refetch, rather than assuming a stale entry disappears on its own because a TTL exists somewhere.

Lost in-flight requests. Streams do not resume. A load balancer that drains connections mid-response will lose the request, and the client must re-issue it with a new ID; make long tools idempotent or hand them a handle so the retry is safe.

Non-deterministic tools/list. A tool list that changes order between calls defeats both client-side caching and the model provider's prompt cache. Sort it.

Production checklist and tradeoffs

Every request carries _meta with protocol version and client capabilities; the HTTP header matches the body.
server/discover is implemented and its result is cacheable.
No modern code path depends on a protocol session; cross-call state on the 2026-07-28 path is represented explicitly, as a handle in tool arguments with an expiry and a model-readable error when stale.
Mid-call input uses MRTR; requestState is integrity protected, verified on return, bound to principal and expiry, and free of secrets; the retry path is tested with a different server instance handling the second request, and any pre-input side effect is reconciled rather than repeated.
tools/list is deterministic and returns ttlMs and cacheScope; changes are announced on subscriptions/listen.
Origin is validated; local servers bind to localhost; the authorization section, where used, follows the 2026-07-28 rules (RFC 9728 discovery, resource indicators, iss validation).
Proxies pass SSE through unbuffered and forward Mcp-* headers untouched.
Roots, Sampling, Logging and Dynamic Client Registration are not adopted in new code; existing uses have a migration ticket dated before July 2027.
If legacy clients still exist, legacy and session behaviour is isolated to the compatibility path and has a written removal date. The legacy wire did not become stateless; it is simply on a clock.
The SDK's support for Tasks, DPoP and the jwt-bearer grant is checked against its release notes before any design depends on them.

If you are migrating today

The short version, in the order it usually gets done:

Delete initialize and notifications/initialized from the modern path.
Stop minting, echoing or expecting Mcp-Session-Id.
Add io.modelcontextprotocol/protocolVersion and clientCapabilities to every request, and mirror the version into the HTTP header.
Implement server/discover and make its result cacheable.
Move anything the server used to ask the client for into MRTR.
Move cross-call state into explicit handles backed by durable storage.
Replace the old GET stream with subscriptions/listen, and plan notification fan-out if you run more than one instance.
Validate mirrored headers against the body, and set ttlMs and cacheScope deliberately.
Isolate legacy compatibility behind one path with a removal date, if you still need it.

If you are writing a new server rather than porting one, the companion piece walks through building an MCP server for internal tools with the Python SDK, including token verification and tool errors a model can act on.

The tradeoffs are real. Stateless requests mean more bytes per call (_meta on every request, capabilities repeated) and no transport-level continuity for free; you pay for that with explicit handles and MRTR retries. Dual-era support doubles your test matrix for a year. And the SDKs are not uniformly ahead of the spec. In exchange, an MCP server becomes something a platform team can host, scale, cache, route and observe with the tools it already has, which is the difference between a demo integration and one you can put a production agent on.

If you are migrating, I would like to hear which 2025-era assumption broke first in your codebase. The next article in this series walks through building a modern MCP server for internal tools end to end: transport, authentication and error handling.

Drafted with AI assistance and reviewed, edited and approved by the author.

RAG Architecture in 2026: A Production Blueprint for Retrieval-Augmented Generation

Suresh B — Mon, 21 Sep 2026 11:07:39 +0000

The notebook worked. Retrieval found the right passages, the model wrote a good answer, and the demo landed. Then the same design met real users and four things happened: first tokens took ten seconds or more, a document the user was never allowed to see turned up in the prompt, a wrong answer could not be traced to retrieval or to generation, and the monthly bill stopped being predictable.

Most of those are not primarily model-selection problems. They are architecture problems, and they are the reason this article treats retrieval-augmented generation as a system with eight layers rather than a pattern with two steps. What follows is a blueprint: the layers, the contract each one owes the next, where authorization has to live, how a first-token budget is worked backwards from a target, and where it is worth buying a layer instead of building it. Where JarvisBitz has verified production evidence, I say so; everywhere else the guidance is engineering reasoning you should test against your own corpus.

Where RAG demos break in production

Most production failures fall into five groups, and each one maps to a layer that the demo skipped.

Stale index. Documents change; the index does not. Users get last quarter's policy with a confident citation. The demo had no ingestion pipeline, only a one-time load.

Leaked documents. The vector store returns the nearest chunks regardless of who is asking. A filter applied after retrieval helps until the top results are all restricted and the user sees an empty answer, or until someone forgets the filter on one code path. The demo had one user.

Unattributable errors. An answer is wrong. Was the right chunk never indexed, never retrieved, retrieved but ranked out, or retrieved and then contradicted by the model? Without per-layer logs there is no way to know, so the fix is guesswork. The demo had no observability.

Latency that compounds. Authorization lookup, embedding, approximate nearest neighbour search, reranking and model time to first token each add their own share of latency, and the demo measured none of them.

Unpredictable spend. Long prompts, no caching, and a retrieval step that returns fifty chunks because fifty was the default. The demo ran twenty queries.

The reference architecture: eight layers and their contracts

 user request
      |
 [1] Ingest        raw documents -> parsed, cleaned, versioned text + metadata
      |
 [2] Index         chunks -> embeddings + lexical index + ACL metadata per chunk
      |
 [3] Query         request -> intent, rewritten query (or no retrieval at all)
     understanding
      |
 [4] Authorized    query + principal -> candidate chunks the user may see
     retrieval
      |
 [5] Ranking       candidates -> ordered, deduplicated shortlist
      |
 [6] Context       shortlist -> token-budgeted prompt with citation payloads
     assembly
      |
 [7] Grounded      prompt -> answer + citations, or an honest "not in the sources"
     generation
      |
 [8] Evaluation    every layer -> logs, metrics, offline evals, alerts
     and observability

A layer contract is the shape of what a layer promises to hand to the next one. Writing them down is the single most useful thing a team can do before scaling, because it turns "the RAG is wrong" into "layer 4 returned no candidates for this principal".

Layer	Owes the next layer	Must record
1 Ingest	Parsed text with structure preserved, a document version, source metadata	Parse status per page, pages dropped, parser used
2 Index	Chunks with stable IDs, embeddings, lexical terms, ACL and tenant metadata	Chunk count per document, embedding model version
3 Query understanding	Intent label, rewritten query or "skip retrieval", confidence	Intent, confidence, rewrite applied
4 Authorized retrieval	Candidates the principal is allowed to see, with scores	Filter applied, candidate count, latency
5 Ranking	Ordered shortlist, duplicates removed	Ranker, top scores, items dropped
6 Context assembly	Prompt within a token budget, one citation payload per passage	Tokens per source, passages included
7 Grounded generation	Answer with citations, or a refusal	Model, cached tokens, time to first token, citations emitted
8 Evaluation and observability	Attribution for any answer, drift alerts	Everything above, joined by request ID

Query understanding and intent routing

The cheapest retrieval is the one you do not run. A production assistant receives greetings, follow-ups that only need the conversation so far, requests for actions, and questions that the corpus cannot answer at all. Sending every one of those through embedding, search and reranking wastes latency and, worse, gives the model irrelevant passages to be confidently wrong with.

Intents that skip retrieval

Define a small set of intents up front and classify each request before anything else runs: informational (retrieve), transactional (route to a tool or workflow), conversational (answer from the dialogue), and out of scope (decline and log). Classification is a short model call or a lightweight classifier over a few example phrasings per intent; it costs far less than the retrieval it prevents. The intent also decides the rest of the path: which corpus, which filters, and whether a person needs to be involved.

When to rewrite

Rewriting the query helps when users write the way people talk and the corpus is written the way documents are written. "Why does my export keep failing since the update" becomes "export failure after version upgrade" and retrieval improves. Rewriting hurts when the query already contains exact identifiers (error codes, part numbers) that a rewrite paraphrases away. A workable rule: rewrite conversational queries, pass identifier-heavy queries through unchanged, and log both forms so you can see which one retrieval actually used. The managed retrieval tools now expose this directly; OpenAI's vector store search accepts rewrite_query=true and returns the rewritten form in the result, which is convenient for exactly this kind of logging.

Authorized retrieval

This is the layer that separates a demo from a system you can put in front of customers, and the rule is short: authorization is a retrieval filter, never only a prompt instruction. A sentence in the system prompt saying "only use documents the user may access" is a request to a model that has already been shown the documents. By then the leak has happened.

Filter before rank

The filter belongs inside the search query, applied by the store before similarity scoring, so that restricted chunks are never candidates. Filtering after retrieval has two failure modes: the result set can be exhausted by restricted items (the user gets nothing, or the system quietly widens the search), and every new code path is a place to forget the filter. The common retrieval stacks discussed in this article all support authorization-aware filtering, though the exact mechanism differs by product: pgvector through ordinary SQL predicates and Postgres row-level security, Qdrant and Weaviate through payload and tenant filters, Elasticsearch through document-level security, and the managed tools through attribute or metadata filters (OpenAI vector stores, Gemini File Search). Metadata filtering in Google's RAG Engine was documented as Preview on 18 September 2026; treat Preview features as things to test, not to assume.

ACL metadata on every chunk

For the filter to work, each chunk must carry what the filter needs. A minimal record:

{
  "chunk_id": "doc_8812:p14:c03",
  "doc_id": "doc_8812",
  "doc_version": "2026-09-04T10:12:00Z",
  "tenant_id": "t_acme",
  "acl": {
    "allow_groups": ["g_service_advisors", "g_admins"],
    "allow_users": [],
    "classification": "internal"
  },
  "source": { "uri": "s3://kb/manuals/2026/brake-service.pdf", "page": 14, "section": "3.2 Torque specifications" },
  "text": "...",
  "embedding_model": "text-embedding-x@2026-06",
  "lexical_terms": ["torque", "caliper", "bracket"]
}

And the retrieval call that uses it, in framework-free pseudocode:

def retrieve(query: str, principal: Principal, k: int = 40) -> list[Chunk]:
    # 1. cache: same principal scope + same normalised query => reuse
    cache_key = hash(principal.tenant_id, sorted(principal.groups), normalise(query))
    if (hit := retrieval_cache.get(cache_key)) is not None:
        return hit

    # 2. authorization filter is part of the query, not a post-step
    acl_filter = {
        "tenant_id": principal.tenant_id,
        "any_of": {"acl.allow_groups": principal.groups, "acl.allow_users": [principal.user_id]},
        "doc_version": {"current": True},
    }
    q_vec = embed(query)
    dense = store.search(vector=q_vec, filter=acl_filter, limit=k)
    sparse = store.lexical_search(terms=tokenize(query), filter=acl_filter, limit=k)

    # 3. fuse, then hand to the ranking layer; nothing unauthorised can reach it
    candidates = reciprocal_rank_fusion(dense, sparse)
    retrieval_cache.set(cache_key, candidates, ttl=seconds(300))
    return candidates

Two details in that snippet matter more than they look. The cache key includes the principal's authorization scope, so a cached result can never be served to someone with narrower access. And the filter includes the document version, so a superseded chunk cannot outrank its replacement.

Group expansion (turning a user into the set of groups they belong to) happens before retrieval, is cached briefly, and is invalidated when memberships change. Test this layer adversarially: seed a canary document that only one group may see and assert, in CI, that every other principal gets zero results for a query that matches it exactly.

Ranking and context assembly

Retrieval returns candidates; ranking decides what the model sees. Hybrid retrieval (dense plus lexical, fused with reciprocal rank fusion) fixes the case where a query contains an exact term the embedding blurs. A cross-encoder reranker over the fused top forty or so lifts precision further. Reranking adds another latency stage, and the cost varies materially with provider, candidate count and model. Measure it on the actual request path rather than assuming a fixed budget. Whether reranking pays off is an evaluation question, not a belief: measure recall at k with and without it on your own query set.

Token budget per source

Context assembly gets a fixed budget and spends it deliberately. A practical scheme allocates a maximum share per source document so that one long manual cannot crowd out three short ones that each contain part of the answer, and orders passages deliberately. Where the highest-ranked evidence should sit is model-dependent, so test passage ordering on the chosen model with your own evaluation set rather than assuming a universal rule.

Deduplication

Overlapping chunks from the same page, near-identical paragraphs across document versions, and the same table captured by two parsers all waste budget and mislead the model into treating repetition as corroboration. Deduplicate on normalised text and on (doc_id, page) before assembly.

Citation payloads

Every passage in the prompt carries a citation payload: chunk ID, document title, page or section, version. The model is instructed to cite by ID, and the application resolves IDs to links after generation. This keeps citations verifiable (the ID either exists in the prompt or it does not) and gives layer 8 the join key it needs to attribute a bad answer to a specific passage.

The first-token latency budget

Users judge a chat interface by when the first token appears. Work the budget backwards from a target and give each layer a share. The figures below are illustrative allocations for a two-second first-token target, not measurements; your numbers come from your own traces.

Step	Illustrative allocation	What removes it
Authorization and group expansion	50 ms	Short-lived principal cache
Intent classification	150 ms	Skip for follow-ups with a cached intent
Query embedding	100 ms	Embedding cache on normalised query
ANN plus lexical search	150 ms	Retrieval cache keyed by principal scope and query
Reranking	150 ms	Rerank fewer candidates; skip when top scores are separated
Context assembly	50 ms	Precomputed citation payloads
Model time to first token	1,000 ms	Prompt caching of the stable prefix; smaller model for simple intents
Network and serialisation	100 ms	Streaming from the first token

Three different caches are doing three different jobs here, and conflating them is a common mistake.

Prefix caching is provided by the model API and removes the cost of re-reading the stable part of the prompt (system instructions, tool definitions, long standing context). It is enabled by default on OpenAI's platform with a minimum cacheable prefix of 1,024 tokens on GPT-5.6 and later, and cached reads are discounted by up to 90 percent; Anthropic offers automatic caching or explicit breakpoints with 5-minute or 1-hour lifetimes; standard cache hits are priced at 0.1x the base input price, with a 0.025x rate on Claude Fable 5.1 and Claude Mythos 5.1. Prefix caching only helps if the prompt is built so that the stable part comes first and the retrieved passages come last.

Retrieval caching stores the candidate set for a normalised query within an authorization scope. It removes embedding and search time on repeated questions, which in support-style traffic are a large share.

Semantic caching stores whole answers for questions judged equivalent. It is the biggest saver and the most dangerous: a wrong equivalence serves a wrong answer, and a cached answer can outlive the document it was built from. If you use it, key it by principal scope and document version, and expire it on ingestion events.

Build or buy, one layer at a time

The decision is rarely all or nothing. In 2026 the practical options per layer are a self-managed search stack, a managed retrieval tool attached to a model API, or, for some corpora, no retrieval at all.

Layer	Self-managed	Managed	Notes
Ingest and parse	Docling, Unstructured, your own parsers	Document AI layout parser and LLM parser in Google's RAG Engine; automatic chunking in OpenAI vector stores and Gemini File Search	Managed parsing is fast to adopt and hard to inspect page by page
Index and search	pgvector for vector search combined with PostgreSQL full-text search and result fusion (RRF or reranking) in your own code; Qdrant, Weaviate and Elasticsearch with built-in hybrid search; all support pre-filtering	OpenAI vector stores (hybrid search weights, attribute filters, up to 50 results), Gemini File Search (chunking config, metadata filters, file citations), RAG Engine (RagManagedDb, Vector Search, Pinecone, Weaviate)	Managed stores tie you to one model provider's request path
Authorization	Your filter, your tests	Attribute or metadata filters you populate	Either way the ACL data is yours to maintain
Ranking	Cross-encoder rerankers, provider rerank APIs	Built-in rankers with limited tuning	Measure before paying for it
Generation	Any model	Any model	Cache the prefix regardless
Evaluation	Open-source eval frameworks plus your own labelled set	Provider tracing dashboards	The labelled query set is the asset; tools are interchangeable

Long context as a layer

The frontier models now offer roughly one million tokens of input: Claude Opus 5, Sonnet 5 and Fable 5.1 list a 1M token window (Haiku 4.5 stays at 200K); OpenAI's gpt-6-astra has a 1,050,000 token window with 922,000 maximum input tokens, and prompts above 272K input tokens are billed at twice the input rate and 1.5 times the output rate for the whole request; Gemini 3.8 Flash accepts 1,048,576 input tokens, with promotional pricing that runs until 31 December 2026. Many 300-page text-heavy manuals can fit within these context windows, depending on extraction, images and tokenization. Whether they should is a different question.

Long context replaces layers 2 through 5 with "send everything", which is attractive for small, static, single-tenant corpora where every user may see every document. It is the wrong tool when the corpus is larger than the window (a 40,000 document library is not going in), when different users may see different documents (there is no filter to apply to a prompt), when the same content is read on every request (you pay the full input each time unless the cache holds), or when you need to attribute an answer to a passage. In practice the strongest 2026 designs use both: retrieval to select an authorized, versioned subset, and a large window to give the model generous, well-ordered context from that subset.

Failure modes and the detectors that catch them

Failure	What the user sees	Detector
Retrieval miss	"I could not find that", but the document exists	Offline recall at k on a labelled query set; alert on recall drop after index rebuilds
Ranking miss	The right chunk was retrieved but not shown	Log positions; measure "gold chunk rank" distribution; compare with and without reranker
Faithfulness failure	Answer contradicts or exceeds the passages	Faithfulness eval on sampled traffic; citation ID validation (every cited ID must exist in the prompt)
Permission leak	A restricted passage appears in a citation	Canary documents; assert zero cross-scope results in CI; audit log per request
Index drift	Answers cite superseded versions	Version field on every chunk; alert when retrieved doc_version is not current
Cache poisoning	A stale or wrong answer keeps being served	Cache keys include scope and version; invalidate on ingestion events; sample cached answers into evals
Cost spike	The bill jumps with no traffic change	Track tokens per request by layer; alert on prompt length percentile and on cache hit rate drops

The common thread: every detector depends on a log line from a specific layer, keyed by the request ID. If layer 8 is bolted on after launch, most of these detectors cannot be built.

Production checklist

Every chunk carries tenant, ACL, version and source metadata; retrieval filters on all four inside the store query.
Intent classification runs before retrieval; at least one intent skips retrieval entirely.
Hybrid retrieval is fused, deduplicated and reranked; recall at k is measured on a labelled set before and after each change.
Context assembly has a token budget per source and emits one citation payload per passage.
The prompt is ordered stable-prefix-first so prefix caching applies; retrieval and semantic caches are keyed by principal scope and document version.
The first-token budget is written down per layer and compared with traces weekly.
Canary documents and cross-scope assertions run in CI.
Every request produces per-layer logs joined by a request ID, and a bad answer can be attributed to a layer within minutes.
Index rebuilds and re-embedding are events that invalidate caches and trigger the recall eval.

The lesson from a production system

JarvisBitz rebuilt the AI assistant inside a US automotive SaaS product whose knowledge base held more than 120 GB of content across roughly 40,000 PDF documents. Before the rebuild, answers could take approximately 10 to 30 seconds to begin, depending on the query, and intent accuracy was poor. The redesign was the architecture above: intent detection and routing before retrieval, authorization-aware document retrieval, caching, targeted retrieval and optimised context construction, built in Python on Google Vertex AI Platform (now Gemini Enterprise Agent Platform) with Gemini. After it, first-token latency in typical interactions was under three seconds, the system achieved approximately 98 percent accuracy across intent recognition and information retrieval in the client's evaluated use cases, and operating cost fell by roughly 30 percent. The full account is in the case study on our site.

What that project taught, and what this blueprint is really about, is that the difficult part of production RAG is rarely the model call. It is the surrounding system: retrieval, permissions, caching, context management, latency controls, verification and monitoring. These are the kinds of production AI engineering problems JarvisBitz Engineering focuses on, and they are the ones that decide whether a RAG system survives contact with real users.

The rest of this series covers each layer in turn. If your open problems are exact-term queries or tenant isolation, start with hybrid search and multi-tenant retrieval.

Drafted with AI assistance and reviewed, edited and approved by the author.

AI voice agent for customer service: what stops callers hanging up?

Suresh B — Mon, 21 Sep 2026 07:42:03 +0000

Every support team that asks us about a voice agent has already heard a demo. The demo is always good. A pleasant voice answers a clean question in a quiet room, and somebody in the room says the word deflection. Then the thing goes live on a real number, and within a week somebody pulls the recordings and finds that a third of callers are pressing zero or simply hanging up.

I have looked at enough of those recordings to be fairly confident about where the damage happens, and it is almost never where the team expects. The voice gets blamed, the accent gets blamed, and speed gets blamed most of all because it is the number on the dashboard. In the recordings, callers leave when the agent has been allowed to start something it cannot finish, and then cannot pass them to a person without losing everything they just said.

The hangup happens at a specific moment

Average handling time will not show you this. Averages smooth over the exact thing you need to see. What you want is the turn number at which the call died, counted across a few hundred real calls, and once you plot that you tend to get two spikes rather than a slope.

The first spike is early, usually turn one or two. That is the caller deciding they have reached a machine and they would rather not. Some of that is unavoidable. Some of it is your greeting promising more than the agent can do, which makes the caller test it immediately with the hardest thing they have.

The second spike is the expensive one. It sits somewhere in the middle of the call, after the caller has explained their problem properly, given an account number, answered two clarifying questions, and then hit a wall. The agent cannot do the thing. It says something polite. The caller has now invested two minutes and is further from a human than when they started. That is the call that turns into a complaint, and it is entirely a design decision rather than a technology limit.

Narrow the job until it is boring

My opinion, and it is an opinion rather than a measurement, is that most voice agent projects fail at scoping and then spend their budget on engineering. The question teams usually ask is what the model can handle, and the answer is a lot, which is why it is the wrong question. The useful one is which calls have a definite end that the agent can reach on its own.

A call has a definite end when there is a system the agent can read from or write to, and a sentence it can say afterwards that closes the matter. Where is my order. Move my appointment. Am I still under warranty. Reset this. Those end.

Calls that do not end are the ones where the resolution lives in somebody's judgement: refunds outside policy, complaints, anything where the caller is upset about a decision rather than confused about a fact. An agent can take the details, and that is genuinely useful, but the moment it implies it might resolve the matter you have built the middle of call hangup on purpose.

A test I use when a team hands me their intent list: for each intent, write the exact final sentence the agent says when it goes well. If you cannot write that sentence without a conditional in it, the intent is not ready for the agent to own.

Build the transfer before the conversation

The handoff is treated as an edge case in most projects I have seen, and it is the single biggest driver of whether callers tolerate the agent at all. A caller who reaches a person in fifteen seconds, with the agent having already told that person the account number and the problem, has had a decent experience even though the automation technically failed. A caller who has to start again from the beginning has had a worse experience than if you had never deployed anything.

So I would build the transfer path first, before a single intent is wired up. Concretely that means three things exist on day one.

A warm transfer that carries state. The agent writes a short structured summary plus the raw transcript into the ticket or the CRM record before the call moves, so the agent picking up is reading rather than asking. If your telephony platform cannot attach data to a transferred call, fix that before anything else, because everything downstream depends on it.
An escape that always works. Saying agent, saying representative, pressing zero, or simply repeating themselves twice. All of these route out immediately, with no retention attempt and no "I can help with that" in between. Retention attempts on a caller who has already asked for a human are where the recordings get genuinely unpleasant.
A rule for what happens out of hours. When there is nobody to transfer to, the agent needs an honest fallback such as taking a callback with a stated window. An agent that keeps trying because the queue is closed is the worst version of this product.

There is a cost argument here too, and it cuts against the usual business case. If you scope narrowly and transfer quickly, your containment rate will look modest. Somebody will be disappointed by that number. I would rather have a modest containment rate on calls that were genuinely finished than a high one that is mostly callers giving up, which is what a containment metric counts if you do not separate the two.

What to actually measure

Support dashboards are built for human queues and the default metrics mislead you here. These are the ones I would put on the wall instead.

Metric	Why it tells you something	What the usual metric hides
Hangups by turn number	Separates people who never wanted a machine from people the agent lost mid call	Average handling time averages the two together
Resolved without transfer, sampled and listened to	Confirms the call actually ended rather than the caller leaving	Containment counts an abandoned call as a success
Time from escape request to a human voice	This is the number angry callers remember	Queue time measured from call start, not from the handoff
Repeat contact within 48 hours on the same issue	Catches confident wrong answers that closed the call falsely	First contact resolution measured on the agent side only

The repeat contact number is the one I would watch most closely in the first month. A voice agent that is wrong and definite will look excellent on every other metric for about two weeks, because the calls are short and they end. The cost arrives later, in the second call and in the trust the caller no longer extends to your support line.

The parts that are genuinely hard

Interruptions

Callers talk over agents constantly, especially when correcting a misheard account number. If the agent keeps talking while being interrupted, the call is effectively over even if nobody hangs up yet. Stopping instantly and discarding the rest of the planned sentence is table stakes, and it is worth testing explicitly rather than assuming your platform does it.

Numbers and names

Order references, postcodes, spelled surnames. Recognition on these degrades in exactly the conditions real support calls happen in, which is a caller in a car or a kitchen. Where you can, verify against a system rather than asking the caller to repeat: a partial match against the account on record is faster and less annoying than a third attempt at the full string.

The greeting

I keep coming back to how much weight sits in the first eight seconds. An agent that names what it can do, in plain words, and mentions that a person is available, gets a noticeably calmer conversation than one that opens with "how can I help you today". The open question invites the whole problem, including the parts the agent cannot take.

How I would start if this were my support line

Pick the two highest volume calls that have a definite end. Build the transfer and the logging first. Run the agent on a share of traffic rather than the whole line, with the old routing one config change away. Listen to twenty calls yourself in week one, which nobody ever wants to do and which tells you more than the dashboard will for a month. Expand the intent list only after the hangup curve stops showing a bump in the middle.

The volume and cost side of the business case depends entirely on your call mix, so I will not put a number on it here. Working that out for a specific line is part of the audit below.

If you have a voice agent live and the recordings are not matching the dashboard, our free AI audit includes listening to a sample of real calls and marking where each one was lost, and how we build voice systems covers the architecture underneath. If you would rather talk it through first, get in touch.

Suresh B is the Founder & CEO of JarvisBitz Tech, an AI and software engineering company helping businesses build and scale production AI systems and digital products.

How to build an AI sales assistant that does not make things up

Suresh B — Mon, 21 Sep 2026 06:47:43 +0000

I have tested a lot of website chatbots over the past two years, and the failure I see most often is confidence. An assistant quotes a price nobody agreed to. It promises a feature that does not exist. It explains a refund policy the business never wrote down. Every one of these answers arrives in the same calm, helpful tone as the correct ones, which is exactly what makes them dangerous.

We have been building a configurable assistant platform at JarvisBitz for a while now, and the question I get asked most about it is how we stop it inventing answers. This article is my answer. It is an engineering answer rather than a product pitch, because the fix lives in the architecture, and it applies whether you build your own assistant or buy one.

Where invented answers actually come from

A language model answers every question it is given. That is what it was trained to do, and it is very good at it. Put it behind a chat window, give it your web pages as context, and it will answer the questions your pages cover. It will also answer the questions your pages do not cover, using whatever it absorbed during training, in the same confident voice.

Retrieval augmented generation, the technique most teams reach for, narrows the problem without removing it. Retrieval finds the passages from your material that look most similar to the question, and the model writes an answer from them. The trouble starts when retrieval comes back thin. The passages are only loosely related, but the model still has to say something, so it fills the gap from general knowledge. The output looks grounded. It has the shape of an answer drawn from your documents, and part of it is fiction.

I have seen this with prices, with features that did not exist, and with policies the business never wrote down. In each case the retrieval step had worked exactly as designed, which is the uncomfortable part. The design itself was the problem.

Rule one: not enough evidence is an outcome, and it needs its own path

The most important decision we made was to treat "we do not have a reliable answer" as a real result with its own branch in the system, instead of something the model should smooth over.

In practice it works like this. After retrieval and before any text is generated, a separate check scores the evidence. How well do the retrieved passages cover the question. Whether they come from approved sources. Whether there are enough of them and whether they agree with each other. If that score falls under a threshold the business has set, the model is never asked to write an answer. The conversation takes a different route: the assistant says plainly that it does not have that information, offers to connect the visitor with a person, and, if the visitor is willing, collects a way to follow up.

It sounds like a small change, and it changes almost everything downstream, because once the model only ever generates from evidence that passed a check, it is no longer in the position of having to guess. The invented price problem mostly disappears on its own.

The threshold is a business decision rather than an engineering one. A law firm wants it strict. A company selling gift boxes can afford to be looser. So we expose it as a setting, and we let the business watch what happens at each level during testing, which I will come back to.

Give "I do not know" its own path through the system. Most of the other rules follow from that one.

Rule two: the assistant may only know what the business gave it

The second rule is about the boundary of knowledge. The assistant answers from approved sources and from nothing else. The website, the product documentation, the policy PDFs, the notes someone in sales wrote down. Whatever the business attaches is the whole world as far as the assistant is concerned.

This is easy to say and surprisingly easy to get wrong. The most common leak I see is a system prompt that tells the model to use its general knowledge if the documents do not cover the question. Someone added it because the assistant felt unhelpful in testing, and it is the single instruction most responsible for confident nonsense in production. We do not allow it. If the business wants the assistant to answer a question, the business adds a source that answers it.

The second leak is subtler. The model is allowed to reason from the documents, and reasoning drifts. Asked whether the product integrates with a tool the documents never mention, a model reading a page that lists eight integrations will sometimes conclude that a ninth is probably supported as well. The evidence check catches most of this, because the passages do not actually contain the claim. Requiring the model to cite the passage behind every factual statement catches most of the rest, since a claim with nothing to cite stands out immediately.

Rule three: business rules belong in code, where a visitor cannot argue with them

Grounding governs what the assistant knows. There is a separate question of what it is allowed to say, and this is where I see a second family of failures that have nothing to do with hallucination.

Whether this assistant may discuss pricing at all. Whether the support assistant may suggest an upgrade, or that belongs to the sales assistant alone. Which topics are off limits. At what point in a conversation it should stop and bring a person in. These are policies. Most teams put them in the system prompt, and I understand why, because it takes two minutes. But a prompt instruction is advice the model can be talked out of by a determined visitor, and visitors are more determined than you expect. A rule checked in code on every turn cannot be talked out of anything.

So in our architecture the persona layer has two parts. Tone, which the prompt handles perfectly well. And policy, which ordinary code checks before and after generation. If the policy says no pricing, the draft answer is checked for pricing and stopped before it reaches the visitor, who gets a polite redirect and, where it makes sense, an offer to talk to someone who can quote.

Rule four: the hard cases go to a person, and that is a feature

An assistant that hands off to a human is often described as one that failed. I think that view is wrong, and it pushes teams into making the assistant answer things it should not.

A visitor who is angry, a question that touches a contract, a request for a discount, a medical or legal detail, a question the knowledge base cannot support: these should reach a person, and the assistant's job is to get them there gracefully and with context. The handoff carries the conversation so far, the intent it detected, and whatever contact details the visitor offered, so the person picks it up without asking anyone to start again.

Designed this way, the handoff is where the highest value conversations end up. It deserves to be built properly rather than treated as the apology screen.

The part most teams skip: learning from the questions you could not answer

Every time the evidence check fails, we record the question. Over a month, that log becomes one of the most useful documents the business owns.

If thirty visitors ask whether you work with a particular accounting system and your site never mentions it, that is thirty pieces of market research delivered for free. It tells the business what to write next, what belongs in the FAQ, what sales keeps hearing on calls, and sometimes what to build. An assistant that had invented a plausible answer would have destroyed that signal. The visitor would have left, satisfied or misled, and nobody would have learned that the question was ever asked.

We surface this as a report: unanswered questions grouped by theme and ranked by how often they came up. I expect it to be the feature businesses value most once they have lived with it, and it only exists because we refused to let the model guess.

Test it before it meets a customer

None of the above is worth much if the business cannot see what the assistant is doing. Our test environment shows the decision behind every reply: which intent matched and with what confidence, which sources were retrieved, whether the evidence threshold passed, which policy rule fired, which lead detail was being collected.

This is how a non technical owner tunes the system. A wrong answer with no trace is a mystery that needs an engineer. A wrong answer with a trace usually shows that a document was missing or a threshold was too loose, and the owner fixes that directly. It is also how they choose the evidence threshold with some confidence, by watching real questions flow through at different settings before anything goes live.

What this means if you are buying rather than building

If you are evaluating an assistant for your website, from us or from anyone else, the rules above turn into a short checklist. Ask the vendor:

What happens when the knowledge base does not contain the answer. Ask to see it, using a question you know is not covered.
Whether you can restrict the assistant to your sources only, with no fallback to general knowledge.
Whether rules like "never discuss pricing" are enforced in code or only written into a prompt.
Whether you can see why the assistant gave a particular answer.
Whether you get a report of the questions it could not answer.

A vendor who cannot answer those clearly is selling you the chat window, and the chat window is the easy part. If you already run an assistant and want a second opinion on it, our free AI audit covers exactly these checks.

Where this leaves us

We describe what we are building as a platform for domain specific assistants, and it is still a prototype, so I am not going to claim results we have not measured. What I will claim is that this architecture is the only approach I have seen hold up once real visitors arrive. The model handles the execution: understanding the question, finding the answer, collecting the details. The business keeps control of what the assistant knows, what it may say, and when it must stop. If you take one thing from this piece, make it the first rule. Give "I do not know" its own path, and most of the rest follows.

Suresh B is the Founder & CEO of JarvisBitz Tech, an AI and software engineering company helping businesses build and scale production AI systems and digital products.

How we took a SaaS AI assistant from 30 second answers to under 3, across 40,000 PDFs

Suresh B — Fri, 18 Sep 2026 13:56:13 +0000

The client is a SaaS company in the US automotive space. I will keep them anonymous, but the shape of the problem will be familiar to anyone who added an AI assistant to a product in the first wave. They had done it early, and they had done it seriously: the assistant was meant to help customers find information, work out what they wanted, respond to it, spot sales opportunities in the conversation, search a very large document library, and eventually talk out loud.

By the time we were involved it did most of those things badly. It behaved like a fixed workflow with a chat window on the front. It often misread what the customer was asking. And depending on the question, an answer could take anywhere from 10 to 30 seconds to appear, which in a chat interface feels like the product has died.

The knowledge base was the reason for the last part. More than 120 GB of material, roughly 40,000 PDF documents, and the original system was searching far too much of it for every single question.

What we found when we looked at the latency

Everyone's first instinct with a slow assistant is to blame the model. In this case the model was maybe a fifth of the problem. Most of the 10 to 30 seconds was spent before any text generation started, and it broke down into four habits that reinforced each other.

Retrieving too much. Every question pulled a wide set of passages from across the library, because nobody had decided what kind of question it was first.
Sending too much. All of those passages went into the prompt. Bigger prompts mean slower generation and higher cost, and they also make the model worse at picking out the passage that actually matters.
Checking permissions late. Which documents a given user was allowed to see was handled after retrieval, which meant retrieving material only to throw it away, and in a few cases meant material that should have been filtered leaking into the context.
Caching nothing. Two people asking the same question an hour apart paid the full cost twice.

The intent problem and the latency problem turned out to be the same problem. The system did not know what the customer wanted, so it retrieved everything, and retrieving everything is slow.

What we rebuilt

We redesigned the system around a custom retrieval augmented generation pipeline with agentic orchestration on top, running on Google Vertex AI with Gemini, in Python, with speech to text and text to speech for the voice path. The important decisions were not the tools. They were the order things happen in.

Layer	What it decides	Why it comes before the model
Intent detection	What kind of request this is, and which path it should take	Everything downstream depends on it: what to retrieve, which rules apply, whether this is a lead
Authorization aware retrieval	Which documents this user is entitled to, applied as a constraint on the search itself	Filtering after retrieval wastes time and can leak; filtering inside retrieval does neither
Context construction	The smallest set of passages that can answer the question well	Smaller prompts are faster, cheaper and more accurate
Caching	What can be reused safely across users and sessions, and what must never be	Repeated questions stop paying the full cost
Agentic execution	What happens next: answer, ask a follow up, route a lead, hand to a person	The assistant responds to the request instead of walking a fixed script

Intent first, then retrieval

The single biggest change was putting intent detection at the front. The business defined the intents that mattered to them, with example phrasings for each, and every incoming message is classified against those by meaning before anything else happens. Once the system knows it is looking at, say, a question about a specific product's specification rather than a general browse, retrieval can be targeted at the right slice of the library instead of the whole thing.

This is also where the lead generation logic lives. Certain intents carry commercial signal. When one of those fires, the assistant can move naturally from answering into asking a qualifying question or two, and route the conversation on. A customer who simply wants a fact gets the fact and is left alone.

Permissions inside the search, not after it

Different users on this platform are entitled to different documents. The original design retrieved first and filtered second, which is the common pattern and the wrong one. We moved authorization into the retrieval step itself, so the search only ever runs over documents the current user may see. It is faster, because there is less to search, and it closes a category of leak that is very hard to test for from an admin account, because admins can see everything and every answer looks fine.

If your retrieval system is tested only by the people who built it, using their own accounts, you have not tested permissions at all. Test it with the narrowest account you have.

Less context, chosen better

With intent known and the candidate set already narrowed by authorization, context construction becomes a question of how little you can send rather than how much. We were fairly aggressive here. The prompt carries the passages that answer the question and not much else. This is where a lot of the latency went, and it is also where a surprising amount of the accuracy came from: a model reading three relevant passages is more reliable than a model reading thirty, most of which are noise.

The numbers

	Before	After
Time to first token, typical interaction	10 to 30 seconds	Under 3 seconds
Accuracy across intent recognition and retrieval	Frequent misreads	Approximately 98% on the client’s evaluated use cases
AI system operating cost	Baseline	Roughly 30% lower
Knowledge base	120 GB, about 40,000 PDFs	Unchanged

Two honest notes on those figures. The 98% is measured on the set of use cases the client defined and evaluated, and I would not present it as a universal rate across every possible question. And "under three seconds" is time to the first token of the answer in typical interactions, which is what a person in a chat window actually feels; long answers still take longer to finish streaming.

The cost reduction was mostly a side effect. Smaller prompts and cached answers mean fewer tokens, and fewer tokens is the whole cost model.

The voice path

The same engine sits behind a voice interface, with speech to text on the way in and text to speech on the way out. Voice is far less forgiving of latency than text; a three second pause in a spoken conversation is an eternity. The work described above is what made a voice path viable at all. Without the intent routing and context trimming, the assistant could not have answered fast enough to hold a conversation.

What I would tell a team starting this today

Find out what kind of question you are answering before you go looking for the answer. Intent detection is not a nice to have on top of RAG; it is what makes retrieval targeted.
Put permissions inside retrieval from day one. Retrofitting it is painful and the leak you are preventing is one your own testing will never find.
Measure where the time actually goes before touching the model. In our experience the model is rarely the largest slice.
Keep humans in charge of the decisions that matter. The assistant here answers questions, collects details and routes leads on its own. It does not close deals or change records without a person in the loop, and that is a design choice, not a limitation.

The hard part of enterprise AI is rarely calling the model. It is the retrieval, the permissions, the context management, the caching, the latency controls and the verification around it. That surrounding system is what we build, and it is the difference between an assistant that demos well and one that runs inside a real product with a substantial production user base. If you have an assistant that is slow, wrong, or both, the free AI audit is a reasonable place to start, and this page describes how we approach retrieval systems in general.

Suresh B is the Founder & CEO of JarvisBitz Tech, an AI and software engineering company helping businesses build and scale production AI systems and digital products. This article first appeared on jarvisbitz.com.

DEV Community: Suresh B

How to Build an MCP Server for Internal Tools: Architecture, Authentication and Error Handling

What the server owns, and what it does not

Architecture

A server that does something real

Authentication

Scopes are not enough

Error handling

Running it

Failure modes worth knowing about

Readiness checklist

Production LLM Application Architecture in 2026: An Engineering Checklist Beyond the Demo

Why the demo proves nothing

The shape that survives contact with users

Controls that earn their place

Authorize before you retrieve

Treat external content as data, not instructions

Treat model output as untrusted input

Put explicit approval in front of high impact actions

Reliability is mostly boring

Cost, latency and the thing you cannot see

Evaluation is a release gate, not a research project

Make every change reversible

Make traces answer questions

The moving parts are genuinely moving

The checklist

What you can safely leave until later

Production Voice AI Architecture in 2026: How Realtime Conversational Agents Are Built

Architecture 1: a single realtime model

Architecture 2: full-duplex with a delegated backend

Architecture 3: a chained pipeline

The components every voice agent needs

Testing the conversation and the task

Choosing

Production considerations

When not to build a voice agent

MCP in 2026: What Changed in the 2026-07-28 Specification and How to Design Production Integrations

MCP in September 2026

What 2026-07-28 removed

Sessions and Mcp-Session-Id

The initialize handshake

The GET stream, resumability and a few utilities

Server-initiated requests

What replaced it

Per-request _meta

server/discover

subscriptions/listen

Multi Round-Trip Requests and InputRequiredResult

Hosting consequences

Stateful work without sessions

Compatibility matrix

Deprecations and the lifecycle

Authorization hardening

Extensions, and an integration decision table

Failure modes

Production checklist and tradeoffs

If you are migrating today

RAG Architecture in 2026: A Production Blueprint for Retrieval-Augmented Generation

Where RAG demos break in production

The reference architecture: eight layers and their contracts

Query understanding and intent routing

Intents that skip retrieval

When to rewrite

Authorized retrieval

Filter before rank

ACL metadata on every chunk

Ranking and context assembly

Token budget per source

Deduplication

Citation payloads

The first-token latency budget

Build or buy, one layer at a time

Long context as a layer

Failure modes and the detectors that catch them

Production checklist

The lesson from a production system

AI voice agent for customer service: what stops callers hanging up?

The hangup happens at a specific moment

Narrow the job until it is boring

Build the transfer before the conversation

Sessions and `Mcp-Session-Id`

The `initialize` handshake

Per-request `_meta`

`server/discover`

`subscriptions/listen`

Multi Round-Trip Requests and `InputRequiredResult`