DEV Community: G POORNA PRUDHVI

Vector Databases for RAG: How to Choose One and Wire It In

G POORNA PRUDHVI — Fri, 09 Oct 2026 17:38:35 +0000

Originally published at aiengineerinsights.com

In short: Does your RAG app need a dedicated vector database? pgvector vs Chroma vs Qdrant vs Weaviate vs Milvus vs Pinecone — open-source vs managed, and how to choose.

What a vector database does in a RAG pipeline

In a retrieval-augmented generation pipeline, the vector database is the store that sits between your chunking step and the language model. It does two jobs.

At ingest time, each chunk is run through an embedding model to produce a vector, and that vector is written to the database together with an id, the original text, and whatever metadata you attach (source file, section, tenant, date).

At query time, the user's question is embedded with the same model, and the database returns the top-k stored vectors closest to it under a distance metric such as cosine, inner product, or L2. Those k chunks are the "context" that gets pasted into the prompt.

The part that makes a vector database more than a table with a float array column is the approximate nearest-neighbour (ANN) index. Exact search compares the query against every stored vector, which is fine for thousands of chunks and painful for millions.

ANN indexes like HNSW (a graph-based index) and IVFFlat (a clustering-based index) trade a little recall for a large speed-up, and every option in this guide is built around one or more of them (pgvector README — indexing; Qdrant docs — indexing).

Two consequences follow. First, the database can only return what you indexed, so chunking quality caps retrieval quality before the database is ever involved. Second, the database's filtering and index behaviour directly shape what "top-k" means in practice — which is why the choice matters more than "it stores vectors" suggests.

Do you actually need a dedicated vector database?

This is the decision most teams get wrong in one of two directions: either they stand up a new distributed system for a corpus of ten thousand chunks, or they bolt vectors onto an existing database and only discover its limits under production filtering load. The honest answer is that a Postgres extension is enough for a lot of RAG apps, and a dedicated database earns its place under specific, nameable conditions.

The case for pgvector. pgvector adds a vector column type to Postgres, performs exact nearest-neighbour search by default (perfect recall), and lets you add an HNSW or IVFFlat index for approximate search when the table grows.

It supports L2, inner product, and cosine distance, and — because it is just Postgres — you keep ACID transactions, joins against your existing tables, point-in-time recovery, and every client library you already use (pgvector README).

If your chunks live next to the rows they came from, a single SELECT ... ORDER BY embedding <=> $1 LIMIT 10 with a WHERE clause on tenant or document id is a complete retrieval layer with no second system to operate.

A dedicated vector database earns its keep when:

Scale. Your vector count is heading past what a single Postgres node's memory and HNSW build time handle comfortably, and you need horizontal sharding or a distributed deployment designed around vectors (Milvus's distributed mode, for example — Milvus docs).
Filtering at scale. Most real queries combine "nearest to this vector" with "and only from this tenant / date range / document type". Dedicated databases design their ANN index around payload filtering so the filter doesn't collapse recall; Qdrant's filterable HNSW is the canonical example (Qdrant docs — filtering).
Hybrid search. You want keyword (BM25) and vector scores fused in one query, which Weaviate ships natively (Weaviate docs — hybrid search). In Postgres you can combine tsvector full-text search with pgvector, but you are assembling the fusion yourself.
Index tuning. You need to pick between HNSW, IVF, product quantisation, disk-based, or GPU indexes per collection and tune them independently of the rest of your database workload.
Zero ops. Nobody on the team wants to run or tune a database at all, and a managed service is worth the vendor dependency.

If none of those apply yet, start with pgvector (or Chroma for a local prototype) and keep the retrieval interface behind a thin function, so swapping the store later is a contained change rather than a rewrite.

The main options compared

Below are the six options that come up in almost every "best vector database for RAG" conversation. The table sticks to documented characteristics — licence, hosting model, and what each is designed for — rather than benchmark rankings, which vary wildly with dataset, dimensionality, filter selectivity, and hardware.

pgvector

Type: Postgres extension (open source, PostgreSQL licence)
Hosting: Wherever your Postgres runs — self-hosted or any managed Postgres that ships the extension
Best for: Teams already on Postgres who want vectors next to their relational data, with SQL joins and ACID transactions, without a second system

Chroma

Type: Open-source vector database (Apache 2.0), embedded / local-first
Hosting: In-process with a persistent client, self-hosted server, or Chroma Cloud
Best for: Prototyping and small-to-mid apps where a pip install and a local folder is the whole deployment story

Qdrant

Type: Open-source vector database (Apache 2.0), written in Rust
Hosting: Self-host (Docker / Kubernetes) or Qdrant Cloud
Best for: Production workloads that lean hard on metadata filtering alongside vector search, with an option to move to managed later

Weaviate

Type: Open-source vector database (BSD-3-Clause) with a managed cloud
Hosting: Self-host or Weaviate Cloud
Best for: Teams that want built-in hybrid (BM25 + vector) search and pluggable vectoriser modules out of the box

Milvus

Type: Open-source vector database (Apache 2.0), LF AI & Data project
Hosting: Milvus Lite (pip), Standalone (Docker), Distributed (Kubernetes), or Zilliz Cloud
Best for: Large-scale deployments that need a distributed architecture and a wide choice of index types, including disk- and GPU-based ones

Pinecone

Type: Fully managed, closed-source vector database
Hosting: Managed service only — no self-host option
Best for: Teams that want zero infrastructure to run and are comfortable with a vendor-hosted, proprietary store

A few details worth knowing beyond the table. Chroma can run entirely in-process (ephemeral or persisted to a directory) or as a client-server deployment, and its query API supports metadata filters via where and document-text filters via where_document (Chroma docs).

Qdrant stores a JSON "payload" alongside each vector and supports sparse vectors and quantisation to shrink memory (Qdrant docs).

Weaviate offers HNSW, flat, and dynamic index types plus vectoriser modules that call an embedding provider for you at import time (Weaviate docs).

Milvus exposes the broadest index menu — FLAT, IVF variants, HNSW, DiskANN, and GPU indexes — and three deployment modes from a pip-installable Lite to a Kubernetes cluster (Milvus docs).

Pinecone is serverless and managed only, with namespaces for tenant isolation and metadata filtering on query (Pinecone docs).

Open source vs managed: the real trade-off

Five of the six options above are open-source vector databases or extensions, and four of those (Chroma, Qdrant, Weaviate, Milvus via Zilliz) also sell a managed cloud tier. That makes the choice less "open source vs managed" than "who runs it, and what do you give up either way".

Self-hosting an open source vector database gives you control over index parameters, data locality (the vectors never leave your VPC), and predictable infrastructure cost at steady load.

The price is operations: you own upgrades, backups, replication, memory sizing for HNSW graphs, and re-indexing when you change embedding models.

For a team that already runs Postgres or Kubernetes this is often marginal work; for a two-person team shipping a product it can be the largest single chunk of non-feature effort.

A managed service — Pinecone, or the cloud tier of an open-source option — removes that operational load entirely and usually scales without a re-architecture.

What you give up is control and portability: index internals are the vendor's, cost scales with usage rather than with hardware, and migrating out means re-embedding or bulk-exporting your corpus.

Choosing a managed tier of an open-source database keeps an exit hatch open, since the same API is available self-hosted; a closed-source service does not.

A reasonable rule: prototype on something you can run locally, and make the self-host vs managed call when you know your actual query volume, filter patterns, and who will be on-call.

A minimal vector database example for RAG (Chroma)

Here is the entire chunk → embed → store → query loop using the Chroma vector database, because it needs nothing but pip install chromadb and runs in-process. The same four calls — create a client, get a collection, add documents, query — map almost one-to-one onto every other option in this guide (Chroma docs — getting started).

import chromadb

# 1. Persistent client: vectors are stored on disk at ./rag_db and survive restarts
client = chromadb.PersistentClient(path="./rag_db")

# 2. A collection is a named index. Chroma embeds documents with its
#    default embedding function unless you pass embedding_function=...
collection = client.get_or_create_collection(name="docs")

# 3. Chunks from your chunking step (one string per chunk)
chunks = [
    "pgvector adds a vector type plus HNSW and IVFFlat indexes to Postgres.",
    "Chroma can run in-process with a persistent client, no server needed.",
    "Qdrant is an open-source vector database written in Rust.",
]

# 4. Add: each chunk is embedded, indexed, and stored with an id and metadata
collection.add(
    ids=[f"chunk-{i}" for i in range(len(chunks))],
    documents=chunks,
    metadatas=[{"source": "notes.md", "chunk": i} for i in range(len(chunks))],
)

# 5. Query: the question is embedded the same way, then the top-k nearest
#    chunks come back. 'where' applies an optional metadata filter.
results = collection.query(
    query_texts=["Which option runs inside Postgres?"],
    n_results=2,
    where={"source": "notes.md"},
)

for doc, dist in zip(results["documents"][0], results["distances"][0]):
    print(f"{dist:.3f}  {doc}")

Three things to notice. You never called an embedding model directly — Chroma applies a default embedding function to documents on both add and query, and you can swap in OpenAI, Sentence Transformers, or your own function by passing embedding_function= when you create the collection.

Whatever you choose, ingest and query must use the same model or the distances are meaningless. Metadata is not optional in practice — the where filter is how you scope retrieval to a tenant, a document, or a date range, and it is the feature you will lean on most as the corpus grows.

The distances come back with the documents, which is what you will log when you start measuring retrieval quality in the next section.

Swapping this for pgvector means a CREATE EXTENSION vector, a table with a vector(n) column, an INSERT per chunk, and an ORDER BY embedding <=> query_vector LIMIT k query — with the embedding call done in your own code (pgvector README). For Qdrant, Weaviate, Milvus, and Pinecone the shape is the same: a client, a collection/index, an upsert of vectors plus payload, and a search with a filter.

How to choose: a decision guide by stage

Prototype / local development. Chroma with a persistent client, or pgvector if you already have Postgres running. Optimise for iteration speed — you will change chunking and embedding models several times, and re-indexing a local store is free.
Already on Postgres, up to millions of vectors. pgvector with an HNSW index. You get joins against your application data, one backup story, and no new system. Revisit when index build time, memory, or filtered-query recall becomes a measurable problem rather than a hypothetical one.
Production with heavy metadata filtering or hybrid search. Qdrant (filterable HNSW, payloads, sparse vectors), Weaviate (native hybrid search, vectoriser modules), or Milvus (distributed deployment, wide index choice). All three are open source, so you can self-host now and move to their managed cloud later without changing the client code.
Zero-ops, willing to accept a vendor. Pinecone, or the managed tier of one of the open-source databases. Pick the managed tier of an open-source option if you want the ability to self-host later; pick Pinecone if you never intend to.
Very large scale (hundreds of millions of vectors and up). Milvus Distributed or a managed service designed for that scale. This is the only tier where a dedicated distributed architecture is a requirement rather than a preference.

Whatever you pick, put the retrieval call behind a single function that takes a query string and filter and returns chunks with scores. Every option here fits that interface, and it is the difference between a one-afternoon migration and a week of untangling.

How to know your vector DB retrieval is actually good

The database returns top-k vectors by distance — that is a mechanical guarantee, not a quality one. Whether those k chunks are the right chunks depends on the embedding model, the chunking, the index's recall at your settings, and how well filters interact with the index.

None of that is visible from the query response, so treat the store the way you'd treat any other pipeline component: hold everything else constant and measure retrieval.

The two retrieval-side metrics to track are context precision (of the chunks returned, how many were relevant) and context recall (of the chunks that were relevant, how many were returned), computed against a labelled eval set of questions and their expected source chunks. Frameworks such as RAGAS implement both (RAGAS docs), and our guide to RAG evaluation metrics walks through how they're calculated and what thresholds teams commonly use.

Two database-specific checks are worth adding.

First, compare ANN recall against exact search on a sample: pgvector lets you drop the index (or raise hnsw.ef_search) and re-run the same query to see how much recall the index is costing you (pgvector README — query options); Qdrant and the others expose equivalent search-time parameters.

Second, re-run your eval set with your most selective production filter applied — a store that scores well unfiltered and badly filtered is telling you something important about how its index handles that filter. A vector database is only as good as the retrieval quality you can measure through it.

Frequently Asked Questions

What is a vector database in RAG?

A vector database is the store that holds the embeddings of your document chunks and answers nearest-neighbour queries against them. In a RAG pipeline it sits between chunking and generation: at ingest time each chunk is embedded and written to the database with an id and metadata, and at query time the user's question is embedded with the same model and the database returns the top-k most similar chunks, which become the context the language model answers from.

Do I need a vector database for RAG?

Not always. For small-to-mid scale — up to a few million vectors — pgvector inside Postgres is often enough, especially if you already run Postgres and want vectors next to your relational data.

A dedicated vector database earns its place when you need to scale beyond what a single Postgres node handles comfortably, when you rely heavily on metadata filtering combined with vector search, or when you want hybrid (keyword plus vector) search and finer control over the ANN index.

What is the best open-source vector database?

There is no single winner — it depends on where you are. Chroma is the easiest to start with because it runs in-process from a pip install and persists to a local folder, which makes it ideal for local development and prototyping.

Qdrant, Weaviate, and Milvus are all open-source options built for self-hosted production: Qdrant is known for filtering alongside vector search, Weaviate for built-in hybrid search and vectoriser modules, and Milvus for a distributed architecture and a wide choice of index types.

Pick based on your deployment model and the features you will actually use, not on a leaderboard.

Is pgvector good enough for RAG?

Yes, for many production applications. pgvector adds a vector column type, exact nearest-neighbour search by default, and HNSW and IVFFlat indexes for approximate search, and it inherits Postgres transactions, joins, and backups.

It is a strong choice up to millions of vectors, particularly if you are already on Postgres.

The caveats are that ANN index build time and memory (especially for HNSW) grow with the table, and that scaling beyond one node is a Postgres scaling problem rather than a vector-database feature, so very large or very filter-heavy workloads may be better served by a dedicated store.

Chroma vs Pinecone: which should I use for RAG?

They solve different problems. Chroma is open source and local-first: it runs in-process with a persistent client, so it is great for development, prototyping, and small self-hosted apps, and it also offers a server mode and a managed cloud.

Pinecone is a fully managed, closed-source service: you never run infrastructure, it scales without you operating anything, and in return you accept a proprietary store and a vendor dependency.

Use Chroma to build and iterate; move to Pinecone (or a managed tier of an open-source database) when you want someone else to run production.

How do I know if my vector database is returning good results?

Measure retrieval directly rather than judging the final answers by eye.

Build a small eval set of questions with the chunks that should be retrieved, then compute context precision (how much of what was retrieved is relevant) and context recall (how much of what was relevant was retrieved) for each configuration — index type, top-k, filters, embedding model.

Our guide to RAG evaluation metrics covers how those scores are computed. The database choice only matters to the extent that retrieval quality you can measure improves.

References

MCP vs API: What the Model Context Protocol Actually Is

G POORNA PRUDHVI — Fri, 09 Oct 2026 17:37:57 +0000

Originally published at aiengineerinsights.com

In short: MCP vs API: an open, model-facing standard that makes tools reusable across AI apps — turning M×N integrations into M+N. How it works and when to use each.

What is MCP (Model Context Protocol)?

MCP is an open protocol for connecting AI applications to external tools and data sources. It was introduced and open-sourced by Anthropic in late 2024 and adopted across the industry through 2025.

The problem it solves is boring but expensive: before MCP, every AI app needed custom integration code for every tool it wanted to use. Ten apps and ten tools meant a hundred bespoke integrations.

MCP fixes that the way USB-C fixed chargers, or the way the Language Server Protocol fixed editor tooling: a single standard connector.

You expose a capability once as an MCP server, and any MCP host — Claude, ChatGPT, an IDE like Cursor, or your own AI agent — can discover and use it without custom glue.

Anthropic's own phrase for it is a "USB-C port for AI applications." If you want the full plain-English explainer — what an MCP server is, the host/client/server roles, and the architecture — see What Is MCP? This article focuses on how MCP compares to a plain API.

MCP vs API: the real difference

The question "MCP vs API" is slightly the wrong frame, because MCP is built on top of APIs — an MCP server almost always calls a REST or database API underneath. The useful contrast is what each is for: an API is a general interface a developer codes against; MCP is a model-facing protocol that makes those interfaces discoverable and reusable by any AI app.

Aspect	Traditional API	MCP
What it is	A general interface between two pieces of software	A specific open protocol for connecting LLM apps to tools and data
Who it's for	Developers, who write integration code	The model / agent, which calls tools at runtime
Discovery	Read the docs, hard-code each endpoint	Self-describing — the host lists available tools automatically
What it exposes	Endpoints (REST, GraphQL, RPC…)	Standard Tools, Resources, and Prompts
Integration cost	M×N — every app wired to every tool	M+N — write one server, any host uses it
Transport	Usually HTTP	JSON-RPC 2.0 over stdio or streamable HTTP
Relationship	The thing being called	Usually wraps an API so a model can use it

The single biggest win is the last-but-one row. Custom integrations scale as M×N — every AI app times every tool. MCP collapses that to M+N: write a server once, and every host that speaks MCP gets it for free.

How MCP works: architecture

MCP is a client-server protocol built on JSON-RPC 2.0. Three roles (shown in the diagram above):

Host. The AI application the user interacts with — Claude Desktop, an IDE, an agent runtime.
Client. Lives inside the host and holds one connection to each server.
Server. Exposes capabilities over MCP, usually wrapping an API, database, or filesystem.

Each server offers three standard primitives: Tools (actions the model can call, like "create a GitHub issue"), Resources (data/context the app can read, like a file or record), and Prompts (reusable templates a user can invoke).

Servers run over stdio for local processes or streamable HTTP for remote ones. Because the primitives are standard and self-describing, the host can list a server's tools at runtime — no hard-coding.

How MCP evolved — and what was hard at first

MCP shipped fast and changed fast, so the version you read about in an early post is not quite the one you build on today. The first release (November 2024) supported two transports — stdio for local servers and an HTTP+SSE transport for remote ones — with a deliberately minimal spec. That minimalism is what got it adopted, but it also left real gaps the next revisions had to close.

The early pain points:

The HTTP+SSE transport was awkward. It needed a long-lived server-sent-events stream plus a separate POST endpoint — stateful, hard to run on serverless, and with no way to resume a dropped connection.
Auth was underspecified. Early remote servers had no standard way to authenticate, so everyone rolled their own.
Security gaps. Because tool descriptions are fed to the model, a malicious server can smuggle instructions ("tool poisoning" / prompt injection), and over-permissioned servers or token pass-through opened confused-deputy and token-theft risks.
Breaking changes. The spec moved so quickly that early adopters had to chase transport and SDK churn.

The revisions that fixed most of it:

2025-03-26. The clunky HTTP+SSE transport was replaced by Streamable HTTP — a single endpoint that works statelessly (serverless-friendly) and supports resumable streams — and a formal OAuth 2.1 authorization framework was added.
2025-06-18. MCP servers were defined as OAuth Resource Servers with Resource Indicators (RFC 8707) so tokens can't be reused where they shouldn't be, plus structured tool output, an elicitation flow for servers to ask the user for input, removal of JSON-RPC batching, and a dedicated security best-practices document.

Governance opened up too, moving toward a public specification process rather than a single-vendor project. Net effect: the "USB-C for AI" idea survived, but the wiring behind it got a lot more production-ready.

RAG vs MCP: not the same thing

These get compared a lot, but they solve different problems at different layers. RAG (retrieval-augmented generation) is a technique for pulling relevant knowledge into a model's context before it answers. MCP is a protocol for connecting tools and data sources to AI apps. They're complementary, not competing.

In fact, the cleanest way to ship RAG to an agent is through MCP: wrap your retrieval pipeline (embeddings, vector search, reranking) in an MCP server that exposes a search_knowledge_base tool. Now any MCP host can do retrieval against your data with zero custom integration. RAG is the what; MCP is one clean how to deliver it.

When to use MCP vs a plain API

Reach for MCP when an LLM or agent needs to discover and call tools, especially if you want the same tool to work across multiple AI hosts, or you're building agents that plug into many capabilities.
Reach for a plain API when it's deterministic app-to-app integration with no model in the loop, when you need maximum control or performance, or when only one consumer will ever call it.
Usually you use both: the MCP server is a thin, model-friendly wrapper around your existing API, adding discovery, standard primitives, and reuse.

What developers actually say about MCP

MCP's reception has been loud on both sides — worth knowing before you build on it. The praise: it solved a real, expensive problem (the M×N integration mess), and adoption was extraordinary — thousands of community servers within months, plus cross-vendor support from OpenAI, Google, and Microsoft, made it a de facto standard faster than almost any recent protocol.

The criticism clusters in three places:

Security is the loudest. Researchers repeatedly warned that MCP makes it easy to wire an agent into what Simon Willison calls the "lethal trifecta" — access to private data, exposure to untrusted content, and a way to exfiltrate — so a poisoned tool description or malicious server can hijack an agent. The 2025 auth and security revisions were a direct response.
Context and token burn. Every connected server injects its tool definitions — names, descriptions, and schemas — into the model's context on every call. Wire up a dozen servers and you can spend thousands of tokens before the user asks anything, driving up cost and latency; worse, too many tools measurably degrades the model's tool selection. The practical fix is to load only the servers a task needs (and lean on dynamic/lazy tool loading as hosts add it).
Transport churn. Replacing HTTP+SSE so soon frustrated early adopters who'd already built on it.
"Do we even need it?" Some argued function calling plus OpenAPI already covered most cases and that MCP re-invents the wheel; the counter is that runtime discovery, standard primitives, and cross-host reuse are exactly what ad-hoc function calling lacks.

The honest read in 2026: MCP won the standardization battle, and the sharp early edges (transport, auth, security) are mostly sanded down — but you still own the security of what you connect. Treat every third-party server as untrusted, scope permissions tightly, and keep a human in the loop for risky actions.

MCP examples and ecosystem

There's a large and growing library of ready-made MCP servers — filesystem, GitHub, Google Drive, Slack, Postgres, Puppeteer/browser, and web search among them — plus SDKs to build your own in TypeScript, Python, and other languages.

On the host side, Claude Desktop, popular IDEs and coding agents (see our ranked AI coding agents), and custom agent frameworks all speak MCP. That two-sided adoption is exactly why writing one server is worth it: it lights up everywhere at once.

MCP vs A2A, ADK, and other agent protocols

MCP isn't the only standard in this space, and the others mostly complement it rather than compete. The trick is to notice which connection each one is about:

MCP — agent ↔ tools and data. What this article is about.
A2A (Agent2Agent) — agent ↔ agent. Google's protocol for letting independent agents discover and delegate to each other, later donated to the Linux Foundation. MCP gives an agent its tools; A2A lets agents talk to one another. See our deep dive on A2A.
ADK (Agent Development Kit) — not a protocol but Google's open-source framework for building agents; it speaks both MCP (for tools) and A2A (for agent-to-agent), which is a clean picture of how the layers stack.
ACP and others — additional agent-communication efforts exist (e.g. IBM/BeeAI's ACP), but momentum in 2026 has consolidated around MCP for tools and A2A for agent-to-agent.

How to build an MCP server

The minimum path:

Pick an SDK — the official TypeScript or Python SDK is the usual starting point.
Define your primitives. List the Tools, Resources, and Prompts you want to expose, with clear names and descriptions (the model reads these). 3.

Wrap the backend. Each tool calls your existing API, database, or filesystem.

Choose a transport. stdio for local, streamable HTTP for remote.
Connect a host — point Claude Desktop, your IDE, or your agent at the server, and the tools appear automatically.

Building tool-using agents is core AI-engineering work in 2026. If you're leveling up toward it, our AI engineering roadmap and skills checklist cover the fundamentals underneath MCP and agents.

Frequently Asked Questions

What is MCP in simple terms?

MCP (Model Context Protocol) is an open standard for connecting AI apps to external tools and data. Instead of writing custom glue code for every tool in every AI app, you expose a tool once as an MCP server, and any MCP-compatible host — Claude, ChatGPT, an IDE, your own agent — can use it. Anthropic describes it as a 'USB-C port for AI applications.'

What is the difference between MCP and an API?

An API is a general interface you write code against, one integration at a time. MCP is one standardized, model-facing protocol that makes tools self-describing and reusable across every AI app. They aren't competitors: an MCP server usually calls an API under the hood — MCP is the layer that lets an LLM discover and use that API in a uniform way.

Is MCP better than a REST API?

It's not better or worse — it operates at a different layer. Use a plain API for deterministic app-to-app integration with no model in the loop. Use MCP when you want an LLM or agent to discover and call tools, especially across multiple AI hosts. In practice MCP servers wrap REST APIs, so you often use both together.

What is the difference between RAG and MCP?

RAG (retrieval-augmented generation) is a technique for pulling relevant knowledge into a model's context. MCP is a protocol for connecting tools and data sources. They're at different layers and are often combined — you can expose a retrieval/RAG capability as an MCP server so any agent can search your knowledge base as a standard tool.

Who created MCP and is it open?

MCP was introduced and open-sourced by Anthropic in late 2024. The specification is public, with SDKs in several languages, and it saw broad adoption across the industry through 2025 — including support from other model providers and many IDEs and agent frameworks.

Is MCP secure?

MCP is as secure as how you deploy it. Early versions had gaps — underspecified auth and prompt-injection / 'tool poisoning' risks — which the 2025 revisions addressed with an OAuth 2.1 framework, resource-server semantics, and a security best-practices spec.

But you still own the risk of what you connect: treat third-party servers as untrusted, scope permissions tightly, and keep a human in the loop for sensitive actions.

What is the difference between MCP and A2A?

They cover different connections. MCP connects an agent to tools and data. A2A (Agent2Agent, from Google) connects agents to each other so they can discover and delegate work. They're complementary — an agent might use MCP for its tools and A2A to hand off to another agent. Google's ADK framework speaks both.

How do I build an MCP server?

Pick an official MCP SDK (TypeScript or Python are common), define the Tools, Resources, and Prompts you want to expose, wrap whatever API or data source they call, and run the server over stdio (for local) or HTTP (for remote). Point an MCP host — Claude Desktop, an IDE, or your agent — at it, and the tools become available automatically.

References

LLM Routing Explained: How an LLM Router Picks the Right Model

G POORNA PRUDHVI — Fri, 09 Oct 2026 17:36:11 +0000

Originally published at aiengineerinsights.com

TL;DR: LLM routing sends each request to the cheapest model that can answer it acceptably, using a decision made before the expensive call — by rules, embedding similarity, a trained router (RouteLLM-style), or a calibrated decision model (Jev-style). Published routers report cost cuts from roughly 2× (RouteLLM) up to 98% (FrugalGPT cascades) on their own benchmarks; the number you get depends on how much of your traffic is genuinely easy. Default anything uncertain to the stronger model, log every decision, and measure router accuracy and quality delta on your own logs.

What is LLM routing, and what does an LLM router do?

LLM routing (also called model routing) is the practice of choosing which model answers a request at runtime rather than hard-wiring your application to one model. The router is whatever makes that choice.

It sits between your application and your model providers, inspects each incoming request — the text, its length, the user, the tools in play — and dispatches it to a model tier: a small, fast model for the easy cases; a frontier model for the hard ones; sometimes a middle tier in between.

The reason routing exists is an uncomfortable fact about production traffic: a large share of it is easy. Password resets, "what's your refund policy," classify-this-ticket, rewrite-this-sentence.

Sending those to the same model you use for multi-step agent planning is paying frontier prices for work a model a tenth the size does fine.

The FrugalGPT paper framed the economics bluntly in 2023: API fees across popular models "can differ by two orders of magnitude," which is exactly the gap a router exploits.

One distinction to fix before anything else, because the term "LLM router" is used for two different things:

Model routing decides which model should answer, based on the request. This is what RouteLLM, Semantic Router, NVIDIA's router, OpenRouter's auto router, and Jev do, each with a different signal. It is the subject of this post.
Gateway routing decides which deployment of a model serves a request — which region, which provider key, which copy has rate-limit headroom — plus retries, fallbacks, and cooldowns. LiteLLM's Router is the canonical open-source example. It does not look at what the request is asking.

You usually want both, layered: a model router in front choosing the tier, a gateway behind it keeping each tier reliable. Conflating them is how teams buy a gateway, call it a router, and wonder why their bill didn't move.

Does LLM routing actually save money? What the papers measured

Yes, with a caveat that matters: every published number is on the authors' own benchmark and model pair. Three results anchor the field:

FrugalGPT (Chen, Zaharia, Zou, 2023) reports that it "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction" and can "improve the accuracy over GPT-4 by 4% with the same cost," using three strategies — prompt adaptation, LLM approximation, and the LLM cascade: try a cheap model first, score its answer, and only escalate if the score is low.
Hybrid LLM (Ding et al., 2024) trains a router on predicted query difficulty with a tunable quality level, and reports "up to 40% fewer calls to the large model, with no drop in response quality."
RouteLLM (Ong et al., 2024) trains routers on human preference data to choose between a stronger and a weaker LLM, and reports it "significantly reduces costs — by over 2 times in certain cases — without compromising the quality of responses." Notably, the routers kept their performance "even when the strong and weak models are changed at test time."

The spread — 2× to 98% — is not the papers disagreeing. It is the savings being a function of three things you control: how much of your traffic is genuinely easy, how much cheaper your small tier is than your big one, and how accurately your router can tell the two apart.

A router with perfect accuracy on traffic that is 90% easy saves close to 90% of frontier spend. The same router on agent-planning traffic that is 90% hard saves almost nothing and adds a hop.

The routing research keeps pushing the accuracy term — a 2026 NVIDIA paper, "LLM Router: Rethinking Routing with Prefill Activations", routes on a model's internal prefill activations instead of surface features and reports closing 45.58% of the gap to an oracle router at 74.31% cost savings relative to the most expensive model — but the traffic-mix term is yours, and it is the one to measure first.

The four routing signals: rules, embeddings, learned routers, decision models

Every router, open source or hosted, makes its decision from one of four kinds of signal — or a stack of them. The signal determines what the router can and cannot see, which is more important than which product wraps it.

Rules and metadata

What it decides on: Token count, user tier, whether a tool is required, language, SLA — plain code
Cost of the decision: Free, microseconds, deterministic
Strength: Zero risk of being 'wrong' in a way you can't explain; perfect for hard constraints
Weakness: Cannot tell an easy question from a hard one by content; rules rot as traffic shifts

Embedding similarity (semantic router)

What it decides on: Which of your example-utterance groups the query is closest to in vector space
Cost of the decision: One embedding call per request (often local), tens of milliseconds
Strength: No training loop — add a route by adding example sentences; good for intent-shaped traffic
Weakness: Measures topic, not difficulty; a 'billing' question can be trivial or brutal

Learned difficulty / preference router

What it decides on: A trained model predicts which tier will answer well (or which model 'wins') for this query
Cost of the decision: A small model forward pass; needs labeled or preference data to train
Strength: Directly optimizes the thing you care about — quality vs cost; strongest published results
Weakness: Training data effort; drift when your models or traffic change; opaque

Calibrated decision model (Jev-style)

What it decides on: A typed Choice — simple / complex — plus a confidence trained to track accuracy
Cost of the decision: One non-autoregressive pass; TypeSafe quotes ~70–500 ms and $0.042 per million input tokens
Strength: No training data, labels defined at request time, and a confidence you can threshold on
Weakness: Calibration is imperfect in the 0.3–0.8 band; prompt injection can move the verdict

Rules are where to start and where to put anything that is actually a policy: requests over N tokens go to the long-context model, requests that need a tool go to the model that calls tools reliably, enterprise-tier users always get the frontier model. Rules can't judge difficulty from content, but they never surprise you.

Embedding similarity is what Semantic Router does: you define each route with a handful of example utterances, the library embeds them, and at request time it embeds the query and returns the closest route (or None).

It is fast and needs no training loop, and it is the right tool when your traffic is intent-shaped — billing vs support vs sales. Its blind spot is that it measures topic, not difficulty. "Why was I charged twice" and "reconcile these three invoices against the contract amendment" are both billing.

Learned routers fix that by training a model to predict the thing you care about: RouteLLM predicts, from preference data, whether the weak model's answer would be preferred; Hybrid LLM predicts query difficulty against a quality target; NVIDIA's prefill-activation router predicts per-model correctness from internal activations. They have the strongest published results and the highest setup cost — you need labeled or preference data, and the router is a model that drifts when your traffic or your model pair changes.

Calibrated decision models are the newest option and the one most relevant to teams who want a learned-style signal without a training project.

A model like Jev takes the request plus a label set you define at call time (simple / complex, or small / medium / frontier) and returns one typed Choice with a confidence trained to track accuracy.

The confidence is the point: it lets you route on a threshold instead of a guess. We come back to the practical rules for that in the Jev section below.

A fifth pattern deserves a name because it is a different topology, not a different signal: the cascade. Instead of predicting difficulty up front, FrugalGPT's cascade calls the cheap model first, scores the answer with a separate checker, and escalates only if the score is low.

Cascades don't need a difficulty predictor, but they pay the cheap call on every request and add its latency to the hard ones. Routers and cascades compose well: route the obvious cases, cascade the ambiguous ones.

RouteLLM vs Semantic Router vs NVIDIA vs Jev vs LiteLLM vs OpenRouter: what each actually routes on

The names below get thrown into the same "LLM router" bucket in search results and Reddit threads. They are not interchangeable. Read the third column: it tells you which signal from the previous section each one is built on, and therefore what it can't see.

RouteLLM (LMSYS / UC Berkeley)

What it is: Learned router, open source
Routes on: Predicted win-rate of a strong vs weak model, trained on human preference data
Worth knowing: Paper reports cost reductions of over 2× in certain cases without quality loss; routers transferred when the strong/weak pair was swapped

Semantic Router (Aurelio Labs)

What it is: Embedding router, open source
Routes on: Similarity between the query embedding and example utterances per route
Worth knowing: Returns a route name or None; positioned as a 'superfast decision-making layer' that avoids waiting on an LLM generation

NVIDIA LLM Router blueprint

What it is: Reference architecture
Routes on: v2: intent routing with a small Qwen 1.7B model, or 'auto-routing' with CLIP embeddings plus a trained network
Worth knowing: v2 only returns a model recommendation (it does not proxy the call); the repo now carries a deprecation notice pointing to NeMo Switchyard

Jev (TypeSafe AI)

What it is: Calibrated decision model, API
Routes on: A Choice you define at request time (e.g. simple / complex) with a calibrated confidence
Worth knowing: Used in LangChain's ModelRouterMiddleware with the instruction 'Choose the least costly model that can complete the task'

LiteLLM Router

What it is: AI gateway / proxy, open source
Routes on: Deployment-level load balancing: simple-shuffle, latency-based, usage-based, least-busy, cost-based, custom
Worth knowing: Balances copies of the same model across providers and regions with cooldowns, fallbacks, retries; does not judge query difficulty

OpenRouter Auto Router (openrouter/auto)

What it is: Hosted gateway feature
Routes on: A lightweight classifier assigns ~30 task types, then ranks models by what the OpenRouter community spent on that task over a trailing 7-day window
Worth knowing: Cost tiers (low → max), session stickiness, allowed/excluded model filters, graceful degradation to a default set

Two of these deserve a closer look because they are the ones most often mistaken for a difficulty router. LiteLLM's Router is a gateway: its documented strategies — simple-shuffle (the default, weighted by RPM/TPM), latency-based, usage-based, least-busy, cost-based, and custom — all pick a deployment from a pool, and its reliability features are cooldowns, fallbacks, timeouts, and retries.

It is excellent at that job. It will not notice that a request is easy.

OpenRouter's Auto Router (openrouter/auto) is a genuine model router, but note what it optimizes.

Per its docs, "a fast, lightweight classifier assigns each prompt one of ~30 fine-grained task types," then the router "looks up which models the OpenRouter community actually spends on over a trailing 7-day window" for that task, filtered by the cost tier you choose (low through max).

That is routing on task type and market behavior, not on the difficulty of your specific request — useful if you want a sensible default per task without building anything, less useful if your goal is "send 70% of my support traffic to the small model." It also keeps a conversation on the model it landed on and degrades to a default set if classification is unavailable.

The NVIDIA blueprint is a reminder that this space moves fast: the llm-router repo went from v1 (task/complexity classifiers) to v2 (intent routing via a small Qwen model, or CLIP embeddings plus a trained network, returning recommendations only) and now carries a deprecation notice pointing at NeMo Switchyard. Treat any specific router product as replaceable; the signal taxonomy is what lasts.

How do you build an LLM router? A pattern that holds up in production

The router you ship is almost never one signal. It is rules first, then one content signal, then a threshold with a safe default, then logging. In order:

Define the tiers and what "acceptable" means per tier. Two tiers is enough to start (small, frontier). Write down the quality bar — the eval score or human rubric — that a small-tier answer must clear. Without this, "the router works" is unfalsifiable. 2.

Pull every hard constraint into rules. Context length, tool requirements, tenant policy, regulated content. These never go through a model; they run first and short-circuit.

Pick one content signal for the rest. Intent-shaped traffic: embeddings. Difficulty-shaped traffic with labeled data: a learned router.

Difficulty-shaped traffic with no labels and a deadline: a calibrated decision model. Start with the one you can ship this week; you can swap it later because the interface — request in, tier out — doesn't change.
4.

Threshold with the expensive model as the default. Route to the small tier only when the signal is confidently "easy." Everything uncertain goes up. A router that errs toward the frontier model costs you a little money; one that errs toward the small model costs you quality you may not notice for weeks.
5.

Feed the router structured fields, not just raw text. Token count, turn count, whether retrieval found anything, user tier. It gives rules something to act on and shrinks the surface an attacker can write into.
6.

Log the decision, the confidence, the tier, and the outcome. This log is your eval set, your drift detector, and the training data for a learned router later.

Put a gateway behind each tier. Retries, fallbacks, cooldowns, provider keys — the LiteLLM job.

If the small tier is down, the router's decision should fall through to the frontier tier, not fail.

The shape of it as pseudocode. Illustrative only — the control flow, not real SDK calls:

# ILLUSTRATIVE PSEUDOCODE — control flow only, not real API syntax.

def route(request):
    # 1. hard rules run first, in plain code
    if request.tokens > SMALL_CTX or request.needs_tools or request.user.tier == "enterprise":
        return FRONTIER

    # 2. one content signal — swap the implementation, keep the interface
    decision = router.decide(
        question="simple or complex?",
        input=structured(request),          # fields your code assembled
        labels=["simple", "complex"],
    )
    log(request.id, decision.label, decision.confidence)

    # 3. threshold with the expensive model as the default
    if decision.label == "simple" and decision.confidence >= 0.9:
        return SMALL
    return FRONTIER                          # the uncertain middle goes UP, not down

tier = route(request)
answer = gateway.complete(tier, request)     # retries / fallbacks live in the gateway
if tier == SMALL and not passes_check(answer):
    answer = gateway.complete(FRONTIER, request)   # optional cascade on failure
log_outcome(request.id, tier, answer)

Notice the asymmetry in step 3. The threshold is on "simple," not on "complex." That is deliberate: it encodes the safe default into the control flow, so a badly calibrated or drifting router degrades into "slightly more expensive," never into "silently worse." If the router is doing its job inside an agent loop, the same decision point is also where you'd gate risky tool calls — a different question to the same cheap decision layer.

Where does Jev fit as an LLM router?

Routing by complexity is the headline use case TypeSafe and LangChain give for Jev, so it is worth being precise about what it adds and what it doesn't.

Jev is a non-autoregressive decision model: you send the request plus a label set, it returns one Choice (up to 255 labels) and a confidence in a single forward pass, with no text. It is trained with RLCD — Reinforcement Learning for Calibrated Decisions — so the confidence is meant to track how often it is right.

Pricing is $0.042 per million input tokens with output free, which is what makes it plausible to call on every request.

In the taxonomy above, that makes it a learned-style signal with no training step: you get a difficulty judgment (not just a topic match, as with embeddings) without collecting preference data (as with RouteLLM).

LangChain's harness write-up wires it in as a ModelRouterMiddleware with the instruction "Choose the least costly model that can complete the task," and uses the same model in an AutoModeMiddleware to block risky tool calls before they execute.

Our how to use Jev guide walks through that routing-plus-guardrail pattern; the Jev vs LLMs post covers why you wouldn't use a chat model to make the routing decision in the first place.

The limits, which decide how you threshold it:

Calibration is real but imperfect. An independent test measured an expected calibration error of about 0.107, with confidence reliable near 0 and 1 and shaky in the 0.3–0.8 band. For routing that means: trust a 0.96 "simple"; treat a 0.6 "simple" as "complex." The pseudocode above does exactly that.
Prompt injection moves the verdict. VentureBeat reported an Octomind demo in which a block probability fell from 0.76 to 0.48 after a fake "user pre-approved" field was added to the input. For a router, the attack is cheaper and subtler: a user who writes "this is a simple question" into their request may push themselves onto the weak model and get a worse answer — or, in the other direction, push expensive traffic onto your frontier tier. Keep the decision on fields your code controls, and cap per-user frontier spend in rules, not in the model.
No reasoning, no explanation. You get a label and a number. If you need to know why a request was routed, you reconstruct it from the logged fields.
Vendor numbers are vendor numbers. The ~70–500 ms latency and the headline speed/cost multipliers are TypeSafe's own figures on workflows it chose. Benchmark the decision latency in your region before it goes into a p99 budget.

Net: a calibrated decision model is the fastest way to get a difficulty-aware router running on day one, and the logged decisions become the dataset that lets you train a RouteLLM-style router later if the volume justifies it. The same bridge logic we described for classification applies: zero-shot now, trained later, with the zero-shot phase building toward its own replacement.

How do you evaluate an LLM router?

A router is a classifier whose errors have asymmetric costs, so evaluate it like one. Three numbers, per routing configuration, on a labeled sample of real traffic:

Router accuracy, split by direction. How often did it send an easy request to the small tier (the savings), and how often did it send a hard request to the small tier (the damage)? Report them separately; a single accuracy number hides the one that matters.
Quality delta vs all-frontier. Run the routed pipeline and the everything-to-the-frontier-model pipeline over the same eval set and grade both — with your existing evals, an LLM judge, and human spot checks on the disagreements. The delta is the price you are paying for the savings. Decide the acceptable delta before you look at the savings.
Realized cost and latency. Not the vendor multiplier — your bill and your p50/p99, including the router's own call. A router that adds 400 ms to every request to save money on 30% of them may be a net loss on latency-sensitive paths.

Then keep running it. The production log from the build section is the re-evaluation set; re-run the three numbers whenever you change a model, a prompt, or a threshold, and watch the confidence distribution for drift.

If you already have an eval harness for RAG or agents — the kind we describe in RAG evaluation metrics — the router eval slots into it as one more configuration to compare.

Common LLM routing mistakes

Using a chat LLM as the router. "Rate this query's difficulty from 1 to 10" costs an autoregressive call per request, returns a number that is generated text rather than a calibrated probability, and can take longer than the small-model answer it was supposed to save. Use a cheap signal.
Thresholding toward the cheap model. If "complex" needs 0.9 confidence to reach the frontier model, every uncertain request gets the weak answer. Flip it: "simple" needs the confidence; everything else goes up.
Buying a gateway and calling it a router. Load balancing across deployments of the same model will not reduce frontier spend. Check which signal the product routes on.
Routing on topic when the problem is difficulty. Embedding routers are fast and useful, but an intent label doesn't tell you whether the small model can handle this instance of that intent.
Letting user text control the decision. Any model-based router can be nudged by what the user writes. Rules on fields you control — tier, spend caps, tool requirements — are the parts an attacker can't talk past.
Shipping without the quality delta. Savings are visible on the bill the same week; quality loss shows up as churn a quarter later. Measure both on day one.
Not logging confidence. A log of labels without confidences cannot tell you whether the router is drifting or whether the threshold is in the right place. Store the number.

Frequently Asked Questions

What is LLM routing?

LLM routing is a cheap decision made before an expensive model call: for each incoming request, a router picks which model (or model tier) should answer it, based on difficulty, intent, cost, latency, or policy. The goal is to send the easy majority of traffic to a small, cheap model and reserve the frontier model for the hard minority, so you cut cost and latency without a visible drop in quality.

What is the difference between an LLM router and an AI gateway?

An AI gateway (LiteLLM, Portkey, OpenRouter's core proxy) sits in front of many providers and handles keys, rate limits, retries, fallbacks, and load balancing across deployments of a model — it decides which copy of a model serves a request.

An LLM router decides which model should answer at all, based on the content of the request. Many products do both; the routing signal (rules, embeddings, a learned router, or a calibrated decision model) is the part that determines quality and cost.

How much does LLM routing actually save?

Published results, on the authors' own benchmarks: RouteLLM (Ong et al., 2024) reports cost reductions of over 2× in certain cases without compromising response quality; Hybrid LLM (Ding et al., 2024) reports up to 40% fewer calls to the large model with no drop in response quality; FrugalGPT (Chen, Zaharia, Zou, 2023) reports matching the best individual LLM with up to 98% cost reduction using cascades and related tricks. Your number depends on how much of your traffic is genuinely easy, so measure it on your own logs before putting it in a budget.

What is RouteLLM?

RouteLLM is an open-source framework and paper from LMSYS and UC Berkeley (Ong et al., 2024) for training routers that dynamically choose between a stronger and a weaker LLM at inference time. The routers are trained on human preference data (plus data augmentation) to predict when the weaker model's answer would be preferred, and the paper reports that trained routers kept working even when the strong and weak models were swapped at test time.

Can I use Jev as an LLM router?

Yes — routing by complexity is one of the use cases TypeSafe and LangChain document for it. You ask Jev a Choice question (simple vs complex, or small / medium / frontier) and get back a label plus a calibrated confidence in one non-autoregressive pass, with no training data.

The practical rules: route to the cheap model only on high-confidence 'simple', send every mid-range confidence (roughly 0.3–0.8) to the stronger model by default, and keep attacker-controlled text out of the fields the decision hinges on, because prompt injection has been shown to move Jev's verdicts.

How do I evaluate an LLM router?

Build a labeled eval set of real requests where you already know which tier answers acceptably, then measure three things for each routing configuration: router accuracy (how often it picks the cheapest acceptable tier), the quality delta versus sending everything to the frontier model (graded by your existing evals or an LLM judge with human spot checks), and the realized cost and latency. Log every production decision with its confidence and outcome so you can re-run that evaluation as traffic and models drift.