MCP Discovery Servers for AI Agents

The future of AEO will be dominated by businesses utilizing MCP discovery servers that feed accurate on-demand data to AI agents.

The businesses that win in AI answers will be the ones set up to feed LLMs precise, accurate and up to date information. To do that, an MCP discovery server is going to be a critical asset. This guide covers why, what one is, and the general setup involved in building one and getting AI agents to find it.

Why live data matters for AEO

Most AEO work today is about getting your content into what an LLM already knows: training data, search indexes, and pages it can crawl and quote. That works well for questions with stable answers. It doesn’t work for questions where the answer changes by the hour.

An LLM’s training data is months old by the time anyone uses it. Even when a chat tool searches the web, it’s reading pages that were cached at some point in the past. For anything time sensitive, both of those sources are guessing.

An MCP server closes that gap. Instead of hoping an AI tool has seen a recent copy of your data, you give it a direct, structured way to ask your systems for the current answer.

Efficiency is going to matter more over time, too. AI crawlers and agents will be limited on resources: compute, bandwidth and time all cost money at the scale they run. Crawling and parsing thousands of HTML pages to find one answer is expensive. Calling a single MCP tool that returns exactly the answer is cheap. As those limits tighten, AI systems will look for the most efficient way to find information, and businesses that offer a direct, structured source will be the ones they go to first.

MCP servers vs. MCP discovery

“MCP discovery server” covers two separate pieces, and it helps to keep them apart:

Piece	What it is	What it does
MCP server	An HTTP endpoint that speaks the Model Context Protocol (JSON-RPC 2.0)	Exposes tools, resources and prompts that AI agents call to get your data
MCP discovery	Files, tags and records on your domain that point to the server	Lets agents find your server from your domain alone, without anyone hardcoding it: /.well-known/ files, HTML <link> tags, DNS records and registries

You need both. A server nobody can find doesn’t get called, and discovery files that point to nothing don’t help anyone.

It’s also worth being clear about who calls an MCP server. Training crawlers like GPTBot and ClaudeBot collect static pages for training future models. They don’t call live tools. The callers are AI agents working at answer time: ChatGPT, Claude and Gemini when they use tools, agentic search, and assistants acting for a user. Discovery helps both groups, since crawlers learn your server exists and agents use it.

Questions that need live data

Think about a sports league management app. It creates an MCP discovery server that AI agents and chat assistants call directly while they’re answering a question. When someone asks something that can only be answered with live, up to date information, the AI calls that business’s data directly instead of guessing.

Here are a few examples of questions like this:

• When is the next upcoming men’s golf league in Medfield, Mass?
• Our flight was just cancelled, what alternative Delta flights exist from NYC to SFO right now?
• I’m looking for reservations for a group of 12 at a high end restaurant in downtown Boston, who has availability right now?

None of these can be answered accurately from training data. The answer changes by the hour, sometimes by the minute. Here’s what each one actually needs:

Question	Data needed	Who has it	How fast it changes
Next men’s golf league in Medfield	League schedules, divisions, open spots	The league management app	Daily
Alternative Delta flights, NYC to SFO	Flight schedules, seat inventory, rebooking options	The airline	Minutes
Table for 12 in downtown Boston	Restaurant availability by party size and time	Restaurants and their reservation systems	Minutes

In every case the accurate answer lives in one business’s database. If that business exposes it through an MCP server, the AI can answer correctly and point the user back to that business. If it doesn’t, the AI either guesses or sends the user to whoever does expose it.

What an MCP server does

To answer questions like these accurately, you need a way to give AI chat tools a means to interact with your data in real time, efficiently. That’s what an MCP server does. It relies on systems like Cloudflare to cache, handle high volumes and return accurate, relevant information on demand.

So when the LLM receives that question, instead of referencing its training data it knows to reference your specific MCP API to gather the latest and most accurate data. That’s the critical piece. Many businesses are already ramping up to do this now, and the future will be dominated by the ones who are in place to do it.

How MCP works

MCP (Model Context Protocol) is an open spec that defines how AI applications talk to outside systems. Messages are JSON-RPC 2.0, and an MCP server exposes three kinds of things:

• Tools: functions the AI can call with arguments, like get_upcoming_golf_leagues(town, division) or check_availability(party_size, time).
• Resources: read-only data the AI can load as context, like a menu, a price list or a policy document.
• Prompts: reusable templates that guide the AI through a common task.

For AEO, tools are the main event. When an AI agent connects to your server, it requests the list of tools (tools/list) and reads their names, descriptions and input schemas. When a user asks something that matches one of those descriptions, the agent calls the tool (tools/call), gets your data back, and uses it to write the answer.

That’s why the descriptions matter so much. The agent decides whether to use your server based on how clearly each tool says what it does.

The transport: Streamable HTTP

Remote MCP servers use the Streamable HTTP transport, which replaced the older HTTP plus SSE transport. The 2026-07-28 version of the spec simplified it further by removing protocol sessions, so each request stands on its own. In practice that means:

• Your server exposes a single endpoint, like https://mcp.yourdomain.com/mcp.
• Agents send JSON-RPC messages to it as HTTP POST requests.
• Requests carry an MCP-Protocol-Version header so the server knows which version of the spec the client speaks.
• Requests also mirror the JSON-RPC method into an Mcp-Method header (like tools/call), and calls to a named tool add an Mcp-Name header. Gateways like Cloudflare can route and cache on those headers without reading the body.

Stateless requests and routing headers are what make MCP practical to cache at the edge, which comes up again below.

The general setup

At a high level, there are two parts:

1. Create the MCP server, with Cloudflare DNS in front of it for security and, most important, caching.
2. Configure DNS and onsite AEO so AI crawlers and agents know about your MCP server and how they can access the information.

1. Creating the MCP server

The technology behind this depends heavily on your current stack and how your data can be accessed. It will often involve giving LLMs a way to authenticate and gather information from you. Here’s how I’d approach it.

Decide what data to expose

Start with the questions you want AI tools to answer about your business, then work backwards to the data that answers them. For most businesses that’s a short list:

• Availability, schedules and inventory
• Pricing and product or service details
• Locations, hours and contact details
• Status information like order tracking or event updates

A good rule: if it’s already on your website or you’d tell anyone who called and asked, it’s a strong candidate.

Design the tools

Each tool should do one clear thing. A few things I’ve found help:

• Write descriptions for an AI, not a developer. Say what the tool returns and when to use it.
• Keep inputs simple and typed. Use enums for fixed values (divisions, party sizes, cabin classes) so the AI can’t send something your system doesn’t understand.
• Bound every query. Never expose a tool that can dump a whole table. Require specific filters like a town or a date, and cap results (the top 5 open slots, not every slot).
• Return only what answers the question. Smaller responses are faster, cheaper to cache and less likely to leak something you didn’t mean to share.
• Start read-only. Looking things up is low risk. Actions like booking a table or rebooking a flight need authentication and should come later.

Here’s the difference in the actual tool definition an MCP server returns from tools/list. A weak definition gives the agent almost nothing to go on:

{
  "name": "search",
  "description": "League search endpoint",
  "inputSchema": {
    "type": "object",
    "properties": {
      "q": { "type": "string" }
    }
  }
}

A strong one for the Medfield golf league example tells the agent exactly what it gets and what to send:

{
  "name": "get_upcoming_golf_leagues",
  "description": "Returns upcoming golf leagues in Massachusetts filtered by town and division, including start dates, registration fees and real-time open roster spots.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "town": {
        "type": "string",
        "description": "Town name, for example Medfield, Walpole or Boston",
        "x-mcp-header": "Town"
      },
      "division": {
        "type": "string",
        "enum": ["mens", "womens", "mixed", "senior"],
        "description": "League division",
        "x-mcp-header": "Division"
      }
    },
    "required": ["town"]
  }
}

The x-mcp-header annotations are part of the 2026-07-28 spec. They tell clients to copy those arguments into Mcp-Param-Town and Mcp-Param-Division HTTP headers, so your edge cache can key on them without parsing the JSON body.

Return links that convert

An answer in a chat window is only worth something if it leads back to your business. Don’t return a block of plain text. Return structured results that include the canonical URL for the next step, tagged so you can attribute the visit:

{
  "leagues": [
    {
      "name": "Medfield Men's Thursday Night League",
      "starts": "2027-04-15",
      "open_spots": 6,
      "fee": 275,
      "register_url": "https://example.com/leagues/medfield-mens-thursday?utm_source=mcp_agent&utm_medium=ai"
    }
  ]
}

Assistants will usually render that as a direct “register” or “book” link in the chat, and the UTM tags show up in your analytics as AI agent traffic.

Put Cloudflare in front of it

Once AI tools start calling your server, request volume can grow fast and unpredictably. Cloudflare handles two jobs here: caching and security.

Caching is the most important part. Most questions repeat. Hundreds of people asking about Medfield golf leagues this week should not mean hundreds of database queries.

There’s a catch, though. MCP tool calls are HTTP POST requests, and CDN caching works on GET requests by URL. A normal Cloudflare cache rule won’t cache MCP responses at all. The fix is to do the caching in a Cloudflare Worker that sits in front of your server (or is your server):

1. Read the Mcp-Method, Mcp-Name and Mcp-Param-* headers on each request.
2. Build a cache key from them, like tools/call:get_upcoming_golf_leagues:Medfield:mens.
3. Serve the response from Workers KV or the Cache API when the key exists, and only pass the request to your origin when it doesn’t.

Cache each lookup for as long as the data can safely be stale:

Data type	Example	Reasonable cache time
Rarely changes	Locations, hours, service descriptions	Hours to a day
Changes daily	League schedules, event listings	5 to 15 minutes
Changes constantly	Seats, tables, inventory counts	Use a write-through cache (below)

Use a write-through cache for high volume servers

Time based caching has a tradeoff: either the data is a few minutes stale, or the cache expires often and your database takes the hit. For a high volume MCP server, a write-through cache gets rid of that tradeoff.

With write-through caching, the cache is updated at the same moment your data changes, not when someone asks for it. Every time your application writes to the database, it also writes the new value to the cache. The MCP server only ever reads from the cache.

	Time based (read-through)	Write-through
When the cache updates	When a request finds it expired	When the data changes
Freshness	Stale until the TTL runs out	Current as soon as the write lands
Database load from AI traffic	Grows with request volume	Near zero, reads never touch it
Traffic spikes	Many requests can miss at once and stampede the origin	No misses to stampede

That last row is the big one for high volume servers. When a flight gets cancelled, thousands of people ask the same question within minutes. With a write-through cache, every one of those AI requests is answered from the edge and your booking system never sees them.

Here’s how I’d set it up:

1. Keep the database as the source of truth. The cache is a copy that’s always kept current, never the place data is created.
2. Hook every write. Wherever your app changes availability, schedules or inventory, write the same record to the cache right after the database write succeeds. Model events, queue jobs or database triggers all work.
3. Store responses in the shape the tool returns. Cache the finished answer per key (for example availability:restaurant-123), so the MCP tool does a single lookup with no joins or formatting.
4. Use predictable keys. If the key can be built from the tool’s inputs (or the Mcp-Param-* headers), the tool never has to search.
5. Fall back to the database on a miss. If a key is ever missing, read from the origin, write it to the cache, and return it. That covers new records and cache resets.
6. Rebuild periodically. A nightly job that rewrites the cache from the database catches anything a missed write left behind.

On Cloudflare, Workers KV is a good fit for data that changes a few times an hour, since reads are fast everywhere and writes take up to about a minute to reach every location. For data that changes constantly, like seat or table counts, a Durable Object gives you a single, immediately consistent copy instead.

// In your app: write to the database, then the cache
async function updateAvailability(env, restaurantId, slots) {
  await db.saveAvailability(restaurantId, slots);
  await env.AVAILABILITY.put(`availability:${restaurantId}`, JSON.stringify(slots));
}

// In the MCP tool: read from the cache only
const slots = await env.AVAILABILITY.get(`availability:${restaurantId}`, "json");

If you want to host the whole server on Cloudflare, Workers has an MCP agent class built for this. Here’s a minimal read-only example for the golf league case, caching the origin API response for five minutes:

import { McpAgent } from "agents/mcp";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";

export class LeagueMCP extends McpAgent {
  server = new McpServer({ name: "league-schedule", version: "1.0.0" });

  async init() {
    this.server.tool(
      "get_upcoming_golf_leagues",
      "Returns upcoming golf leagues in Massachusetts filtered by town and division, with start dates and open spots.",
      { town: z.string(), division: z.enum(["mens", "womens", "mixed", "senior"]) },
      async ({ town, division }) => {
        const url = `https://api.example.com/leagues?town=${encodeURIComponent(town)}&division=${division}&limit=5`;
        const res = await fetch(url, { cf: { cacheTtl: 300, cacheEverything: true } });
        return { content: [{ type: "text", text: await res.text() }] };
      }
    );
  }
}

export default LeagueMCP.serve("/mcp");

Balance security and access

It’s a careful, strategic balance of security and access restriction. An MCP endpoint is easily scanned by malicious actors, and they can gain access to the same data you’re handing to AI agents. As long as you fully understand what the technology is doing and how it’s feeding data to outsiders, it shouldn’t be an issue. You just need to know this going in so you don’t expose sensitive business data.

Things I check before an MCP server goes live:

• Every tool queries through a limited, read-only database user or API key, never an admin connection.
• Responses are built from an allowlist of fields, not by returning whole database rows.
• Every query is bounded: required filters and a result cap.
• Nothing customer specific (names, emails, order details) is available without authentication.
• Tools that take actions require OAuth so the AI acts on behalf of a signed in user.
• The server validates the Origin header on incoming requests, which the MCP spec requires.
• Cloudflare rate limits are in place per IP, per ASN or per auth token, so one script can’t exhaust your API.
• Free text that came from users (reviews, comments, notes) is either left out or clearly separated as data, so nobody can plant instructions for the AI in your responses.

Don’t be too stringent

At the same time, I’d caution most people not to be too stringent about what they supply. If the data is something you already provide publicly to people, there’s a lot of incentive to provide it through an MCP server too. Ask yourself:

Is this data being available live truly harmful, or does it expose anything for my business?

If it doesn’t, I would consider including it. A restaurant hiding its availability from AI agents doesn’t protect anything. It just means the agent recommends the restaurant down the street that didn’t.

Choose a stack

I’ve seen MCP servers set up with Laravel, and I’ve worked on some myself. I’ve seen others written in Go, and even more written in Node.js. It can be built in any stack. The important thing is what you’re doing: creating a gateway into your business data that understands your data structures and handles requests from AI agents. It’s much like creating a public API, except this API follows the structure and format defined by the common MCP spec.

Stack	When it fits
Laravel	Your data already lives in a Laravel app. Laravel has an official MCP package, so tools can reuse your existing models and auth.
Go	High volume, low latency, or a team that already runs Go services. There’s an official Go SDK.
Node.js	The most examples and tooling, and it runs on Cloudflare Workers.
WordPress or WooCommerce	Your content, products or bookings already live in WordPress. The WordPress MCP adapter turns registered abilities into MCP tools on a REST endpoint.

This site is an example of the last one. kevinleary.net runs an MCP server through the WordPress MCP adapter that exposes content lookup and a business profile to agents.

If you already have a public or partner API, the fastest path is usually a thin MCP layer that calls it, rather than connecting the MCP server straight to your database.

Log every tool call

An MCP server gives you something a web page never does: the exact questions AI agents are asking about your business. Log every tools/call with the tool name, arguments, client and cache hit or miss, and send it somewhere you can query, like BigQuery or GA4 through the Measurement Protocol. That shows which agents use your data, what customers are asking for in real time, and which tools are worth expanding.

2. DNS and onsite AEO configuration

Once the server exists, AI crawlers and agents need to be told it’s there and how to use it. This involves some of the methods shown when you run your site through isitagentready.com, which is a great starting point for understanding how to connect your MCP server to AI agents.

The scan groups its checks into five categories:

Category	What it checks
Discoverability	robots.txt, sitemap, Link response headers, and DNS records for AI discovery
Content accessibility	Whether your pages can be served as Markdown to agents that ask for it
Bot access control	AI bot rules in robots.txt, content signals, and Web Bot Auth
Protocol discovery	API catalog, OAuth discovery, OAuth protected resource metadata, MCP server card, A2A agent card, agent skills, WebMCP
Commerce	Agent payment protocols like x402 and ACP

You don’t need all of it. For an MCP discovery server, these are the pieces that matter most.

Allow AI bots in robots.txt

If you block AI crawlers, they never learn your server exists. Add AI bot rules and a sitemap directive.

Publish an MCP server card

A server card is a JSON file at a well-known URL that describes your server: its name, endpoint, transport and what it offers. It’s how an agent finds your MCP server from your domain alone. Here’s the start of the one on this site, at /.well-known/mcp/server-card.json:

{
  "$schema": "https://static.modelcontextprotocol.io/schemas/v1/server-card.schema.json",
  "name": "net.kevinleary/wordpress",
  "title": "Kevinleary.net MCP Server",
  "version": "1.0.0",
  "websiteUrl": "https://www.kevinleary.net",
  "transport": {
    "type": "streamable-http",
    "endpoint": "https://www.kevinleary.net/wp-json/mcp/mcp-adapter-default-server"
  }
}

I serve the same card at /.well-known/mcp.json too, since agents check both.

Point to it from your HTML

Link tags in the <head> tell crawlers and browser agents where your machine-readable files live. These are the ones on every page of this site:

<link rel="ai-catalog" href="https://www.kevinleary.net/.well-known/ai-catalog.json" type="application/json">
<link rel="api-catalog" href="https://www.kevinleary.net/.well-known/api-catalog" type="application/linkset+json">
<link rel="service-desc" href="https://www.kevinleary.net/llm-api/" type="application/json">

The AI catalog lists everything an agent can use on the site, including the API and the MCP server. The same links can also be sent as HTTP Link response headers so agents find them without parsing HTML.

Publish OAuth metadata if you use auth

The protected resource metadata file (/.well-known/oauth-protected-resource) tells an agent where to sign in before calling tools that need a user.

Add a DNS record

There’s an IETF draft for MCP discovery through DNS: a TXT record at _mcp.yourdomain.com that points to your endpoint. It lets an agent find your server with a single DNS lookup, before it ever loads a page. It’s still an individual draft rather than a standard, so treat it as an extra, not a replacement for the well-known files.

List it in the MCP registry

The official MCP registry at registry.modelcontextprotocol.io is where clients and directories look up servers. Servers are named in reverse DNS form, so kevinleary.net becomes net.kevinleary/wordpress, and you prove you own the domain when you publish.

After making changes, run the isitagentready.com scan again. The report includes instructions you can hand to a developer or a coding agent to fix whatever is missing.

Testing it

Before announcing anything, test the server the way an AI agent will use it:

1. Connect with the MCP Inspector (npx @modelcontextprotocol/inspector) and call each tool by hand.
2. Add the server as a custom connector in Claude or ChatGPT and ask the real questions you built it for.
3. Watch your logs to see which tools get called, what arguments come in, and how often the cache answers.
4. Rewrite any tool description the AI ignores or misuses. That’s almost always the fix.

FAQ

What is the difference between an API and an MCP server for AEO?

An API is built for developers who read its docs and write code against it. An MCP server is built for AI agents: it describes its own tools in a standard format, so any MCP client can find them and call them without custom code. Many MCP servers are a thin layer over an existing API.

Does an MCP server replace Schema.org structured data?

No. Schema.org markup describes the content on your pages for search engines and crawlers. An MCP server answers live questions that pages can’t, like current availability. Most businesses should have both.

Do AI training crawlers use MCP servers?

Not directly. Training crawlers collect static pages. MCP servers are called by AI agents at answer time, when a user asks something that needs live data. Discovery files help both: crawlers learn your server exists, and agents use it.

Can MCP responses be cached on a CDN?

Not with a normal cache rule, because MCP requests are HTTP POSTs. Cache in a Cloudflare Worker instead, keyed on the Mcp-Method, Mcp-Name and Mcp-Param-* headers, with a write-through cache for data that changes constantly.