DEV Community

Cover image for We went from 33 to 87 on Cloudflare's agent-readiness scan. Here's what each check actually does.
Alex
Alex

Posted on AI-assisted

We went from 33 to 87 on Cloudflare's agent-readiness scan. Here's what each check actually does.

#seo #mcp #ai #webdev

Cloudflare has a free scanner at isitagentready.com that grades how easy your site is for AI agents to find, read and use. We build Fuxux, a social media scheduler with a public REST API and a hosted MCP server, so I assumed we'd do fine.

We scored 33/100, Level 1 ("Basic Web Presence").

A day later we were at 87/100, Level 5 ("Agent-Native"). None of it was hard, but most of it wasn't obvious either, so here's every check that failed, what it means, and the exact thing we now serve. Two checks still fail on purpose, and I'll explain why at the end.

The starting point

Category Before After
Discoverability 2/4 4/4
Content 1/1 1/1
Bot Access Control 1/2 2/2
API, Auth, MCP & Skill Discovery 1/8 6/8

What we already had: a valid robots.txt, a sitemap, and Markdown versions of our pages for agents (that's the Content check). What we didn't have was any machine-readable way to say "there's an API here, and an MCP server, and this is how you sign in". We had all of those things. We just never told agents where they were.

1. Link headers on the homepage (RFC 8288)

The cheapest fix. Agents that only fetch your homepage can't find anything that isn't linked from it, and HTML links are noisy. A Link response header points straight at the useful resources:

Link: </.well-known/api-catalog>; rel="api-catalog",
      </openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json",
      </docs>; rel="service-doc"; type="text/html",
      </.well-known/mcp/server-card.json>; rel="describedby"; type="application/json",
      </llms.txt>; rel="describedby"; type="text/plain"
Enter fullscreen mode Exit fullscreen mode

We're on Next.js, so this is one entry in headers() in next.config.ts, applied to / and our translated homepages.

2. Content Signals in robots.txt

robots.txt says who may crawl. Content Signals say what they may do with it: show it in search, use it as input to an AI answer, or train on it.

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Enter fullscreen mode Exit fullscreen mode

This is a business decision, not a technical one. We want AI assistants to know Fuxux exists and recommend it, so everything is yes. A news site would probably answer differently.

3. An API catalog (RFC 9727)

/.well-known/api-catalog is a small JSON "linkset" listing your APIs and where their descriptions and docs live. Ours lists the REST API and the MCP server:

{
  "linkset": [
    {
      "anchor": "https://www.fuxux.com/api",
      "service-desc": [{ "href": "https://www.fuxux.com/openapi.json", "type": "application/vnd.oai.openapi+json" }],
      "service-doc": [{ "href": "https://www.fuxux.com/docs", "type": "text/html" }]
    },
    {
      "anchor": "https://www.fuxux.com/api/mcp",
      "service-desc": [{ "href": "https://www.fuxux.com/.well-known/mcp/server-card.json", "type": "application/json" }],
      "service-doc": [{ "href": "https://www.fuxux.com/mcp/docs", "type": "text/html" }]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Serve it as application/linkset+json.

4. The MCP server card was missing one field

We already published /.well-known/mcp.json, and the scanner found it, then failed it: "missing required fields (need serverInfo.name or name)". We'd written it for one client months ago and never checked it against the draft server-card format.

We added name, version and a serverInfo block, kept every old field so nothing that read it before breaks, and also serve it at /.well-known/mcp/server-card.json:

{
  "name": "com.fuxux/mcp",
  "title": "Fuxux",
  "version": "1.3.0",
  "serverInfo": { "name": "fuxux", "title": "Fuxux", "version": "1.3.0" },
  "transport": { "type": "streamable-http", "endpoint": "https://www.fuxux.com/api/mcp" },
  "authentication": {
    "required": true,
    "schemes": ["oauth2", "bearer"],
    "protected_resource_metadata": "https://www.fuxux.com/.well-known/oauth-protected-resource"
  }
}
Enter fullscreen mode Exit fullscreen mode

(Trimmed: the real one also has a description, icons and docs links.)

5. An Agent Skills index

Agent Skills are Markdown instruction files (SKILL.md) that teach an agent how to use your product. We already had two for the ChatGPT plugin and ClawHub. The discovery index just lists them, with a hash so a client can tell whether a file changed:

{
  "$schema": "https://schemas.agentskills.io/discovery/0.2.0/schema.json",
  "skills": [
    {
      "name": "social-media-posts",
      "type": "skill-md",
      "url": "/.well-known/agent-skills/social-media-posts/SKILL.md",
      "digest": "sha256:4f7590cc…"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

One gotcha: compute the digest from the exact bytes you serve. We first hashed the file before our build changed its line endings, and the digest didn't match.

6. An ARD manifest

ARD (Agentic Resource Discovery) is one catalog at /.well-known/ai-catalog.json that lists everything an agent can use on your domain: MCP servers, OpenAPI specs, agents. Each entry can carry representativeQueries, example requests the resource is good for:

"representativeQueries": [
  "schedule a post to Instagram and LinkedIn for tomorrow at 9am",
  "what posts do I have scheduled this week",
  "why did my TikTok post fail"
]
Enter fullscreen mode Exit fullscreen mode

7. WebMCP: two read-only tools in the browser

WebMCP lets a page register tools that an AI agent running in the user's browser can call. It's still a draft and few browsers support it, so we kept it tiny: two read-only tools on marketing pages, one returning an overview, one returning pricing, both from Markdown we already serve.

const ctx = document.modelContext ?? navigator.modelContext;
if (!ctx || typeof ctx.registerTool !== "function") return; // most browsers today
for (const tool of TOOLS) {
  try {
    ctx.registerTool(tool, { signal: controller.signal });
  } catch {
    // an older draft of the API: leave the page alone
  }
}
Enter fullscreen mode Exit fullscreen mode

No writes, no auth, nothing that can change an account. If the API changes shape, the try keeps the page working.

8. DNS for AI discovery, and DNSSEC

This one took the longest, and it isn't code. DNS-AID lets an agent find your services with a DNS lookup instead of a web request:

_mcp._agents.fuxux.com.   HTTPS 1 www.fuxux.com. alpn="h2" port=443
Enter fullscreen mode Exit fullscreen mode

Adding the record took a minute. The scanner still failed it, with a precise message: "records found, but DNSSEC was not validated." A discovery record only counts if a resolver can prove it wasn't forged.

Our DNS is on Cloudflare but the domain is registered at GoDaddy, so DNSSEC has two halves:

  1. Turn DNSSEC on in Cloudflare (DNS → Settings). It signs the zone and shows you a DS record.
  2. Add that DS record at the registrar (GoDaddy → DNS → DS Records): key tag, algorithm 13, digest type 2, digest.

About 20 minutes later cloudflare-dns.com returned AD: true for the record, and Discoverability went to 4/4. Side benefit: the whole domain is now protected against forged DNS answers, which we should have done anyway.

The two checks we skip on purpose

OAuth Protected Resource Metadata at the root. We do serve RFC 9728 metadata, but ours names the MCP endpoint as the resource:

{ "resource": "https://www.fuxux.com/api/mcp", "authorization_servers": ["https://www.fuxux.com"] }
Enter fullscreen mode Exit fullscreen mode

The scanner expects the document at the root to name the origin itself. That's a reasonable reading of the RFC. But when we submitted our connector to Claude's and ChatGPT's directories, both reviews needed resource to equal the MCP URL exactly. One scanner point isn't worth risking two live directory listings, so this stays as it is.

auth.md. This lets an agent register an account on its own, without a person going through sign-up. For a product where each published X post costs us real money, and where our abuse limits (one trial per person, email and card checks) assume a human signs up, automatic account creation is a product decision, not a checkbox. We'll revisit it when assistants actually use it.

What I'd do first if you're starting from Level 1

In order of effort versus score:

  1. Content-Signal in robots.txt (two minutes)
  2. Link headers on the homepage (ten minutes)
  3. /.well-known/api-catalog if you have an API
  4. Check any MCP server card you already publish against the current format
  5. DNSSEC, if your DNS provider makes it one click

Then decide, deliberately, which checks you don't want. The score is a checklist, not a goal. A couple of the things it asks for change who can do what on your product, and those deserve a real decision.

If you want to try the MCP server this was all about, it's at https://www.fuxux.com/api/mcp, and the setup for Claude, ChatGPT and Cursor is at fuxux.com/mcp/docs. Questions about any of the files above are welcome in the comments.

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to write a short YouTube comment as a regular developer, reacting to this specific video. Should be one or two sentences, casual, maybe ask a question about the checks. Must not be polished, no marketing. No URLs. Must avoid certain phrases. No double hyphens. Use straight quotes only. Ensure no em-dash, no ellipsis char. Provide comment text only. Let's craft: "lol the 'robots.txt check' actually mattered more than i thought, any tips on getting the score up without breaking