TL;DR
To make your site agent-readable you need four layers — readable, discoverable, callable, payable — and three of them are files you generate at build time, not code that runs per request. A Markdown twin of each page is 94% smaller than the HTML on this site (13,698 bytes against 227,990), a /.well-known/api-catalog linkset makes those twins findable, and only the callable layer needs a server. Get that last part wrong and you meter every human pageview too, which is how this site once earned a Cloudflare Error 1027.
What is an agent-readable site?
An agent-readable site is one that publishes a machine-facing copy of its content and a machine-readable index of that copy, at their own URLs, so an agent never has to parse HTML to use it. That is the whole definition. It is a property of your URL space, not of your server — which is why almost all of it can be prerendered.
Cloudflare's agentic-internet post from 6 August 2026 splits the problem into four layers. It is a vision document, so it names products more than paths. Stripped to implementation, the layers are:
| Layer | The question it answers | What it is on disk |
|---|---|---|
| Readable | Can an agent consume this page without parsing HTML? | A .md twin of every page |
| Discoverable | Can an agent find what exists before fetching it? |
/llms.txt, /.well-known/api-catalog
|
| Callable | Can an agent do something, not just read? | An MCP endpoint, or in-page WebMCP tools |
| Payable | Can the agent compensate you for the fetch? |
x402, agent-facing ad slots |
The ordering matters. Each layer is only worth building if the one above it exists — a callable endpoint no agent can discover is a URL nobody types, and a payable layer on content no agent can read is a toll booth on a closed road.
Layer 1: Readable means a Markdown twin, not a redirect
The llms.txt spec that Jeremy Howard published on 3 September 2024 contains the whole idea in one line: serve a clean Markdown version of a page at the same URL with .md appended. That is the readable layer. Everything else is elaboration.
The reason to bother is bytes. I measured one post on this site three ways:
| URL | Bytes |
|---|---|
/blog/how-to-write-claude-md (HTML) |
227,990 |
/blog/how-to-write-claude-md.md |
19,149 |
/clean/blog/how-to-write-claude-md.md |
13,698 |
The clean Markdown is 94% smaller than the HTML — a factor of 16.6. That gap is not compression, it is deletion: inlined CSS, the framework's hydration payload, the header and footer, three JSON-LD blocks, and the theme switcher. All of it is load-bearing for a human and dead weight for a model with a context window to fill.
Two implementation details are easy to get wrong.
Give the Markdown its own URL. Do not branch on User-Agent at the edge to decide what to serve. That is cloaking in the search-engine sense, and it also forces every single pageview through a dynamic route — see the routing section below for what that costs. /blog/post.md as a separate, publicly fetchable document is not cloaking; it is publishing two files.
Generate it at build time. The Markdown twin is a pure function of source you already have. Rendering it on demand buys you nothing and costs you a request. The two-URL split also gives you somewhere clean to put the honest version: this site keeps an unsponsored copy of every post at /clean/blog/<slug>.md precisely so the machine-facing document stays auditable — the reasoning is in the teardown of TIME's agent ads.
Layer 2: Where the actual standards live
Readable is a convention. Discoverable has real specifications, and using them costs nothing.
/llms.txt is the human-authored index: an H1, a summary blockquote, and H2-delimited lists of links with one-line descriptions. Think of it as a sitemap written for something that reads prose. The /llms-full.txt variant most sites also ship — the entire corpus concatenated — is a community convention, not part of the spec. Ship both; they serve different context budgets.
/.well-known/api-catalog is the standardised one. RFC 9727, published June 2025, defines it as a well-known URI returning application/linkset+json — a machine-readable list of every API and document surface you publish, each with a link relation and media type. It is the difference between an agent guessing at five well-known paths and reading one document that names them all.
The full discovery set on this site is nine static files:
/llms.txt text/plain
/llms-full.txt text/plain
/.well-known/api-catalog application/linkset+json (RFC 9727)
/.well-known/mcp application/json
/.well-known/mcp/server-card.json application/json (SEP-1649)
/.well-known/agent-skills/index.json application/json
/.well-known/oauth-protected-resource application/json (RFC 9728)
/auth.md text/markdown
/.well-known/security.txt text/plain
Every one of those is prerendered. Zero of them need a server.
One point of honesty is worth more here than a fabricated auth flow: /.well-known/oauth-protected-resource on this site declares that /mcp is public and requires no authentication, and the authorization-server document advertises no token endpoint because none exists. Publishing OAuth metadata that describes an auth server you have not built is worse than publishing nothing — it sends agents into a handshake that will never complete.
Layer 3: Why callable is the only layer that needs a server
An MCP endpoint is where "agent-readable" turns into "agent-usable". This site exposes four tools over JSON-RPC at /mcp:
curl -s -X POST https://umesh-malik.com/mcp \
-H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
That returns search_posts, get_post, list_topics, and get_profile. Note what they are: thin wrappers over the same build artifacts the readable layer already produces. get_post returns the Markdown twin. search_posts reads the JSON feed. The callable layer added no new data — it added an interface to data that was already static, which is why it fits in one small Worker.
There is a second, cheaper form of callable that most write-ups skip. WebMCP registers tools from inside the page itself via navigator.modelContext.provideContext(), so a browser-driving agent already on your site can invoke them with no round-trip to your server at all. It is a feature-detected no-op in browsers that lack it, which makes it close to free to add. If your agent story is "someone's browser agent is on my page right now", WebMCP is the layer you want before a hosted MCP server.
Do not skip the write-safety question if your tools do more than read. Read-only tools over public content are the easy case; the moment a tool mutates something, you need explicit controls, which is a separate design problem. And if you are building the endpoint itself, the deployment walkthrough covers the Worker side.
Layer 4: Payable, and why I have not shipped it
The payable layer — x402, agent wallets, per-fetch micropayments — is the one Cloudflare is most excited about and the one with the least deployed surface. The protocol works and the mechanics are genuinely interesting, but almost no agent in the wild carries a budget yet, so a paywall keyed on HTTP 402 today mostly just returns 402 to agents that then leave.
What is deployable right now is the weaker version: a single labeled sponsored block in the machine-facing documents only, never in the human HTML, declared openly in robots.txt and a manifest. That is monetisation without a payment rail, and it is the honest interim step.
The architecture constraint that will take your site down
Here is the part no vendor post mentions, and it is the reason this article exists.
All four layers are tempting to implement as dynamic routes. Serving Markdown from a Worker is three lines. Generating llms.txt per request is easy. Once you do that, every request to your site — including ordinary human pageviews, if your routing is broad — is metered against your platform's request quota.
This site learned that the expensive way. An earlier version had a custom analytics endpoint that browsers polled every 10 seconds per open tab. That is 360 Worker requests per hour per idle tab. It exhausted the Cloudflare Free plan's 100,000 requests/day ceiling and the whole site started returning Error 1027 — not a degraded feature, the entire domain down.
The fix is one line of wrangler.toml:
run_worker_first = ["/mcp"]
not_found_handling = "404-page"
Only /mcp touches the Worker. Every other URL — all nine discovery files, every Markdown twin, every HTML page, the 404 — is prerendered into the build output and served by Cloudflare's asset layer, which is free, unlimited, and edge-cached. Setting that value to true or "/*" meters every pageview and reintroduces the exact failure above.
The general rule holds on any platform: the agentic layers are content, not compute. Anything that is a pure function of your source belongs in the build, not in a request handler. If you find yourself writing a route handler for the readable or discoverable layer, you have converted a free file into a billed request for no gain.
What to build first to make your site agent-readable
A defensible order of operations:
-
Markdown twins at
<url>.md, generated in your build. Highest ratio of agent value to effort, and it is one build step. -
/llms.txt, hand-written. It is an index, not a dump — the curation is the value. -
/.well-known/api-cataloglisting what you now have. This is the step that makes layers 1 and 2 findable rather than guessable. - WebMCP tools in-page, if agents visit your site in a browser. Feature-detected, no server.
- A hosted MCP endpoint, only when you have an action worth exposing, and scoped to exactly one route.
- Payable, when the agents that reach you actually carry budgets. Not yet.
Steps 1 through 4 add zero requests to your bill. Step 5 adds one route. That is the whole architecture: the agentic web, as far as most sites need it, is a build step and a well-known directory — and if your implementation needs a server for anything except the callable layer, you have built it wrong.
FAQ
What does it mean to make a site agent-readable?
It means publishing a machine-facing copy of your content and a machine-readable index of it, so an AI agent does not have to parse your HTML to use your site. In practice that is three things: a Markdown twin of every page, a discovery document at a well-known URL that lists what exists, and optionally a callable endpoint the agent can invoke instead of scraping. None of it requires changing what humans see.
Is llms.txt an actual standard?
It is a proposal, not an IETF standard. Jeremy Howard published the llms.txt spec on 3 September 2024, and it defines a Markdown file at /llms.txt containing an H1 title, a summary blockquote, and H2-delimited link lists. The companion convention — serving a clean Markdown version of any page by appending .md to its URL — is in the same spec. The widely used /llms-full.txt is a community extension, not part of the spec.
Do I need an MCP server to be agent-readable?
No, and it is the layer to build last. Readable and discoverable are static files you can generate at build time and serve for free. An MCP server is a live JSON-RPC endpoint, which means a running process, a request bill, and an attack surface. Ship the Markdown twins and the discovery documents first; add MCP only when you have an action an agent should take, not just text it should read.
Will serving Markdown to crawlers get me penalised for cloaking?
Not if the Markdown lives at its own URL. Cloaking is serving different content at the same URL based on who is asking, usually by branching on User-Agent at the edge. Publishing /blog/post.md alongside /blog/post is just publishing two documents, both fetchable by anyone including Googlebot. Route on the URL, never on the header, and there is nothing to penalise.
How much smaller is a Markdown page than the HTML version?
On this site, one blog post is 227,990 bytes as HTML and 13,698 bytes as clean Markdown — about 94% smaller, or a factor of 16.6. The gap is inlined CSS, the framework's hydration payload, navigation chrome, and structured-data blocks, none of which an agent can use. That difference is why agents that support the .md convention prefer it.
What is the cheapest way to serve all of this?
Prerender it. Every layer except the live callable endpoint is a static file, so it can be generated at build time and served by a CDN's asset layer at no per-request cost. On Cloudflare, that means keeping run_worker_first scoped to the single dynamic route rather than the whole site, so the metered Worker never sees ordinary traffic.
Sources
- Cloudflare, Building an open Agentic Internet: readable, discoverable, callable, and payable, 6 August 2026 — the four-layer framing this post implements.
- Jeremy Howard, The llms.txt spec, 3 September 2024 — the
/llms.txtformat and the.md-suffix convention. - IETF, RFC 9727: api-catalog, June 2025 — the
/.well-known/api-catalogwell-known URI andapplication/linkset+jsonlinkset format. - Byte counts measured with
curlagainstumesh-malik.comon 7 August 2026.
Originally published at umesh-malik.com
Keep reading on umesh-malik.com:
Top comments (0)