While marketers argue about llms.txt, a much larger machine-readable layer is quietly taking shape at your domain root. Here's an honest map of what exists, what's real, and what's still a draft.
Open a terminal and try this against almost any website:
curl https://example.com/.well-known/agent.json
You will almost certainly get a 404. Try it against a few hundred sites and you will get a few hundred 404s. That emptiness is the most interesting thing happening on the web right now.
There is a directory convention, /.well-known/, that has quietly existed for years holding boring infrastructure files. Over the last eighteen months it has become the staging ground for something bigger: the place where websites declare, in machine-readable form, what an AI agent is allowed to do with them. Not read. Do.
Most of it is empty. Some of it is not even finalized. But the shape of it is now clear enough to map, and knowing the map is worth more than rushing to fill it in.
Short answer: what is the agent web layer?
It's a set of small, machine-readable files published at predictable paths, mostly under /.well-known/, that tell AI agents what your site is, what services it exposes, how to authenticate, and how you want your content used. Where robots.txt says who may crawl and llms.txt says what your content means, this layer says what an agent can do with you.
It matters because AI is shifting from reading the web to acting on it. A model that summarizes your page needs your text. An agent that books, buys, compares, or calls your API needs a contract. That contract lives here.
Key takeaways
-
Three distinct jobs. Permission (robots.txt), meaning (llms.txt), and capability (the
/.well-known/layer). They are complementary, not competing. - Maturity varies wildly. Some of these are settled conventions. Others are IETF drafts. At least one has no ratified path at all and is being probed by convention.
- Adoption is near zero. Which is exactly why the map is more useful than the checklist right now.
- Do the boring parts first. Sitemaps, JSON-LD, crawlable text, and sane robots rules still carry almost all of your actual AI visibility today.
Three layers, three different questions
It helps to separate what these files actually do, because they get lumped together constantly.
Permission: robots.txt. The oldest layer. It answers "which crawlers may access which paths." It is a gate, not a description. It has also grown a new dialect: Content-Signal: directives, which let you state how your content may be used (training versus indexing versus live answers) rather than simply whether it may be fetched. That distinction matters, because "don't train on me" and "don't cite me" are very different wishes.
Meaning: llms.txt. A Markdown file at your root describing what your site is, which pages are canonical, and what to ignore. It gives a model context so it stops guessing which of your pages speaks for you.
Capability: /.well-known/. The newest and least populated layer. Instead of describing your content, these files describe your services: here are my APIs, here is how to authenticate against them, here is what my agent can do, here is how to pay me.
The first two serve a reader. The third serves an actor. Almost every site on the web has addressed the first, some have addressed the second, and virtually none have addressed the third.
What actually lives in /.well-known/
Here's the honest inventory, with maturity stated plainly, because a lot of writing on this subject presents drafts as settled standards.
A2A Agent Card at /.well-known/agent.json. From the Agent-to-Agent protocol. It declares that your organization runs an agent, gives it a name, and describes its capabilities and skills so other agents can discover and talk to it. The probe path is stable enough to rely on, though the spec is still settling.
MCP Server Card at /.well-known/mcp/server-card.json, defined in SEP-1649. Model Context Protocol is how assistants connect to external tools and data. A server card announces that you expose an MCP server and describes it. Worth knowing: the schema is an early draft and real-world implementations already disagree with it. Some cards put name and version at the top level; older drafts nest them under serverInfo. Anyone telling you this is locked down has not looked at production cards.
OAuth discovery and protected-resource declarations. These are the grown-ups in the room. If your APIs require authentication, agents need to discover how to authenticate, and which authorization servers can mint valid tokens for your protected endpoints. This is well-trodden OAuth/OIDC machinery, not speculative. If you run an authenticated API, this is the least experimental item on the list.
API catalog and agent skills index. Machine-readable listings of what your APIs are and what your agent can actually do. Without these, an agent has to guess at your capabilities or rely on a human having documented them somewhere in prose.
Web Bot Auth at /.well-known/http-message-signatures-directory. An IETF draft addressing a real problem: today, a bot claiming to be GPTBot is just a string in a header, and anyone can send it. This publishes cryptographic keys so bots can prove who they are. Genuinely important direction, genuinely still a draft.
x402 at /.well-known/x402. The payments layer, built on the long-dormant HTTP 402 "Payment Required" status code, so agents can pay for access programmatically. Be clear-eyed here: there is no ratified well-known path for x402 yet. That path is a convention that tools probe by agreement, not a standard you can cite. It is the furthest from settled of anything on this list.
Markdown content negotiation. Not a file at all. It's a request: an agent sends Accept: text/markdown to your homepage, and a well-prepared site returns a clean Markdown representation instead of a wall of rendering HTML. Cheap to implement, immediately useful, and almost nobody does it.
Alongside these sit the unglamorous fundamentals that still do most of the work: your sitemap, your JSON-LD structured data, your HTTP link headers, and your robots rules.
Why almost none of this is implemented
Three reasons, and they are all rational.
The specs are moving. Building against a draft that shifts under you is genuine engineering cost with uncertain payoff. Teams that waited on early web standards were often vindicated.
There is no immediate traffic reward. Publishing an agent card today does not send you visitors tomorrow. The payoff is positional, not immediate, and positional bets are the first thing cut from a roadmap.
And most importantly, almost nobody is asking for it yet. The agents that would consume these files are still being built. This is infrastructure for a demand curve that has not arrived.
That is a completely reasonable case for waiting. It is also exactly what the case for waiting sounded like in 1996, and in 2007, and in 2010.
What to actually do about it, in order
Here is the honest priority order, and note that the top of it is deliberately unexciting.
First, fix the boring layer. Crawlable server-rendered text, a working sitemap, correct JSON-LD, and robots rules that do not accidentally block the assistants you want citing you. This is where nearly all of your real AI visibility lives today. A site with a beautiful agent card and a stray Disallow: / is a site nobody can see. Do not skip to the interesting part.
Second, publish an llms.txt. Cheap, useful now, and it makes your canonical sources explicit rather than inferred.
Third, do OAuth discovery properly if you have authenticated APIs. This is stable, standard, and useful regardless of how the agent web shakes out. It is not a speculative bet.
Fourth, publish an agent card or MCP server card if you actually run an agent or an MCP server. Note the condition. Publishing a card describing an agent you do not have is worse than publishing nothing, because it advertises a capability that will fail when something tries to use it.
Fifth, watch the drafts. Web Bot Auth and x402 solve real problems and are worth understanding. They are not worth building against this quarter unless you have a specific reason.
The pattern is straightforward: implement what is stable and useful now, understand what is coming, and resist the urge to build against specifications that are still arguing with themselves.
The map is the point
The reason to know this layer exists is not so you can rush out and populate every path tomorrow. It is so that when a client, a competitor, or a vendor says "you need an agent card," you can ask the right question: is that a settled standard, or a draft, and does it describe something we actually run?
Right now, on this subject, the confident people are usually the ones who have not read the specs. The paths are conventions in flux. The schemas disagree with their own implementations. The honest position is that this layer is real, it is coming, most of it is not ready, and the fundamentals still matter more than any of it.
Knowing precisely which is which is the whole advantage.
FAQ
What is /.well-known/?
A standardized directory at your domain root where services publish machine-readable metadata at predictable paths. It has existed for years for infrastructure purposes and is now being used to describe what AI agents can do with a site.
Is this the same as llms.txt?
No. llms.txt describes your content: what your site is about and which pages are canonical. The /.well-known/ layer describes your capabilities: your APIs, your agent, your authentication, your payment endpoints. One serves a reader, the other serves an actor.
Do I need an A2A agent card?
Only if you actually run an agent that other agents should be able to discover and call. Publishing a card for an agent that does not exist advertises a capability that will fail on first contact.
Is x402 a standard I should implement?
Not yet. x402 is the agent-payments direction built on HTTP 402, but there is no ratified well-known path for it. Tools that probe /.well-known/x402 are following a convention, not a finalized spec. Understand it; do not build production infrastructure against it today.
What is Web Bot Auth solving?
Bot identity. Today any client can send a user-agent string claiming to be GPTBot, and there is no cryptographic way to verify it. Web Bot Auth, an IETF draft, publishes signing keys so bots can prove their identity. Real problem, draft solution.
What should I do first if I only have an hour?
Check that you are not accidentally blocking AI crawlers in robots.txt, confirm your important content is server-rendered text rather than trapped in JavaScript or images, and publish an llms.txt. That hour will do more for your AI visibility than every file in /.well-known/ combined, at least for now.
See where your site actually stands
Sourceable's agent-readiness scanner probes this entire layer, from the fundamentals through to the experimental drafts, and returns a 0-100 score across five pillars with the specific fixes to reach the next level. It tells you which of these files you have, which you are missing, and which ones actually matter for your situation.
Fair warning, and we say this from experience: most sites score well on the first three pillars and near zero on Protocol Discovery. When we ran it on our own site, we scored 78 out of 100, with a 10 on exactly this layer. The gap is normal. Knowing its exact shape is what lets you close it deliberately instead of guessing.
Top comments (0)