If you publish an MCP server or an A2A agent card, a small crowd of bots starts visiting within days. I spent a morning reading a day of our worker logs to see who they are, what they fetch, and what that means for anyone shipping agent-facing endpoints.
Who shows up
Every one of them announces itself in the User-Agent, often with a link and a promise about what it will not do:
- Registries and directories that list agents and MCP servers, re-checking the entries they already hold.
- Liveness monitors that hit the same endpoint on a fixed schedule, every few minutes, around the clock.
- Trust and quality scorers that read your card, check it against the spec and rank you.
- Research probes, including at least one academic study measuring MCP quality of service.
- Onboarding probes for newer agent registries that do one handshake per published endpoint.
Several say outright "read-only", "liveness-only" or "never invokes tools" in their User-Agent. They come to read, not to use.
What they fetch
The same four things, over and over:
-
/.well-known/agent-card.json, the A2A card. -
/llms.txt(and sometimes/llms-full.txt). - MCP
initialize, thentools/list, and stop. - Newer discovery files: an AI catalog or an MCP server card under
/.well-known/.
That is your storefront for machines. A coding agent that later decides whether to use your tool is reading the same files, or a registry's copy of them.
What they never do
In our logs none of them signed up or bought anything. Crawler traffic tells you that you are indexed. It does not tell you anyone is interested. Count it separately from people and real agent calls, or it will flatter every chart you have.
What this means if you ship agent endpoints
-
Keep
tools/listcheap and free of side effects. Crawlers call it all day, and nobody calling it is a user. - Keep the card valid and the facts current. Registries copy what you publish. A stale price or a removed feature in your llms.txt lives on in other indexes after you fix the page.
- Answer probes fast and the same way every time. Monitors mark you down for slow or flaky answers long before any person notices.
- Make the first real call succeed. When an agent finally tries you, the example in your tool description has to work as written. We found ours pointed at demo data that returned a 404 on the obvious first try, and fixed the hint the same day.
- Tag bot traffic by User-Agent before you report anything. Ours turned out to be mostly monitors and our own test clients.
Our setup, for reference
We run Odile Gate, which keeps AI agents inside the calls an API allows (per-app keys, allowed calls, approvals and an audit log). Its hosted MCP server is https://gate.odilelabs.com/mcp, and an agent can create a free project there with no signup. The crawlers above read it all day; that is how we learned all of this.
Top comments (1)
The “four things” list is the killer takeaway here. If crawlers mostly do:
/.well-known/agent-card.json/llms.txt(and maybe/llms-full.txt)initialize→tools/list/.well-known/then that directly implies your “machine storefront” is tiny and very cacheable. Most of the agent-facing truth for your product fits into a couple of static JSON / text resources plus a cheap list call.
The line “Crawler traffic tells you that you are indexed. It does not tell you anyone is interested.” pretty much calls out a new vanity metric class. I’d absolutely split dashboards into at least three buckets: crawlers (User-Agent match), agents actually invoking tools, and humans.
A few things I’m wondering / would check after reading this:
tools/listacross all these bots?agent-card.jsonandllms.txtand keep stale data?Also: your “make the first real call succeed” feels like the agent equivalent of landing page optimization. I’d treat that first example invocation as a test case in CI so it can’t silently rot.