Your robots.txt probably does not block the AI crawler you think it blocks

#ai #webdev #seo #php

Here is a rule that looks like it blocks OpenAI's crawler:

User-agent: *
Disallow: /checkout/
Disallow: /customer/

User-agent: GPTBot
Crawl-delay: 10

It blocks nothing. GPTBot reads its own group, finds one Crawl-delay line and no Disallow, and crawls everything — including /checkout/ and /customer/.

Under RFC 9309, a crawler that matches a specific user-agent group uses only that group. It does not inherit from User-agent: *. Naming a bot anywhere in the file switches off the wildcard rules for that bot entirely.

So the moment you add a named group to be more careful, you become less careful. And the failure is invisible: the file reads as though it says something.

I found this while doing something else, and the something else is the rest of this post.

The something else

I spent a few weeks cataloguing every project that helps an ecommerce store be found, read, trusted or transacted with by an AI system — for Magento specifically, because that is the platform I work on. The result is a list of 42 projects: awesome-magento-aeo, CC0, fork it if it is useful.

The interesting part is not the list. It is the distribution.

Category	Projects
`llms.txt` discovery files	11
Specifications	7
MCP servers	6
Agentic checkout	5
Structured data	4
Crawler policy	3
Product feeds	3
Auditing	3

Eleven implementations of llms.txt. For a file format proposed in 2024 that is, at heart, a Markdown document at your site root.

Three for crawler policy — the layer where the RFC 9309 trap above lives.

The ecosystem built the easiest layer eleven times and the hard layers three times each.

Why eleven is not the joke it looks like

My first read was duplicated effort: everyone shipping the same weekend project. That read is wrong.

Generating llms.txt for a real store is not trivial. You hit multi-store layouts, page-builder content that has to be resolved rather than dumped as markup, CMS directives, customer-group pricing, and catalogues large enough that naive generation exhausts memory.

The eleven implementations differ precisely on those axes — cursor-based pagination and generators for large catalogues, store-scoped entity selection, weighted ranking, blog content feeding the file with IndexNow pinging.

They look identical from outside and diverge completely at the point where the platform gets hard. That is what a young category looks like: several people solving the same problem and disagreeing about which parts matter.

The category that shrank while I was writing

In March 2026 OpenAI pulled back from in-chat Instant Checkout. Published figures put live merchants somewhere between about a dozen and around thirty, against the "over a million" named at launch six months earlier. The model moved toward product discovery inside ChatGPT with the purchase completing on the merchant's own site.

The Agentic Commerce Protocol did not disappear — its role shifted from checkout toward feeds, promotions and availability. OpenAI's developer documentation still describes Instant Checkout for approved partners, so the details are contested. The direction is not.

Not one of the five agentic checkout projects on the list is past a stable 1.0.

If you were scoping a sprint against in-chat checkout, the feed is the realistic route to being surfaced at all. The checkout integration is where the protocols are heading, not where revenue is this quarter.

Where I would start

Measure first. Bing Webmaster Tools reports citation counts and grounding queries under its AI Performance view — still the only free first-party source I know of that reports how often an AI system cited your pages. Google Search Console added generative AI reporting in June 2026, but it shows impressions rather than citations.
Fix crawler policy. Read RFC 9309 before you write a rule, then verify the rule does what you think it does.
Structured data, then llms.txt. In that order. Schema exposes machine-readable price, availability and identifiers; llms.txt points systems at the pages that matter.
Stop. Everything past this point is early, and being early is a cost rather than a badge.

The disclosure

I maintain 11 of the 42 entries on that list. My own projects are listed last within their sections rather than in alphabetical position, where they would otherwise appear first almost everywhere. I read documentation and READMEs rather than source, so I have not independently audited every implementation, and where I was unsure I marked partial rather than complete.

Corrections are welcome and never need justification — especially corrections to mine.

The full write-up, with the inclusion criteria and the parts I left out, is on my site.