AI assistants are starting to shop. ChatGPT, Claude and Perplexity can now open a product page on a user's behalf, compare prices and fill a cart. Whether that works depends on two things a shop controls: what its robots.txt says, and whether its bot protection lets the visitor in at all.
On 3 November 2026 I checked 46 large online shops that sell in Germany: marketplaces (Amazon, eBay, Otto, Kaufland), electronics (MediaMarkt, Saturn, Cyberport, notebooksbilliger), fashion (Zalando, About You, H&M, Breuninger), drugstores and pharmacies (dm, Rossmann, Shop Apotheke, DocMorris), DIY, furniture, sports and pets.
Disclosure: I built the checker used here, the AI Crawler & Agent Access Checker on Apify. How to rerun this is at the end.
Three kinds of AI visitors
It helps to separate the bots, because shops treat them very differently:
- Training crawlers collect pages for model training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), meta-externalagent, plus the opt-out tokens Google-Extended and Applebot-Extended.
- AI search indexers build the index an assistant searches: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- User agents fetch a page live because a person asked: ChatGPT-User, Claude-User, Perplexity-User. This is the one an AI shopping assistant uses.
Blocking (1) keeps your content out of training data. Blocking (3) means an assistant cannot look at your product page when a customer asks it to.
Result 1: most shops block nothing
I could read robots.txt for 32 of the 46 shops (the other 14 refused or timed out for an automated client, more on that below).
- 23 of 32 do not block a single one of the 17 AI agents I check.
- 9 of 32 block at least one.
- Classic search bots (Googlebot, Bingbot, Applebot) are allowed everywhere.
The nine that block something:
| Shop | AI agents blocked in robots.txt |
|---|---|
| amazon.de | 12: all training crawlers, all three AI search indexers, and all three user agents (ChatGPT-User, Claude-User, Perplexity-User) |
| zalando.de | 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot |
| ebay.de | 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot |
| notebooksbilliger.de | 7: GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot |
| home24.de | 2: ClaudeBot, Amazonbot |
| thomann.de | 1: GPTBot |
| rossmann.de | 1: ClaudeBot |
| idealo.de | 1: Applebot-Extended |
| ikea.com | 1: Amazonbot |
Two patterns stand out.
Amazon is the only shop that blocks the user agents. Every other shop with a readable robots.txt lets ChatGPT-User, Claude-User and Perplexity-User in. Amazon.de says no to all three, while not listing its own Amazonbot. That is a clear statement: Amazon does not want third-party assistants shopping on its pages.
Zalando, eBay and notebooksbilliger block training, not shopping. They shut out GPTBot, ClaudeBot and CCBot but leave OAI-SearchBot, Claude-SearchBot and the user agents alone. So their pages can show up in ChatGPT or Claude answers and be opened on request, but should not end up in training data. All three also block Amazonbot, which is hard to read as anything other than "not for a competitor".
Result 2: the real wall is bot protection
robots.txt is a polite request. Bot protection (Cloudflare, Akamai and similar) is enforcement. For each shop I also loaded the home page as a simple automated client:
| Home page response | Shops |
|---|---|
| 200 OK | 30 |
| 403 Forbidden | 12 |
| No answer (timeout) | 2 |
| 429 Too Many Requests | 1 |
| 404 | 1 |
16 of 46 shops (35%) did not serve their home page to a plain automated client. Among them: MediaMarkt, Saturn, Kaufland, Conrad, Cyberport, Douglas, Galeria, Bauhaus, Decathlon, H&M and eBay.
Four of them (Douglas, H&M, Decathlon and apotheke.de) have a readable robots.txt that blocks zero AI agents. Their stated policy says "welcome", their firewall says "no". For an AI shopping assistant the firewall wins.
Caveat: real assistants do not send the same request as my checker. Some run a full browser, some come from IP ranges the bot vendors whitelist, and a 403 for one client is not a 403 for all. Treat this as "the shop runs aggressive bot protection", not as a precise list of who gets in.
Result 3: llms.txt has arrived in German retail
llms.txt is a proposed Markdown file that gives language models a clean map of a site. 9 of 46 shops serve one: Otto, Zalando, Tchibo, Alternate, Douglas, dm, toom, Mytheresa and the Shopify brand Ankerkraut.
Douglas is an odd case: it publishes an llms.txt but its home page returned 403 to my automated client.
Structured data is thin on home pages: Organization JSON-LD on 17 of 46, WebSite on 11. That says little about product pages, where Product markup normally lives, but it is the first thing an agent sees.
What this means if you run a shop
- Decide per bot type, not "AI yes or no". Blocking training crawlers while allowing search indexers and user agents is a coherent policy, and it is what Zalando and eBay do.
-
Check that your firewall matches your robots.txt. If you want AI assistants to recommend your products, a 403 on the home page undoes a permissive
robots.txt. - Write the decision down. Of the 32 readable files, 23 have no AI-specific rules at all. Most of those are probably no decision rather than a decision.
How to rerun this
The checker takes a list of domains and returns one row per site with the status of 17 agents, the blocked list, the home page status, llms.txt, sitemap and JSON-LD types:
curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~ai-crawler-access-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"shops": ["amazon.de", "zalando.de", "otto.de"]}'
Pricing is $5 per 1,000 sites, so the 46 shops here would cost $0.23. The run took about two minutes. Counting is then a few lines:
from collections import Counter
readable = [r for r in rows if r["robotsTxtFound"]]
print(sum(1 for r in readable if r["aiAgentsBlockedCount"] == 0), "of", len(readable), "block no AI agent")
print(Counter(a for r in readable for a in r["aiAgentsBlocked"]).most_common())
Run it monthly and diff the results: shops change these rules quietly, and the first shop to block or unblock the user agents will be a story.
Data: robots.txt and home page checks of 46 shop domains from one automated client, 3 November 2026. Snapshot only; rules and bot protection change often.
Top comments (0)