On 18 October 2026 we repeated our crawler check on the x402 Bazaar. Of 82 sites that blocked AI training crawlers and are still up, 80 still do.
On 18 October 2026, three weeks after our first look, we sent the same eight requests to the same x402 Bazaar sites. Of the 86 sites that had refused GPTBot and ClaudeBot, the crawlers that collect pages for AI training, on 27 September, 82 were still online, and 80 of those still refused both. Of the 770 sites that had let the training crawlers in, not one had started to block them.
Read the full post on 402registry: https://402registry.com/blog/x402-sites-still-block-training-crawlers?utm_source=devto&utm_medium=social
Top comments (1)
Interesting that the 80 holdouts didn't budge in three weeks, since that suggests the blocks are deliberate policy rather than leftover defaults. One thing I'd be curious about is whether those sites distinguish training bots from retrieval bots like OAI-SearchBot or PerplexityBot. Blocking GPTBot alone doesn't remove a site from ChatGPT search results, so the two policies can be set independently.