Five problems we keep finding on AI-built and x402 sites: empty HTML, soft 404s, two hostnames, no sitemap, a firewall. A curl test and a fix for each.
Twice now we have fetched sites exactly as an AI crawler would: 104 built with AI app builders on 20 September, and 927 x402 Bazaar sites on 27 September. The same five problems came up in both. None of them shows in a browser, so the owner rarely knows. Here is each one, the one-line test, and the fix.
Read the full post on 402registry: https://402registry.com/blog/fix-sites-ai-crawlers-cannot-read?utm_source=devto&utm_medium=social
Top comments (0)