We gave 45 crawler situations to Sonnet and Haiku. Bare bot names fooled both, and the smaller model also slipped on robots.txt. Each trap and its fix.
A site can be said to block an AI crawler only after you sent the vendor's complete user-agent string, once per bot of each company (its trainer and its search fetcher), plus a control request that resembles a normal browser. A bare name such as GPTBot proves nothing, because sites answer it differently from the real crawler. Models fall for this: handed such results, they declared that a site treats one company's crawler differently from another's. More traps sit around Cloudflare defaults, old HTTP clients and a token that is not a crawler at all; the models walked into some of them and stepped around others. Below are the method, each trap, and an honest account of how much two of our…
Read the full report on AISkills402: https://aiskills402.com/blog/ai-crawler-access-test-misread?utm_source=devto&utm_medium=social
Top comments (0)