We run a behavioral observation network that scores how bots and AI agents conduct themselves when they visit websites. We scored 172+ operators. The results were eye-opening.
GPTBot: 8 content requests in 14 seconds
On April 17, 2026, OpenAI s GPTBot visited our network from IP 74.7.241.33 -- verified against OpenAI s own published ranges at openai.com/gptbot.json.
In a single session of 51 seconds, it made 39 requests. 8 of those went to content not visible to human visitors. All 8 in a 14-second burst.
GPTBot does not render CSS. It parses raw HTML and follows every anchor tag it finds -- visible or not. It cannot tell the difference between content meant for users and content that is hidden from the rendered page.
A 00B company s flagship crawler, navigating the web blind.
TwitterBot and ClaudeBot: zero
X Corp s TwitterBot and Anthropic s ClaudeBot visited the same pages. Same HTML. Same content -- visible and hidden.
Neither followed any hidden content.
Three crawlers. Three of the biggest tech companies in the world. Same test. Two understood what humans can see. One didn t.
The full leaderboard
This is not a cherry-picked comparison. We scored 172+ bot operators on behavioral conduct. Here is how the named operators rank:
The pattern: the biggest name does not mean the best behavior. Some of the most well-funded AI companies run crawlers less sophisticated than open-source projects with zero budget.
What happens when a crawler can t see
Hidden content exists everywhere on the web: honeypots, bot detection systems, anti-scraping layers, admin panels, internal tooling. A crawler that follows everything blindly will:
- Trigger every honeypot it encounters
- Get flagged by every bot detection system
- Scrape content it was never meant to access
- Get blocked, rate-limited, and blacklisted
This is not about ethics. This is about engineering. Rendering CSS is a solved problem. Google s crawler does it. Anthropic s does it. X s does it. OpenAI s does not.
We contacted OpenAI
We emailed opt-out@openai.com on April 17, 2026 with 48 hours notice before publication. No response as of this writing. If they respond, we will update this post.
This is Part 1 of 5
We are publishing one finding per day:
- Part 1 (today): GPTBot and hidden content
- Part 2: 194 rotating IPs with a fake iPhone User-Agent. Six days. One cloud provider.
- Part 3: The crawler that ignored its own standard
- Part 4: What bot traffic actually costs you
- Part 5: A free tool to see what is hitting YOUR site right now
Full report with research disclaimer: botconduct.org/report/april-2026/part-1
Want to see what bots do on your site? Free sensor, 30 seconds, one line of code: botconduct.org/sensor.html
Top comments (0)