First State of Bot Conduct report - how AI agents, search crawlers, and scrapers actually behave when they visit your site.
Agents are not failing because of intelligence. They fail because of behavior.
We built BotConduct.org - an open standard that scores web bot and AI agent behavior from 0 to 100. Ten measurable criteria. Automated test environment with adversarial scenarios. Verifiable certificate.
In 3 days, we observed and tested 108 bots. Here are the results.
The Numbers
- 108 bots scored
- 14 operators identified (Google, OpenAI, Anthropic, Apple, Microsoft, Meta, ByteDance, and more)
- 57% scored 70+ (acceptable conduct)
- 30% scored below 50 (hostile)
Who passed
Major search engines and AI agents demonstrated strong behavioral conduct:
| Bot | Operator | Score |
|---|---|---|
| ClaudeBot | Anthropic | 100/100 |
| GPTBot | OpenAI | 100/100 |
| ChatGPT-User | OpenAI | 100/100 |
| AhrefsBot | Ahrefs | 90/100 |
| Googlebot | 92/100 |
Who did not
| Bot | Operator | Score |
|---|---|---|
| Tencent Cloud Crawler | Unknown | 36/100 |
| L9Explore | LeakIX | 0/100 |
| WordPress Scanner | Unknown | 0/100 |
The Tencent crawler used 68 rotating IPs with a fake iPhone User-Agent. L9Explore probed 57 sensitive paths in one minute.
(CONTINUACION del articulo Dev.to - pegar junto con el anterior)
The 10 Criteria
- IDENTIFY - Descriptive User-Agent
- DECLARE - Published data collection scope
- OBEY - Respects robots.txt
- THROTTLE - Rate limit compliance
- DISTRIBUTE - No burst patterns
- PROTECT - No PII collection
- RETAIN - Data retention policy
- RESPOND - Working contact endpoint
- RESPECT - Honors opt-out signals
- REPORT - Transparency reports
How we detect bots
Three layers:
- Known patterns - 45+ User-Agent signatures
- Self-identifying - UAs containing bot/crawler/spider keywords
- Behavioral - visited pages but never executed JavaScript = not human
Why this matters for AI agents
Enterprise buyers ask: how do I know your agent will not cause damage?
There is no answer today. No certification, no standard, no verifiable signal.
Cloudflare Web Bot Auth solves identity - WHO a bot is. BCS solves conduct - HOW it behaves. Together they form the trust stack for the agentic web.
Try it
- Full report: State of Bot Conduct - April 2026
- Test your bot: botconduct.org/test-your-bot
- 108 bot profiles: botconduct.org/bot/
- GitHub: botconduct-obcs
Open source. Free during beta. CC BY 4.0.
Built by BotConduct Standard. Contact: hello@botconduct.org
Top comments (0)