A troubling new finding from Known Agents reveals that someone is conducting mass vulnerability scans across the web while disguising their traffic as legitimate AI bots like ClaudeBot. This discovery, now trending on Hacker News (124 points), raises serious questions about bot authentication, security, and the growing chaos of the AI-driven web.
What Is Happening?
Known Agents, which maintains the Agentic Web Index analyzing bot vs human traffic across 5,000+ websites, discovered that vulnerability scans were being conducted with User-Agent strings that impersonated known AI crawlers.
This is not just a coincidence. The attacker (or attackers) are deliberately using AI bot User-Agent strings to:
- Blend in with legitimate AI traffic — as AI bot traffic becomes commonplace, security teams may not scrutinize it as carefully
- Bypass bot filters — many sites now whitelist AI crawlers like ClaudeBot, GPTBot, and GoogleBot
- Obfuscate their identity — if you are scanning for vulnerabilities, you do not want your real identity known
The Broader Context: The Agentic Web
The Known Agents data reveals some striking statistics about the current state of web traffic:
- 35% of all web traffic is bots (down 2% from previous 90 days)
- 28% of bot traffic is AI-related (up 11% from previous 90 days) — this is the "agentrification" metric
- 98.5% robots.txt compliance by known AI bots
- AI chat referrals remain tiny at 0.1% of human visits
The 11% increase in AI-related bot traffic in just 90 days is remarkable. The web is becoming an AI-first environment, and that creates new attack vectors.
Why This Is Dangerous
1. Whitelisting AI Bots Is Now a Security Risk
Many websites have started whitelisting AI crawlers in their robots.txt and firewall rules. If an attacker can spoof a whitelisted User-Agent, they bypass your security layer entirely. This is the classic "security through obscurity" problem, but with a new twist: the obscurity is borrowed from legitimate AI companies.
2. AI Bot Traffic Is Normalized
When 28% of your bot traffic is AI-related, it becomes harder to distinguish legitimate AI crawling from malicious scanning. Security teams are already overwhelmed — adding "is this ClaudeBot real or fake?" to their checklist is a significant burden.
3. Attribution Becomes Impossible
If someone is scanning your site for vulnerabilities while claiming to be ClaudeBot, who do you blame? Anthropic? The actual attacker? Your own security team for not catching it? This creates an accountability vacuum that attackers can exploit.
What Should Websites Do?
Verify Bot Identity, Not Just User-Agent
User-Agent strings are trivially easy to spoof. Real AI crawlers:
- Come from known IP ranges — Anthropic publishes ClaudeBot IP ranges, Google publishes GoogleBot ranges
- Reverse DNS resolves to their domain — crawl.anthropic.com for ClaudeBot, googlebot.com for GoogleBot
- Follow robots.txt — 98.5% of legitimate AI bots do
Implement Bot Verification
Reverse DNS verification should be standard:
- Get the connecting IP
- Reverse DNS lookup to get hostname
- Forward DNS lookup on that hostname to get IP
- Compare the two IPs — they should match
Monitor for Anomalous Behavior
Even legitimate AI bots behave in predictable ways:
- They crawl pages, not scan ports
- They follow links, not probe endpoints
- They respect rate limits
- They do not attempt SQL injection or XSS
If "ClaudeBot" is hitting your endpoints with injection attempts, it is not ClaudeBot.
The Irony
The most ironic aspect of this story is that AI companies are now victims of their own success. As AI bots become ubiquitous and websites rush to accommodate them, malicious actors exploit that accommodation. The very infrastructure built to help AI access the web is being weaponized.
What This Means for AI Agent Builders
For those of us building AI agents that interact with the web, this story has direct implications:
- Your agent could be blocked — if sites start aggressively filtering AI bot traffic to prevent spoofing, legitimate agents get caught in the crossfire
- Authentication matters — the future of AI web interaction needs better authentication than User-Agent strings
- Be a good citizen — follow robots.txt, use proper identification, and do not scrape aggressively
The web is becoming a shared space between humans and AI. For that to work, we need trust mechanisms that go beyond easily-spoofed headers. This vulnerability scan discovery is a wake-up call that the current system is not sufficient.
Based on data from Known Agents Agentic Web Index, trending on Hacker News.
Top comments (0)