DEV Community

jf lu
jf lu

Posted on

I tracked AI crawlers on my new GEO site for 30 days — here's what I learned

#ai

I launched a site focused on Generative Engine Optimization (GEO) about 2 months ago. Instead of just tracking Google rankings, I set up Cloudflare D1 logging to track every AI crawler visit.

Here's what 30 days of data shows:

Crawler breakdown:

  • Bingbot: 98 hits/day average (highest volume)
  • ClaudeBot: 18-22 hits/day (steady, consistent)
  • GPTBot: 8-12 hits/day (sporadic bursts)
  • PerplexityBot: 3-5 hits/day (small but high-quality)
  • Bytespider: 500+ hits/day (massive scanning, likely training data)

Key insights:

  1. Bing is the new Google for AI. Bingbot volume dwarfs everything else. If you're optimizing for GEO, Bing Webmaster Tools is your best free analytics source.

  2. ClaudeBot is remarkably consistent. Same crawl pattern every day, suggesting scheduled indexing rather than on-demand. If you publish content, expect 24-48hr lag before Claude "sees" it.

  3. Bytespider (ByteDance) is aggressive. 500+ hits/day on a small site. They're clearly doing massive data collection. If you want visibility in Doubao/DeepSeek, your content needs to be crawlable.

  4. External referrers matter more than I expected. A single Reddit post drove 8 AI crawler visits from a domain I'd never heard of (sahammurah.com). External mentions = AI discovery signals.

What I'm doing differently now:

  • Added llms.txt (AI-specific robots.txt equivalent)
  • Every article now has FAQPage + Article Schema
  • Answer-first paragraphs (first sentence = direct answer)
  • Verifiable data points with source citations

Anyone else tracking AI crawlers? What patterns are you seeing?

Top comments (0)