In 2026, the major AI companies collectively crawl billions of pages per day. OpenAI's GPTBot, Anthropic's ClaudeBot, Google's crawlers, and Meta's AI training scrapers are among the largest consumers of web bandwidth after traditional search engines.
But this crawling isn't free — it costs the AI companies real money, and it costs website operators in bandwidth and compute. We've been tracking the economics on our own site, global-chat.io, and the numbers are revealing.
Cost to the Crawler
Running a web crawler at scale requires serious infrastructure:
- Compute for HTTP requests and HTML parsing: $0.01–0.05 per 1,000 pages
- Bandwidth for downloading content: $0.05–0.12 per GB
- Storage for indexing and processing: $0.02 per GB/month
- Legal/compliance teams to handle robots.txt and copyright
At billion-page scale, this adds up to millions per month for each major AI company.
Cost to Website Operators
Every bot visit consumes server resources. For sites on serverless platforms like Vercel or Cloudflare, each bot request costs $0.000001–0.00001 in compute. Sounds tiny, but a popular site getting 100K bot visits/day pays $30–300/month just serving bots. Larger sites report 40–60% of their traffic being bots.
The robots.txt Economy
robots.txt has become the de facto negotiation tool between sites and AI crawlers. Some publishers block all AI bots. Others allow specific crawlers in exchange for partnerships — like news organizations granting access to Google in exchange for traffic.
The legal status is still evolving. Several lawsuits are testing whether ignoring robots.txt constitutes trespass.
Who Actually Benefits?
The economics are asymmetric. AI companies extract billions in training data value while website operators bear the bandwidth costs.
Some sites are fighting back: serving different content to bots, implementing CAPTCHAs, or demanding licensing fees. Others see bot traffic as free promotion — if ChatGPT or Claude references your site, that drives real human traffic.
Our Data
We've been tracking this on global-chat.io. Here's what we found:
- 10 unique AI bots visit our site regularly
- 264 total bot visits tracked so far
- 34 visits today across a small site with ~75 pages
- Bandwidth cost: negligible (~$0.01/month)
- The real question: can we turn bot visits into referral traffic?
We're running experiments to find out — testing what content attracts which bots, and whether getting crawled translates to getting cited.
This is part of our ongoing research into AI bot behavior. See our first post: How We Detect AI Bots on Our Website.
Full data and methodology at global-chat.io.
Top comments (0)