AI Scrapers Are Now Breaking Open Source Infrastructure — Gentoo's Bugzilla Is Just the Beginning
Gentoo Linux had to shut down its Bugzilla this week. Not because of a security breach, not because of funding cuts, not because of a dispute among maintainers. Because AI scrapers overwhelmed it.
The announcement came from Gentoo developer Michał Górny, who took the bug tracker offline after it became clear that aggressive AI bot scraping was making the service unusable for actual developers and contributors.
What Happened
Gentoo's Bugzilla — the issue tracker that Gentoo developers use to manage bugs, patches, and development work — was taken down because AI scraping bots were generating so much traffic that the service couldn't function for its intended users. This isn't a theoretical concern about AI bots being annoying. This is a critical piece of open source infrastructure being forced offline.
Gentoo is one of the oldest and most respected Linux distributions. Its package manager, Portage, was the inspiration for many modern package managers. Its bug tracker is essential infrastructure for the distribution's development workflow. When that goes down, development slows or stops.
This Is a Pattern, Not an Isolation
Gentoo's Bugzilla is the latest in a growing list of open source and community infrastructure that has been overwhelmed by AI scrapers:
- Wikipedia has reported increasing costs from AI bot traffic, forcing the Foundation to invest in additional infrastructure
- GitHub has had to implement rate limiting and authentication requirements to manage automated scraping at scale
- Project Gutenberg and other public text repositories have seen dramatic increases in automated downloads
- Stack Overflow has implemented API restrictions partly in response to AI training data collection
- Common Crawl, the non-profit that provides web crawl data, has itself been overwhelmed by the downstream effects of AI training demand
The pattern is clear: AI companies are scraping everything they can reach, as fast as they can, with no regard for the infrastructure they're loading. Open source projects, with their limited resources and community-maintained servers, are the most vulnerable.
Why AI Scraping Has Gotten This Aggressive
The current wave of AI scraper aggression is driven by several converging factors:
Model training hunger. Every AI lab wants more data. The publicly available, high-quality text on the internet has largely already been scraped. Now companies are going deeper — scraping bug trackers, mailing list archives, documentation wikis, IRC logs, and anything else that contains human-written text.
Agent-based browsing. The rise of AI agents that browse the web autonomously means the volume of automated requests is exploding. An agent doesn't just fetch a page — it follows links, fills forms, clicks buttons, and navigates through entire sites. One agent session can generate dozens of requests.
Competitive pressure. As more companies build AI products, the demand for training data has outstripped the supply of easily accessible text. Companies that don't scrape aggressively risk falling behind competitors who do.
No economic accountability. AI companies don't pay for the infrastructure they're loading. The cost of running a Bugzilla instance, a mailing list server, or a documentation wiki is borne by the open source project or its community. The AI company gets the data for free; the project pays the hosting bill.
The Real Cost
When AI scrapers take down infrastructure like Gentoo's Bugzilla, the cost isn't just server resources. It's:
- Developer time lost — bugs go unreported, patches go unreviewed, development stalls
- Contributor frustration — when the bug tracker is down, contributors can't participate, and some may not come back
- Project momentum — open source projects live on momentum. Disruptions like this can kill momentum that took years to build
- Security risk — unreported security vulnerabilities sit unfixed while the tracker is down
What Can Be Done
Open source projects are fighting back in several ways:
Rate limiting. The most immediate response — limiting requests per IP or per time window. Effective but crude, and it can block legitimate users who happen to share an IP with a scraper.
Bot detection. Tools like Cloudflare's bot management can identify and block automated traffic. But this adds complexity and can create a barrier for legitimate automated tools (CI systems, package managers, etc.).
Authentication walls. Requiring login to access content stops scrapers but also stops indexing, reduces transparency, and goes against the open source ethos.
AI-specific blocks. Some projects are blocking known AI crawler user agents (GPTBot, ClaudeBot, Google-Extended, etc.) via robots.txt or at the firewall level. This is effective for compliant crawlers but doesn't stop those that ignore robots.txt.
The nuclear option: going offline. Gentoo took Bugzilla offline entirely. This is the last resort — it stops the scraping but also stops all legitimate use.
The Bigger Picture
The AI scraping crisis highlights a fundamental tension in the current AI economy: the companies building AI models benefit from access to the entire internet's content, but they don't pay for the infrastructure that hosts that content. Open source projects, maintained by volunteers and funded by donations, are subsidizing the AI industry's training data supply chain.
This isn't sustainable. Either AI companies need to start paying for the infrastructure they're loading (through API fees, scraping licenses, or infrastructure contributions), or open source projects need better tools to protect themselves from automated scraping at scale.
The situation is also creating a perverse incentive: projects that want to remain open and accessible are the most vulnerable to being scraped into oblivion, while projects that put up walls and require authentication are protected. The open web is being punished for being open.
What This Means for You
If you maintain any open source infrastructure — a bug tracker, a wiki, a documentation site, a mailing list archive — you should be thinking about AI scraper protection now, not after your service goes down. Check your access logs. You might be surprised how much of your traffic is already automated.
And if you're building AI tools that scrape the web, consider the infrastructure you're loading. A polite rate limit and respect for robots.txt costs you nothing and saves the projects you depend on from going dark.
Gentoo's Bugzilla is back online now. But the next project might not be so lucky.
Top comments (0)