Cloudflare reported in July 2026 that AI training represented 52% of the crawler requests it identified in June, up from 22% in spring 2025. Mixed-use crawlers—combining search, agent, and training purposes—represented more than 36%.
I work with the 98IP team. My engineering takeaway is that a large IP pool is no longer a sufficient operating model. Data-collection systems need explicit crawler governance.
Give every job an identity
Store a business owner, technical owner, documented purpose, approved targets, user agent, contact address, and retention policy for every automated workflow. Do not rotate identity strings to disguise traffic.
Apply limits at the destination level
Global concurrency is too blunt. Use per-domain rate and concurrency limits, bounded queues, exponential backoff, and a small retry budget. One hundred primary requests plus thirty retries means the destination sees 130 requests, not 100.
Treat denial as a control signal
A rise in 403, 429, CAPTCHA, or challenge responses should reduce or pause traffic. It should not trigger more identity rotation. Re-check permission, authentication requirements, robots directives, and target terms.
Measure validated value
Track validated records per thousand requests, first-attempt success, p95 latency, bytes per usable record, and cost per validated result. Raw request volume rewards the wrong behavior.
Keep an immediate stop path
Every job needs an owner-accessible kill switch and an incident log. If the purpose, target, or rules change, the job should return to review before resuming.
The wider lesson is simple: proxies can support authorized regional coverage and reliability, but they should preserve—not erase—the destination's ability to manage traffic.
Disclosure: I work with 98IP. More about our infrastructure: https://en.98ip.com/?k=dev
Source: Cloudflare, “Content Independence Day, one year on: building the business model for the agentic Internet,” July 1, 2026.
Top comments (0)