Proxies for AI training data: throughput, sizing and rules

Originally published on proxymint.

How to size proxies for AI training data: why datacenter ports carry most of the crawl, where residential fits, opt-out rules and a sized 10M page job.

TL;DR

• Training-data crawls are breadth work on mostly lenient targets. Millions of pages across thousands of hosts, most of them without an anti-bot vendor, which is the job dedicated datacenter IPv4 ports are built for.
• Bandwidth is rarely the bottleneck. 10 million pages at the 22 KB median HTML transfer size is 220 GB; spread over 30 days that is about 3.9 pages per second, so politeness per host sets the pace, not the proxy.
• Datacenter is the default for training crawls. 20 dedicated IPv4 ports are a flat monthly line with no bandwidth cap and a fixed IP list to publish, while per-GB residential scales with every page.
• Opt-outs now come from three places. robots.txt tokens such as GPTBot and Google-Extended, the IETF draft Content-Usage rule with train-ai=n, and Cloudflare, which since September 15, 2026 blocks training bots by default on ad-supported pages of new domains.
• Verification favours a fixed address. Cloudflare verifies a bot by signature or by a published IP list used for nothing else, which rotating home connections cannot provide.

Key takeaways

• Size the crawl from page size, not page count. Multiply pages by the median HTML transfer size, then compare the GB total with a flat monthly port cost before choosing a tier.
• Filter Common Crawl before crawling anything. Run a custom crawl only for the hosts, pages or freshness the free archive does not cover.
• Gate every host on robots.txt and Content-Usage. Skip hosts that disallow your token, the known AI training tokens, or state train-ai=n, and treat an edge block as the site's answer.
• Run one rate limit per host across all IPs. A shared token bucket per host keeps the crawl polite no matter how many ports it has.
• Log every fetch. URL, time, exit IP, user agent, robots.txt state and status code make the corpus defensible when a rightsholder or regulator asks.

Read the full guide on proxymint, with the commands, tables and sources.