Originally published on proxymint.
How Cloudflare pay per crawl uses HTTP 402, the crawler-price headers and Web Bot Auth, what sites and crawler operators set up, and where proxies fit.
TL;DR
- Pay per crawl charges AI crawlers per page. A Cloudflare site sets one price; an unpaid crawler gets HTTP 402 with a crawler-price header, a paying one gets the page and a crawler-charged receipt.
- Payment needs a signature. The crawler retries with crawler-exact-price or crawler-max-price inside a Web Bot Auth signature tied to a registered Ed25519 key.
- Site owners pick Charge, Allow or Block per crawler. One default price per zone, dynamic prices from the origin, and a few paths such as robots.txt that are always free.
- A proxy does not change who pays. Identity sits in the signature, so the exit network has no bearing on a charge or a block.
Key takeaways
- Log every crawler-charged value. Cloudflare leaves spend limits to the crawler operator, and every re-crawl of a page is billed again.
- Sign the payment header. crawler-exact-price or crawler-max-price must be a covered component in Signature-Input, or the paid request fails.
- Retry only InternalError. CrawlerForbidden, signature errors and PaymentFailed will not change on a retry; stop or fix the setup.
- Keep search crawlers on Allow. Cloudflare warns that charging or blocking search engine crawlers can hurt indexing; charge training crawlers instead.
- Test your own zone from outside. Check which paths return 402 and which stay free from more than one network and country before widening the charge.
Read the full guide on proxymint, with the commands, tables and sources.
Top comments (0)