DEV Community

Cover image for robots.txt for web scraping: rules, law and a pre-check
proxymint
proxymint

Posted on Originally published at proxymint.com

robots.txt for web scraping: rules, law and a pre-check

Originally published on proxymint.

What robots.txt means for web scraping: groups and wildcards, Crawl-delay, status codes, what the law says in 2026, and a script that checks URLs first.

TL;DR

  • Before your first request, fetch /robots.txt and find the group that names your crawler. Nothing enforces it, since RFC 9309 says the rules are not access authorization, but it's the first thing a site owner will check you against.
  • Don't trust Python's built-in parser on its own. On Python 3.11 urllib.robotparser blocks /shop/public/a even when an Allow rule covers it and lets /list?sort=asc through a wildcard Disallow, so use Protego or a release with the May 2026 fix.
  • The status code changes the meaning. A 404 means no rules apply, but a 503 or a timeout means you assume everything is disallowed and pause, which is an easy one to get backwards.
  • Legally it's mostly evidence, not a contract. The December 2025 Ziff Davis v. OpenAI ruling said robots.txt is not a DMCA access control, yet ignoring it still shows you knew the owner objected, and terms of service do the binding.

Key takeaways

  • Put the robots.txt check in front of the queue, not after the fetch. Cache it per origin for 24 hours at most, and log every URL it blocks so someone can see what the job skipped.
  • Name your crawler honestly. A product token plus a contact URL lets a site write a group for you, while borrowing Googlebot's name gets you Google's rules and a quick block after a reverse DNS check.
  • Pace per site, never per IP. Crawl-delay: 5 across fifty proxy exits still means five seconds between your requests to that site, so keep one budget per target in the scheduler.
  • Treat blocks and captchas as an answer. A 429 or a challenge on an allowed path means slow down or stop and look for the site's API, not retry from another exit.
  • If the data trains a model, store the AI signals. Parse Content-Signal and Content-Usage lines and keep them next to each document, because they're the machine-readable reservations EU rules care about.

Read the full guide on proxymint, with the commands, tables and sources.

Top comments (0)