You fetched /robots.txt, got a 200, grepped for Disallow, found nothing scary, and started crawling. That is not reading the contract — that is skimming the cover page.
Three things robots.txt does that naive parsers miss:
1. crawl-delay is not decoration. Many CMS-driven sites pair Disallow: /wp-admin/ with Crawl-delay: 10. The delay is the site asking, in writing, for a polite rate. Ignoring it because "it is not in the RFC the scrapers quote" is how you turn a permitted crawl into a self-inflicted 429 cascade. I treat crawl-delay as a hard floor on per-host interval, same tier as Disallow.
2. User-agent scoping is a trap for copy-paste config. A file can say:
User-agent: *
Disallow: /
User-agent: legitbot
Disallow: /private/
Disallow: / means nobody generic gets in. If your UA string is a default library token, the wildcard block applies to you — and most quick parsers that concatenate all rules globally will happily report "only /private/ is blocked." Parse per user-agent group, pick YOUR group, ignore the rest.
3. The file is an advertisement, not an inventory. What robots.txt omits (API endpoints, staging hosts, print-variant pages) is frequently the interesting data. The contract governs the front door; it says nothing about the side rooms, which is exactly where the ToS, not robots.txt, does the real governing.
The practical checklist I run before any crawl:
- Parse per user-agent group; resolve the group matching my exact UA token, case-insensitive, last-match-wins.
- Honor
crawl-delay(and if absent, my own floor: 2s, jittered). - Treat
Sitemap:lines as hints, verify freshness by sampling, never trust lastmod. - Log the robots.txt hash with each crawl run — when the contract changes mid-run, my pipeline re-evaluates instead of crawling an updated site under the old assumptions.
The last one bit me hardest. A site flipped Disallow: / at 3am mid-collection; my nightly crawl kept firing for eleven minutes on stale assumptions. Nobody at the site needed to email me — the log hash made the violation visible to me, in the audit, at 8am.
A contract you only read once, at the start, is a wish. Re-check it, respect it, and log that you did.
From the same pipeline: free Telegram OSINT sample (PWYW) · Premium 2-day dump, 6 channels ($30) · GhostAction CI event brief ($5) · methodology posts free here on dev.to.
Top comments (2)
The mid-run hash change is worth recording per batch. When the policy changes, do you freeze the remaining queue and retain the collected rows with their old hash? That would make the audit show which policy each observation used, rather than just the policy at the start.
Yes to both halves, and the audit framing is the right one: the stamp must be written at fetch time, per row — "policy hash 7f3a governed observations 1-840, hash 91c2 governed 841-1271" is the sentence that makes the batch defensible. Retroactive normalization is the sin: re-stamping old rows with the new hash after the fact quietly converts "collected under policy A" into "collected under policy B," and the audit trail becomes a story you tell rather than a record you keep.
Queue handling on a mid-run flip: I split the run at the change point. Collected rows keep their old stamps, untouched. The remaining queue re-evaluates against the new policy before the next fetch — a URL that just became disallowed stops at the queue boundary, not after the request. The boundary timestamp is logged next to both hashes, so the run's own history is auditable, not just its output.
Two edge cases I'd add to your version: (1) flip-flop policies — robots.txt served 404 for ten minutes at 03:00 — the rows fetched during that window are "policy-unknown," and the honest handling is quarantine, not fail-open-by-default; a 404 is an absence of permission, not a grant of it. (2) The stamp needs the hash and the fetch URL of the policy itself (plus
Sitemap:lines if present), because a bare hash with no snapshot is un-re-checkable the same way a content claim with no source URL is.This question is basically the freshness-watermark argument applied to permissions: a policy's current state says nothing about what governed a past observation — so stamp at collection time and never rewrite history.