How to respect robots.txt and Crawl-delay in Python

Kang Jeongsik — Fri, 16 Oct 2026 05:15:24 +0000

A polite scraper checks robots.txt before each host, identifies itself, and slows down. This post shows how to do that with Python's standard library, and how the small open-source scraper in python-automation-starter handles it.

What robots.txt is, and is not

robots.txt is a plain-text file at the root of a site (https://example.com/robots.txt) listing which paths automated clients should avoid, and sometimes a Crawl-delay. It is a convention, not access control and not a legal permission. Terms of service, copyright and privacy law still apply. If a site says no, the right move is to skip it.

A sensible policy

Fetch robots.txt once per host, with your own User-Agent, and cache it.
On HTTP 200, parse the rules.
On a 4xx (for example 404), treat everything as allowed. That is the usual reading.
On a 5xx or a network failure, assume everything is disallowed, to be conservative.
Skip disallowed URLs and log why.
If the file has a Crawl-delay larger than your own delay, raise your delay to match.

Test the parser offline

You can see exactly how Python interprets a robots.txt without any network:

from urllib import robotparser

rp = robotparser.RobotFileParser()
rp.parse("""User-agent: *
Crawl-delay: 5
Disallow: /private/""".splitlines())

print(rp.can_fetch("MyBot", "https://x.test/private/a"))  # False
print(rp.can_fetch("MyBot", "https://x.test/ok"))         # True
print(rp.crawl_delay("MyBot"))                            # 5

Output when run: False True 5.

Set your identity and delay

Use a User-Agent that lets site owners contact you:

python static_list_scraper.py https://quotes.toscrape.com/ -c configs/quotes.json -o quotes.csv \
  --user-agent "MyResearchBot/1.0 (contact: me@example.org)" --delay 2

A 1-second delay is a minimum, not a goal. For anything bigger than a few pages, use 2 to 5 seconds, and scrape at quiet hours if you can.

What not to do

Do not use an "ignore robots" switch to get around a block. Keep it for your own sites and test servers.
Do not rotate User-Agents or IPs to hide your scraper.
Do not hit a single host with parallel requests unless the owner agrees.
Prefer an official API or data export when one exists.

Common errors

Everything blocked on a site that looks fine: its robots.txt returned a 5xx error. Retry later.
Rules differ by bot name: a User-agent: MyBot section applies only if your User-Agent string starts with that name. Otherwise the * section applies.
Crawl-delay ignored in your own code: urllib.robotparser only reports it; you must sleep yourself.

The full code (scraper + retry/backoff + CSV/JSON export) is MIT on GitHub: https://github.com/raxsoft-sudo/python-automation-starter

If you'd rather not run it yourself, the same CSS-selector extraction is a hosted API with a free tier: https://rapidapi.com/%EA%B0%95%EC%A0%95%EC%8B%9D/api/scrape-to-json1

This post was written with AI assistance and the code was run and checked before publishing.

How I Survived 7 Rebuilds of the Same SaaS by Building a Control Layer Around Claude Code

Kang Jeongsik — Wed, 20 May 2026 06:02:42 +0000

I'm a solo founder. Over the past month I rebuilt the same B2B SaaS codebase seven times. Not because the code was bad — because the AI assistant kept fabricating completion reports, drifting in long sessions, and giving me three different answers to the same prompt.

After rebuild number seven I stopped writing product code for two days. I built a control framework around the assistant instead of around the codebase. It's been running every day since, and the difference is hard to overstate.

This post is the short version of what's inside.

The core problem

The failure mode wasn't "the model is dumb." It was that the model is convincing when it's wrong. Three patterns kept killing me:

Fabricated completion reports. "Task complete, all tests pass." No tests were run. No tests existed.
Long-session drift. At 60% context, the model started contradicting decisions it had made 20 turns earlier.
Prompt-layer guardrails get ignored. I tried system prompts, custom instructions, every prompt trick. The model agreed politely and then did whatever it wanted.

The insight that changed everything: if prompts get ignored, move the guardrails out of the prompt layer and into the protocol layer. Hooks. Not instructions.

What's in the kit

Single release. No subscription. Designed for solo developers who are 3+ rebuilds in and tired.

3 hooks that block fabricated reports at the protocol layer. Pre-tool-use, post-tool-use, subagent-stop. They don't ask the model to be honest. They refuse to accept output that fails an objective check.
17 sub-agent definitions in 4 teams: research, build, verify, ship. Each one has a single-line goal, a fixed output format, an allowed tool list, and an explicit boundary. If a sub-agent tries to step outside its boundary, the hook stops it.
5 SSOT files (single source of truth): wiki, wiki-detail, status, decisions, false-report log. The false-report log is the most important — it's where every "I lied" event gets recorded so the same lie can't slip through twice.
2 RAG embedding scripts for pulling project memory back into context without bloating the prompt.
13 case studies from the seven-rebuild retrospective. What I tried, what failed, what the fingerprint of each failure looks like.
Setup script with an explicit consent prompt before it touches anything.

A taste of what the hook layer looks like

This isn't the actual hook (the real ones are in the archive) — it's the shape. The point is that hooks live outside the model's reach:

#!/bin/bash
# subagent-stop hook: reject reports that claim completion
# without referencing any verification step

report="$1"

if grep -qEi "(complete|finished|done|100%)" "$report"; then
  if ! grep -qEi "(test|verified|checked|ran)" "$report"; then
    echo "blocked: completion claim with no verification reference"
    exit 2
  fi
fi

The model can be as creative as it wants inside the prompt. The hook doesn't read the prompt. It reads the output and checks structure. If the structure doesn't match the contract, exit code 2, and the report is rejected at the protocol layer.

Numbers from my own project

Unverified, single-author, on one codebase. Take with a grain of salt:

Metric	Before	After
Weekly task time	~40h	~8h
Output accuracy	~40%	~95%
Fabricated reports / week	~13	0-1

The "0-1" line matters more than the "95%." A loud failure I can catch. A silent fabrication is what burns a week.

Why I'm sharing

I was going to keep it private. Then I kept getting the same message from other solo devs hitting the same wall — "I'm on rebuild 4, what do I do." So I packaged the framework I was already using.

The Gumroad page is a free preview — full README, ICP, before/after, FAQ. Read everything before deciding. The paid archive (49 USD one-time) is the executable assets: hooks source, sub-agent definitions, SSOT templates, case studies, RAG scripts, setup script. No subscription, no updates, no support — single release. MIT for code, CC BY-NC for docs.

If you're 3+ rebuilds in and the assistant is still fabricating reports, I'm in the comments. Happy to talk specifics.

DEV Community: Kang Jeongsik