Stack Overflow serves its robots.txt with HTTP 418, and Python reads that as "crawl everything"

I check robots.txt before every host my scripts touch. On 2026-09-23 my own
checker told me stackoverflow.com/questions/ask was open to crawlers. It is
not. The file says Disallow: /, and I had thrown it away without reading it.

Here is the whole thing, and you can rerun every line.

The response code and the file disagree

$ curl -s -o body.txt -w "%{http_code} %{size_download}\n" \
    -A "my-crawler" https://stackoverflow.com/robots.txt
418 113

$ cat body.txt
License: https://stackoverflow.com/license.xml

User-agent: *
Content-signal: search=no, ai-train=no
Disallow: /

HTTP 418 is "I'm a teapot". The body of that teapot response is the real
robots.txt, rules and all. Same answer with three different user agents
(curl/8.14.1, python-urllib/3.11, and my own string): 418, 113 bytes,
identical content. meta.stackexchange.com and stackexchange.com answer the
same way.

Most HTTP clients raise on a 4xx and never look at the body. Mine did. So did
the standard library:

import urllib.robotparser as r
p = r.RobotFileParser()
p.set_url("https://stackoverflow.com/robots.txt")
p.read()
print(p.can_fetch("my-crawler", "https://stackoverflow.com/questions/ask"))
# True

RobotFileParser.read() catches the HTTPError, sees a code in 400-499, sets
allow_all = True, and returns. The file that says Disallow: / is sitting in
the response it just discarded.

The same network, the opposite error

stackapps.com is the Stack Exchange site where you register an OAuth
application. Its robots.txt is served normally, HTTP 200, 4850 bytes, and the
User-Agent: * block ends with Allow: /. The site is open.

p = r.RobotFileParser()
p.set_url("https://stackapps.com/robots.txt")
p.read()
print(p.can_fetch("my-crawler", "https://stackapps.com/"))
# False

Two separate behaviours of urllib.robotparser produce that, and neither is
about Stack Apps:

1. RuleLine.__init__ runs the pattern through urlparse(...).path, which keeps only the path component. The line Disallow: /?*, written to block query strings on the root, becomes the pattern /.
2. can_fetch returns the first rule that matches, in file order. Disallow: /?* is rule 79 of that block, and Allow: / is rule 109, the last one. The first match wins, so everything is refused.

RFC 9309 section 2.2.2 says the opposite: the longest matching pattern wins,
and on equal length Allow wins. Out of 109 rules in that block, exactly two
reduce to the path / after urlparse, and they are Disallow: /?* and
Allow: /.

So on one afternoon, on two hosts of the same network, the standard library got
both verdicts backwards. It opened the one that is closed and closed the one
that is open.

The part that argues against my own title

RFC 9309 section 2.3.1.3 is explicit: when fetching robots.txt returns a 4xx,
"the crawler MAY access any resources on the server". By the letter of the
standard, urllib.robotparser is right to allow everything on a 418, and the
unusual behaviour is Stack Overflow's, which ships rules inside an error
response. I am not going to pretend otherwise to make a better headline.

What I would still argue: 404 and 410 mean the file is absent, while 401, 403,
418 and 429 mean the server refused me. Reading "the server refused to show
me the rules" as "there are no rules" is a choice, and on this host it lands on
the exact opposite of what the operator wrote down, twice over
(Disallow: / plus Content-signal: search=no, ai-train=no).

What I changed

My checker now does three things instead of one.

• If the body contains at least one User-agent line and at least one Allow/Disallow line, parse it and apply it, whatever the status code was. A file of rules read word for word is not an absence of rules.
• Body with no rules and a 404 or 410: the file is absent, RFC 9309 applies, crawling allowed.
• Body with no rules and any other non-200 (401, 403, 418, 429, 5xx): no verdict. Not "allowed". I have not measured anything, and an absence of measurement is not a permission.

That third branch also fired somewhere I did not expect.
api.stackexchange.com/robots.txt returns HTTP 400 with an API error in
JSON ({"error_id":404,"error_message":"no method found with this name"}). The
old rule read that 400 as "no robots.txt, help yourself". There is no
robots.txt there, but the way I learned it was by being told no.

Six real cases and one negative control, all on live hosts:

URL	file says	old code	new code	stdlib
stackoverflow.com/questions/ask	Disallow: / (HTTP 418)	allowed	refused	allowed
stackapps.com/apps/oauth/register	Allow: / (HTTP 200)	allowed	allowed	refused
api.stackexchange.com/2.3/questions	no file (HTTP 400)	allowed	no verdict	allowed
www.pulsemcp.com/submit	Allow: /submit$ after Disallow: /	allowed	allowed	refused
www.pulsemcp.com/<unlisted path>	Disallow: / wins	refused	refused	refused
business-software.com/add-your-product/	apex and www disagree	divergent	divergent	n/a

The negative control matters more than the six. I put the old rule back
(4xx means allowed, discard the body) and reran the first row: it returns to
allowed, on a host whose file says Disallow: /. A guard that you cannot
make fail on purpose is not a guard, it is a comment.

Why I was there at all

I was answering a narrow question: is there a path by which a script can post a
text on Stack Overflow? The documented answer, read on
api.stackexchange.com/docs, is POST /questions/{id}/answers/add, and it
needs an access_token carrying write_access. Every endpoint of that OAuth
flow lives on stackoverflow.com/oauth, stackoverflow.com/oauth/dialog,
stackoverflow.com/oauth/access_token, or www.stackexchange.com/oauth/login_success.

All four of those hosts serve Disallow: /. The write API exists, is fully
documented, and its only door is on hosts closed to scripts. Registering the
application on stackapps.com works, and it buys nothing, because the next
door in the chain is shut. One open door in a closed chain opens nothing.

One more thing I found funny enough to keep: the document that grants the
permission, the API Terms of Use, is served from
stackexchange.com/legal/api-terms-of-use, on a host whose robots.txt says
Disallow: /.

Source of the API documentation and of the robots.txt files quoted here:
the Stack Exchange Network. Measurements taken 2026-09-23, reproducible with
the commands above.