A few days ago I wrote about getting your public IP from the command line. In the Python section I mentioned, almost in passing, that the default urllib gets a 403 from ipcomet while curl doesn't. I wrote that line from memory. It kept bothering me, because "bot protection rejects it" isn't an explanation. So I tested it properly.
Same laptop, same connection, same IP, same endpoint:
curl https://ipcomet.com/api/ip -> 200
Python urllib, default settings -> 403
Something differs between those two requests. Finding out exactly what took less time than I expected, and the answer is narrower than "Python gets blocked".
First suspect: the headers
If a server tells two clients apart, it's looking at what they send. So I wrote a tiny server that echoes back the request headers it receives, ran it on localhost (no third party involved), and hit it with four clients.
# echo_headers.py: returns the headers each client sends, in order
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
class Echo(BaseHTTPRequestHandler):
def do_GET(self):
body = json.dumps(list(self.headers.items())).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
HTTPServer(("127.0.0.1", 8777), Echo).serve_forever()
What came back, in the order the headers arrived:
| Client | Headers sent | User-Agent |
Accept-Encoding |
|---|---|---|---|
| curl 8.7.1 | host, user-agent, accept (3) | curl/8.7.1 |
not sent |
| curl --compressed | host, user-agent, accept, accept-encoding (4) | curl/8.7.1 |
deflate, gzip |
| Python 3.9 urllib | accept-encoding, host, user-agent, connection (4) | Python-urllib/3.9 |
identity |
| Node 24 fetch | host, connection, accept, accept-language, sec-fetch-mode, user-agent, accept-encoding (7) | node |
gzip, deflate |
Nothing here looks more "bot-like" for urllib than for curl. curl sends the fewest headers of all and sails through, so the header count isn't it. urllib does send two things curl doesn't by default: Accept-Encoding: identity, which tells the server not to compress anything, and Connection: close. Neither looks unusual enough to justify a block.
That left the User-Agent, the one header whose value differs in an obvious way.
Same client, different User-Agent
To separate the header from everything else, I stopped changing the client and changed only the string. Same curl, same machine, one request per User-Agent, 1.5 seconds apart, against https://ipcomet.com/api/ip:
User-Agent sent |
Status |
|---|---|
curl/8.7.1 (default) |
200 |
Python-urllib/3.9 |
403 |
Python-urllib/3.12 |
403 |
Python-urllib |
403 |
python-urllib/3.9 (lowercase) |
200 |
urllib |
200 |
my-urllib-script/1.0 |
200 |
python-requests/2.31.0 |
200 |
python-httpx/0.27.0 |
200 |
Python/3.9 aiohttp/3.9.1 |
200 |
Go-http-client/1.1, Wget/1.21.4, node, axios/1.6.0, okhttp/4.12.0, Java/17.0.2, PostmanRuntime/7.36.0
|
200 (all) |
empty (no User-Agent header) |
200 |
my-script/1.0, with or without contact details appended |
200 |
Twenty strings, 22 requests. Exactly one family was blocked: anything containing Python-urllib, with that capitalisation. The lowercase version passed. So did urllib alone and my-urllib-script/1.0, and so did every other non-browser client I tried, including an empty User-Agent.
Two reverse checks agree. curl sending Python-urllib/3.9 got the 403 all three times I tried it, and Python's urllib sending my-script/1.0 got a 200.
What the 403 looks like
The response is a 403 with a body of 17 bytes that contains the code 1010. That is Cloudflare's error for access denied based on the browser's signature, and ipcomet is served through Cloudflare. I didn't trace which rule produced it. The matrix already tells a client everything it needs to know about what the rule matches. It's a plain string match on a known default, which is why it's so easy to hit by accident and so confusing to debug. Nothing about the response says "change your User-Agent".
What I'd do about it
- Set a descriptive
User-Agenton every script, something likemy-tool/1.0 (+you@example.com). It costs nothing, and it gives a site owner someone to contact instead of a reason to block you. - Don't copy a browser's string to get around a block. A blocked default is a hint to identify yourself properly, and pretending to be Chrome sends the wrong signal.
- When a script gets a 403 with a tiny body, check the
User-Agentfirst. It takes ten seconds.
In urllib the fix is one argument:
import urllib.request
req = urllib.request.Request(
"https://ipcomet.com/api/ip",
headers={"User-Agent": "my-tool/1.0 (+you@example.com)"},
)
print(urllib.request.urlopen(req, timeout=10).read().decode().strip())
Run the same test on your own site
This is the script I used, rewritten to use only the standard library. Only point it at a site you own or have permission to test. It sends one request per string with a pause between them.
import sys, time, urllib.error, urllib.request
URL = sys.argv[1]
AGENTS = [
"curl/8.7.1", "Python-urllib/3.9", "Python-urllib/3.12", "Python-urllib",
"python-urllib/3.9", "urllib", "my-urllib-script/1.0", "python-requests/2.31.0",
"python-httpx/0.27.0", "Go-http-client/1.1", "Wget/1.21.4", "node",
"axios/1.6.0", "okhttp/4.12.0", "Java/17.0.2", "PostmanRuntime/7.36.0", "my-script/1.0",
]
def status(agent):
req = urllib.request.Request(URL, headers={"User-Agent": agent})
try:
return urllib.request.urlopen(req, timeout=20).status
except urllib.error.HTTPError as e:
return e.code
except Exception as e: # timeouts and resets are results too
return type(e).__name__
for agent in AGENTS:
print(f"{agent:<26} {status(agent)}")
time.sleep(1.5)
Run it with python3 ua_matrix.py https://your-site.example/. I ran exactly this version against ipcomet as well, with urllib doing the asking instead of curl, and every status matched the table above.
What this doesn't tell you
- It's one site's configuration. Other sites, including most of the ones you scrape, will behave differently. This maps one rule.
- I tested the libraries by their strings.
requestsandhttpxweren't installed on this machine, so those rows come from curl sending theirUser-Agent. The real libraries also send other headers of their own, which could matter on a stricter site. - Rules change. This is what one site did when I tested it.
If your lookups to ipcomet fail with a 403, the API reference already lists this under client notes.
I maintain ipcomet, a free IP checker. AI-assisted: I used Claude to help draft this post and to run the tests. The results come from my own machine, and every request in the matrix went to my own site.
Top comments (1)
Running the echo server on localhost to compare what each client actually sends is a neat way to take guessing out of it. Thanks for testing it properly instead of leaving it as a throwaway line.