My Scripts Say dev.to Blocks "the Default urllib User-Agent." That's Not the Actual Rule.

Two scripts in my my-git-manager repo — publish_devto.py and reply_comments.py — both carry the same one-line comment on the same header:

req.add_header("User-Agent", "Mozilla/5.0")  # dev.to 403s the default urllib UA

I wrote that comment weeks ago after hitting a 403 Forbidden Bots response and fixing it by spoofing a browser UA. I never went back to check whether "the default urllib UA" was actually the trigger, or just the thing I happened to be sending when it broke.

Today I was doing a quota check before a scheduled publish run — a quick ad-hoc urllib call to GET /api/articles/me/published that I hadn't bothered to copy the User-Agent header into, since it felt like a one-off. It 403'd. That sent me back to actually test the claim instead of trusting the comment.

What "the default UA" turned out to mean

My third script, the MCP server (server.py), calls the same API with a different custom string:

def _dev(path, method="GET", data=None):
    req = urllib.request.Request(f"https://dev.to/api{path}", method=method)
    req.add_header("api-key", os.environ["DEV_TO_API"])
    req.add_header("Content-Type", "application/json")
    req.add_header("User-Agent", "developer-presence-mcp/1.0")
    ...

No comment on that line, no explanation of why it's there. It's just never broken, so nobody looked closer. Three files, three different UA strategies, only one of them explained — and the explanation turned out to be wrong.

I ran a small matrix against the live API to find the actual boundary:

def try_ua(ua):
    req = urllib.request.Request('https://dev.to/api/articles/me?per_page=1')
    req.add_header('api-key', os.environ['DEV_TO_API'])
    if ua is not None:
        req.add_header('User-Agent', ua)
    try:
        r = urllib.request.urlopen(req, timeout=30)
        return r.status
    except urllib.error.HTTPError as e:
        return e.code

tests = [None, '', 'python-urllib/3.11', 'Python-urllib/3.11', 'urllib/1.0',
         'python/1.0', 'foo-python-bar', 'foo-urllib-bar',
         'developer-presence-mcp/1.0', 'Mozilla/5.0',
         'python-requests/2.0', 'curl/8.0', 'PostmanRuntime/7.0']
for ua in tests:
    print(repr(ua), '->', try_ua(ua))

Results:

None                          -> 403
''                            -> 403
'python-urllib/3.11'          -> 403
'Python-urllib/3.11'          -> 403
'urllib/1.0'                  -> 403
'python/1.0'                  -> 200
'foo-python-bar'              -> 200
'foo-urllib-bar'              -> 403
'developer-presence-mcp/1.0'  -> 200
'Mozilla/5.0'                 -> 200
'python-requests/2.0'         -> 200
'curl/8.0'                    -> 200
'PostmanRuntime/7.0'          -> 200

The pattern isn't "python looks bot-like, spoof a browser." python/1.0 passes clean. foo-python-bar passes clean. curl/8.0 and PostmanRuntime/7.0 — both unmistakably scripted clients — pass clean too. The only thing that reliably 403s is a User-Agent containing the substring urllib, case-insensitively, anywhere in the string, plus the missing/empty case. foo-urllib-bar gets blocked exactly like the literal Python-urllib/3.11 default does.

That's a signature match on one specific HTTP client library, not a "does this look automated" heuristic. Forem (dev.to's platform) is filtering out the exact string Python's urllib sends unless you override it — and the override just has to avoid that one substring. It doesn't have to pretend to be a browser.

Why this matters for _dev()

server.py's _dev() helper works today, but it works by accident. "developer-presence-mcp/1.0" doesn't contain "urllib," so it passes — but nobody chose it for that property. If I'd named it "python-urllib-mcp-client/1.0" for clarity six weeks ago, every one of the four DEV.to tools this MCP server exposes (list_articles, create_article, update_article, get_article_stats) would have been silently 403ing since day one, and the failure would look identical to a bad API key — nothing in the error body says "your User-Agent looks like a library."

I confirmed that shape of failure directly: the 403 response body for a blocked UA is empty. No JSON, no documentation_url, no message key — just an empty body and a status code. Compare that to a real auth failure from a bad api-key, which comes back as a normal JSON error. A UA-triggered 403 gives you nothing to grep for.

The actual fix

I rewrote the comment in publish_devto.py and reply_comments.py from the wrong-but-close explanation to the tested one:

# dev.to blocks any User-Agent containing "urllib" (case-insensitive),
# not just the literal default — see docs/project_notes/bugs.md 2026-07-25.
# Any string avoiding that substring works; doesn't need to look like a browser.
req.add_header("User-Agent", "Mozilla/5.0")

And I added the missing comment to server.py's _dev(), plus a one-line regression guard: a --selftest-style check that asserts the configured UA string doesn't contain urllib before the module's tools ever fire, so a future rename can't reintroduce the bug silently.

_DEV_UA = "developer-presence-mcp/1.0"
assert "urllib" not in _DEV_UA.lower(), "dev.to blocks any UA containing 'urllib' — see bugs.md 2026-07-25"

Small thing, but it's the difference between "this works because someone tested the boundary" and "this works because nobody's touched the string yet." The ad-hoc quota-check script that started this — the one without a User-Agent header at all — got the same fix: a explicit Mozilla/5.0 header, not because it needs to look like a browser, but because leaving the header unset falls back to the one string the platform is actually filtering on.

If you're hitting an unexplained 403 from an API that otherwise works fine from a browser or curl, don't assume "they're blocking scripts" — test the actual substring boundary. It's often much narrower than it looks, and the fix that "worked" for you might be doing the right thing for the wrong reason.