Wikimedia blocks python-requests and waves through an empty User-Agent

Quick answer

Wikimedia's REST API returns 403 to python-requests/2.31.0, 403 to Python-urllib/3.11, and 200 to a completely empty User-Agent header.

The 403 body politely asks you to "set a user-agent." Sending no user-agent string at all satisfies it. Sending your HTTP library's honest default does not. It isn't a requirement — it's a denylist of library defaults, and the error message describes the wrong rule.

The measurement 📏

Same URL, same machine, one variable changed. This is curl, so there's no impersonation layer quietly rewriting anything:

U="https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/\
en.wikipedia/all-access/all-agents/Python_(programming_language)/daily/20260901/20260903"

curl -s -o /dev/null -w "%{http_code}\n" -H "User-Agent;" "$U"                        # empty  -> 200
curl -s -o /dev/null -w "%{http_code}\n" "$U"                                          # curl/8 -> 200
curl -s -o /dev/null -w "%{http_code}\n" -H "User-Agent: python-requests/2.31.0" "$U"  # -> 403
curl -s -o /dev/null -w "%{http_code}\n" -H "User-Agent: Python-urllib/3.11" "$U"      # -> 403

(-H "User-Agent;" is curl's syntax for send the header with an empty value, not omit it.)

The empty-UA result reproduced three times in a row, so it isn't a cache artifact. And the 403 body reads:

Please set a user-agent and respect our robot policy
https://w.wiki/4wJS

Which is genuinely good advice, and also not what the server is actually enforcing.

Why this matters more than it looks 🕵️

Two very different debugging sessions come out of this.

If you believe the error message, you go looking for what makes a user-agent valid. Is there a format? Does it need a contact URL? Do I need to register? That's an afternoon.

If you measure it, you find the actual rule in four requests: a handful of well-known library defaults are on a list, and everything else is fine. Now the fix is one line and you understand its blast radius.

The general shape — and we keep re-learning this one — is that an error message is a claim by the server about the server, and it gets audited like any other measurement. It's usually approximately true. It is very often not precisely true, and "precisely" is where your bug lives.

To be clear about what we shipped: we send a real descriptive UA anyway.

USER_AGENT = "DevilScrapes/1.0 (https://apify.com/DevilScrapes)"

Not because an empty string fails — it doesn't — but because Wikimedia's robot policy asks for a way to contact whoever's generating the traffic, and that's a reasonable thing to ask of anyone running volume against a donation-funded service. The denylist is the mechanism; the policy is the point. Comply with the policy, but know the mechanism, because the mechanism is what your error handler has to reason about.

The second trap: a 404 that means four different things 🚧

Here's the same API answering two completely different questions:

GET .../Python_(programming_language)/daily/20990101/20990102   # real article, future dates
-> HTTP 404

GET .../Zzzz_No_Such_Article_Xyzzy/daily/20260901/20260903      # article that doesn't exist
-> HTTP 404

Both bodies are byte-identical:

{"detail":"The date(s) you used are valid, but we either do not have data for
those date(s), or the project you asked for is not loaded yet.
Please check documentation for more information"}

So from the response alone you cannot distinguish:

• this article doesn't exist
• this article exists but had no traffic in this window
• this project isn't loaded
• you typo'd the title

That has real consequences for how you write the client:

Never retry a 404 here. It isn't transient. Five attempts with exponential backoff on a nonexistent article is 30 seconds of sleeping to arrive at the same answer, multiplied by every bad title in a customer's list.

Never fail the run on a 404. If one title in a batch of 500 is misspelled, the other 499 are still perfectly good data. We raise a dedicated WikipediaNoDataError, log the title, and move to the next one:

class WikipediaNoDataError(RuntimeError):
    """The API answered with 404 — a genuine data gap or unknown article."""

class WikipediaApiError(RuntimeError):
    """The API was unreachable or answered with an unrecoverable status."""

Two exception types, because they deserve two different behaviours. 429 and 5xx get retried with backoff and Retry-After honoured. 404 gets logged and skipped, immediately.

This is the single most common failure mode we fix across our fleet: a recoverable, per-item error crashing the whole run. One bad row in the input shouldn't cost the customer the other four hundred and ninety-nine.

What the Actor gives you

The Wikipedia Pageviews Scraper is a traffic feed, not a content scraper — one clean row per article per day (or month):

• view counts split by access method (desktop, mobile app, mobile web)
• split by agent type (human, spider, automated) — so you can strip bot traffic out of a trend
• any date range, daily or monthly granularity
• title normalization that actually works: Beyoncé and AC/DC are underscore-normalized and percent-encoded exactly once, which is the step naive clients do twice and then wonder why everything 404s

The honest limitations 🚧

• Pageviews data starts in July 2015; earlier ranges have no data anywhere upstream.
• Recent days lag — Wikimedia finalises daily counts with a delay, so "yesterday" may legitimately be absent.
• Per-article only. Top-articles and per-project aggregate endpoints aren't exposed in this version.

Pricing

$0.20 per run plus $0.002 per row — about $2.20 per 1,000 results. A run that finds nothing costs the start fee and nothing else.

→ Wikipedia Pageviews Scraper on Apify

---

Built by Devil Scrapes. We handle the user-agent denylists, the ambiguous 404s, the encoding that has to happen exactly once, and the one bad row that shouldn't kill your run.