DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Chess.com splits one player across four endpoints, and only answers Firefox

#webscraping #python #cloudflare #api

Quick answer

Chess.com's Published-Data API is free, keyless and documented — and it will still hand you two surprises:

  1. One player lives at three or four different URLs. There is no "give me this player" endpoint. Profile, per-game-type ratings, and games are separate resources you stitch yourself.
  2. It 403s a Chrome TLS fingerprint and answers a Firefox one. Same URL, same IP, same second, no auth involved.

If your Chess.com client is returning a Cloudflare "Just a moment..." page, stop buying proxies and change your impersonation profile first.

The endpoint that doesn't exist ♟️

When we built the Chess.com Player Scraper, the thing that shaped the whole design was an absence. You would expect this:

GET /pub/player/{username}          -> everything about a player
Enter fullscreen mode Exit fullscreen mode

What you actually get is a resource tree that models Chess.com's internal storage, not your question:

GET /pub/player/{username}                      -> profile: country, joined, status, avatar
GET /pub/player/{username}/stats                -> ratings, one object per game type
GET /pub/player/{username}/games/{YYYY}/{MM}    -> games, one month per request
Enter fullscreen mode Exit fullscreen mode

So "one row per player" — the shape every analyst actually wants — is a client-side join across three endpoints, and the third one is paginated by calendar month. Want the last 90 days of games? That is three more requests, and you have to do the date arithmetic yourself, including the year rollover in January.

The ratings response is its own small puzzle. It is not a list. It is an object whose keys are the game types, and which keys are present depends on what the player has actually played:

{
  "chess_blitz":  { "last": { "rating": 1543, "date": 1757000000 }, "best": {...}, "record": {...} },
  "chess_rapid":  { "last": { "rating": 1488, ... } },
  "tactics":      { "highest": { "rating": 2210, ... } },
  "puzzle_rush":  { "best": { "score": 41 } }
}
Enter fullscreen mode Exit fullscreen mode

Three things to notice, because each one breaks a naive parser:

  • tactics and puzzle_rush have a different internal shape than the chess_* keys. tactics has no last.rating; it has highest. puzzle_rush has no rating at all — it has a score.
  • Absent means "never played", not zero. A player with no chess_bullet key is not a 0-rated bullet player. If you default missing ratings to 0 you will publish a dataset that says thousands of people are terrible at a format they have never opened.
  • Timestamps are Unix epoch integers, not the ISO-8601 strings the profile endpoint uses. The same API disagrees with itself about time in two adjacent responses.

Then Cloudflare 🛡️

Our HTTP layer defaults to curl_cffi with Chrome impersonation, because that is the repo-wide default for everything we build. The first live run went out as chrome131:

HTTP 403
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>
server: cloudflare
Enter fullscreen mode Exit fullscreen mode

The useful move is not to reach for a residential proxy. It is to change exactly one variable and re-run — same URL, same machine, same minute:

from curl_cffi import requests

url = "https://api.chess.com/pub/player/hikaru"
for profile in ("chrome131", "chrome124", "firefox133", "firefox135", "safari184"):
    r = requests.get(url, impersonate=profile, timeout=25)
    print(f"{profile:12} -> HTTP {r.status_code}")
Enter fullscreen mode Exit fullscreen mode 
chrome131    -> HTTP 403   Just a moment...
chrome124    -> HTTP 403   Just a moment...
firefox133   -> HTTP 200   {"player_id":15448422,"@id":"https://api.chess.com/pub/player/hikaru",...
firefox135   -> HTTP 200   {"player_id":15448422,...
safari184    -> HTTP 200   {"player_id":15448422,...
Enter fullscreen mode Exit fullscreen mode

The IP never changed. The headers never changed. Only the TLS and HTTP/2 fingerprint did.

Why is Chrome the blocked one, when Chrome is the most common browser on earth? Because that is precisely why. Fingerprint checks don't ask "is this a plausible browser." They ask "does this TLS fingerprint match the rest of this request?" Real Chrome arrives with a Chrome fingerprint and Chrome's full header set, ordering, sec-ch-ua hints and navigation context. An impersonation library reproduces the fingerprint precisely and the surrounding context imperfectly — and Chrome, being the most-impersonated profile in existence, is the one whose mismatch has been tuned most carefully. Firefox and Safari carry less scrutiny not because they are stealthier, but because almost nobody bothers to fake them.

The most popular disguise is the most closely inspected one.

What we shipped

Not a hardcoded firefox133. One profile is one point of failure — the day Cloudflare re-tunes, every run dies at once.

# Chess.com blocks Chrome curl-cffi impersonation on this host (403 Cloudflare
# challenge) but allows Firefox — verified 2026-09-04, see spec.md "Target and
# Inputs". Never add a chrome* profile here (REQ-5).
BROWSER_PROFILES = ("firefox133", "firefox135")
Enter fullscreen mode Exit fullscreen mode

Worth stealing from that comment:

  1. Rotate across a known-good tuple, don't pin one profile.
  2. Date the finding. Anti-bot posture moves; "verified 2026-09-04" tells the next reader whether to trust it.
  3. Say what is forbidden and why. The next person to "simplify" this back to the repo default should hit the reason before they hit the 403.

And a 404 is treated as data, not as failure. A renamed or closed account is a completely normal thing to find in a list of 5,000 usernames — it lands as its own row with a skip reason, and the other 4,999 still ship.

The part that generalises 🧭

This is now at least the seventh distinct target where a Chrome fingerprint was the block, on services we had variously written off as IP problems. Grepping our own fleet for the pattern turns up: the ECB's data portal (every chrome* profile gets a 503, firefox133/safari17_0 get the full 1.3 MB envelope), Docker Hub's /v2 API, Glassdoor's Cloudflare (0/6 Chrome profiles passed), leboncoin, two separate Reddit endpoints — and now Chess.com.

Seven is the number we can document, because each of those has a dated comment in the source. The real number is higher, because for every one we probed there are targets we dismissed as "blocked" without ever changing the profile.

The durable lesson isn't about Chrome, though. It is about the sentence in our own spec that said this keyless public API needed no anti-bot handling. That was written as a statement of fact about the world, it was load-bearing, and nothing in the pipeline ever re-checked it. Mocked tests all passed. Code review doesn't catch it, because it doesn't read like a claim about our code.

Written assertions about third-party services decay, and nothing re-probes them. When a spec line asserts something about a live service, spend thirty seconds curling it — with two different fingerprints — before building on top.

What the Actor gives you

One row per username, already joined:

  • profile fields — country, join date, last-online, title, status
  • every rating record the player actually has, normalised across the chess_* / tactics / puzzle_rush shape differences, with absent formats left absent rather than zeroed
  • optional recent-games slice, with the month-by-month pagination and the January year rollover handled for you
  • ISO-8601 timestamps throughout, so the epoch-vs-string disagreement never reaches your table

One bad or renamed username never crashes the run — it is skipped and named in the run summary.

The honest limitations 🚧

  • Public data only. The Published-Data API exposes what a logged-out visitor can see; no private games, no email addresses.
  • Games are month-scoped at the source, so a wide historical sweep is genuinely many requests. The row cap is there to keep that predictable.
  • Chess.com's rating objects are theirs. If they add a new game type tomorrow, it lands as a new rating record rather than a new typed column.

Pricing

$0.20 per run plus $0.002 per row — about $2.20 per 1,000 results. A run that finds nothing costs the start fee and nothing else.

→ Chess.com Player Scraper on Apify

Built by Devil Scrapes. We handle the fingerprints, the three-endpoint joins, the month-by-month pagination and the assumptions that quietly went stale, so you get a flat table instead of a weekend.

Top comments (0)