A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.

#webscraping #python #api #cloudflare

Quick answer: {"useApifyProxy": true} with no group specified doesn't resolve to "some proxy, best effort." It resolves to the standard/datacenter group. On Patreon, that group gets a Cloudflare interactive challenge — HTTP 403, <title>Just a moment...</title>, 6 KB — on every single creator. Our laptop, testing the exact same code against the exact same URLs, saw clean 200s with a full __NEXT_DATA__ payload every time. The bug wasn't in the parser. It was in what IP address "the default" meant.

Green on the laptop, red in the cloud

Local dev looked finished. curl-cffi impersonating a browser TLS fingerprint, window.__NEXT_DATA__ present on the page, patron_count: 15820 parsed straight out of it, tests passing, the whole thing felt shippable.

Then we ran it against the actual Apify cloud build for the first time, and it failed — not one flaky creator, not a rate limit, but a non-retryable 403 on every creator in the batch. Same code. Same target. Different egress.

What the tiers actually showed

Rather than guess, we probed each proxy tier directly against the same creator page:

Tier	Status	Body
Direct (no proxy)	200	434 KB, full `__NEXT_DATA__`
Apify DATACENTER	403	6 KB, `<title>Just a moment...</title>`
Apify RESIDENTIAL-US	200	full page
Apify RESIDENTIAL-DE	200	full page

That table is the whole bug report. Direct from a residential home connection: fine. Apify's datacenter pool: a Cloudflare interactive challenge page, every time. Apify's residential pool, from two different countries: fine again.

We didn't stop at one clean run either — one clearance is not proof of reach, it's a coin flip you got lucky on. We repeated the residential probe 10 times across two different creators. 10 for 10, clean. That's the bar we hold ourselves to before calling a tier reliable: not "it worked," but "it worked ten times in a row on more than one target."

The actual root cause

The Actor's proxy input defaulted to a bare {"useApifyProxy": true}. That looks like "use Apify's proxy, whatever's sensible." It isn't a sensible-default flag — it's shorthand for the standard/datacenter pool specifically. If you want residential, you ask for it explicitly:

DEFAULT_PROXY_CONFIGURATION: dict[str, Any] = {
    "useApifyProxy": True,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US",
}

So the shipped default was the blocked tier. The code was correct the whole time — the parser worked, the retries worked, the fingerprint impersonation worked. The thing that was wrong was three keys in a config dict that nobody thought to question because the flag was named useApifyProxy: true and that sounded like enough.

Why the laptop couldn't have caught this

A local run doesn't go through Apify's proxy pool at all — it goes out your home or office IP, which Cloudflare has no reason to challenge. That means a local "it works" tells you the parser is correct and the fingerprint is plausible. It tells you nothing about what a specific production egress IP will see. "No anti-bot surface" was never a fact about Patreon; it was an artifact of which network we happened to be testing from. Once we pointed the exact same code at the exact same URLs through Apify's datacenter pool, the surface reappeared immediately.

That's the general lesson worth taking: a probe is only evidence for the network path it actually took. If your probe ran locally and your production run goes through a shared datacenter pool, you've tested a different program.

Three real edge cases while we were in there

Batch-input scraping means creators fail independently, and Patreon gives you three distinct flavors of "not a normal campaign":

Renamed or deleted creators — plain 404. Skip and continue; don't let one bad vanity URL abort the batch.
No active campaign — campaign: null in the payload for creators like jacksepticeye, who have a Patreon presence but no live campaign object to read patron/pledge numbers off of.
Page-builder redirects — some vanity URLs (veritasium, GameGrumps) 302 to a /cw/<vanity> custom page-builder template that carries no __NEXT_DATA__ blob at all, because it isn't the standard creator page.

None of those are proxy problems — they're just what "a batch of real creator URLs" looks like once you stop testing against two happy-path handles.

What we ship on top of all that

We rotate through Chrome / Firefox / Safari TLS fingerprints per request, retry with exponential backoff on 408/429/5xx, and — as of this bug — route through residential proxies by default rather than a datacenter pool that a target can and will challenge on sight. Fault isolation means one 404'd or renamed creator never takes down the rest of your batch.

🔥 Patreon Creator Scraper turns a list of creator handles into clean rows — campaign name, patron count, paid member count, pledge sum, and currency — for competitor research and creator benchmarking. $6.20 per 1,000 results, and you only pay for rows that land.

FAQ

Does {"useApifyProxy": true} give you a random or "best" proxy tier?
No. With no group specified it resolves to the standard/datacenter pool. If a target blocks datacenter ranges, you have to request apifyProxyGroups: ["RESIDENTIAL"] explicitly.

Why did local testing show a clean 200 when the cloud run 403'd?
Local runs go out your own IP, which the target has no reason to challenge. The block was tied to Apify's datacenter proxy pool specifically, and a local run never touches that pool.

Is one clean proxy test enough to trust a tier?
No — treat a single clearance as a coin flip. We repeat probes (10x across multiple targets, in this case) before calling a tier reliable.

What happens when a creator on my list has no active Patreon campaign?
The row reflects campaign: null rather than failing the run — one inactive or renamed creator never aborts the rest of your batch.

DEV Community