Quick answer
This Actor was shelved with a note that read "Cloudflare 403 on all WebShare residential exits" — which sounds like Wellfound blocking us. It wasn't. That note named a third-party proxy vendor, and the scraper's own code had a leftover environment variable that let that vendor's pool silently override Apify's proxy whenever it happened to be set. The Wellfound Startup Jobs Scraper pulls startup job listings — title, company, funding-stage badge, remote flag, compensation range, tags — at $0.20 per run plus $0.0015 per result, about $1.70 per 1,000 listings.
The wall that was a leftover env var 🔌
The original shelve verdict was gathered entirely through a third-party residential gateway (WebShare), and every exit in that pool came back blocked. That's a real, measured result — but it's a result about one proxy vendor's IP reputation against this target, not a verdict about Wellfound's own defenses. The shelve note read like the latter.
The deeper problem was in the code, not the recon: _build_proxy_factory checked for a WEBSHARE_PROXY_URL environment variable and, if it was set, used it instead of Apify Proxy — silently, with no log line calling out which proxy path a run actually took. A scraper wired that way can request RESIDENTIAL in its input, get validated, look correctly configured end to end, and still ship every request through a completely different, unrelated pool. You'd only find out by checking which proxy a run actually used, not which one you asked it to use.
Verify the tier you got, not the one you asked for 🔍
That's the generalizable trap: a proxy fallback that swaps tiers silently produces a "this site blocks us" verdict that's void by construction, because the verdict was never tested against the tier you meant to use. The only honest check is the proxy line in the run's own billing, after the fact — not the config object you passed in.
We re-ran the probe with exactly one variable changed: same HTTP client, same browser-fingerprint impersonation, same URL shapes, same parser — only the proxy pool swapped, from WebShare to Apify's own RESIDENTIAL group pinned to country_code=US, with a fresh exit IP requested on every single page rather than reusing one session. Result: 20 out of 20 pages came back HTTP 200, 705 rows parsed, and no degradation between page 1 and page 20. The env-var override is gone from the shipped code — not reordered, removed — so a stray environment variable can't quietly re-arm the same trap on a future run.
What you get per row 📋
| Field | What it is |
|---|---|
job_title, company
|
The listing and who's hiring |
company_stage |
Funding-stage badge (e.g. "Early Stage", "Series B"), when Wellfound publishes one |
location, remote
|
Listing location and a clean boolean remote flag |
salary_range |
Posted compensation/equity range, when published |
tags |
Company + role badges |
job_url, posted_at
|
Canonical URL and ISO-8601 posting timestamp |
Funding stage riding along on every row is the point — a role posting plus a funding badge is a buying signal most job-board scrapers don't bother joining together.
Is Wellfound hard to scrape? 🛡️
It has a real anti-bot layer, and we don't skip past that: every request goes out impersonating a genuine browser TLS/HTTP fingerprint through a residential proxy, with retries and fresh sessions on any block signal — the same handling this Actor would need even if the WebShare episode had never happened. What this specific case proves is narrower and, honestly, more useful to know: before you write off a target as blocked, check which proxy pool the failing run actually used. "Blocked" and "misconfigured" look identical from a dashboard; they only look different when you read the run's own billing line.
What it costs
Pay-Per-Event: a $0.20 actor-start fee plus $0.0015 per job listing. A thousand listings runs about $1.70. The proven run (705 rows, zero degradation) settled at a fraction of a cent per row before margin, since this Actor's HTTP-client engine carries none of a headless browser's egress overhead — pricing leaves real room above that floor. No API key, no login, and Apify gives every new account $5 of free credit to try it with no card on file.
Was Wellfound actually easier to reach than expected?
No — it sits behind a real anti-bot layer, which is exactly why this Actor drives fingerprint impersonation and residential proxy rotation on every request. The fix wasn't that the target dropped its guard; it was that our own proxy routing had a silent failure mode that made a working path look like a dead one.
Can I search a specific city instead of remote?
Yes — set location to any city name. Remote searches clear the fewest anti-bot retries within the shared budget, so they're the fastest, most reliable default; city-specific searches work but may return fewer rows per run.
Top comments (0)