Quick answer: our Upwork scraper cleared Cloudflare 4/4 on a laptop and then scored 0 out of 19 on the first nineteen cloud attempts. Nothing about Upwork changed in between. The fix was one boolean — geoip=True on the browser — and the next ten attempts scored 9. The full 2x2: bare cloud IP 0/7, residential proxy without geoip 0/12, residential proxy with geoip 9/10. Both conditions are required and neither one is sufficient.
Why does a residential proxy alone score zero against Cloudflare?
Because the proxy fixes your IP and says nothing about your browser, and Cloudflare compares the two.
An anti-bot browser like Camoufox spoofs a coherent fingerprint — timezone, locale, screen, WebGL. By default it spoofs a plausible one, not one that matches wherever your proxy happens to exit. So you present a browser that says "I am in New York" over an IP that geolocates to Frankfurt. A real user cannot be in two places at once, and the session gets parked on an interstitial forever.
geoip=True tells Camoufox to look up the exit IP's actual geolocation and build the fingerprint around it. Exit and fingerprint agree, and the challenge resolves.
Here is the part worth internalizing: the library warned us the entire time. Camoufox emits a LeakWarning on every launch — "When using a proxy, it is heavily recommended that you pass geoip=True." We had spent a week on proxy-tier theories with the answer printing in our own logs on every run.
Why did the laptop pass when the cloud failed on identical code?
This is the control that saved the whole investigation, and it is worth copying.
The first cloud result was 0/4. That is a very natural place to file "Upwork blocks the Apify cloud" and move on. Two controls stopped us:
- Same code on the laptop: 4/4 cleared. So the code works and Upwork had not gone hostile that morning.
- The laptop's browser, routed through the same residential proxy pool: 3/3 cleared.
Control 2 is the one that mattered. The identical IP pool that fails 0/12 from a cloud container succeeds 3/3 from a laptop. That single result rules out the IP as the cause and points the finger squarely at browser configuration — which is what motivated testing the geoip flag at all.
A reachability verdict is only valid for the instrument that produced it. "Upwork blocks us" was never a fact about Upwork; it was a fact about one misconfigured browser.
Our recon was scoring its own successes as failures
While auditing the instrument we found a bug that had been silently corrupting the numbers.
Playwright's page.content() raises page is navigating and changing the content — and it raises it precisely when Cloudflare's interstitial hands off to the real page. In other words, it throws on the success path. Uncaught, it had already recorded two genuine local clearances as failures.
A recon that counts its own successes as failures will file a NO-GO, and nobody ever re-checks a NO-GO. This is how targets get written off permanently for reasons that were never about the target.
Why a 403 from Upwork is not a block
Read this before you write a retry rule.
A cleared page came back with HTTP 403 — and 1,295,295 bytes of fully rendered job listings, title Search Freelance Jobs on Upwork, ten distinct job IDs in the DOM. Cloudflare serves the challenge under 403 and the challenge resolves in place, so the status of the first navigation tells you nothing at all. The parsed record count is the only honest signal.
The failures look different, and the difference is the useful part. Every failure was byte-identical in kind: 403, ~345 KB, title Just a moment.... Not one said Access denied.
A stall body is a challenge still working. A denial body is a decision already made. Retrying the first one pays; retrying the second one just burns proxy budget.
Across 29 cloud attempts and 20+ distinct exit IPs, Upwork never once hardened into a denial — which is exactly why retry-on-a-fresh-context is the right strategy here.
Does the clearance survive pagination?
Yes, and this is what decides whether the thing is a product rather than a demo.
Page 1 arrives under 403-with-data. Page 2, in the same session, returns a clean 200 — 10 job IDs, zero overlap with page 1. The clearance cookie persists, so the challenge is paid once per session, not once per page. That single fact drives the architecture: walk as many pages as you can per cleared context instead of re-minting one.
It also drives the economics. The dominant cost is residential proxy transfer at roughly 4.3 MB per cleared attempt — so amortizing one clearance over many pages is the difference between a viable price and an absurd one.
What you get per row
| Field | Notes |
|---|---|
job_id / title / source_url
|
Listing identity and canonical Upwork URL |
posted_age_text / posted_at
|
Raw verbatim text plus a parsed timestamp |
price_type / price_min / price_max
|
hourly or fixed, with the parsed range |
experience_level |
Entry / Intermediate / Expert, as Upwork labels it |
search_query / page_number / scraped_at
|
Provenance for every row |
What does it actually cost?
Pay-Per-Event: $0.20 per run start + $0.004 per job — $4.20 per 1,000 jobs.
The start fee only fires once a query has actually delivered a row. A run that cannot clear Cloudflare bills you nothing at all — not the start fee, not a partial. We shipped that gate deliberately: at a ~90% clearance rate, roughly one run in ten will fail, and charging a flat fee for a zero-row run is indefensible.
FAQ
Does this need an Upwork login?
No. It reads the public job-search pages only.
Why does it default to residential proxy instead of something cheaper?
Because it was measured, not assumed. Bare cloud IPs scored 0/7 across two separate configurations. Residential is not a preference here; it is the only tier that has ever worked.
What happens when a page gets stalled mid-run?
It retries on a fresh browser context. A Just a moment... body is a challenge still in progress, not a ban, so retrying is correct — and when retries are genuinely exhausted the run fails loudly instead of quietly returning a short dataset.
Will one bad query ruin a multi-query run?
No. Each query is isolated — one failure never discards the rows the others already delivered.
Do I ever get charged twice for the same job?
No. Rows are de-duplicated by job_id within a run before anything is billed. We found and fixed exactly that defect in a pre-release build, where a 40-row run contained 37 distinct jobs and billed for 40.
😈 Upwork Job Leads Scraper turns Upwork's job search into structured lead rows — title, budget range, experience level, posted date, canonical URL — across as many keyword searches as you want in a single run. We handle the Cloudflare challenge, pin the proxy tier and align the browser fingerprint to it, amortize each clearance across pages, and isolate a stalled query from the rest of your batch. $4.20 per 1,000 results, and a run that can't get through bills you nothing.
Top comments (0)