Indeed 403s Chrome on every proxy tier — and our laptop said it was fine

Quick answer

Indeed's Cloudflare rules do not care which proxy tier you pay for nearly as much as they care which browser your TLS handshake claims to be. We ran the same GET against https://www.indeed.com/jobs?q=python&l=remote from an Apify container across two proxy tiers and two curl-cffi impersonation profiles — four combinations — and exactly one of the four returned data. Chrome impersonation was refused on residential. Firefox impersonation was refused on datacenter. Only Firefox-on-residential got through. And the laptop we developed on, with no proxy at all, happily returned 200 for both profiles, which is precisely why this class of bug ships.

The 2x2 that only has one green cell 🔬

One probe, one target URL, four runs, minutes apart:

Proxy tier	Impersonation	Status	Bytes	Job cards
Datacenter	chrome131	403	28,061	0
Datacenter	firefox133	403	27,848	0
Residential	chrome131	403	28,083	0
Residential	firefox133	200	1,532,810	48

Note the 403 bodies: ~28 KB, not 500 bytes. That is not a terse edge-node refusal, it is a full Cloudflare interactive-challenge document — <title>Security Check - Indeed.com</title>, with window.INDEED_CLOUDFLARE_STATIC_PAGE and PAGE_TYPE:"captcha" inside. Both blocked combinations got byte-for-byte the same challenge page, which reads as a deliberate per-request decision rather than a poisoned exit IP. (Four total requests were made. You cannot burn a proxy pool with four requests.)

The interesting cell is the one that isn't in the table. From a developer machine on an ordinary consumer connection, no proxy, chrome131 and firefox133 both returned 200 with 1,837,625 bytes and 96 job titles. So the local test suite is green, the parser is correct, the selectors are right — and the moment the same code runs from a data centre, Chrome's fingerprint gets it refused on every tier you can buy. A local pass proves your parser. It does not prove your reach.

This is why the shipped Actor leads its rotation with firefox133 and keeps Chrome out of it entirely:

PRIMARY_IMPERSONATION_PROFILE = "firefox133"
IMPERSONATION_PROFILES = ("firefox133", "firefox135", "firefox147", "safari184")

The fallbacks exist for retry diversity if firefox133 itself gets challenged mid-run — not because anyone pretends they were independently proven.

Indeed doesn't render job cards. It ships a JSON blob. 📦

Once you're past the challenge, the HTML is a red herring. The full result set for the page is assigned to a JavaScript global inside a <script> tag:

window.mosaic.providerData["mosaic-provider-jobcards"] = {...}

That is not valid JSON on its own and it isn't delimited by anything you can regex safely — the blob contains nested braces, quoted braces, and escaped quotes. We find the anchor, skip forward past the = and whitespace, then let the JSON decoder find its own end:

anchor_idx = html.find(MOSAIC_BLOB_ANCHOR)
json_start = _skip_to_value_start(html, anchor_idx + len(MOSAIC_BLOB_ANCHOR))
blob, _ = json.JSONDecoder().raw_decode(html, json_start)

raw_decode reads one complete JSON value starting at an offset and tells you where it stopped. It's the right tool whenever a payload is embedded in a larger document, and it beats every brace-counting regex anyone has ever written at 2am.

Two return types carry the whole "is this a block or a zero-match search" distinction, and they're deliberately different:

• extract_mosaic_blob() returns None — the anchor was missing or the JSON didn't parse. Something is wrong: challenged, redesigned, truncated. Loud failure.
• parse_job_cards() returns [] — the blob was well-formed and simply had no results. Your query genuinely matched nothing. That's a real answer, not an error.

A scraper that collapses those two into "no rows" is a scraper that will one day tell a customer there are no nursing jobs in Chicago.

The indefinite article that splits a salary in half ✂️

Indeed's salary snippets are free text: "$25 - $32 an hour", "Up to $145,000 a year", "From $62,000 a year". Parsing them means matching the period word after the amount — and the obvious pattern, a\s+(\w+), has a quiet bug. Against "an hour", it matches the a, then captures n as the period word. You get a period of "n", which maps to nothing, and the row silently loses its salary. The fix is one alternation:

_PERIOD_PHRASE = r"(?:a|an)\s+(\w+)"

When a snippet doesn't match any known shape, parse_salary returns all-nulls and keeps salary_text verbatim. It never guesses a currency or invents an annualisation — a wrong number in a salary column is worse than an empty one, because nobody audits a number that looks plausible.

Output

One row per job posting, JSON/CSV/Excel:

job_key, title, company, location, salary_text, salary_min, salary_max,
salary_currency, salary_period, job_type, posted_date_text, posted_date,
snippet, job_url, search_keyword, search_location, scraped_at

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/indeed-jobs-scraper").call(
    run_input={
        "queries": [
            {"keyword": "software engineer", "location": "Austin, TX"},
            {"keyword": "registered nurse", "location": "Remote"},
        ],
        "maxResultsPerQuery": 100,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], "—", item["company"], "—", item["salary_text"])

Pricing is $0.20 to start a run plus $0.003 per job row — $3.20 for a thousand postings. Queries are isolated from each other: one keyword/location pair that gets challenged is logged and skipped, and the rest of your queries still land their rows.

→ Indeed Jobs Scraper on Apify

---

Built by Devil Scrapes. We run the residential exits and the Firefox TLS fingerprint that this target actually accepts, retry with backoff on every transient status, and tell the difference between "blocked" and "no results" instead of reporting both as an empty dataset.