An unset proxy group cost us 75% of Glassdoor's customer runs

Quick answer

An unset proxyConfiguration.apifyProxyGroups on Apify doesn't mean "use whatever's available" — it silently resolves to the account's datacenter tier. We shipped Glassdoor's salary scraper with that field unset, on the design assumption that Apify's default would fall through to residential automatically. It didn't. Thirty days of customer runs later, the success rate was 25% — 9 failures out of 12 — and our own reproduction run's billing showed zero residential-proxy transfer on a Cloudflare-blocked request. The fix was one line: hard-set apifyProxyGroups: ["RESIDENTIAL"] as the actual default, not just a suggestion in the input schema's example.

Why did "no proxy setting" not mean "pick one for me"? 📉

Glassdoor's salary pages sit behind a Cloudflare Managed Challenge, which datacenter IPs don't clear. The original design reasoned that if a caller didn't specify a proxy group, Apify would reach for something sensible. What actually happens is the opposite: an empty apifyProxyGroups list resolves to the plain datacenter pool, which is cheaper, not smarter, and Cloudflare 403s it on the first request.

The diagnosis needed a billing read, not a log read, because the run itself looked unremarkable — a 403, some retries, a timeout. What confirmed the actual cause was checking run_usage.py's proxy-transfer line for the reproduction run: zero bytes of residential transfer, on a target that requires residential to pass at all. That's the tell that the requested tier was never actually in play — the run had quietly degraded to a tier that was never going to work, and every symptom downstream (403, retries exhausting, eventual FAILED) was just that root cause playing out.

# glassdoor-salaries-scraper CHANGELOG, 0.1.0
Root cause: 30-day customer success rate was 25% (9/12 FAILED). Our own
reproduction run hit a Cloudflare Managed-Challenge 403 on the first
request, then timed out on every rotated retry — billing showed zero
residential-proxy transfer, proving the unset-groups default resolves
to the account's datacenter tier, not RESIDENTIAL as design.md assumed.

The fix made RESIDENTIAL the actual, hard-set default in both models.py's DEFAULT_PROXY_CONFIGURATION and the input schema — not a recommendation a caller could skip past.

Which browser fingerprint actually gets through? 🛡️

The same reproduction pass turned up a second, independent fix. A live 6-trial probe against this Actor's own target URLs, run over a residential proxy, found Chrome impersonation profiles 403'd on every single attempt — 0 for 6 — while Firefox cleared cleanly 4 for 4. BROWSER_PROFILES now leads with firefox147/firefox144, Safari kept as a secondary fallback, Chrome dropped from the rotation for this target entirely. It's the same signal we'd already documented on Reddit, applied here once we went and measured it rather than assuming Chrome — the most common real-world browser — is also the safest impersonation choice. It usually isn't, on targets that fingerprint aggressively: the most popular disguise draws the most scrutiny.

Where does the actual salary data live on the page? 📊

Once the request clears, the harder problem is structural. Glassdoor's salary pages are a Next.js App Router build, and Next.js App Router doesn't ship a single __NEXT_DATA__ JSON blob the way older Next.js pages do — the data arrives across many self.__next_f.push([N, "..."]) script calls, one per React Server Components "flight" chunk, in whatever order the server decided to stream them.

Two live-fixture-confirmed traps sit inside that stream:

1. The core salary fields are split across two separate chunks. One chunk carries employerId/totalPay/currencyCode/payPeriod; a different chunk carries shortName/jobTitle/twentyFifth/seventyFifth. Neither chunk alone is a complete row — parse_salary_detail_page locates both by their own anchor key and merges them.
2. The pay-mix array's id field isn't id. Design assumed a chartData array keyed by id; the live fixture keys it payType instead (BASEPAY/BONUS/STOCK/COMMISSION/PROFITSHARING/TIPS). The parser normalizes both spellings into one "id" key rather than assuming either is authoritative:

def _normalize_pay_entry(entry: Any) -> dict[str, Any] | None:
    if not isinstance(entry, dict) or "median" not in entry:
        return None
    comp_id = entry.get("id") or entry.get("payType")
    if comp_id is None:
        return None
    return {"id": comp_id, "median": entry["median"], "p25": entry.get("p25"), "p75": entry.get("p75")}

Locating either chunk means scanning the RSC stream structurally — by which key the decoded object contains — rather than by array index or chunk number, because there's no stable numbering to rely on. A quote-aware regex extracts each push([...]) call's string argument without truncating early on the first unescaped ] a large percentile array happens to contain, the same class of bug a naive non-greedy match would introduce silently.

The part that generalises 🧭

Two unrelated failure modes, same root lesson: an assumption written into a spec doesn't get re-checked once it's shipped. "Unset proxy groups falls through to something sane" and "the pay-mix array is keyed by id" were both plausible, both wrong, and both survived until real customer failures and a real fixture capture forced a re-look. Read your own billing data before trusting your own config; read a live fixture before trusting your own schema guess.

What the Actor gives you

• Base-pay and total-pay percentiles (25th/median/75th), currency, pay period, and a pay-mix breakdown per job title — optionally scoped to a specific company via title_at_company mode.
• RESIDENTIAL proxy hard-set by default, Firefox-led fingerprint rotation — both fixes from the incident above, not defaults you have to remember to set yourself.
• Pydantic-validated rows, no Glassdoor login required.

Honest limitations 🚧

base_pay_25th/base_pay_75th stay null — only the BASE pay-mix median is confirmed present on every page checked. locations is a best-effort echo of your filter, not a server-side query parameter Glassdoor actually accepts.

FAQ

Why did my run come back with fewer rows than maxResults?
Glassdoor rate-limits aggressively under bursts. We rotate fingerprints and back off automatically, but a specific title/company can genuinely have no match.

Do I need a Glassdoor account?
No — the percentile data is visible on the public salary pages without logging in.

What's the difference between title and title_at_company mode?
title pulls national aggregate benchmarks; title_at_company resolves a specific (title, company) pair, costing one extra resolution request unless you supply employerIds directly.

Why are base_pay_25th/base_pay_75th sometimes null?
Glassdoor's pay-mix breakdown confirms a BASE median but not always a 25th/75th split for that component on every page — we surface exactly what's present, never an estimate.

Pricing

$0.20 per run plus $0.002 per result row — $2.20 per 1,000 salary benchmarks.

→ Glassdoor Salaries Scraper on Apify

---

Built by Devil Scrapes. We handle the RESIDENTIAL proxy pin, the fingerprint rotation, and the two-chunk RSC merge, so you get a flat table instead of a weekend. 😈