Quick answer
Glassdoor doesn't run one anti-bot challenge on its review pages. It runs two, and they don't look like each other.
Land on a company's Overview page and a hard block shows a Cloudflare-branded "Just a moment..." interstitial. Clear that, navigate to the Reviews page for the same company in the same browser session, and a second challenge can appear — this one titled "Security | Glassdoor", styled as the site's own page rather than Cloudflare's. Same domain, same session, same visit, two different disguises. A scraper that only recognizes the first one will report the second as an empty product page instead of a block, and ship a customer a dataset of confident nulls.
Does a cold request straight to the Reviews URL work? 🛡️
No. This is the finding that shaped the whole Glassdoor Company Reviews Scraper's navigation order.
The obvious approach is the fast one: build the Reviews URL for a company and fetch it directly.
https://www.glassdoor.com/Reviews/{company}-Reviews-E{employer_id}.htm
Live recon against that URL, cold, gets a 403 — even routed through a residential exit. What actually clears it is a same-context warm-up: navigate to the company's Overview page first, let it settle, then navigate to the Reviews page from the same browser Page/context object. Two hops, one session, in that order, every time:
OVERVIEW_PATH_TEMPLATE = "/Overview/Working-at-{company_slug}-EI_IE{employer_id}.htm"
REVIEWS_PATH_TEMPLATE = "/Reviews/{company_slug}-Reviews-E{employer_id}.htm"
# navigate_with_warmup is the ONLY function in this Actor that calls
# page.goto() for an Overview/Reviews URL — no other function does.
Skip the warm-up hop and the Reviews page treats you as a fresh, unvetted visitor and challenges you before you ever see a review. This is the load-bearing reason a plain HTTP client — even one impersonating a real browser's TLS fingerprint — isn't enough here: the check isn't only about how the request looks, it's about the navigation history the session carries into it.
Why does the same block need two different detectors? 🔍
Because the two pages don't run the same defense. Live evidence across cloud QA runs shows the Overview page's challenge titles itself "Just a moment..." — Cloudflare's own generic interstitial — while the Reviews page's challenge, when it fires, titles itself "Security | Glassdoor", styled as Glassdoor's page rather than Cloudflare's stock one.
A detector written against only the first string works perfectly in testing (every Overview fetch gets caught correctly) and then silently fails in production the first time a Reviews-page challenge fires with the second one — because a single-marker check returns "not blocked" on its very first look. The fix is a marker set, not a marker:
CHALLENGE_TITLE_MARKERS: tuple[str, ...] = (
"just a moment",
"security | glassdoor",
"attention required",
"access denied",
"are you a human",
)
The practical failure mode this prevents is worse than a crash. page.goto() succeeds cleanly — HTTP 200, no timeout, no exception. Nothing in a naive pipeline flags this as abnormal. Without an explicit challenge check, a 200-status interstitial page parses to zero reviews and reports itself as a company with no reviews, which is a believable, unremarkable, wrong answer. We raise a dedicated exception for it instead of letting it fall through as "zero rows parsed," which is what triggers a fresh-proxy-and-retry rather than a false "no data" result.
Where do the actual review fields live? 📦
Not in visible HTML you could select() your way through. Glassdoor's Overview and Reviews pages are Next.js App Router pages, and the data ships inside React Server Components "flight" payloads — script tags that look like this:
<script>self.__next_f.push([7,"7:[\"$\",\"div\",null,{\"ratings\":{\"overallRating\":4.1,...
Each reviews array entry we confirmed against live-captured fixtures carries fields like ratingWorkLifeBalance, ratingCareerOpportunities, jobTitle.text, lengthOfEmployment, countHelpful, and reviewDateTime — but the container key wrapping that array isn't something we anchor on, deliberately. We locate the review list structurally: any JSON array whose every element carries a reviewId field, rather than by whatever Glassdoor happens to name the enclosing object this month. Container keys in RSC payloads are an internal implementation detail of the React tree; anchoring a parser to one is a bet that Glassdoor's frontend team never refactors, which is not a bet worth making.
The part that generalizes 🧭
Two rules came out of building this one:
A 200 status code is not evidence of real content. The Reviews-page challenge never times out and never raises — it just quietly hands back a page that looks successful and is an interstitial. If your only success signal is the HTTP status, you will ship "successful" runs that scraped nothing.
The same target can run more than one distinct defense, and they can look nothing alike. Testing against one company's Overview page and calling the anti-bot detection "handled" only proves you handled that one. The fix that actually holds up is a marker set with each entry traceable to a dated, live observation — not a single string that happened to work in the first ten runs.
What the Actor gives you
One typed row per company (company_summary) plus one row per individual review, from a single run:
- overall rating and all six sub-ratings — work-life balance, culture, career opportunities, compensation, senior management, diversity
- pros, cons, advice to management, job title, employment status, review date, helpful-vote counts
- company-level aggregates: CEO approval, business outlook, recommend-to-friend percentage
- per-company fault isolation — one company that fails to resolve or parse is skipped and logged; the rest of the run still completes
- Pydantic-validated rows; a field Glassdoor doesn't expose comes back
null, never guessed
The honest limitations 🚧
Anonymous access only, always — no login, no reviewer de-anonymization, ever. Glassdoor renders roughly 3 full reviews per page load for unauthenticated visitors, so collecting real volume means real page-navigation compute, not a cheap bulk export. Company-name-to-internal-ID resolution is still being hardened; passing a known employerId directly is the most reliable path today.
FAQ
Does Glassdoor block every request, or only some?
Only the pages carrying reviews and ratings. The block is a genuine two-stage challenge, not a rate-limit courtesy — cold requests get turned away regardless of how fast or slow you go.
Why would a run report fewer reviews than I asked for?
Either the anonymous-access ceiling of roughly 3 rendered reviews per page load, or a challenge that didn't clear on that attempt. The run's status message reports which, rather than silently under-delivering.
Can I get reviewer names or emails?
No — Glassdoor never exposes reviewer identity on its public pages, and this Actor doesn't attempt to de-anonymize anyone.
Do I need to supply my own proxy?
No, a proxy is wired in by default with automatic rotation on a challenge; bringing your own is optional.
Pricing
$0.20 per run, $0.01 per review row, $0.01 per company-summary row — about $10.00 per 1,000 review rows. That's the top of our usual range on purpose: full review text renders a few rows per page load without a login, and we won't use one, so real volume costs real browser-navigation compute rather than a cheap illusion of bulk.
→ Glassdoor Company Reviews Scraper on Apify
Built by Devil Scrapes. We handle the two-stage challenges, the same-context warm-up, and the React flight payloads, so you get a flat table instead of a weekend.
Top comments (0)