Nearly every company's careers page is a skin over one of a dozen applicant tracking systems: Workday, Greenhouse, Lever, Ashby, Oracle, SAP SuccessFactors, iCIMS, Eightfold, Phenom and a few more. Each of them publishes the open jobs as JSON, because job boards and Google need to index them. Scraping a career page is mostly two problems: knowing which system and which board a company uses, and reading that board completely. Both have traps that return plausible, wrong data.
Trap 1: the name on the board is not always the company
Guessing a board from a company name ("stripe" on Greenhouse) works for startups. For big brands it goes wrong in ways that look right:
- A Greenhouse board that names itself "LinkedIn" belongs to Obsidian Security; it is the feed they post to LinkedIn. Ask a naive scraper for LinkedIn's jobs and you get Obsidian's account executives.
- "Meta" on Recruitee is a one-job account of a much smaller company. "Honda" on Paylocity is a dealership in Akron with two openings. American Honda's real site is careers.honda.com, which runs on Phenom.
- Google, Apple, IBM, Meta, LinkedIn and Tesla run careers software of their own. The honest answer for those is "not on a system I can read", not the nearest board with the right word in it.
What works is a directory built from boards that have actually answered, with one answer per name, plus a list of brands whose name must never be matched to somebody else's board.
Trap 2: the same job, twice
Citi and Netflix put Eightfold, a careers front end, over their Workday. A name finds both, and a scraper that reads both returns every job twice: Citi's "Young Apprentice - C00 - MUMBAI" came back from each. The front end serves the underlying system's requisition id, so (company, job id) identifies the job on either side. Keep one.
Trap 3: filtering after you read misses most matches
Big employers have thousands of open roles, and most scrapers read the newest N and filter by title afterwards. Measured against the newest 1,000 (300 for EY):
| Company | Title | Found by reading, then filtering | Found by the site's own search |
|---|---|---|---|
| EY (SAP) | senior consultant | 30 | 690 |
| NVIDIA (Workday) | software engineer | 196 | 442 |
| Microsoft (Eightfold) | software engineer | 177 | 415 |
| Amazon | product manager | 15 | 492 |
Every one of these systems has a keyword search in the same public feed: Workday's searchText, Amazon's base_query, Eightfold's query, SAP's q. Send the title there, keep the rows whose title actually holds it, and stop paging once a few pages in a row hold no match. The searches rank title matches first, so that rule kept 3,264 of 3,264 matches on Workday while reading 239 of the 502 pages.
Or skip all of it
I maintain Company Career Page Jobs Scraper on Apify, which does the above for 78,000 company names across 24 systems: type Stripe, stripe.com or Home Depot, add a title or a city, and get the employer's own postings. Brands that publish on their own software are named as such instead of guessed, a job seen through two systems is charged once, and filtered-out rows are not charged.
{
"companies": ["EY", "NVIDIA", "Amazon"],
"titleKeywords": ["software engineer", "senior consultant"],
"locationKeywords": ["London"]
}
It is also callable from an AI agent through the Apify MCP server.
Top comments (0)