DEV Community

Birchwood Data
Birchwood Data

Posted on

How to pull jobs from any company's careers page (Greenhouse, Lever, Ashby, Workday) without scraping HTML

#webscraping #api #automation #python

Most company career pages aren't really websites. They're a thin front end over an applicant tracking system (ATS), and almost every ATS exposes the same public JSON that powers its own job widget. If you need job data for a job board, a recruiting tool or labor-market research, reading that JSON is much more robust than parsing HTML.

This post shows the endpoints, the gotchas, and a ready-made way to run it on a schedule.

1. Identify the ATS from the careers URL

If the careers link looks like… ATS Board identifier
boards.greenhouse.io/<slug> or job-boards.greenhouse.io/<slug> Greenhouse <slug>
jobs.lever.co/<slug> Lever <slug>
jobs.ashbyhq.com/<slug> Ashby <slug>
<tenant>.wd5.myworkdayjobs.com/<Site> Workday <tenant> + <Site>

If the company's own /careers page doesn't show one of these, view the source. The embed script usually reveals it.

2. The public endpoints

Greenhouse:

curl "https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=true"
Enter fullscreen mode Exit fullscreen mode

Lever:

curl "https://api.lever.co/v0/postings/palantir?mode=json"
Enter fullscreen mode Exit fullscreen mode

Ashby:

curl "https://api.ashbyhq.com/posting-api/job-board/ramp?includeCompensation=true"
Enter fullscreen mode Exit fullscreen mode

Workday (a POST with paging, 20 per page):

curl -X POST "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs" \
  -H "Content-Type: application/json" \
  -d '{"appliedFacets":{},"limit":20,"offset":0,"searchText":"data engineer"}'
Enter fullscreen mode Exit fullscreen mode

Then GET .../wday/cxs/<tenant>/<site><externalPath> for each job's full description.

3. Gotchas you'll hit

  • Different shapes everywhere. Locations, remote flags and salary fields are all modeled differently, so you'll want one normalized schema.
  • Workday dates are relative ("Posted 3 Days Ago", "Posted 30+ Days Ago"), so you have to derive an approximate date.
  • Workday search caps out at about 2,000 results per query. Split big employers by keyword.
  • Monitoring needs state. To alert on new jobs, store the IDs you've already seen between runs.
  • Be polite. These are public endpoints, but keep concurrency modest and retry with backoff.

4. Or skip the plumbing

I packaged all of this as Apify Actors, which you can call from the API, schedule, or plug into n8n/Make/Zapier:

Calling one from Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("birchwood-data/workday-jobs-scraper").call(run_input={
    "careerSiteUrls": ["https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"],
    "searchText": "data",
    "maxJobsPerSite": 20,
})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["title"], "|", job["location"], "|", job["url"])
Enter fullscreen mode Exit fullscreen mode

Set onlyNewJobs: true and a daily schedule, and each run returns only the jobs that appeared since the last one.

5. Bonus: build the company list first

If you don't have a target list yet, the Y Combinator Scraper (https://apify.com/birchwood-data/yc-companies-scraper) exports the YC directory. With detectCareersPage: true, it tries to find each startup's ATS board, which you can feed straight into the job scrapers. Detection only works when the board link is visible in the page HTML.

Disclosure: I built and maintain these Actors. They're paid per result on Apify (from $0.001 per result at the time of writing).

Top comments (0)