DEV Community

SteadyScrape
SteadyScrape

Posted on Edited on

How to Pull Job Postings Straight From Company Career Sites (Greenhouse, Lever, Ashby, Workday) as JSON

#webscraping #python #jobs #api

If you want job postings from specific companies (for a niche job board, a hiring-signal dashboard, salary research, or just an alert for your dream employer), you usually don't need to scrape HTML at all.

Most company careers pages are powered by an applicant tracking system (ATS) like Greenhouse, Lever, Ashby or Workday. These systems publish the job board through public endpoints, because the whole point is for the jobs to be seen. The careers page you look at in a browser is often just a thin front end over one JSON call.

This post shows how to call those endpoints directly with a few lines of Python, and what tends to go wrong once you do it for more than one company. Every endpoint below was tested with curl and the Python snippets were run on October 26, 2026.

Step 0: find the ATS and the "slug"

Open the company's careers page and click any job. The URL usually gives it away:

ATS Job URL looks like Slug
Greenhouse job-boards.greenhouse.io/stripe/jobs/... stripe
Lever jobs.lever.co/spotify/... spotify
Ashby jobs.ashbyhq.com/ramp/... ramp
Workday nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/... tenant nvidia, data center wd5, site NVIDIAExternalCareerSite

If the careers page lives on the company's own domain, the "Apply" button often links out to the ATS.

Greenhouse 

import requests

HEADERS = {"User-Agent": "my-job-feed/0.1 (contact: you@example.com)"}

def greenhouse(slug):
    url = f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs"
    data = requests.get(url, params={"content": "true"}, headers=HEADERS, timeout=30).json()
    for j in data["jobs"]:
        yield {"id": f"greenhouse:{slug}:{j['id']}", "title": j["title"],
               "location": j["location"]["name"], "url": j["absolute_url"],
               "posted": j.get("first_published")}
Enter fullscreen mode Exit fullscreen mode

greenhouse("stripe") returned 714 jobs in one request when I ran it (data["meta"]["total"] tells you the count). Leave out content=true if you don't need the HTML description; the response is much smaller.

Lever 

def lever(slug):
    url = f"https://api.lever.co/v0/postings/{slug}"
    data = requests.get(url, params={"mode": "json"}, headers=HEADERS, timeout=30).json()
    for j in data:
        yield {"id": f"lever:{slug}:{j['id']}", "title": j["text"],
               "location": j["categories"].get("location"), "url": j["hostedUrl"],
               "posted": j["createdAt"]}  # epoch milliseconds, not ISO
Enter fullscreen mode Exit fullscreen mode

lever("spotify") returned 89 postings as a plain JSON array. Lever also accepts skip and limit if you want to page through it. Note that createdAt is a Unix timestamp in milliseconds, and EU-hosted boards live on api.eu.lever.co instead.

Ashby 

def ashby(slug):
    url = f"https://api.ashbyhq.com/posting-api/job-board/{slug}"
    data = requests.get(url, params={"includeCompensation": "true"}, headers=HEADERS, timeout=30).json()
    for j in data["jobs"]:
        yield {"id": f"ashby:{slug}:{j['id']}", "title": j["title"],
               "location": j["location"], "url": j["jobUrl"],
               "posted": j["publishedAt"],
               "pay": (j.get("compensation") or {}).get("compensationTierSummary")}
Enter fullscreen mode Exit fullscreen mode

ashby("ramp") returned 173 jobs, descriptions included, and with includeCompensation=true many of them carry a pay range like $211.4K – $290.6K • Offers Equity. Ashby is the friendliest of the bunch for salary data.

Workday (the hard one)

Workday has no documented public API for this, but every Workday careers site calls an internal JSON endpoint you can see in your browser's network tab. It's a POST, it's paginated, and it has quirks:

import time

def workday(tenant, dc, site, max_jobs=100):
    url = f"https://{tenant}.{dc}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs"
    offset, total = 0, None
    while offset < max_jobs:
        body = {"appliedFacets": {}, "limit": 20, "offset": offset, "searchText": ""}
        page = requests.post(url, json=body, headers=HEADERS, timeout=30).json()
        for j in page["jobPostings"]:
            yield {"id": f"workday:{tenant}:{site}:{j['externalPath']}",
                   "title": j["title"], "location": j.get("locationsText"),
                   "path": j["externalPath"], "posted": j.get("postedOn")}
        if total is None:
            total = page["total"]  # only the first page reports it; later pages say 0
        offset += 20
        if offset >= total:
            break
        time.sleep(1)  # be polite

jobs = list(workday("nvidia", "wd5", "NVIDIAExternalCareerSite", max_jobs=60))
Enter fullscreen mode Exit fullscreen mode

Things I hit while testing this against NVIDIA:

  • limit maxes out at 20. Asking for 50 returned HTTP 400.
  • total is only set on the first page. Pages after that report total: 0, so a naive while offset < page["total"] loop stops after page two. (I made exactly that mistake in my first version.)
  • The listing is thin. You get a title, a location string, a relative date like "Posted Today", and a path. The description, exact date and job ID need one extra request per job to .../wday/cxs/{tenant}/{site}{externalPath}.
  • Large boards seem to cap at 2,000. NVIDIA reports exactly total: 2000. Use searchText or facets to narrow it.
  • The data center (wd1, wd3, wd5, ...) and site name differ per company, so you can't guess them from the company name.

Pitfalls once you go beyond one company

Bad slugs return 404. Greenhouse, Lever and Ashby all return 404 for an unknown board. Handle it per company so one typo doesn't kill the whole run. Companies also switch ATS now and then, so a board that worked last month can 404 today.

Be polite. These are public endpoints, not an invitation to hammer them. Send a descriptive User-Agent, keep concurrency low (a handful of parallel requests), retry 429 and 5xx with exponential backoff, and don't poll more often than you need to. For most use cases, once a day is plenty.

Normalize early. Every ATS names things differently (text vs title, epoch ms vs ISO 8601, a location string vs an object). Map each one into your own schema right away and keep the raw payload around in case you need a field later.

Dedupe with a stable key. Use platform:slug:job_id, as in the snippets above. Titles aren't unique, and the same job can appear in several locations.

Track new and removed jobs. For alerts, what you usually want is the diff, not the full list. Store the IDs you've seen and compare:

import json, pathlib

def diff(board_key, jobs, state_file=pathlib.Path("seen.json")):
    state = json.loads(state_file.read_text()) if state_file.exists() else {}
    seen = set(state.get(board_key, []))
    current = {j["id"]: j for j in jobs}
    new = [j for jid, j in current.items() if jid not in seen]
    removed = sorted(seen - current.keys())
    state[board_key] = sorted(current)
    state_file.write_text(json.dumps(state))
    return new, removed

new, removed = diff("greenhouse:stripe", list(greenhouse("stripe")))
Enter fullscreen mode Exit fullscreen mode

One catch: only mark a job as "removed" if you fetched the complete board. If you stopped paginating early (a Workday cap, a timeout, a keyword filter), missing jobs aren't necessarily gone.

Then there are the other ATSs. BambooHR, Workable, Recruitee, SmartRecruiters and Personio each have their own public endpoint, their own field names and their own pagination. None of them are hard on their own. Keeping nine of them working, normalized and tested is where the time goes.

If you'd rather not maintain this yourself

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

We packaged all of the above into an Apify Actor: ATS Job Feed (steady-scrape/ats-job-feed). What it does:

  • Covers 9 ATS platforms: Greenhouse, Lever, Ashby, Workday, BambooHR, Workable, Recruitee, SmartRecruiters and Personio, all mapped into one output schema.
  • Accepts board URLs, job URLs, platform:slug, or a company name or domain that it looks up in a bundled list of 381 boards (e.g. stripe.com). Names it can't find are reported, not guessed.
  • Has an "Only new jobs" mode that remembers what it already delivered and can also output removed-job records (only when it saw the full board listing, per the catch above).
  • Supports a per-company cap (maxJobsPerCompany), title/location/department/date filters, and optional full descriptions.
  • Pricing is pay-per-event: $1.50 per 1,000 jobs delivered. Removed-job records and the run summary are free.

Example input:

{
  "companies": [
    "https://job-boards.greenhouse.io/stripe",
    "lever:spotify",
    "ashby:ramp",
    "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"
  ],
  "keywords": ["engineer"],
  "excludeKeywords": ["intern"],
  "locations": ["US", "Canada"],
  "maxJobsPerCompany": 100,
  "includeDescription": true,
  "onlyNewSinceLastRun": true,
  "emitRemovedJobs": true
}
Enter fullscreen mode Exit fullscreen mode

One output record. This is real data from one of our test runs (October 25, 2026), with the descriptions cut short here:

{
  "jobId": "greenhouse:discord:8806482002",
  "sourceJobId": "8806482002",
  "platform": "greenhouse",
  "boardSlug": "discord",
  "boardUrl": "https://job-boards.greenhouse.io/discord",
  "company": "Discord",
  "title": "Commercial Policy Lead",
  "department": "Policy",
  "team": null,
  "locations": [
    {"raw": "San Francisco Bay Area", "city": "San Francisco Bay Area", "region": "California",
     "country": "United States", "countryCode": "US", "isRemote": false}
  ],
  "locationText": "San Francisco Bay Area",
  "isRemote": false,
  "workplaceType": null,
  "employmentType": null,
  "salary": {"min": 192000.0, "max": 216000.0, "currency": "USD", "interval": "YEAR",
             "source": "structured", "raw": "Base Salary"},
  "postedAt": "2026-09-15T16:52:11Z",
  "updatedAt": "2026-10-21T00:27:20Z",
  "jobUrl": "https://job-boards.greenhouse.io/discord/jobs/8806482002",
  "applyUrl": "https://job-boards.greenhouse.io/discord/jobs/8806482002",
  "requisitionId": "R-107406",
  "descriptionHtml": "<div class=\"content-intro\"><p>Discord has a highly engaged c...",
  "descriptionText": "Discord has a highly engaged community of millions of daily ...",
  "changeType": null,
  "scrapedAt": "2026-10-25T21:07:46Z"
}
Enter fullscreen mode Exit fullscreen mode

Fields the ATS doesn't publish come back as null rather than guessed (Greenhouse, for example, doesn't expose employment type here). For scale: a run with the default input (Stripe, Spotify, Ramp and NVIDIA, capped at 100 jobs per company) returned 389 jobs in about 29 seconds on Apify. Most of that time is the per-job detail requests on Workday.

It's not a search engine for every job on the internet. Like the scripts above, it fetches the boards you list.

Wrapping up

For one or two companies, the snippets above are all you need: one GET for Greenhouse, Lever or Ashby, and a paginated POST for Workday. Once you're tracking dozens of boards across several ATSs, the work moves to normalization, error handling, politeness and change tracking. You can build that yourself (the diff() function above is a decent start) or use a hosted tool if your time is worth more.

If you've found other ATS endpoints or quirks worth knowing about, please share them in the comments.

Top comments (2)

Collapse
 
double_chen_70da460344c73 profile image
Double CHEN •

The Workday generator yields no id, but diff() requires one. Fetch the detail ID or build a stable key from tenant, site and externalPath before change detection.

Collapse
 
steadyscrape profile image
SteadyScrape •

Good catch, thank you. You're right: the Workday snippet doesn't yield an id, so diff() would throw a KeyError on it. The simplest fix is a stable key built from the listing itself:

yield {"id": f"workday:{tenant}:{site}:{j['externalPath']}", ...}
Enter fullscreen mode Exit fullscreen mode

externalPath is unique per posting on a given site, so it works as a dedupe key without the extra detail request. If you're already fetching details, the job's own ID works too. I've updated the post.