DEV Community

Web Obsidians
Web Obsidians

Posted on

How to Scrape Job Listings from Greenhouse, Lever and Ashby Career Pages (Python and No-Code)

If you want the open jobs of a list of companies (for a job board, market research, a recruiting tool or your own job search), you do not need to scrape HTML career pages one by one. Most companies host their career page on an applicant tracking system (ATS), and several of them publish the open jobs as public JSON.

This post shows the three easiest ones in Python, the catches you will hit after that, and a no-code option for when you need many companies and many systems.

Option 1: public job board APIs in Python

Greenhouse, Lever and Ashby each have a public, read-only endpoint per company. You only need the company's board name, which is usually in its careers URL (boards.greenhouse.io/stripe, jobs.lever.co/palantir, jobs.ashbyhq.com/ramp).

import json
import urllib.request


def get(url):
    # some job boards reject Python's default User-Agent, so send a normal one
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (job-list-script)"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)


def greenhouse(board):
    data = get(f"https://boards-api.greenhouse.io/v1/boards/{board}/jobs")
    return [{"title": j["title"], "location": j["location"]["name"], "url": j["absolute_url"]}
            for j in data["jobs"]]


def lever(board):
    data = get(f"https://api.lever.co/v0/postings/{board}?mode=json")
    return [{"title": j["text"], "location": j["categories"].get("location"), "url": j["hostedUrl"]}
            for j in data]


def ashby(board):
    data = get(f"https://api.ashbyhq.com/posting-api/job-board/{board}")
    return [{"title": j["title"], "location": j.get("location"), "url": j["jobUrl"]}
            for j in data["jobs"]]


for name, jobs in [("stripe", greenhouse("stripe")), ("palantir", lever("palantir")), ("ramp", ashby("ramp"))]:
    print(name, len(jobs), "open jobs, e.g.", jobs[0]["title"], "-", jobs[0]["location"])
Enter fullscreen mode Exit fullscreen mode

That is enough for a handful of companies on these three systems. Things to know before you scale it:

  • Full descriptions are optional. Greenhouse returns them with ?content=true; Lever includes descriptionPlain; Ashby has descriptionPlain and, with ?includeCompensation=true, salary ranges when the company publishes them.
  • Every system has its own shape. Locations, departments, remote flags, dates and salaries live in different fields, so you will write a normalizer per system.
  • You first have to know which ATS a company uses. A list of company names does not tell you whether to call Greenhouse or Workday.
  • Workday is the hard one. Large companies often use Workday, which has no simple per-company feed: each company has its own tenant, data center number and site name, and the job list is a paged POST endpoint.
  • Monitoring needs state. To get "only jobs that are new since yesterday", you have to remember what you already saw.

Option 2: a ready-made Actor for many companies and systems

I packaged all of that as an Apify Actor: ATS Jobs Scraper. You paste company names, domains or careers URLs, and it finds the system behind each one (Workday, Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio and Breezy) and returns one clean row per job:

{
  "company": "Linear",
  "ats": "ashby",
  "title": "Senior / Staff Fullstack Engineer",
  "department": "Product",
  "team": "Engineering",
  "location": "Europe",
  "isRemote": true,
  "workplaceType": "remote",
  "employmentType": "Full-time",
  "salary": null,
  "url": "https://jobs.ashbyhq.com/linear/d3bc1ced-3ce4-4086-a050-555055dbb1ff",
  "applyUrl": "https://jobs.ashbyhq.com/linear/d3bc1ced-3ce4-4086-a050-555055dbb1ff/application",
  "descriptionText": "At Linear, we're building the product development system for teams and agents. ..."
}
Enter fullscreen mode Exit fullscreen mode

On top of the script above it gives you:

  • Filters for title keywords (or keywords in the description), excluded words, locations, remote only, departments and "posted within the last N days"
  • Salary in one format (min, max, currency, interval) when the company publishes it
  • Return only new jobs: schedule it daily and each run returns only the jobs that earlier runs did not return
  • Export to JSON, CSV or Excel, or push results to Google Sheets, Slack or a webhook with Apify integrations

From code, use the Apify client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("webobsidians/ats-jobs-scraper").call(
    run_input={"companies": ["stripe", "linear.app", "palantir"], "keywords": ["engineer"], "remoteOnly": True}
)
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["company"], "|", job["title"], "|", job["location"], "|", job["url"])
Enter fullscreen mode Exit fullscreen mode

It costs $1.00 per 1,000 jobs plus $0.00005 per run start, with platform usage included, and Apify's free plan includes monthly credit to try it.

Which one should you use?

  • A few companies on Greenhouse, Lever or Ashby: the Python script.
  • Many companies, Workday, a daily "new jobs" feed, or a spreadsheet for a non-developer: the Actor.

Either way, you are reading job posts that the companies publish for everyone, so keep request rates reasonable.

Top comments (0)