DEV Community

Big Man
Big Man

Posted on

How to get job listings from Lever, Ashby, Workday and other career sites with an API (Python, Node or no code)

Most companies don't post every opening to the big job boards, but they all list them on their own careers page. Those pages usually run on an applicant tracking system such as Lever, Ashby, SmartRecruiters, Workday or Workable, and each one works differently. If you track hiring across a list of companies, you end up writing five scrapers.

This tutorial uses Career Site Jobs Scraper, an Apify actor I built that covers all five with one input. For each job it returns the title, company, location, remote flag, salary when published, department, posted date, full description and apply link. You can filter by title, location, department, remote and date, and an only-new mode suits daily feeds. The full examples are on GitHub: auto-actor/career-site-jobs-scraper.

What people use it for

  • One daily feed of new openings across a list of target companies, whatever platform each one hires on, with onlyNew: true.
  • Remote-job boards and newsletters, using remoteOnly and postedWithinDays.
  • Tracking which teams a company is growing (departments) as a sales or investment signal.

Input

Give it the careers URL, for example https://jobs.lever.co/spotify, or the short form ashby:notion. You can mix platforms in one run.

{"companies": ["https://jobs.lever.co/spotify", "https://jobs.ashbyhq.com/notion"], "maxItems": 50}
Enter fullscreen mode Exit fullscreen mode
Field What it does
companies Career sites to scrape: the careers URL (e.g. https://jobs.lever.co/spotify) or ats:name (e.g. ashby:notion). A bare name is looked up on Lever, Ashby, SmartRecruiters and Workable.
titleSearch Keep only jobs whose title contains any of these words.
titleExclusionSearch Drop jobs whose title contains any of these words.
locationSearch Keep only jobs whose location contains any of these words.
departmentSearch Keep only jobs whose department or team contains any of these words. Workday boards don't publish departments, so this drops Workday jobs.
remoteOnly Keep only jobs marked remote.
postedWithinDays Keep only jobs posted in the last N days.
includeDescription Return the full description as text and HTML (one extra request per job on SmartRecruiters, Workday and Workable).
maxItemsPerCompany Stop after this many jobs from any one career site.
maxItems Stop after this many jobs in total.
onlyNew Skip jobs already returned by earlier runs (remembered for 180 days in the career-site-jobs-seen store of your account). Ideal for schedules.

Python

"""Run Career Site Jobs Scraper on Apify and print the results.

    pip install apify-client
    export APIFY_TOKEN=...   # Apify Console -> Settings -> Integrations
    python example.py
"""
import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("daedal_quintuplet/career-site-jobs-scraper").call(
    run_input={
        "companies": [
            "https://jobs.lever.co/spotify",
            "https://jobs.ashbyhq.com/notion"
        ],
        "maxItems": 50
    }
)

jobs = client.dataset(run["defaultDatasetId"]).iterate_items()
for job in jobs:
    print(job["title"], "|", job["organization"], "|", job["location"], "|", job["source"])
Enter fullscreen mode Exit fullscreen mode

Node.js

// Run Career Site Jobs Scraper on Apify and print the results.
//
//   npm install apify-client
//   APIFY_TOKEN=... node example.js     (token: Apify Console -> Settings -> Integrations)

const { ApifyClient } = require('apify-client');

async function main() {
    const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

    const run = await client.actor('daedal_quintuplet/career-site-jobs-scraper').call({
        "companies": [
            "https://jobs.lever.co/spotify",
            "https://jobs.ashbyhq.com/notion"
        ],
        "maxItems": 50
    });

    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    for (const job of items) {
        console.log(job.title, '|', job.organization, '|', job.location, '|', job.source);
    }
}

main().catch((err) => {
    console.error(err);
    process.exit(1);
});
Enter fullscreen mode Exit fullscreen mode

No code: one HTTP request

Works from n8n, Make, Zapier or curl:

curl -X POST "https://api.apify.com/v2/acts/daedal_quintuplet~career-site-jobs-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"companies": ["https://jobs.lever.co/spotify", "https://jobs.ashbyhq.com/notion"], "maxItems": 50}'
Enter fullscreen mode Exit fullscreen mode

What you get

One row per job, with the same fields whichever platform it came from:

Field Meaning
id The career site's job id
title Job title
organization Company
organization_url The company's career site
url Link to the job ad
apply_url Link to the application form
date_posted When the job was posted
location Location text
locations Every location listed
countries Countries
remote_derived True when the job is marked remote or says remote in its location
workplace_type On-site, Hybrid or Remote, when the site says
employment_type Employment type
departments Departments or teams
salary Salary when the career site publishes one
description_text Full description as plain text
description_html Full description as HTML
source Which platform it came from: lever, ashby, smartrecruiters, workday or workable
company_slug The company's id on that platform
scrapedAt When it was scraped

Notes

  • Covers Lever, Ashby, SmartRecruiters, Workday and Workable. For Greenhouse boards, use our Greenhouse Jobs Scraper.
  • Turn off includeDescription for faster, smaller runs when you only need titles and links.

Price

$0.0015 per job ($1.50 per 1,000), paid per result, with no monthly fee. Apify's free plan includes $5 of usage a month, which is enough to try it. For companies on Greenhouse, use the matching Greenhouse Jobs Scraper.

Links: Store page ยท GitHub examples

Top comments (0)