Scrape Workday, Greenhouse and Lever jobs into SQLite with Python (and get only the new ones)

Disclosure: I built and sell the Apify actor used here (Career Site Jobs Scraper). It is pay-per-result at $4 per 1,000 jobs saved; companies not found, empty results and filtered-out jobs are free. All code below was executed against the live actor on 2026-10-30 and the outputs shown are real.

Job boards are a gold mine for market research, lead generation and personal job hunting, but every applicant-tracking system (ATS) has a different format. Greenhouse, Lever, Ashby, SmartRecruiters and Workday all expose public career pages, and the data you want (title, location, salary range when published, apply URL) is the same everywhere. This tutorial shows how to get one normalized schema from all of them, store it in SQLite, and run a "new jobs since last run" monitor.

Setup

pip install requests
export APIFY_TOKEN=apify_api_xxx   # Apify Console > Settings > Integrations

Step 1: fetch jobs with filters

The companies input accepts bare slugs (stripe is tried on Greenhouse, Lever, Ashby and SmartRecruiters) or full career-page URLs. Workday needs the full URL, for example https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite. Filters such as titleKeywords, excludeKeywords, locations, remoteOnly and postedWithinDays are applied for you, and filtered-out jobs are not billed.

import os
import requests

TOKEN = os.environ["APIFY_TOKEN"]
ACTOR = "flyingnimbus~career-site-jobs-scraper"
BASE = "https://api.apify.com/v2"
HEADERS = {"Authorization": f"Bearer {TOKEN}"}


def fetch_jobs(companies, **filters):
    """Run the actor synchronously; `filters` are passed through as input fields."""
    payload = {"companies": companies, "includeDescription": False, **filters}
    resp = requests.post(
        f"{BASE}/acts/{ACTOR}/run-sync-get-dataset-items",
        headers=HEADERS,
        json=payload,
        timeout=300,
    )
    resp.raise_for_status()
    return resp.json()


if __name__ == "__main__":
    jobs = fetch_jobs(
        ["stripe", "https://jobs.lever.co/spotify"],
        titleKeywords=["engineer"],
        locations=["Toronto", "Canada"],
        maxJobsPerCompany=20,
    )
    print(len(jobs), "jobs")
    for j in jobs[:8]:
        print(j["company"], "|", j["ats"], "|", j["title"], "|", j["location"], "|", j["url"])

Real output (trimmed) for Toronto and Canada engineering roles:

21 jobs
Stripe | greenhouse | Android Engineer, Terminal - CA | Toronto, CA | https://stripe.com/jobs/search?gh_jid=8274893
spotify | lever | Senior Software Engineer | Toronto | https://jobs.lever.co/spotify/e04833c0-...

Each record has ats, jobId, company, title, location, department, employmentType, workplaceType, salary, postedAt, url, applyUrl and, if includeDescription is true, descriptionText and descriptionHtml. I turn descriptions off here to keep rows small.

Step 2: store in SQLite, de-duplicated

(ats, jobId) is a stable key, so INSERT OR IGNORE makes the script idempotent.

import sqlite3
from jobs_fetch import fetch_jobs

db = sqlite3.connect("jobs.db")
db.execute(
    """CREATE TABLE IF NOT EXISTS jobs (
         ats TEXT, job_id TEXT, company TEXT, title TEXT, location TEXT,
         url TEXT, posted_at TEXT, salary TEXT,
         PRIMARY KEY (ats, job_id))"""
)

jobs = fetch_jobs(
    ["stripe", "https://jobs.lever.co/spotify"],
    titleKeywords=["engineer"],
    maxJobsPerCompany=25,
)

new = 0
for j in jobs:
    if j.get("status") != "ok":          # board not found / failed rows carry a status
        print("skipped:", j.get("input"), j.get("status"))
        continue
    cur = db.execute(
        "INSERT OR IGNORE INTO jobs VALUES (?,?,?,?,?,?,?,?)",
        (j["ats"], j["jobId"], j["company"], j["title"], j["location"],
         j["url"], j["postedAt"], str(j["salary"]) if j["salary"] else None),
    )
    new += cur.rowcount
db.commit()
total = db.execute("SELECT COUNT(*) FROM jobs").fetchone()[0]
print(f"{new} new rows, {total} total in jobs.db")

Running it twice gave:

50 new rows, 50 total in jobs.db
0 new rows, 50 total in jobs.db

Note the status check: the actor emits a free status row for boards it could not read or that had no matches (for example no_matching_jobs), so you can tell "nothing found" from "something broke".

Step 3: monitor mode (new jobs only)

Rather than diffing yourself, set onlyNewJobs: true. The actor remembers job IDs per company in a named key-value store in your Apify account and returns only unseen jobs, so you pay only for new postings. This script uses the async API (start a run, poll it, read the dataset), which also works for runs longer than the 5-minute sync limit:

import time
import requests
from jobs_fetch import ACTOR, BASE, HEADERS

# Monitor mode: the actor remembers job IDs per company and returns only unseen ones.
# Run this on a schedule (Apify Console > Schedules, or cron) to get "new jobs since last run".
run_input = {
    "companies": ["stripe", "https://jobs.lever.co/spotify"],
    "onlyNewJobs": True,
    "includeDescription": False,
    "titleKeywords": ["engineer", "developer"],
}
run = requests.post(f"{BASE}/acts/{ACTOR}/runs", headers=HEADERS, json=run_input, timeout=60)
run.raise_for_status()
run = run.json()["data"]

while run["status"] in ("READY", "RUNNING"):
    time.sleep(3)
    run = requests.get(f"{BASE}/actor-runs/{run['id']}", headers=HEADERS, timeout=60).json()["data"]

print("status:", run["status"], "|", run.get("statusMessage"))
items = requests.get(
    f"{BASE}/datasets/{run['defaultDatasetId']}/items", headers=HEADERS,
    params={"clean": 1}, timeout=60,
).json()
items = [j for j in items if j.get("status") == "ok"]   # other rows are free status markers
print(len(items), "new job(s) since last run")
for j in items[:5]:
    print("-", j["company"], "|", j["title"], "|", j["url"])

My first run seeded the state and returned 93 matching jobs. Running it again straight afterwards printed:

status: SUCCEEDED | Done: 0 jobs; 2 boards read, 0 not found, 0 failed.
0 new job(s) since last run

Schedule it daily in the Apify Console (Schedules) or from cron, and send the printed lines to email, Slack or a webhook.

Caveats

• Coverage is only what the ATS publishes. Companies on other systems, or behind a login, will not appear.
• Salary is often null. It is only filled where the board publishes ranges, which is required in some jurisdictions but far from universal.
• Date fields vary by ATS. postedWithinDays relies on dates from Greenhouse, Lever and Ashby; do not expect them on every board.
• Respect the data. These are public postings, but check the terms of use of the sites you target and keep request volumes reasonable.

Cost

At $4 per 1,000 jobs, the 50-row store run above costs about 20 cents, and a daily monitor on 20 companies usually pays for a few dozen new jobs, which is pennies. Apify's free plan includes some monthly credit; check their pricing page for current numbers.

Next steps

• Add excludeKeywords: ["intern"] and remoteOnly for a personal job alert.
• Group by department and count to see which teams a company is hiring for, a decent growth signal.
• Join with salary data to build a compensation benchmark by city.

Questions or a board that does not parse? Open an issue on the actor page.