DEV Community: Rakshit Solanki

Scrape Workday, Greenhouse and Lever jobs into SQLite with Python (and get only the new ones)

Rakshit Solanki — Sat, 31 Oct 2026 19:55:09 +0000

Disclosure: I built and sell the Apify actor used here (Career Site Jobs Scraper). It is pay-per-result at $4 per 1,000 jobs saved; companies not found, empty results and filtered-out jobs are free. All code below was executed against the live actor on 2026-10-30 and the outputs shown are real.

Job boards are a gold mine for market research, lead generation and personal job hunting, but every applicant-tracking system (ATS) has a different format. Greenhouse, Lever, Ashby, SmartRecruiters and Workday all expose public career pages, and the data you want (title, location, salary range when published, apply URL) is the same everywhere. This tutorial shows how to get one normalized schema from all of them, store it in SQLite, and run a "new jobs since last run" monitor.

Setup

pip install requests
export APIFY_TOKEN=apify_api_xxx   # Apify Console > Settings > Integrations

Step 1: fetch jobs with filters

The companies input accepts bare slugs (stripe is tried on Greenhouse, Lever, Ashby and SmartRecruiters) or full career-page URLs. Workday needs the full URL, for example https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite. Filters such as titleKeywords, excludeKeywords, locations, remoteOnly and postedWithinDays are applied for you, and filtered-out jobs are not billed.

import os
import requests

TOKEN = os.environ["APIFY_TOKEN"]
ACTOR = "flyingnimbus~career-site-jobs-scraper"
BASE = "https://api.apify.com/v2"
HEADERS = {"Authorization": f"Bearer {TOKEN}"}


def fetch_jobs(companies, **filters):
    """Run the actor synchronously; `filters` are passed through as input fields."""
    payload = {"companies": companies, "includeDescription": False, **filters}
    resp = requests.post(
        f"{BASE}/acts/{ACTOR}/run-sync-get-dataset-items",
        headers=HEADERS,
        json=payload,
        timeout=300,
    )
    resp.raise_for_status()
    return resp.json()


if __name__ == "__main__":
    jobs = fetch_jobs(
        ["stripe", "https://jobs.lever.co/spotify"],
        titleKeywords=["engineer"],
        locations=["Toronto", "Canada"],
        maxJobsPerCompany=20,
    )
    print(len(jobs), "jobs")
    for j in jobs[:8]:
        print(j["company"], "|", j["ats"], "|", j["title"], "|", j["location"], "|", j["url"])

Real output (trimmed) for Toronto and Canada engineering roles:

21 jobs
Stripe | greenhouse | Android Engineer, Terminal - CA | Toronto, CA | https://stripe.com/jobs/search?gh_jid=8274893
spotify | lever | Senior Software Engineer | Toronto | https://jobs.lever.co/spotify/e04833c0-...

Each record has ats, jobId, company, title, location, department, employmentType, workplaceType, salary, postedAt, url, applyUrl and, if includeDescription is true, descriptionText and descriptionHtml. I turn descriptions off here to keep rows small.

Step 2: store in SQLite, de-duplicated

(ats, jobId) is a stable key, so INSERT OR IGNORE makes the script idempotent.

import sqlite3
from jobs_fetch import fetch_jobs

db = sqlite3.connect("jobs.db")
db.execute(
    """CREATE TABLE IF NOT EXISTS jobs (
         ats TEXT, job_id TEXT, company TEXT, title TEXT, location TEXT,
         url TEXT, posted_at TEXT, salary TEXT,
         PRIMARY KEY (ats, job_id))"""
)

jobs = fetch_jobs(
    ["stripe", "https://jobs.lever.co/spotify"],
    titleKeywords=["engineer"],
    maxJobsPerCompany=25,
)

new = 0
for j in jobs:
    if j.get("status") != "ok":          # board not found / failed rows carry a status
        print("skipped:", j.get("input"), j.get("status"))
        continue
    cur = db.execute(
        "INSERT OR IGNORE INTO jobs VALUES (?,?,?,?,?,?,?,?)",
        (j["ats"], j["jobId"], j["company"], j["title"], j["location"],
         j["url"], j["postedAt"], str(j["salary"]) if j["salary"] else None),
    )
    new += cur.rowcount
db.commit()
total = db.execute("SELECT COUNT(*) FROM jobs").fetchone()[0]
print(f"{new} new rows, {total} total in jobs.db")

Running it twice gave:

50 new rows, 50 total in jobs.db
0 new rows, 50 total in jobs.db

Note the status check: the actor emits a free status row for boards it could not read or that had no matches (for example no_matching_jobs), so you can tell "nothing found" from "something broke".

Step 3: monitor mode (new jobs only)

Rather than diffing yourself, set onlyNewJobs: true. The actor remembers job IDs per company in a named key-value store in your Apify account and returns only unseen jobs, so you pay only for new postings. This script uses the async API (start a run, poll it, read the dataset), which also works for runs longer than the 5-minute sync limit:

import time
import requests
from jobs_fetch import ACTOR, BASE, HEADERS

# Monitor mode: the actor remembers job IDs per company and returns only unseen ones.
# Run this on a schedule (Apify Console > Schedules, or cron) to get "new jobs since last run".
run_input = {
    "companies": ["stripe", "https://jobs.lever.co/spotify"],
    "onlyNewJobs": True,
    "includeDescription": False,
    "titleKeywords": ["engineer", "developer"],
}
run = requests.post(f"{BASE}/acts/{ACTOR}/runs", headers=HEADERS, json=run_input, timeout=60)
run.raise_for_status()
run = run.json()["data"]

while run["status"] in ("READY", "RUNNING"):
    time.sleep(3)
    run = requests.get(f"{BASE}/actor-runs/{run['id']}", headers=HEADERS, timeout=60).json()["data"]

print("status:", run["status"], "|", run.get("statusMessage"))
items = requests.get(
    f"{BASE}/datasets/{run['defaultDatasetId']}/items", headers=HEADERS,
    params={"clean": 1}, timeout=60,
).json()
items = [j for j in items if j.get("status") == "ok"]   # other rows are free status markers
print(len(items), "new job(s) since last run")
for j in items[:5]:
    print("-", j["company"], "|", j["title"], "|", j["url"])

My first run seeded the state and returned 93 matching jobs. Running it again straight afterwards printed:

status: SUCCEEDED | Done: 0 jobs; 2 boards read, 0 not found, 0 failed.
0 new job(s) since last run

Schedule it daily in the Apify Console (Schedules) or from cron, and send the printed lines to email, Slack or a webhook.

Caveats

Coverage is only what the ATS publishes. Companies on other systems, or behind a login, will not appear.
Salary is often null. It is only filled where the board publishes ranges, which is required in some jurisdictions but far from universal.
Date fields vary by ATS. postedWithinDays relies on dates from Greenhouse, Lever and Ashby; do not expect them on every board.
Respect the data. These are public postings, but check the terms of use of the sites you target and keep request volumes reasonable.

Cost

At $4 per 1,000 jobs, the 50-row store run above costs about 20 cents, and a daily monitor on 20 companies usually pays for a few dozen new jobs, which is pennies. Apify's free plan includes some monthly credit; check their pricing page for current numbers.

Next steps

Add excludeKeywords: ["intern"] and remoteOnly for a personal job alert.
Group by department and count to see which teams a company is hiring for, a decent growth signal.
Join with salary data to build a compensation benchmark by city.

Questions or a board that does not parse? Open an issue on the actor page.

Pull Google Trends data into Python with one API call (no pytrends, no cookies)

Rakshit Solanki — Sat, 31 Oct 2026 19:54:20 +0000

Disclosure: I built and sell the Apify actor used in this post (Google Trends Scraper). It is pay-per-use ($2 per 1,000 search terms that return data; failed or empty terms are not charged). Everything below was run against the real actor on 2026-10-30; the numbers shown are from that run.

If you have ever scripted Google Trends, you know the pain: there is no official API, pytrends breaks whenever Google changes something, and rate limits (HTTP 429) hit quickly from datacenter IPs. In this post I will show a small, dependency-light Python workflow that handles the retries and proxy rotation on the server side, so your code is just "POST JSON, get JSON".

You will end up with a reusable fetch_trends() function, a comparison of several keywords in one Canada-only query, a list of rising related queries, and a CSV you can open in a spreadsheet.

Setup

Create a free Apify account and copy your API token from Settings → Integrations.
pip install requests
Export the token: export APIFY_TOKEN=apify_api_xxx (never hard-code it).

Step 1: a reusable fetch function

Apify's run-sync-get-dataset-items endpoint starts the actor, waits for it to finish, and returns the results in the HTTP response. It is ideal for short jobs (it times out after 5 minutes, so for large batches use the async start-poll-read pattern shown in the second tutorial of this series, or the Apify client library).

import os
import requests

TOKEN = os.environ["APIFY_TOKEN"]
ACTOR = "flyingnimbus~google-trends-scraper"
URL = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items"


def fetch_trends(terms, geo="", time_range="past_12_months", compare=False):
    """Run the actor synchronously and return its dataset items as a list of dicts."""
    payload = {
        "searchTerms": terms,
        "compareTerms": compare,
        "geo": geo,                      # "" = worldwide, "CA", "US-CA", ...
        "timeRange": time_range,
        "interestOverTime": True,
        "interestByRegion": True,
        "relatedQueries": True,
        "proxyConfiguration": {"useApifyProxy": True},
    }
    resp = requests.post(
        URL,
        headers={"Authorization": f"Bearer {TOKEN}"},
        json=payload,
        timeout=300,
    )
    resp.raise_for_status()
    return resp.json()


if __name__ == "__main__":
    items = fetch_trends(["python", "ruby"], geo="CA", compare=True)
    print(len(items), "item(s)")
    item = items[0]
    print(item["status"], item["geo"], item["timeRange"])
    print(item["averageInterest"])

Key input fields: searchTerms (list), geo ("" for worldwide, or CA, US-CA, and so on), timeRange (past_7_days, past_12_months, past_5_years, all, ...), compareTerms (put up to 5 terms on one shared 0-100 scale, like the Trends "compare" view) and searchProperty (web, youtube, news, images, shopping).

My run of the __main__ block printed:

1 item(s)
ok CA today 12-m
{'python': 58.13, 'ruby': 25.83}

With compareTerms=True you get one item per comparison group, not one per term. With compareTerms=False you get one item per term.

Step 2: weekly series, rising queries, CSV

Each result item contains interestOverTime (a list of {time, values} points), interestByRegion, relatedQueries (keyed by term, each row tagged top or rising) and averageInterest. Here is a small report script:

import csv
from trends_fetch import fetch_trends


def weekly_series(item):
    """Yield (period_start, {term: value}) from one result item."""
    for point in item.get("interestOverTime", []):
        yield point["time"][:10], point["values"]


def rising_queries(item, limit=5):
    """Return {term: [rising related queries]} for a result item."""
    out = {}
    for term, rows in (item.get("relatedQueries") or {}).items():
        rising = [r for r in rows if r["list"] == "rising"]
        out[term] = [(r["query"], r["value"]) for r in rising[:limit]]
    return out


def to_csv(item, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        terms = list(item["averageInterest"].keys())
        writer.writerow(["period_start", *terms])
        for day, values in weekly_series(item):
            writer.writerow([day, *[values.get(t, "") for t in terms]])


if __name__ == "__main__":
    items = fetch_trends(["python", "ruby", "php"], geo="CA", compare=True)
    for item in items:
        if item["status"] != "ok":
            print("skipped:", item["searchTerms"], item["status"])
            continue
        print("average interest:", item["averageInterest"])
        print("rising:", rising_queries(item))
        to_csv(item, "trends_ca.csv")
    print("wrote trends_ca.csv")

Output from my run (Canada, last 12 months, three terms compared):

average interest: {'python': 57.4, 'ruby': 25.81, 'php': 20.98}
wrote trends_ca.csv   # 53 weekly rows

The CSV starts period_start,python,ruby,php followed by rows like 2025-10-26,63,23,21.

Gotchas I hit (so you do not have to)

Ambiguous terms. The "rising" list for ruby was dominated by celebrity names, and python surfaced restaurant queries. Google Trends measures search terms, not topics. Use the category field or more specific phrases like ruby on rails.
Relative, not absolute. Values are scaled 0-100 against the peak in your query. They are not search volumes. Comparing across separate queries is invalid; use compareTerms to share a scale.
Check status. Each item has a status field. Skip anything that is not "ok", as the script above does, rather than assuming data is present.
Partial last point. The latest period can be flagged isPartial: true; drop it before computing growth rates.

Compare mode versus separate queries

Should you pass compareTerms=True or query each term alone? Use compare mode when you care about relative popularity: the values for every term share one scale, so "python 57 vs ruby 26" is a meaningful statement. Use separate queries when you care about each term's own shape over time, for example to spot seasonality of a niche keyword that would look flat next to a giant like python. The cost is per comparison group in compare mode and per term otherwise, so comparing five terms at once is also the cheapest way to get a shared scale.

Ideas to build on this

Run it weekly on a keyword list and chart your niche's seasonality before you plan content.
Feed rising queries into a content calendar or a keyword-research spreadsheet.
Use interestByRegion to pick which provinces or states to target for local ads.
Schedule it in the Apify Console (Schedules) and push results to a webhook or Google Sheet.

What it costs

At $2 per 1,000 terms, the three-term comparison above is one comparison group, so roughly a fifth of a cent. A 500-keyword weekly refresh is about $1 per run. Apify's free plan includes some monthly platform credit; check their current pricing page for the exact amount.

Wrap-up

The trick is moving the fragile parts (sessions, proxies, backoff) out of your codebase and behind a stable JSON interface. If you hit an edge case or want another field exposed, open an issue on the actor page.