DEV Community

Siftwright Data
Siftwright Data

Posted on

Track new job openings at specific companies without scraping Indeed or LinkedIn

Scraping Indeed or LinkedIn Jobs looks like the obvious way to watch who's hiring. In practice you fight Cloudflare challenges, logins, rate limits and HTML that changes every few weeks, and the listings you do get are often days old, reposted from somewhere else.

That somewhere else is usually the company's own careers page, and most careers pages are powered by an applicant tracking system (ATS) that publishes a clean, public JSON feed. Watching those feeds directly is faster, more reliable and fully public.

Step 1: find which ATS a company uses

Open the company's careers page and look at where the "Apply" links point:

If links go to... The ATS is Public feed
boards.greenhouse.io/acme or job-boards.greenhouse.io/acme Greenhouse https://boards-api.greenhouse.io/v1/boards/acme/jobs
jobs.lever.co/acme Lever https://api.lever.co/v0/postings/acme?mode=json
jobs.ashbyhq.com/acme Ashby https://api.ashbyhq.com/posting-api/job-board/acme
apply.workable.com/acme Workable https://apply.workable.com/api/v1/widget/accounts/acme
jobs.smartrecruiters.com/Acme SmartRecruiters https://api.smartrecruiters.com/v1/companies/Acme/postings

The part after the domain (acme) is the board slug.

Step 2: poll the feed and keep only new jobs

import json, pathlib, requests

SEEN = pathlib.Path("seen.json")
seen = set(json.loads(SEEN.read_text())) if SEEN.exists() else set()

def greenhouse(slug):
    r = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", timeout=30)
    for j in r.json()["jobs"]:
        yield j["id"], j["title"], j["location"]["name"], j["absolute_url"]

new = []
for slug in ["stripe", "databricks", "notion"]:
    for job_id, title, loc, url in greenhouse(slug):
        key = f"gh:{slug}:{job_id}"
        if key not in seen:
            seen.add(key)
            new.append((slug, title, loc, url))

SEEN.write_text(json.dumps(sorted(seen)))
for row in new:
    print(" | ".join(row))
Enter fullscreen mode Exit fullscreen mode

Run it from cron once a day and you have a "new openings" alert for any list of companies, with no browser, proxies or logins.

Step 3: scale past a handful of companies

The manual approach breaks down when you want hundreds of companies across several ATSs, because every ATS has a different schema and you first need to know each company's slug.

That's the problem we built ATS Jobs Search on Apify to solve. It covers seven ATSs (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio), uses one output schema, has a built-in directory of about 7,900 companies' boards, and has an "only new jobs since last run" mode:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright_data/ats-jobs-search").call(run_input={
    "searchDirectory": True,
    "keywords": ["data engineer", "analytics engineer"],
    "locations": ["Remote"],
    "onlyNewJobs": True,
})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["companyName"], "|", job["title"], "|", job["url"])
Enter fullscreen mode Exit fullscreen mode

It costs $1 per 1,000 jobs, and Apify's free monthly credit covers a lot of testing.

Prefer a file or a ready-made feed?

  • Everything at once: today's full snapshot (about 400,000 open jobs from 7,500+ companies, one CSV) is a $39 instant download.
  • A daily niche feed (for example "remote data jobs" or "healthcare roles in Texas") delivered to a spreadsheet or your inbox: email siftwright.data@gmail.com and we'll quote it.

Happy hunting.

Top comments (0)