DEV Community: Zahar Baranenko

Turning a jobs scraper into an AI agent's tool with the Apify MCP server

Zahar Baranenko — Fri, 11 Sep 2026 12:07:54 +0000

There is a lot of good advice around on making your Actor agent-friendly. Almost none of it shows you the thing an agent actually sends over the wire — and that turned out to be the part that changed how I build.

I build Actors on Apify under the name primeflowio. One of them, ATS Jobs Scraper & Change Monitor, takes a list of companies and returns every open job they have, normalized into one schema — no matter which applicant tracking system the company uses. A few weeks after publishing it, I connected it to the Apify MCP server and an agent called it like a native tool: discovered it, filled the input, ran it, read the results back. No glue code on my side.

The Origin column says MCP: that run was started by an agent, not by me clicking Start.

So this article does two things. First it walks the protocol by hand — every request, every real response — so you can see exactly what your Actor is answering to. Then it goes through what I changed once I had read those responses, including a bug that would have told a customer that every job at a company had disappeared overnight.

The problem the Actor solves

Job postings are scattered across ATS platforms. If you track 200 target companies — as recruiters, B2B sales teams, and serious job seekers do — you'd need to visit 200 career pages, sitting on Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee or Personio, each with its own markup.

The twist most people miss: you don't need to scrape any of those pages. The major ATS platforms expose documented, public endpoints for their job boards. Here is every platform the Actor supports, with what each one actually returned when I re-checked it by hand this week:

ATS	Endpoint	Live check (Sep 2026)
Greenhouse	`boards-api.greenhouse.io/v1/boards/{slug}/jobs`	stripe: 611, gitlab: 230
Lever	`api.lever.co/v0/postings/{slug}?mode=json`	spotify: 73
Ashby	`api.ashbyhq.com/posting-api/job-board/{slug}`	ramp: 143, linear: 28
Personio	`{company}.jobs.personio.de/xml`	personio: 1 open role
Recruitee	`{company}.recruitee.com/api/offers/`	yource: 1 open role
Workday	`{tenant}.{wd}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs`	nvidia: 2,000 — and slow
SmartRecruiters	`api.smartrecruiters.com/v1/companies/{slug}/postings`	see below

No API keys, no proxies, no anti-bot arms race. The hard part isn't access — it's detection (which ATS does this company use?), normalization (a different response shape per platform, one schema out), and change tracking.

Three of those rows deserve a footnote, and they are the same lesson in different clothes.

Workday is the expensive one. It answers, and it answers with everything — two thousand postings for one tenant — but a small run against it took 83 seconds where Ashby takes two. That is fine for a watchlist on a schedule and a bad idea for anything with a stopwatch on it, which is why my prefilled input does not point at Workday.

Recruitee gets the failure mode right, and it is worth copying. An unknown company returns 404 {"error":"Not Found"}. A real company with nothing open returns 200 {"offers":[]}. Those are different sentences, and a caller can act on the difference.

Which brings us to SmartRecruiters, which does not. It answers 200 {"totalFound": 0, "content": []} for every identifier I tried — including companies that visibly run their careers site on SmartRecruiters, and including identifiers that do not exist at all. An endpoint that cannot distinguish "no openings" from "wrong name" from "this company never switched the public API on" is not a data source; it is a coin flip with a JSON wrapper. The Actor now reports that board as unreadable instead of returning an empty list that looks like an answer — which turns out to be the same bug I had shipped myself, one layer up. More on that in lesson 6.

So the Actor does three things:

Auto-detect — given a company slug or careers URL, it probes the supported endpoints and figures out which ATS answers.
Normalize — every job becomes one record: title, department, team, location, remote, employment_type, salary_min/max, url, published_at, updated_at.
Monitor — in monitor mode it keeps the previous run's state in the Actor's key-value store and emits new / removed / changed records as a diff. Users point a webhook or an n8n flow at those and get "company X just opened 3 engineering roles" without re-downloading everything.

That last part is the reason the Actor exists at all: the incumbent job-data Actors make you re-pull the full dataset every run and charge per job every time. If what you actually want is changes, you're paying for the same rows daily.

Every Actor is already an MCP tool

Here's the part that cost me zero extra work. The Apify MCP server exposes any public Actor as a tool over the Model Context Protocol. You don't register anything, you don't write a manifest — the Actor's input schema is the tool definition. An agent connects to:

https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper

and gets a tool named primeflowio--ats-jobs-scraper, with parameters generated from the same input schema that renders the form in Apify Console. (?tools= is the parameter in Apify's docs; ?actors= is accepted too — I checked both today and they return the same five tools.)

I wanted to see exactly what an agent sees, so I did the whole handshake by hand with curl before touching any framework. MCP over streamable HTTP is three steps.

1. Initialize a session:

curl -s -D - -X POST "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize",
       "params":{"protocolVersion":"2025-03-26","capabilities":{},
                 "clientInfo":{"name":"demo-agent","version":"1.0"}}}'

The response headers contain mcp-session-id — every later call carries it in an Mcp-Session-Id header, plus a notifications/initialized message to finish the handshake.

2. List tools. {"method":"tools/list"} returns, in my session:

primeflowio--ats-jobs-scraper
get-actor-run
get-dataset-items
get-key-value-store-record
abort-actor-run

Note what the server adds for free: the agent doesn't just get my Actor, it gets the helpers to check a run's status and page through the results dataset. That completes the loop — an agent can start a run, poll it, and fetch data without any Apify-specific code.

3. Call the tool:

curl -s -X POST "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Mcp-Session-Id: $SESSION" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call",
       "params":{"name":"primeflowio--ats-jobs-scraper",
                 "arguments":{"companies":["https://jobs.lever.co/spotify"],
                              "mode":"pull","maxJobsPerCompany":3}}}'

The run finished in under two seconds and the MCP response came back with a summary the agent can act on directly:

"status":"SUCCEEDED",
"statusMessage":"1/1 companies, 3 jobs, 0 changes, 0 errors"
...
SUCCEEDED in 1.635s. 4 items; 24 fields available.
Use get-dataset-items with datasetId=pl8b6nClKueN3efCO and limit
(for example 20) to fetch items (4 total). Available fields (dot notation):
record_type, company, ats, job_id, title, department, team, location,
remote, employment_type, salary_min, salary_max, salary_currency, url, ...

The whole exchange from a plain client: five tools discovered, one called, a summary back in under three seconds.

Read that response again from an LLM's point of view. It's not a blob of JSON rows — it's an instruction: here's what happened, here's the tool call to make next, here are the fields you can ask for. The Apify MCP server writes that guidance itself. My only job was to make sure the run summary (statusMessage) and the dataset fields were worth reading.

The same flow in Python

For anything scripted I use ~60 lines of plain requests — no MCP SDK needed for the happy path. Full file in the repo; the core is:

MCP_URL = "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper"

def rpc(session, method, params=None, msg_id=None):
    payload = {"jsonrpc": "2.0", "method": method}
    if params is not None: payload["params"] = params
    if msg_id is not None: payload["id"] = msg_id
    headers = dict(HEADERS)
    if session: headers["Mcp-Session-Id"] = session
    r = requests.post(MCP_URL, headers=headers, json=payload, timeout=300)
    r.raise_for_status()
    return r

init = rpc(None, "initialize", {...}, msg_id=1)
session = init.headers["mcp-session-id"]
rpc(session, "notifications/initialized")

tools = parse_sse(rpc(session, "tools/list", msg_id=2).text)

result = parse_sse(rpc(session, "tools/call", {
    "name": "primeflowio--ats-jobs-scraper",
    "arguments": {"companies": ["https://jobs.lever.co/spotify"],
                  "mode": "pull", "maxJobsPerCompany": 3},
}, msg_id=3).text)

One implementation detail that trips people up: responses arrive as server-sent events (Content-Type: text/event-stream), even for a single JSON-RPC reply. parse_sse just takes the last data: line and json.loads it.

Plugging the same URL into Claude, Cursor, or any MCP-capable client is a config entry instead of code — the point of the protocol is that the curl session above is exactly what those clients do under the hood.

What building for agents changed in my Actor design

I originally designed the input schema for humans filling a form in Apify Console. Agents read the same schema, but they punish different mistakes. Six lessons I'd apply to any Actor now:

1. The input schema is a tool contract — keep required fields to one. My schema has exactly one required field, companies. Everything else (mode, maxJobsPerCompany, includeSnapshot) has a sane default. An agent that has to guess six required parameters will guess some of them wrong; an agent that has to provide one list of company names almost can't fail.

2. Give agents a cost throttle. maxJobsPerCompany exists so a cautious caller can cap the bill of an exploratory run. My test cost under a cent: with pay-per-event pricing, 1 company check plus 3 job results is $0.005. An agent (or the human supervising it) can try the tool for pocket change before pointing it at a 500-company watchlist. If your Actor can generate unbounded output, give the caller a knob to bound it.

3. Make statusMessage machine-readable prose. "1/1 companies, 3 jobs, 0 changes, 0 errors" is short enough to fit in any context window and precise enough that an agent can decide what to do next without fetching the dataset at all. I treat that one string as part of the API surface now.

4. Put bulk data in the dataset, not the response. The MCP server already nudges this: it returns a summary plus "use get-dataset-items with this ID" rather than inlining rows. Fighting that — returning huge payloads inline — wastes the agent's context and your users' tokens. Design the output so the first page of dataset items answers the most common question.

One schema out, whichever ATS answered. This is the page the agent fetches after the summary tells it what's there.

5. Your defaults are load-bearing: the platform runs them. Apify's automated QA runs every public Actor with its prefilled input and expects success within about 5 minutes. Mine was ["linear", "https://jobs.lever.co/spotify"] — two boards, ~130 postings, chosen back when I was thinking about humans clicking "Start" and wanting to see something substantial. In August an upstream endpoint got slow, three consecutive QA runs failed, and the Actor was flagged "under maintenance" in the Store until the runs went green again. I found out from the notification emails, days later. Worth knowing what that clock is: under Apify's publishing terms, an Actor left broken for 30 days can be deprecated and removed.

The lesson isn't "make the Actor faster". It's that the prefill is a health check the platform runs on your behalf, so it should exercise the cheapest, most reliable path you have. I've since cut mine to a single small board: 28 postings, 2.2 seconds against a five-minute budget. The same defaults make agent test-calls cheap, which is a happy coincidence — QA bots and cautious agents want exactly the same thing from you.

Ivan Solovyev landed on the same wall from a different direction — his prefill hit four marketplaces at once and the test gave up waiting while paying users ran the Actor happily all day. Two of us tripped over the same default in the same month, which says something about how easy it is to write a prefill for the demo rather than for the robot.

6. An empty answer is not "everything disappeared". This one I found because of that article, and it is the most expensive bug I have shipped.

In monitor mode the Actor compares this run's jobs against the previous run's state and emits new / removed / changed. The failure path was already handled: if a job board errors out, the run records an error for that company, keeps the old state, and moves on. What I had not separated was a board that answers 200 with an empty list — which happens during migrations, when a board is briefly unpublished, or when the provider hiccups.

That path was ok, jobs = True, []. The diff then did exactly what it was told: marked every known job as removed, and overwrote the state with nothing. A customer watching Stripe would have received "611 jobs removed" one morning and "611 jobs new" the next, with every webhook firing twice on fiction.

The guard is small, and the shape of it generalises to any stateful Actor:

# A board that answers 200 with zero jobs is usually the source blinking, not
# every role closing at once. Keep the old state, say so, and only believe the
# zero once it repeats.
if prev_jobs and not jobs and streak + 1 < EMPTY_RUNS_BEFORE_TRUSTED:
    push_items([{"record_type": "warning", "company": slug, "ats": ats,
                 "detail": f"{ats} returned an empty board while "
                           f"{len(prev_jobs)} jobs were known; previous state "
                           f"kept, no changes emitted",
                 "empty_streak": streak + 1}])
    state_put(store_id, cache_key,
              {"ats": ats, "jobs": prev_jobs, "empty_streak": streak + 1})
    continue

I tested it by seeding the state with two known jobs for a board that currently answers empty. Before: two removed records and a wiped state. After: 1/1 companies, 0 jobs, 0 changes, 0 errors, 1 stale, one warning record, and both jobs still in state. Real mass closures are slow; sources blink instantly — so a zero has to earn its credibility over three consecutive runs before the Actor will act on it.

The general rule, which is the same one Solovyev arrives at from the billing side: an Actor with memory must distinguish what the world says from what it managed to hear. For a stateless scraper an empty result is a result. For anything that diffs, an empty result is an accusation.

Where this goes next: monitor mode plus agents

Pull mode answers "who is company X hiring right now?" — a natural one-shot agent query ("did Spotify open any Android roles this month?"). Monitor mode is more interesting for agent workflows because it inverts the direction: instead of the agent polling, the Actor runs on Apify's scheduler, emits only new / removed / changed records, and a webhook or n8n flow wakes the agent only when something changed. "Head of Data role appeared at a target account" is a buying signal; an agent that gets only diffs can react to it without burning tokens re-reading yesterday's 611 postings.

One platform detail makes that possible, and it is the part I expected to be hard. Monitor mode needs memory that survives between runs, and those runs belong to whoever calls the Actor, not to me. Actors run with limited permissions by default, which sounds like the end of the idea. The docs say a limited-permissions Actor may "create any additional storage, and write to that storage" and "read and write to storages created in previous runs", and my own runs — the log line reads Running under "LIMITED_PERMISSIONS" — create a named key-value store, then read that same state back on later runs and diff against it. So the state can live on the caller's account rather than in a database of mine, and I never have to ask anyone for a token. I have only exercised that on my own account, so treat the cross-account half as documented rather than as something I measured; the design decision it points to is what matters. If you are building anything stateful, look at named storage before you reach for infrastructure of your own.

The diff mode earns its keep on live data, by the way. While I was writing this, a watchlist run caught Databricks editing a marketing role 29 minutes after they touched it — one changed record out of thirteen companies and roughly three and a half thousand postings. That is the argument for a change feed in a single line: the agent reads one row instead of 3,500.

That's the pattern I'd summarize the whole experience with: the MCP server made my Actor callable by agents in an afternoon, but making it worth calling — one required input, bounded costs, a status line an LLM can parse, diffs instead of dumps — was ordinary API design, applied to a new consumer.

FAQ

Do I need to modify an Actor to expose it via MCP? No. Any public Actor works through mcp.apify.com immediately; the input schema becomes the tool signature. Whether it's a good tool is a design question — see the six lessons above.

What does a call cost? Whatever the Actor's pricing says — same as running it from Console or API. This Actor is pay-per-event: $2 per 1,000 companies checked, $1 per 1,000 job results, so the demo call above cost $0.005.

Is scraping ATS job boards allowed? The endpoints used here are the platforms' own public job-board APIs — Greenhouse Job Board API, Lever Postings API, Ashby Posting API, Recruitee's offers endpoint, Personio's job XML feed. They exist precisely so postings can be syndicated to aggregators and career-site widgets, which is why none of them need a key.

Can the agent also fetch the results? Yes — the MCP session includes get-actor-run and get-dataset-items helpers automatically, so the agent can page through results without any Apify SDK.

Try it yourself

Point an agent at your own watchlist. Add the MCP server to any MCP-capable client:

https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper

…then ask it something you'd actually want to know — "who is Stripe hiring in Berlin right now?" — and watch it pick the arguments out of the input schema on its own. A single company check plus a handful of postings costs half a cent, so the first call is cheap enough to be curious with. Run it a second time a week later with mode: "monitor" and you get the diff instead of the dump.

The Actor is ATS Jobs Scraper & Change Monitor on Apify Store. The demo client and the full curl transcript from this article are at github.com/strongboxr/ats-mcp-agent-demo.

I build data Actors on Apify as primeflowio. If a number in this article has a caveat, the caveat is in the sentence next to it.

Suggested meta description: How an Apify Actor that scrapes ATS job boards became an AI agent tool via the Apify MCP server — the full handshake, working code, and six design lessons.

How to pull every open job from Workday, Greenhouse, Lever, Ashby and 3 more ATS with public APIs (and monitor changes)

Zahar Baranenko — Thu, 20 Aug 2026 15:56:55 +0000

Job postings are one of the most underrated public data sources on the internet. Recruiters use them to spot placement opportunities, B2B teams read them as buying signals (a new Head of Data means data-tooling budget), and job seekers want to apply on day one — not when a posting finally reaches the aggregators.

The usual instinct is to scrape career pages. Don't. Most companies host their careers page on one of a handful of Applicant Tracking Systems (ATS), and the big ones — Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio — all expose public JSON (or XML) endpoints. No auth. No proxies. No brittle HTML selectors. The career page itself loads the same data you're about to fetch.

In this tutorial we'll build a single-file Python tool that:

fetches every open job for a company from any of the seven ATS,
auto-detects which ATS a company uses,
normalizes everything into one clean schema,
monitors changes — run it on a schedule and get only new / removed / changed postings.

The seven endpoints

ATS	Endpoint
Greenhouse	`GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true`
Lever	`GET https://api.lever.co/v0/postings/{slug}?mode=json`
Ashby	`GET https://api.ashbyhq.com/posting-api/job-board/{slug}`
SmartRecruiters	`GET https://api.smartrecruiters.com/v1/companies/{slug}/postings` (paginated)
Recruitee	`GET https://{slug}.recruitee.com/api/offers/`
Personio	`GET https://{slug}.jobs.personio.de/xml` (XML feed)
Workday	`POST https://{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs` (paginated)

The first six are documented by the vendors. The Workday one is the CXS endpoint every Workday career site loads its own listings from — public and unauthenticated, just not in a docs portal.

The {slug} is the company identifier you see in career-page URLs: boards.greenhouse.io/stripe → stripe, jobs.lever.co/spotify → spotify, jobs.ashbyhq.com/linear → linear, careers.smartrecruiters.com/Visa → Visa.

Try one right now — no API key needed:

curl -s "https://api.ashbyhq.com/posting-api/job-board/linear" | head -c 400

Step 1 — fetchers, one per ATS

Each API returns a different shape, so we normalize as we fetch. Here are the first four (Python 3, only requests):

import requests

UA = {"User-Agent": "ats-jobs-tutorial/1.0"}

def get_json(url, params=None):
    r = requests.get(url, params=params, headers=UA, timeout=30)
    r.raise_for_status()
    return r.json()

def fetch_greenhouse(slug):
    d = get_json(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs",
                 {"content": "true"})
    return [{
        "job_id": str(j["id"]),
        "title": j.get("title"),
        "location": (j.get("location") or {}).get("name"),
        "url": j.get("absolute_url"),
        "published_at": j.get("first_published"),
    } for j in d.get("jobs", [])]

def fetch_lever(slug):
    d = get_json(f"https://api.lever.co/v0/postings/{slug}", {"mode": "json"})
    return [{
        "job_id": str(j["id"]),
        "title": j.get("text"),
        "location": (j.get("categories") or {}).get("location"),
        "url": j.get("hostedUrl"),
        "published_at": j.get("createdAt"),  # milliseconds since epoch!
    } for j in d]

def fetch_ashby(slug):
    d = get_json(f"https://api.ashbyhq.com/posting-api/job-board/{slug}")
    return [{
        "job_id": str(j["id"]),
        "title": j.get("title"),
        "location": j.get("location"),
        "url": j.get("jobUrl") or j.get("applyUrl"),
        "published_at": j.get("publishedAt"),
    } for j in d.get("jobs", []) if j.get("isListed") is not False]

def fetch_smartrecruiters(slug):
    out, offset = [], 0
    while True:
        d = get_json(f"https://api.smartrecruiters.com/v1/companies/{slug}/postings",
                     {"limit": 100, "offset": offset})
        items = d.get("content", [])
        for j in items:
            loc = j.get("location") or {}
            out.append({
                "job_id": str(j["id"]),
                "title": j.get("name"),
                "location": ", ".join(filter(None, [loc.get("city"), loc.get("country")])) or None,
                "url": f"https://jobs.smartrecruiters.com/{slug}/{j['id']}",
                "published_at": j.get("releasedDate"),
            })
        offset += len(items)
        if not items or offset >= d.get("totalFound", 0):
            break
    return out

Three real-world quirks worth knowing (each cost me a debugging session):

Lever returns timestamps in milliseconds (createdAt: 1721900000000), not ISO strings. Divide by 1000 before datetime.fromtimestamp.
Ashby includes unlisted postings — filter out isListed: false or you'll "discover" jobs the company never published.
SmartRecruiters answers 200 OK with an empty list for any slug, even one that doesn't exist. An empty SmartRecruiters response is not proof the company uses SmartRecruiters.

Step 1.5 — Recruitee, Personio and Workday

Recruitee is another plain JSON GET:

def fetch_recruitee(slug):
    d = get_json(f"https://{slug}.recruitee.com/api/offers/")
    return [{
        "job_id": str(j["id"]),
        "title": j.get("title"),
        "location": j.get("location"),
        "url": j.get("careers_url"),
        "published_at": j.get("published_at"),
    } for j in d.get("offers", [])]

Personio publishes an XML feed (stdlib handles it fine):

import xml.etree.ElementTree as ET

def fetch_personio(slug):
    r = requests.get(f"https://{slug}.jobs.personio.de/xml", headers=UA, timeout=30)
    r.raise_for_status()
    out = []
    for p in ET.fromstring(r.content).iter("position"):
        jid = p.findtext("id")
        out.append({
            "job_id": jid,
            "title": p.findtext("name"),
            "location": p.findtext("office"),
            "url": f"https://{slug}.jobs.personio.de/job/{jid}",
            "published_at": p.findtext("createdAt"),
        })
    return out

Workday is the interesting one — it's the ATS of most large enterprises, and it can't be probed from a bare slug. A Workday career URL carries three parts: the tenant, the instance (wd1, wd5, …) and the site name — e.g. https://adobe.wd5.myworkdayjobs.com/external_experienced. All three go into the CXS endpoint, which is a POST with pagination:

import re

def fetch_workday(career_url):
    m = re.search(r"([\w-]+)\.(wd\d+)\.myworkdayjobs\.com(/[^?#]*)?", career_url)
    tenant, wd, path = m.group(1), m.group(2), m.group(3) or ""
    site = next((s for s in path.strip("/").split("/")
                 if s and not re.match(r"^[a-z]{2}([-_]\w{2,4})?$", s)), tenant)
    base = f"https://{tenant}.{wd}.myworkdayjobs.com"
    out, offset, total = [], 0, None
    while True:
        r = requests.post(f"{base}/wday/cxs/{tenant}/{site}/jobs",
                          json={"limit": 20, "offset": offset,
                                "searchText": "", "appliedFacets": {}},
                          headers=UA, timeout=30)
        r.raise_for_status()
        d = r.json()
        if total is None:
            total = d.get("total", 0)   # only present on the first page!
        posts = d.get("jobPostings", [])
        for j in posts:
            out.append({
                "job_id": (j.get("bulletFields") or [j.get("externalPath")])[0],
                "title": j.get("title"),
                "location": j.get("locationsText"),
                "url": base + (j.get("externalPath") or ""),
                "published_at": None,  # the list API only gives "Posted N Days Ago"
            })
        offset += len(posts)
        if not posts or offset >= total:
            break
    return out

Three more quirks, all Workday:

limit is capped at 20. Ask for 100 and you get 400 Bad Request. A 2,000-job board is 100 requests — budget for it.
total is only present on the first page — later pages report total: 0. Cache it from page one or your loop stops after 40 jobs (mine did).
The site name can't be guessed. nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite — nobody types that from memory. Always start from the career-page URL.

FETCHERS = {
    "greenhouse": fetch_greenhouse,
    "lever": fetch_lever,
    "ashby": fetch_ashby,
    "smartrecruiters": fetch_smartrecruiters,
    "recruitee": fetch_recruitee,
    "personio": fetch_personio,
}
# workday is separate: it needs a URL, not a slug

Step 2 — auto-detect the ATS

You usually don't know (or care) which ATS a company uses. Probe them in order and keep the first that answers with jobs:

def detect_and_fetch(slug):
    for ats, fetch in FETCHERS.items():
        try:
            jobs = fetch(slug)
        except Exception:
            continue
        # SmartRecruiters gives 200 + [] for any slug — an empty board
        # during detection means "not found", not "no openings".
        if ats == "smartrecruiters" and not jobs:
            continue
        return ats, jobs
    return None, []

>>> detect_and_fetch("linear")
('ashby', [{'job_id': '...', 'title': 'Senior / Staff Fullstack Engineer', ...}])

In production you'd cache the detected ATS per company so you don't re-probe on every run — four HTTP calls when one is enough.

Step 3 — monitor changes

Pulling all jobs is a one-liner now. The genuinely useful part is knowing what changed: which postings appeared today, which quietly disappeared, which got edited. That's a diff against the previous state.

Fingerprint each job, store {job_id: fingerprint} between runs, compare:

import hashlib, json, pathlib

STATE = pathlib.Path("state.json")

def fingerprint(job):
    src = "|".join(str(job.get(f, "")) for f in ("title", "location", "url"))
    return hashlib.sha1(src.encode()).hexdigest()[:16]

def diff(prev, jobs):
    cur = {j["job_id"]: j for j in jobs}
    changes = []
    for jid, job in cur.items():
        if jid not in prev:
            changes.append({"change": "new", **job})
        elif prev[jid]["fp"] != fingerprint(job):
            changes.append({"change": "changed", **job})
    for jid, snap in prev.items():
        if jid not in cur:
            changes.append({"change": "removed", "job_id": jid, "title": snap["title"]})
    return changes

def monitor(companies):
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    for slug in companies:
        ats, jobs = detect_and_fetch(slug)
        if not ats:
            print(f"{slug}: no supported board found")
            continue
        prev = state.get(slug)
        if prev is None:
            print(f"{slug} [{ats}]: baseline saved, {len(jobs)} jobs")
        else:
            for c in diff(prev, jobs):
                print(f"{slug}: {c['change'].upper()} — {c.get('title')}")
        state[slug] = {j["job_id"]: {"fp": fingerprint(j), "title": j.get("title")}
                       for j in jobs}
    STATE.write_text(json.dumps(state))

monitor(["stripe", "linear", "spotify"])

First run saves a baseline. Every later run prints only the delta:

stripe: NEW — Backend Engineer, Payments
linear: REMOVED — Account Executive, Growth

Put it on cron (or GitHub Actions on a schedule) and pipe the output into Slack, a spreadsheet, or an n8n/Make webhook — you'll know a company is hiring the day the posting goes live.

What it takes to run this seriously

The 100-line version above works. Running it reliably for a real watchlist grows the usual operational tail: state storage that survives machines, retry/backoff when an API hiccups, distinguishing "board is gone" from "request failed" (so you don't fire 200 false REMOVED alerts), department/salary/remote fields where each ATS hides them differently, caching ATS detection, and a scheduler that doesn't silently die.

All of that is maintenance, not insight. If you'd rather not own it, I packaged this exact pipeline — all seven ATS, Workday included — as an Apify Actor: ATS Jobs Scraper & Change Monitor:

paste slugs or career-page URLs, get the normalized dataset (department, salary where exposed, remote flags included),
monitor mode with hosted state and per-change pricing ($0.001 per job) — watching 100 companies daily costs a few dollars a month,
native @apify scheduling, dataset exports (CSV/JSON), webhooks, and it's callable by AI agents via Apify MCP.

The DIY script above gets you 80% of the way for $0 — start there (full runnable version on GitHub). When babysitting it stops being fun, the Actor is the same logic with the ops solved.

Questions about a specific ATS or an edge case? Drop a comment — I've probably hit it.