Turning a jobs scraper into an AI agent's tool with the Apify MCP server

There is a lot of good advice around on making your Actor agent-friendly. Almost none of it shows you the thing an agent actually sends over the wire — and that turned out to be the part that changed how I build.

I build Actors on Apify under the name primeflowio. One of them, ATS Jobs Scraper & Change Monitor, takes a list of companies and returns every open job they have, normalized into one schema — no matter which applicant tracking system the company uses. A few weeks after publishing it, I connected it to the Apify MCP server and an agent called it like a native tool: discovered it, filled the input, ran it, read the results back. No glue code on my side.

The Origin column says MCP: that run was started by an agent, not by me clicking Start.

So this article does two things. First it walks the protocol by hand — every request, every real response — so you can see exactly what your Actor is answering to. Then it goes through what I changed once I had read those responses, including a bug that would have told a customer that every job at a company had disappeared overnight.

The problem the Actor solves

Job postings are scattered across ATS platforms. If you track 200 target companies — as recruiters, B2B sales teams, and serious job seekers do — you'd need to visit 200 career pages, sitting on Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee or Personio, each with its own markup.

The twist most people miss: you don't need to scrape any of those pages. The major ATS platforms expose documented, public endpoints for their job boards. Here is every platform the Actor supports, with what each one actually returned when I re-checked it by hand this week:

ATS	Endpoint	Live check (Sep 2026)
Greenhouse	boards-api.greenhouse.io/v1/boards/{slug}/jobs	stripe: 611, gitlab: 230
Lever	api.lever.co/v0/postings/{slug}?mode=json	spotify: 73
Ashby	api.ashbyhq.com/posting-api/job-board/{slug}	ramp: 143, linear: 28
Personio	{company}.jobs.personio.de/xml	personio: 1 open role
Recruitee	{company}.recruitee.com/api/offers/	yource: 1 open role
Workday	{tenant}.{wd}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs	nvidia: 2,000 — and slow
SmartRecruiters	api.smartrecruiters.com/v1/companies/{slug}/postings	see below

No API keys, no proxies, no anti-bot arms race. The hard part isn't access — it's detection (which ATS does this company use?), normalization (a different response shape per platform, one schema out), and change tracking.

Three of those rows deserve a footnote, and they are the same lesson in different clothes.

Workday is the expensive one. It answers, and it answers with everything — two thousand postings for one tenant — but a small run against it took 83 seconds where Ashby takes two. That is fine for a watchlist on a schedule and a bad idea for anything with a stopwatch on it, which is why my prefilled input does not point at Workday.

Recruitee gets the failure mode right, and it is worth copying. An unknown company returns 404 {"error":"Not Found"}. A real company with nothing open returns 200 {"offers":[]}. Those are different sentences, and a caller can act on the difference.

Which brings us to SmartRecruiters, which does not. It answers 200 {"totalFound": 0, "content": []} for every identifier I tried — including companies that visibly run their careers site on SmartRecruiters, and including identifiers that do not exist at all. An endpoint that cannot distinguish "no openings" from "wrong name" from "this company never switched the public API on" is not a data source; it is a coin flip with a JSON wrapper. The Actor now reports that board as unreadable instead of returning an empty list that looks like an answer — which turns out to be the same bug I had shipped myself, one layer up. More on that in lesson 6.

So the Actor does three things:

1. Auto-detect — given a company slug or careers URL, it probes the supported endpoints and figures out which ATS answers.
2. Normalize — every job becomes one record: title, department, team, location, remote, employment_type, salary_min/max, url, published_at, updated_at.
3. Monitor — in monitor mode it keeps the previous run's state in the Actor's key-value store and emits new / removed / changed records as a diff. Users point a webhook or an n8n flow at those and get "company X just opened 3 engineering roles" without re-downloading everything.

That last part is the reason the Actor exists at all: the incumbent job-data Actors make you re-pull the full dataset every run and charge per job every time. If what you actually want is changes, you're paying for the same rows daily.

Every Actor is already an MCP tool

Here's the part that cost me zero extra work. The Apify MCP server exposes any public Actor as a tool over the Model Context Protocol. You don't register anything, you don't write a manifest — the Actor's input schema is the tool definition. An agent connects to:

https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper

and gets a tool named primeflowio--ats-jobs-scraper, with parameters generated from the same input schema that renders the form in Apify Console. (?tools= is the parameter in Apify's docs; ?actors= is accepted too — I checked both today and they return the same five tools.)

I wanted to see exactly what an agent sees, so I did the whole handshake by hand with curl before touching any framework. MCP over streamable HTTP is three steps.

1. Initialize a session:

curl -s -D - -X POST "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize",
       "params":{"protocolVersion":"2025-03-26","capabilities":{},
                 "clientInfo":{"name":"demo-agent","version":"1.0"}}}'

The response headers contain mcp-session-id — every later call carries it in an Mcp-Session-Id header, plus a notifications/initialized message to finish the handshake.

2. List tools. {"method":"tools/list"} returns, in my session:

primeflowio--ats-jobs-scraper
get-actor-run
get-dataset-items
get-key-value-store-record
abort-actor-run

Note what the server adds for free: the agent doesn't just get my Actor, it gets the helpers to check a run's status and page through the results dataset. That completes the loop — an agent can start a run, poll it, and fetch data without any Apify-specific code.

3. Call the tool:

curl -s -X POST "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Mcp-Session-Id: $SESSION" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call",
       "params":{"name":"primeflowio--ats-jobs-scraper",
                 "arguments":{"companies":["https://jobs.lever.co/spotify"],
                              "mode":"pull","maxJobsPerCompany":3}}}'

The run finished in under two seconds and the MCP response came back with a summary the agent can act on directly:

"status":"SUCCEEDED",
"statusMessage":"1/1 companies, 3 jobs, 0 changes, 0 errors"
...
SUCCEEDED in 1.635s. 4 items; 24 fields available.
Use get-dataset-items with datasetId=pl8b6nClKueN3efCO and limit
(for example 20) to fetch items (4 total). Available fields (dot notation):
record_type, company, ats, job_id, title, department, team, location,
remote, employment_type, salary_min, salary_max, salary_currency, url, ...

The whole exchange from a plain client: five tools discovered, one called, a summary back in under three seconds.

Read that response again from an LLM's point of view. It's not a blob of JSON rows — it's an instruction: here's what happened, here's the tool call to make next, here are the fields you can ask for. The Apify MCP server writes that guidance itself. My only job was to make sure the run summary (statusMessage) and the dataset fields were worth reading.

The same flow in Python

For anything scripted I use ~60 lines of plain requests — no MCP SDK needed for the happy path. Full file in the repo; the core is:

MCP_URL = "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper"

def rpc(session, method, params=None, msg_id=None):
    payload = {"jsonrpc": "2.0", "method": method}
    if params is not None: payload["params"] = params
    if msg_id is not None: payload["id"] = msg_id
    headers = dict(HEADERS)
    if session: headers["Mcp-Session-Id"] = session
    r = requests.post(MCP_URL, headers=headers, json=payload, timeout=300)
    r.raise_for_status()
    return r

init = rpc(None, "initialize", {...}, msg_id=1)
session = init.headers["mcp-session-id"]
rpc(session, "notifications/initialized")

tools = parse_sse(rpc(session, "tools/list", msg_id=2).text)

result = parse_sse(rpc(session, "tools/call", {
    "name": "primeflowio--ats-jobs-scraper",
    "arguments": {"companies": ["https://jobs.lever.co/spotify"],
                  "mode": "pull", "maxJobsPerCompany": 3},
}, msg_id=3).text)

One implementation detail that trips people up: responses arrive as server-sent events (Content-Type: text/event-stream), even for a single JSON-RPC reply. parse_sse just takes the last data: line and json.loads it.

Plugging the same URL into Claude, Cursor, or any MCP-capable client is a config entry instead of code — the point of the protocol is that the curl session above is exactly what those clients do under the hood.

What building for agents changed in my Actor design

I originally designed the input schema for humans filling a form in Apify Console. Agents read the same schema, but they punish different mistakes. Six lessons I'd apply to any Actor now:

1. The input schema is a tool contract — keep required fields to one. My schema has exactly one required field, companies. Everything else (mode, maxJobsPerCompany, includeSnapshot) has a sane default. An agent that has to guess six required parameters will guess some of them wrong; an agent that has to provide one list of company names almost can't fail.

2. Give agents a cost throttle. maxJobsPerCompany exists so a cautious caller can cap the bill of an exploratory run. My test cost under a cent: with pay-per-event pricing, 1 company check plus 3 job results is $0.005. An agent (or the human supervising it) can try the tool for pocket change before pointing it at a 500-company watchlist. If your Actor can generate unbounded output, give the caller a knob to bound it.

3. Make statusMessage machine-readable prose. "1/1 companies, 3 jobs, 0 changes, 0 errors" is short enough to fit in any context window and precise enough that an agent can decide what to do next without fetching the dataset at all. I treat that one string as part of the API surface now.

4. Put bulk data in the dataset, not the response. The MCP server already nudges this: it returns a summary plus "use get-dataset-items with this ID" rather than inlining rows. Fighting that — returning huge payloads inline — wastes the agent's context and your users' tokens. Design the output so the first page of dataset items answers the most common question.

One schema out, whichever ATS answered. This is the page the agent fetches after the summary tells it what's there.

5. Your defaults are load-bearing: the platform runs them. Apify's automated QA runs every public Actor with its prefilled input and expects success within about 5 minutes. Mine was ["linear", "https://jobs.lever.co/spotify"] — two boards, ~130 postings, chosen back when I was thinking about humans clicking "Start" and wanting to see something substantial. In August an upstream endpoint got slow, three consecutive QA runs failed, and the Actor was flagged "under maintenance" in the Store until the runs went green again. I found out from the notification emails, days later. Worth knowing what that clock is: under Apify's publishing terms, an Actor left broken for 30 days can be deprecated and removed.

The lesson isn't "make the Actor faster". It's that the prefill is a health check the platform runs on your behalf, so it should exercise the cheapest, most reliable path you have. I've since cut mine to a single small board: 28 postings, 2.2 seconds against a five-minute budget. The same defaults make agent test-calls cheap, which is a happy coincidence — QA bots and cautious agents want exactly the same thing from you.

Ivan Solovyev landed on the same wall from a different direction — his prefill hit four marketplaces at once and the test gave up waiting while paying users ran the Actor happily all day. Two of us tripped over the same default in the same month, which says something about how easy it is to write a prefill for the demo rather than for the robot.

6. An empty answer is not "everything disappeared". This one I found because of that article, and it is the most expensive bug I have shipped.

In monitor mode the Actor compares this run's jobs against the previous run's state and emits new / removed / changed. The failure path was already handled: if a job board errors out, the run records an error for that company, keeps the old state, and moves on. What I had not separated was a board that answers 200 with an empty list — which happens during migrations, when a board is briefly unpublished, or when the provider hiccups.

That path was ok, jobs = True, []. The diff then did exactly what it was told: marked every known job as removed, and overwrote the state with nothing. A customer watching Stripe would have received "611 jobs removed" one morning and "611 jobs new" the next, with every webhook firing twice on fiction.

The guard is small, and the shape of it generalises to any stateful Actor:

# A board that answers 200 with zero jobs is usually the source blinking, not
# every role closing at once. Keep the old state, say so, and only believe the
# zero once it repeats.
if prev_jobs and not jobs and streak + 1 < EMPTY_RUNS_BEFORE_TRUSTED:
    push_items([{"record_type": "warning", "company": slug, "ats": ats,
                 "detail": f"{ats} returned an empty board while "
                           f"{len(prev_jobs)} jobs were known; previous state "
                           f"kept, no changes emitted",
                 "empty_streak": streak + 1}])
    state_put(store_id, cache_key,
              {"ats": ats, "jobs": prev_jobs, "empty_streak": streak + 1})
    continue

I tested it by seeding the state with two known jobs for a board that currently answers empty. Before: two removed records and a wiped state. After: 1/1 companies, 0 jobs, 0 changes, 0 errors, 1 stale, one warning record, and both jobs still in state. Real mass closures are slow; sources blink instantly — so a zero has to earn its credibility over three consecutive runs before the Actor will act on it.

The general rule, which is the same one Solovyev arrives at from the billing side: an Actor with memory must distinguish what the world says from what it managed to hear. For a stateless scraper an empty result is a result. For anything that diffs, an empty result is an accusation.

Where this goes next: monitor mode plus agents

Pull mode answers "who is company X hiring right now?" — a natural one-shot agent query ("did Spotify open any Android roles this month?"). Monitor mode is more interesting for agent workflows because it inverts the direction: instead of the agent polling, the Actor runs on Apify's scheduler, emits only new / removed / changed records, and a webhook or n8n flow wakes the agent only when something changed. "Head of Data role appeared at a target account" is a buying signal; an agent that gets only diffs can react to it without burning tokens re-reading yesterday's 611 postings.

One platform detail makes that possible, and it is the part I expected to be hard. Monitor mode needs memory that survives between runs, and those runs belong to whoever calls the Actor, not to me. Actors run with limited permissions by default, which sounds like the end of the idea. The docs say a limited-permissions Actor may "create any additional storage, and write to that storage" and "read and write to storages created in previous runs", and my own runs — the log line reads Running under "LIMITED_PERMISSIONS" — create a named key-value store, then read that same state back on later runs and diff against it. So the state can live on the caller's account rather than in a database of mine, and I never have to ask anyone for a token. I have only exercised that on my own account, so treat the cross-account half as documented rather than as something I measured; the design decision it points to is what matters. If you are building anything stateful, look at named storage before you reach for infrastructure of your own.

The diff mode earns its keep on live data, by the way. While I was writing this, a watchlist run caught Databricks editing a marketing role 29 minutes after they touched it — one changed record out of thirteen companies and roughly three and a half thousand postings. That is the argument for a change feed in a single line: the agent reads one row instead of 3,500.

That's the pattern I'd summarize the whole experience with: the MCP server made my Actor callable by agents in an afternoon, but making it worth calling — one required input, bounded costs, a status line an LLM can parse, diffs instead of dumps — was ordinary API design, applied to a new consumer.

FAQ

Do I need to modify an Actor to expose it via MCP? No. Any public Actor works through mcp.apify.com immediately; the input schema becomes the tool signature. Whether it's a good tool is a design question — see the six lessons above.

What does a call cost? Whatever the Actor's pricing says — same as running it from Console or API. This Actor is pay-per-event: $2 per 1,000 companies checked, $1 per 1,000 job results, so the demo call above cost $0.005.

Is scraping ATS job boards allowed? The endpoints used here are the platforms' own public job-board APIs — Greenhouse Job Board API, Lever Postings API, Ashby Posting API, Recruitee's offers endpoint, Personio's job XML feed. They exist precisely so postings can be syndicated to aggregators and career-site widgets, which is why none of them need a key.

Can the agent also fetch the results? Yes — the MCP session includes get-actor-run and get-dataset-items helpers automatically, so the agent can page through results without any Apify SDK.

Try it yourself

Point an agent at your own watchlist. Add the MCP server to any MCP-capable client:

https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper

…then ask it something you'd actually want to know — "who is Stripe hiring in Berlin right now?" — and watch it pick the arguments out of the input schema on its own. A single company check plus a handful of postings costs half a cent, so the first call is cheap enough to be curious with. Run it a second time a week later with mode: "monitor" and you get the diff instead of the dump.

The Actor is ATS Jobs Scraper & Change Monitor on Apify Store. The demo client and the full curl transcript from this article are at github.com/strongboxr/ats-mcp-agent-demo.

---

I build data Actors on Apify as primeflowio. If a number in this article has a caveat, the caveat is in the sentence next to it.

Suggested meta description: How an Apify Actor that scrapes ATS job boards became an AI agent tool via the Apify MCP server — the full handshake, working code, and six design lessons.