DEV Community

Zakaria Blefkih
Zakaria Blefkih

Posted on

12 job boards, one table, no API key

#api #python #tutorial #webscraping

The short answer

You can query 12 job boards in one run without an API key by using a single Apify Actor that searches LinkedIn, Indeed, Glassdoor and The Muse by default and merges eight keyless boards on request. It returns one row per role, deduplicated across boards, with 54 identical columns every run. A measured run on 8 Aug 2026 turned 90 raw listings into 89 billed rows in 39 seconds.

That is the whole article in one paragraph. The rest is the paste-ready input, the numbers behind it, and the parts that do not work.

Why "one table" is the hard part

Scraping one job board is a solved problem — there are dozens of single-board scrapers. The pain starts at board number two:

  1. The same job appears on three boards. A "Senior Data Analyst" at one company is on LinkedIn, on Indeed and on Glassdoor. Naively concatenating three scrapers' output gives you three rows, three CSV lines, three cold emails to the same recruiter.
  2. Every board names its fields differently. postedDate vs date_posted vs created_at; salary as a string on one board, as min/max integers on another, absent on a third.
  3. Company names do not match. Wipro on one board is Wipro Limited on the next, so a naive group by company never collapses them.

Deduplication has to happen across boards, on normalised company names, before you are charged for the rows.

The input, copy-paste

This is a complete run. Every other field has a working default:

{
  "searchTerm": "data analyst",
  "location": "New York, NY"
}
Enter fullscreen mode Exit fullscreen mode

Defaults you are getting for free: boards ["linkedin","indeed","glassdoor","muse"], 20 results per board, full LinkedIn job details on, Apify's datacenter proxy.

Same thing from the shell:

curl -X POST "https://api.apify.com/v2/acts/flash_scraper~multi-jobboard-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"searchTerm":"data analyst","location":"New York, NY","maxResults":30}'
Enter fullscreen mode Exit fullscreen mode

Pass the token as a header, not a query parameter — query strings end up in server logs, proxy logs, stack traces and your own shell history. The Apify API accepts Authorization: Bearer on every endpoint.

The HTTP response body is the rows. No second call to fetch a dataset.

From Python:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])   # never hardcode the token
run = client.actor("flash_scraper/multi-jobboard-scraper").call(run_input={
    "searchTerm": "data analyst",
    "location": "New York, NY",
    "maxResults": 30,
})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(rows), "jobs")
print(rows[0]["title"], "|", rows[0]["company"], "|", rows[0]["found_on_sites"])
Enter fullscreen mode Exit fullscreen mode

maxResults is per board, not per run — the single most common surprise. Four boards at 30 is up to 120 raw listings before deduplication.

What the deduplication is worth, measured

From the Actor's own README, with the dates it publishes:

Run Result
data analyst / New York, NY, 30 per board, 3 boards — 8 Aug 2026 90 raw listings -> 89 billed rows in 39 s, 1 duplicate merged across boards
Untouched form, then-default 3 boards at 20 each — 8 Aug 2026 60 rows in 27 s
Same run, salary coverage salary_min_annual filled on 74 of 89 rows (83%)
Two identical runs a day apart, before only-new mode existed 47% of rows were the same job — a daily schedule was re-buying half its data

Two columns show the merge instead of hiding it:

{
  "title": "Senior Data Analyst",
  "company": "Acme Corp",
  "site": "linkedin",
  "found_on_sites": ["linkedin", "indeed", "glassdoor"],
  "duplicate_count": 3,
  "salary_min": 158100.0,
  "salary_max": 213800.0,
  "salary_interval": "yearly",
  "date_posted": "2026-08-06",
  "job_url": "https://..."
}
Enter fullscreen mode Exit fullscreen mode

One row, one charge, and you can still see which three boards carried it. Salary fields are filled on 59-70% of rows on a default run and 56-60% with LinkedIn detail fetching off (README's own fill table, measured 2026-08-07 and re-checked across all 5,936 delivered rows on 2026-08-23) — Glassdoor publishes pay on 95-100% of its rows, Indeed on 50-80%, LinkedIn on about 35% and only with detail fetching.

The board list, including the ones that do not work

Default: LinkedIn, Indeed, Glassdoor, The Muse.
Optional keyless boards: Remotive, Jobicy, Himalayas, Hacker News "Who is hiring?", Remote OK, We Work Remotely, Working Nomads, DevITjobs US — the remote-only ones join automatically when you tick remote. DevITjobs UK is still selectable but was discontinued upstream on 2026-08-29 (re-probed 2026-09-05: its public endpoint redirects to a signup page), so 11 of the 12 selectable working boards return rows today.

Blocked at the source today and reported as blocked rather than silently returning nothing: Google Jobs, ZipRecruiter, Bayt, BDJobs, Naukri. Glassdoor also tops out at roughly 28-30 rows per query no matter what cap you set — a board-side limit, not a bug in the scraper.

This matters more than it looks: a scraper that returns 0 rows for a board and calls the run a success is indistinguishable from "there are no jobs". Check that whatever tool you use writes a per-board outcome somewhere. This one writes a RUN_SUMMARY record naming each board's result.

Remote-only searches: use the leaner Actor

If your query has no city in it, the big three boards mostly get in the way. There is a separate Actor for that shape — Remote Jobs Aggregator — which sweeps 10 keyless remote boards (RemoteOK, We Work Remotely, Working Nomads, DevITjobs US and UK, The Muse, Remotive, Jobicy, Himalayas, HN "Who is hiring?") with no key, no proxy and no login — DevITjobs UK has been dead upstream since 2026-08-29, so nine deliver rows today, and the run says so:

{ "searchTerms": ["python"], "maxItems": 100 }
Enter fullscreen mode Exit fullscreen mode

Measured default run on 15 Aug 2026: 100 deduplicated jobs from 9 boards, 6 cross-board duplicates merged, 35% of rows carrying a salary, 88% a description snippet, 100% a posting date.

One switch is worth knowing before you run it. matchDescriptions also matches your keyword inside the first 500 characters of the posting, not just the title. Measured on 15 Aug 2026 with ["python"]: 194 rows wide vs 28 title-only — roughly 7x the recall, and a marketing role whose blurb mentions "our Python stack" now matches too. Wide by default in the form, title-only over the API.

Every row carries job_id, a 16-hex fingerprint of the canonical URL — that is your upsert key for Sheets, Airtable or a CRM.

Turn it into a daily alert (the part people actually keep)

Both Actors have an only-new mode that remembers what a given search has already delivered, in a private store in your own account, for 90 days:

{
  "searchTerm": "data analyst",
  "location": "New York, NY",
  "onlyNewJobs": true,
  "webhookUrl": "https://hooks.slack.com/services/T000/B000/XXXX"
}
Enter fullscreen mode Exit fullscreen mode

First run seeds the memory. Every run after that delivers — and bills — only postings it has not seen. A measured pair: 15 rows on the first run, 0 on an identical rerun. Schedule it from the run page (Actions -> Schedule), point webhookUrl at Slack, Discord, or an n8n/Make/Zapier catch hook, and you have a job alert feed you own.

Remember the 47% number above: without only-new mode, a daily schedule pays twice for half its rows.

What it costs

Pay-per-result, $0.005 per delivered row for the multi-board Actor and $0.002 for the remote one (free-plan rates, read from the Store on 5 Sep 2026 — the Pricing tab on each Actor page is always authoritative). The remote Actor has a pricing record scheduled for 14 Sep 2026 that raises it to $0.003 per row, so if you are reading this after that date, take the figure off the Pricing tab, not off this paragraph. Deduplicated and filtered rows are never billed, and a run that delivers nothing bills only a $0.00005 run start.

Practical maths: a 4-board search at 20 per board is at most 80 rows = $0.40. Apify's free plan needs no card and its monthly credit covers several runs of that size.

What neither Actor does

  • No login, anywhere. Only public data. Nothing behind a login wall, no auto-apply.
  • No historical archive. Each run reads what the boards are showing today.
  • No guessed salary. A salary column is empty when the board published no figure — it is never inferred.
  • Five boards are blocked (listed above), and the run says so instead of pretending.

FAQ

Can I scrape LinkedIn and Indeed jobs without an API key?

Yes. Neither board offers a public jobs API to most developers, but both publish their search results as public pages, and the Actor above reads them without logging in — you only need an Apify API token to call the platform, not a key from LinkedIn or Indeed.

How do I remove duplicate jobs across job boards?

Match on the canonical job URL first, then on (normalised company name, title, location). Normalising the company is the step most home-made pipelines skip, which is why Wipro and Wipro Limited stay two rows. The Actor above does both and exposes the result in found_on_sites and duplicate_count.

How many rows can one run deliver?

A measured stress run on 7 Aug 2026 returned 258 rows from 3 boards at 120 per board in 226 seconds (that run predates a 29 Aug 2026 LinkedIn paging fix, so its 118 LinkedIn rows were not the newest 118; it has not been re-measured since). Per-board inventory, not the cap, is usually the binding limit — Glassdoor stops near 30 per query.

Is there a free way to try it?

Apify's free plan needs no card and carries a monthly usage credit; a default 4-board run costs about $0.40 of it. Both Actors also publish example tasks you can open and run as-is.

Which is the right one for remote work?

If your query has a city, use the multi-board Actor and tick remote — it applies each big board's own remote filter and enforces LinkedIn's client-side (LinkedIn's public search accepts a remote flag and ignores it, so the Actor filters those rows itself before billing). If your query has no city at all, the remote aggregator is cheaper and quieter.

Disclosure: I build and maintain both Actors (Multi Job Board Scraper, Remote Jobs Aggregator, publisher flash_scraper on Apify). Every figure in this post is published in the Actors' own READMEs with the date it was measured; Store user and run counts were read from the public Apify Store API on 5 Sep 2026. If you reproduce a run and get different numbers, tell me in the comments and I will update the post.

Top comments (0)