DEV Community

quiethand098
quiethand098

Posted on

Pulling remote jobs from 5 free sources into one deduplicated feed

#api #jobs #webdev #tutorial

If you build a job board, a newsletter or a salary dataset, you quickly find out that "remote jobs" is not one source. Each board has its own shape, and several of them have free public endpoints you can use without scraping HTML.

Free endpoints worth knowing

  • RemoteOK: https://remoteok.com/api returns JSON. The first element is a legal notice, skip it. Link back to the job URL, their terms ask for attribution.
  • Remotive: https://remotive.com/api/remote-jobs?category=software-dev. Documented, with a fair-use note about polling rarely.
  • Arbeitnow: https://www.arbeitnow.com/api/job-board-api paginated JSON, mostly Europe.
  • We Work Remotely: RSS feeds per category.
  • Hacker News "Who is hiring?": the monthly thread is on the HN Algolia API; each top-level comment is one posting.

The normalization problem

Every source names things differently: position vs title, tags vs category, salary as a string, a range or missing. Pick one schema early:

{ "source": "remoteok", "title": "...", "company": "...", "location": "Worldwide",
  "tags": ["python"], "salaryMin": null, "salaryMax": null, "postedAt": "2026-10-01T00:00:00Z", "url": "..." }
Enter fullscreen mode Exit fullscreen mode

Deduplicate

The same job often appears on 2-3 boards. A key of lowercase company + title + normalized location catches most of them. Keep the earliest postedAt and the list of sources.

If you do not want to maintain it

I run this pipeline as an Apify actor that merges seven sources into one deduplicated feed, with keyword, tag and date filters. You pay per job delivered, and it uses official APIs and feeds only, so it works without proxies.

Remote Jobs Aggregator on Apify

Code and notes: https://github.com/quiethand098/remote-jobs-aggregator

Top comments (1)

Collapse
 
saadhazz profile image
Saad | Data APIs •

Solid list — RemoteOK’s “skip the first element” note is exactly right (it’s the legal/last_updated object, not a job).

A small dedupe tip that saved me false merges: don’t key only on title+company. Title text drifts (“Senior” ↔ “Sr.”) and the same role can appear on Remotive and RemoteOK with different ids. Prefer the canonical apply/job URL (normalized: strip tracking params, lowercase host) as the primary key, and fall back to (company, title) only when a source has no stable URL.

Also: Remotive’s category filter returns a small page (I just got 16 for software-dev) — if you need breadth, drop the category and filter client-side, and keep the poll interval gentle per their fair-use note.