DEV Community

Aurelix Solutions
Aurelix Solutions

Posted on

Remote jobs from Remote OK and We Work Remotely in one deduplicated list with Python

Remote job newsletters, niche job boards and job-search tools all start from the same few boards. Remote OK and We Work Remotely both publish public feeds, but in different formats, with different categories and with the same job often posted on both.

This post pulls both boards into one deduplicated list in Python, filters to remote roles open to one country with a minimum salary, takes a quick salary snapshot, and sets up a daily feed of new jobs.

Disclosure: I'm Aurelix, and I built Remote Job Board Scraper, the tool used here. It's a paid tool (prices below).

What each row holds

28 fields per job:

  • title, company, company_logo_url, tags.
  • category (one of 12 normalised categories) and the board's own category_raw.
  • employment_type (normalised, e.g. FULL_TIME, CONTRACT) and employment_type_raw.
  • locations (the allowed regions as published) and a worldwide flag.
  • salary_min, salary_max, salary_currency, salary_interval where the board publishes a salary.
  • posted_at, expires_at, job_url, apply_url, source, an attribution line, and also_listed_on when the same job is on the other board.
  • With includeDescription on, description_text and description_html.

Python and data jobs open to Canada or the US

pip install apify-client
Enter fullscreen mode Exit fullscreen mode
import csv
import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/remote-jobs-scraper").call(
    run_input={
        "keywords": ["python", "data engineer"],
        "categories": ["software-development", "data-ai"],
        "regions": ["Canada", "USA"],
        "minSalary": 80000,
    }
)
jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items())

fields = ["title", "company", "category", "locations", "salary_min", "salary_max", "salary_currency", "posted_at", "source", "job_url"]
with open("remote_jobs.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
    w.writeheader()
    w.writerows(jobs)
Enter fullscreen mode Exit fullscreen mode

regions matches words in the published location text (country names, two-letter codes such as GB, or time zones such as UTC), and includeWorldwide (on by default) keeps jobs open to anyone anywhere. minSalary compares the top of the published range, converted to yearly, and drops jobs without a salary.

A quick salary snapshot

from collections import Counter
from statistics import median

paid = [j for j in jobs if j["salary_max"] and j["salary_interval"] == "year" and j["salary_currency"] == "USD"]
if paid:
    print(f"{len(paid)} of {len(jobs)} jobs publish a yearly USD salary")
    print(f"median top of range: ${median(j['salary_max'] for j in paid):,.0f}")

print("by category:", Counter(j["category"] for j in jobs).most_common())
print("companies hiring most:", Counter(j["company"] for j in jobs).most_common(5))
Enter fullscreen mode Exit fullscreen mode

New jobs every morning

run_input = {
    "keywords": ["python", "data engineer"],
    "regions": ["Canada"],
    "incremental": True,
    "stateKey": "remote-python-canada",
    "slackWebhookUrl": "https://hooks.slack.com/services/...",
}
Enter fullscreen mode Exit fullscreen mode

With incremental on, each run returns only jobs no earlier run returned, matched across both boards. Save the input as a task in the Apify Console and schedule it daily: Slack gets a short digest of the new jobs, and nothing is sent when there are none. webhookUrl posts the same digest as JSON to Zapier, Make or n8n.

Why this tool

  • Two boards, one schema, with a normalised category and employment type.
  • A job on both boards comes back once.
  • Region and minimum-salary filters.
  • An only-new mode matched across both boards, with a Slack or webhook digest.
  • Cheaper than the leading tools.

Limits

  • Coverage is what the boards publish: Remote OK's feed gives its ~100 newest jobs and We Work Remotely's feeds their newest jobs per category. By default each board returns up to 100 jobs; maxResultsPerSource: 0 returns everything they publish.
  • Categories come from keyword rules on titles and board categories; unusual titles can land in other.
  • Region matching is by word, synonym and country code, not geography: Europe does not match a list of European countries.
  • Neither board publishes seniority, so that column stays empty.
  • Apply links go to the job page on the source board, as the boards require.

Cost

$0.90 per 1,000 jobs on Apify's free plan, from $0.48 per 1,000 on paid plans, plus a start fee per run. A job on both boards is charged once. The current prices are on the Store page; Apify's free plan includes $5 of credit a month, about 5,500 jobs.

Use it from an AI assistant

The tool also works through Apify's hosted MCP server, so Claude, ChatGPT or Cursor can run it from a request like "remote data engineer jobs open to Canada paying at least $100k, posted this week". Server URL: https://mcp.apify.com/?tools=aurelix/remote-jobs-scraper.

Links

Top comments (0)