DEV Community

Get Anything
Get Anything

Posted on

Track new job postings the week they appear, with a scheduled scraper and a diff

#python #automation #career #webscraping

One scrape of a careers page tells you what's open. Two scrapes, a week apart, tell you what changed — and the change is where the value is.

A role that appeared three days ago has a handful of applicants. The same role a month later has hundreds. If you are job hunting, that gap is most of the advantage you can actually control. If you are selling to companies, a role that just opened is a signal that a team is growing and has budget.

Here's how to build it.

The idea

Run the same scrape on a schedule. Store each run's job IDs. Compare consecutive runs:

  • IDs in the new run but not the old one → newly posted
  • IDs in the old run but not the new one → closed or filled
  • Count per company over time → hiring momentum

That's the whole trick. The engineering is small; the discipline is in keeping a stable identifier.

Use a stable ID, not the title

The most common mistake is diffing on job title. Titles get edited. "Senior Engineer" becomes "Senior Software Engineer" and suddenly your diff reports one role closed and one opened, on the same day, for the same job.

Every applicant tracking system returns a stable ID per posting. Use it, and keep the URL as a tiebreaker:

def job_key(job):
    return job.get("jobId") or job["url"]
Enter fullscreen mode Exit fullscreen mode

Titles and locations belong in the output, never in the key.

The diff 

import json
from pathlib import Path

STATE = Path("last_run.json")

def diff(current_jobs):
    current = {job_key(j): j for j in current_jobs}
    previous = json.loads(STATE.read_text()) if STATE.exists() else {}

    opened = [current[k] for k in current.keys() - previous.keys()]
    closed = [previous[k] for k in previous.keys() - current.keys()]

    STATE.write_text(json.dumps(current))
    return opened, closed
Enter fullscreen mode Exit fullscreen mode

On the very first run, previous is empty and everything looks new. Handle that explicitly rather than emailing yourself 500 "new" jobs:

first_run = not STATE.exists()
opened, closed = diff(jobs)
if first_run:
    print(f"Baseline stored: {len(opened)} jobs. No alerts on the first run.")
else:
    print(f"{len(opened)} new, {len(closed)} closed")
Enter fullscreen mode Exit fullscreen mode

Two failure modes that will lie to you

A failed fetch looks like mass closure. If one company's board times out and you write the partial result as your new state, next week's diff reports every one of that company's jobs as "closed," then as "newly opened" the week after. Never persist state from a run where any source errored:

if errors:
    print("Skipping state update — incomplete run")
    return opened, []          # report opens, suppress the false closures
Enter fullscreen mode Exit fullscreen mode

This one bit me. A source silently returned an empty list, and the diff cheerfully announced that a company had closed every role it had.

Pagination that stops early looks the same. If your scraper quietly collects 40 of 2,000 jobs because a paginated endpoint throttled you, the diff will report 1,960 closures. Assert that the count you fetched matches the total the API reports, and fail loudly when it doesn't.

Reading momentum

Once you have a few weeks of runs, the count per company is more interesting than any single posting:

from collections import Counter

def momentum(runs):                       # runs: list of {company: count}
    latest, prior = runs[-1], runs[-2]
    return {c: latest.get(c, 0) - prior.get(c, 0) for c in set(latest) | set(prior)}
Enter fullscreen mode Exit fullscreen mode

A company that added six roles this week and four the week before is expanding. One that closed eight without opening any either finished a hiring round or stopped hiring. Both are worth knowing, and neither shows up in a single snapshot.

Weight by seniority if you want a sharper signal. A new VP posting says something different from a new intern posting: senior hires usually mean a new function is being built, not a backfill.

Scheduling it

Any scheduler works: cron, GitHub Actions, a hosted platform. Two rules regardless:

  1. Run at the same time each week. Diffs across uneven intervals aren't comparable.
  2. Keep the raw output of every run, not just the diff. When a number looks wrong — and eventually one will — you need the underlying rows to check it against. Every aggregate I've published that surprised me turned out to be a bug, and the raw data is what proved it.

Doing it without writing any of this

My ATS scraper reads five applicant tracking systems and returns a normalized record per job, with a stable ID. Point it at a list of companies, schedule it weekly, and diff the datasets. It's free to run.

The hiring-signal tracker takes it further: it does the diff for you, aggregates per company, and scores hiring momentum by role volume, seniority and week-over-week change.

Both are on my Apify page

What this is not

A diff tells you a role appeared. It doesn't tell you the team is good, the salary is fair, or the posting isn't a pipeline-filling exercise that's been reposted quarterly for two years. Being early is a real edge, and it is the only part of this a scraper can give you.

Top comments (0)