DEV Community

Cover image for Automate your job search: a daily LinkedIn jobs pipeline in Python
Andrew
Andrew

Posted on

Automate your job search: a daily LinkedIn jobs pipeline in Python

#python #automation #career #api

Job hunting has a timing problem that nobody warns you about.

A posting goes up. For the first few hours it has a handful of applicants. By day three it has two hundred, a recruiter has stopped reading carefully, and your carefully tailored application is row 187 in an ATS.

You cannot control how good the other 186 candidates are. You can control whether you were row 8 instead. That is an automation problem, and it takes about forty lines of Python.

The two filters that actually matter

LinkedIn's job search has a lot of knobs. For this purpose, two of them do most of the work:

  • Posted in the last 24 hours. Anything older is already contested.
  • Under 10 applicants. LinkedIn exposes this, and it is the single best proxy for "you will actually be read."

Everything else — seniority, remote, job type — narrows the funnel to roles you would actually take. But the two above are what turn a job feed into an edge.

Checking that by hand every morning is exactly the kind of thing you stop doing on day four. So let's not do it by hand.

One call for the data

I am using an Apify Actor that queries LinkedIn's job search and returns structured results. The endpoint below starts a run, waits, and hands back the rows — no polling loop, no login, no cookie to extract from your browser.

curl -X POST \
  "https://api.apify.com/v2/acts/data_pool~linkedin-jobs-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "keywords": ["backend engineer", "platform engineer"],
    "maxItems": 100,
    "location": "Berlin, Germany",
    "workplaceType": ["remote", "hybrid"],
    "datePosted": "24h",
    "sortBy": "date",
    "under10Applicants": true
  }'
Enter fullscreen mode Exit fullscreen mode

Each keyword runs as its own search, and results are de-duplicated across them, so overlapping search terms do not produce duplicate rows.

What comes back 

{
  "jobId": "4012345678",
  "jobUrl": "https://www.linkedin.com/jobs/view/4012345678/",
  "title": "Senior Backend Engineer",
  "location": "Berlin, Germany",
  "workplaceType": "hybrid",
  "listedAtIso": "2026-08-11T09:12:00Z",
  "easyApply": true,
  "fewApplicants": true,
  "isPromoted": false,
  "company": {
    "name": "Acme GmbH",
    "profileUrl": "https://www.linkedin.com/company/acme-gmbh"
  },
  "insights": ["Actively hiring"],
  "matchedKeyword": "backend engineer"
}
Enter fullscreen mode Exit fullscreen mode

Three fields deserve attention:

  • jobId is stable across runs. That is your deduplication key — without it, a daily job will keep showing you Monday's postings all week.
  • workplaceType comes back as a raw enum (on_site, remote, hybrid), not a pretty label. Map it before it reaches a human.
  • isPromoted tells you the company paid to boost the listing. Not automatically bad, but a promoted post that is three weeks old is a different signal from a fresh organic one.

The daily script

This is the whole thing: fetch, drop anything seen before, print a digest.

import json
import os
import pathlib
import requests

ACTOR = "data_pool~linkedin-jobs-scraper"
URL = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items"
SEEN = pathlib.Path("seen_jobs.json")

WORKPLACE = {"on_site": "On-site", "remote": "Remote", "hybrid": "Hybrid"}

SEARCH = {
    "keywords": ["backend engineer", "platform engineer"],
    "maxItems": 100,
    "location": "Berlin, Germany",
    "workplaceType": ["remote", "hybrid"],
    "datePosted": "24h",
    "sortBy": "date",
    "under10Applicants": True,
}


def fetch_jobs():
    resp = requests.post(
        URL,
        headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
        json=SEARCH,
        timeout=300,
    )
    resp.raise_for_status()
    return resp.json()


def main():
    seen = set(json.loads(SEEN.read_text())) if SEEN.exists() else set()

    fresh = []
    for job in fetch_jobs():
        job_id = job.get("jobId")
        if not job_id or job_id in seen:
            continue
        seen.add(job_id)
        fresh.append(job)

    for job in fresh:
        company = (job.get("company") or {}).get("name", "unknown")
        flags = []
        if job.get("easyApply"):
            flags.append("Easy Apply")
        if job.get("fewApplicants"):
            flags.append("<10 applicants")
        suffix = f"  [{', '.join(flags)}]" if flags else ""
        print(f"{job['title']} — {company}")
        print(f"  {job.get('location', '')} · {WORKPLACE.get(job.get('workplaceType'), '')}{suffix}")
        print(f"  {job.get('jobUrl', '')}\n")

    SEEN.write_text(json.dumps(sorted(seen)))
    print(f"{len(fresh)} new posting(s).")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it, and you get only what appeared since yesterday.

One gotcha: this endpoint returns HTTP 408 if the run takes more than 5 minutes. For a daily search that is not a concern, but if you ever crank maxItems into the thousands, switch to the async pattern (POST /runs, then poll).

Actually running it every day

The script is deliberately boring so you can schedule it however you like:

  • cron, if you have a machine that is always on: 0 8 * * * cd ~/jobsearch && APIFY_TOKEN=... python daily.py >> log.txt
  • GitHub Actions on a schedule: trigger, with the token as a repository secret — free, and no machine to maintain.
  • n8n / Make / Zapier, if you would rather the output land in Google Sheets or a Slack DM than a terminal. The same call works as a single HTTP Request node.

The digest is more useful pushed somewhere you already look. Slack DM, a Telegram bot, or an email to yourself all beat a log file you forget to open.

Filters worth knowing about

Field Why you'd use it
under10Applicants The highest-signal filter here. Apply while the pile is small.
datePosted: "24h" Keeps a daily run cheap and the results genuinely new.
easyApply One-click applications, if you're playing a volume strategy.
workplaceType remote is often geo-restricted anyway — pair it with location.
seniority entry_level through executive; filters out the mismatches.
company Watch one employer's hiring. Useful when you're targeting a shortlist.

What it costs

$0.50 per 1,000 job postings returned, with no per-run fee and no subscription.

A realistic daily search pulling 100 postings costs about $1.50 a month — comfortably inside Apify's free monthly credit, so a personal job search runs at no cost. The pricing only starts to matter if you are feeding a job board rather than a job hunt.

The honest caveats

  • LinkedIn caps what any search returns. Realistically a few hundred results per query, not every posting in existence. Several narrow searches beat one enormous one.
  • Results are not deterministic. LinkedIn's ranking shifts between runs, so treat a daily digest as "what surfaced today", not a complete index.
  • under10Applicants reflects LinkedIn's own counter, which updates on its own schedule. Treat it as a strong hint, not a guarantee.
  • The applicant count is not the whole story. Being early gets you read. It does not get you hired — that part is still on you and your CV.

Wrapping up

The interesting part of this was not the scraping. It was noticing that "posted in the last 24 hours" plus "under 10 applicants" is a genuinely different product from "job search", and that the only thing standing between those two things was a scheduled script and a stable ID to deduplicate on.

If you are hunting right now: automate the finding, and spend the time you save on the applications themselves.

Disclosure: I built the Actor used in this post. Costs quoted are its public list price in August 2026, and the caveats above are the same ones I would give you in person.

Top comments (0)