Building a Job Market Data Pipeline: From ATS Endpoints to Salary Normalization

Job market data is one of the most requested scraping targets I've ever encountered — and one of the most misjudged. Everyone starts with the same idea: "aggregate postings from the big boards, build a search engine." Then reality arrives: LinkedIn has litigated aggressively against scraping, several large boards run Cloudflare or Akamai with enterprise configs, postings duplicate and rot at a horrifying rate, and the actual analytical value turns out to live less in any single post and more in the aggregation: what's being hired, where, at what pay, and how fast roles open and close.

This post walks through building a job-market data pipeline that survives contact with the real web — source selection, legal posture, extraction, deduplication, salary normalization, and the freshness loop that keeps it honest.

Source Strategy: Start Where the Friction Is Lowest

Before writing a line of code, rank your sources by (a) legal risk, (b) technical difficulty, and (c) uniqueness of data. My rough ranking after several of these projects:

1. Aggregators and company career pages. Greenhouse, Lever, Ashby, Workable and similar ATS platforms expose structured job-board JSON for every customer company. It's fast, stable, legal-risk-light, and the posting is the source of truth — posted by the employer directly. For many "who is hiring" analyses, ATS coverage plus a crawl of careers pages gets you 80% of the signal.
2. RSS and official APIs. Some boards and aggregators publish feeds or offer partner APIs.
3. Public sector and academic job boards. Often no anti-bot at all, high-quality structured data.
4. The big consumer boards. Highest volume, highest anti-bot, highest legal sensitivity. If your project touches these, read their terms carefully, throttle hard, and — this is genuine advice, not a disclaimer — consider whether the marginal data justifies it, because regulators and courts in several jurisdictions have treated job-posting data as carrying commercial restrictions.

The ATS route deserves a concrete example, because it's the best-kept non-secret in this space. A company using Greenhouse has a public board at boards-api.greenhouse.io/v1/boards/{company}/jobs:

import httpx, asyncio
from datetime import datetime, timezone

ATS_ENDPOINTS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{org}/jobs",
    "lever": "https://api.lever.co/v0/postings/{org}?mode=json",
    "ashby": "https://api.ashbyhq.com/posting-api/job-board/{org}",
}

async def fetch_atlas_org(client: httpx.AsyncClient, provider: str, org: str) -> list[dict]:
    url = ATS_ENDPOINTS[provider].format(org=org)
    r = await client.get(url, timeout=30)
    r.raise_for_status()
    data = r.json()
    jobs = data.get("jobs", data if isinstance(data, list) else [])
    for job in jobs:
        job["_source"] = f"{provider}:{org}"
        job["_fetched_at"] = datetime.now(timezone.utc).isoformat()
    return jobs

One request per company, no rendering, no proxies needed, and the payloads include location, department, and — with Greenhouse's ?questions=true — even the application form structure. Maintain a registry of company → ATS mappings (a few thousand well-known tech companies covers an enormous share of hiring signal), and you have a professional-grade feed.

When You Do Need the Heavy Boards

If your analysis genuinely requires consumer-board volume — e.g., non-tech occupations, gig platforms, regional coverage — the technical playbook looks like this. Expect TLS/HTTP2 fingerprinting, per-IP quotas that are low, geo-variance in results (the same search returns different postings by request location), and aggressive session invalidation. The architecture that holds up:

• Geo-matched residential proxies, sticky per session. Search results are localized; collect each region through an exit IP in that region, and keep that IP for the whole session so the cookie jar and IP stay in lockstep. Thordata's username-suffix sticky sessions work well here — one session ID per region per crawl run.
• Search-oriented crawling, not exhaustive crawling. You cannot re-crawl every posting daily. Crawl the search index (which gives you title, company, location, salary band, posted-date cheaply) at high frequency, and visit individual posting pages only for new or changed entries.
• A browser fallback lane. Some pages need rendering; keep a small Playwright pool for those, routed through the same proxy sessions, and treat it as an escalation path rather than the default.

import random, time

def crawl_region(session, region: str, query: str, max_pages: int = 5):
    """Search-index crawl for one region. One sticky proxy session per region."""
    results = []
    for page in range(0, max_pages):
        params = {"q": query, "l": region, "start": page * 10}
        r = session.get(SEARCH_URL, params=params, timeout=30)
        if r.status_code in (403, 429):
            backoff_and_rotate(session)   # yield region, resume later
            return results
        results.extend(parse_cards(r.text))
        time.sleep(random.uniform(5, 12))  # search pages are watched harder
    return results

Deduplication: The Actual Hard Problem

Here is what nobody warns you about: the same job posting exists in many places — the employer's ATS, their careers page, two or three aggregators, and a big board — each with a different ID, slightly different titles, and different timestamps. Naive pipelines count these as separate postings and systematically overestimate demand. Dedup is not a nice-to-have; it's the core of the pipeline.

Deduplicate on two levels. First, exact-ish matching on normalized (company, title, location) tuples:

import re
from hashlib import sha256

def norm_text(s: str) -> str:
    s = re.sub(r"[^a-z0-9 ]", " ", s.lower())
    return re.sub(r"\s+", " ", s).strip()

TITLE_CANON = {"sr.": "senior", "sr": "senior", "jr.": "junior", "jr": "junior"}

def posting_key(company: str, title: str, location: str) -> str:
    t = norm_text(title)
    for short, full in TITLE_CANON.items():
        t = t.replace(f" {short} ", f" {full} ")
    blob = f"{norm_text(company)}|{t}|{norm_text(location)}"
    return sha256(blob.encode()).hexdigest()

Second, fuzzy matching for the residue — near-duplicate descriptions. MinHash or SimHash over the description text catches reposted-with-edits listings. Keep a cluster ID: canonical posting → N source observations, each with its own URL and first/last-seen timestamps.

Salary Normalization and Structured Extraction

Job postings encode pay in a hundred dialects: ranges, floors, "up to," hourly vs. yearly, "competitive," currency-dependent on country. Build a small parser pipeline:

SALARY_RE = re.compile(
    r"\$?(?P<lo>[\d,.]+)(?:\s*(?:-|–|to)\s*\$?(?P<hi>[\d,.]+))?\s*(?P<unit>k|/hr|/hour|per hour|/yr|annually)?",
    re.I,
)

def parse_salary(text: str) -> dict | None:
    m = SALARY_RE.search(text)
    if not m:
        return None
    lo = float(m.group("lo").replace(",", ""))
    hi = float(m.group("hi").replace(",", "")) if m.group("hi") else lo
    unit = (m.group("unit") or "").lower()
    if unit == "k" or lo < 1000 and hi < 1000:  # "$120k" or "$25-$35"
        annual = "k" in unit or lo >= 40
        lo = lo * 1000 if ("k" in unit or lo >= 40) else lo * 2080  # hourly→annual
        hi = hi * 1000 if ("k" in unit or hi >= 40) else hi * 2080
        return {"lo_usd": lo, "hi_usd": hi}
    return {"lo_usd": lo, "hi_usd": hi}

(Yes, the heuristic "≥40 means annual, <40 means hourly" is crude — tune per market and add currency conversion for multi-country pipelines.) For titles and skills, a small canonical occupation taxonomy (I use a lightly customized O*NET mapping) turns free-text titles into analyzable categories. From there the aggregations write themselves: postings per occupation per metro per week, median posted salary, time-to-fill proxies from first-seen to disappearance.

The Freshness Loop

Postings are short-lived. A pipeline that doesn't track disappearance systematically will accumulate a growing pile of zombie postings. The pattern that works: every crawl, record (posting_key, seen_at). A posting not observed for N consecutive crawls of its own region/query — where N accounts for search-index volatility, so 3–5 misses, not 1 — is marked closed, with last_seen as the closing time bound. Because search-index crawls are cheap and deterministic (same queries, same regions), misses mean something. And this is another reason the search-index-first architecture wins: you can afford the revisit cadence that disappearance detection requires.

Wrapping Up

A job-market pipeline is 20% scraping and 80% everything else: source strategy weighted toward the low-friction ATS endpoints, session-stable geo-matched crawling for the boards you truly need, two-tier dedup with canonical posting clusters, dialect-tolerant salary parsing, and a disappearance loop that keeps the inventory honest. Get the identity layer right and even modest volume yields real labor-market insight — hiring velocity by metro, pay compression by role, which companies are quietly building teams.

Disclosure: I use Thordata's residential proxies for the geo-distributed board crawling described in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*