Job board scraping is the collection of public job postings from job boards, aggregators and employers' careers pages into a structured dataset, usually to measure hiring demand by role, skill, location or company. The collection is the easy part. The hard part is that one vacancy can appear five times, under five slightly different titles, and a naive count turns one job into five.
_This post covers the pipeline that makes job data countable: a clean posting model, normalisation of titles, companies, locations and salaries, layered deduplication and expiry detection. Every snippet below was run against sample postings.
_
Why duplicates happen
A single vacancy typically spreads like this:
The employer publishes it on its own careers page.
It is posted, or syndicated, to several job boards.
Aggregators pick it up from the boards and from the careers page.
It is refreshed or reposted to stay near the top of results, often with a new posting date.
Each copy may have a different title format, a shortened description, a different location string and its own posting date. Count them all and demand looks several times higher than it is.
Start with a posting model
Keep the model to posting fields: what the job is, where, for whom, at what pay, and when. Carry lineage on every record so you can always trace a figure back to its source.
{
"source_url": "https://board.example/jobs/88213",
"source_type": "board",
"source_job_id": "88213",
"title": "Sr. Data Engineer (m/f/d) - Remote",
"company": "Acme Analytics Ltd.",
"city": "London",
"country": "GB",
"work_arrangement": "remote",
"employment_type": "full_time",
"salary_text": "£45k - £55k per annum",
"description": "We are looking for a senior data engineer to ...",
"date_posted": "2026-10-03",
"valid_through": "2026-11-02",
"collected_at": "2026-10-26T05:00:00Z"
}
Where a page publishes schema.org JobPosting markup, read it first: title, hiringOrganization, jobLocation, jobLocationType, baseSalary, employmentType, datePosted and validThrough map almost directly onto this model.
Normalise before you compare
Deduplication only works on normalised values. Raw strings almost never match.
Titles and companies
Titles carry noise: gender tags common in some markets, "Remote" or "Urgent" labels, abbreviations and punctuation. Company names carry legal suffixes. Strip both to a comparable form, and capture work arrangement as its own field before the label is removed from the title.
import re
import unicodedata
SENIORITY = {"sr": "senior", "snr": "senior", "jr": "junior", "mgr": "manager", "eng": "engineer"}
NOISE = re.compile(r"\((?:m/f/d|f/m/d|m/w/d|w/m/d|all genders)\)|\b(?:remote|hybrid|urgent|new)\b", re.I)
def _fold(text: str) -> str:
text = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode()
return re.sub(r"\s+", " ", re.sub(r"[^a-z0-9 ]", " ", text.lower())).strip()
def normalise_title(title: str) -> str:
words = _fold(NOISE.sub(" ", title)).split()
return " ".join(SENIORITY.get(w, w) for w in words)
LEGAL = re.compile(r"\b(?:ltd|limited|plc|llc|inc|incorporated|gmbh|ag|sa|sas|bv|pvt|co)\b")
def normalise_company(name: str) -> str:
return re.sub(r"\s+", " ", LEGAL.sub(" ", _fold(name))).strip()
"Sr. Data Engineer (m/f/d) - Remote" and "Senior Data Engineer" both become senior data engineer. "Acme Analytics Ltd." and "ACME Analytics Limited" both become acme analytics.
For analysis, you will usually also map normalised titles to a standard occupation taxonomy, but keep that as a separate step: dedup on the normalised title, classify afterwards.
Locations
Split locations into city, region and country, with country as an ISO code. Treat "remote" and "hybrid" as a work arrangement, not a place. A remote role open to one country is still tied to that country for analysis.
Salaries
Salary text comes in every shape: ranges, "k" shorthand, hourly, daily and monthly rates. Parse to an annualised minimum and maximum, and keep the period that was quoted:
import re
from decimal import Decimal
PERIODS = {"hour": 2080, "hourly": 2080, "day": 260, "daily": 260,
"week": 52, "weekly": 52, "month": 12, "monthly": 12,
"year": 1, "annum": 1, "annual": 1, "yearly": 1}
CURRENCIES = {"£": "GBP", "€": "EUR", "₹": "INR"} # "$" is ambiguous: take currency from the source
AMOUNT = re.compile(r"(\d[\d,]*(?:\.\d+)?)\s*(k)?", re.I)
def parse_salary(text: str, default_currency: str) -> dict | None:
"""'£45k - £55k per annum' -> {'min': 45000, 'max': 55000, 'currency': 'GBP', 'period': 'year'}"""
amounts = [Decimal(n.replace(",", "")) * (1000 if k else 1)
for n, k in AMOUNT.findall(text)]
if not amounts:
return None
period = next((p for p in PERIODS if re.search(rf"\b{p}\b", text, re.I)), "year")
symbol = next((s for s in CURRENCIES if s in text), None)
factor = PERIODS[period]
return {
"min": min(amounts) * factor,
"max": max(amounts) * factor,
"currency": CURRENCIES.get(symbol, default_currency),
"period_quoted": period,
}
"$38 - $42 per hour" becomes 79,040 to 87,360 a year, assuming 2,080 working hours. Store that assumption with the data, and only parse the salary field itself: run this over a full description and numbers like "40 hours a week" will leak in.
Deduplicate in layers
No single rule catches every duplicate without merging distinct jobs. Use three layers, cheapest first.
Layer 1: identifiers. The same source job ID or the same canonical URL means the same posting. This catches refreshes on a single site.
Layer 2: a normalised key. Company, title, city and country, all normalised, form a blocking key. Postings that share it are candidate duplicates:
def dedup_key(posting: dict) -> str:
return "|".join([
normalise_company(posting["company"]),
normalise_title(posting["title"]),
_fold(posting.get("city", "")),
posting.get("country", "").upper(),
])
Layer 3: description similarity. An employer hiring three data engineers in London for different teams will produce three postings with the same key. Compare descriptions to separate them, using word shingles and Jaccard similarity:
def shingles(text: str, k: int = 5) -> set[str]:
words = re.findall(r"[a-z0-9]+", text.lower())
return {" ".join(words[i:i + k]) for i in range(max(len(words) - k + 1, 1))}
def similarity(a: str, b: str) -> float:
sa, sb = shingles(a), shingles(b)
return len(sa & sb) / len(sa | sb) if sa | sb else 0.0
Then cluster: within each key, group postings whose descriptions are similar and whose posting dates are close.
from collections import defaultdict
def cluster_postings(postings: list[dict], threshold: float = 0.8) -> list[list[dict]]:
"""Group postings that describe the same job."""
by_key = defaultdict(list)
for p in postings:
by_key[dedup_key(p)].append(p)
clusters = []
for group in by_key.values():
group.sort(key=lambda p: p["date_posted"])
assigned: list[list[dict]] = []
for p in group:
for c in assigned:
close_in_time = abs((p["date_posted"] - c[0]["date_posted"]).days) <= 30
if close_in_time and similarity(p["description"], c[0]["description"]) >= threshold:
c.append(p)
break
else:
assigned.append([p])
clusters.extend(assigned)
return clusters
def canonical(cluster: list[dict]) -> dict:
"""Prefer the employer's own careers page, then the earliest, most complete record."""
return min(cluster, key=lambda p: (
p["source_type"] != "employer",
p["date_posted"],
-sum(v not in (None, "") for v in p.values()),
))
In testing, a board copy with "Apply today." appended scored 0.93 against the careers-page original and joined its cluster, while a same-titled posting for a different team stayed separate. The careers-page record was chosen as canonical. Pairwise comparison is fine within blocking keys; at very large scale, swap the Jaccard step for MinHash with locality-sensitive hashing.
Keep every source in the cluster rather than discarding duplicates. Which boards carried a job, and when, is useful data in its own right.
Detect when a job closes
Demand metrics depend on knowing which jobs are still open. Combine three signals:
validThrough in structured data, where the employer provides it.
Removal: the posting returns a 404 or 410, or redirects to a search page.
Absence: the posting has not been seen on any source for several consecutive collection runs.
Mark a cluster closed only when every source agrees, and record the close date. "Time to fill" estimates come from exactly this field.
What you get
After normalisation and deduplication, each cluster is one real vacancy with a clean title, company, location and annualised salary range, a list of every place it appeared, and an open or closed status. That is what makes counts trustworthy: open roles by occupation, new roles per week, salary ranges by city, hiring momentum by company.
Collect from public pages only, at a polite rate, and respect each site's terms and robots.txt. If you need this across many boards and countries without maintaining the pipeline, managed job data aggregation delivers postings already normalised and deduplicated.
FAQ
How do you remove duplicate job postings?
Use layers. First match on source job IDs and canonical URLs. Then group postings by a normalised key of company, title, city and country. Finally compare descriptions within each group, using shingle-based similarity and a posting-date window, so that genuinely separate roles with the same title are not merged.
What fields should a job posting dataset include?
Title, company, city, region, country, work arrangement, employment type, salary range with currency and period, description, date posted, closing date, and the source URL and collection time for every record.
How do you normalise salaries from job postings?
Parse the salary field into minimum and maximum amounts, take the currency from the symbol or the source, identify the quoted period (hourly, daily, monthly or annual) and convert to an annual figure using stated assumptions, while keeping the original period alongside.
Top comments (0)