Your hiring chart may measure who posts most often

Dataspryva — Thu, 22 Oct 2026 13:13:53 +0000

Your Friday investigation needs an industry ranking. On Thursday evening, industry.value_counts() looks reassuringly tidy. The uncomfortable question is whether those bars measure hiring demand or visibility in a recent-results feed.

Totaljobs serves anonymous searches newest-first. Relevance ordering requires a signed-in session and can't be combined with the filters, so the Actor always returns newest-first results.

A capped collection therefore selects through posting recency. Recently posted adverts can enter the collection while older vacancies remain open beyond its reach. A chart can describe the collected adverts perfectly and still answer the wrong question.

Fresh adverts get more chances to appear

The tempting assumption is that matching an occupational keyword produces a representative cross-section of that occupation. Matching and representation are separate properties.

An agency might repeatedly advertise similar roles while a direct employer keeps an advert open through a longer application cycle. A recent-results feed can favour the former pattern. That's a mechanism to investigate, not evidence that agencies dominate your collection.

Keep these boundaries beside your notebook.

Coverage boundary	What you can establish
Public listing scope	UK vacancies on Totaljobs, reached through keyword and area searches, area browsing or supplied URLs
Primary key	`jobId` identifies a listing, not an underlying hiring requirement
Search ceiling	Each query exposes at most 100 jobs; larger requests trigger automatic splitting across facets
Collection ceiling	`maxItems` accepts up to 5000; requesting that ceiling doesn't establish complete coverage
Field availability	Empty fields are omitted; detail fetching and published information determine availability

Start with an explicit search boundary and retain full snapshots.

Example: this input requests nursing vacancies in Manchester, with a chosen posting window and collection ceiling.

import os

import duckdb
import pandas as pd
from apify_client import ApifyClient

run_input = {
    "mode": "search",
    "keywords": ["registered nurse"],
    "location": "Manchester",
    "postedWithin": "7",
    "includeDetails": True,
    "onlyNewJobs": False,
    "maxItems": 500,
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "dataspryva/totaljobs-uk-vacancy-finder-by-keyword-area"
).call(run_input=run_input)

if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection ended with status {run['status']}")

The input defines an observable slice. It doesn't establish its share of the underlying vacancy population.

Where the search boundary bites

Leaving location empty searches UK-wide; specifying a town, city, county or postcode narrows the search. That requested area isn't proof of a precise workplace boundary, particularly for remote or multi-site roles.

Keywords are searched separately. A keyword match doesn't establish an occupational classification, and industry carries the site's classification rather than an independently verified industry assignment.

postedWithin accepts 1, 3, 7 or 14 days. Every returned job is checked against its posting date, including where the site's date control is unavailable.

That excludes older adverts that may still be open.

Salary filtering creates another boundary: it keeps overlapping annualised advertised pay ranges and excludes jobs without published pay. An industry chart made from that collection describes salary-disclosing adverts.

The legacy employment inputs full-time, internship and apprenticeship are accepted but ignored because the site no longer offers those filters. An accepted input isn't evidence that the restriction happened.

The collection path looks like this:

matching public adverts
        ↓
newest-first searches across reachable slices
        ↓
job-ID deduplication and collection ceiling
        ↓
observed adverts available for analysis

Splitting doesn't create a random sample. Reaching your ceiling means collection stopped there; falling short doesn't establish that every matching advert was reachable.

Neither outcome supplies a population coverage percentage.

Missing dates can make a clean chart misleading

Opening job pages enables exact posting dates, descriptions, parsed salary bands and company details. Coordinates and employer profile information still depend on what's published.

When a chart drops records without industry or companyId, its denominator becomes “adverts with that field available”. That may be a different slice from the collection as a whole.

Dates need particular care. Without detail fetching, postedDate can contain a relative label instead of an ISO timestamp. Silently discarding those labels can make the remaining records look unusually well dated.

firstSeenAt records when onlyNewJobs first returned the vacancy. It isn't the employer's posting timestamp.

Keep comparable snapshots together

For repeated collection, preserve the input and keep onlyNewJobs disabled. Incremental returns suppress previously seen IDs, so they can't measure repeated visibility across full snapshots.

The next cell combines the current dataset with optional earlier dataset IDs from the comma-separated environment variable PREVIOUS_DATASET_IDS. Use earlier datasets collected with the same input.

Streaming avoids treating a response page as the entire dataset.

previous_ids = [
    value.strip()
    for value in os.environ.get("PREVIOUS_DATASET_IDS", "").split(",")
    if value.strip()
]
dataset_ids = list(dict.fromkeys(
    previous_ids + [run["defaultDatasetId"]]
))

rows = [
    {**item, "_dataset_id": dataset_id}
    for dataset_id in dataset_ids
    for item in client.dataset(dataset_id).iterate_items()
]

columns = [
    "_dataset_id", "jobId", "title", "company", "companyId",
    "companyUrl", "location", "locationPostcode", "industry",
    "postedDate", "postedDateText", "scrapedAt",
]
frame = pd.DataFrame(rows).reindex(columns=columns).astype("string")
frame = frame.replace(r"^\s*$", pd.NA, regex=True)

Names beginning with _ belong to this analysis. They aren't returned fields.

With only the current dataset, you can audit dates and field availability. Comparisons across collection times require earlier snapshots.

How old are the adverts you received?

Use scrapedAt as the record's observation timestamp, keeping exact dates separate from relative labels.

This cell parses numeric day labels without pretending they identify an exact posting instant.

frame["_posted_at"] = pd.to_datetime(
    frame["postedDate"], format="ISO8601", errors="coerce", utc=True
)
frame["_scraped_at"] = pd.to_datetime(
    frame["scrapedAt"], format="ISO8601", errors="coerce", utc=True
)
frame["_age_days"] = (
    frame["_scraped_at"] - frame["_posted_at"]
).dt.total_seconds() / pd.Timedelta(days=1).total_seconds()

relative_text = frame["postedDateText"].fillna(frame["postedDate"])
frame["_label_days"] = pd.to_numeric(
    relative_text.str.extract(
        r"(?i)^\s*(\d+)\s+days?\s+ago\s*$", expand=False
    ),
    errors="coerce",
).astype("Int64")

Other label formats remain unparsed. We'd rather expose the gap than manufacture a timestamp.

The following cell calculates field coverage per pull and separate distributions for exact posting ages and relative labels.

coverage = frame.groupby("_dataset_id").agg(
    saved_rows=("jobId", "size"),
    distinct_job_ids=("jobId", "nunique"),
    missing_job_ids=("jobId", lambda values: values.isna().sum()),
    dated_rows=("_posted_at", "count"),
    timestamped_rows=("_scraped_at", "count"),
    industry_rows=("industry", "count"),
    usable_age_rows=("_age_days", lambda values: values.ge(0).sum()),
    negative_ages=("_age_days", lambda values: values.lt(0).sum()),
).reindex(dataset_ids).fillna(0).astype("int64")

coverage["rows_without_exact_posting_date"] = (
    coverage["saved_rows"] - coverage["dated_rows"]
)

age_by_day = (
    frame.loc[frame["_age_days"].ge(0)]
    .assign(_age_day=lambda data: data["_age_days"].floordiv(1).astype("Int64"))
    .groupby(["_dataset_id", "_age_day"])
    .size()
    .rename("adverts")
)

relative_labels = (
    frame.groupby(["_dataset_id", "_label_days"])
    .size()
    .rename("adverts")
)

Inspect coverage before interpreting age_by_day. Missing timestamps never enter the age bins, and negative ages need investigation.

These calculations describe returned records. They can't reveal the posting ages of adverts excluded by the search ceiling.

Different listing IDs don't prove a repost

Deduplication removes repeated appearances of the same jobId across overlapping search slices. A replacement advert with a different ID remains a different listing.

Conversely, a familiar title doesn't prove deletion and reposting. Similar positions can represent different shifts, sites or hiring requirements.

This cell constructs repeat_candidates, a grouped review table with counts and distinct listing IDs calculated from your snapshots.

frame["_employer_key"] = (
    ("id:" + frame["companyId"])
    .fillna("url:" + frame["companyUrl"])
    .fillna(
        "name:" + frame["company"].str.casefold()
        .str.replace(r"\s+", " ", regex=True).str.strip()
    )
)
frame["_title_key"] = (
    frame["title"].str.casefold()
    .str.replace(r"\s+", " ", regex=True).str.strip()
)
frame["_place_key"] = (
    frame["locationPostcode"].fillna(frame["location"])
    .str.casefold().str.replace(r"\s+", " ", regex=True).str.strip()
)

connection = duckdb.connect()
connection.register("observations", frame)

repeat_candidates = connection.sql("""
    SELECT
        _employer_key,
        _title_key,
        _place_key,
        count(*) AS observed_rows,
        count(DISTINCT _dataset_id) AS pulls_seen,
        count(DISTINCT jobId) AS distinct_listing_ids,
        list(DISTINCT jobId ORDER BY jobId) AS listing_ids
    FROM observations
    WHERE _employer_key IS NOT NULL
      AND _title_key IS NOT NULL
      AND _place_key IS NOT NULL
      AND jobId IS NOT NULL
    GROUP BY _employer_key, _title_key, _place_key
    HAVING count(DISTINCT _dataset_id) > 1
       AND count(DISTINCT jobId) > 1
    ORDER BY distinct_listing_ids DESC, pulls_seen DESC
""").df()

Each row groups a matching advertiser, title and place. Its counts distinguish repeated observations from distinct IDs, while the ID list gives you the listings to inspect.

Employer-name fallbacks are weaker matches than profile identifiers. Missing place information excludes a record from this check, so an empty queue wouldn't establish an absence of reposting.

An agency comparison also needs a checked advertiser classification. applyType distinguishes internal from external application routes. It doesn't identify an agency, and an advertiser profile needn't identify the organisation employing the successful applicant.

A convenient field is rarely grateful when asked to do a different job.

Compare industry shares before writing the caption

The collector used here's Totaljobs UK Vacancy Finder by Keyword, Area & Pay.

Before publication, compare industry percentages from pooled feed observations with percentages after keeping each listing ID once. This measures how repeated observations affect your chart; it doesn't remove replacement adverts carrying new IDs.

This cell creates the comparison table, including counts, percentages and their difference in percentage points, while retaining unpublished industry classifications as a visible category.

identified = frame.loc[frame["jobId"].notna()].copy()
identified["_industry"] = identified["industry"].fillna(
    "(industry not published)"
)

distinct_listings = (
    identified.sort_values(
        "_scraped_at", na_position="first", kind="stable"
    )
    .drop_duplicates("jobId", keep="last")
)

industry_comparison = pd.DataFrame(
    index=pd.Index(identified["_industry"].unique(), name="industry")
)
industry_comparison["raw_observations"] = (
    identified.groupby("_industry").size()
    .reindex(industry_comparison.index, fill_value=0)
)
industry_comparison["distinct_listings"] = (
    distinct_listings.groupby("_industry").size()
    .reindex(industry_comparison.index, fill_value=0)
)
industry_comparison = industry_comparison.astype("int64")

raw_total = industry_comparison["raw_observations"].sum()
listing_total = industry_comparison["distinct_listings"].sum()

industry_comparison["raw_share_pct"] = (
    industry_comparison["raw_observations"]
    / (raw_total if raw_total else float("nan")) * 100
)
industry_comparison["listing_share_pct"] = (
    industry_comparison["distinct_listings"]
    / (listing_total if listing_total else float("nan")) * 100
)
industry_comparison["share_change_pp"] = (
    industry_comparison["listing_share_pct"]
    - industry_comparison["raw_share_pct"]
)
industry_comparison = industry_comparison.sort_values(
    "distinct_listings", ascending=False
)

Both denominators exclude missing listing IDs, which coverage counts separately. Where an ID has different industry values across observations, the distinct-listing calculation prefers its latest timestamped observation.

With comparable full snapshots, raw shares weight adverts by their observed appearances. Distinct-listing shares give each observed ID equal weight. With only the current collection, this comparison can't diagnose persistence across pulls.

Neither column is a corrected estimate of hiring demand.

A defensible caption can say:

Distinct observed vacancy adverts returned for the specified keyword, area and posting window through newest-first searches.

Beside the chart, disclose the collection ceiling, filters, observation period and missing classifications. If you pool snapshots, identify whether the bars count observations or distinct listing IDs.

Counting identified advertisers offers another unit of analysis, provided you disclose identity coverage. It doesn't repair unseen vacancies.

The hiring decision remains off the page

Public adverts don't establish headcount, filled positions, hiring urgency or whether recruitment represents expansion, replacement or an ongoing requirement.

The posting date doesn't establish when an employer first needed someone. Similar adverts with different identities can support further checking, but can't prove deliberate feed manipulation.

You can describe which industries occupy the collected feed and how that composition changes under comparable search conditions. Claiming which industries are aggressively hiring requires evidence beyond the feed.

A posting timestamp measures an advert's recency. It doesn't measure an employer's appetite.

DEV Community: Dataspryva