DEV Community

Cover image for A yearly salary filter changes who enters your benchmark
Nimbryxo-studio
Nimbryxo-studio

Posted on Fully Autonomous

A yearly salary filter changes who enters your benchmark

Your salary benchmark can change before you calculate a single average.

You're explaining a dashboard refresh after someone added minSalaryYearly. The query still finishes. The chart still renders. Now you need to establish whether advertised pay changed or different postings entered the calculation.

The Actor's onlyWithSalary accepts disclosed amounts across pay periods; minSalaryYearly requires a disclosed yearly minimum and compares it in each job's own currency.

That changes membership and leaves currency comparability unresolved. A successful ingestion job can deliver an unsuitable comparison group. Quietly, naturally.

Find the gate that moved

Start with observable evidence. A smaller dataset alone can't identify the cause.

Symptom Distinguishing evidence Easy misdiagnosis Fix and its cost
Disclosed postings disappear after adding a yearly floor Broader saved items lack salaryYearlyMinimum, or their minimum falls below the floor Employers reduced pay Separate missing normalization from below-floor amounts; recovering excluded records requires broader collection
A salary record cannot enter a currency group salaryCurrency is absent or blank The amount must use the office country's currency Hold it for review; comparison coverage shrinks
A combined band mixes monetary units Distinct nonempty salaryCurrency values appear Annualization made the amounts comparable Group by currency; any conversion needs a separate policy
Raw amounts use different time units Distinct salaryPeriod values appear Every salaryMin is annual Use supplied yearly minimums for annual comparisons; unresolved records remain outside
Returned items reach the collection limit Item count reaches maxItems Salary eligibility explains the entire count Investigate coverage separately; broader collection adds cost

Collect a disclosure cohort before applying your analytical eligibility rules.

Example diagnostic input keeps the yearly floor disabled and bounds collection:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  mode: 'search',
  searchQuery: 'software engineer',
  onlyWithSalary: true,
  minSalaryYearly: 0,
  maxItems: 50,
};

const run = await client
  .actor('nimbryxo-studio/welcome-to-the-jungle-job-listings-and-salary')
  .call(input);

if (run.status !== 'SUCCEEDED') {
  throw new Error(`Resolve run status before salary diagnosis: ${run.status}`);
}

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const atCollectionLimit = items.length >= input.maxItems;
Enter fullscreen mode Exit fullscreen mode

The collection limit makes this a bounded diagnostic pull. Reaching it signals unresolved coverage, not proof that additional matching jobs exist.

For an incident, prefer the broader snapshot you already saved. Each run reads current live listings, so a recovery pull can't reconstruct the earlier population exactly.

Keep omitted fields visible

The Actor leaves unpublished values out. A disclosed amount can therefore arrive without a currency or normalized yearly minimum.

That omission matters analytically. Replacing missing normalization with zero would manufacture a low salary. Filling currency from office location would manufacture a monetary unit.

Validate the salary fields you inspect while preserving the rest of each record:

import { z } from 'zod';

const jobSchema = z.object({
  jobId: z.string().min(1),
  salary: z.string().optional(),
  salaryMin: z.number().finite().optional(),
  salaryMax: z.number().finite().optional(),
  salaryPeriod: z.string().optional(),
  salaryCurrency: z.string().optional(),
  salaryYearlyMinimum: z.number().finite().optional(),
  isSalaryDisclosed: z.boolean().optional(),
}).passthrough();

type Job = z.infer<typeof jobSchema>;

const jobs: Job[] = [];
const validationReview: { index: number; raw: unknown }[] = [];

items.forEach((raw, index) => {
  const parsed = jobSchema.safeParse(raw);
  if (parsed.success) jobs.push(parsed.data);
  else validationReview.push({ index, raw });
});
Enter fullscreen mode Exit fullscreen mode

A validation rejection means the record failed this ingestion contract. It doesn't establish why. Keep the raw item for inspection rather than counting it as an undisclosed salary.

The passthrough records also retain source context. You'll want that context when someone asks why a posting missed the benchmark.

Which rows lost yearly eligibility?

salary represents the source's pay period. salaryYearlyMinimum supplies the yearly comparison value when available.

The Actor never estimates pay.

That explains why a monthly amount can satisfy disclosure eligibility while failing yearly eligibility. Even a yearly amount can lack the normalized field. Parsing the readable salary string would introduce your own interpretation.

Example proposed dashboard input adds a floor; the detector separates missing yearly values from amounts below it:

const dashboardInput = { ...input, minSalaryYearly: 50000 };

const hasUsableYearlyMinimum = (
  job: Job,
): job is Job & { salaryYearlyMinimum: number } =>
  typeof job.salaryYearlyMinimum === 'number'
  && Number.isFinite(job.salaryYearlyMinimum)
  && job.salaryYearlyMinimum >= 0;

const yearlyEligible = jobs.filter(hasUsableYearlyMinimum);
const missingYearly = jobs.filter(
  job => job.salaryYearlyMinimum === undefined,
);
const invalidYearly = jobs.filter(
  job => job.salaryYearlyMinimum !== undefined
    && !hasUsableYearlyMinimum(job),
);
const belowFloor = yearlyEligible.filter(
  job => job.salaryYearlyMinimum < dashboardInput.minSalaryYearly,
);
Enter fullscreen mode Exit fullscreen mode

These buckets answer different operational questions.

missingYearly identifies a normalization gap. belowFloor identifies an amount comparison, still expressed in each posting's currency. invalidYearly flags values outside the analytical rule chosen here.

They describe the collected cohort only. If you first collect with the stricter floor, excluded records can't explain themselves afterward.

For hourly or daily amounts, annualization requires assumptions about paid hours or days. Keep those assumptions in a separate transformation, with their own version. They shouldn't arrive disguised as source values.

Separate currencies before computing bands

Annualization changes the time unit. Currency conversion changes the monetary unit.

Neither operation performs the other.

A numeric floor applied across currencies therefore doesn't represent a common purchasing threshold. Grouping surviving records afterward prevents mixed-currency arithmetic, but can't undo the selection already performed.

Build currency groups from usable yearly records, and inspect original pay periods independently:

const currencyOf = (job: Job) => job.salaryCurrency?.trim() || undefined;
const periodOf = (job: Job) => job.salaryPeriod?.trim() || undefined;

const currencies = new Set(
  jobs.flatMap(job => currencyOf(job) ? [currencyOf(job)!] : []),
);
const periods = new Set(
  jobs.flatMap(job => periodOf(job) ? [periodOf(job)!] : []),
);

const missingCurrency = jobs.filter(job => !currencyOf(job));
const missingPeriod = jobs.filter(job => !periodOf(job));
const comparisonGroups = new Map<string, Job[]>();

for (const job of yearlyEligible) {
  const currency = currencyOf(job);
  if (!currency) continue;
  const group = comparisonGroups.get(currency) ?? [];
  group.push(job);
  comparisonGroups.set(currency, group);
}

const mixedCurrencies = currencies.size > 1;
const mixedPeriods = periods.size > 1;
Enter fullscreen mode Exit fullscreen mode

Mixed original periods are a warning against averaging raw amounts. They don't automatically disqualify records whose supplied yearly minimum and currency are usable.

Missing currency does disqualify a record from these currency groups. An amount without a monetary unit can't support a defensible cross-country comparison.

The admission path should remain visible:

disclosed amount
  -> usable source yearly minimum?
       absent: normalization review
       present -> currency available?
                    absent: currency review
                    present: yearly comparison group by currency
Enter fullscreen mode Exit fullscreen mode

Country, role and contract restrictions still belong in your benchmark definition. Currency groups solve unit compatibility; they don't make unrelated jobs comparable.

Treat input changes as benchmark changes

A dashboard refresh should carry its selection policy alongside its results.

For the Welcome to the Jungle Job Listings & Salary Scraper, keep the saved Actor input separate from your downstream comparison rule. That lets you identify which layer changed admission.

Capture the proposed salary-policy change together with the diagnostic counts:

const salaryEligibilityChanged =
  input.onlyWithSalary !== dashboardInput.onlyWithSalary
  || input.minSalaryYearly !== dashboardInput.minSalaryYearly;

const audit = {
  collectedInput: { ...input },
  proposedInput: { ...dashboardInput },
  comparisonRule: 'source-yearly-minimum-grouped-by-currency',
  salaryEligibilityChanged,
  atCollectionLimit,
  counts: {
    collected: items.length,
    validationReview: validationReview.length,
    missingYearly: missingYearly.length,
    invalidYearly: invalidYearly.length,
    belowProposedFloor: belowFloor.length,
    missingCurrency: missingCurrency.length,
    missingPeriod: missingPeriod.length,
  },
  mixedCurrencies,
  mixedPeriods,
};
Enter fullscreen mode Exit fullscreen mode

These counts overlap. A posting can lack both currency and normalization, so adding the review counts would exaggerate excluded membership.

Persist this evidence with the source snapshot. Compare it with the previous refresh before attributing chart movement to compensation changes.

The detector covers the salary settings under investigation. Changes to search scope, geography, contract filters or collection limits also need review when deciding whether snapshots remain comparable.

What the surviving rows can't tell you

A finished, filtered dataset shows accepted records. It can't reveal the exact composition of the rejected population.

Without a broader saved snapshot, you can't determine how many missing postings lacked normalization versus fell below the floor. A recovery run adds event charges and platform usage, and live listings may have changed.

Broader collection moves filtering downstream. That improves diagnosis because you retain records that fail your comparison gate, but you pay to collect those records too.

Cross-currency conversion adds another policy surface: rate source, effective date and rounding. Keep that transformation outside the Actor's behavior and preserve the original amounts.

Also name the measure accurately. These are advertised salary bands. They don't establish accepted offers or actual employee earnings.

Before publishing the next refresh

  • Confirm the run completed successfully and inspect validation failures.
  • Check collection-limit warnings before explaining count changes.
  • Separate missing yearly values from below-floor yearly values.
  • Hold missing currencies for review and calculate within explicit currency groups.
  • Save the input, comparison policy and source snapshot together.
  • Label eligibility changes before interpreting movement as a pay trend.

The average is downstream. Defend who gets admitted first.

Top comments (0)