DEV Community: northbell

My scraper returned 660 jobs. There were 880. Nothing in the output said so.

northbell — Tue, 01 Sep 2026 11:36:14 +0000

The first version of my LinkedIn company-jobs Actor looked like it worked. Point it at a company, get back every open role. The dataset had hundreds of rows, the fields were populated, the run finished green.

It was returning about three quarters of the jobs, and a different three quarters each time.

Worse, I had built change tracking on top of it — "these 12 roles opened since your last run, these 5 closed" — and that feature was manufacturing closures out of nothing. One test run reported 113 jobs closed and 117 opened, 75 seconds apart.

Nothing in the output distinguished this from a correct answer. That is the specific failure I want to write about, because "did my scrape actually finish?" turns out to be a question you can measure, not just hope about — using a method that ecologists use to count fish.

LinkedIn's public job search does not paginate

The public endpoint takes a start offset. I assumed start=0 gives you jobs 1–10, start=10 gives you 11–20, and so on.

It doesn't. It gives you ten jobs. Which ten is not stable.

I ran the same query twice and compared the returned job IDs:

3 pages deep: 33% of the IDs were different
10 pages deep: 61% of the IDs were different

So a straightforward loop — walk start from 0 upward until a page comes back short — does not collect a company's roster. It collects a sample of it. On a small company the sample happens to be everything. On a large one it isn't, and nothing tells you which case you're in.

Two things were broken by this, and only one of them was obvious.

The obvious one: change tracking. If run A samples 660 of 880 and run B samples a different 660, roughly 220 IDs are in A and not in B. Those get reported as closed. They were never closed. They were never absent — they just weren't in this draw.

The one I nearly missed: the basic listing was wrong too. Not the fancy feature — the core product. I was selling "every open role at this company" and shipping a list missing a quarter of it, with no indication that anything was missing.

Two stopping rules that don't work

My first attempt at "am I done" was the obvious heuristic: stop when four consecutive requests add nothing new.

Here's why that's not a measurement. Say the company has 700 roles and I currently hold 650. Each request returns 10 roles drawn from the pool. The chance that all ten are ones I already have:

// P(a page adds nothing) with 650 of 700 held, 10 drawn:
(650/700) ** 10   // ≈ 0.478

Nearly half. Four in a row happens about 5% of the time at that exact point — and more often as you get closer. So the rule does fire eventually. It just fires at a completeness level that depends on the fraction you already hold, which is precisely the unknown you were trying to determine. The stopping rule is circular: it tells you you're done by assuming you're nearly done.

My second attempt went the other way: collect the roster twice and only accept it if the two passes match exactly. Rigorous, and useless. Two random samples of a large pool are essentially never identical. The condition never fired, so the feature never produced a number, so I had built an elaborate way of saying "I don't know."

The answer was in the data I already had

The thing that unstuck me was reframing what those repeated draws are. They aren't retries. They are samples.

Ecologists count fish in a pond by catching some, tagging them, releasing them, then catching a second batch and seeing how many carry tags. Few tags means a big pond. That's Lincoln–Petersen:

N ≈ (size of sample A × size of sample B) / (number in both)

I already had two samples — I just had to stop merging them into one bucket. So the Actor now splits its own requests into two sets, tracks them separately, and estimates the total from their overlap:

export function estimateTotal(sampleA, sampleB) {
  const a = sampleA.size, b = sampleB.size;
  let overlap = 0;
  for (const id of sampleA) if (sampleB.has(id)) overlap += 1;
  return { total: Math.round((a * b) / overlap), a, b, overlap };
}

Then coverage is just what I hold divided by what I estimate exists, and "how many am I still missing" falls out of the same number.

Here is a real run against a company page, 800 requests, recording the unique jobs held at each point:

requests	unique jobs
40	345
80	617
160	795
240	844
320	867
400	878
520	880
520–800	880 (zero new in 280 requests)

Estimated total from the overlap: 880. Actual point of exhaustion: 880.

The pool wasn't unreachable. I had been stopping at 93 requests.

Bug 1: I split the samples along the wrong axis

The first version of the split used request parity — odd-numbered requests into sample A, even into sample B. The estimate came back at 1,334 for a company with 880 roles. Fifty percent too high, and confidently so.

The reason took me longer than it should have. The offset also advances once per request. So odd requests were looking at odd pages, and even requests at even pages. The two samples were drawing from different parts of the pool, which makes their overlap unnaturally small, which inflates N. Lincoln–Petersen assumes both samples come from the same population; mine didn't.

The fix is to split by sweep — one full pass over all offsets — rather than by request:

const sweep = Math.floor(r / pages);
const start = (r % pages) * PAGE_SIZE;
...
(sweep % 2 === 0 ? evenSample : oddSample).add(j.jobId);

Now both samples cover every offset. The estimate landed on 880.

Bug 2: the formula collapses, and it collapses quietly

This one reached production before I caught it, and it produced the worst kind of output: a confident, plausible, wrong number.

The two samples were 20 and 681 items, with an overlap of 20. Run the formula:

N = 20 × 681 / 20 = 681

The estimate equals the count I already had. Coverage: 100%. The Actor concluded it had collected everything, stopped, compared against the previous run, and reported 199 jobs closed. None had closed.

When one sample is entirely contained in the other, the formula returns the larger sample. It isn't an error — it's what the math says when your "second sample" is really just a subset of the first. And an overlap of 20 comfortably passes a naive "overlap must be at least 10" check.

So the guards aren't about size, they're about shape:

const MIN_SAMPLE = 20;
const MIN_BALANCE = 0.5;   // smaller ÷ larger

if (Math.min(a, b) < MIN_SAMPLE) return { total: null, reason: 'sample-too-small' };
if (Math.min(a, b) / Math.max(a, b) < MIN_BALANCE) {
  return { total: null, reason: 'samples-unbalanced' };
}
if (overlap < 10) return { total: null, reason: 'overlap-too-small' };

Plus one more condition outside the function: don't trust an estimate until both samples have completed at least one full sweep. The degenerate case above happened when the run stopped after one sweep plus two requests, leaving one sample with two requests' worth of data.

Note what total: null means here. It is not zero and it is not 100%. It means I cannot tell, and it travels to the output with a reason attached.

Bug 3: I fixed the wrong variable

My first instinct on the collapse was to raise the minimum sample size from 20 to 50. Bigger samples, less degeneracy, done.

That broke companies with about 25 open roles. Too many to fit in one page (10 per request), too few to ever reach a 50-item sample. Mid-sized companies became undecidable — the Actor could neither confirm nor estimate, so it withheld everything.

The collapse was never caused by absolute size. It was caused by imbalance — 20 versus 681. Raising the floor treated a symptom and disabled a whole class of correct cases. So the absolute minimum went back down to 20, and the real protection lives in the balance ratio and the two-sweep rule.

Small companies get a separate path entirely, based on a fact rather than an estimate: if no response ever filled a full page, there is no second page, and what I have is everything. Measured page churn at one page deep is 0%, so this is safe — and it matters, because mark-and-recapture needs an overlap of 10+ to work at all, which means the smaller the company, the less able it is to estimate. Exactly backwards, if you don't handle it separately.

Bug 4: my own request cap looked like a wave of layoffs

A production run reported closed: 20. Nothing had closed. I had changed the per-company cap from 30 to 20 between runs.

Truncate a list, compare it to an untruncated one, and the rows you didn't collect appear as rows that vanished. Same for changing a filter: narrow the location, and everything outside it "closes."

So before any comparison, the Actor now checks whether the two observations are comparable at all, and refuses to produce numbers when they aren't:

if (ctx.truncated) return blank('truncated-this-run');
if (prevPoint.truncated) return blank('previous-run-was-truncated');
if (ctx.filterKey !== prevPoint.filterKey) return blank('filters-changed');

opened and closed come back as null with a reason, instead of numbers that look real.

What the Actor reports now

Every company row carries the measurement alongside the data:

openJobs: 880
estimatedTotal: 880
coverage: 1
estimatedMissing: 0
changeMarginOfError: 0
collectionComplete: true
requestsUsed: 523

changeMarginOfError is the one I'd argue hardest for. If this run missed m roles and the previous run missed m', the apparent change is muddied by up to m + m'. A run at 99% coverage on 880 roles is missing about 9 — so a reported change of ±18 is inside the noise. That is why the target coverage for change tracking defaults higher than the target for just listing jobs. A number without its error bar invites a conclusion it can't support.

One more measurement, since it costs nothing to state: LinkedIn returns HTTP 400 for start values of 1000 and above. In an 800-request experiment, exactly 20 requests failed — 1000, 1010, … 1190 — and every other request succeeded. The offsets below 1000 are enough, because each pass returns a different draw. Circling 0–990 repeatedly reaches everything.

What I'd take from this

"No new results" is not evidence of completeness. It's evidence about the fraction you already hold, which is the thing you were trying to measure. Any stopping rule built on it is circular.

Repeated requests are samples, not retries. The moment I stopped merging them into one bucket, the data I already had answered the question I thought I needed a new experiment for.

A degenerate formula returns a number, not an error. 20 × 681 / 20 = 681 is arithmetically fine and semantically garbage. If a formula has a collapse mode, guard its shape — and make the guard return "I don't know" rather than a default.

Silence and completeness are indistinguishable unless you report the difference. Before this work, "660 jobs" and "880 jobs" looked identical from the outside. Now the row says which one it is, and how sure it is.

The estimator, the guards and the tests that pin down each of these four bugs are at northbell-dev/honest-scraping — no dependencies, npm test runs the 40 tests with nothing to install.

The Actor is LinkedIn Company Jobs Scraper. It reads only public pages — no login, no cookies, and the request headers are a frozen object that cannot carry one.

northbell builds honest web scrapers on Apify. If a derived number has an error margin, the margin ships with it.

I scraped 89 job postings to find out how my salary parser was lying to me

northbell — Tue, 01 Sep 2026 10:50:34 +0000

I shipped a LinkedIn jobs scraper that returns salary as structured numbers — salaryMin, salaryMax, currency, and a yearly figure so an hourly warehouse job and a director role sit on the same axis. The whole point was to make "show me jobs paying over $150k" a question you can actually ask.

Then I ran it against 89 real postings and found six separate ways it was wrong.

None of them threw an error. Every single one produced a confident-looking number. That is the part worth writing about: a salary parser doesn't fail loudly, it fails plausibly, and the only way to find out is to sit down with real postings and check them one by one.

Here's what I found, in the order I found it, including the one where the platform had already solved my problem and I broke it by "fixing" it.

Why job descriptions are hostile to salary parsing

My first assumption was that the salary would live in a structured field. It doesn't.

I pulled 64 postings across eight roles and countries — software engineer in the US, registered nurse in California, retail associate in Texas, truck driver, marketing manager in New York, software developer in the UK, data analyst in Canada, accountant in the US — and looked for where the money actually was:

The top-card salary box: 3 out of 64
JSON-LD baseSalary: 0 out of 64

So the salary is in the free-text description, mixed in with every other dollar figure a company might mention. And companies mention a lot of dollar figures:

"We're a Series A company with $20M+ raised from Headline, Village Global"
"serving an $800 billion market"
"New Hire Bonus: $3,000 after 90 days"
"[$5,000] signing bonus available to external hires"
"Annual maximum out of pocket $3,000 per individual / $7,500 per family"
"we have grown from $1 million in sales in 1983 to more than $500 million today"

Not one of those is a salary. A naive regex that grabs $[\d,]+ returns all of them.

My first version had a plausibility gate for exactly this reason: normalize any candidate to a yearly figure first, and if it lands outside $8,000–$2,000,000, it isn't a wage. That one rule kills $20M raised, $800 billion market, and $1 million in sales instantly, because nobody is paid $800 billion a year.

It also killed a real salary, which is where the trouble started.

Bug 1: the bonus in the next sentence poisoned the salary

Here's a real posting:

Salary Range: $46.00 - $50 per hour   New Hire Bonus: $3,000

My parser rejected it. Not the bonus — the salary.

The reason: I was testing my noise words (bonus, signing, raised, market) against a window of context around the candidate. $46.00 - $50 per hour had "New Hire Bonus" sitting 8 characters to its right, so the whole match got thrown out.

The fix was to stop treating context as a blob and start treating it as directional. A number's label is what comes before it. What comes after belongs to the next number:

// Drop it only if this number's own label is noise.
// Whatever sits on the right belongs to the next number, not this one.
if (NOISE_BEFORE.test(left)) return null;

NOISE_BEFORE is anchored to the end of the left context:

const NOISE_BEFORE = /(bonus|signing|sign[-\s]?on|relocation|referral|stipend|allowance|raised|funding|valuation|revenue|market|worth|in\s+(sales|arr)|series\s+[a-e])\b[\s:_*\-]*$/i;

The $ anchor is the whole trick. "Bonus:" immediately before a number means that number is a bonus. "Bonus" thirty characters later means nothing about this number at all.

Once I made that change, the pay range came back and the $3,000 bonus stayed rejected — which is exactly right.

Bug 2: a following word's first letter became a magnitude suffix

This one was invisible until I checked a number I had no reason to suspect.

1st year compensation plans range from $55,000-$95,000 Medical, dental, life

Parser said: no salary found.

I instrumented the candidate loop and printed what it actually matched:

raw=|$55,000-$95,000 M|

There it is. My money regex ended with [kKmM]? to catch $150K and $20M. The word after the range was Medical. The regex ate its first letter as a magnitude suffix, read $95,000 M as ninety-five billion, and the plausibility gate — correctly, given what it was handed — threw the whole thing away.

A parser that turns $95,000 into $95,000,000,000 and then silently drops the job is worse than one that crashes. The job just quietly isn't in your results.

The fix is a negative lookahead: a k or m only counts as a suffix if a letter doesn't follow it.

// k/m is a magnitude suffix only when it follows the digits AND no letter follows it,
// so "95,000 Medical" keeps its M.
const NUM_RE = String.raw`\d[\d,]*(?:\.\d+)?\s?[kKmM]?(?![A-Za-z])`;

Regression test, because I never want to relearn this:

test('regression: does not eat a following word initial M/K as a suffix ($95,000 Medical)', () => {
  const p = ok('compensation plans range from $55,000-$95,000 Medical, dental');
  assert.equal(p.salaryFound, true);
  assert.equal(p.salaryMax, 95000); // must not read "$95,000 M" as 95,000,000,000
  const q = ok('base pay $80,000 Monthly meetings and $18/hr overtime');
  assert.equal(q.salaryFound, true); // does not eat the M of "Monthly"
});

Bugs 3 and 4: ranges that collapsed to a single number

Two postings looked fine in the output. They weren't.

Salary Range: $81,000 USD - $105,000 USD    →  min 81000, max 81000
expected hiring range is $36.79/hr - $58.50/hr  →  min 36.79, max 36.79

Both reported the bottom of the range as if it were the whole range. If you filter on "pays at least $100k", the first job is wrongly excluded. Quiet, plausible, wrong.

The cause was the same in both cases: my range pattern was positional, NUMBER separator NUMBER, and both postings put something between the first number and the separator — a currency word in one, a unit in the other. The pattern stopped at the first number and treated it as a lone value.

I could have kept patching the pattern with more optional groups. Instead I stopped matching positions and started counting numbers:

// Take at most two numbers out of raw. A positional range regex breaks as soon as a unit
// or currency word sits between them ("$X/hr - $Y/hr", "$X USD - $Y"), so extract the
// numbers instead and let the count decide.
const nums = [...raw.matchAll(/(\d[\d,]*(?:\.\d+)?)\s?([kKmM])?(?![A-Za-z])/g)]
  .map((m) => toNumber(m[1], m[2])).filter((v) => v != null);

Two numbers inside one money token means a range. One means a single value. It stopped mattering what was sitting between them.

That refactor immediately exposed bug 4: C$90,000 - C$110,000 also collapsed, because a leftover separator check didn't recognise C$ after the dash. Deleting the check fixed it — the tokenizer already guaranteed what the check was testing for.

Bug 5: the insurance benefit that looked like a director's salary

An electrician job came back at $200,000 a year.

Electricians in that market make about $80k. The parser had found this:

Annual maximum out of pocket $3,000 per individual / $7,500 per family
... Health Insurance (In-network) $1,000 individual deductible
... Life Insurance plan with coverage up to $200,000

The word Annual was doing the damage. My period detector saw it, concluded "yearly", and the plausibility gate happily accepted $200,000 as a yearly wage. It is a plausible salary. It is not this job's salary — it's an insurance payout ceiling.

The fix was another directional rule, this time for benefits vocabulary:

// Benefits and insurance words disqualify a number only when they sit on the LEFT,
// the label side. "out of pocket $200,000" -> drop it.
// "salary $120,000 plus health insurance" -> the label is salary, so keep it.
if (BENEFIT_NEAR.test(left)) return null;

I wrote it as a both-sides check first and caught myself: base salary $120,000 plus health insurance and 401k match is an extremely normal sentence, and a both-sides rule throws that real salary away. Left-side only keeps it.

Bug 6: "40 hours per week" is not a pay period

A dental hygienist posting listed both a full-time salary and a part-time rate:

Full time hours-40 hours per week salary-$60,428.52.
Part-time-work up to 29 hours per week, $45/hour.

The parser returned the part-time hourly rate and ignored the full-time salary entirely.

Why: $60,428.52 had "hours per week" sitting in its left context. My period detector read per week, called it a weekly wage, multiplied by 52, got $3.1M, and the plausibility gate threw it out. The part-time rate survived, so the output looked complete.

Schedule language and pay-period language use the same words. So I strip schedule phrases before reading the period:

// A work schedule ("40 hours per week", "5 days a week") is not a pay period, so remove it
// from the context first. Leaving it in makes the $60,428 in "hours per week salary-$60,428"
// look like a weekly wage.
context = String(context).replace(/\b(hours?|hrs?|days?|shifts?)\s*(?:\/|per\s|a\s|an\s)\s*(?:week|day|month|hour|year)/gi, ' ');

With that, the full-time salary came back at $60,429/year — and Base pay $800 per week, a genuine weekly wage, still parses as weekly. Both are in the test suite.

What I got wrong about being wrong

I want to be honest about a seventh thing, because it wasn't a parser bug — it was me.

Separately from the salary work, I was making my Actors usable as tools for AI agents through Apify's MCP server. I compared my input schemas against what the MCP tools/list response exposed, and noticed that required didn't appear in the tool definitions. Three Actors, same result. I concluded the agent never learns which fields are mandatory, edited six Actors to prepend "Required." to those field descriptions, and deployed all six.

Then I looked at the description text an agent actually receives:

**REQUIRED** LinkedIn job URLs (https://www.linkedin.com/jobs/view/...)

Apify's MCP server already injects **REQUIRED** into the description of required fields. The information was reaching the agent the whole time. My "fix" produced **REQUIRED** Required. LinkedIn job URLs... on three live Actors before I caught it. I reverted all six and redeployed.

The mistake wasn't the edit. It was that I checked the field I expected to carry the information instead of the payload the consumer actually receives. Which is the same mistake as bug 5, one level up: I trusted a signal (Annual, required) without checking what it meant in context.

What the parser refuses to do

The finished version outputs a number only when it can defend it:

Normalize first, judge second. Everything becomes a yearly figure before the plausibility gate. Hourly × 2080, monthly × 12. $800B market never reaches the output because $800B/year is not a wage.
A number needs its own label. A bare $600,000 in a sentence about project value is not pay. Something must mark it: a unit (per hour, /yr), a pay word before it (salary, pay range, compensation), or a period word in context.
Direction matters. Labels sit to the left. Words to the right belong to the next number.
When it can't tell, it says so. salaryFound: false and no invented figure.

And every run reports the fill rate, because the honest answer to "why did I only get 74 jobs" is usually "because 6 in 10 US postings state pay, and the rest didn't write a number down":

examined: 12   withSalary: 8 (67%)   returned: 7
droppedBelowMin: 1   droppedNoSalary: 4
annualSalaryMedian: 250000   range: 135000-350000

That 67% matched my earlier measurement of 63% across a different sample, which was the first thing that made me trust the pipeline.

If you're building something similar

Three things I'd do from the start next time:

Build the corpus before the parser. I spent an afternoon collecting money phrases with their surrounding context from eight different job types. Nurses and retail workers are quoted hourly, engineers yearly, accountants monthly, and each format breaks a different assumption. If I had only tested on software jobs I would have shipped four of these six bugs.

Print what your regex actually matched. Bug 2 was undiagnosable from the output — the field was just empty. One line dumping raw=|$55,000-$95,000 M| made it obvious in seconds.

Write the regression test in the same commit as the fix. All six are in the suite. When I later replaced the range logic entirely, those tests caught a fifth bug I'd introduced in the process, before it shipped.

The parser and every regression test in this post are at northbell-dev/honest-scraping — no dependencies, npm test runs the 40 tests with nothing to install.

The Actor is LinkedIn Jobs Salary Data. It reads only public pages — no login, no cookies, and the request headers are a frozen object that cannot carry one.

northbell builds honest web scrapers on Apify. If a derived number has an error margin, the margin ships with it.