The first version of my LinkedIn company-jobs Actor looked like it worked. Point it at a company, get back every open role. The dataset had hundreds of rows, the fields were populated, the run finished green.
It was returning about three quarters of the jobs, and a different three quarters each time.
Worse, I had built change tracking on top of it — "these 12 roles opened since your last run, these 5 closed" — and that feature was manufacturing closures out of nothing. One test run reported 113 jobs closed and 117 opened, 75 seconds apart.
Nothing in the output distinguished this from a correct answer. That is the specific failure I want to write about, because "did my scrape actually finish?" turns out to be a question you can measure, not just hope about — using a method that ecologists use to count fish.
LinkedIn's public job search does not paginate
The public endpoint takes a start offset. I assumed start=0 gives you jobs 1–10, start=10 gives you 11–20, and so on.
It doesn't. It gives you ten jobs. Which ten is not stable.
I ran the same query twice and compared the returned job IDs:
- 3 pages deep: 33% of the IDs were different
- 10 pages deep: 61% of the IDs were different
So a straightforward loop — walk start from 0 upward until a page comes back short — does not collect a company's roster. It collects a sample of it. On a small company the sample happens to be everything. On a large one it isn't, and nothing tells you which case you're in.
Two things were broken by this, and only one of them was obvious.
The obvious one: change tracking. If run A samples 660 of 880 and run B samples a different 660, roughly 220 IDs are in A and not in B. Those get reported as closed. They were never closed. They were never absent — they just weren't in this draw.
The one I nearly missed: the basic listing was wrong too. Not the fancy feature — the core product. I was selling "every open role at this company" and shipping a list missing a quarter of it, with no indication that anything was missing.
Two stopping rules that don't work
My first attempt at "am I done" was the obvious heuristic: stop when four consecutive requests add nothing new.
Here's why that's not a measurement. Say the company has 700 roles and I currently hold 650. Each request returns 10 roles drawn from the pool. The chance that all ten are ones I already have:
// P(a page adds nothing) with 650 of 700 held, 10 drawn:
(650/700) ** 10 // ≈ 0.478
Nearly half. Four in a row happens about 5% of the time at that exact point — and more often as you get closer. So the rule does fire eventually. It just fires at a completeness level that depends on the fraction you already hold, which is precisely the unknown you were trying to determine. The stopping rule is circular: it tells you you're done by assuming you're nearly done.
My second attempt went the other way: collect the roster twice and only accept it if the two passes match exactly. Rigorous, and useless. Two random samples of a large pool are essentially never identical. The condition never fired, so the feature never produced a number, so I had built an elaborate way of saying "I don't know."
The answer was in the data I already had
The thing that unstuck me was reframing what those repeated draws are. They aren't retries. They are samples.
Ecologists count fish in a pond by catching some, tagging them, releasing them, then catching a second batch and seeing how many carry tags. Few tags means a big pond. That's Lincoln–Petersen:
N ≈ (size of sample A × size of sample B) / (number in both)
I already had two samples — I just had to stop merging them into one bucket. So the Actor now splits its own requests into two sets, tracks them separately, and estimates the total from their overlap:
export function estimateTotal(sampleA, sampleB) {
const a = sampleA.size, b = sampleB.size;
let overlap = 0;
for (const id of sampleA) if (sampleB.has(id)) overlap += 1;
return { total: Math.round((a * b) / overlap), a, b, overlap };
}
Then coverage is just what I hold divided by what I estimate exists, and "how many am I still missing" falls out of the same number.
Here is a real run against a company page, 800 requests, recording the unique jobs held at each point:
| requests | unique jobs |
|---|---|
| 40 | 345 |
| 80 | 617 |
| 160 | 795 |
| 240 | 844 |
| 320 | 867 |
| 400 | 878 |
| 520 | 880 |
| 520–800 | 880 (zero new in 280 requests) |
Estimated total from the overlap: 880. Actual point of exhaustion: 880.
The pool wasn't unreachable. I had been stopping at 93 requests.
Bug 1: I split the samples along the wrong axis
The first version of the split used request parity — odd-numbered requests into sample A, even into sample B. The estimate came back at 1,334 for a company with 880 roles. Fifty percent too high, and confidently so.
The reason took me longer than it should have. The offset also advances once per request. So odd requests were looking at odd pages, and even requests at even pages. The two samples were drawing from different parts of the pool, which makes their overlap unnaturally small, which inflates N. Lincoln–Petersen assumes both samples come from the same population; mine didn't.
The fix is to split by sweep — one full pass over all offsets — rather than by request:
const sweep = Math.floor(r / pages);
const start = (r % pages) * PAGE_SIZE;
...
(sweep % 2 === 0 ? evenSample : oddSample).add(j.jobId);
Now both samples cover every offset. The estimate landed on 880.
Bug 2: the formula collapses, and it collapses quietly
This one reached production before I caught it, and it produced the worst kind of output: a confident, plausible, wrong number.
The two samples were 20 and 681 items, with an overlap of 20. Run the formula:
N = 20 × 681 / 20 = 681
The estimate equals the count I already had. Coverage: 100%. The Actor concluded it had collected everything, stopped, compared against the previous run, and reported 199 jobs closed. None had closed.
When one sample is entirely contained in the other, the formula returns the larger sample. It isn't an error — it's what the math says when your "second sample" is really just a subset of the first. And an overlap of 20 comfortably passes a naive "overlap must be at least 10" check.
So the guards aren't about size, they're about shape:
const MIN_SAMPLE = 20;
const MIN_BALANCE = 0.5; // smaller ÷ larger
if (Math.min(a, b) < MIN_SAMPLE) return { total: null, reason: 'sample-too-small' };
if (Math.min(a, b) / Math.max(a, b) < MIN_BALANCE) {
return { total: null, reason: 'samples-unbalanced' };
}
if (overlap < 10) return { total: null, reason: 'overlap-too-small' };
Plus one more condition outside the function: don't trust an estimate until both samples have completed at least one full sweep. The degenerate case above happened when the run stopped after one sweep plus two requests, leaving one sample with two requests' worth of data.
Note what total: null means here. It is not zero and it is not 100%. It means I cannot tell, and it travels to the output with a reason attached.
Bug 3: I fixed the wrong variable
My first instinct on the collapse was to raise the minimum sample size from 20 to 50. Bigger samples, less degeneracy, done.
That broke companies with about 25 open roles. Too many to fit in one page (10 per request), too few to ever reach a 50-item sample. Mid-sized companies became undecidable — the Actor could neither confirm nor estimate, so it withheld everything.
The collapse was never caused by absolute size. It was caused by imbalance — 20 versus 681. Raising the floor treated a symptom and disabled a whole class of correct cases. So the absolute minimum went back down to 20, and the real protection lives in the balance ratio and the two-sweep rule.
Small companies get a separate path entirely, based on a fact rather than an estimate: if no response ever filled a full page, there is no second page, and what I have is everything. Measured page churn at one page deep is 0%, so this is safe — and it matters, because mark-and-recapture needs an overlap of 10+ to work at all, which means the smaller the company, the less able it is to estimate. Exactly backwards, if you don't handle it separately.
Bug 4: my own request cap looked like a wave of layoffs
A production run reported closed: 20. Nothing had closed. I had changed the per-company cap from 30 to 20 between runs.
Truncate a list, compare it to an untruncated one, and the rows you didn't collect appear as rows that vanished. Same for changing a filter: narrow the location, and everything outside it "closes."
So before any comparison, the Actor now checks whether the two observations are comparable at all, and refuses to produce numbers when they aren't:
if (ctx.truncated) return blank('truncated-this-run');
if (prevPoint.truncated) return blank('previous-run-was-truncated');
if (ctx.filterKey !== prevPoint.filterKey) return blank('filters-changed');
opened and closed come back as null with a reason, instead of numbers that look real.
What the Actor reports now
Every company row carries the measurement alongside the data:
openJobs: 880
estimatedTotal: 880
coverage: 1
estimatedMissing: 0
changeMarginOfError: 0
collectionComplete: true
requestsUsed: 523
changeMarginOfError is the one I'd argue hardest for. If this run missed m roles and the previous run missed m', the apparent change is muddied by up to m + m'. A run at 99% coverage on 880 roles is missing about 9 — so a reported change of ±18 is inside the noise. That is why the target coverage for change tracking defaults higher than the target for just listing jobs. A number without its error bar invites a conclusion it can't support.
One more measurement, since it costs nothing to state: LinkedIn returns HTTP 400 for start values of 1000 and above. In an 800-request experiment, exactly 20 requests failed — 1000, 1010, … 1190 — and every other request succeeded. The offsets below 1000 are enough, because each pass returns a different draw. Circling 0–990 repeatedly reaches everything.
What I'd take from this
"No new results" is not evidence of completeness. It's evidence about the fraction you already hold, which is the thing you were trying to measure. Any stopping rule built on it is circular.
Repeated requests are samples, not retries. The moment I stopped merging them into one bucket, the data I already had answered the question I thought I needed a new experiment for.
A degenerate formula returns a number, not an error. 20 × 681 / 20 = 681 is arithmetically fine and semantically garbage. If a formula has a collapse mode, guard its shape — and make the guard return "I don't know" rather than a default.
Silence and completeness are indistinguishable unless you report the difference. Before this work, "660 jobs" and "880 jobs" looked identical from the outside. Now the row says which one it is, and how sure it is.
The estimator, the guards and the tests that pin down each of these four bugs are at northbell-dev/honest-scraping — no dependencies, npm test runs the 40 tests with nothing to install.
The Actor is LinkedIn Company Jobs Scraper. It reads only public pages — no login, no cookies, and the request headers are a frozen object that cannot carry one.
northbell builds honest web scrapers on Apify. If a derived number has an error margin, the margin ships with it.
Top comments (0)