How to know which job postings were removed, when nothing tells you

Every ATS job board API answers the same question: what is open right now.
Greenhouse, Ashby, Workable, Workday, all of them hand you the current list.

None of them tell you what disappeared. There is no closed_at, no
deletions feed, no tombstone. A role that was filled yesterday is simply not
in today's response, indistinguishable from a role that never existed.

And disappearance is usually the more interesting half. A company that opened
40 roles and closed 38 is not the same company as one that opened 40 and
closed none, and the open-roles endpoint reports both as 40.

The mechanic is a diff, and it is not hard

You cannot ask for closures. You have to have been watching.

CREATE TABLE seen (
    platform TEXT, boardToken TEXT, jobId TEXT,
    firstSeen TEXT,          -- first run this job appeared in
    lastSeen  TEXT,          -- most recent run it appeared in
    closedOn  TEXT,          -- set when it stops appearing
    PRIMARY KEY (platform, boardToken, jobId)
);

Each run: upsert everything you fetched with today's date in lastSeen, then
mark the rest closed.

UPDATE seen SET closedOn = :today
 WHERE closedOn IS NULL
   AND lastSeen < :today
   AND boardToken IN (SELECT boardToken FROM fetched_ok_today);

That last line is the one people leave out, and it is the difference between
a closure feed and a random number generator.

The three ways this goes wrong

A failed fetch is not a closure. If a board 500s, times out, or you get
rate limited, every job on it vanishes from your response. Diff naively and
you have just recorded 2,000 closures at one company on the day their CDN had
a bad afternoon. Only diff boards that answered successfully this run.
Everything else keeps yesterday's state untouched.

A paginated board that stopped early is the same bug wearing a hat. If you
read 3 of 7 pages and treat the result as the complete current state, pages
4 to 7 all "closed". Track per-board fetch completeness, not just HTTP 200.

You cannot know how long a job was open if you started watching after it
was posted. This is the subtle one. A job you first saw on the day you
started watching may have been open for a week or a year. Computing
daysOpen = closedOn - firstSeen on that row invents a number.

The fix is to record it as unknown rather than compute it anyway:

board_first_seen = min_first_seen_for(platform, board_token)
knowable = job.first_seen > board_first_seen   # strictly after, not equal
days_open = (closed_on - job.first_seen).days if knowable else None

"Open for 3 days" and "we do not know how long it was open" must never look
the same, because one of them is a signal about the company and the other is
an artefact of when you happened to start.

The same discipline applies upward: a company's closed30 should be absent
until you have watched it for 30 days, not 0. "Closed nothing" and "we have
not been looking long enough" send a reader to two different conclusions.

What it gets you once it runs

Closures turn a job board into a time series. Net change per company. Roles
that close in four days, which usually means an internal candidate. Roles that
sit open for six months, which usually means the requisition is a placeholder.
A company whose closures suddenly outrun its openings, which is a churn signal
worth a phone call if they are your customer.

None of that is visible in the endpoint everyone reads.

If you would rather not run it for a month first

That is the real cost here: this data does not exist until you have collected
it, and the first month produces nothing.

I have been running the diff across 24,280 company boards on 10 ATS platforms,
and the index currently holds 688,711 open roles and 29,027 recorded
closures, rebuilt nightly. It is on Apify as
ATS Jobs Search API, with
company-level opening and closing counts in
B2B Leads Finder.

daysOpen is null wherever it is not knowable, for the reason above. If you
find a row where that rule looks wrong, tell me, because that is the exact
mistake this is trying not to make.

Comments

[Mads Hansen]

The completeness guard is the crucial part. One extra state helps with eventually consistent upstreams: active → suspected_missing → closed, where closure requires two complete observations (or a grace interval). Store missing_first_observed_at separately from closure_confirmed_at; the true removal time is interval-censored between lastSeen and that first complete miss, not necessarily “today.” If the same stable key reappears, reopen/merge it instead of creating a new job. That keeps one clean-but-transient snapshot from becoming a false closure and makes downstream duration metrics much more honest.