I run Job Opportunities API (JOA), a jobs API that only reads employers' own career sites and applicant-tracking systems. The hard part is not finding jobs. It is knowing when a job is gone.
Why closing jobs matters
A job board full of dead listings wastes everyone's time. So every job we serve gets re-checked at its source, and when the employer removes it, it leaves the live list and moves to a separate closed list with a date and a reason (for example expired_upstream). Our job pages for closed roles answer HTTP 410, not a soft "sorry".
The rule we thought was enough
Our checker is polite per host: one request per second, back off on 429, respect robots.txt, an honest User-Agent. Per host.
What actually happened
Last night our own data-quality review noticed a spike of "slow down" (429) answers. 90% came from one vendor. Many small employers use an applicant-tracking system that gives each company its own subdomain. We had 17,651 of those subdomains, and every one was its own "host" to our pacer. One request per second each looked polite. But they all sit behind one edge, so the vendor saw 3 to 8 requests per second from us. Polite per host, rude per server.
The fix
We now group hosts into lanes by the server that actually answers, not by the hostname. Shared-edge vendors get one lane, one budget, one penalty box. For this vendor that means one request per second in total, and a full re-check takes about 11.6 hours, still inside our 18-hour target.
The second bug it exposed
A "blocked" answer was counted as a strike against the job, the same as "gone". Blocked never closes a job on its own, but enough blocked strikes plus one real "gone" could have closed a live job. We reset those strikes and are giving "gone" its own counter. Nothing was closed by mistake, but it was closer than I like.
Lessons, if you crawl anything at scale
- Rate-limit by the server you hit, not the name you typed.
- A 429 is information about the whole vendor, not one page.
- "We couldn't check" and "it's gone" are different answers. Store them differently.
If you want to see how a job's status is exposed, /v1/jobs?status=closed returns the closed list with closed_at and closed_reason. Free key, no card: https://jobopportunitiesapi.org
What's the worst rate-limit surprise you've hit?
Top comments (0)