A job postings API over 18,000 company career sites, measured: one night's crawl, its cost and one search

Most companies publish their open roles through an applicant tracking system (ATS), and the ATS serves each company's job board through a public API or feed so that the company's own careers page can show it. We read about 18,000 of those boards on eight systems, seven of the systems every night and Workday every second night, and keep the postings in one index that a single API call searches. This is the index as it stood at 16:26 UTC on 7 November 2026, what one night of reading it took, and one search run against it from start to finish. Every figure comes from the runs' own records; times and dates are UTC.

Where the boards come from

No ATS vendor publishes its customer list. Common Crawl's URL index is the next best thing: it records every page Common Crawl's crawler captured, keyed in SURT form (the host reversed, then the path), so the boards of one system sit together in one sorted run of keys. Greenhouse boards are under io,greenhouse,boards)/ and io,greenhouse,job-boards)/, Ashby's under com,ashbyhq,jobs)/, Personio's under de,personio,jobs, and Workday's under com,myworkdayjobs,. A binary search of each crawl's cluster.idx, by HTTP range requests, finds the compressed blocks that hold a prefix; only those blocks are fetched and unzipped, and a board's name is the first path segment or the subdomain of each URL in them.

Four crawls (CC-MAIN-2026-30, -34, -39 and -43), keeping the boards whose captures had answered 200, gave 20,784 boards on eight systems. Asked on the nights of 6 and 7 November:

System	Boards asked	Read or unchanged	Gone	Kept out by robots.txt	Postings in the index
Workday	5,036	4,752	76	0	992,634
Greenhouse	5,116	4,835	281	0	153,025
Ashby	4,140	3,441	684	0	60,641
Personio	3,775	2,600	968	207	21,706
Recruitee	1,217	1,205	3	1	19,229
JazzHR	1,283	1,065	204	0	18,583
Pinpoint	136	122	0	12	4,872
Lever	81	79	2	0	3,436

Workday's 5,036 career sites are what is left of 6,342 addresses once the same site spelt with different capitals is counted once. The rest of the boards asked failed after retries, were refused, or were names that are not boards. Lever's 81 says more about Common Crawl than about Lever: Lever's main host refuses Common Crawl's crawler, and 69 of the 81 boards are on Lever's EU instance.

Two things in that table are worth knowing before you build your own. About one board in nine that Common Crawl had seen answering in recent months was gone when we asked: a quarter of Personio's, one in six of Ashby's. And postings are not spread like boards. A Workday career site averages about 190 postings (880,653 from 4,630 sites in its last whole read), a Greenhouse board about 32, so Workday is a quarter of the boards and 78% of the postings.

One night, and its manners

The seven systems other than Workday are read in full every night. On the night of 7 November that was 15,611 requests in 1 hour 29 minutes: 99.3% answered at the first attempt, 188 retries, no HTTP 429, and 253,505 postings read. The rules the crawl keeps:

• It says who it is. Every request carries Mozilla/5.0 (compatible; AdderleyData-index/0.1; +https://adderleydata.com).
• robots.txt first. Each host's robots.txt is read before its boards are asked: 6,413 hosts that night, because every Personio, Recruitee, JazzHR and Pinpoint customer has a subdomain of its own. 220 boards whose robots.txt disallows the path were not asked, 207 of them on Personio. A host's Crawl-delay is kept where it is longer than our own spacing; Lever's asks for one second.
• Limits per host, not per crawl. At most four requests in flight and 90 a minute to any one host, and 32 in flight across the whole crawl. Greenhouse serves all 5,116 boards from one API host, so at 90 a minute its boards alone take at least 57 minutes.
• A refusal is an answer. A 401, 403 or 429 is never retried from another address. The board is counted as refused or limited, and a 429 halves that host's rate for the rest of the run.
• Ask only for what changed. Ashby's API sends an ETag. Each board is asked with the last one in If-None-Match, and 2,377 Ashby boards answered 304 Not Modified, so their postings were carried over without a body.
• Public endpoints only. No login, no key and no cookies: only the API or feed each ATS serves for its customers' careers pages.

Workday is read differently

Workday's job board API returns 20 postings a page and stops at 2,000 per query, so reading every posting takes a lot of requests: the last whole read, at the end of October, took 80,322 requests and 6 hours 18 minutes, and found 880,653 postings on 4,630 sites. So Workday is read newest first, every second night: each site's list from the top, stopping at the first posting already held. On the night of 6 November that took 5,153 requests and 15 minutes, with 99.5% answered at the first attempt; it read 4,752 sites and found 13,900 postings newer than any it held.

What that gives up is in the last section.

What a night costs

The crawl runs on Apify like any other Actor, at our cost. The seven systems take about two hours on 2 GB of memory: US$0.89, US$1.14 and US$1.16 on the three runs we have costed. Workday's newest-first read costs with the gap it covers: US$0.12 for 17 hours of new postings, US$1.27 for two and a half days. A whole Workday read was US$5.36, which is why it is not done every night: that alone would be about US$160 a month. At these figures a month of nights comes to about US$27–35 for the seven systems and US$8–15 for Workday's new postings. That is an estimate from six runs, not a bill.

What the index holds

789 shards of gzipped NDJSON, by system and by the country each board's list gives (large ones in parts), and a manifest that lists every shard with its row count and SHA-256: 1,274,126 postings, 136 MB gzipped, 1.45 GB unzipped. A row is the posting as its board's list gives it, in one schema: title, company, location, department, work arrangement, employment type, dates and the posting's own link. No posting text, and no person's name or contact details.

The country is where most of the work is. Two in three postings name no country in their board's list:

Country in the board's list	Postings
None	855,699
United States	234,645
United Kingdom	20,670
India	17,696
Canada	17,023
Germany	14,710
Netherlands	11,164
Singapore	9,855
114 other countries	92,664

Workday accounts for most of the first line: 778,470 of its 992,634 postings name no country. Its lists give a city and a US state code ("Austin, TX"), a count ("3 Locations"), a campus or a hospital's name, and keep the country for each posting's own record. So a search for a country reads that country's shards and the ones with no country, and matches state codes, provinces and large cities by name.

One search, from start to finish

A search is an Actor run with no companies named. Through Apify's API:

curl -X POST "https://api.apify.com/v2/acts/adderleydata~career-site-jobs-scraper/run-sync-get-dataset-items?format=json" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keywords":["product manager"],"locations":["Germany"],"sources":["workday"],"includeDescription":true,"maxResults":10}'

Run at 20:00 UTC on 7 November, on that night's index:

1. The run read the manifest and chose 33 shards: Workday's postings placed in Germany, and Workday's postings with no country.
2. It read those 783,408 postings, 80.6 MB, and matched 30: titles that hold both "product" and "manager" as whole words, at a place in Germany.
3. Newest first by posted date, it asked Workday for each match's own record, which holds the full location, the posted date and the text that the list leaves out.
4. Workday refused 5 of the 15 records it was asked for, with HTTP 403. The run treats a refused record as it treats a closed one: left out, not charged, and the next match takes its place. Were ten or more, and over half, of a run's records refused, it would stop and report a block instead.
5. 10 postings were saved, each placed in Germany by its own record. Two email addresses in their text were replaced before saving. 21 seconds, 15 requests, none retried.

A saved row, trimmed (the company is made up):

{
  "schemaVersion": "job.v1",
  "id": "workday:global:acme/Product-Manager_R-01234",
  "source": "workday",
  "url": "https://acme.wd5.myworkdayjobs.com/External/job/Berlin-Germany/Product-Manager_R-01234",
  "title": "Product Manager",
  "company": { "name": "Acme GmbH", "sourceCompanyId": "acme", "url": "https://acme.wd5.myworkdayjobs.com/External" },
  "location": { "raw": "Berlin, Germany", "city": "Berlin", "region": null, "country": "DE" },
  "workArrangement": "hybrid",
  "employmentTypes": ["full_time"],
  "postedAt": "2026-11-04T00:00:00.000Z",
  "expiresAt": "2026-12-31T00:00:00.000Z",
  "description": { "text": "...", "contactsRedacted": true }
}

With "incremental": true and the same input on a daily schedule, each run returns only the postings that are new or changed since the last one, and unchanged postings are not charged: a job postings feed for the titles and places you choose.

Where it stops

• Closed Workday postings. A newest-first read adds postings but cannot see one close, so between whole reads the index keeps Workday postings that have since come down: Workday's part grew from 880,653 to 992,634 in the week after its last whole read. A search reads each Workday posting's own record before saving it, which is where the five above were caught.
• Text for four systems. With includeDescription, a search reads the text of matched Greenhouse, Workday, Pinpoint and JazzHR postings. Lever, Ashby, Recruitee and Personio postings come without it for now.
• Places are the board's words. A place that is not a country matches as whole words, so "London" also finds London, Ontario. Filter on location.country where that matters.
• No EXPIRED rows from a search. A board that fails on a given night drops out of the index for that night, so a posting missing from the index has not been proven closed. A run that names the company's board reports closed postings; a search reports new and changed ones only.
• Only what Common Crawl has seen, on eight systems. A company on another ATS, or one whose board Common Crawl never captured, is not in the index.

The search above is Career Site Jobs Scraper on the Apify Store with Companies left empty, priced per posting saved; with companies named, it reads their boards directly, on fifteen systems. The reference page has the index's counts by system and country and the measured runs behind them.

Disclosure: Adderley Data, who wrote this, builds and sells that Actor and runs the crawl; neither link carries a referral or affiliate code. We are not affiliated with Workday, Greenhouse, Ashby, Personio, Recruitee, JazzHR, Pinpoint, Lever or Common Crawl, and their names are used only to say which job boards and which index this describes.