174,000 open jobs from four public ATS APIs: the board list, the bill, and why storage cost more than the crawl

Update, 30 October 2026: the index has grown since this first run. The refresh of 30 October holds 469,423 open jobs from 9,345 company career sites, and SmartRecruiters is now the fifth system. The numbers below are from the first full run on 22 October. You can search the current index with Career Jobs Search.

Greenhouse, Ashby, Lever and Personio each publish every company's open jobs through a public, documented endpoint that needs no key. Reading one company is easy. The question people actually ask is a different one: "which companies are hiring a data engineer in Berlin this week?" That needs every board read in advance and kept fresh, so a search takes seconds instead of a crawl.

We built that index this month. These are the measured numbers from the first full run on 22 October 2026, and the three things that cost more than expected.

The four endpoints

Greenhouse  https://boards-api.greenhouse.io/v1/boards/<slug>/jobs
Ashby       https://api.ashbyhq.com/posting-api/job-board/<slug>
Lever       https://api.lever.co/v0/postings/<slug>?mode=json
Personio    https://<slug>.jobs.personio.de/xml

One request per company returns the whole board on all four; none of them pages. Lever's robots.txt asks for Crawl-delay: 1, so Lever boards are read one per second. Greenhouse's listing call without ?content=true leaves out the job text, which is what you want for an index: the text is the heavy part, and it is only needed for the jobs a search returns.

Where 5,637 board names came from

None of these vendors publishes a list of its customers. The slug is the hard part.

Most of them came from Common Crawl's public URL index (crawl CC-MAIN-2026-39): we looked up the captured URLs on each vendor's job-board hosts and kept the company part of each URL, the first path segment (or the subdomain, for Personio). The rest came from a board index we had already verified for an earlier tool. That gives slugs, not jobs. Common Crawl is used only as a list of hints; every job is read from the vendor's own endpoint above, and we never read the archived page content.

System	Boards found	Boards with open jobs	Jobs
Greenhouse	3,236	2,405	117,782
Ashby	1,732	1,488	36,417
Personio	544	349	3,817
Lever	125	117	15,946
Total	5,637	4,359	173,962

About one board in four (1,278) answered with nothing: a company that moved to another system, a board with no openings today, or a slug that only ever existed in a URL somebody typed. They cost one request each and stay on the list, because a board that is empty in October can be hiring in November.

Lever looks small because the URL index gave us only 125 Lever slugs, and that is a limit of this discovery method, not a measure of Lever. Those that were found were nearly all live (117 of 125), and they are big boards: 136 open jobs each on average.

The bill: the crawl was cheap, the storage was not

The first full run read all 5,637 boards in 29 minutes with no failed requests, and cost $1.30. Compute was $0.10 of it. The other $1.19 was storage: $0.87 of dataset writes, $0.23 of request-queue writes and $0.09 of reads. Nine tenths of the bill went to the platform storing 174,000 rows and 5,637 queue entries that the run threw away a minute later, once the index was built.

The fix was to keep both in memory for the crawl and write only the finished index:

import { Configuration } from 'crawlee';
import { MemoryStorage } from '@crawlee/memory-storage';

// queue and rows live in RAM; only the index below is written to the platform
Configuration.getGlobalConfig().useStorageClient(
    new MemoryStorage({ persistStorage: false, writeMetadata: false }),
);

and to open the one store that must survive with forceCloud, so it still lands on the platform:

const store = await Actor.openKeyValueStore('dododata-career-index', { forceCloud: Actor.isAtHome() });

The trade: a run that crashes halfway keeps nothing, so it starts again from the top. For a 29-minute read of public endpoints that costs about $0.10 of compute, that is a better deal than paying twelve times as much to store rows nobody reads.

Eight gzip shards instead of a dataset

A search over the index needs every row, every time, so a paged dataset is the wrong shape for it. The index is eight gzipped JSON arrays in a key-value store, sharded by a stable checksum of the job id:

const hash = (s: string) => {
    let h = 0;
    for (let i = 0; i < s.length; i++) h = (h * 31 + s.charCodeAt(i)) % 1_000_000_007;
    return h;
};
shards[hash(id) % 8].push(row);

Each shard is about 1.46 MB gzipped and about 12 MB unpacked, so the whole index is 11.7 MB to download. A search downloads all eight in parallel and filters in memory; the index part of a search takes a few seconds; fetching the full text of the rows returned, at five requests a second, is most of a run.

Three smaller files ride along:

• SEEN, a map of job id to the date we first saw it. It is read at the start of every refresh and carried forward, so first_seen_at means "first time this job appeared", not "first time this run saw it". It is a date we set ourselves, the same way on all four systems, next to whatever date the board publishes.
• CLOSED, the ids that were listed last time and are gone now, with the date. Ids only: a removed posting's text is not kept.
• MANIFEST, row counts per system and the refresh time, so a reader can check freshness with one small request.

One guard that is easy to forget: a refresh that reads zero rows (a network fault, a changed endpoint) must not overwrite a good index with an empty one. The writer refuses to write when the row count is zero.

Reading it from somewhere else

The store is set to "anyone with the id can read", which is what lets a search run in another account read it without our token. It holds listing fields only: title, company, locations, remote flag, pay as numbers where the board publishes them, dates and the employer's own apply link. No job text and no personal data.

The text is fetched live, per job, only for the rows a search returns, from the same public endpoints. A measured search on 22 October for "data engineer" or "analytics engineer" matched 1,061 jobs. Returning the newest 100 with their full text took 27 seconds (the text is fetched at five requests a second) and covered 78 companies. All 100 came with full text, and 51 had pay: 27 from the board's own salary field and 24 read from the job text.

What we would tell you before you build one

1. Discovery is a list of hints, not a list of customers. One in four slugs is dead. Keep them anyway and re-read them.
2. On Apify, count storage operations before compute. Our crawl was mostly waiting on the network; writing rows nobody reads cost twelve times as much as reading the boards.
3. Keep first-seen dates yourself. Each board's own date means something slightly different; a date you record when the job first appears means the same thing everywhere.
4. Link back. Every row carries the employer's own posting URL, and a job the employer removes leaves the index at the next refresh.

Since that first run, a wider pass over Common Crawl found 4,236 more boards with open jobs; on 26 October the index held 271,718 jobs from 8,486 boards. On 29 October SmartRecruiters joined as a fifth system, read through its public Posting API: 198,046 jobs from 915 companies, which took the index to 469,585 jobs from 9,346 boards. It refreshes daily. If you want to search it rather than build your own, it is the Career Site Jobs Search Actor on Apify: keywords, locations, companies, time range, remote and pay filters, from $3.00 per 1,000 jobs returned ($4.00 on Apify's free plan).