We run DeUnemployment, a free job search over 6,811,344 indexed
postings, 3,588,091 of them with full descriptions. It runs on one 4-vCPU VPS with 8 GB of RAM.
That is not where this kind of thing usually gets built, and the constraint decided most of what
follows.
The first assumption was wrong
We assumed the database would be the bottleneck. It wasn't. Profiling a single query put the CPU at
246% — the embedding model and the dot products were the entire cost. Storage and SQL were nowhere
near the top of the profile.
The fixes were unglamorous. A warm cache. Keeping garbage collection off the response path.
Separating workers so they stop competing. A warm query now returns in 0.0s; a cold one takes
7–12.5s.
The more interesting part is what we did not build. We priced a real inverted index: 84 MB of extra
state, two to three days of build, and 0.1–0.3s saved per warm query. On a large machine that trade
might be obvious. On ours the memory and the operational surface were not worth a third of a second,
so we skipped it and spent the effort on the cache instead. If your index is the bottleneck, build
the index. Ours was not.
"Still listed" is not "still open"
We measured listing age, not liveness. Two numbers from the September snapshot are worth keeping
apart: 29.8% of postings carried a source-reported publish date older than 90 days, and 50.79% of
those same postings were seen again at the source within the previous week.
Older is not automatically dead. A hard-to-fill requisition, a pipelined role, a slow approval chain
and a listing somebody forgot to close look identical from the outside. We can measure age. We cannot
measure intent, so we label it "still listed" rather than "open", and we publish the definitions next
to the numbers (methodology).
What pay does to shelf life
Across 6,564,432 sampled postings, the 812,943 that state any pay figure have a median listing age of
22 days, against 27 days for the 5,751,489 that don't, and a lower stale-90 share (25.7% vs 28.27%).
Only 12.4% of postings state pay at all. This is a correlation measured on a minority of the corpus,
not a causal claim, and we say so where we publish it.
The crawlers found us before the users did
The index went public and the AI crawlers arrived the same day. Not a trickle. ClaudeBot alone has
105,670 requests on disk, 4,907 of them today; Googlebot 10,409; Applebot 5,181. Eleven distinct
crawlers in total.
On a 4-core box this is not a traffic story, it is a resources story: a crawler walking every /job/
URL holds the CPU that a person's search needs.
The rule is one line long, and we have not made an exception to it: an indexer can wait, an assistant
answering a person cannot. Throttling is therefore per crawler (not per request), applied only to bulk
indexers and only on /job/ pages, as a token bucket with a burst of 10 so a genuine re-visit is never
shed:
| crawler | allowed |
|---|---|
| ClaudeBot | 0.2 req/s (12/min) |
| CCBot | 0.3 req/s |
| Amazonbot, Meta | 0.5 req/s |
| Bytespider | 0.1 req/s |
Nothing is ever blocked. A shed request gets 429 with a Retry-After and the crawler comes back
later. Googlebot, Bingbot, Applebot and every question-answering agent — OAI-SearchBot, PerplexityBot,
ChatGPT-User, Claude-SearchBot, Gemini — are exempt by name, because slowing them down is slowing our
own discovery. Claude-SearchBot spent one day inside the throttled table by mistake, which would have
put a 429 in front of the one Anthropic agent that sends us readers; that is what the exemption list is
for. ClaudeBot also gets Crawl-delay: 3 in robots.txt, and behind all of it there is a load-shed:
when more than 6 cold SEO renders are in flight at once, bulk crawlers are answered with 503 instead
of queueing in front of a person's search.
The numbers moved more than we expected. Pages served to crawlers went from 644/min to 49/min,
process memory from 3,885 MB to 765 MB, 69 crawler requests shed in total — and no 429 for anyone
actually looking for a job.
Five ways into the same corpus
The web search is the obvious door, but it is not the only one, and the others exist because people
kept asking for them.
- The site. Plain-language search, no account. Anonymous traffic gets a small number of cold searches per minute; cache hits are not counted against it.
- A chat agent on the site. Ask for a job in words and it searches the same index and answers with postings. Rate-limited, for the same reason as everything else here: an assistant that is polite enough to wait beats one that takes the index down.
-
The public API.
GET /api/public/search?q=…&limit=10, JSON, no key, 30 requests a minute per IP. Same index as the site. - An MCP server. Keyless, so an assistant that speaks MCP can query the corpus instead of guessing at it. Rate-limited like the rest of the public surface, because the alternative is an index that stops answering for everyone.
- A local clone. The companion repo is written for agents: you point your agent at it, answer a few questions about the job you want, and it embeds that description, pulls the nearest groups of the corpus onto your disk and builds a self-contained search page. Search then runs on your own machine, and your notes and résumé stay there. The data endpoints need an access key, so this route is gated on purpose.
There is also a browser extension for filling application forms, still in development, so I will not
link it yet.
Why the size of the corpus matters more than anything clever
The site is free, with no ads and no accounts, because the search is not the whole point. The point is
the dataset underneath it: 82 countries, monthly snapshots, published as JSON and CSV under CC BY 4.0
at deunemployment.com/report.
Our target is a corpus of 50M+ postings. Not for the search, which does not need it — for the
cross-tabs. At 6.8M rows a question like "does stated pay shorten a listing, by country and by source"
is still noisy. More rows is how that becomes answerable, and answerable is the only reason we keep
crawling. Every number we publish stays tied to its definition and its date.
What we would tell someone starting here
- Measure before you architect. The bottleneck was not where we assumed, and it took a profiler, not an opinion, to find out.
- Publish the definition next to the metric. The definition is the part that makes the number usable.
- Spend the budget where the users are. We throttle indexers, not assistants: an indexer that gets a 429 comes back tomorrow, a person asking a question does not.
You can search the corpus at deunemployment.com/jobs.
Top comments (0)