Most "search APIs" are a thin layer over Google or Bing results. NestDaddy is not. It has its own crawler, its own inverted index, its own ranking and its own news collection, and it all runs on a single server with 48 GB of RAM.
This post walks through how the pieces fit together, the problems we hit in production, and the numbers we used to decide what to fix. If you are thinking about building search yourself, or you want to know what sits behind the API you are calling, this is for you.
The shape of the system
There are four parts, and each one is allowed to fail without taking the others down:
- Crawler. Fetches pages, obeys robots.txt, extracts the readable text and stores it in PostgreSQL.
- Indexer. Turns new pages into posting lists in a RocksDB index that lives next to the live one.
- Swap. Once a day, the new index replaces the live index in one atomic step, with an automatic rollback.
- Search. A BM25 ranker reads the live index, read-only, behind the web site, the REST API and an MCP server for AI agents.
The rule that ties them together: the search path never waits for the crawler or the indexer, and neither of them can starve it of CPU or memory.
The crawler: polite, bounded, fenced
A crawler that misbehaves gets blocked, and a crawler that runs out of memory takes the machine with it. So ours is built around limits.
- robots.txt first. One fetch per site, cached for 24 hours. A 401 or 403 on robots.txt means "disallow everything".
-
Spread across hosts. Each batch claims URLs from the queue with
FOR UPDATE SKIP LOCKEDand keeps at most 3 URLs per host. A batch of 500 URLs touches about 500 different sites instead of hammering one. TheSKIP LOCKEDclaim also means several crawler processes can share one queue safely. -
Bounded downloads. Every response is streamed and cut at 5 MB or 20 seconds. We learned this one the hard way: the first version read
response.contentand only then trimmed it, so a server that streamed an endless chunked reply pushed the whole body into memory. The fixed version stops at 5.2 MB in 0.07 seconds on the same endless stream. - A fence around it. The crawler runs as a systemd service inside its own slice: pinned to 4 of the server's CPUs, capped at 1.5 cores and 3 GB of memory, with a low priority. If it misbehaves, Linux slows it down or stops it, and the search side does not notice.
That last point matters more than it sounds. On one day this month the crawler hit its memory cap and restarted 36 times. Search latency, the database and every other service on the machine were untouched. The fence turned a production incident into a log line.
The index: RocksDB and posting lists
The index is an inverted index stored in RocksDB: for every term, a posting list of document IDs, plus per-document statistics for BM25.
We picked RocksDB over a search server for three reasons. It is a library, so there is no extra service to run. It is built for fast point lookups and multi-gets, which is exactly what a query does. And it can be opened read-only by many processes while another process builds the next version.
Text goes through a tokenizer (lowercasing, stop words, Porter stemming with a memoised stemmer) before it reaches the index, and the same tokenizer runs on every query.
Replacing the index without breaking it
Our first pipeline copied index directories while they were open. That produced two corrupted index files before we threw it away. The current swap is boring on purpose:
- Take the writer lock so the indexer pauses.
- Hard-link the immutable SST files into a staging directory and copy the small metadata files (MANIFEST, CURRENT, OPTIONS, WAL). This takes about a second and uses no extra disk.
- Release the lock and verify the staging copy read-only: the document counter must not go down, sample terms must resolve, the newest indexed pages must be readable.
- Rename the live directory to a "previous" directory and the staging directory to live.
- Reload the search workers and run a live search check. If anything fails, roll back automatically.
Two previous generations are kept, so going back is a rename, not a restore.
Ranking: BM25, measured
The ranker is BM25 with a few practical additions: a two-stage pre-score (rare terms first, then the top 3,000 candidates re-ranked by term frequency with one RocksDB multi-get), a domain match signal for navigational queries, and image documents kept out of web results.
We do not change ranking by feel. A small evaluation set of 40 real queries, each with a judgment ("the right answer is this URL pattern" or "the homepage of this site"), runs against the live index after every change. The fixes we shipped in November moved:
| Metric | Before | After |
|---|---|---|
| Right answer at #1 (P@1) | 45% | 70% |
| Right answer in the top 3 | 60% | 77.5% |
| Right answer in the top 10 | 70% | 87.5% |
| Mean reciprocal rank | 0.536 | 0.754 |
The changes behind that were unglamorous: true document frequencies for IDF instead of estimates, an average document length measured from real documents, deterministic sampling instead of random, and keeping images out of web search.
Latency: the bugs were not where we expected
When search got slow, the ranker was rarely the problem. Profiling live workers found three surprises.
A regular expression. The tokenizer stripped email addresses with \S+@\S+. On some inputs that pattern backtracks quadratically, and one query held the Python GIL for over a minute. The fix was a pattern with identical output and linear time: (?<!\S)(?=\S+@\S)\S+.
The index falling out of RAM. A cold read of the index from disk turned a 0.2 second query into several seconds. We added a small service that checks which parts of the index file are resident (mincore) every ten minutes and reads back only what is missing, and we gave it a cgroup memory guarantee so the kernel prefers to evict other files first. One detail cost us an afternoon: cgroup v2 memory protection is hierarchical. Setting MemoryMin on the service did nothing until the parent slice had the same guarantee.
A forgotten cron job. Even with that protection, the whole index still vanished from RAM once an hour. The cause was a one-line script in /etc/cron.hourly that ran echo 1 > /proc/sys/vm/drop_caches, an old "free up memory" trick. Dropping caches ignores cgroup protection entirely. Removing it brought the median query time on our evaluation set from 0.59 seconds back to 0.22 seconds.
The lesson we keep relearning: measure on the real machine, and read every scheduled job before you trust a graph.
What we have not solved yet
Being independent means owning the weak spots too. These are the ones we are working on now:
- Long, specific queries. The current index weighs titles, descriptions and headings. Pages whose answer lives deep in the body text can rank below a page that merely shares one word with the query. The next index version adds body text, which also means the index outgrows what we can keep in RAM on one machine, so it needs a tiered design first.
- Numbers and non-Latin scripts. The current tokenizer drops most numbers ("windows 11", "iphone 17") and handles Chinese, Japanese and Thai poorly. A new tokenizer that keeps short numbers and splits CJK and Thai text is written and waiting for the next rebuild.
- Freshness. Most of the index was crawled months ago. A targeted recrawl of the most searched results and the most linked sites is running now, and the crawler is next in line to run as a continuous pipeline with several processes.
Why build it at all
Because an index you own is an index nobody can switch off. Microsoft retired the Bing Search APIs in August 2025 and many apps lost their search overnight. Products that need raw results, news by country, or readable page text for AI agents need a source that answers in plain JSON and does not depend on someone else's roadmap.
That is what NestDaddy is for. You can try the search, news and page extraction APIs with a free 7-day key at nestdaddy.com/apis, or connect it to Claude, Cursor or any MCP client through nestdaddy.com/mcp.
If you have questions about any part of this, or you have built something similar and want to compare notes, we would like to hear from you.
Originally published on the NestDaddy blog. NestDaddy is an independent search engine with its own index: nestdaddy.com.
Top comments (0)