AI search runs on indexes. Somebody has to build them.
Search is changing hands. Per an SE Ranking study, AI search traffic grew 16 times from 2024 to 2026, and now accounts for about 1 in every 312 website visits. Google's share of search fell below 90 percent in late 2024, for the first time since 2015, per StatCounter. In early 2024, Gartner predicted traditional search engine volume would fall 25 percent by 2026. The ten blue links are giving way to a generated answer with a handful of citations attached.
Here is the part that gets less attention. Every answer engine runs on an index, and a handful of companies own the indexes the answer engines draw from. If a page is not in the index, it does not exist in the answer. No amount of content tweaking fixes that. Indexing is the bottleneck now, not writing.
The numbers back that up. Only 14.4 percent of ChatGPT responses contained a fan-out query (a web search ChatGPT runs to find sources), per an Ahrefs study of 66.6 million free, logged-out responses from August and September 2026. The funnel narrows at the top. Being findable is getting harder even as more of the web gets read.
Meanwhile the crawlers doing the reading are most of the web. Automated traffic already drives more than half of all web traffic: bots at 53 percent in 2025, up from 51 percent the year before, per the Imperva 2026 Bad Bot Report. That is not a forecast. That is the report describing the web as it is today.
What that looks like from inside the index
We run cl0q.com, a search engine that maps the open web. We have 56.4 million domains mapped and 36.8 million sites crawled so far (cl0q /api/stats, generated 2026-11-16 00:35Z, sampled 2 percent, approximate), working toward 100 million. We live in the layer the trend depends on, and this week gave us a close look at what it costs to keep it running.
On November 11 we served 3,189 busy 503 errors. On November 13 it was 7,447, on November 14 it was 3,394, and on November 15 it was 1,024. Visitors hit a wall while the servers were fine, which is the worst kind of outage to explain. We traced it to one operator, a scraper fleet running out of a single cloud provider, hammering database-heavy pages. The fix was a limit, not a block. That fleet's requests to heavy pages now get through only a fraction of the time, and the rest get a 429 with a retry time. Most of November 15's came from a database stall at 02:45Z, our own cache rebuild at 03:56Z, and a database host reboot at 22:34Z.
The same week we put live a tighter limit on database-heavy pages and a disk cache. The disk cache now holds about 170,000 pages under a 3 GB cap. None of that is the story. The story is what the work is for: keeping an independent index alive while the traffic on it doubles and the machines asking for it stop being people.
Why an independent index matters now
A handful of companies own the indexes the answer engines draw from, each with its own crawlers, its own retrieval, its own idea of what counts. That fragmentation is the opening for smaller indexes. A page that one engine never crawls can still be found by another, and the more independent indexes exist, the less the whole web depends on any single company's crawl.
The OSINT world already works this way. Market research firms put the open source intelligence market anywhere from about 3 to 16 billion dollars for 2026, and every tool in it, from link analysis to breach exposure to threat intel, sits on the same shared layer: collecting live public web data at scale. The tools get the headlines. The collection layer does the work.
That is the bet behind cl0q. The web is not getting smaller, and the machines reading it are not going away. Somebody has to keep mapping it in the open, page by page, domain by domain. The code is going open source soon. For now the site is the product.
If you want to poke at it, the site is live. Search a domain you know and tell us what looks wrong.
Every cl0q dataset and the API credit pack are 30 percent off for everyone, 50 percent off for followers, November 27 to 30: https://cl0q.com/black-friday.html?utm_source=devto
This post was written with AI assistance and reviewed by a human before publishing.
Top comments (0)