DEV Community

Cover image for My AI agent lost its search engine. Here's what I learned about who actually owns a web index.
Eber Cruz Fararoni
Eber Cruz Fararoni

Posted on

My AI agent lost its search engine. Here's what I learned about who actually owns a web index.

#ai #architecture #opensource #selhhosted

One year ago today — August 11, 2025 — Microsoft shut off the Bing Search APIs. I didn't notice at the time. I noticed last week, when our AI agent's search stopped working and I went looking for a replacement.

The immediate failure was mundane. DuckDuckGo started answering our requests with anti-bot challenge pages instead of results. A few dozen automated queries from one IP was enough. What started as a scraping incident turned into a much more interesting lesson: about how the web search industry actually works, and about a phrase we all repeat without examining it.

First surprise: almost nobody has an index of the web

Researching alternatives, I found something that should be better known: DuckDuckGo doesn't have its own full index either. This isn't a leak or an accusation — DDG says so on its own help pages. It maintains its own crawler (DuckDuckBot) and several indexes for Instant Answers, but the traditional links and images "largely" come from Bing. Yahoo resells Bing too.

The instinct is to sort search engines into "has an index" and "doesn't." That's the wrong axis, because it flattens three orders of magnitude. Sorted by scale instead:

  • Tens of billions of pages, independently crawled: Google, Bing, Brave, Yandex, Baidu. Brave's product page currently states 30B+ pages with 100M+ daily updates; its February 2026 blog post says 40B. Take the number with a date attached.
  • Emerging, regional: Staan, the index built by Qwant and Ecosia through their European Search Perspective joint venture (formed November 2024). It started serving real queries to French Ecosia users on August 6, 2025, targeting 30% of French queries by year end. Worth noting what this actually is: Ecosia spent sixteen years reselling other people's results and then built its own index, explicitly because Microsoft raised API prices. It's the clearest example of the escape path existing.
  • Billions, independent, deliberately smaller: Mojeek (~5B pages, its own MojeekBot, running since 2004), Yep, Stract, Seekport, Petal Search, Marginalia. These are not small Googles. Several are intentionally biased toward the non-commercial web, which for some workloads is a feature.

The uncomfortable conclusion: "build your own Google" doesn't mean crawling the web. Crawling a billion pages means roughly 100 TB of transfer per pass (~100 KB of HTML per page, before you store or parse anything). It means a ranking function that needs the whole web graph to mean anything. It means an adversarial war against the SEO spam industry.

And it means the moat nobody mentions: permission. Googlebot gets through because sites want to be in Google. A new crawler hits the same anti-bot walls that blocked me, multiplied by millions of domains. Brave is unusually candid about this — its documentation notes that its crawler deliberately doesn't advertise a distinctive user-agent, precisely to avoid discrimination from sites that only allow Google. That reputation took decades to build, and it isn't for sale.

The substrate is closing — on both sides

Here's what makes this a 2026 story rather than a personal anecdote. When I started looking for a paid API to replace scraping, I found that both sanctioned routes had been closed:

  • Bing Search APIs: retired August 11, 2025. Existing instances decommissioned completely, no new signups. New resource creation had already been disabled months earlier, so by the time the announcement landed in May 2025 you couldn't provision a fresh key at all. Microsoft's suggested replacement, Grounding with Bing Search, is an Azure AI Agents feature — not a SERP API you can point at your own retrieval pipeline.
  • Google Custom Search JSON API: discontinued January 1, 2027. Closed to new customers in 2025; the shutdown date was announced in January 2026. Google points developers to Vertex AI Search, which is not a drop-in replacement — it's an enterprise product with configuration minimums in the range of 1,000 queries per minute and 50 GiB of indexing.

To be clear about my own incident: these two things are not causally linked. DuckDuckGo blocking my agent had nothing to do with the Bing retirement — DDG was explicitly unaffected, because it holds a longstanding syndication contract with Microsoft rather than an API subscription. Wired confirmed that with them at the time. Which is the sharper version of the same point: access to a web index isn't bought from a catalogue, it's negotiated. Whoever has a contract survives the shutdown. Everyone else gets a challenge page. My rate-limit incident wasn't a consequence of the retirement — it was the other end of the same asymmetry.

So the third-party scraper market absorbed the demand, and then that got contested too. On December 19, 2025, Google sued SerpApi, framing the claim as DMCA §1201 circumvention — alleging SerpApi defeated its SearchGuard anti-bot protections at a scale of hundreds of millions of queries per day. On July 20, 2026, the Northern District of California granted SerpApi's motion to dismiss, rejecting the attempt to extend the DMCA to cover access to public pages. The court left Google room to amend, so this is not settled.

I'm not offering a legal opinion; I'm pointing at a supply risk. If you are building an agent in 2026, "we'll just call a search API" is a dependency on a market that lost both of its official suppliers in eighteen months and is now litigating the unofficial ones. That belongs in your architecture decision record, not in a footnote.

Second surprise: "if it's free, you're the product" has a huge exception

While evaluating SearXNG — an open-source metasearch engine that aggregates something like 200 configurable engines — my first reflex was the classic suspicion: if it's free, you're the product.

That's the conceptual error worth taking apart in public.

The phrase applies to free services. SearXNG is not a service. It's free software you host in your own container — exactly like PostgreSQL or NATS, which are also free and don't profit from anyone's data. No intermediary. No telemetry. Nobody sees anything.

…except the upstream engines. Google, Bing, and the rest still receive your queries, from your IP. That is the same exposure as searching by hand, minus the cookies and minus the fingerprint — but it is not zero.

The full honesty matters here, because half of it is the part that gets used as marketing: your data isn't monetized, but your queries do reach third parties. Only two things eliminate that. A commercial API contract (Brave sells access to its independent index; Tavily and others sell retrieval shaped for agents), or your own index. Note that even the API route has been tightening: Brave removed its traditional free tier for new signups in early 2026, replacing it with ~$5/month in credits, roughly 1,000 queries. There is no longer a free ramp for prototyping.

What we actually built: a ladder, not a leap

Our decision wasn't to marry any one option. It was to own the seam — a search port with a swappable provider chain:

  1. Self-hosted (SearXNG). Immediate resilience. One engine getting banned stops being the whole system getting banned.
  2. Contracted APIs (Brave, Tavily, …). For production, with each client's own key. The cost lands on the party who owns the workload.
  3. Own index — as a ladder, not a moonshot. First vertical: 1–10M pages of a single niche, where a curated index genuinely beats Google. Then regional, on top of Common Crawl — billions of pages already crawled, free, in the open. This is not theoretical: Stract started from Common Crawl and has since developed its own independent crawler and index. Only by accumulation does a first-party crawler make sense, earning reputation one domain at a time.

Total sovereignty is a legitimate goal. What's unworkable is treating it as step one.

The engineering rule the incident actually left us

This is the part I'd want if I were reading someone else's post. The bug wasn't that search failed. The bug was that our agent couldn't tell the difference between an empty web and a closed door.

A challenge page returns HTTP 200. It has a <title>. It parses. Our extractor found zero results and reported, truthfully and uselessly, "no results." The agent concluded the query was bad, rephrased it, and tried again. And again. Reformulating queries forever against a wall, believing the world was empty.

Three states, not two:

from dataclasses import dataclass
from enum import Enum


class SearchOutcome(Enum):
    OK = "ok"                  # results, trust them
    EMPTY = "empty"            # provider answered, genuinely nothing matched
    BLOCKED = "blocked"        # provider refused us: challenge, 403, 429, captcha
    DEGRADED = "degraded"      # answered, but the shape is wrong (parse failed, truncated)
    UNAVAILABLE = "unavailable"  # timeout, DNS, 5xx


@dataclass
class SearchResponse:
    outcome: SearchOutcome
    provider: str
    results: list
    detail: str = ""

    @property
    def should_rephrase(self) -> bool:
        # ONLY EMPTY justifies changing the query.
        return self.outcome is SearchOutcome.EMPTY

    @property
    def should_failover(self) -> bool:
        # Everything else is a provider problem. Rotate, don't rewrite.
        return self.outcome in {
            SearchOutcome.BLOCKED,
            SearchOutcome.DEGRADED,
            SearchOutcome.UNAVAILABLE,
        }
Enter fullscreen mode Exit fullscreen mode

The classifier that fills this in is the load-bearing part, and it cannot rely on status codes alone:

CHALLENGE_MARKERS = (
    "unusual traffic", "verify you are human", "captcha",
    "enable javascript and cookies",
)


def classify(status: int, body: str, parsed: list) -> SearchOutcome:
    if status in (401, 403, 429) or status == 503:
        return SearchOutcome.BLOCKED
    if status >= 500:
        return SearchOutcome.UNAVAILABLE
    if status != 200:
        return SearchOutcome.DEGRADED

    lowered = body[:4000].lower()
    if any(marker in lowered for marker in CHALLENGE_MARKERS):
        return SearchOutcome.BLOCKED          # a 200 that is really a refusal

    if parsed:
        return SearchOutcome.OK
    if _looks_like_a_results_page(body):
        return SearchOutcome.EMPTY            # the page rendered, it just had nothing
    return SearchOutcome.DEGRADED             # we can't tell — don't claim emptiness
Enter fullscreen mode Exit fullscreen mode

The default for the ambiguous case is DEGRADED, never EMPTY. An honest system refuses to assert that the web is empty unless it can prove it looked.

Once BLOCKED and EMPTY are distinct, the provider chain writes itself: BLOCKED rotates to the next provider and trips a per-provider circuit breaker with backoff; EMPTY propagates to the agent as real information. Only then does swapping SearXNG for Brave for Tavily become a config change instead of a rewrite. The abstraction isn't valuable because it hides providers — it's valuable because it makes their failures legible.

Sources

Primary where possible, with dates, because the whole point of this post is that these facts have expiry dates.

Google's Custom Search JSON API discontinuation (January 1, 2027) is documented in Google's own developer notices; verify the current date against Google's docs before relying on it, since sunset schedules move.

If you're building search for agents, I'd genuinely like to know which rung of the ladder you stopped on — and whether your retrieval layer can currently tell an empty web from a closed door.

Top comments (0)