DEV Community

Philip Stayetski
Philip Stayetski

Posted on

Run a Local Search Engine as an Agent Capability: No SERP Scraping, No API Keys

#ai #agents #llm #opensource

Your agent needs to look things up. The default move is scraping a search engine results page — parse HTML, dodge rate limits, pray the markup doesn't change under you. The second move is wiring a hosted search API: sign up, generate a key, embed it in your runtime, watch the per-query bill. There's a third path that skips both: run a local search engine as an agent capability — a grounded web search tool installed once on your machine, then called as typed JSON, with no SERP scraping and no API keys to juggle. Here's the recipe, end to end.

Why the Agent Shouldn't Scrape the SERP

Scraping a results page from inside an agent loop fails in quiet, annoying ways. The HTML changes and your parser breaks — not with an error, but with subtly wrong results. Rate limits kick in mid-run and your agent stalls on a 429 instead of finishing the task. And the results you get are optimized for a human eyeballing a screen, not for a model trying to extract a citation.

The hosted-API path fixes the scraping but introduces its own tax: key provisioning, key rotation, per-query pricing, and a different SDK to learn for every search vendor you evaluate. None of that is the task you're trying to do, which is: give the agent ground truth, fast.

Run a Local Search Engine as an Agent Capability

The pattern that actually holds up in production is treating search as a capability: a tool the agent owns, installed locally, discoverable and callable without any browser or REST plumbing. Pilot Protocol's agent app store is built exactly around this loop — discover → install → call — and it takes three commands.

Discover what's available:

pilotctl appstore catalogue
Enter fullscreen mode Exit fullscreen mode

Install the search app:

pilotctl appstore install io.pilot.cosift
Enter fullscreen mode Exit fullscreen mode

Call it:

pilotctl appstore call io.pilot.cosift cosift.search '{"q":"grounded web search for AI agents"}'
Enter fullscreen mode Exit fullscreen mode

That last line is the whole point. The app runs locally on your daemon as a typed IPC service — JSON in, JSON out — and returns ranked hits with URLs and excerpts. Your agent never touches a search engine's HTML, never holds a search API key, and never leaves the local call boundary.

What "Local" Actually Means Here

It's worth being precise about the architecture, because it's what makes this different from yet another API wrapper.

  • The capability lives on your machine. The app is installed locally and auto-spawned by the daemon on demand. There's no browser session, no REST round-trip from your agent to a third-party endpoint you have to authenticate.
  • Installs are signature-verified. The app manifest pins a sha256 hash and an ed25519 signature, re-checked each time the app spawns. You're not curling a random script into your runtime.
  • Permissions are grant-scoped. The app asks for what it needs at install time, rather than inheriting ambient authority over your whole environment.
  • Capabilities are runtime-discoverable. Every app exposes a help contract, so the agent can learn what a tool does before calling it: 
pilotctl appstore call io.pilot.cosift cosift.help '{}'
Enter fullscreen mode Exit fullscreen mode

This is a genuinely different shape from a search MCP server or a plugin: the adapter is local and thin, the trust boundary is explicit, and the heavy retrieval backend stays out of your agent's critical path. MCP servers are a fine pattern for many things — the difference here is where the adapter lives and how installs are verified.

The Full Recipe: Search → Read → Answer

A single search call gets you ranked hits, but the recipe gets better when you chain the capability. The cosift app's methods are built for exactly this flow — search, then read a hit, then synthesize an answer over the corpus:

pilotctl appstore call io.pilot.cosift cosift.search '{"q":"x25519 key exchange"}'
Enter fullscreen mode Exit fullscreen mode

Take a hit's URL from the results and pull its full text:

pilotctl appstore call io.pilot.cosift cosift.contents '{"url":"https://example.com/..."}'
Enter fullscreen mode Exit fullscreen mode

Or skip the manual reading and ask for a grounded answer with citations over the same corpus:

pilotctl appstore call io.pilot.cosift cosift.answer '{"q":"what is x25519 used for?"}'
Enter fullscreen mode Exit fullscreen mode

And when one answer isn't enough, cosift.research runs a multi-step research pass instead of a single lookup. The pattern generalizes beyond search: any capability you install this way becomes a local tool your agent can compose — search for evidence, pull the source, answer with citations, all without leaving typed JSON.

When You'd Still Self-Host an Index

Honesty clause: "local search engine" can mean two different things, and the distinction matters. If you need to search your own corpus — internal docs, a codebase, a dataset — the right tool is an embedded engine over your files, not a web search app. The same app-store pattern covers that case too: the catalogue includes in-process engines like DuckDB for analytical queries over local files, no server to provision.

If your agent needs grounded web retrieval — current information, sources it can cite — that's what the search capability is for. Pick the tool by what you're searching, not by what's easiest to install.

Get Started

The whole loop — install the CLI, browse the store, install a search app, call it from your agent — starts with one command:

curl -fsSL https://pilotprotocol.network/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Then:

pilotctl appstore catalogue
Enter fullscreen mode Exit fullscreen mode

The store is discoverable by the 243k+ agents on the network, which is the other half of the payoff: build a capability once, and any agent can find and install it. Scraping a SERP is a workaround for not having a tool. Now you have the tool.

Top comments (0)