I Built a Better Web Search MCP for AI Agents — No API Key Required

Krystian — Sun, 27 Sep 2026 09:35:18 +0000

BetterWebSearch MCP — smarter web search, better data extraction, flexible and extensible, open source

Web search for AI agents sounds simple:

Search the web
Open the results
Extract useful information
Feed it back to the model

In reality, it gets messy pretty quickly.

Some websites work with a simple HTTP request. Others render almost everything through JavaScript. Search APIs often require accounts, API keys or paid plans, and dumping entire webpages into an LLM context wastes a huge amount of tokens.

That's why I built BetterWebSearch MCP.

👉 https://www.npmjs.com/package/better-web-search-mcp

👉 https://github.com/PhantomPixelDev/BetterWebSearch-MCP

It's a local, open-source Model Context Protocol server for web search, extraction and deep research.

And the default setup requires no API key, no account and no cloud service.

Zero-config web search

BetterWebSearch uses DuckDuckGo by default, so you can get started immediately.

Optional providers are also supported:

DuckDuckGo — keyless
Brave Search — optional
Tavily — optional

The easiest way to run it is:

npx -y better-web-search-mcp

Or add it directly to your MCP configuration:

{
  "mcpServers": {
    "better-web-search-mcp": {
      "command": "npx",
      "args": ["-y", "better-web-search-mcp"]
    }
  }
}

That's it.

No .env file is required for the default setup.

The bigger problem isn't search — it's extraction

Getting ten URLs isn't particularly useful if your agent can't actually read them.

Modern websites can require very different extraction strategies.

BetterWebSearch uses a three-tier extraction pipeline:

Tier 1: Fast HTTP fetch
       ↓
Tier 2: Structured / hydration data
       JSON-LD, __NEXT_DATA__, __NUXT__
       ↓
Tier 3: Playwright browser
       full rendering + API interception

The important part is that it only escalates when necessary.

If normal HTTP extraction works, there's no reason to launch Chromium.

If the page is JavaScript-heavy, BetterWebSearch can progressively fall back to structured application data and finally Playwright.

This keeps simple pages fast while still supporting modern web apps.

Built for research, not just search results

The MCP server currently exposes tools including:

Tool	Purpose
`web_search`	Search across providers, deduplicate and rerank results
`web_research`	Search, extract and return cited research passages
`deep_search`	Alias for `web_research`
`web_extract`	Extract clean content from a URL
`web_find`	Search inside a specific website
`web_news`	Search recent news with diversity filtering

The tool I find especially useful is web_research.

Instead of making the agent manually do:

web_search
↓
web_extract result 1
↓
web_extract result 2
↓
web_extract result 3
↓
web_extract result 4
↓
web_extract result 5

it can perform the search, extraction and passage selection inside the MCP server and return a much smaller research payload.

It can dramatically reduce agent context usage

This was one of the main things I wanted to improve.

I ran a benchmark comparing two workflows.

Traditional workflow

The agent:

performs a search
opens the top five pages
receives their extracted contents

BetterWebSearch research workflow

The agent makes one web_research call and receives selected passages with citations.

Across 12 live-web research questions, the results were:

	Traditional	`web_research`
Payload	820,229 chars	110,973 chars
Estimated tokens	~205,000	~28,000
Wall clock	137.3s	49.6s

That's an 86.5% reduction in returned text overall.

The median per-question reduction was 83.8%.

Of course, returning less text isn't useful if the important answer disappears.

So I added a second benchmark for answer retention.

Across 34 questions with known factual markers:

Numbers             8 / 8
HTTP status codes   6 / 6
Acronym expansions 12 / 12
Facts               7 / 8

Overall: 33 / 34
Retention: 97.1%

The benchmark source is included in the repository, so you can reproduce it yourself:

npm run build
npm run bench

Query expansion

Another useful feature is automatic query expansion.

Instead of searching only the exact sentence given by the model, BetterWebSearch can rewrite it into several related searches.

For example:

Unlimited mobile internet Germany

might also produce queries around:

unbegrenztes Datenvolumen Deutschland
unlimited data SIM Germany
German unlimited mobile plans

Those queries can run in parallel before the results are deduplicated and reranked.

This helps especially with searches where terminology differs between websites or languages.

Source independence

Search result count can also be misleading.

If the same wire story is republished by five websites, that isn't really five independent sources.

BetterWebSearch clusters similar content before citation so duplicated stories are less likely to be treated as separate evidence.

Self-learning extraction

Different domains require different extraction techniques.

BetterWebSearch remembers which extraction strategy worked for a domain.

So if the first visit discovers that a site only works properly through a particular structured-data path or browser fallback, later requests can skip unnecessary stages.

Security matters too

Web content is untrusted input, especially when it's being consumed automatically by AI agents.

BetterWebSearch includes several protections.

SSRF protection

URLs are validated and requests to things like:

localhost
private IP ranges
link-local addresses
reserved addresses

are rejected.

Redirects are checked again at every hop.

Prompt-injection detection

Extracted content is treated as untrusted.

web_extract can flag text that appears to contain instructions attempting to manipulate an agent.

The original content isn't rewritten, but the result carries security information so the calling model can treat it appropriately.

It runs locally

Another design goal was avoiding unnecessary infrastructure.

BetterWebSearch runs locally over MCP stdio.

There is:

no BetterWebSearch cloud
no proxy server
no telemetry
no analytics service
no mandatory account

Your queries go to the search provider and target websites, not through another hosted intermediary.

Works with most MCP clients

It should work with any MCP client supporting stdio, including:

Claude Desktop
Claude Code
Cursor
VS Code Copilot
Windsurf
Cline
Zed
OpenCode

I personally built it primarily around coding-agent and AI research workflows, but there's nothing client-specific about the server.

Try it

The fastest way:

npx -y better-web-search-mcp

NPM:

https://www.npmjs.com/package/better-web-search-mcp

GitHub:

https://github.com/PhantomPixelDev/BetterWebSearch-MCP

Documentation:

https://phantompixeldev.github.io/BetterWebSearch-MCP/

It's MIT licensed, so feel free to use it, fork it or contribute.

If you're using MCP agents heavily, I'd especially be interested in hearing about:

websites where extraction fails
unusual JavaScript-heavy sites
search-quality issues
agent workflows that consume too much context
providers you'd like to see supported

Issues and PRs are welcome.

TL;DR

BetterWebSearch MCP gives AI agents:

🔍 Keyless web search with DuckDuckGo

🧠 Deep research with citations

⚡ HTTP-first extraction

💧 Structured-data extraction

🎭 Playwright fallback when required

📉 Up to ~86% smaller research payloads in my benchmark

🔐 SSRF and prompt-injection protections

🔌 Support for popular MCP clients

🏠 Local execution

💸 No mandatory paid API

🛠️ Open source / MIT

npx -y better-web-search-mcp

DEV Community: Krystian