날다람쥐

Posted on Jul 7

I turned a Claude Code-only web reader into a normal MCP server

#mcp #ai #python #opensource

I kept running into the same problem while building with agents:

The page is public. A browser can read it. But my agent gets blocked, redirected, rate-limited, or handed a shell of HTML with no useful content.

There was already a fun project in this space: insane-search. It is clever, aggressive, and useful, but the original workflow is centered around Claude Code.

I wanted the same kind of public-web resilience as a regular MCP server, so any MCP-compatible client could call it.

That became unlimited-search.

flyingsquirrel0419 / unlimited-search

MCP server and CLI for reading public web pages through resilient public routes.

unlimited-search

English | 한국어 | 中文 | 日本語 | Español

From public pages to usable signal.

unlimited-search is a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool.

It is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls.

Quickstart

Prerequisite: Python 3.12 or newer.

python -m pip install unlimited-search
unlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800

The command returns JSON with page content, a verdict, request metadata, and an attempt trace.

For MCP clients, install the package and register the stdio server:

{
  "mcpServers": {
    "unlimited-search": {
      "command": "unlimited-search"
      "args"

…

View on GitHub

TL;DR

unlimited-search is a Python CLI and MCP server for reading public web pages through multiple public routes and fallbacks.

It can:

read one public URL
read many public URLs in one MCP call
diagnose why a page is hard to access
extract public media metadata with yt-dlp
fall back through public platform routes, browser-like HTTP fetching, public archives, feeds, and metadata extraction

It does not try to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, or real access controls.

Install

python -m pip install unlimited-search

Then add it to your MCP client:

{
  "mcpServers": {
    "unlimited-search": {
      "command": "unlimited-search",
      "args": ["serve"]
    }
  }
}

Restart your MCP client and you should get these tools:

read_public_url
read_public_urls
diagnose_access
extract_media

Quick test

You can also use it directly from the terminal:

unlimited-search read https://example.com --max-content-chars 1000
unlimited-search diagnose https://example.com

The output is JSON with the recovered content, final URL, verdict, metadata, and a trace of what was attempted.

Why I built it

Most agents are good at reasoning over text once the text is in context.

The annoying part is getting the text.

Modern public websites are not consistent:

some have useful public APIs
some render almost everything client-side
some expose RSS or Atom feeds
some have Open Graph or JSON-LD metadata but thin HTML
some work only with a browser-like TLS identity
some are easier to read through a public archive snapshot
some media pages are better handled as metadata, not raw HTML

So instead of pretending every URL is the same, unlimited-search tries a layered strategy.

How it works

For a URL, the reader tries the least invasive public route first:

Platform-specific public routes for sites like Reddit, X/Twitter, YouTube metadata, Bluesky, Mastodon, Hacker News, Stack Overflow, Wikipedia, GitHub, npm, PyPI, Naver Blog, Google Scholar, Amazon, and more.
Browser-like HTTP fetching with multiple identities, URL variants, referer strategies, and HTTP transport fallback.
Content fallbacks such as Jina Reader JSON, RSS/Atom discovery, common feed paths, OGP, JSON-LD, Schema.org, and Next.js metadata.
Public archive fallbacks through Wayback and archive.today/archive.ph best-effort snapshots.
Media metadata extraction through yt-dlp for public media URLs.

Every result includes a verdict:

strong_ok: a high-confidence public route worked
weak_ok: content was recovered, but may be partial or fallback-based
suspect_ok: something useful was found, but the caller should inspect it
failure verdicts for blocked, unsafe, login-only, CAPTCHA, private-network, or unusable results

Example MCP call

Ask your MCP client something like:

Use unlimited-search to read this public URL and summarize the page:
https://example.com

Behind the scenes, the model can call:

{
  "url": "https://example.com",
  "max_content_chars": 4000
}

And get back content plus a trace, not just a silent failure.

What makes it different from a normal fetch?

A normal fetch asks:

Can I GET this URL?

unlimited-search asks:

Is there a legitimate public route to useful text for this URL?

That difference matters for agent workflows.

If the direct HTML fetch is bad, the tool can try a public API. If the page is metadata-rich but content-light, it can salvage metadata. If the live page is flaky, it can try public archive routes. If the page is media, it can return structured media metadata instead of dumping a giant HTML shell.

The goal is not to be magical. The goal is to be practical.

Safety boundary

This is important: unlimited-search is a public-content reader.

It is designed to stop when the target requires:

authentication
payment
CAPTCHA
private network access
account-specific authorization
hard anti-abuse bypassing

It also rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default.

That boundary makes the tool useful for agents without turning it into a bypass tool.

The insane-search inspiration

I like the spirit of insane-search: make public web reading less fragile for agent workflows.

The difference is packaging and interface.

Instead of keeping the workflow tied to one agent environment, unlimited-search exposes the capability as a regular MCP server:

install with pip
run over stdio
call from any MCP-compatible client
inspect traces and verdicts
use it as a CLI when you do not need MCP

That makes it easier to plug into different local agent setups.

Try it

python -m pip install unlimited-search
unlimited-search read https://dev.to --max-content-chars 1500

GitHub:

https://github.com/flyingsquirrel0419/unlimited-search

PyPI:

https://pypi.org/project/unlimited-search/

If this saves you from writing yet another fragile page-fetching tool, a GitHub star would help the project get discovered.

And if you test it against a weird public site, I would love to hear what worked, what failed, and what fallback should be added next.

DEV Community