How I turned any website into an MCP server (and what I learned building it)

#agents #ai #mcp #showdev

MCP (Model Context Protocol) is becoming the standard for how AI agents
interact with the world. But there's a problem almost nobody is talking about:
99% of websites don't have MCP tools.

Which means AI agents can't use them.

I spent the last few months building AgentLayer to fix this. Here's what
I learned.

The Problem

If you want Claude or Cursor to interact with a website, you have three options:

1. Web scraping — works until the site changes its HTML. Brittle and
unreliable at scale.

2. Build a custom API — months of work, requires access to the codebase.
Not an option for third-party sites.

3. Write MCP tools by hand — slow, requires understanding every endpoint,
breaks when the site updates.

None of these are good enough.

The Approach

AgentLayer automates the entire pipeline:

Step 1: Crawl

We crawl the target URL using Playwright. We follow internal links, capture
page structure, identify navigation patterns.

Step 2: Analyze

We run the crawled content through an LLM to identify what actions and
data queries make sense for this site. What can a user do here? What
information is available?

Step 3: Generate

We generate MCP tool definitions — proper JSON schemas with input parameters,
descriptions, and expected outputs.

Step 4: Validate (the hard part)

This is where most tools fail. Generated tool definitions can look correct
but not actually work. We execute every tool in an E2B sandbox before
serving it. If it fails, we regenerate. You only ever get tools that
have been proven to work.

Step 5: Serve

A live MCP endpoint is ready. Copy it into your Claude Desktop config
or Cursor settings. Done.

What Actually Works

After testing on hundreds of sites, here's what I've found:

✅ Documentation sites (ReadTheDocs, GitBook, custom docs)

✅ Portfolio sites and blogs

✅ Content-heavy sites with clear navigation

✅ Open source project sites

⚠️ Heavy JavaScript SPAs (partial support)

❌ Sites requiring authentication

❌ Real-time data (prices, live feeds)