FlyCrawl – Fast Web-to-Markdown scraper for LLMs (Pay-as-you-go crypto)

Feeding live web content into LLMs (like Claude 3.5, Cursor AI, or local DeepSeek/Llama models) has become a core requirement for modern AI agents and RAG pipelines.

However, anyone who has built web-scraping pipelines in production knows the two biggest headaches:

1. Token Bloat: A 1,000-word blog post typically balloons into 12,000+ tokens when converted to standard markdown because of navigation menus, header links, cookie consent banners, and tracking scripts.
2. Expensive Paywalls & Banking Restrictions: Most commercial scraping APIs require Western credit cards (Stripe) and force a minimum $99/month subscription tier, locking out indie developers and teams who just want pay-as-you-go.

To solve this, we built FlyCrawl — an ultra-fast, compiled Web-to-Markdown scraper and deep crawler engine designed specifically for AI contexts.

---

📉 How Fit-Markdown Cuts 88% of LLM Token Bills

Instead of running generic HTML-to-text converters, FlyCrawl implements Fit-Markdown: a heuristic DOM analyzer that strips non-semantic boilerplate before emitting markdown.

We benchmarked 200 diverse websites comparing raw HTML, standard markdown, and FlyCrawl:

• Raw HTML average: ~11,400 tokens
• Standard Markdown: ~6,200 tokens
• FlyCrawl Fit-Markdown: ~1,320 tokens (88.4% token reduction)

Vector retrieval recall also improves because embeddings aren't diluted by footer boilerplate and navigation bars.

---

💻 Quickstart: Using FlyCrawl in 60 Seconds

1. Native MCP Server (Cursor & Claude Desktop)

FlyCrawl is officially published on npm as a standalone Model Context Protocol (MCP) server:

bash
npx -y flycrawl-mcp

In Cursor (Settings -> Features -> MCP Servers) or Claude Desktop (claude_desktop_config.json):

json
{
  "mcpServers": {
    "flycrawl": {
      "command": "npx",
      "args": ["-y", "flycrawl-mcp"],
      "env": {
        "FLYCRAWL_API_KEY": "YOUR_FLYCRAWL_API_KEY"
      }
    }
  }
}
Now Claude or Cursor can scrape and search the live web directly without token bloat!

2. Official Python SDK (Drop-in Firecrawl Compatibility)
If you already have code written for Firecrawl, FlyCrawl's Python library is 100% API compatible:

bash
pip install flycrawl
python
from flycrawl import FlyCrawl
client = FlyCrawl(api_key="fc_live_YOUR_API_KEY")
# Scrape a page to clean, LLM-ready markdown
result = client.scrape({
    "url": "https://news.ycombinator.com",
    "formats": ["markdown"],
    "onlyMainContent": True
})
print(result.markdown)
🪙 Sovereign Crypto Payments (No $99/Month Lock-in)
No credit card required. You can top up your account with as little as $5 using USDT (TRC-20) or TON with instant, on-chain confirmation. Pricing is pure pay-as-you-go at $0.0019 per scraped page.

🎁 Free Playground
You can test any URL directly on our live playground without creating an account: 👉 https://flycrawl.net

We offer 100 free credits upon sign-up so you can test edge-case websites, dynamic JavaScript rendering, and anti-bot challenges.

Feel free to test challenging URLs and leave your benchmark feedback in the comments below!