DEV Community

Cover image for Harvest: Free Alternative to Firecrawl with Semantic Cache and MCP
Joseph Postman
Joseph Postman

Posted on

Harvest: Free Alternative to Firecrawl with Semantic Cache and MCP

#agents #ai #opensource #showdev

I spent the last month building something I couldn't find: a web scraper that doesn't
break every time a website updates its HTML.

Three weeks ago I open-sourced it. It's called Harvest,
it's MIT-licensed, and this is the honest tour — what it does, why it exists, and
the three lessons that cost me the most time.

The problem that started it

Every scraper I tried had the same failure mode: the website changes one CSS class,
and suddenly you're debugging at 2 AM. Firecrawl costs $50/month. Crawl4AI (72K stars,
great library) has no MCP server and no semantic cache. ScrapeGraphAI requires a
paid API.

I wanted something that bypasses Cloudflare, extracts data by plain-language
description (not CSS selectors), and works as an MCP server for AI agents. Free.

So I built it.

Three features that changed how I scrape

1. Semantic Cache — same meaning, zero tokens

Every AI-based scraper burns tokens on repeated queries. Ask "get all prices" and
then "extract product prices" — most tools process both from scratch.

Harvest caches by meaning, not exact text. Use sentence embeddings to compare
queries. Same intent? Cache hit. Zero tokens.

# First call: 2K tokens consumed
harvest llm-extract https://shop.com --prompt "Get all product prices"

# Second call: instant, 0 tokens
harvest llm-extract https://shop.com --prompt "Extract prices"

# Third call: instant, 0 tokens
harvest llm-extract https://shop.com --prompt "Find prices on page"
Enter fullscreen mode Exit fullscreen mode

The cache auto-invalidates when the HTML hash changes. I saw 50-70% token reduction
on real workloads.

2. Self-Healing Parsers — scrapers that repair themselves

This is the one I'm proudest of. When a website changes its HTML structure, Harvest
doesn't crash — it regenerates the CSS selectors via LLM.

Website updated →
  → Old selectors fail validation
  → LLM gets: old HTML fragment + new HTML + old selectors
  → LLM returns: new working selectors
  → New selectors validated against schema
  → Selector saved. User notified.
Enter fullscreen mode Exit fullscreen mode

All selector history is stored in ~/.harvest/self_healing/. You can roll back
any time.

# Enable self-healing
harvest llm-extract https://shop.com --prompt "Get prices" --self-healing
Enter fullscreen mode Exit fullscreen mode

3. Structural Diff — git diff for web pages

The most underrated feature. Before Harvest, when a parser broke I had no idea
what changed. Now:

# Capture a snapshot
harvest snapshot https://shop.com --name v1.0

# Compare later
harvest diff https://shop.com v1.0 latest
Enter fullscreen mode Exit fullscreen mode

Output:

📊 Structural Diff for https://shop.com

🆕 Added:
  • Block "Recommendations" (after description)
  • Field "Delivery" (in sidebar)

❌ Removed:
  • Field "SKU" (was in header)

🔄 Changed:
  • Price: <span class="price"> → <div class="price-container">

💡 Recommendation:
  Update your extractor: .price → .price-container .price-value
Enter fullscreen mode Exit fullscreen mode

The Script Generator trick

This one saved me the most operational cost. One LLM call analyzes a page and
generates a standalone Python script with hardcoded CSS selectors. Zero tokens
at runtime. Pure Scrapling + BeautifulSoup.

# One-time: 4K tokens
harvest generate https://catalog.com --fields title price image

# Forever: 0 tokens per run
./scrape_generated.py https://catalog.com/page/1

# Batch mode
./scrape_generated.py urls.txt --csv prices.csv
Enter fullscreen mode Exit fullscreen mode

Before: 1000 runs = 2M tokens ($0.30-2.00 in LLM costs).
After: 1000 runs = $0.00.

Why MCP matters

Most scraping tools are libraries. Harvest runs as an MCP server — any
MCP-compatible client (Claude, Cursor, Hermes, custom agents) can use it
out of the box.

pip install -e ".[mcp]"
harvest-mcp
Enter fullscreen mode Exit fullscreen mode

Available tools: scrape, extract, llm_extract, batch, crawl, monitor,
contacts, snapshot, diff, cache-stats, generate.

Honest limitations

I'm not going to pretend this solves everything:

  • Turnstile checkbox (behavioral biometrics) can still block. Regular Cloudflare JS challenges work fine.
  • Requires Chromium (auto-downloaded by Scrapling, ~300MB)
  • LLM extraction needs an endpoint — Ollama, OmniRoute, or any OpenAI-compatible API
  • Brand new — 0 GitHub stars as of writing. No community yet.

But within those bounds, it works. I use it daily for price monitoring, content
extraction, and web research.

Comparison

Feature Harvest Crawl4AI Firecrawl
Semantic Cache ✅ Meaning-based ❌ URL-only ❌
Self-Healing Parsers ✅ Auto-LLM repair ❌ ❌
Structural Diff ✅ DOM change detection ❌ ❌
Script Generator (0 tokens) ✅ ❌ ❌
MCP Server ✅ ❌ ❌
Cloudflare bypass ✅ Built-in ⚠️ Basic ✅
LLM extraction (plain language) ✅ ✅ ✅
Price Free Free $50/mo

Quick start 

pip install scrapling aiohttp
git clone https://github.com/zad111ak-ai/harvest
cd harvest
pip install -e .

# Full page content
harvest scrape https://news.ycombinator.com

# AI extraction — just describe
harvest llm-extract https://books.toscrape.com \
  --prompt "Get all book titles and prices"

# Monitor for changes
harvest monitor https://example.com/pricing

# Zero-token scraper
harvest generate https://shop.com --fields title price
./scrape_generated.py https://shop.com/page/1
Enter fullscreen mode Exit fullscreen mode

What I'd do differently

If I were starting today:

  1. Better docs first. I wrote README after the code. Docs-first would have caught API inconsistencies earlier.
  2. More tests earlier. 116 tests now, but the first 30 would have prevented three regression bugs.
  3. Release earlier. The first public version was v0.5.0. I should have shipped at v0.1.0 and iterated.

pip install harvest-agent

Or clone from github.com/zad111ak-ai/harvest

MIT licensed. 38 commits, 9.5K lines of Python. Built in 3 weeks.

Donation addresses in README if you find it useful — but the real contribution is feedback and issues.

Top comments (0)