DEV Community

Cover image for Open Source Project #127: wigolo — Zero API Key, $0/Query Local-First Web Intelligence for AI Agents, Compared Against Tavily / Exa / Firecrawl
WonderLab
WonderLab

Posted on

Open Source Project #127: wigolo — Zero API Key, $0/Query Local-First Web Intelligence for AI Agents, Compared Against Tavily / Exa / Firecrawl

#opensource #agents #crawler #claude

Introduction

"The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query."

This is article #127 in the "One Open Source Project a Day" series. Today's project is wigolo — a local-first web intelligence tool built specifically for AI agents, delivering search, crawl, fetch, and research capabilities over MCP with no API keys and zero per-query cost.

Equipping an AI agent with "web search" is a problem that looks simple on the surface but has surprisingly deep layers. Tavily, Exa, and Firecrawl have each carved out mature niches in the cloud space — but they share a fundamental trait: per-query billing, data leaving your machine, and mandatory account registration. wigolo's premise: search engines' raw data is public, ranking and extraction algorithms can run locally, and the agent's query history can be cached into a local semantic index. Why should every agent search cost money and travel to the cloud?

1,218 Stars. Created April 2026. Public beta.

What You'll Learn

  • wigolo's 10 tools and the design logic behind each
  • What "byte-pinned source provenance" and "explainable scores" mean, and why they matter
  • Local-first architecture: what runs on-device, what optionally reaches the cloud
  • Head-to-head comparison with Tavily, Exa, Firecrawl — each tool's differentiated positioning
  • What scenarios favor wigolo, and where cloud tools are still the better choice

Prerequisites

  • Experience with Claude Code or any AI agent
  • Basic familiarity with MCP (Model Context Protocol)
  • General understanding of AI agent tool-calling mechanics

Project Background

Overview

wigolo is a Node.js process that exposes 10 web-related tools as an MCP server, letting AI agents call search, fetch, crawl, extract, and other functions over the standard MCP protocol.

Core design principles:

  • Local-first: cache, vector index, embedding model, and reranker all run locally (~/.wigolo/)
  • No forced API keys: the six core tools — search, fetch, crawl, extract, cache, find_similar — require zero API keys
  • Explicit failure: bot blocks, dead engines, and stale cache hits are reported in results rather than silently dropped or disguised as empty results

Project Info

  • Author: KnockOutEZ
  • Language: TypeScript
  • License: AGPL-3.0
  • Status: Public Beta
  • Website: knockoutez.github.io/wigolo

Project Stats

  • ⭐ GitHub Stars: 1,218+
  • 🍴 Forks: 83+
  • 📄 License: AGPL-3.0
  • 📅 Created: 2026-04-12

Quick Setup 

# One command: download browser engine and local models, write MCP config
npx wigolo init --agents=claude-code

# Configure multiple agents at once
npx wigolo init --agents=claude-code,cursor,codex

# Check all component health
npx wigolo doctor
Enter fullscreen mode Exit fullscreen mode

Requires Node ≥ 20 and about 1.5 GB of disk space (browser engine + local ML models).

All six core tools work immediately after setup — no API keys needed. If you want research and agent tools to produce a synthesized answer (rather than raw data for the host LLM to process), a free Gemini key is the single biggest quality upgrade:

export WIGOLO_LLM_PROVIDER=gemini
export GEMINI_API_KEY=<free key from aistudio.google.com>
Enter fullscreen mode Exit fullscreen mode

For fully air-gapped operation, use a local Ollama model instead: WIGOLO_LLM_PROVIDER=ollama.

10 Tools, Explained

Core Retrieval Tools

search — Multi-engine parallel search

  • 18 direct engine adapters (no intermediate proxy API)
  • Rank fusion across engines + ML reranking
  • Pass an array of queries to fan out multiple searches in a single MCP call
  • Each result includes an explainable score: semantic + lexical + engine_consensus broken out separately

fetch — Tiered-routing page retrieval

Plain HTTP request
    ↓ (fails or detects SPA signals)
Headless browser (handles JS-rendered content)
    ↓ (encounters bot challenge)
Challenge clearance flow
    ↓ (clearance fails)
Flags blocked_by_challenge — does NOT pretend it succeeded
Enter fullscreen mode Exit fullscreen mode

wigolo learns which tier each domain requires and skips unnecessary escalations on repeat visits.

crawl — Multi-page crawling

Supports BFS, DFS, sitemap mode, or map-only (just build the site map, don't download content). Respects robots.txt, enforces per-domain rate limits, and deduplicates boilerplate templates.

extract — Structured extraction

Extracts from a page: tables, JSON-LD, metadata, brand assets, named schemas (Article / Recipe / Product / …), or any arbitrary structure via a custom JSON Schema.

Memory and Discovery Tools

cache — Local semantic cache

Every visited page is automatically cached. Supports both keyword and semantic similarity queries, works offline. Also exposes cache stats, clear, and change detection.

find_similar — Similar content discovery

Find pages similar to a given URL or concept, via three-way fusion: keyword + semantic + live web.

Research and Autonomous Tools

research — Deep research

Decomposes a question → fires parallel sub-queries → fetches sources → synthesizes a cited report (or generates a structured summary for the host LLM to write).

agent — Autonomous gather loop

Self-plans → searches → fetches → extracts → synthesizes, with configurable time budget and optional output JSON Schema.

Monitoring Tools

diff + watch — Page change detection

Check what changed on a URL since the last visit; set up recurring monitoring and push changes to a webhook.

Deep Dive: Two Unique Features

Byte-Pinned Source Provenance

This is wigolo's most distinctive technical feature — one that no other tool in the comparison set offers. Each search result carries not just a snippet, but:

{
  "excerpt": "Logical replication is a method of replicating data objects…",
  "citation_id": "src-1",
  "source_span": { "start": 1042, "end": 1305 },
  "evidence_score": {
    "final": 0.86,
    "semantic": 0.91,
    "lexical": 0.78,
    "engine_consensus": 3
  }
}
Enter fullscreen mode Exit fullscreen mode

source_span is a byte-offset range pinning the excerpt to its exact location in the original document — not "somewhere in this article," but a precise, verifiable position.

evidence_score breaks down into three independently interpretable components: semantic relevance, lexical match, and engine consensus (how many engines returned this result). Weak results are flagged as junk by wigolo's own scorer and surfaced in the output rather than silently filtered.

Why it matters: When an AI agent builds a RAG pipeline or answers questions that need citations, byte-level provenance means references are actually verifiable rather than plausibly attributed. The "AI might be fabricating citation locations" uncertainty drops by one layer.

Local-First Architecture 

AI agent (any MCP client / REST / SDK)
    ↓
wigolo process (single Node process)
    │
    ├── Search layer
    │   18 direct engine adapters
    │   rank fusion + ML reranking (local model)
    │
    ├── Fetch layer
    │   Tiered routing (HTTP → headless browser → challenge clearance)
    │   Per-domain learning + cache reuse
    │
    ├── Local storage (~/.wigolo/)
    │   Keyword index + vector index (local embedding model)
    │   Browser engine + ML models
    │
    └── Optional cloud (dashed)
        LLM API (only for research/agent synthesis)
        Aggregator backend (widen search funnel, legacy mode)
Enter fullscreen mode Exit fullscreen mode

What runs locally:

  • ✅ Direct search engine adapters
  • ✅ ML reranking model
  • ✅ Vector embedding model (all-MiniLM-L6-v2 etc.)
  • ✅ Headless browser engine
  • ✅ Local cache (SQLite + vector index)

What's optionally cloud:

  • 🔲 LLM synthesis (when research/agent needs a synthesized answer)
  • 🔲 Aggregator backend (widen the search funnel, optional)

Head-to-Head Comparison: wigolo vs Tavily / Exa / Firecrawl

This is the main event. wigolo's README includes a comparison table, but stops short of explaining each tool's actual positioning and optimal use case. Here's a deeper reading.

Feature Matrix

Capability wigolo Tavily Exa Firecrawl
Multi-engine web search ✅ ✅ ✅ ✅
Page fetch + structured extract ✅ ✅ ✅ ✅
Full-site crawl ✅ ✅ — ✅
Byte-pinned source provenance ✅ — — —
Explainable score decomposition ✅ — — —
Persistent local cache (offline re-query) ✅ — — —
Query data stays on your machine ✅ — — —
API key / account required None Required Required Required
Per-query cost $0 Metered Metered Metered

Each Tool's Actual Positioning

Tavily — Purpose-built search API for AI agents; currently the most widely integrated paid service in the "RAG + agent search" space. Strengths: production-grade quality out of the box, native support in LangChain/LlamaIndex and most major frameworks, search quality tuned specifically for agent use cases. Weakness: per-query billing, which can add up quickly in high-frequency agent workflows.

Exa — Positions itself as "semantic web search," designed to understand the intent behind a query rather than just matching keywords. Notable strength: high-quality retrieval of technical documentation and academic content — the benchmark section in wigolo's README notes that Exa successfully rendered the full comparison matrix from an official docs page that other tools missed. Like Tavily: requires registration + metered billing.

Firecrawl — Focused on web crawling and structured extraction, not search. Primary strength: handling complex JS-rendered pages, large-scale crawl jobs, and extracting structured data from arbitrary websites. The right tool for "given a set of URLs, batch-extract structured content" — not for "given a question, search for relevant content."

wigolo — Attempts to combine all three capabilities into a single local tool while eliminating API keys and per-query costs. The differentiated bets: byte-pinned provenance, explainable scores, local cache with offline re-query capability, and $0 core tooling cost.

Scenario Recommendations

Choose wigolo when:

  • Frequent technical documentation queries during development (no billing constraints)
  • Source traceability matters (byte-level pinning → agent citations are verifiable)
  • Privacy requirements prohibit query data leaving the machine
  • High agent query volume makes cloud API costs a concern
  • Building "memory-enabled" research tasks that benefit from local caching across sessions

Choose Tavily when:

  • Production-grade quality out of the box, no local dependencies to maintain
  • Already in the LangChain/LlamaIndex ecosystem needing framework-native integration
  • Query volume is manageable and per-query billing is acceptable

Choose Exa when:

  • Primary use case is technical documentation or academic content (semantic relevance priority)
  • Official documentation retrieval quality is critical (strong docs page rendering)

Choose Firecrawl when:

  • Batch crawling specific websites to extract structured content (not a search use case)
  • Large-scale web content processing pipelines

An Honest Limitation

wigolo's README flags one: data-center IP reputation scores lower than residential IPs on some sites' bot-detection systems, which can reduce challenge-clearance rates when running on a cloud server vs. a local machine. Self-hosted deployments should account for this; the README documents optional proxy configurations.

Other Integration Options

wigolo isn't just a Claude Code plugin. It supports multiple invocation paths:

REST API

wigolo serve   # starts on 127.0.0.1:3333

curl -sX POST http://127.0.0.1:3333/v1/search \
  -H 'Content-Type: application/json' \
  -d '{"query":"local-first software","max_results":5}'
Enter fullscreen mode Exit fullscreen mode

Cross-loopback connections require a bearer token; fail-closed by design.

TypeScript / Python SDKs

from wigolo import local_client

with local_client() as client:
    res = client.search(query="local-first web search", max_results=5)
    for r in res["results"]:
        print(r["title"], r["url"])
Enter fullscreen mode Exit fullscreen mode

The SDK automatically reuses a running daemon, or starts one if needed, and shuts it down as appropriate.

Framework Integrations

Framework Package What it exposes
LangChain wigolo-langchain Each tool as BaseTool + BaseRetriever for RAG
CrewAI wigolo-crewai wigolo_tools() passed directly to a crew
LlamaIndex wigolo-llamaindex BaseReader loading pages as Documents
Vercel AI SDK wigolo-vercel-ai-sdk Tool factory for generateText/streamText

Resources

Official Links

Summary

wigolo's core thesis: an AI agent's web access layer shouldn't be a perpetually-billed cloud API black box. Search engine raw data is public. Reranking and extraction algorithms can run locally. Query history can be cached into a local semantic index that persists across sessions. Put these together and you get a zero-cost, on-device, offline-capable web intelligence layer.

Two design decisions are worth singling out:

Byte-pinned source provenance: Unique among all the tools compared. When an agent cites a search result, source_span provides a byte-offset range that ties the excerpt to its exact position in the source document. For research tasks that require verifiable citations, this removes one layer of "the AI might be fabricating where this came from" uncertainty.

Explicit failure over silent pretense: Bot blocks surface as blocked_by_challenge. Dead engines report themselves. Weak results are scored and flagged by wigolo's own evaluator, then shown in the output rather than quietly discarded. This "always tell you what ground you're standing on" philosophy is significantly more reliable in agent contexts than tools that present silence or empty results when something goes wrong.

If your Claude Code workflow involves heavy documentation querying, technical information search, or if you're building an agent pipeline that needs web search — npx wigolo init --agents=claude-code is currently the lowest-cost entry point in the most literal sense: zero dollars, zero API keys, zero per-query billing.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.

Top comments (0)