I Built a Self-Hosted Web Layer for AI Agents

#webdev #ai #node #opensource

The Problem

If you're building AI agents that need to interact with the web, you're probably wiring together:

A search API (Tavily, Brave, etc.)
Playwright or Puppeteer for browsing
A cookie manager for auth
Custom scripts for each platform you want to post to

I got tired of this. So I built Spectrawl — one Node.js package that does all of it.

What It Does

Search across 8 engines with automatic fallback:

gemini-grounded → tavily → brave

Gemini Grounded Search gives you Google-quality results through their API. 5,000 free queries/month.

Browse with stealth anti-fingerprinting:

playwright-extra + stealth plugin (default)
Camoufox binary (optional, stronger anti-detect)
Auto-escalation when basic browsing gets blocked

Auth with persistent cookies:

SQLite cookie store
Multi-account support
Auto-refresh cron for expiring sessions

Act on 24 platforms from one API:

Post, comment, upvote, delete
X, Reddit, Dev.to, GitHub, LinkedIn, HN, and more
Rate limiting + dedup built in

Quick Start

npm install spectrawl
export GEMINI_API_KEY=your-free-key

const { Spectrawl } = require('spectrawl')
const web = new Spectrawl()

// Search — returns sources for your agent to process
const result = await web.deepSearch('Node.js web scraping libraries')
console.log(result.sources) // [{ title, url, content, score }]

// Browse a page with stealth
const page = await web.browse('https://example.com')
console.log(page.content)

// Post to a platform
await web.act('github, 'create_issue, {
  repo: 'myuser/myrepo,
  title: 'Bug report,
  body: 'Details here\n})

Speed

Honest numbers: ~6-10s per search. Tavily does ~2s. The difference is what you get back — Spectrawl returns full page content (scrapes the top results), Tavily returns snippets. Different tradeoff.

Cached queries return in <1ms.

MCP Server

Spectrawl ships with an MCP server, so any MCP-compatible agent can use it as a tool:

npx spectrawl mcp

Or run it as an HTTP server:

npx spectrawl serve --port 3900

DEV Community