I Built an AI-Native Local Crawler to Audit My Clients' Websites

KR Shanto — Fri, 18 Sep 2026 04:30:35 +0000

Let's be honest. Our AI coding agents can inspect our code, understand the project structure, and help us change a shared layout. But it cannot automatically see that the website has broken canonicals, orphan pages, redirect chains, or inconsistent metadata across hundreds of URLs.

An SEO crawler can find those problems, but the results usually live in a separate dashboard. I still have to read the report, find the relevant source code, explain the issue to the agent, apply a fix, and then repeat the crawl.

For client websites, there is another concern. The site may be running locally, sitting behind a staging URL, or containing information I do not want to send to another cloud service.

I wanted a tighter loop:

Ask an AI agent to audit a website.
Run the crawl locally.
Let the agent query the findings it actually needs.
Fix the related code.
Run the check again.

I could not find a tool that worked exactly this way, so I started building Black Sparrow.

The tool I wanted to use

Black Sparrow is an open-source, local-first website crawler and technical SEO audit engine written in Rust.

It works as a regular command-line tool, so I can audit a site directly:

blacksparrow audit https://example.com --max-pages 500

The important part, however, is that Black Sparrow also runs as a Model Context Protocol server. That gives AI coding agents a structured way to start crawls, monitor progress, inspect pages, filter issues, and retrieve concise reports.

Instead of copying a large SEO export into a chat, I can ask the agent something closer to this:

Audit the SEO of this website. And fix them.

That's it.

The crawler gathers evidence from the website. The agent connects that evidence to the codebase.

That connection is the main reason I am building Black Sparrow.

Why I built it in Rust

Performance was one reason, but the larger reason was learning.

Building a crawler touches networking, asynchronous execution, streaming parsers, storage, graph algorithms, and error handling. Rust makes me think carefully about the boundaries between those parts.

Black Sparrow uses Tokio for asynchronous work and Reqwest for HTTP requests. It processes HTML with Cloudflare's lol_html streaming parser, extracting titles, descriptions, headings, links, canonicals, directives, images, and structured data without building a large DOM for every page.

Each page moves through a pipeline:

Fetch the response.
Parse the relevant HTML.
Run page-level SEO rules.
Add new internal links to the frontier.
Store the findings in SQLite.
Build the site graph for post-crawl analysis.

The current rule catalog covers 120 technical checks across metadata, indexability, links, canonicals, structured data, security, international SEO, and site architecture.

Giving AI agents tools instead of a data dump

Calling a project “AI-native” can easily become a marketing label. For Black Sparrow, it means designing the crawler around the way agents actually use tools.

A complete website crawl may take longer than an agent tool call is allowed to run. Returning thousands of raw findings can also waste most of the model's context window.

Black Sparrow's MCP server handles those constraints by exposing focused operations.

An agent can:

Start an audit in the background and receive a session ID.
Check the crawl's progress without blocking.
Run a quick audit on one page.
Query issues by severity, category, or URL.
Request a compact Markdown report.
Validate structured data.
Check whether AI search crawlers are allowed by robots.txt.

The agent asks for the information it needs instead of loading the entire audit into context.

A practical workflow could look like this:

The agent starts a crawl against http://localhost:3000.
Black Sparrow audits the pages and stores the results locally in SQLite.
The agent queries only critical and alert-level issues.
It traces repeated findings back to a shared component or layout.
After changing the code, it runs a single-page check for quick verification.
It starts another crawl to confirm the site-wide result.

The SEO tool and the coding agent remain separate systems, but MCP gives them a useful contract for working together.

Why local-first matters

Local-first is not only about avoiding an account or subscription.

When I work on a website, I don't want it to first publish, then use some online crawlers to judge the SEO. I want to test that locally with my AI agents so they can autonomously catch the issues and fix them.

The audit history stays in a local SQLite database. Reports can be exported as terminal output, JSON, Markdown, CSV, or interactive HTML when I need to share them.

It also makes the tool easier to include in development and CI workflows. The same engine can be used by a person in the terminal, a script in a pipeline, or an AI agent through MCP.

Trying Black Sparrow with an AI agent

Black Sparrow can be installed on macOS or Linux with:

curl -fsSL https://raw.githubusercontent.com/Shantodotdev/blacksparrow/main/install.sh | bash

An MCP-compatible coding agent can launch it with this configuration:

{
  "mcpServers": {
    "blacksparrow": {
      "command": "blacksparrow",
      "args": ["mcp"]
    }
  }
}

Then the agent can start audits and query the results through Black Sparrow's MCP tools.

The raw HTTP crawler is the main engine today. JavaScript-rendered analysis through the Chrome DevTools Protocol is planned for websites where the meaningful content only appears after client-side rendering. I also plan to build a desktop interface, while keeping the CLI and agent workflows lightweight.

Building it in public

Black Sparrow began with a simple frustration: my coding tools could understand the code, and my SEO tools could understand the website, but they could not work together.

Building the missing connection has turned into a deeper project involving Rust, networking, crawling, graph analysis, local storage, and agent-tool design.

If you work on client websites, try Black Sparrow on a local or staging project and tell me where the workflow feels rough. The most useful feedback at this stage is not praise. It is the point where the tool fails to match real work.

You can find the project here:

Shantodotdev / blacksparrow

Blazingly fast, local-first website crawler and 120-rule technical SEO audit engine with native MCP support.

Black Sparrow

High-performance, local-first website crawler, 120-rule technical SEO audit engine, and AI-native auditor written in Rust.

Overview

Black Sparrow is a lightweight, blazing-fast website crawler and technical SEO auditor designed as a modern, local-first alternative to heavy legacy tools like Screaming Frog and expensive cloud crawlers.

Built from the ground up in Rust, Black Sparrow streams HTML responses through Cloudflare's zero-copy lol_html parser, dynamically adjusts crawl speed using network congestion algorithms (AIMD), persists audit history in an embedded SQLite database, and exposes a native Model Context Protocol (MCP) server for autonomous AI coding agents.

Key Features

⚡ Blazingly Fast: Crawls ~500–800 pages/second on raw HTTP using under 50 MB of RAM.
🔒 Local-First & Private: All data stays on your machine in an embedded SQLite WAL database. No accounts, no cloud dependencies, zero telemetry.
🧠 AI-Native (MCP): Native stdio Model Context Protocol server. AI agents in…

View on GitHub

I will keep sharing what I learn while building it, including the crawler architecture, technical SEO rules, MCP design, and the mistakes behind each improvement. If those topics interest you, follow the project and follow me here on Dev.to.

DEV Community: KR Shanto