SEO audits should not require uploading your entire website crawl to a third-party platform.
That is why I built Scary Spider SEO, a free, open-source desktop crawler for SEO, accessibility, and technical website audits.
It runs locally on Windows, macOS, and Linux. There are no crawl credits, hosted projects, or required accounts.
What It Does
Scary Spider SEO crawls a starting URL and helps you find issues such as:
- Missing or poorly optimized title tags
- Missing meta descriptions
- Broken links and resources
- Duplicate content
- Canonical URL problems
- Missing or duplicate H1 headings
- Redirects and HTTP errors
- Structured data issues
- Security header problems
- Accessibility violations
- Mobile usability problems
The application explains each issue in plain language and suggests what to fix.
Why Build a Desktop Crawler?
Many SEO crawlers are hosted services. That can be useful, but it also means:
- Your crawl data leaves your machine
- You may need an account
- Large crawls may require paid credits
- Audits are often presented as reports instead of actionable data Scary Spider SEO takes a different approach.
The crawler runs on your computer, and the results are displayed in a searchable, sortable interface. You can inspect individual pages, filter issues, save crawl snapshots, and export data to CSV.
Built With Rust, React, and Tauri
The application uses:
- Rust for the crawler backend
- React for the interface
- Tauri for the desktop application
- SQLite-compatible data structures for crawl results
- Headless browser rendering for JavaScript-enabled audits
Rust handles crawling, concurrency, resource collection, and page parsing. React provides the interface for exploring the results.
This combination makes it possible to keep the application lightweight while still supporting serious crawl workflows.
Configure Each Crawl
You can configure settings such as:
- Crawl depth
- Maximum page count
- Request concurrency
- Request timeout
- Request delay
- JavaScript rendering
- Sitemap seeding
- External link checking
- Image checking
- Accessibility auditing
- Mobile usability auditing
- Robots.txt compliance
JavaScript rendering is optional. Basic crawls can stay lightweight, while rendered crawls can inspect pages that depend on client-side JavaScript.
Useful for More Than SEO
Although the project started as an SEO crawler, a website audit often involves more than search rankings.
A page can have a correct title and still provide a poor experience for users. That is why Scary Spider SEO also supports accessibility and mobile usability checks when JavaScript rendering is enabled.
This makes it useful for:
- SEO practitioners
- Web developers
- Site owners
- Accessibility specialists
- Technical marketing teams
- Agencies performing client audits
Open Source and Local by Default
The project is available on GitHub:
GitHub repository link
You can inspect the code, report issues, request features, or contribute improvements.
The goal is not to replace every enterprise SEO platform. It is to provide a transparent, practical tool for people who want to inspect their websites without sending crawl data to another service.
Try Scary Spider SEO
Download the latest version for your operating system:
Download link
If you work on websites, build SEO tools, or care about local-first software, I would love to hear your feedback.
Try it, report an issue, and let me know which audit checks would be most useful in your workflow.
Top comments (0)