Building a Free, Local-First SEO Crawler for Developers and Small Businesses

Siamese_Cat_Dev — Sun, 02 Aug 2026 03:26:08 +0000

Screaming Toad , not a frog. An alternative to Screaming Frog and Ashrefs have been built. Opensource and self hosted.

A technical SEO crawler can save hours of manual work.

It can find broken links, missing titles, indexing problems, duplicate pages, redirect issues, and pages that search engines may struggle to understand.

The problem is that many professional SEO tools require a recurring subscription.

That may be reasonable for a large SEO agency, but it can be difficult to justify for:

Independent developers
Students
Small agencies
Local businesses
Open-source projects
Founders managing only a few websites

This was the motivation behind SEO Screaming Toad — Not Frog, an open-source technical SEO crawler designed to run locally.

The goal is simple:

Crawl a website, identify technical problems, preserve the evidence, and help the user understand what should be fixed.

The project combines a Go crawler, SQLite persistence, a React dashboard, a local API, a command-line interface, and an MCP server for AI-assisted workflows.

But the main benefit is not the technology.

The benefit is being able to inspect and improve a website without paying for every crawl or uploading the crawl database to a third-party platform.

Why Build Another SEO Crawler?

There are already excellent commercial SEO tools available.

They are mature, powerful, and widely used.

However, many developers and small businesses do not need an entire enterprise SEO platform.

Sometimes they only need answers to a few practical questions:

Are there broken links?
Can search engines reach the important pages?
Are page titles missing or duplicated?
Are canonical tags configured correctly?
Are redirects creating unnecessary chains?
Is important content visible only after JavaScript runs?
Did the latest website update introduce new problems?

A local crawler can answer these questions without requiring a cloud account or recurring subscription.

It also gives developers more control over the data and the implementation.

Testing the Crawler on Real Websites

The crawler was tested on two websites with very different purposes:

djai.academy
siamesecat.cafe

DJAI Academy provides AI education, software-development services, and coding-related content.

Siamese Cat Café is a local business in Bangkok that depends on customers discovering its location, services, menu, and activities through search.

At first, both websites appeared functional.

The pages loaded, the navigation worked, and the content was visible.

But crawling the sites revealed issues that were easy to miss during normal development.

These included:

Broken and outdated links
Weak or duplicated page metadata
Pages that were difficult for search engines to discover
Indexing and sitemap inconsistencies
Internal linking problems
Page titles that did not clearly describe the content
Technical differences between raw HTML and rendered content

After reviewing and fixing these issues, both websites showed better technical SEO health.

The improvements included:

Fewer broken links
Cleaner indexing signals
Better internal navigation
Improved SEO audit scores
Stronger search visibility
Higher rankings for relevant pages

The main lesson was that a website can look correct to a visitor while still sending confusing signals to search engines.

The Real Value of a Technical Audit

An SEO audit score is useful, but the score itself does not improve a business.

The value comes from what the score represents.

For example, fixing a broken link may lead to:

Fewer visitors reaching dead pages
More users reaching a service or product page
Better internal distribution of link authority
Easier crawling for search engines
A more professional user experience

Fixing an indexing problem may allow a page that already exists to finally appear in search results.

Improving a title may help both search engines and users understand the page more clearly.

The benefit is not simply “better SEO.”

The benefit is helping the right person find the right page.

Local-First by Design

The crawler stores its working data locally using SQLite.

This means the crawl database stays on the user’s machine.

That provides several advantages:

No required cloud account
No URL-credit system
No subscription needed to reopen previous crawl data
Better control over client or internal website information
Easier experimentation for developers
A simpler setup than running several external infrastructure services

SQLite was selected because the application is intended to be easy to run locally.

A user should not need to configure Redis, PostgreSQL, Kafka, or a distributed queue before checking a small or medium-sized website.

SQLite also makes the application portable.

The crawler can store:

Discovered URLs
Fetch state
Page metadata
Links
Audit findings
Crawl events
Checkpoints
Comparison data

Write operations are controlled carefully, while the crawling workers can continue fetching pages concurrently.

Recovering Long Crawls

One of the more important design goals was recovery.

A crawler should not lose all progress because:

The application closed
The machine restarted
The network failed
The crawl was paused
The user cancelled the job
A rendering worker crashed

The crawler persists its frontier and crawl state.

It supports:

Pause
Resume
Cancel
Checkpoints
Interrupted-run recovery
Crawl history
Recrawl comparison

This makes the crawl database more than a temporary cache.

It becomes a record of what was discovered, what was processed, and what still remains.

For larger websites, that is much more useful than restarting from zero after every interruption.

Raw HTML and Rendered Content Are Different

Modern websites often rely heavily on JavaScript.

A search crawler may receive one version of a page from the server and a different version after JavaScript executes.

For example:

The server response may contain no title
JavaScript may insert a title later
Canonical tags may be added client-side
Links may appear only after rendering
Important content may not exist in the original HTML

It is tempting to render the page and treat the result as the only version.

However, that can hide useful evidence.

SEO Screaming Toad keeps raw and rendered evidence separate.

This allows the user to inspect:

What the server originally returned
What appeared after rendering
Which version triggered an audit finding
Whether JavaScript improved or damaged the page’s SEO signals

This distinction is especially useful when debugging React, Vue, Next.js, and other JavaScript-heavy websites.

Evidence Instead of Only Scores

Many audit tools present a list of problems, but it may not always be obvious why a page was flagged.

The crawler attempts to preserve evidence for each finding.

A finding can include:

The affected URL
Rule identifier
Rule version
Severity
Extracted evidence
Suggested remediation
Known limitations

For example, a duplicate-title finding should not only say that duplicates exist.

It should identify the affected pages and preserve the titles that caused the rule to trigger.

A canonical issue should show the observed canonical value and the target involved.

This makes the audit easier to verify.

It also reduces the risk of blindly applying automated recommendations.

Audit Coverage

The current audit engine checks several common areas of technical SEO.

These include:

HTTP errors
Broken internal links
Redirects
Page titles
Meta descriptions
Heading structure
Canonical tags
Indexability
Robots directives
Robots.txt
Sitemap coverage
Hreflang
Duplicate content signals
Image alt attributes
Broken image resources
Crawl depth
Internal linking
Mixed content
Selected security headers
JSON-LD structured data
Raw-versus-rendered differences

The project does not claim that every warning automatically affects ranking.

Technical findings should be reviewed in context.

For example, a missing page from a sitemap may be intentional. A duplicate title may be acceptable for certain filtered pages. A blocked page may be blocked by design.

The tool should provide evidence, not pretend to replace human judgment.

MCP Support for AI Agents

The project also includes a Model Context Protocol server.

This allows compatible AI clients to interact with the crawler through bounded tools.

An AI client can perform tasks such as:

Create a crawl profile
Preview crawl scope
Start a crawl
Pause or resume a job
List pages
Inspect links
Review findings
Explain a rule
Compare two crawls
Generate an export

The important word is bounded.

The MCP server does not provide:

General shell execution
Arbitrary SQL queries
Unrestricted filesystem access
Unrestricted HTTP requests

Instead, it exposes specific SEO and crawl operations.

This allows an AI assistant to help analyze an audit without giving it unnecessary access to the host machine.

A possible workflow could look like this:

Start a bounded crawl.
Wait for completion.
List high-severity findings.
Group issues by page type.
Explain the most common rule.
Export the results.
Compare the crawl after fixes are deployed.

The goal is not to let AI control everything.

The goal is to give AI structured access to trustworthy crawl evidence.

Crawl Safety Matters

A crawler is also a network tool, so safety controls are important.

The project includes controls for:

Robots.txt enforcement
Per-host politeness
Scope restrictions
URL limits
Depth limits
Response-size limits
Redirect validation
Timeout budgets
Retry limits
DNS and IP validation
TLS hostname verification
Crawler-trap protection

These controls help prevent a simple website audit from turning into an unbounded or unsafe crawl.

Users should only crawl websites they own or are authorized to inspect.

What About the 100M+ URL Claim?

The repository includes an experimental architecture for segmented crawling campaigns.

The idea is to avoid treating a very large site as one enormous in-memory crawl.

Instead, work can be divided into bounded segments and processed over time.

Synthetic persistence and analysis campaigns have completed at 1 million and 5 million URLs.

However, the 100M+ figure should be understood carefully.

It is a theoretical architectural target.

It is not:

A verified live crawl of 100 million URLs
A guarantee
A laptop benchmark
A claim that every website can be crawled at that scale
A promise of commercial support

Real-world capacity depends on:

Storage
CPU
Memory
Network speed
Rendering requirements
Website behavior
Crawl limits
Politeness settings

The large-scale architecture is experimental and should be evaluated separately from the normal local crawler.

What Improved on the Real Websites?

The most useful outcome came from applying the crawler to real projects.

For djai.academy, the audit helped identify technical problems affecting content discovery and page clarity.

Fixing those issues improved:

Broken-link coverage
Page metadata
Internal linking
Indexability
Search visibility
Rankings for relevant pages

For "siamesecat.cafe", the improvements were connected directly to local discovery.

A local business website needs search engines to clearly understand:

What the business is
Where it is located
What services it offers
Which pages are important
How users can visit or contact it

Fixing technical problems gave the site a stronger foundation for appearing in relevant local searches.

The crawler did not create the business value by itself.

It revealed where existing value was being hidden by technical mistakes.

Reflection:

Building and using the crawler changed the way I think about website maintenance.

A site is not finished when it is deployed.

Every new page, redesign, framework update, CMS change, or content edit can introduce problems.

Technical SEO should therefore be part of the development cycle.

A useful workflow is:

Crawl the current site.
Save the results.
Make a change.
Crawl it again.
Compare the findings.
Fix regressions before they affect users or search visibility.

This is similar to testing code.

Developers would not normally deploy major changes without checking whether the application still works.

The same idea should apply to website structure, metadata, indexing, and internal links.

Project Status

SEO Screaming Toad is currently a release candidate.

It is not a stable replacement for every commercial SEO platform.

It also does not provide the proprietary keyword, competitor, and backlink databases available in broader tools such as Ahrefs.

Its purpose is narrower:

Provide an open, local, inspectable way to crawl websites and understand technical SEO problems.

The project is MIT licensed and open to technical review.

Repository:

https://github.com/lovecatisgood-sudo/Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential

Websites used for real-world testing:

https://djai.academy

https://siamesecat.cafe

Final Thoughts

Small technical problems can quietly reduce the value of an otherwise good website.

A business may already have useful content, good services, and a strong product.

But if links are broken, pages are difficult to index, or search engines receive unclear signals, that value becomes harder to discover.

The most important benefit of a crawler is not producing more data.

It is showing developers and business owners where their existing work is being wasted.

A free, local-first crawler lowers the barrier to finding those problems.

And once the problems are visible, they become much easier to fix.

opensource` #seo #webdev #golang

DEV Community: Siamese_Cat_Dev

Building a Free, Local-First SEO Crawler for Developers and Small Businesses

opensource` #seo #webdev #golang