Screaming Toad , not a frog. An alternative to Screaming Frog and Ashrefs have been built. Opensource and self hosted.
A technical SEO crawler can save hours of manual work.
It can find broken links, missing titles, indexing problems, duplicate pages, redirect issues, and pages that search engines may struggle to understand.
The problem is that many professional SEO tools require a recurring subscription.
That may be reasonable for a large SEO agency, but it can be difficult to justify for:
- Independent developers
- Students
- Small agencies
- Local businesses
- Open-source projects
- Founders managing only a few websites
This was the motivation behind SEO Screaming Toad — Not Frog, an open-source technical SEO crawler designed to run locally.
The goal is simple:
Crawl a website, identify technical problems, preserve the evidence, and help the user understand what should be fixed.
The project combines a Go crawler, SQLite persistence, a React dashboard, a local API, a command-line interface, and an MCP server for AI-assisted workflows.
But the main benefit is not the technology.
The benefit is being able to inspect and improve a website without paying for every crawl or uploading the crawl database to a third-party platform.
Why Build Another SEO Crawler?
There are already excellent commercial SEO tools available.
They are mature, powerful, and widely used.
However, many developers and small businesses do not need an entire enterprise SEO platform.
Sometimes they only need answers to a few practical questions:
- Are there broken links?
- Can search engines reach the important pages?
- Are page titles missing or duplicated?
- Are canonical tags configured correctly?
- Are redirects creating unnecessary chains?
- Is important content visible only after JavaScript runs?
- Did the latest website update introduce new problems?
A local crawler can answer these questions without requiring a cloud account or recurring subscription.
It also gives developers more control over the data and the implementation.
Testing the Crawler on Real Websites
The crawler was tested on two websites with very different purposes:
djai.academysiamesecat.cafe
DJAI Academy provides AI education, software-development services, and coding-related content.
Siamese Cat Café is a local business in Bangkok that depends on customers discovering its location, services, menu, and activities through search.
At first, both websites appeared functional.
The pages loaded, the navigation worked, and the content was visible.
But crawling the sites revealed issues that were easy to miss during normal development.
These included:
- Broken and outdated links
- Weak or duplicated page metadata
- Pages that were difficult for search engines to discover
- Indexing and sitemap inconsistencies
- Internal linking problems
- Page titles that did not clearly describe the content
- Technical differences between raw HTML and rendered content
After reviewing and fixing these issues, both websites showed better technical SEO health.
The improvements included:
- Fewer broken links
- Cleaner indexing signals
- Better internal navigation
- Improved SEO audit scores
- Stronger search visibility
- Higher rankings for relevant pages
The main lesson was that a website can look correct to a visitor while still sending confusing signals to search engines.
The Real Value of a Technical Audit
An SEO audit score is useful, but the score itself does not improve a business.
The value comes from what the score represents.
For example, fixing a broken link may lead to:
- Fewer visitors reaching dead pages
- More users reaching a service or product page
- Better internal distribution of link authority
- Easier crawling for search engines
- A more professional user experience
Fixing an indexing problem may allow a page that already exists to finally appear in search results.
Improving a title may help both search engines and users understand the page more clearly.
The benefit is not simply “better SEO.”
The benefit is helping the right person find the right page.
Local-First by Design
The crawler stores its working data locally using SQLite.
This means the crawl database stays on the user’s machine.
That provides several advantages:
- No required cloud account
- No URL-credit system
- No subscription needed to reopen previous crawl data
- Better control over client or internal website information
- Easier experimentation for developers
- A simpler setup than running several external infrastructure services
SQLite was selected because the application is intended to be easy to run locally.
A user should not need to configure Redis, PostgreSQL, Kafka, or a distributed queue before checking a small or medium-sized website.
SQLite also makes the application portable.
The crawler can store:
- Discovered URLs
- Fetch state
- Page metadata
- Links
- Audit findings
- Crawl events
- Checkpoints
- Comparison data
Write operations are controlled carefully, while the crawling workers can continue fetching pages concurrently.
Recovering Long Crawls
One of the more important design goals was recovery.
A crawler should not lose all progress because:
- The application closed
- The machine restarted
- The network failed
- The crawl was paused
- The user cancelled the job
- A rendering worker crashed
The crawler persists its frontier and crawl state.
It supports:
- Pause
- Resume
- Cancel
- Checkpoints
- Interrupted-run recovery
- Crawl history
- Recrawl comparison
This makes the crawl database more than a temporary cache.
It becomes a record of what was discovered, what was processed, and what still remains.
For larger websites, that is much more useful than restarting from zero after every interruption.
Raw HTML and Rendered Content Are Different
Modern websites often rely heavily on JavaScript.
A search crawler may receive one version of a page from the server and a different version after JavaScript executes.
For example:
- The server response may contain no title
- JavaScript may insert a title later
- Canonical tags may be added client-side
- Links may appear only after rendering
- Important content may not exist in the original HTML
It is tempting to render the page and treat the result as the only version.
However, that can hide useful evidence.
SEO Screaming Toad keeps raw and rendered evidence separate.
This allows the user to inspect:
- What the server originally returned
- What appeared after rendering
- Which version triggered an audit finding
- Whether JavaScript improved or damaged the page’s SEO signals
This distinction is especially useful when debugging React, Vue, Next.js, and other JavaScript-heavy websites.
Evidence Instead of Only Scores
Many audit tools present a list of problems, but it may not always be obvious why a page was flagged.
The crawler attempts to preserve evidence for each finding.
A finding can include:
- The affected URL
- Rule identifier
- Rule version
- Severity
- Extracted evidence
- Suggested remediation
- Known limitations
For example, a duplicate-title finding should not only say that duplicates exist.
It should identify the affected pages and preserve the titles that caused the rule to trigger.
A canonical issue should show the observed canonical value and the target involved.
This makes the audit easier to verify.
It also reduces the risk of blindly applying automated recommendations.
Audit Coverage
The current audit engine checks several common areas of technical SEO.
These include:
- HTTP errors
- Broken internal links
- Redirects
- Page titles
- Meta descriptions
- Heading structure
- Canonical tags
- Indexability
- Robots directives
- Robots.txt
- Sitemap coverage
- Hreflang
- Duplicate content signals
- Image alt attributes
- Broken image resources
- Crawl depth
- Internal linking
- Mixed content
- Selected security headers
- JSON-LD structured data
- Raw-versus-rendered differences
The project does not claim that every warning automatically affects ranking.
Technical findings should be reviewed in context.
For example, a missing page from a sitemap may be intentional. A duplicate title may be acceptable for certain filtered pages. A blocked page may be blocked by design.
The tool should provide evidence, not pretend to replace human judgment.
MCP Support for AI Agents
The project also includes a Model Context Protocol server.
This allows compatible AI clients to interact with the crawler through bounded tools.
An AI client can perform tasks such as:
- Create a crawl profile
- Preview crawl scope
- Start a crawl
- Pause or resume a job
- List pages
- Inspect links
- Review findings
- Explain a rule
- Compare two crawls
- Generate an export
The important word is bounded.
The MCP server does not provide:
- General shell execution
- Arbitrary SQL queries
- Unrestricted filesystem access
- Unrestricted HTTP requests
Instead, it exposes specific SEO and crawl operations.
This allows an AI assistant to help analyze an audit without giving it unnecessary access to the host machine.
A possible workflow could look like this:
- Start a bounded crawl.
- Wait for completion.
- List high-severity findings.
- Group issues by page type.
- Explain the most common rule.
- Export the results.
- Compare the crawl after fixes are deployed.
The goal is not to let AI control everything.
The goal is to give AI structured access to trustworthy crawl evidence.
Crawl Safety Matters
A crawler is also a network tool, so safety controls are important.
The project includes controls for:
- Robots.txt enforcement
- Per-host politeness
- Scope restrictions
- URL limits
- Depth limits
- Response-size limits
- Redirect validation
- Timeout budgets
- Retry limits
- DNS and IP validation
- TLS hostname verification
- Crawler-trap protection
These controls help prevent a simple website audit from turning into an unbounded or unsafe crawl.
Users should only crawl websites they own or are authorized to inspect.
What About the 100M+ URL Claim?
The repository includes an experimental architecture for segmented crawling campaigns.
The idea is to avoid treating a very large site as one enormous in-memory crawl.
Instead, work can be divided into bounded segments and processed over time.
Synthetic persistence and analysis campaigns have completed at 1 million and 5 million URLs.
However, the 100M+ figure should be understood carefully.
It is a theoretical architectural target.
It is not:
- A verified live crawl of 100 million URLs
- A guarantee
- A laptop benchmark
- A claim that every website can be crawled at that scale
- A promise of commercial support
Real-world capacity depends on:
- Storage
- CPU
- Memory
- Network speed
- Rendering requirements
- Website behavior
- Crawl limits
- Politeness settings
The large-scale architecture is experimental and should be evaluated separately from the normal local crawler.
What Improved on the Real Websites?
The most useful outcome came from applying the crawler to real projects.
For djai.academy, the audit helped identify technical problems affecting content discovery and page clarity.
Fixing those issues improved:
- Broken-link coverage
- Page metadata
- Internal linking
- Indexability
- Search visibility
- Rankings for relevant pages
For "siamesecat.cafe", the improvements were connected directly to local discovery.
A local business website needs search engines to clearly understand:
- What the business is
- Where it is located
- What services it offers
- Which pages are important
- How users can visit or contact it
Fixing technical problems gave the site a stronger foundation for appearing in relevant local searches.
The crawler did not create the business value by itself.
It revealed where existing value was being hidden by technical mistakes.
Reflection:
Building and using the crawler changed the way I think about website maintenance.
A site is not finished when it is deployed.
Every new page, redesign, framework update, CMS change, or content edit can introduce problems.
Technical SEO should therefore be part of the development cycle.
A useful workflow is:
- Crawl the current site.
- Save the results.
- Make a change.
- Crawl it again.
- Compare the findings.
- Fix regressions before they affect users or search visibility.
This is similar to testing code.
Developers would not normally deploy major changes without checking whether the application still works.
The same idea should apply to website structure, metadata, indexing, and internal links.
Project Status
SEO Screaming Toad is currently a release candidate.
It is not a stable replacement for every commercial SEO platform.
It also does not provide the proprietary keyword, competitor, and backlink databases available in broader tools such as Ahrefs.
Its purpose is narrower:
Provide an open, local, inspectable way to crawl websites and understand technical SEO problems.
The project is MIT licensed and open to technical review.
Repository:
Websites used for real-world testing:
Final Thoughts
Small technical problems can quietly reduce the value of an otherwise good website.
A business may already have useful content, good services, and a strong product.
But if links are broken, pages are difficult to index, or search engines receive unclear signals, that value becomes harder to discover.
The most important benefit of a crawler is not producing more data.
It is showing developers and business owners where their existing work is being wasted.
A free, local-first crawler lowers the barrier to finding those problems.
And once the problems are visible, they become much easier to fix.
Top comments (0)