When you build a website, one of the easiest things to assume is:
If the page looks fine in the browser, the website is fine.
But a browser session only shows you one path through the application.
A website can look completely normal while still having broken links, missing metadata, invalid structured data, crawlability problems, or performance issues across pages you never opened.
That was the idea behind a project I’ve been building called sitebot.
The problem
As a frontend developer, I spend a lot of time checking websites from the browser.
I usually have several things to verify:
Does the page render correctly?
Are the SEO tags there?
Is the canonical correct?
Are Open Graph tags present?
Are there broken links?
Is structured data available?
Are important pages actually discoverable?
How does performance look?
The problem is that doing this manually doesn't scale very well.
Checking one page is easy.
Checking hundreds of pages is a different story.
And that's where crawling becomes useful.
What is sitebot?
sitebot is a CLI website crawler and technical auditing tool.
Instead of opening a website and manually inspecting pages, you can run an audit from the terminal.
The basic idea is simple:
sitebot https://example.com
The crawler discovers pages and resources, then runs different checks against what it finds.
The goal isn't to replace every specialized SEO or performance platform.
It's to make technical website checks accessible from a developer-friendly CLI.
What does it check?
The project currently focuses on several areas.
SEO metadata
It can inspect important metadata such as:
</p>
<p>meta descriptions</p>
<p>canonical URLs</p>
<p>robots-related metadata</p>
<p>Open Graph metadata</p>
<p>These are small pieces of HTML, but missing or incorrect values can have a noticeable impact on how pages are interpreted and shared.</p>
<p>Broken links</p>
<p>A website can accumulate broken links over time.</p>
<p>This becomes particularly difficult to notice on larger websites where some pages may rarely be visited.</p>
<p>A crawler can follow links across the site and identify URLs that don't return the expected response.</p>
<p>Structured data</p>
<p>Structured data is another area where manually checking every page becomes tedious.</p>
<p>sitebot can inspect structured data found in pages and help surface pages where it is missing or potentially problematic.</p>
<p>Crawling and page discovery</p>
<p>One of the interesting parts of building a crawler is realizing that the list of URLs you think a website has isn't necessarily the list a crawler can actually discover.</p>
<p>The crawler follows links and builds a picture of the site's reachable pages.</p>
<p>This can reveal pages that behave differently from what you expected.</p>
<p>Core Web Vitals</p>
<p>Performance is also part of the audit.</p>
<p>Core Web Vitals provide useful signals around the actual user experience of a page, so I wanted performance to be part of the same developer-oriented workflow rather than something completely disconnected from the rest of the audit.</p>
<p>Why a CLI?</p>
<p>There are already many excellent website auditing tools.</p>
<p>So why build another one?</p>
<p>For me, the interesting part was the developer workflow.</p>
<p>I wanted something that could fit naturally into the terminal.</p>
<p>For example:</p>
<p>sitebot <a href="https://mywebsite.com">https://mywebsite.com</a></p>
<p>And eventually:</p>
<p>Crawling website...</p>
<p>Pages discovered: 142</p>
<p>SEO<br>
✓ 138 pages have titles<br>
⚠ 4 pages have missing descriptions</p>
<p>Links<br>
✓ 136 pages checked<br>
⚠ 6 broken links found</p>
<p>Structured Data<br>
✓ 120 pages checked<br>
⚠ 8 pages with issues</p>
<p>Performance<br>
✓ Core Web Vitals collected</p>
<p>The exact output is still evolving, but the idea is to make the result easy to understand and useful enough to act on.</p>
<p>Building a crawler is more interesting than I expected</p>
<p>The auditing rules themselves are only part of the problem.</p>
<p>Once you start crawling real websites, you quickly run into questions like:</p>
<p>How should duplicate URLs be handled?</p>
<p>What happens with URL fragments?</p>
<p>How should redirects be treated?</p>
<p>How do you avoid crawling the same page repeatedly?</p>
<p>What should happen with external links?</p>
<p>How should query parameters be handled?</p>
<p>How do you deal with pages that are generated dynamically?</p>
<p>How much concurrency is reasonable?</p>
<p>How should failures and timeouts be represented?</p>
<p>A simple crawler can start with:</p>
<p>URL → request → parse links → repeat</p>
<p>But making it reliable requires much more thought.</p>
<p>That's one of the parts I've enjoyed most about working on this project.</p>
<p>What I want to improve</p>
<p>There are still plenty of things I want to explore.</p>
<p>Some of the areas I'm interested in are:</p>
<p>Better crawl reports</p>
<p>More SEO checks</p>
<p>Better structured-data validation</p>
<p>Improved performance reporting</p>
<p>More useful error classification</p>
<p>Smarter crawling</p>
<p>CI/CD integration</p>
<p>Machine-readable output for automation</p>
<p>Better support for JavaScript-heavy websites</p>
<p>The last point is particularly interesting.</p>
<p>Traditional crawling and modern frontend applications don't always see the web in the same way.</p>
<p>A page that depends heavily on client-side rendering can behave very differently depending on how it is accessed.</p>
<p>That's something I want to explore further.</p>
<p>What I learned</p>
<p>The biggest lesson so far isn't about crawling.</p>
<p>It's about building developer tools.</p>
<p>A tool can technically detect hundreds of problems and still be frustrating to use.</p>
<p>The hard part is turning raw technical information into something that answers:</p>
<p>What is wrong?</p>
<p>Where is it wrong?</p>
<p>Why does it matter?</p>
<p>What should I check next?</p>
<p>That's the direction I want to keep pushing sitebot toward.</p>
<p>Not just more checks.</p>
<p>Better diagnostics.</p>
<p>Try it</p>
<p>The project is open source and still under active development.</p>
<p>If you're interested in website crawling, SEO, frontend performance, or developer tooling, I'd genuinely appreciate feedback.</p>
<p>GitHub: <a href="https://github.com/Abdelrahman5243/sitebot-cli">https://github.com/Abdelrahman5243/sitebot-cli</a></p>
<p>If you build websites yourself, I'm especially interested in one question:</p>
<p>What is something you regularly check manually that you wish you could just run from the terminal?</p>
The "browser session only shows one path" framing is the right starting point, because that's exactly where most manual QA stops.
One thing I'd watch: as soon as the crawler meets a page whose content depends on JavaScript, the raw HTML parse will silently report missing metadata that is actually injected at runtime. You can either shell out to a headless browser for the pages that need it, or flag them as "needs render" instead of "missing" — otherwise the report trains people to distrust it after the first false positive.
On politeness, three things saved me from quietly hammering sites in my own crawls: honoring robots.txt before queueing a URL, a global concurrency cap with a per-host delay, and a hard stop on 429/503 rather than a retry loop. If sitebot doesn't have those as defaults yet, they're worth adding early; auditors get blocked fast when they look like scraping.
How are you handling URL normalization and dedup, especially parameters that only change session tokens? That's the bug that makes a 40-page site look like 4000.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Top comments (1)
The "browser session only shows one path" framing is the right starting point, because that's exactly where most manual QA stops.
One thing I'd watch: as soon as the crawler meets a page whose content depends on JavaScript, the raw HTML parse will silently report missing metadata that is actually injected at runtime. You can either shell out to a headless browser for the pages that need it, or flag them as "needs render" instead of "missing" — otherwise the report trains people to distrust it after the first false positive.
On politeness, three things saved me from quietly hammering sites in my own crawls: honoring robots.txt before queueing a URL, a global concurrency cap with a per-host delay, and a hard stop on 429/503 rather than a retry loop. If
sitebotdoesn't have those as defaults yet, they're worth adding early; auditors get blocked fast when they look like scraping.How are you handling URL normalization and dedup, especially parameters that only change session tokens? That's the bug that makes a 40-page site look like 4000.