The latest articles on DEV Community by Abdelrahman Ragab (@abdelrahman_ragab_20). https://dev.to/abdelrahman_ragab_20 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4118299%2F23c64756-972e-4b8b-9b90-5d79d05b8fbe.png https://dev.to/abdelrahman_ragab_20 en Abdelrahman Ragab Wed, 09 Sep 2026 23:48:49 +0000 https://dev.to/abdelrahman_ragab_20/i-built-a-website-crawler-because-it-works-in-the-browser-isnt-enough-54ji https://dev.to/abdelrahman_ragab_20/i-built-a-website-crawler-because-it-works-in-the-browser-isnt-enough-54ji <p>When you build a website, one of the easiest things to assume is:</p> <blockquote> <p>If the page looks fine in the browser, the website is fine.</p> </blockquote> <p>But a browser session only shows you one path through the application.</p> <p>A website can look completely normal while still having broken links, missing metadata, invalid structured data, crawlability problems, or performance issues across pages you never opened.</p> <p>That was the idea behind a project I’ve been building called sitebot.</p> <h2> The problem </h2> <p>As a frontend developer, I spend a lot of time checking websites from the browser.</p> <p>I usually have several things to verify:</p> <ul> <li>Does the page render correctly?</li> <li>Are the SEO tags there?</li> <li>Is the canonical correct?</li> <li>Are Open Graph tags present?</li> <li>Are there broken links?</li> <li>Is structured data available?</li> <li>Are important pages actually discoverable?</li> <li>How does performance look?</li> </ul> <p>The problem is that doing this manually doesn't scale very well.</p> <p>Checking one page is easy.</p> <p>Checking hundreds of pages is a different story.</p> <p>And that's where crawling becomes useful.</p> <h2> What is sitebot? </h2> <p>sitebot is a CLI website crawler and technical auditing tool.</p> <p>Instead of opening a website and manually inspecting pages, you can run an audit from the terminal.</p> <p>The basic idea is simple:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>sitebot https://example.com </code></pre> </div> <p>The crawler discovers pages and resources, then runs different checks against what it finds.</p> <p>The goal isn't to replace every specialized SEO or performance platform.</p> <p>It's to make technical website checks accessible from a developer-friendly CLI.</p> <p>What does it check?</p> <p>The project currently focuses on several areas.</p> <p>SEO metadata</p> <p>It can inspect important metadata such as:</p> &lt;/p&gt; &lt;p&gt;meta descriptions&lt;/p&gt; &lt;p&gt;canonical URLs&lt;/p&gt; &lt;p&gt;robots-related metadata&lt;/p&gt; &lt;p&gt;Open Graph metadata&lt;/p&gt; &lt;p&gt;These are small pieces of HTML, but missing or incorrect values can have a noticeable impact on how pages are interpreted and shared.&lt;/p&gt; &lt;p&gt;Broken links&lt;/p&gt; &lt;p&gt;A website can accumulate broken links over time.&lt;/p&gt; &lt;p&gt;This becomes particularly difficult to notice on larger websites where some pages may rarely be visited.&lt;/p&gt; &lt;p&gt;A crawler can follow links across the site and identify URLs that don't return the expected response.&lt;/p&gt; &lt;p&gt;Structured data&lt;/p&gt; &lt;p&gt;Structured data is another area where manually checking every page becomes tedious.&lt;/p&gt; &lt;p&gt;sitebot can inspect structured data found in pages and help surface pages where it is missing or potentially problematic.&lt;/p&gt; &lt;p&gt;Crawling and page discovery&lt;/p&gt; &lt;p&gt;One of the interesting parts of building a crawler is realizing that the list of URLs you think a website has isn't necessarily the list a crawler can actually discover.&lt;/p&gt; &lt;p&gt;The crawler follows links and builds a picture of the site's reachable pages.&lt;/p&gt; &lt;p&gt;This can reveal pages that behave differently from what you expected.&lt;/p&gt; &lt;p&gt;Core Web Vitals&lt;/p&gt; &lt;p&gt;Performance is also part of the audit.&lt;/p&gt; &lt;p&gt;Core Web Vitals provide useful signals around the actual user experience of a page, so I wanted performance to be part of the same developer-oriented workflow rather than something completely disconnected from the rest of the audit.&lt;/p&gt; &lt;p&gt;Why a CLI?&lt;/p&gt; &lt;p&gt;There are already many excellent website auditing tools.&lt;/p&gt; &lt;p&gt;So why build another one?&lt;/p&gt; &lt;p&gt;For me, the interesting part was the developer workflow.&lt;/p&gt; &lt;p&gt;I wanted something that could fit naturally into the terminal.&lt;/p&gt; &lt;p&gt;For example:&lt;/p&gt; &lt;p&gt;sitebot &lt;a href="https://mywebsite.com"&gt;https://mywebsite.com&lt;/a&gt;&lt;/p&gt; &lt;p&gt;And eventually:&lt;/p&gt; &lt;p&gt;Crawling website...&lt;/p&gt; &lt;p&gt;Pages discovered: 142&lt;/p&gt; &lt;p&gt;SEO&lt;br&gt; ✓ 138 pages have titles&lt;br&gt; ⚠ 4 pages have missing descriptions&lt;/p&gt; &lt;p&gt;Links&lt;br&gt; ✓ 136 pages checked&lt;br&gt; ⚠ 6 broken links found&lt;/p&gt; &lt;p&gt;Structured Data&lt;br&gt; ✓ 120 pages checked&lt;br&gt; ⚠ 8 pages with issues&lt;/p&gt; &lt;p&gt;Performance&lt;br&gt; ✓ Core Web Vitals collected&lt;/p&gt; &lt;p&gt;The exact output is still evolving, but the idea is to make the result easy to understand and useful enough to act on.&lt;/p&gt; &lt;p&gt;Building a crawler is more interesting than I expected&lt;/p&gt; &lt;p&gt;The auditing rules themselves are only part of the problem.&lt;/p&gt; &lt;p&gt;Once you start crawling real websites, you quickly run into questions like:&lt;/p&gt; &lt;p&gt;How should duplicate URLs be handled?&lt;/p&gt; &lt;p&gt;What happens with URL fragments?&lt;/p&gt; &lt;p&gt;How should redirects be treated?&lt;/p&gt; &lt;p&gt;How do you avoid crawling the same page repeatedly?&lt;/p&gt; &lt;p&gt;What should happen with external links?&lt;/p&gt; &lt;p&gt;How should query parameters be handled?&lt;/p&gt; &lt;p&gt;How do you deal with pages that are generated dynamically?&lt;/p&gt; &lt;p&gt;How much concurrency is reasonable?&lt;/p&gt; &lt;p&gt;How should failures and timeouts be represented?&lt;/p&gt; &lt;p&gt;A simple crawler can start with:&lt;/p&gt; &lt;p&gt;URL → request → parse links → repeat&lt;/p&gt; &lt;p&gt;But making it reliable requires much more thought.&lt;/p&gt; &lt;p&gt;That's one of the parts I've enjoyed most about working on this project.&lt;/p&gt; &lt;p&gt;What I want to improve&lt;/p&gt; &lt;p&gt;There are still plenty of things I want to explore.&lt;/p&gt; &lt;p&gt;Some of the areas I'm interested in are:&lt;/p&gt; &lt;p&gt;Better crawl reports&lt;/p&gt; &lt;p&gt;More SEO checks&lt;/p&gt; &lt;p&gt;Better structured-data validation&lt;/p&gt; &lt;p&gt;Improved performance reporting&lt;/p&gt; &lt;p&gt;More useful error classification&lt;/p&gt; &lt;p&gt;Smarter crawling&lt;/p&gt; &lt;p&gt;CI/CD integration&lt;/p&gt; &lt;p&gt;Machine-readable output for automation&lt;/p&gt; &lt;p&gt;Better support for JavaScript-heavy websites&lt;/p&gt; &lt;p&gt;The last point is particularly interesting.&lt;/p&gt; &lt;p&gt;Traditional crawling and modern frontend applications don't always see the web in the same way.&lt;/p&gt; &lt;p&gt;A page that depends heavily on client-side rendering can behave very differently depending on how it is accessed.&lt;/p&gt; &lt;p&gt;That's something I want to explore further.&lt;/p&gt; &lt;p&gt;What I learned&lt;/p&gt; &lt;p&gt;The biggest lesson so far isn't about crawling.&lt;/p&gt; &lt;p&gt;It's about building developer tools.&lt;/p&gt; &lt;p&gt;A tool can technically detect hundreds of problems and still be frustrating to use.&lt;/p&gt; &lt;p&gt;The hard part is turning raw technical information into something that answers:&lt;/p&gt; &lt;p&gt;What is wrong?&lt;/p&gt; &lt;p&gt;Where is it wrong?&lt;/p&gt; &lt;p&gt;Why does it matter?&lt;/p&gt; &lt;p&gt;What should I check next?&lt;/p&gt; &lt;p&gt;That's the direction I want to keep pushing sitebot toward.&lt;/p&gt; &lt;p&gt;Not just more checks.&lt;/p&gt; &lt;p&gt;Better diagnostics.&lt;/p&gt; &lt;p&gt;Try it&lt;/p&gt; &lt;p&gt;The project is open source and still under active development.&lt;/p&gt; &lt;p&gt;If you're interested in website crawling, SEO, frontend performance, or developer tooling, I'd genuinely appreciate feedback.&lt;/p&gt; &lt;p&gt;GitHub: &lt;a href="https://github.com/Abdelrahman5243/sitebot-cli"&gt;https://github.com/Abdelrahman5243/sitebot-cli&lt;/a&gt;&lt;/p&gt; &lt;p&gt;If you build websites yourself, I'm especially interested in one question:&lt;/p&gt; &lt;p&gt;What is something you regularly check manually that you wish you could just run from the terminal?&lt;/p&gt; seo typescript npm node