The latest articles on DEV Community by Vikram Yadav (@vikram_yadav_2323b2df8bda). https://dev.to/vikram_yadav_2323b2df8bda https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4198282%2F1122bd4d-fe9e-4423-aac3-10163b4e6e25.jpg https://dev.to/vikram_yadav_2323b2df8bda en Vikram Yadav Thu, 22 Oct 2026 04:29:58 +0000 https://dev.to/vikram_yadav_2323b2df8bda/i-built-sitescope-a-free-technical-seo-audit-tool-that-ranks-what-to-fix-first-1k1i https://dev.to/vikram_yadav_2323b2df8bda/i-built-sitescope-a-free-technical-seo-audit-tool-that-ranks-what-to-fix-first-1k1i <p>For my college course I built <strong>SiteScope</strong> (<a href="https://vikramyadav.me" rel="noopener noreferrer">vikramyadav.me</a>), a free technical SEO audit tool. You enter a site, it crawls every page it can reach, and it tells you what to fix first. No sign-up needed to run an audit. If you want, you can create an account to have your sites re-audited daily or weekly.</p> <h2> The problem </h2> <p>Two things bugged me about free SEO checkers.</p> <p>First, most test a single URL and hand back an unordered list. SiteScope follows internal links and the XML sitemap and checks every page it finds (up to 300 per audit). Then it ranks issues with one rule:</p> <p><strong>priority = severity × share of pages affected</strong></p> <p>A missing alt text on every page outranks a broken link on one page; a 5xx on half the site outranks both.</p> <p>Second, a lot of sites load their content with JavaScript. SiteScope reads raw HTML, the way most AI crawlers do, so you can check whether your content needs JavaScript to show up. The tool itself uses no JavaScript framework. Every page, including the report, is server-rendered, because a tool that talks about JavaScript SEO should be readable by a crawler that runs none.</p> <p>It also checks broken links, duplicate and near-duplicate pages (SimHash), the internal link graph (click depth, PageRank, orphan pages), titles, meta descriptions, H1s, canonicals, noindex and more. Reports download as PDF or JSON, and re-running an audit shows what was fixed and what is new since last time.</p> <h2> Tech stack </h2> <ul> <li> <strong>FastAPI</strong> (Python 3.12) with Jinja2 templates; audits run as FastAPI background tasks</li> <li> <strong>PostgreSQL 16</strong>, where each finished result is one JSONB column</li> <li> <strong>Docker</strong> Compose with an Nginx reverse proxy and Certbot</li> <li> <strong>Azure VM</strong>: one small VM (2 vCPU, 1 GiB RAM) running two containers</li> <li> <strong>Cloudflare</strong> for DNS and proxying</li> <li>httpx + asyncio for crawling, BeautifulSoup for parsing, NetworkX for the link graph</li> </ul> <h2> What I learned </h2> <ul> <li> <strong>A URL fetcher is an SSRF hole.</strong> If a stranger can type any URL, they will try the cloud metadata service or localhost. The guard refuses private, loopback and reserved addresses, checks every redirect hop, and connects to the exact IP it checked, so DNS rebinding can't swap the address in between.</li> <li> <strong>Don't trust a header anyone can send.</strong> My per-IP limit read Cloudflare's <code>CF-Connecting-IP</code>, but anyone hitting the server's IP directly could send that header with a new value each time. Now it only believes that header when the connection really comes from Cloudflare's IP ranges, and the limit lives in Postgres so a restart doesn't reset it.</li> <li> <strong>Keep dependencies small on a small VM.</strong> NetworkX's PageRank pulled in numpy and scipy, so I wrote my own PageRank and tested it against the reference. The scheduled re-audits run as a background task inside the app, with no cron, Celery or Redis.</li> <li> <strong>Severity alone gets the order wrong.</strong> Multiplying by the share of pages affected fixed it.</li> <li> <strong>An SEO tool shouldn't create thin pages.</strong> Report pages send noindex and are disallowed in robots.txt.</li> <li> <strong>DNS moves need care.</strong> Switching nameservers to Cloudflare replaces the whole zone, so I recreated the existing records first and issued SSL before turning on the proxy.</li> </ul> <h2> What's next </h2> <p>Some things are still open: alert emails come from a Gmail address for now (next step is sending from the site's own domain), and external link checks stop at 200 links per audit.</p> <p>Try it at <a href="https://vikramyadav.me" rel="noopener noreferrer">vikramyadav.me</a> and see the code on <a href="https://github.com/vikramyadav1404/sitescope" rel="noopener noreferrer">GitHub</a>. Feedback welcome!</p> seo python webdev showdev Vikram Yadav Wed, 21 Oct 2026 12:01:51 +0000 https://dev.to/vikram_yadav_2323b2df8bda/i-built-sitescope-a-free-technical-seo-audit-tool-that-ranks-what-to-fix-first-3j43 https://dev.to/vikram_yadav_2323b2df8bda/i-built-sitescope-a-free-technical-seo-audit-tool-that-ranks-what-to-fix-first-3j43 <p>For my college course I built <strong>SiteScope</strong> (<a href="https://vikramyadav.me" rel="noopener noreferrer">vikramyadav.me</a>), a free technical SEO audit tool. You enter a site, it crawls up to 300 pages, and it tells you what to fix first. No sign-up.</p> <h2> The problem </h2> <p>Two things bugged me about free SEO checkers.</p> <p>First, many test a single URL and hand back an unordered list. SiteScope follows internal links and the XML sitemap and checks every page it finds (up to 300 per audit). Then it ranks issues with one rule:</p> <p><strong>priority = severity × share of pages affected</strong></p> <p>A missing alt text on every page outranks a broken link on one page; a 5xx on half the site outranks both.</p> <p>Second, a lot of sites load their content with JavaScript. SiteScope reads raw HTML, the way the major AI crawlers do, so you can check whether your content needs JavaScript to show up. The tool itself uses no JavaScript framework. Every page, including the report, is server-rendered, because a tool that talks about JavaScript SEO should be readable by a crawler that runs none.</p> <p>It also checks broken links, duplicate and near-duplicate pages (SimHash), the internal link graph (click depth, PageRank, orphan pages), titles, meta descriptions, H1s, canonicals, noindex and more.</p> <h2> Tech stack </h2> <ul> <li> <strong>FastAPI</strong> (Python 3.12) with Jinja2 templates; audits run as FastAPI background tasks</li> <li> <strong>PostgreSQL 16</strong>, where each finished result is one JSONB column</li> <li> <strong>Docker</strong> Compose with an Nginx reverse proxy and Certbot</li> <li> <strong>Azure VM</strong>: one small VM (2 vCPU, 1 GiB RAM) running two containers</li> <li> <strong>Cloudflare</strong> for DNS and proxying</li> <li>httpx + asyncio for crawling, BeautifulSoup for parsing, NetworkX for the link graph</li> </ul> <h2> What I learned </h2> <ul> <li> <strong>A URL fetcher is an SSRF hole.</strong> If a stranger can type any URL, they will try the cloud metadata service or localhost. The guard refuses private, loopback and reserved addresses, and it checks every redirect hop, not just the first request.</li> <li> <strong>Keep dependencies small on a small VM.</strong> NetworkX's PageRank pulled in numpy and scipy, so I wrote my own PageRank and tested it against the reference.</li> <li> <strong>Severity alone gets the order wrong.</strong> Multiplying by the share of pages affected fixed it.</li> <li> <strong>DNS moves need care.</strong> Switching nameservers to Cloudflare replaces the whole zone, so I recreated the existing records first and issued SSL before turning on the proxy.</li> </ul> <h2> What's next </h2> <p>Some things are still open: a DNS-rebinding gap in the SSRF guard, and no PDF export or saved audits yet.</p> <p>Try it at <a href="https://vikramyadav.me" rel="noopener noreferrer">vikramyadav.me</a> and see the code on <a href="https://github.com/vikramyadav1404/sitescope" rel="noopener noreferrer">GitHub</a>. Feedback welcome!</p> seo python webdev showdev