I Built an SEO Tool That Lied to Me. So I Rebuilt It.

This is a submission for the GitHub Finish-Up-A-Thon Challenge.

What I Built

SEOCORE is the project I came back to finish for this challenge.

It started as a small SEO crawler I built for myself because I wanted a tool I could understand end to end. The first version looked more complete than it really was: it printed scores, generated reports, and even had a few useful ideas like crawl graph analysis and snapshot diffs.

But I eventually realised I could not trust its output.

This rebuild added a lot of new capabilities, but the most important change was that I finally made the tool trustworthy. I turned that abandoned prototype into a TypeScript CLI that can crawl real sites, handle redirects and robots.txt, render JavaScript-heavy pages when needed, and report issues with context and suggested fixes instead of hiding everything behind one misleading score.

The Before

The old script was roughly 700 lines. It had classes, interfaces, config objects, sitemap parsing, structured output, an HTML report, and a snapshot diff system.

Running it on example.com looked like this:

Starting SEO audit of https://example.com

https://example.com
Score: 80/100
Grade: B
Title: Example Domain
Meta description: missing
Canonical: missing

Audit complete. Report saved to ./audit-output/

And this was the kind of report it produced:

Before screenshot placeholder: old report with a misleadingly good score

That output was the problem.

A page with no meta description, no canonical tag, and no robots.txt should not feel "basically fine". But the tool wrapped weak logic in clean output, so I trusted it.

Two parts of the first version were actually useful:

• a crawl graph that mapped internal links and orphan pages
• a snapshot diff system that compared audits over time

Those good parts are exactly why I missed the bad foundation for so long.

Where v1 was wrong

The bugs were not dramatic. The script did not crash. It just produced plausible-looking answers.

1. Redirects were treated like failures

const CONFIG = {
  followRedirects: false,
};

if (res.status >= 300 && res.status < 400) {
  return;
}

That meant normal 301/302 behavior could stop the crawl.

2. Meta description extraction accepted invalid patterns

const metaDescription =
  $('meta[name="description"]').attr('content') ||
  $('meta[property="description"]').attr('content') ||
  $('meta[name="og:description"]').attr('content') ||
  undefined;

Those fallback selectors are wrong. A page could fail the real check but still look fine in my report.

3. Scoring was just arbitrary deductions

let score = 100;
if (!page.title) score -= 15;
if (!page.metaDescription) score -= 10;
if (!page.h1) score -= 15;

No real severity model. No category breakdown. No evidence. Just a number that looked authoritative.

That was the worst part of the old tool: not that it was incomplete, but that it was confidently wrong.

Why I stopped working on it

I did not abandon the first version because I got bored with SEO. I abandoned it because I lost confidence in the code.

Every time I tried to improve it, I ran into another blocker:

• redirects broke assumptions in the crawl flow
• site restrictions and real-world variance made naive checks unreliable
• some ideas I wanted, like better JavaScript rendering, stronger rules, and more trustworthy scoring, felt hard or impossible inside that codebase
• fixing one weak part usually exposed two more

At some point the project stopped feeling like "one more weekend and it is done" and started feeling like a pile of compromises I no longer trusted.

That killed my motivation more than the size of the code ever did.

Finishing it meant rebuilding it

For this challenge, I did not "polish the old script". I kept the useful ideas, then rebuilt the project around one rule:

correctness before polish

Here is the real before/after:

Area	Before	After
Crawl handling	Naive fetch() flow	rate limiting, retries, redirect-chain handling
Extraction	Fragile selectors	validated extractors and cross-checks
Link checking	false positives	better status handling and concurrency control
Scoring	one magic number	category-based scoring with severities
Output	score first	findings first, with fixes
Good ideas kept	crawl graph, snapshot diff	both retained and expanded

What I actually accomplished in the rebuild

The rebuilt version became SEOCORE, a TypeScript CLI I can actually use on real sites instead of just demo locally.

What shipped was much bigger than a simple script. SEOCORE now spans 20+ commands and feature areas across crawling, analysis, reporting, and workflow tooling.

These are the core pillars of the rebuild:

⚙️ 1. High-Performance Crawl Engine

• Concurrent Crawler: Built-in rate-limiting (Bottleneck) and queue control (p-queue) to handle large sites safely.
• Execution Tier System: Four distinct tiers (Fast, Standard, Deep, Enterprise) that dynamically adjust crawl budgets, rule sets, and scoring behavior.
• JS Rendering (Playwright): Full headless browser execution to audit single-page apps (SPAs) and client-side hydration.
• Redirect Loop Tracer: Intercepts 3xx responses, maps complete redirect chains, and flags circular loops.
• Compliance Guard: Automatic robots.txt parsing, sitemap.xml URL extraction, and path filtering (wildcard inclusions/exclusions).
• Visual Screenshot Capture: Automatically captures full-page and multi-breakpoint (mobile, tablet, desktop) screenshots using Playwright device descriptors.

🧠 2. Deep SEO & Entity Analyzers

• Structured Data Graph: Extracts Schema.org (JSON-LD, Microdata, RDFa), stitches nodes into an Entity Graph, resolves deep referencing pointers, and exports interactive Mermaid diagrams.
• E-E-A-T & Quality Scorer: Analyzes content readability (Flesch-Kincaid), internal link density, keyword stuffing, and authoritativeness.
• AI Visibility Auditor: Validates llms.txt rules and crawler directives for GPTBot, ClaudeBot, and PerplexityBot.
• Mobile & CWV Scorer: Audits viewport meta, tap targets, and scores mobile performance using throttled LCP/CLS metrics.
• Hreflang Validator: Deep-crawls and validates bidirectional hreflang links, x-default configurations, and language code formats.
• Outbound Authority & Rank Checker: Extracts backlink domain metrics and checks Google Top 10 organic rankings for target keywords.

🔍 3. Advanced Diagnostic & Strategy Tools

• JS SEO Impact Report: Compares raw source HTML against rendered DOM to flag metadata, link, or content parity issues caused by client-side JS.
• Dedicated Image Auditor: Audits images for weight, alt text, responsive srcset, lazy-loading, and CLS risk. Decodes dimensions with sharp.
• Tech Stack Detector: Evidence-based framework, CDN, and CMS detection using deterministic confidence weights.
• Business Directory Auditor: Checks local business listings (NAP consistency) across directories using a resilient search cascade.
• Internal Link Planner: Generates actionable internal linking recommendations, identifying orphan pages and suggesting source/target pairs with anchor text themes.
• Search Opportunities Analyzer: Combines crawl findings with optional GSC/CrUX data to prioritize page-level opportunities by business impact and ease of fix.
• Competitive Site Comparer: Compares health metrics, performance budgets, metadata, and link structures across two different URLs or exported JSON audits.

💼 4. Workflows & CI/CD Integration

• Snapshots & Diff System: Saves audit snapshots automatically and compares them over time.
• CI Regression Mode: Fails build pipelines only on SEO regressions (--diff --ci).
• Multi-Format Reports: Real-time terminal logs, structured JSON, interactive HTML, SARIF, and Mermaid diagrams.
• Dry-Run & Explain UX: Preview config without crawling (--dry-run) or explain rules/tiers in detail.

More importantly, the output became more usable: instead of one vague score, the tool now surfaces findings by category, severity, and suggested fix.

That does not mean every result is perfect on every site. Real websites are messy, edge cases are real, and some findings still need human verification. But the difference now is that the tool is designed to surface evidence and uncertainty more honestly instead of hiding weak logic behind a confident-looking score.

That was my definition of "finished": not flawless, but broad enough and trustworthy enough to use on real sites.

How GitHub Copilot helped

Copilot helped most when I already understood the target shape and needed a fast first draft.

1. Parser and matcher scaffolding

The robots.txt matcher and related parsing logic had a lot of repetitive branching. Copilot was useful for drafting the first pass, especially around wildcard and suffix handling. It did not get every edge case right, but it gave me something concrete to test and refine.

2. Type and interface scaffolding

During the rebuild, I split logic across packages and needed shared types like Finding, CrawlResult, and rule context objects. Copilot was good at generating the boring first version quickly. I still had to simplify and correct those types, but it removed a lot of mechanical typing.

3. Test skeletons

Copilot also helped generate initial Vitest test scaffolding. That saved time, especially for regression tests based on bugs from v1. The generated tests were not enough on their own, but they were a useful starting point.

What still required human judgment

GitHub Copilot sped up drafts, scaffolding, and test setup, but it did not decide what was actually correct.

It could suggest clean-looking code for selectors, scoring, and parser logic, but it could not tell me whether those choices matched real SEO behavior. I still had to compare against real tools, read specs, test edge cases, and decide what should count as severe.

That part stayed human:

• compare against real tools
• read specs
• test edge cases
• decide what should count as severe

Copilot accelerated implementation. Trust still had to be earned through validation.

Demo

These screenshots show sample output from the audit command. SEOCORE also includes 20+ commands and feature areas beyond audit.

Project: github.com/codepurse/SEOCORE
Package: npmjs.com/package/seocore

Final thoughts

The lesson from this project was simple:

a polished tool that gives wrong answers is worse than a rough tool that tells the truth

I stopped working on the first crawler when every fix revealed another bad assumption. Finishing this project did not mean forcing that codebase a little farther. It meant admitting the foundation was wrong, keeping the useful ideas, and rebuilding the rest so the output could be trusted.

That is why this challenge fit so well. I did not just reopen an abandoned project. I finally finished the hard part: making it honest enough to use.