And what building it taught me about technical SEO that no course ever could.
Let me start with an embarrassing confession.
I was sitting in front of Screaming Frog, staring at a wall of red errors for a client site, and I genuinely did not understand why half of them were flagged. I knew what a missing H1 was. I knew what a canonical tag was supposed to do. But the connection between "this tag is wrong" and "this is why your traffic is dying" — that gap was very real for me.
So I did what any slightly obsessive person would do.
I built my own audit tool from scratch. Not because Screaming Frog or Ahrefs wasn't good enough. But because the only way I was going to truly understand what was happening under the hood of a website was to build something that looked under that hood myself.
That tool is CrawlIQ. And this is the story of how it works — and what building it taught me about technical SEO that I never learned from reading about it.
The Problem Nobody Talks About
Here's the thing about most SEO audit tools — they tell you what is wrong but not why it matters right now for your specific site.
You run an audit. You get a list of 47 issues. Meta descriptions too long. Missing H2s. Duplicate titles. Great. Now what?
Most tools treat every issue with equal urgency. But a missing H1 on your homepage is not the same as a missing H1 on a paginated archive page nobody visits. A duplicate meta description across two product pages is not the same as duplicate meta across your homepage and your most-trafficked landing page.
The priority matters. The context matters. And the fix matters even more.
I wanted a tool that didn't just find issues — it explained them, prioritised them intelligently, and told you exactly what to change. Not a generic recommendation. An actual rewritten title tag. An actual improved meta description. Ready to use.
That's what CrawlIQ does. And here's exactly how it works under the hood.
Layer 1 — Data Collection (aiohttp + BeautifulSoup)
Every audit starts with a simple question: what does this page actually contain?
To answer that at scale — across 50, 100, 500 URLs — you need speed. Sending HTTP requests one by one, waiting for each response before moving to the next, would take forever. On a 100-page site that could mean 5-10 minutes of waiting. Nobody has time for that.
This is where aiohttp comes in.
aiohttp fires async HTTP requests — meaning it sends requests to multiple URLs simultaneously without waiting for each one to finish before starting the next. Think of it like sending 20 WhatsApp messages at once instead of waiting for a reply to each one before sending the next. CrawlIQ processes 100 URLs per minute this way.
Once the raw HTML response comes back from each URL, BeautifulSoup takes over. It parses that raw HTML string into a navigable tree — like turning a wall of text into a structured document you can actually query.
For each URL, CrawlIQ extracts exactly this:
{
"url": "https://example.com/page",
"title": "Page Title Here",
"meta_description": "The meta description content",
"h1": "Main Heading",
"h2s": ["Subheading One", "Subheading Two"],
"canonical": "https://example.com/page"
}
That structured object — one per URL — gets passed to the next layer.
Layer 2 — The Rule Engine (issues.py + ai_analysis.py)
This is the layer that most people skip when they think about how SEO tools work. They assume AI is doing the detection. It isn't — at least not in CrawlIQ.
The detection is pure Python. Rule-based. Fast. Deterministic.
For every extracted data object, the rule engine runs a series of threshold checks:
| Check | Rule | Priority |
|---|---|---|
| Title length |
len(title) > 60 → "Title Too Long" |
Low |
| Missing title |
title is None → "Missing Title" |
High |
| Meta length |
len(meta) > 160 → "Meta Too Long" |
Low |
| Missing H1 |
h1 is None → "Missing H1" |
Medium |
| Duplicate meta | Same meta on 2+ URLs | High |
| Duplicate title | Same title on 2+ URLs | High |
| Missing canonical | canonical is None |
Medium |
Priority assignment uses Python set intersection — genuinely elegant in its simplicity:
_HIGH = {"Broken Page", "Missing Title", "Duplicate Title"}
_MEDIUM = {"Missing H1", "Missing Meta Description", "Duplicate Meta"}
def assign_priority(issues):
if set(issues) & _HIGH:
return "High"
if set(issues) & _MEDIUM:
return "Medium"
return "Low"
If any detected issue intersects with the HIGH set, the page gets flagged High priority. Medium next. Everything else is Low.
No AI. No black box. Just rules that mirror what Google's quality guidelines actually care about.
The output of this layer is a prioritised issues list per URL — which gets handed to the final layer.
Layer 3 — AI Fix Generation (Claude, Groq, Gemini, GPT-4o)
Here's where it gets interesting.
The rule engine knows what is wrong. The AI knows how to fix it — specifically, for this page, with this content, for this target keyword.
The issues list from layer 2 gets formatted into a structured prompt:
"This page has the following SEO issues: [list].
The current title is: [title].
The current meta description is: [meta].
Generate an optimised title (under 60 chars) and meta description
(under 160 chars) that fixes these issues. Return only valid JSON."
That prompt goes to whichever AI provider the user selects — Claude, Groq (Llama 3), Gemini, or GPT-4o. The response comes back as structured JSON with ready-to-use fixes.
Temperature is set to 0.15 — deliberately low. SEO fix recommendations need to be consistent and precise, not creative. You don't want your title tag rewritten in a different voice every time you run the audit.
The entire response pipeline looks like this:
URL input
↓
aiohttp → raw HTML
↓
BeautifulSoup → structured data object
↓
Rule engine → prioritised issues list
↓
LLM prompt → AI-generated fix recommendations
↓
Dashboard → exportable report
Clean. Traceable. Every decision explainable.
What Building This Taught Me About Technical SEO
Honestly? More than any course I've taken.
Crawlability is not binary. When you write the crawler yourself, you realise very quickly that "can Google crawl this page" is not a yes or no question. It's a spectrum — blocked by robots.txt, blocked by noindex, blocked by canonical mismatch, blocked by redirect chains, blocked by slow response times. Each one is a different problem with a different fix.
Meta descriptions don't affect rankings directly — but they affect everything else. I used to treat meta descriptions as a ranking signal. They're not. But when you're looking at hundreds of pages and seeing the same 160-character boilerplate repeated across 46 of them, you understand immediately why CTR tanks. Duplicate meta tells users — and AI engines — that these pages are interchangeable. They're not.
Priority is a business decision, not a technical one. The rule engine assigns technical priority. But whether fixing a missing H1 on a paginated archive matters more than fixing a duplicate title on your two highest-traffic pages — that's a business call. The tool surfaces the data. The SEO makes the call.
AI is a layer, not a replacement. The most common misconception I see is that AI-powered SEO tools are somehow smarter than traditional ones. They're not smarter — they're faster at generation. The intelligence is still in the rules, the structure, the prompts. If you feed bad data into an LLM, you get beautifully written bad recommendations.
Try It Yourself
CrawlIQ is free, open source, and requires no signup.
👉 Live tool: bhavani5a8.github.io/crawliq.io
👉 GitHub: github.com/Bhavani5A8
Enter any URL and it'll crawl up to 50 pages, detect issues across 50+ dimensions, and generate AI fix recommendations using your choice of Claude, Groq, Gemini, or OpenAI.
No credit card. No account. Just paste a URL and see what's actually happening on your site.
CrawlIQ auditing mockers.in — 51 pages crawled, 47 issues detected, 8% health score. Homepage flagged Medium priority with missing H1, missing canonical, and duplicate meta.
If you found this useful, drop a comment — I'm actively improving CrawlIQ and would love to hear what features matter most to you.
Top comments (0)