Why we gave the audit away
We run a full-stack SEO and GEO agency. Before any strategy work could matter, clients kept failing the same handful of checks: a crawler blocked in robots.txt, a page with three H1s, a sitemap full of 404s, JSON-LD that parsed fine but was missing the one property the schema actually required. We were running the same manual checks at the start of every engagement, and they were the same checks anyone could run if they knew where to look.
So we turned them into tools. Seven of them, all free, all no-signup. You paste a domain or a page URL, and you get results in seconds. They are the same checks we run in our own client audits, which is the whole point: there is no watered-down public version. This post is the engineering write-up. For each tool we will cover what it actually checks, and the one design decision we had to get right (a few we got wrong first).
There was a quieter reason too. We were tired of explaining the same problems over email. "Your GPTBot is blocked" lands very differently when the prospect can run the check themselves and see the line number than when it arrives as an assertion in a deck. A tool that shows its work is more persuasive than a consultant who says trust me, and it costs the reader nothing to verify. In practice the tools do a chunk of the audit conversation before we ever get on a call.
The framing: find, read, cite
The tools map to a pipeline, not a menu. We kept coming back to three questions about any page, and they have to be asked in order. Can an AI system find it? Can it read it? Can it cite it? Access gates structure, structure gates extraction. There is no value in validating schema on a page a crawler cannot reach, so the suite is built to be run roughly in that sequence.
The architecture is deliberately boring. A FastAPI backend, and each tool is a self-contained analyzer sitting behind one endpoint shape: take a URL, fetch what you need, return a structured result. The analyzers do not share state with each other, which makes each one easy to reason about and test in isolation. The page fetches run with strict timeouts, because the user is staring at a spinner and a slow upstream site cannot be allowed to hang the request. The API is rate limited, and beyond that we will keep the operational details out of this post. The interesting decisions are inside the analyzers, so that is where we will spend the words.
Seven tools, seven decisions
AI Crawler Checker
What it checks: you give it a root domain, it fetches robots.txt (HTTPS first, HTTP fallback), parses the file line by line, and returns a per-bot status for ten AI crawlers:
- GPTBot and ChatGPT-User: OpenAI
- Google-Extended and Googlebot: Google search and AI Overviews grounding
- PerplexityBot: Perplexity
- Anthropic-AI and ClaudeBot: Anthropic
- Bytespider: ByteDance
- CCBot: Common Crawl
- Applebot-Extended: Apple Intelligence
Each verdict is allowed, blocked, or partial, where partial means a path-specific Disallow.
The decision: every verdict has to cite the exact directive and its line number, or nobody acts on it. Our first internal version just returned "blocked" per bot. It was correct and useless. When we showed it to a client, the immediate response was "blocked by what?" and we had no answer in the UI. People do not edit a production robots.txt on the strength of a red label. They edit it when you can say "line 14, Disallow: / under the GPTBot group." So we rebuilt the parser to carry line numbers and the source directive all the way through to the result. The honest limitation we state plainly: this reads robots.txt only. If a CDN or WAF is blocking a bot at the network layer, we cannot see it, because it never appears in the file.
Meta Tag Analyzer
What it checks: a page URL, scored across five weighted dimensions to a 0 to 100 number.
- Basic Meta, 30 percent: title 50 to 60 chars, description 150 to 160
- Headings, 20 percent: exactly one H1, hierarchy skips flagged
- Open Graph, 20 percent
- Twitter Card, 15 percent
- Technical, 15 percent: canonical, html lang, viewport, robots meta, where a noindex is a fail
It returns up to three prioritized recommendations.
The decision: weighted dimensions instead of a flat pass/fail. A pass/fail checklist treats a missing twitter:card as equal to a missing title, which is wrong. They are not the same severity, and an analyst with limited time needs to know what to fix first. We landed on 30/20/20/15/15 because it reflects how much each dimension moves real visibility. Basic meta is the page's pitch to both search and assistants, so it carries the most weight. Headings are structural and matter almost as much. Open Graph governs how a shared citation renders. Twitter Card and the technical tags matter but are the smaller levers, hence 15 each. The single 0 to 100 score plus three ranked recommendations is what makes the output actionable rather than a 20-item list nobody triages.
Content Structure Analyzer
What it checks: a page URL across six weighted dimensions to 0 to 100.
- Heading Structure, 25 percent: minus 40 for a missing H1, minus 20 for multiple H1s, minus 15 per skipped level
- Content Depth, 20 percent: paragraph word count banded, with under 300 scoring 30, the 300 to 800 range scoring 60, 800 to 1500 scoring 80, and 1500-plus scoring 100
- Q&A Patterns, 20 percent: question headings, definition lists, and FAQ sections, scored zero, one to two, three to five, and six-plus as 20, 50, 75, 100
- Semantic HTML, 15 percent
- Lists, 10 percent
- Media, 10 percent: alt coverage
The decision: model the headings as a tree, not a list. Our first pass collected the headings into a flat array and counted them. That tells you there are four H2s, which is almost meaningless. The problem with headings is structural: an H2 that jumps straight to an H4 has skipped a level, and a flat list cannot see that gap. So we build a nested heading tree and walk it, which is the only way to detect skipped levels and to dock 15 points each time one shows up. The tree is also what lets us assert exactly one H1 cleanly. It reads server-rendered HTML only, which we flag in the results, because a page whose body is injected by client-side JavaScript looks empty to us and frequently looks empty to crawlers too.
Sitemap Checker
What it checks: a root domain. Discovery follows the crawler's path: the Sitemap: directive in robots.txt, then /sitemap.xml, then /sitemap_index.xml. It parses both url sets and sitemap index files (following up to five children), reports total URLs, lastmod coverage as a percentage, and the freshest and stalest entries. Then it health-checks: it samples min(50, total) random URLs, fires parallel HEAD requests, and classifies each as healthy, redirected, or broken with the status code. It flags duplicate URLs and files that breach the 50,000-URL limit.
The decision: sample 50 URLs in parallel rather than crawl everything. A real sitemap can list tens of thousands of URLs. Fetching all of them is slow, hammers the target site, and is pointless when the user wants results in seconds. A random sample of fifty is enough to tell you whether the sitemap is broadly healthy or broadly broken, which is the question that actually matters at audit time. We use HEAD instead of GET so we get the status code without pulling response bodies, and we run them in parallel so the whole health check finishes inside the same few-second budget as the rest of the suite. If the sample comes back clean, the sitemap is almost certainly fine; if a chunk of it 404s, you have a problem worth a deeper look.
Schema Generator
What it checks: a page URL. It extracts signals (title, meta description, canonical, up to thirty headings, up to ten images, author selectors, dates, breadcrumbs, price markers, FAQ markers), detects the page type, then generates ready-to-paste JSON-LD with confidence scores. Output types span Article/BlogPosting, Product, FAQPage, LocalBusiness, HowTo, BreadcrumbList, Organization, and WebPage/WebSite.
The decision: heuristics pick the page type before any LLM call. This was the most important call in the whole suite. The naive build is to hand the page to Gemini and ask "what type is this and write the schema." It works, sometimes, and it is expensive and unpredictable. Instead we run cheap deterministic heuristics first: price markers mean product, FAQ markers mean an FAQ page, an author plus a date means a blog post, an about-or-team page means organization, a contact page means local business, two or more "Step N" headings mean how-to, and everything else falls back to generic. Only then do we call Gemini Flash, and we call it with the type already decided, so the model is writing schema for a known shape rather than guessing the shape and the content at once. That grounds the output and cuts cost. The rule we enforce hard: any fact the model cannot read off the page becomes an explicit placeholder. It never invents a value. A tool you paste into production cannot hallucinate an author name or a price.
Schema Validator
What it checks: a page URL or pasted JSON-LD. It extracts every ld+json block and runs three layers. JSON syntax. schema.org type rules, with required and recommended properties for fifteen common types, plus ISO 8601 date validation. Then content alignment. Each block scores syntax 40 + required 30 + recommended 20 + alignment 10, and the page averages its blocks.
The fifteen types group sensibly:
- Articles:
Article,BlogPosting,NewsArticle(require headline and author), andFAQPage - Commerce:
Product(requires name) - Local:
LocalBusiness(requires name and address) andEvent - Media:
Recipe,VideoObject, andHowTo - Site structure:
BreadcrumbList,Organization,WebSite, andWebPage
The decision: valid JSON is not valid schema, and the output has to make that distinction loud. Early on we leaned on a JSON parse and called it a day. The trouble is that a block can be perfectly valid JSON and still be useless markup, because it is missing the headline an Article requires or the address a LocalBusiness requires. Those are the failures that actually keep you out of rich results and clean citations. So we built per-type required-property tables for the fifteen types and score against them, with required properties weighted far above recommended (30 versus 20) because a missing required property is a hard failure and a missing recommended one is a nudge. Gemini suggests markup you are missing on top of that, but those suggestions never block the result. A syntactically clean, type-complete block validates clean regardless of what the model would add.
OG Previewer
What it checks: a page URL. It extracts Open Graph and Twitter tags, falls back to title and meta description where tags are missing, and renders previews for Facebook, X/Twitter, LinkedIn, and Slack using each platform's real fallback chain. It checks og:image accessibility server-side and recommends 1200×630 (1.91:1). A missing og:title, description, image, or url is an error; a missing twitter:card is a warning.
The decision: implement each platform's real fallback chain rather than a single generic preview. It is tempting to read the OG tags once and draw one card. But the platforms do not agree. Each one has its own order for what it falls back to when a tag is missing, and a card that looks fine in one place can render blank in another. Since these cards are the first impression when an AI-cited link gets shared into a channel, "looks fine somewhere" is not good enough. So we model Facebook, X, LinkedIn, and Slack separately, each with its own fallback logic, and render four cards. The server-side image accessibility check exists because a perfectly tagged og:image that returns a 403 still produces a broken card, and you cannot see that by reading the HTML alone.
What we deliberately left out
There are no signups. No "enter your email to see results." No PDF gate, no "book a demo" wall between you and the numbers. The entire value of these tools is that you can run one on a hunch in the middle of an audit, and anything that interrupts that with a form kills the use case. The hard constraint we held to: results in seconds, or people leave. Every design decision above bends toward that, from the parallel HEAD requests in the Sitemap Checker to the heuristic type detection that keeps the Schema Generator from making unnecessary LLM calls.
We also left out a "score history" feature, at least for now. It is the obvious next ask: run a page weekly and chart the trend. We skipped it because it would have forced accounts, and accounts would have reintroduced the exact friction we were trying to delete. A tool you can run anonymously in ten seconds is a different product than a dashboard you log into, and we decided the anonymous tool was the one worth shipping first.
The honest trade-off in all of this is that these tools check readiness, not outcomes. They tell you whether AI systems can find, read, and cite a page. They cannot tell you your actual citation share inside ChatGPT or Perplexity, or the sentiment of how an assistant describes you. That is a different system, watching live answers over time, and we were careful not to over-promise it in the tool copy. A readiness check that is honest about its boundary is more useful than one that implies it measures the thing it cannot see.
Run it on your own site
The tools are meant to be run as a sequence, find then read then cite, and the whole pass takes about half an hour.
Start with access. Run the AI Crawler Checker on your root domain and clear any blocked AI bots, then run the Sitemap Checker on the same domain to catch broken URLs and stale lastmod coverage. Move to readability: put your two or three most important pages through the Meta Tag Analyzer and the Content Structure Analyzer, fix the title and description lengths, enforce one H1, and turn vague section headings into the questions your buyers actually ask. Then make yourself citable: run your top page through the Schema Generator, paste the JSON-LD into the page, confirm it with the Schema Validator, and check how the shared card renders with the OG Previewer.
All seven live on the tools hub. They are free and there is nothing to sign up for. If you build something with them or find an edge case we missed, we want to hear about it.
Mehul Jain is an AI entrepreneur and product builder working on Geology.
Top comments (0)