DEV Community

Cover image for I run 16 CLI commands, 47 content checks, and 27 AI crawler tests. Here's which ones actually matter
Juan Camilo Auriti
Juan Camilo Auriti

Posted on AI-assisted

I run 16 CLI commands, 47 content checks, and 27 AI crawler tests. Here's which ones actually matter

GEO Optimizer runs 16 CLI commands, 47 content checks, and 27 AI crawler tests per audit. After 1,800+ audits and a calibration study that came back null, I have an honest ranking of which checks matter and which are table stakes. Here is that ranking.

Tier 1: These checks predict something real

These are the checks where a failure means you are almost certainly invisible to at least one AI engine. Fix these first.

robots.txt blocking AI crawlers. The audit checks 27 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-Web, and 22 more). If any are blocked, that engine cannot read your page. 30% of audited sites block at least one AI bot, often through legacy security plugin rules (Wordfence, Sucuri) that block unknown user agents by default. This is the single most common critical issue. If GPTBot is blocked, ChatGPT's 14.4% web search rate cannot reach you. If all 27 are blocked, you are invisible to every AI engine's live retrieval.

JavaScript-only content. The audit checks whether the main content is in the raw HTML or rendered client-side. If the page is an SPA that renders content via JavaScript, AI crawlers see an empty shell. The HN crawler study of 1,500 sites found 40% of marketing sites rely so heavily on client-side rendering that they appear as empty shells to non-hydrating AI agents. This is not a score issue. It is a visibility issue. If the crawler sees nothing, every other check is irrelevant.

Canonical URL missing or mismatched. The audit checks whether the page has a canonical URL and whether it matches the requested URL. The WashU study (55,393 queries) found that canonical correctness correlates with AI Overview citation. A missing or mismatched canonical means the AI engine may not attribute the content to the right URL, which means it may not cite the right page.

Tier 2: These checks matter but are not critical

These checks contribute to the score and align with what the data says about citation. A failure here means you are missing a signal, not that you are invisible.

Published and updated dates. The audit checks for visible dates on the page. The Babel42 study found that pages ignored by AI Overviews were thinner and less likely to have dates. The WashU study found dates correlate with citation. Missing dates cost you up to 4 points in the Google AI readiness category.

Named author byline. The audit checks for a named author. The WashU study found that credible sources with clear authorship are cited more. A missing byline costs you Google AI readiness points.

Schema markup (Organization, WebSite). 76% of audited sites ship some JSON-LD. The sites that do not are missing a signal that helps AI engines understand what the page is about. Organization and WebSite schema alone cover most of the value. Adding a sixth or seventh schema type has diminishing returns.

Meta description and title tag. Basic SEO. The audit checks for presence and quality. Missing these is table stakes that should have been fixed years ago, but 15-20% of audited sites still miss them.

Tier 3: These checks are honest about their limits

These checks are in the tool because they measure something real, but the calibration study showed they do not predict citation. They measure citability, not citation.

llms.txt presence and validity. Worth 6 points in rubric v2 (down from 18 in v1). 58% of audited sites have one. Google says it does nothing for AI Overviews. Some AI crawlers read it. The calibration study showed no relationship between llms.txt points and Perplexity citation. It is the easiest fix and the lowest-impact one. I keep it in the tool because it is a real signal for some crawlers, but I weight it honestly at 6 points.

Citation signals (quotable passages, statistics, quotes). The audit checks for content patterns that are easy for AI to extract: self-contained answer blocks, statistics with sources, quotable sentences. These align with the Babel42 finding that 67% of AI Overview quotes come from the first 30% of the page. But the calibration study showed no significant relationship between this category and citation. The content structure helps extractability. It does not guarantee citation.

Brand entity clarity. The audit checks whether the page has consistent entity references (same name, same description, same links) across schema, metadata, and content. This helps AI engines disambiguate the brand. The calibration study showed no significant relationship (p = 0.2227, Holm-corrected p = 1.0000). It is still a useful check for brand consistency, but it does not predict citation.

Tier 4: These checks are structural, not predictive

These checks exist because they measure something about the page's technical health. They do not predict citation, and they never claimed to.

AI discovery signals (RSS, sitemap references). Worth 3 points in rubric v2 (down from 6). These check whether the site has discovery mechanisms for AI crawlers. Minor signal. I keep it because it is a real technical check, but it is the lowest-weighted category.

Google AI readiness (8 sub-signals). Worth 20 points. This is the newest category, added in v4.19.0. It checks canonical, dates, byline, links, viewport, sitemap, snippets, and index. The calibration study showed the closest-to-significant result here (p = 0.1220 before Holm correction), trending in the wrong direction (OR = 0.8345). This category is based on the WashU study's findings about what correlates with AI Overview citation. The calibration study tested it against Perplexity, not Google. The relationship may be different on Google AI Overviews. I need to run the study again on Google before I can say whether this category predicts anything.

What I would remove

Nothing. Every check measures something real about the page. The question is how to weight them, not whether to keep them. The calibration study told me the weights do not predict citation. The next step is to run the study on Google AI Overviews, where the source selection mechanism is different from Perplexity.

If the Google study also comes back null, I will need to reconsider whether a 0-to-100 score is the right output. A checklist of "fix this, this, and this" may be more honest than a single number that implies predictive power it does not have.

Try it

If you want to see which tier your site falls into, run a free audit on geoready.dev. The score tells you where you are. The fix list tells you what to do. The tier ranking above tells you which fixes to prioritize.

The CLI is on GitHub, MIT licensed, 1,137 stars, 2,000+ tests. The full scoring rubric is in SCORING_RUBRIC.md. If you disagree with a tier ranking, open an issue. The data is open. The calibration study is open. The disagreement should be too.

Top comments (0)