DEV Community: RankCLI

Cloudflare May Be Blocking Googlebot On Your Site, And robots.txt Will Not Tell You

RankCLI — Wed, 30 Sep 2026 18:48:59 +0000

On 2026-09-15 Cloudflare changed what "blocked" means for AI traffic. New domains, new sites on existing accounts, and free-tier accounts that had not changed the setting now start with AI Training crawlers disallowed and AI Agents blocked on ad-bearing pages. Search was supposed to stay open.

It did not, for a lot of people.

The part that catches everyone: mixed-use crawlers

Googlebot crawls for Search and collects data that feeds AI training. Cloudflare classes it as mixed-use, and a mixed-use crawler is treated under whichever rule is most restrictive.

So if Training is blocked, including through the older "Block AI Bots" toggle, the block applies to all of Googlebot's functions on ad-supported pages. Googlebot starts getting 403 on pages and on sitemap fetches. The same applies to Bingbot and Applebot.

Your robots.txt still says Allow. Nothing in your repository changed. Search Console reports fetch failures and nobody can find the cause, because everyone looks at robots.txt first.

Cloudflare's own network data puts Googlebot's crawl share at 27.49% in Q2 2026, down from 57.20% a year earlier.

Why robots.txt checks cannot see this

This is the part worth internalising, because it applies well beyond Cloudflare.

robots.txt is a request. A CDN rule is enforcement. They live in different layers and they fail in different places. robots.txt is served by your origin and read by a crawler that chooses to obey it. A bot rule at the edge refuses the connection before your origin is ever consulted.

A crawler can be explicitly allowed in robots.txt and still receive a 403. Every "AI visibility" checker I know of, including ours until this week, stops at robots.txt and reports the site as open.

Checking it by hand

Two curl requests, one as a browser and one as Googlebot:

curl -s -o /dev/null -w "%{http_code}\\n" \\
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \\
  https://your-site.dev/

curl -s -o /dev/null -w "%{http_code}\\n" \\
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/131.0" \\
  https://your-site.dev/

Browser 200 and Googlebot 403 means an edge rule, not robots.txt. Check the sitemap the same way, because that is often where it shows first.

One honest caveat: real crawlers are verified by IP as well as user-agent, so a spoofed agent is indicative rather than conclusive. Confirm with Search Console's URL Inspection, which fetches as the verified Googlebot.

Or in one command

npx @rankcli/cli@latest audit -u https://your-site.dev

It asks as Googlebot, bingbot, GPTBot, ClaudeBot and PerplexityBot, compares each against a browser request, and reports two separate findings:

SEARCH_CRAWLER_BLOCKED_AT_EDGE, an error, when a search crawler is refused while a browser is served
AI_CRAWLER_BLOCKED_AT_EDGE, a warning, for the AI crawlers

Separate because the costs are different. Losing Googlebot costs you the index. Losing GPTBot costs you citations in AI answers. One is an emergency, the other is a decision you might have made deliberately.

No signup, nothing leaves your machine.

What it will not do

It will not report anything if a browser cannot fetch your site either. A site that is simply down is not a bot-rule problem, and turning one outage into six confident findings about rules that do not exist is exactly the kind of false alarm that teaches people to ignore their tools.

Tested against reddit.com, which refuses Googlebot, bingbot and GPTBot while serving browsers: it reports all three, split correctly. Tested against nytimes.com, which refuses browsers too: it reports nothing.

The fix, if you are affected

It is in your CDN, not your repository. On Cloudflare: zone Security → Bots, and any "Block AI Bots" or AI Crawler Control toggle. Because Googlebot is mixed-use, you need to allow Search crawlers explicitly or opt out of the default on ad-supported pages.

Existing paid customers were notified in advance and can opt out in zone security settings. New domains and free-tier accounts got the default without asking for it, which is why this keeps surprising people who never changed a setting.

Originally published at rankcli.dev.

Your SEO Fixes Now Land Where Your Framework Actually Serves Them

RankCLI — Mon, 28 Sep 2026 21:01:08 +0000

Five releases went out over two days. Here is what changed, and why each one exists.

Detection rebuilt: 57% to 94%

A robots.txt written to the wrong directory is worse than no robots.txt at all. The audit stops reporting it missing, while crawlers still never see it. So before an auto-fix can write a file, it has to know what it is looking at.

The old detector returned the first signal that matched. That is fast and wrong: signals conflict, and the first one to fire is not the most reliable one. It also matched on short substrings, which produced answers like classifying nginx as Gin because server contained "gin", and classifying any page with style="width: 50%" as Spring Boot because a Thymeleaf pattern matched th:.

Detection now weighs every signal on the page. Each contributes evidence at one of three strengths, the highest total wins, and a score below the floor returns unknown rather than a guess. On a 51-site corpus that moved accuracy from 57% to 94% — with no incorrect claims. When it names a framework, it is right; when it is unsure, it says so.

It recognises 32 frameworks now, including Docusaurus and Sphinx.

Fixes land in the right directory

Knowing the framework is only useful if you then act on it. The served directory is a property of the framework, not of whether you happen to have created it yet:

Framework	Static files go to
React, Vue, Next.js, Nuxt, Astro, Remix, Rails, Laravel	`public/`
SvelteKit, Hugo, Gatsby, Docusaurus, Fresh	`static/`
ASP.NET Core	`wwwroot/`
Phoenix	`priv/static/`
Spring Boot	`src/main/resources/static/`
Jekyll, Pelican	the app root itself

An earlier version looked for an existing public/ directory in your repository. That is right for a project that has one and wrong for every project that does not yet — a fresh Next.js app, a fresh Nuxt app and a Docusaurus site all resolved to the repository root, because the directory they serve from had not been created.

Monorepos work too: the app root is found by locating an index.html beside a package.json, so fixes land in the right package rather than at the top of the tree.

A safety layer that refuses to commit

Generated file edits are now checked before they become a commit, and anything carrying a concern is dropped rather than committed. Among what it refuses:

A replacement that lost a <script> tag, a stylesheet, or a search-console verification tag
Invalid JSON, XML or JSON-LD
A sitemap that came back shorter, or a lastmod moved backwards
Invented content — a phone number in the reserved 555 range, "123 Main Street, Anytown", an unfilled your_... placeholder
A canonical added to a single-page-app shell, which would make every route declare itself a duplicate of one URL

Those last two are there because a real generated pull request proposed both. We wrote that one up separately — it is the more interesting story.

Files RankCLI owns from a template — robots.txt, sitemap.xml, llms.txt — may be created but never rewritten by a model. A template cannot hallucinate a date; a model can, and did.

One-click install for Claude, Cursor and VS Code

The MCP server is the lowest-friction thing we ship: no signup, no API key, nothing leaves your machine. Every path we documented for it involved hand-editing a JSON config file, which is a strange way to introduce the easy option.

Claude Desktop now has a downloadable extension bundle — grab the .mcpb from the latest release and double-click it.

Claude Code is one line:

claude mcp add rankcli -- npx -y @rankcli/mcp-server

Cursor and VS Code have one-click install links on the docs page.

All of it drives the same 15 tools — full audits, AI-crawler access checks, structured data validation, Core Web Vitals, security headers, and generators for the fixes.

AI readiness that discriminates

The AI-readiness sub-score used to read near zero for essentially every site, including ones that do it well. A score that says zero for everyone looks like a harsh grader and is actually a broken instrument — it carried no information and could not be acted on.

It is recalibrated. Real sites now spread across a usable range and rank in a defensible order, so the number means something when it moves.

A tier ladder with no flat rung

Auto-fix PR limits now scale properly across plans. The cap on how many pull requests one audit report can open was a flat 3 on every plan — which made the top tier identical to the free one on the limit its customers actually hit, working through a single large report across many sites. It is 3 / 3 / 5 / 10 now, and a test asserts it keeps rising: a rung that does not rise is a rung that means nothing.

Try it

npx @rankcli/cli audit -u https://your-site.dev

No signup, no account, 280+ checks. The CLI, the MCP server and the GitHub Action are free forever.

Originally published at rankcli.dev.

We Ran Our SEO Tool On Ourselves And Found Five Bugs. One Would Have Deindexed Our Site.

RankCLI — Mon, 28 Sep 2026 20:45:48 +0000

We publish rankcli.dev's own audit score every week, whether it went up or not. This week we did something slightly different: instead of reading the number, we read every finding and asked whether it was actually true.

Five of them were not. Here they are, because the failures are more useful than the score.

1. A page that asked not to be indexed, faulted for not being indexed

For the same /login page, in the same run, the audit reported two things:

NOINDEX_TAG - correct, it carries <meta name="robots" content="noindex, nofollow"> on purpose
REACT_CSR_DETECTED, severity error: "Critical - your content is likely invisible to Google"

It knew the page had opted out of search, then raised a critical error because search could not see it.

Every SaaS has a noindex /login, /signup and /account. They are all client-rendered shells. So every one of our users with a login page was collecting a critical SEO error that nobody should ever act on.

Framework checks now carry an indexable_only flag, and 19 of them across 9 frameworks are marked with it. Deliberately not marked: render-blocking scripts, which is about performance, and hydration mismatches, which break the page for real users too. Those still fire on a noindex page, because they are still true there.

2. Every dark-themed page failed colour contrast

The contrast checker read only an element's own inline style. If it found no background, it assumed white.

Our /login fallback is #a1a1aa text painted on #0a0a0a. That is about 9:1 - comfortably past WCAG AA. We reported 2.56:1 and called it critical.

It now reads the background from the nearest ancestor that paints one, and when nothing up the tree paints anything it records the element as undetermined rather than measuring against a guess. Inline styles only, deliberately: resolving a real CSS cascade without a browser is a much larger job, and half-doing it would reintroduce exactly the confident guess we were removing.

3. Our AI readiness score was 1%. So was everyone's.

This is the one that stung, because AI readiness is the thing people come to us for.

We assumed our content was bad. Then we ran the same check against sites that ought to score well:

Site	AI Readiness (before)
stripe.com	13
vercel.com	7
cloudflare.com	0
docs.astro.build	0
rankcli.dev	1

A documentation site - the ideal target for AI citation - scored zero. The metric was saturated. It carried no information at all.

The cause was a category multiplier copied from categories with a tenth of the checks. Between 15 and 27 AI-readiness checks fire on a typical page, most of them aspirational content notices: no comparison table, no testimonials, no original data. At one point per notice and three per warning, multiplied by three, any real page blew past 100 points of deduction before the interesting checks were counted.

The same five pages now spread 52-71, and rank in a defensible order.

4. Whitespace could fail your build

--fail-on error exits 1. So an error in this tool is not a line in a report - it is somebody's broken build.

DOM_SIZE_EXCESSIVE counted every node the parser returns, while its thresholds (1,500 and 3,000) are Lighthouse's - and Lighthouse counts elements.

On tailwindcss.com that was 3,231 counted against 2,230 real elements. The difference: 796 whitespace nodes and 205 comments. Enough to push a warning-sized DOM over the error line.

Which means re-indenting a template, or leaving comments in your output, could have failed your build. It counts elements now, and a test asserts that formatting a page cannot change its severity.

5. Missing image dimensions was rated "your site is broken"

IMAGES_MISSING_DIMENSIONS fired at error for three or more images. That failed svelte.dev, tailwindcss.com, rust-lang.org, laravel.com and fastapi.tiangolo.com - five of the ten well-built sites we swept.

Missing width and height is a real CLS cost and we still report it. It is not "your site is broken". Two other places in our own codebase already treated the same condition as a notice and as info; error was the outlier, not the consensus.

And then the auto-fix PR tried to deindex us

While reviewing an open auto-fix pull request against our own repo before merging it, we found this:

<link rel="canonical" href="https://rankcli.dev/" />

added to the Vite index.html - the shared shell that serves every route. The file carries a comment three lines above the insertion point explaining precisely why that must never happen: Helmet cannot remove static tags, so every page would inherit it.

Merged, every page on the site - pricing, docs, blog - would have declared itself a duplicate of the homepage.

The same PR proposed Organization structured data with a phone number of +1-800-555-0199, an address of "123 Main Street, Anytown, CA 90210", and a social profile of youtube.com/channel/your_youtube_channel. None of it real. It also linked /images/og-image.png and /search, both of which 404.

We closed it. Then we fixed the generator: our safety layer now refuses placeholder content and a canonical added to a single-page-app shell, and any file carrying a safety concern is dropped rather than committed. We tested the new guards against that pull request's actual diff - all six problems are caught, while a real phone number in ordinary prose still passes.

What we actually learned

A confident wrong "critical" is worse than a missing check. A missing check costs you one finding. A false critical teaches people that the red ones are noise - and then the real one goes unread.

Saturated metrics are invisible failures. A score that reads zero for everyone looks like a harsh grader, not a broken instrument. The only way we caught it was running the same check against sites we already believed were good, and being surprised.

Generated fixes need a human read. Ours carry a disclaimer saying so. This week that disclaimer stopped being boilerplate.

The numbers, today

rankcli.dev scores 75/100 with 0 errors and AI Readiness 67, measured with the published CLI. Not because the site improved this week - it barely changed. Because the tool stopped lying about it.

You can run exactly the same audit, with no signup and no account:

npx @rankcli/cli audit -u https://your-site.dev

If it tells you something that is not true, we would genuinely like to know. That is how all five of these were found.

Originally published at rankcli.dev.

RankCLI weekly dogfood report — 2026-09-15

RankCLI — Tue, 15 Sep 2026 10:03:50 +0000

🤖 Weekly dogfood: rankcli.dev scored 88/100 this week (11 open issues) — audited by the same tool we ship. rankcli.dev

Read more or run your own audit: rankcli.dev

Is Your Site Even Visible to ChatGPT?

RankCLI — Sun, 13 Sep 2026 13:02:00 +0000

You've optimized for Google. Sitemap's clean, Core Web Vitals are green, structured data validates. But when someone asks ChatGPT or Perplexity about your product, you're nowhere in the answer — and a competitor with a worse site is.

The likely reason: the crawlers that feed AI answers aren't Googlebot, and they don't see your site the way Googlebot does.

Meet the other crawlers

GPTBot (OpenAI) — crawls for model training and, via a separate retrieval path, for real-time browsing/citation in ChatGPT answers.
ClaudeBot (Anthropic) — same idea, powers Claude's web search and citations.
PerplexityBot — crawls specifically to answer live queries with citations; this is the one most directly tied to "will I get mentioned."
Google-Extended — a separate opt-in signal from regular Googlebot, controls whether Google's AI features can use your content.

Each respects robots.txt, but as a distinct user-agent — blocking Googlebot doesn't block these, and blocking one doesn't block the others. Most sites have never explicitly considered any of them.

Why a site that ranks fine can still be invisible

1. robots.txt blocks it, often by accident. Some site generators and "AI protection" plugins add blanket Disallow rules for GPTBot/CCBot/etc. without the owner realizing it also kills citation eligibility, not just training-data scraping. Check yours:

curl -s https://yoursite.com/robots.txt | grep -A2 -i "gptbot\|claudebot\|perplexitybot"

2. The content is client-rendered only. This is the big one. Traditional SEO crawlers (Googlebot) execute JavaScript and wait for your SPA to render. Several AI crawlers fetch raw HTML and move on — no JS execution. If your content only exists after a React/Vue mount, an AI crawler may see an empty <div id="root"> and nothing else.

Test what a crawler actually receives, vs. what your browser renders:

curl -s -A "GPTBot" https://yoursite.com/ | grep -o '<title>.*</title>'

If that title tag (or the body content you care about) is missing or generic, but your browser shows it fine — that's your JS-rendering gap.

3. No structured data, or broken structured data. AI answer engines lean on schema.org JSON-LD to disambiguate what a page is actually about — product, article, FAQ, organization — rather than inferring it purely from prose. A page with solid content but zero structured data is harder for a retrieval system to confidently cite.

4. No llms.txt. An emerging, not-yet-universal convention — a plain-text file at /llms.txt giving a curated, LLM-readable summary of what your site is and where the important pages are, the same spirit as sitemap.xml but written for language models instead of search indexers. Not every crawler uses it yet, but it costs almost nothing to add and directly addresses the "can a model even summarize what we do" problem.

5. Stale noindex leftovers. Staging-environment meta tags (<meta name="robots" content="noindex">) that never got removed after launch block everything — traditional and AI crawlers alike.

A five-minute self-check

curl -A "GPTBot" yoursite.com/robots.txt — confirm you're not accidentally blocked.
curl -A "GPTBot" yoursite.com/ and diff it against what your browser renders — confirm the content is actually in the raw HTML.
Validate your structured data (Google's Rich Results Test works fine for this even though it's Google-branded — it's just a schema.org validator).
Check if /llms.txt exists. If not, it's a 20-minute add.
Grep your codebase for leftover noindex tags.

Automating this

We ran into this exact problem building RankCLI — the open MCP server (npx @rankcli/mcp-server, no signup) runs GEO checks for all of the above alongside the usual technical-SEO audit, so it's one pass instead of five manual curl commands every time you ship. Free tier is genuinely free; the hosted layer just adds scheduling and auto-fix PRs on top.

But even without any tool, the five checks above take less time than writing this sentence took, and most sites have never run them once.