<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: seotto.com</title>
    <description>The latest articles on DEV Community by seotto.com (@seotto).</description>
    <link>https://dev.to/seotto</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4217922%2Fbc898d99-dcba-488f-842d-42e8e24b69f4.png</url>
      <title>DEV Community: seotto.com</title>
      <link>https://dev.to/seotto</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/seotto"/>
    <language>en</language>
    <item>
      <title>Is structured data injected by JavaScript good enough, or must JSON-LD be in the server HTML? Let's argue about it</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Sat, 21 Nov 2026 13:33:09 +0000</pubDate>
      <link>https://dev.to/seotto/is-structured-data-injected-by-javascript-good-enough-or-must-json-ld-be-in-the-server-html-lets-2am9</link>
      <guid>https://dev.to/seotto/is-structured-data-injected-by-javascript-good-enough-or-must-json-ld-be-in-the-server-html-lets-2am9</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Structured data (usually a &lt;code&gt;&amp;lt;script type="application/ld+json"&amp;gt;&lt;/code&gt; block describing a product, article, FAQ or organisation) can get onto a page in two very different ways.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Server-side:&lt;/strong&gt; the JSON-LD is part of the HTML your server sends. View source, and it's there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Client-side:&lt;/strong&gt; the HTML arrives without it, and JavaScript adds it afterwards, from your app's components, or from a tag manager that marketing controls.&lt;/p&gt;

&lt;p&gt;Google's documentation says it can process structured data added with JavaScript, as long as the page renders properly. So is it fine to inject it, or should it always be in the server response? Let's argue about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for JavaScript-injected structured data
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Google handles it.&lt;/strong&gt; Googlebot renders pages with a recent version of Chrome. If the script tag exists after rendering, it gets read. Google even documents generating structured data with JavaScript and tag managers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's often the only practical option.&lt;/strong&gt; In a pure single-page app, everything is client-rendered anyway. Adding server-side rendering just for a JSON-LD block is a big project.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It decouples SEO from release cycles.&lt;/strong&gt; With a tag manager, an SEO specialist can add or fix structured data without waiting for a deploy. For teams where engineering time is scarce, that's a real advantage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The data is right there in the client.&lt;/strong&gt; Product price, availability and reviews are often fetched by the front end. Building JSON-LD from the same state guarantees the markup matches what's on screen.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for server HTML only
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rendering is a second, later step.&lt;/strong&gt; Google crawls the raw HTML first and renders later, when resources allow. Anything that only exists after rendering depends on that second step succeeding: scripts loading, APIs responding, no errors, within the renderer's time budget. Every one of those is a way to quietly lose the markup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not everyone renders.&lt;/strong&gt; Many other crawlers, link-preview bots and AI crawlers read only the raw HTML. If your structured data matters beyond Google (and for things like organisation info and product data, it increasingly does), client-side markup is invisible to a lot of consumers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tag managers drift.&lt;/strong&gt; JSON-LD defined in a tag manager isn't in version control, isn't reviewed with the code, and isn't tested by CI. The page template changes, the tag's selectors break, and the markup starts describing the wrong price, or nothing at all. These are some of the hardest regressions to notice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Mismatches are risky.&lt;/strong&gt; Structured data that disagrees with visible content (a price that's stale, a review count from yesterday) can lead to rich results being dropped or manual actions for misleading markup. Server-rendered markup built from the same data as the page is the easiest way to keep them in sync.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's harder to test.&lt;/strong&gt; Checking server HTML is a single &lt;code&gt;curl&lt;/code&gt;. Checking client-injected markup needs a headless browser, a wait condition and an assumption about how long rendering takes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;I'd draw the line like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Anything you depend on for rich results goes in the server HTML.&lt;/strong&gt; Product, Article, BreadcrumbList, Organization, Event, JobPosting. If losing it would cost you visible search features, render it on the server, from the same data as the page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Client-side injection is acceptable for experiments and gaps&lt;/strong&gt;, as long as someone owns it and it's on a list to move server-side if it proves useful.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tag-manager markup gets the same review as code.&lt;/strong&gt; Exported, in version control, with a named owner. Otherwise it's unmaintained code running on your production pages.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never do both for the same type.&lt;/strong&gt; A server-rendered &lt;code&gt;Product&lt;/code&gt; block plus a tag-manager &lt;code&gt;Product&lt;/code&gt; block on the same page gives crawlers two different descriptions of the same thing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test the raw HTML in CI.&lt;/strong&gt; A check that the JSON-LD block exists, parses and contains the expected &lt;code&gt;@type&lt;/code&gt; on one page per template catches most regressions before they ship.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Structured data tends to break silently: a template change, a tag manager edit, a renderer timeout.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; keeps checking pages after deploys for technical SEO regressions, so markup that disappeared from the HTML doesn't sit unnoticed for weeks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The parts I'm still unsure about
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Frameworks with streaming or partial hydration.&lt;/strong&gt; Some modern setups put JSON-LD in the initial HTML, some stream it later in the same response, some only add it on hydration. Is streamed-in-the-same-response "server HTML"? I think yes, but I'd test what each crawler actually receives.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prices that change by the minute.&lt;/strong&gt; If price changes faster than pages are cached, server markup can be stale too. Is a client-side update more accurate, or just inconsistent in a different way?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI crawlers.&lt;/strong&gt; It's not clear how much they use structured data, but they almost all read raw HTML only. If you care about that traffic, it's one more argument for the server side.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What about Google's own testing tools?
&lt;/h2&gt;

&lt;p&gt;The Rich Results Test and URL Inspection both render JavaScript, so client-side markup passes them. That's useful, but it can be misleading: a passing test tells you Google &lt;em&gt;can&lt;/em&gt; see the markup when rendering succeeds, not that every crawler sees it, or that rendering succeeds every time on every page. Treat a pass as "it works for Google in good conditions", and check the raw HTML for everything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick test for your site
&lt;/h2&gt;

&lt;p&gt;Pick one URL per template and compare two things:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://www.example.com/products/example &lt;span class="o"&gt;|&lt;/span&gt; grep -c &lt;span class="s1"&gt;'application/ld+json'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;against the count you see in the rendered page (DevTools Elements panel, or the URL Inspection tool's rendered HTML in Search Console). If the first number is lower, some of your structured data only exists after JavaScript runs, and now you know which side of this argument your site is on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/adding-a-paywall-or-login-wall-a-12-point-seo-checklist-so-search-engines-can-still-see-what-they-16n"&gt;Adding a paywall or login wall? A 12-point SEO checklist so search engines can still see what they need to&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/your-structured-data-broke-3-weeks-ago-and-nobody-noticed-monitoring-json-ld-with-a-script-2kb4"&gt;Your structured data broke 3 weeks ago and nobody noticed: monitoring JSON-LD with a script&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/view-source-vs-inspect-element-the-seo-bug-hiding-between-raw-and-rendered-html-4d71"&gt;View source vs Inspect element: the SEO bug hiding between raw and rendered HTML&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/csr-vs-ssr-vs-ssg-vs-isr-an-seo-comparison-without-the-framework-marketing-3eea"&gt;CSR vs SSR vs SSG vs ISR: an SEO comparison without the framework marketing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Where does your structured data come from: templates, components, or a tag manager? And has client-side markup ever silently stopped working for you?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>javascript</category>
      <category>webdev</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Adding a paywall or login wall? A 12-point SEO checklist so search engines can still see what they need to</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Sat, 21 Nov 2026 13:32:46 +0000</pubDate>
      <link>https://dev.to/seotto/adding-a-paywall-or-login-wall-a-12-point-seo-checklist-so-search-engines-can-still-see-what-they-16n</link>
      <guid>https://dev.to/seotto/adding-a-paywall-or-login-wall-a-12-point-seo-checklist-so-search-engines-can-still-see-what-they-16n</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Adding a paywall or a login wall is usually a product and business decision. The SEO consequences get discovered afterwards, when a section that used to bring in thousands of visitors a month quietly drops out of search.&lt;/p&gt;

&lt;p&gt;It doesn't have to go that way. Search engines handle paywalled content fine, as long as you're deliberate about what they can see and you tell them what's going on. Here's the checklist I'd hand to the developer building the wall.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide what should be in search
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Decide per section, not for the whole site.&lt;/strong&gt; Some pages should stay fully public (landing pages, docs overviews, pricing, the first articles in a series). Some should be indexed with a preview. Some, like account pages, shouldn't be in search at all. Write the list down before you build anything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Pick a model: preview, metered or hard wall.&lt;/strong&gt; A &lt;em&gt;preview&lt;/em&gt; shows the first part of the content to everyone. A &lt;em&gt;metered&lt;/em&gt; wall shows a number of full articles before asking for a login. A &lt;em&gt;hard&lt;/em&gt; wall shows nothing without an account. Each one needs a different technical setup, and hard walls generally get the least search visibility, because there's little for a search engine to rank.&lt;/p&gt;

&lt;h2&gt;
  
  
  Show crawlers the same thing you'd show a logged-out user (plus the hidden part, marked up)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;3. Don't redirect logged-out visitors to &lt;code&gt;/login&lt;/code&gt;.&lt;/strong&gt; If &lt;code&gt;/articles/deep-dive&lt;/code&gt; 302-redirects to the login page for anyone without a session, Googlebot gets redirected too, and every article URL looks like a login page. Serve the article URL with a 200, render the preview and the paywall prompt &lt;em&gt;on that page&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. If crawlers get the full text, mark up the paywalled part.&lt;/strong&gt; Google documents a specific structured data pattern for this: on the &lt;code&gt;NewsArticle&lt;/code&gt; or &lt;code&gt;Article&lt;/code&gt; (or other &lt;code&gt;CreativeWork&lt;/code&gt;), set &lt;code&gt;isAccessibleForFree&lt;/code&gt; to &lt;code&gt;false&lt;/code&gt;, and add a &lt;code&gt;hasPart&lt;/code&gt; with a &lt;code&gt;WebPageElement&lt;/code&gt; whose &lt;code&gt;cssSelector&lt;/code&gt; points at the paywalled section, also with &lt;code&gt;isAccessibleForFree: false&lt;/code&gt;. That tells Google the difference between what users and crawlers see is a paywall, not cloaking.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"@context"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://schema.org"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Article"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"headline"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"The deep dive"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"isAccessibleForFree"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"hasPart"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"WebPageElement"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"isAccessibleForFree"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"cssSelector"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;".paywalled"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;5. Verify crawlers properly if you serve them more.&lt;/strong&gt; Serving full text only to Googlebot means checking that requests really come from Googlebot, via reverse DNS or Google's published IP ranges. A user agent check alone lets anyone read your content with a browser extension.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Know the trade-offs of client-side paywalls.&lt;/strong&gt; A paywall that hides content with JavaScript after the full text is in the HTML is easy for search engines, and also easy for users to bypass with "view source". A server-side paywall that only sends the preview is secure but means crawlers see only the preview, unless you do item 5. Choose knowingly.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;One misconfigured paywall rule can turn every article into a redirect to /login for crawlers.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; checks for indexing blocks and deploy regressions like these, so a wall that hides a whole section from search gets caught on the day it ships.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Keep the basics intact
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;7. Keep titles, descriptions, headings and canonicals on the paywalled page.&lt;/strong&gt; These go in the part everyone sees. A paywalled article with a generic title like "Subscribe to read" will rank for nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Keep internal links visible.&lt;/strong&gt; Navigation, related-article links and breadcrumbs should render for logged-out users. Paywall components sometimes replace the whole page body, links included, which cuts the section off from crawling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;9. Make account and checkout pages noindex, not disallowed.&lt;/strong&gt; &lt;code&gt;/account/&lt;/code&gt;, &lt;code&gt;/billing/&lt;/code&gt; and &lt;code&gt;/checkout/&lt;/code&gt; should carry &lt;code&gt;noindex&lt;/code&gt;. Blocking them in robots.txt instead means crawlers can't see that tag, and the URLs can still appear in results from links.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;10. Watch caching.&lt;/strong&gt; If the page varies by login state, make sure your CDN doesn't cache a logged-in version and serve it to everyone, or cache the login redirect and serve it to Googlebot. &lt;code&gt;Cache-Control: private&lt;/code&gt; for personalised responses, or cache keys that include the session state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verify
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;11. Test as a logged-out crawler, from outside.&lt;/strong&gt; Fetch a few paywalled URLs with curl and a crawler user agent, and use Search Console's URL Inspection live test. Check the status (200, not 302), that the title and preview are in the HTML, and that the structured data parses.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;12. Watch the numbers for a month.&lt;/strong&gt; In Search Console, filter the performance report to the paywalled section and compare impressions before and after. A drop in the first week usually means a technical issue (redirects, noindex, missing links). A slower decline may simply be the cost of less visible content, which is a business question, not a bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  A note on metered walls and first visits
&lt;/h2&gt;

&lt;p&gt;Metered walls usually count articles with a cookie. Crawlers don't keep cookies between requests, so they always look like a first-time visitor, which is fine, as long as the meter logic never treats "no cookie" as "over the limit". Test that path explicitly: a request with no cookies at all should get the full article (or the preview you intend), never the hard wall. The same goes for privacy-focused browsers that clear cookies, which is a good reason to make the no-cookie experience sensible anyway.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Decide which sections stay public, preview, metered or hidden.&lt;/li&gt;
&lt;li&gt;No redirects to login on article URLs.&lt;/li&gt;
&lt;li&gt;Paywalled-content structured data if crawlers get more than users.&lt;/li&gt;
&lt;li&gt;Verify Googlebot properly, never by user agent alone.&lt;/li&gt;
&lt;li&gt;Keep titles, links and canonicals visible to everyone.&lt;/li&gt;
&lt;li&gt;Test logged-out, from outside, then watch the section's impressions.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-tell-real-googlebot-from-fakes-in-your-access-logs-reverse-dns-and-ip-ranges-in-python-44n9"&gt;How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/javascript-seo-faq-11-questions-frontend-devs-ask-with-straight-answers-1722"&gt;JavaScript SEO FAQ: 11 questions frontend devs ask (with straight answers)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/out-of-stock-discontinued-or-gone-forever-an-seo-checklist-for-product-pages-developers-actually-1cn7"&gt;Out of stock, discontinued, or gone forever? An SEO checklist for product pages developers actually own&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/is-structured-data-injected-by-javascript-good-enough-or-must-json-ld-be-in-the-server-html-lets-2am9"&gt;Is structured data injected by JavaScript good enough, or must JSON-LD be in the server HTML? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you added a paywall or login wall to a site that relied on search traffic? What happened to the traffic, and what would you do differently?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>javascript</category>
      <category>security</category>
    </item>
    <item>
      <title>How to catch SEO regressions before they ship: diff titles, canonicals and robots tags between production and a preview deploy (Python, for CI)</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Fri, 20 Nov 2026 09:45:36 +0000</pubDate>
      <link>https://dev.to/seotto/how-to-catch-seo-regressions-before-they-ship-diff-titles-canonicals-and-robots-tags-between-504h</link>
      <guid>https://dev.to/seotto/how-to-catch-seo-regressions-before-they-ship-diff-titles-canonicals-and-robots-tags-between-504h</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The SEO incidents that hurt most are rarely strategy mistakes. They're one-line template changes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A &lt;code&gt;noindex&lt;/code&gt; that was added on staging and merged with everything else.&lt;/li&gt;
&lt;li&gt;A canonical URL built from an environment variable that's wrong in production.&lt;/li&gt;
&lt;li&gt;A layout refactor that drops the &lt;code&gt;&amp;lt;h1&amp;gt;&lt;/code&gt; or the meta description from every product page.&lt;/li&gt;
&lt;li&gt;Structured data that disappears because a component stopped rendering server-side.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each of these is easy to spot if you compare the page before and after the change. So let's compare them automatically, on every pull request, against the preview deploy most hosting platforms already give you.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the script compares
&lt;/h2&gt;

&lt;p&gt;For each path, it fetches production and preview and extracts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;HTTP status, and the final URL after redirects&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;&amp;lt;title&amp;gt;&lt;/code&gt; and meta description&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;rel="canonical"&lt;/code&gt; (normalised so the preview host doesn't count as a difference)&lt;/li&gt;
&lt;li&gt;meta robots and the &lt;code&gt;X-Robots-Tag&lt;/code&gt; header&lt;/li&gt;
&lt;li&gt;the first &lt;code&gt;&amp;lt;h1&amp;gt;&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;the number of hreflang alternates&lt;/li&gt;
&lt;li&gt;the &lt;code&gt;@type&lt;/code&gt; values in JSON-LD blocks&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then it reports every difference, and exits non-zero for the dangerous ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;#!/usr/bin/env python3&lt;/span&gt;
&lt;span class="s2"&gt;"""Compare SEO-relevant tags between production and a preview deploy."""&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parser&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;HTMLParser&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urlsplit&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="no"&gt;PROD&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;PREVIEW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="no"&gt;PATHS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="no"&gt;FATAL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"robots"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"x_robots"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"canonical"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Head&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;HTMLParser&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"hreflang"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"jsonld"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_in&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_buf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_starttag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attrs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attrs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"meta"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"meta"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"robots"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"robots"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"canonical"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"canonical"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"href"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;"/"&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"hreflang"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"hreflang"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"h1"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"script"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"application/ld+json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_in&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_buf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_in&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_buf&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_endtag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_in&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"script"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_buf&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
                    &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"jsonld"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"jsonld"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"INVALID"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"\s+"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_buf&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_in&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"seo-diff/1.0"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Head&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;feed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"final_path"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"x_robots"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"X-Robots-Tag"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"jsonld"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"jsonld"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;

&lt;span class="n"&gt;fatal&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;PATHS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;PROD&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;PREVIEW&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"FATAL"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;FATAL&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"warn "&lt;/span&gt;
            &lt;span class="n"&gt;fatal&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"FATAL"&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{level} {path}  {key}: {a.get(key)!r} -&amp;gt; {b.get(key)!r}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{len(PATHS)} paths compared, {fatal} fatal difference(s)"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fatal&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Usage, with a file of representative paths:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/
/pricing/
/blog/
/blog/some-popular-post/
/docs/getting-started/
/products/example-product/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 seo_diff.py https://www.example.com https://pr-142.preview.example.com paths.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FATAL /products/example-product/  robots: None -&amp;gt; 'noindex,nofollow'
warn  /blog/some-popular-post/  description: 'How we cut build times...' -&amp;gt; None
2 paths compared, 1 fatal difference(s)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Why it's built this way
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Compare two live environments, not a snapshot.&lt;/strong&gt; A stored baseline goes stale every time content changes. Comparing production with the preview for the same commit isolates what &lt;em&gt;this change&lt;/em&gt; did.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Canonicals compared by path.&lt;/strong&gt; The preview's canonical will (and should) differ in host if it's built from the request. Comparing the path catches the real bug, a canonical pointing at a different page, without flagging every preview. Add a separate check that production canonicals use the production host.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fatal vs warning.&lt;/strong&gt; A changed title might be intentional. A new &lt;code&gt;noindex&lt;/code&gt;, a status code change or a different canonical almost never is on a pull request that wasn't about SEO. Make the dangerous ones fail the build and the others visible.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A short list of representative paths.&lt;/strong&gt; One URL per template type (home, listing, article, product, docs page, landing page) catches most template regressions in seconds. You don't need to crawl the whole site on every PR.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A CI diff catches what a pull request changes; production can still drift after a CMS edit or a CDN rule.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; keeps checking the live site for indexing blocks and SEO regressions after deploy, so the changes that never went through a pull request get caught too.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Things to watch out for
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Preview deploys are often noindexed on purpose&lt;/strong&gt;, with an &lt;code&gt;X-Robots-Tag: noindex&lt;/code&gt; header from the platform. You'll see this as a fatal difference on every path. Either strip that one header from the comparison when the host is a preview, or have the platform skip the header for requests with a specific user agent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Personalised or A/B-tested pages&lt;/strong&gt; can differ between two fetches of the same environment. Exclude them or pin the variant with a cookie.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Client-rendered tags won't be seen.&lt;/strong&gt; The script reads server HTML, which is deliberate: it's what crawlers get first. If your titles only exist after JavaScript runs, that's a finding in itself.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Wiring it into CI
&lt;/h2&gt;

&lt;p&gt;Run it as a job after the preview deploy finishes, passing the preview URL from your platform's deployment event. Post the output as a PR comment when there are warnings, and fail the check on fatal differences. It takes a few seconds, and it's the cheapest SEO insurance I know.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-duplicate-google-chose-different-canonical-than-user-in-search-console-and-when-to-4b1i"&gt;How to fix 'Duplicate, Google chose different canonical than user' in Search Console (and when to let Google win)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/make-title-and-meta-description-required-props-if-a-page-can-ship-without-them-it-will-509f"&gt;Make title and meta description required props. If a page can ship without them, it will.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/quick-tip-a-tiny-script-that-prints-the-title-canonical-and-robots-tags-of-any-url-so-you-stop-1500"&gt;Quick tip: a tiny script that prints the title, canonical and robots tags of any URL (so you stop opening view-source)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/your-structured-data-broke-3-weeks-ago-and-nobody-noticed-monitoring-json-ld-with-a-script-2kb4"&gt;Your structured data broke 3 weeks ago and nobody noticed: monitoring JSON-LD with a script&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/find-duplicate-and-missing-title-tags-with-a-python-crawl-script-4cfp"&gt;Find duplicate and missing title tags with a Python crawl script&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the worst SEO regression that made it to production on your site? I'd bet a good share of them would have shown up in a diff like this.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>python</category>
      <category>testing</category>
      <category>cicd</category>
    </item>
    <item>
      <title>How to fix 'Duplicate, Google chose different canonical than user' in Search Console (and when to let Google win)</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Fri, 20 Nov 2026 09:45:09 +0000</pubDate>
      <link>https://dev.to/seotto/how-to-fix-duplicate-google-chose-different-canonical-than-user-in-search-console-and-when-to-4b1i</link>
      <guid>https://dev.to/seotto/how-to-fix-duplicate-google-chose-different-canonical-than-user-in-search-console-and-when-to-4b1i</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You add a canonical tag. It points where you want. And Search Console's page indexing report says:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Duplicate, Google chose different canonical than user&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In other words: Google saw your &lt;code&gt;rel="canonical"&lt;/code&gt;, considered it, and picked a different URL as the representative version of this content. The page you inspected won't be shown in search; the one Google chose will.&lt;/p&gt;

&lt;p&gt;This feels like Google ignoring you. It's more accurate to say your site is sending mixed signals, and the canonical tag lost the vote.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Google chooses a canonical
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;rel="canonical"&lt;/code&gt; is one signal among several. Google also looks at:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Redirects&lt;/strong&gt;: if URL A redirects to URL B, B is a strong candidate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal links&lt;/strong&gt;: which version do your own pages link to?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sitemaps&lt;/strong&gt;: which version is listed?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HTTPS vs HTTP&lt;/strong&gt;: HTTPS is preferred.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hreflang annotations&lt;/strong&gt;: which URLs reference each other?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Content similarity&lt;/strong&gt;: are the pages really duplicates, or just similar?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;URL cleanliness&lt;/strong&gt;: shorter URLs without parameters tend to win ties.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When most of these point at one URL and your canonical tag points at another, Google goes with the majority.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: find out which URL Google chose
&lt;/h2&gt;

&lt;p&gt;Use the &lt;strong&gt;URL Inspection&lt;/strong&gt; tool on the affected URL. Under &lt;em&gt;Page indexing&lt;/em&gt;, it shows two values:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;User-declared canonical&lt;/strong&gt;: what your tag says.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google-selected canonical&lt;/strong&gt;: what Google picked.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Write down both. The fix is always about understanding why the second one looks more canonical than the first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: compare the two URLs
&lt;/h2&gt;

&lt;p&gt;Fetch both and compare what you're telling Google about each:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; u &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com/shoes/red"&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com/shoes/red?color=red"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"== $u"&lt;/span&gt;
  curl -sIL &lt;span class="s2"&gt;"$u"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -iE &lt;span class="s1"&gt;'^(HTTP|location|link)'&lt;/span&gt;
  curl -sL &lt;span class="s2"&gt;"$u"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -ioE &lt;span class="s1"&gt;'&amp;lt;link[^&amp;gt;]+rel="canonical"[^&amp;gt;]*&amp;gt;|&amp;lt;meta[^&amp;gt;]+name="robots"[^&amp;gt;]*&amp;gt;'&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then check:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Does each page's canonical tag agree with the other's? If page A says "canonical is A" and page B says "canonical is B", for near-identical content, Google has to pick one, and it might not pick yours.&lt;/li&gt;
&lt;li&gt;Which one is in your sitemap?&lt;/li&gt;
&lt;li&gt;Which one do your navigation, breadcrumbs and in-content links use? Search your templates for the URL pattern.&lt;/li&gt;
&lt;li&gt;Does one of them redirect to the other in some cases (with or without a trailing slash, with &lt;code&gt;www&lt;/code&gt;, with a parameter)?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The usual causes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Internal links point at the "wrong" version.&lt;/strong&gt; The most common one. Your canonical says &lt;code&gt;/shoes/red&lt;/code&gt;, but category pages link to &lt;code&gt;/shoes/red?ref=nav&lt;/code&gt; or &lt;code&gt;/shoes/Red&lt;/code&gt;. Internal links are a strong signal. Fix the links, not just the tag.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. The sitemap lists a different URL.&lt;/strong&gt; Sitemaps generated from the database often include parameter variants or old slugs. Every sitemap URL should be the canonical version, exactly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Pages that aren't really duplicates are canonicalised together.&lt;/strong&gt; If you canonicalise all colour variants to one product page, but each variant has unique content, Google may decide the variant pages are distinct, or pick one of the variants as the representative. Canonicals should join genuinely duplicate content.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. The canonical points to a page that redirects, 404s or is noindexed.&lt;/strong&gt; Google won't use a canonical target it can't index. Check that the target returns 200, is indexable, and itself carries a self-referencing canonical.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Canonical tags added by JavaScript, or conflicting tags.&lt;/strong&gt; If the server HTML has one canonical and JavaScript changes it, or the page has two canonical tags (one from a plugin, one from the theme), Google may ignore both.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Cross-domain duplication.&lt;/strong&gt; If your content is syndicated or scraped, Google may choose the other domain's copy. Ask syndication partners to add a canonical back to your original, and make sure your page was published first and is easy to crawl.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Canonical problems usually start with a template or sitemap change, long before Search Console reports them.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; watches for canonical and indexing inconsistencies after you deploy, so mixed signals get caught while they're still one commit old.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 3: line up the signals
&lt;/h2&gt;

&lt;p&gt;Pick the canonical you want, then make every signal agree with it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Self-referencing canonical on the chosen URL; canonical to it on every duplicate.&lt;/li&gt;
&lt;li&gt;Only the chosen URL in the sitemap.&lt;/li&gt;
&lt;li&gt;All internal links use the chosen URL, exactly, including case and trailing slash.&lt;/li&gt;
&lt;li&gt;Where duplicates have no reason to exist (case variants, old slugs, tracking parameters), 301-redirect them instead of relying on canonical tags.&lt;/li&gt;
&lt;li&gt;Then request reindexing of the chosen URL in URL Inspection, and wait. Canonical selection updates as pages are recrawled, which can take a few weeks.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  When to let Google win
&lt;/h2&gt;

&lt;p&gt;Sometimes Google's choice is fine, or even better:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If Google chose the HTTPS version over an HTTP canonical you forgot to update, fix your tag to match.&lt;/li&gt;
&lt;li&gt;If it chose a cleaner URL over a parameterised one, consider making the clean URL your canonical.&lt;/li&gt;
&lt;li&gt;If the "duplicate" really is a duplicate and Google's choice is the one users would prefer, there's no ranking penalty for letting Google pick. The content still shows up in search; just under the other URL.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The status is a warning, not an error. It matters when Google picks a URL you don't want users landing on: a filtered view, a print version, a staging host, or a page in the wrong language.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/canonical-tags-faq-10-questions-developers-ask-about-relcanonical-with-straight-answers-3hij"&gt;Canonical tags FAQ: 10 questions developers ask about rel=canonical (with straight answers)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/there-is-no-duplicate-content-penalty-but-duplicate-content-still-costs-you-30f5"&gt;There is no duplicate content penalty (but duplicate content still costs you)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/case-study-hypothetical-the-staging-site-that-got-indexed-and-started-outranking-production-41oi"&gt;Case study (hypothetical): the staging site that got indexed, and started outranking production&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-discovered-currently-not-indexed-in-google-search-console-a-developers-list-of-5731"&gt;How to fix 'Discovered – currently not indexed' in Google Search Console (a developer's list of causes)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-catch-seo-regressions-before-they-ship-diff-titles-canonicals-and-robots-tags-between-504h"&gt;How to catch SEO regressions before they ship: diff titles, canonicals and robots tags between production and a preview deploy (Python, for CI)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the most surprising URL you've seen Google pick as canonical? Mine was a &lt;code&gt;?utm_source=newsletter&lt;/code&gt; version that had more internal links than the clean URL, thanks to a template bug.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>beginners</category>
    </item>
    <item>
      <title>Should you noindex your tag and category pages? Let's argue about it</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Thu, 19 Nov 2026 22:10:10 +0000</pubDate>
      <link>https://dev.to/seotto/should-you-noindex-your-tag-and-category-pages-lets-argue-about-it-523k</link>
      <guid>https://dev.to/seotto/should-you-noindex-your-tag-and-category-pages-lets-argue-about-it-523k</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every blog, docs site and content-heavy app ends up with archive pages: &lt;code&gt;/tags/kubernetes/&lt;/code&gt;, &lt;code&gt;/category/tutorials/&lt;/code&gt;, &lt;code&gt;/topics/payments/page/3/&lt;/code&gt;. Content management systems create them automatically, often hundreds of them.&lt;/p&gt;

&lt;p&gt;And there's a long-running argument about what to do with them in search. One camp adds &lt;code&gt;noindex&lt;/code&gt; to every tag and category page as a matter of hygiene. The other says those pages are some of the most useful on the site and should be indexed like anything else.&lt;/p&gt;

&lt;p&gt;Let's argue about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for noindex
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Most tag pages are thin.&lt;/strong&gt; A tag used on one or two posts produces a page with a heading and two excerpts. There's nothing on it that isn't better on the posts themselves. Multiply by every tag someone made up once (&lt;code&gt;#thoughts&lt;/code&gt;, &lt;code&gt;#misc&lt;/code&gt;, &lt;code&gt;#2021&lt;/code&gt;) and you have hundreds of low-value pages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;They duplicate each other.&lt;/strong&gt; &lt;code&gt;/tags/react/&lt;/code&gt;, &lt;code&gt;/tags/reactjs/&lt;/code&gt; and &lt;code&gt;/category/frontend/&lt;/code&gt; may list almost the same posts. Paginated archives add more near-duplicates: page 2 of one tag looks a lot like page 2 of another.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;They compete with real pages.&lt;/strong&gt; A tag page called "Kubernetes" can rank, or try to rank, for "kubernetes" ahead of your actual guide to Kubernetes, which is the page you want people to land on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;They use crawl attention.&lt;/strong&gt; On big sites, thousands of archive pages and their paginated variants can soak up a lot of crawling, especially when combined with sort orders and filters.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for indexing them
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;They're hubs.&lt;/strong&gt; A well-built category page links to every post on a topic. That's exactly the kind of internal linking that helps crawlers discover and understand content. And &lt;code&gt;noindex&lt;/code&gt; pages tend to get crawled less over time, which can mean the links on them get followed less too.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;They match broad searches.&lt;/strong&gt; Someone searching "kubernetes tutorials" is better served by a page listing your 15 Kubernetes tutorials than by any single one. Good category pages are genuine landing pages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Noindex doesn't fix the underlying mess.&lt;/strong&gt; If you have 400 tags, the problem is the 400 tags, not their indexing status. Noindexing them hides the problem from search while leaving it for users.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's easy to overdo.&lt;/strong&gt; Teams that add &lt;code&gt;noindex&lt;/code&gt; to "all archive templates" sometimes catch pages they didn't mean to, like a docs section index or the main blog listing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;I think the answer is "it depends on the page", and a rule that's simple to automate:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Categories (a small, curated set) are indexed, and treated as real pages.&lt;/strong&gt; Give each one an introduction that explains what the section covers, link to the best posts first rather than only newest-first, and keep them in the main navigation. A category page should be worth landing on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tags are indexed only above a threshold.&lt;/strong&gt; A tag with fewer than, say, five posts gets &lt;code&gt;noindex, follow&lt;/code&gt; (crawlers can still follow the links, though as the page drops out of crawling that matters less). Tags above the threshold get the same treatment as categories: a description and curated ordering.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Merge duplicates rather than noindexing them.&lt;/strong&gt; &lt;code&gt;react&lt;/code&gt; and &lt;code&gt;reactjs&lt;/code&gt; should become one tag, with a redirect from the other. That's better than two noindexed pages.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paginated pages stay indexable but plain.&lt;/strong&gt; Self-referencing canonicals on each page (page 2 canonical to page 2, not to page 1), real links between pages, and no &lt;code&gt;noindex&lt;/code&gt; on page 2 and beyond, which would hide older posts from discovery.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never block archive pages in robots.txt&lt;/strong&gt; if you also want &lt;code&gt;noindex&lt;/code&gt; to work. A crawler that can't fetch the page can't see the &lt;code&gt;noindex&lt;/code&gt; tag, and the URL can still end up indexed from links.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Archive templates are where a single template change can noindex hundreds of pages at once.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; checks for indexing blocks and deploy regressions, so a &lt;code&gt;noindex&lt;/code&gt; that lands on the wrong template gets caught before those pages drop out of search.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The parts I'm still unsure about
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Where the threshold is.&lt;/strong&gt; Five posts? Ten? It probably depends on how much each post's excerpt adds. I've seen good arguments for "it should have at least as much unique text as an average post".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Date archives.&lt;/strong&gt; &lt;code&gt;/2024/03/&lt;/code&gt; pages almost never match a search. I'd noindex them or remove them, but some people use them for navigation and like them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Author pages.&lt;/strong&gt; On multi-author sites they matter for trust and for people searching an author's name. On single-author sites they duplicate the blog index.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Faceted category pages in e-commerce.&lt;/strong&gt; "Red shoes, size 42" is a very different problem from blog tags, with its own trade-offs. I'd keep that argument separate.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  One more thing: noindex is slow to undo
&lt;/h2&gt;

&lt;p&gt;Whatever you decide, decide it once. A page that's been noindexed for months gets crawled less and less often, so when you remove the tag, it can take weeks for Google to come back, notice, and index it again. Flip-flopping between "index tags" and "noindex tags" every few months is the worst of both worlds: you pay the recovery cost each time and never see a stable result.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick audit
&lt;/h2&gt;

&lt;p&gt;Export every tag and category URL with its post count, then check three things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How many have fewer than five posts? Those are candidates for merging or &lt;code&gt;noindex&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;How many pairs list mostly the same posts? Candidates for merging.&lt;/li&gt;
&lt;li&gt;Do any get search traffic today? Search Console's performance report, filtered by page, tells you. Don't noindex a tag page that already brings in visitors without a very good reason.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The answer is rarely "all" or "none". It's usually "about 20 of these are good pages, and the other 300 shouldn't exist".&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-couldnt-fetch-and-sitemap-could-not-be-read-in-google-search-console-a-developers-30cf"&gt;How to fix 'Couldn't fetch' and 'Sitemap could not be read' in Google Search Console (a developer's checklist)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/should-every-page-have-a-meta-description-lets-argue-about-it-3gkm"&gt;Should every page have a meta description? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/your-site-search-creates-infinite-urls-an-seo-checklist-for-internal-search-result-pages-2b3e"&gt;Your site search creates infinite URLs: an SEO checklist for internal search result pages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/there-is-no-duplicate-content-penalty-but-duplicate-content-still-costs-you-30f5"&gt;There is no duplicate content penalty (but duplicate content still costs you)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What do you do with tag pages on your site? And has noindexing (or un-noindexing) them ever made a visible difference to your traffic?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>blogging</category>
      <category>discuss</category>
    </item>
    <item>
      <title>How to fix 'Couldn't fetch' and 'Sitemap could not be read' in Google Search Console (a developer's checklist)</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Thu, 19 Nov 2026 22:09:42 +0000</pubDate>
      <link>https://dev.to/seotto/how-to-fix-couldnt-fetch-and-sitemap-could-not-be-read-in-google-search-console-a-developers-30cf</link>
      <guid>https://dev.to/seotto/how-to-fix-couldnt-fetch-and-sitemap-could-not-be-read-in-google-search-console-a-developers-30cf</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You submit &lt;code&gt;https://www.example.com/sitemap.xml&lt;/code&gt; in Search Console, and the status column says &lt;strong&gt;Couldn't fetch&lt;/strong&gt;. Or, a little later, &lt;strong&gt;Sitemap could not be read&lt;/strong&gt;. You click the URL, it opens fine in your browser, and you're left wondering what Google is seeing that you aren't.&lt;/p&gt;

&lt;p&gt;First, some reassurance: "Couldn't fetch" right after submitting is sometimes just Search Console's status not having caught up. Google may not have attempted the fetch yet. If the sitemap is new, wait a day before debugging.&lt;/p&gt;

&lt;p&gt;If it's still there after a day or two, here's how to find the actual cause.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: fetch it the way Googlebot does
&lt;/h2&gt;

&lt;p&gt;Your browser has cookies, a normal user agent and probably a session that passed a bot challenge hours ago. Googlebot has none of those. Request the sitemap with curl, using Googlebot's user agent, from a machine outside your network:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sSL -D - -o /tmp/sitemap.xml \
  -A &lt;span class="s2"&gt;"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"&lt;/span&gt; \
  https://www.example.com/sitemap.xml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at the headers it prints. You want:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A final status of &lt;code&gt;200&lt;/code&gt;, ideally with no redirects before it.&lt;/li&gt;
&lt;li&gt;A &lt;code&gt;Content-Type&lt;/code&gt; of &lt;code&gt;application/xml&lt;/code&gt; or &lt;code&gt;text/xml&lt;/code&gt; (or a gzip type for &lt;code&gt;.xml.gz&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;A body that starts with &lt;code&gt;&amp;lt;?xml&lt;/code&gt; and contains &lt;code&gt;&amp;lt;urlset&lt;/code&gt; or &lt;code&gt;&amp;lt;sitemapindex&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then validate the XML itself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;xmllint --noout /tmp/sitemap.xml &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"well-formed"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Most problems show up in one of these three checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  The usual causes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. A firewall or bot protection is blocking Googlebot.&lt;/strong&gt; A WAF rule, a "bot fight" mode or a rate limit may let browsers through and challenge everything else. If curl gets a 403, a 429, or an HTML challenge page instead of XML, that's it. Check your WAF logs for requests to &lt;code&gt;/sitemap.xml&lt;/code&gt; with Googlebot's user agent. Allow verified Googlebot traffic rather than whitelisting the user agent string, which anyone can fake.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Robots.txt blocks the sitemap.&lt;/strong&gt; It sounds silly, but a broad &lt;code&gt;Disallow: /*.xml$&lt;/code&gt; or a rule that blocks a &lt;code&gt;/sitemaps/&lt;/code&gt; folder will stop Google fetching the file. Check with Search Console's robots.txt report, or simply read the rules that match the sitemap's path.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Redirects.&lt;/strong&gt; A sitemap at &lt;code&gt;http://&lt;/code&gt; that redirects to &lt;code&gt;https://&lt;/code&gt;, or &lt;code&gt;example.com&lt;/code&gt; to &lt;code&gt;www.example.com&lt;/code&gt;, usually works, but submit the final URL anyway. Long chains, redirects to a login page, or a redirect to an HTML page are common causes of "could not be read".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. It's not actually XML.&lt;/strong&gt; A framework error page, a "maintenance" page or a soft 404 served with status 200 looks like a successful fetch but isn't a sitemap. So does a sitemap route that renders through the site's HTML layout by mistake, wrapping the XML in &lt;code&gt;&amp;lt;html&amp;gt;&lt;/code&gt;. The body check above catches all of these.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Malformed XML.&lt;/strong&gt; Unescaped ampersands in URLs are the classic: &lt;code&gt;?a=1&amp;amp;b=2&lt;/code&gt; must be written as &lt;code&gt;?a=1&amp;amp;amp;b=2&lt;/code&gt; inside XML. Also watch for invalid characters copied from content, a byte order mark or whitespace before &lt;code&gt;&amp;lt;?xml&lt;/code&gt;, and the wrong namespace on &lt;code&gt;&amp;lt;urlset&amp;gt;&lt;/code&gt; (it must be &lt;code&gt;http://www.sitemaps.org/schemas/sitemap/0.9&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Gzip confusion.&lt;/strong&gt; A &lt;code&gt;.xml.gz&lt;/code&gt; file served with &lt;code&gt;Content-Encoding: gzip&lt;/code&gt; &lt;em&gt;and&lt;/em&gt; gzipped content is double-compressed from the client's point of view. Either serve a gzipped file as a download (&lt;code&gt;Content-Type: application/gzip&lt;/code&gt;, no &lt;code&gt;Content-Encoding&lt;/code&gt;), or serve plain XML and let the server compress the transfer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Too big.&lt;/strong&gt; A single sitemap may contain at most 50,000 URLs and be at most 50 MB uncompressed. Bigger than that, split it and use a sitemap index.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. It's slow or times out.&lt;/strong&gt; Sitemaps generated on the fly from the database can take many seconds on big sites, especially uncached. Googlebot won't wait forever. Generate sitemaps ahead of time, or cache them aggressively.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A sitemap that worked last week can start failing after a WAF rule or a framework upgrade, and Search Console takes days to tell you.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; checks for crawler-facing problems like blocked or broken sitemaps after you deploy, so you find out the same day.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 2: check the URLs inside
&lt;/h2&gt;

&lt;p&gt;"Could not be read" is about the file. But once Google can read it, the URLs inside matter too. A quick check that every listed URL returns 200 and isn't redirected or noindexed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -oE &lt;span class="s1"&gt;'&amp;lt;loc&amp;gt;[^&amp;lt;]+&amp;lt;/loc&amp;gt;'&lt;/span&gt; /tmp/sitemap.xml &lt;span class="o"&gt;|&lt;/span&gt; sed -E &lt;span class="s1"&gt;'s/&amp;lt;\/?loc&amp;gt;//g'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; head -200 \
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r u&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'%s %s\n'&lt;/span&gt; &lt;span class="s2"&gt;"$(curl -s -o /dev/null -w '%{http_code}' "&lt;/span&gt;&lt;span class="nv"&gt;$u&lt;/span&gt;&lt;span class="s2"&gt;")"&lt;/span&gt; &lt;span class="s2"&gt;"$u"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; grep -v &lt;span class="s1"&gt;'^200'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every URL in a sitemap should be the canonical, indexable, 200 version. A sitemap full of redirects and 404s doesn't break the fetch, but it teaches Google that your sitemap isn't reliable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: resubmit and wait
&lt;/h2&gt;

&lt;p&gt;Once the file passes all the curl checks, resubmit it in Search Console. The status usually updates within a day or two. If it doesn't, use the URL Inspection tool on the sitemap URL itself: the live test shows exactly what Googlebot received, including the HTTP status.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sitemap index files
&lt;/h2&gt;

&lt;p&gt;If you use a sitemap index (a sitemap of sitemaps), every child sitemap has to pass the same checks. A common failure is an index that lists child sitemaps on a different host or protocol (&lt;code&gt;http://&lt;/code&gt; children in an &lt;code&gt;https://&lt;/code&gt; index, or a CDN hostname), or children that were renamed in a deploy while the index still points at the old names. Fetch each &lt;code&gt;&amp;lt;loc&amp;gt;&lt;/code&gt; in the index the same way you fetched the index itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping it fixed
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reference the sitemap in robots.txt&lt;/strong&gt; (&lt;code&gt;Sitemap: https://www.example.com/sitemap.xml&lt;/code&gt;), so crawlers find it even if the Search Console submission is lost.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test the sitemap in CI&lt;/strong&gt;: well-formed XML, right content type, URL count under the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check it from outside after deploys&lt;/strong&gt;, with a crawler user agent, since WAF and CDN rules only exist in production. That's the part a tool like &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; automates.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-check-if-robotstxt-is-blocking-google-and-fix-it-without-breaking-anything-1ck5"&gt;How to check if robots.txt is blocking Google (and fix it without breaking anything)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/postmortem-hypothetical-our-robotstxt-returned-503-for-nine-days-and-crawling-nearly-stopped-15ef"&gt;Postmortem (hypothetical): our robots.txt returned 503 for nine days, and crawling nearly stopped&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-discovered-currently-not-indexed-in-google-search-console-a-developers-list-of-5731"&gt;How to fix 'Discovered – currently not indexed' in Google Search Console (a developer's list of causes)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-blocked-due-to-access-forbidden-403-in-google-search-console-its-usually-your-waf-4eed"&gt;How to fix 'Blocked due to access forbidden (403)' in Google Search Console (it's usually your WAF or bot rules)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/should-you-noindex-your-tag-and-category-pages-lets-argue-about-it-523k"&gt;Should you noindex your tag and category pages? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What was the cause the last time your sitemap showed "Couldn't fetch"? I've seen the ampersand one more times than I'd like to admit.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>xml</category>
    </item>
    <item>
      <title>Should every URL end with a trailing slash? Let's argue about it</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Wed, 18 Nov 2026 21:27:26 +0000</pubDate>
      <link>https://dev.to/seotto/should-every-url-end-with-a-trailing-slash-lets-argue-about-it-4mcf</link>
      <guid>https://dev.to/seotto/should-every-url-end-with-a-trailing-slash-lets-argue-about-it-4mcf</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Few URL questions generate as much quiet frustration as this one. &lt;code&gt;example.com/pricing&lt;/code&gt; or &lt;code&gt;example.com/pricing/&lt;/code&gt;?&lt;/p&gt;

&lt;p&gt;To search engines, those are two different URLs. They can return different content, carry different canonical tags, and get crawled and indexed separately. Frameworks pick different defaults. Static hosts sometimes add a redirect you didn't configure. And every team that has migrated between frameworks has a story about a few thousand redirects that appeared overnight.&lt;/p&gt;

&lt;p&gt;Nearly everyone agrees on the minimum: &lt;strong&gt;pick one form and make the other redirect to it.&lt;/strong&gt; The argument is about which one, so let's have it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for trailing slashes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It matches how static files work.&lt;/strong&gt; On a static host, &lt;code&gt;/pricing/&lt;/code&gt; maps naturally to &lt;code&gt;/pricing/index.html&lt;/code&gt;. Many static site generators produce exactly that directory structure, and servers like Nginx and Apache add the slash automatically when you request a directory without one. Using slashes everywhere matches what the server does anyway.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Relative links behave predictably.&lt;/strong&gt; On &lt;code&gt;/docs/install/&lt;/code&gt;, a relative link to &lt;code&gt;config&lt;/code&gt; resolves to &lt;code&gt;/docs/install/config&lt;/code&gt;. On &lt;code&gt;/docs/install&lt;/code&gt; (no slash), the same link resolves to &lt;code&gt;/docs/config&lt;/code&gt;. Sites that use relative links internally, especially documentation, get fewer surprises with slashes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It signals "this is a section".&lt;/strong&gt; &lt;code&gt;/blog/&lt;/code&gt; reads as a collection that contains things; &lt;code&gt;/blog/my-post/&lt;/code&gt; as a page in it. Some people find that hierarchy easier to reason about.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's the default in several popular tools.&lt;/strong&gt; Hugo and many other static site generators, WordPress (with typical permalink settings) and plenty of hosting platforms default to slashes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case against
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It's how most people type and share URLs.&lt;/strong&gt; Nobody types the trailing slash. Links in emails, chat and social posts usually omit it. Without a slash, the URL people type is the canonical URL, and no redirect is needed for most incoming traffic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It looks like a file path for pages that aren't directories.&lt;/strong&gt; &lt;code&gt;/pricing&lt;/code&gt; is a page, not a folder. For apps where URLs are routes rather than files, the slash is a leftover from the static file era.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's the default in several other popular tools.&lt;/strong&gt; Next.js defaults to no trailing slash, as do many application frameworks and routers. If your stack defaults that way, fighting it adds configuration for no clear benefit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Files and API endpoints never have it.&lt;/strong&gt; &lt;code&gt;/feed.xml&lt;/code&gt;, &lt;code&gt;/sitemap.xml&lt;/code&gt; and &lt;code&gt;/api/users&lt;/code&gt; don't end with slashes. A no-slash convention is consistent across pages, files and endpoints.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually matters for SEO
&lt;/h2&gt;

&lt;p&gt;Either form is fine for search engines. What causes problems is inconsistency:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Both versions returning 200.&lt;/strong&gt; Then you have duplicate URLs, split link signals, and crawlers fetching everything twice. One form must redirect (301 or 308) to the other.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal links using the non-canonical form.&lt;/strong&gt; Every internal link that hits the redirect costs an extra request, for crawlers and users. Links, canonicals, sitemaps and hreflang should all use the canonical form directly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Canonicals that disagree with the redirect.&lt;/strong&gt; A page at &lt;code&gt;/pricing/&lt;/code&gt; with a canonical tag pointing to &lt;code&gt;/pricing&lt;/code&gt;, while &lt;code&gt;/pricing&lt;/code&gt; redirects to &lt;code&gt;/pricing/&lt;/code&gt;, is a loop of mixed signals.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redirect chains after a migration.&lt;/strong&gt; Moving from a slash framework to a no-slash one (or the reverse) on top of an HTTP-to-HTTPS redirect and a &lt;code&gt;www&lt;/code&gt; redirect can produce three hops for old links. Collapse them into one.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Slash rules only work if every URL actually follows them.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; checks for canonical and redirect inconsistencies, including trailing-slash mismatches.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;I lean towards &lt;strong&gt;no trailing slash for application-style sites and slashes for content sites built from static files&lt;/strong&gt;, but honestly the right choice is whichever your framework and host do by default. Fighting your stack's default is a recurring source of bugs: a config change in one layer, a redirect rule in another, and a host that adds its own behaviour on top.&lt;/p&gt;

&lt;p&gt;Whatever you choose:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Configure it in exactly one place, and make sure the other layers don't add their own redirects.&lt;/li&gt;
&lt;li&gt;Check it with curl, for a page, a nested page, a URL with a query string, and the root:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; u &lt;span class="k"&gt;in&lt;/span&gt; /pricing /pricing/ /docs/install /docs/install/ &lt;span class="s2"&gt;"/pricing?ref=x"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'%-22s '&lt;/span&gt; &lt;span class="s2"&gt;"$u"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; curl -s -o /dev/null -w &lt;span class="s1"&gt;'%{http_code} %{redirect_url}\n'&lt;/span&gt; &lt;span class="s2"&gt;"https://example.com$u"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exactly one of each pair should return 200; the other should redirect to it in a single hop, keeping the query string.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Make your sitemap, canonicals and internal links use the canonical form, and check them after every framework upgrade.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Catching duplicate URL forms and redirect chains like these is one of the routine checks we run at &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the setting lives in common stacks
&lt;/h2&gt;

&lt;p&gt;Part of the confusion is that the setting lives in different places. In Next.js it's the &lt;code&gt;trailingSlash&lt;/code&gt; option in &lt;code&gt;next.config.js&lt;/code&gt;. In Hugo, the &lt;code&gt;uglyURLs&lt;/code&gt; setting and your content structure decide it. In Astro, it's the &lt;code&gt;trailingSlash&lt;/code&gt; config option, with &lt;code&gt;build.format&lt;/code&gt; deciding the file layout. In Nginx, requests for a directory without the slash get a 301 automatically, and &lt;code&gt;try_files&lt;/code&gt; rules decide the rest. On static hosts such as Netlify, Cloudflare Pages and Vercel, there's often a separate "pretty URLs" or trailing slash setting that applies on top of your framework's output. Check every layer; two layers disagreeing is how redirect loops happen.&lt;/p&gt;

&lt;h2&gt;
  
  
  The edge cases
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The root.&lt;/strong&gt; &lt;code&gt;example.com&lt;/code&gt; and &lt;code&gt;example.com/&lt;/code&gt; are the same URL to HTTP clients, so you can't redirect between them. No debate needed here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;File-like paths.&lt;/strong&gt; If you use slashes, does &lt;code&gt;/feed.xml/&lt;/code&gt; get a slash? (Please, no.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Migrating an established site.&lt;/strong&gt; If your current form is consistent and indexed, is switching ever worth thousands of redirects? I'd say almost never, but some teams do it for framework consistency.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-blocked-due-to-access-forbidden-403-in-google-search-console-its-usually-your-waf-4eed"&gt;How to fix 'Blocked due to access forbidden (403)' in Google Search Console (it's usually your WAF or bot rules)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/canonical-tags-faq-10-questions-developers-ask-about-relcanonical-with-straight-answers-3hij"&gt;Canonical tags FAQ: 10 questions developers ask about rel=canonical (with straight answers)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/there-is-no-duplicate-content-penalty-but-duplicate-content-still-costs-you-30f5"&gt;There is no duplicate content penalty (but duplicate content still costs you)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/pagination-seo-in-2026-relnext-is-dead-and-these-5-checks-matter-instead-16e0"&gt;Pagination SEO in 2026: rel=next is dead, and these 5 checks matter instead&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So: trailing slash or not? And has your framework ever made the decision for you in a way you didn't notice until the redirects appeared?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>discuss</category>
      <category>nginx</category>
    </item>
    <item>
      <title>How to fix 'Blocked due to access forbidden (403)' in Google Search Console (it's usually your WAF or bot rules)</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Wed, 18 Nov 2026 21:26:54 +0000</pubDate>
      <link>https://dev.to/seotto/how-to-fix-blocked-due-to-access-forbidden-403-in-google-search-console-its-usually-your-waf-4eed</link>
      <guid>https://dev.to/seotto/how-to-fix-blocked-due-to-access-forbidden-403-in-google-search-console-its-usually-your-waf-4eed</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Search Console's Pages report has a reason that looks simple: &lt;strong&gt;Blocked due to access forbidden (403)&lt;/strong&gt;. Googlebot requested the URL and got HTTP 403 Forbidden, so the page can't be indexed.&lt;/p&gt;

&lt;p&gt;The confusing part is that the page usually loads perfectly in your browser. The 403 is aimed at Googlebot specifically, or at requests that look like it, or at requests from where Google crawls. Something between Google and your content decided this visitor wasn't welcome.&lt;/p&gt;

&lt;p&gt;Here's how to find which layer it was and how to fix it safely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: confirm it's real and current
&lt;/h2&gt;

&lt;p&gt;Pick a few example URLs from the report and run &lt;strong&gt;URL Inspection, then Test live URL&lt;/strong&gt;. If the live test shows a 403, the problem is happening now. If it shows 200, the 403 was temporary (a rule that's since been changed, a brief rate limit) and you can click "Validate fix" once you're sure the cause is gone.&lt;/p&gt;

&lt;p&gt;Also check: are these pages you actually want indexed? If they're admin pages, account pages or staging URLs that correctly return 403, the fix is to stop linking them and remove them from the sitemap, not to unblock them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: find which layer returned the 403
&lt;/h2&gt;

&lt;p&gt;A 403 can come from several places. The response itself usually tells you which:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI -A &lt;span class="s2"&gt;"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"&lt;/span&gt; https://example.com/page
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at the headers and body:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CDN or WAF block pages&lt;/strong&gt; have recognisable headers (&lt;code&gt;server: cloudflare&lt;/code&gt; with a &lt;code&gt;cf-ray&lt;/code&gt;, &lt;code&gt;x-amz-cf-id&lt;/code&gt; with an AWS WAF block, &lt;code&gt;x-sucuri-id&lt;/code&gt;, &lt;code&gt;x-iinfo&lt;/code&gt; and so on) and a branded HTML page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your web server&lt;/strong&gt; returns its default 403 page (&lt;code&gt;nginx&lt;/code&gt; or &lt;code&gt;Apache&lt;/code&gt; in the &lt;code&gt;server&lt;/code&gt; header).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your app&lt;/strong&gt; returns its own error template.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A curl request with a Googlebot user agent from your own IP isn't the same as a real Googlebot request. Many bot rules check that "Googlebot" requests come from Google's IP ranges, and correctly block your fake one. That's fine; you're looking for which layer answers, not reproducing Google exactly. The URL Inspection live test is the real thing.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A 403 for Googlebot is the kind of deploy regression that should page you.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; watches for indexing blocks, including access-forbidden responses that only crawlers see.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 3: the usual causes
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Bot protection or WAF rules
&lt;/h3&gt;

&lt;p&gt;The most common cause. A bot-management feature, a "block suspected bots" rule, a rate-limit rule or a custom firewall expression catches Googlebot. It often happens right after someone turns on a stricter security setting, or adds a rule to stop a scraper that happened to use a Googlebot-like user agent.&lt;/p&gt;

&lt;p&gt;Fix: check the WAF or CDN's event log, filtered by the user agent containing "Googlebot" and the time of the failed crawls. Most major CDNs have a built-in "verified bots" category that identifies real search engine crawlers by IP and reverse DNS. Allow that category through before your blocking rules. Never allow by user agent string alone: anyone can send "Googlebot".&lt;/p&gt;

&lt;h3&gt;
  
  
  Rate limiting
&lt;/h3&gt;

&lt;p&gt;Rate limits that count requests per IP will eventually catch a crawler that requests many pages from a small set of IPs. Depending on configuration, the limiter returns 403 or 429. Google treats 429 as "slow down" and comes back, while a 403 reads as "you're not allowed here", so if you rate limit crawlers at all, return 429 (or 503) and exempt verified search crawlers from limits designed for abusive clients.&lt;/p&gt;

&lt;h3&gt;
  
  
  Geo-blocking
&lt;/h3&gt;

&lt;p&gt;If your site blocks countries, remember that Googlebot mostly crawls from US IP addresses. A site that only allows its home country, or blocks the US for licensing reasons, will block Googlebot. Exempt verified crawlers from geo rules, or serve crawlers the same content as allowed visitors.&lt;/p&gt;

&lt;h3&gt;
  
  
  Server-level rules
&lt;/h3&gt;

&lt;p&gt;Old &lt;code&gt;.htaccess&lt;/code&gt; or Nginx rules sometimes block user agents by pattern (&lt;code&gt;deny&lt;/code&gt; lists of "bad bots" copied from a forum years ago, which often include &lt;code&gt;bot&lt;/code&gt; as a substring). Search your server config for user agent matches, and for &lt;code&gt;deny&lt;/code&gt; or &lt;code&gt;return 403&lt;/code&gt; rules that could match crawler requests.&lt;/p&gt;

&lt;h3&gt;
  
  
  File permissions
&lt;/h3&gt;

&lt;p&gt;If only certain paths return 403 for everyone (including you, when you test from a fresh browser), the problem may be file system permissions or a missing index file with directory listing disabled. Nginx returns 403 when it can't read the file or there's no index document; the error log says &lt;code&gt;permission denied&lt;/code&gt; or &lt;code&gt;directory index of ... is forbidden&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Authentication and paywalls
&lt;/h3&gt;

&lt;p&gt;Pages behind a login return 401 or 403 to Googlebot, which can't log in. For paywalled content you want indexed, the usual approach is to serve the full content to verified Googlebot and mark it up with structured data for paywalled content, so it's not treated as cloaking.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: verify the fix with real crawler traffic
&lt;/h2&gt;

&lt;p&gt;After changing a rule:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Run URL Inspection's live test on several affected URLs. They should return 200.&lt;/li&gt;
&lt;li&gt;Check your access or CDN logs over the next day for verified Googlebot requests to those paths, and confirm they get 200.&lt;/li&gt;
&lt;li&gt;Click "Validate fix" in the report. Validation can take a couple of weeks for larger numbers of URLs.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Keep an eye on the Crawl Stats report too. A sudden rise in 403 responses after a security change is a clear signal, and it's worth checking after every WAF rule change.&lt;/p&gt;

&lt;p&gt;Security changes that accidentally lock out search crawlers are a recurring theme in the issues we see at &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;, largely because the site looks fine to everyone who checks it in a browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/what-your-500s-503s-and-429s-tell-search-crawlers-and-how-to-take-a-site-down-for-maintenance-nlh"&gt;What your 500s, 503s and 429s tell search crawlers (and how to take a site down for maintenance without hurting SEO)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/postmortem-hypothetical-our-robotstxt-returned-503-for-nine-days-and-crawling-nearly-stopped-15ef"&gt;Postmortem (hypothetical): our robots.txt returned 503 for nine days, and crawling nearly stopped&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-tell-real-googlebot-from-fakes-in-your-access-logs-reverse-dns-and-ip-ranges-in-python-44n9"&gt;How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/should-every-url-end-with-a-trailing-slash-lets-argue-about-it-4mcf"&gt;Should every URL end with a trailing slash? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What caused the 403s on your site: a bot rule, a geo block, or something nobody could explain? And did anyone find out from Search Console before traffic dropped?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>security</category>
      <category>webdev</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Serving your blog or docs at /blog through a reverse proxy? A 12-point SEO checklist</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Mon, 16 Nov 2026 11:52:19 +0000</pubDate>
      <link>https://dev.to/seotto/serving-your-blog-or-docs-at-blog-through-a-reverse-proxy-a-12-point-seo-checklist-f8a</link>
      <guid>https://dev.to/seotto/serving-your-blog-or-docs-at-blog-through-a-reverse-proxy-a-12-point-seo-checklist-f8a</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A common setup: the main site is one app, and the blog (or docs, or help centre) is a different app, often on a hosted platform, served under the main domain through a reverse proxy. Visitors and crawlers see &lt;code&gt;example.com/blog/my-post&lt;/code&gt;. Behind the scenes, the proxy fetches &lt;code&gt;example-blog.hostingplatform.app/my-post&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Teams do this because they want one domain instead of &lt;code&gt;blog.example.com&lt;/code&gt;, and whichever side of that argument you're on, the proxy itself has to be configured carefully. The most common failure is the origin's hostname or root-relative paths leaking into places crawlers read: canonical tags, sitemaps, redirects and links. Then search engines see two sites, or a site whose canonical tags point somewhere else entirely.&lt;/p&gt;

&lt;p&gt;Here's the checklist.&lt;/p&gt;

&lt;h2&gt;
  
  
  The proxy
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Make sure paths map cleanly in both directions
&lt;/h3&gt;

&lt;p&gt;With Nginx, the trailing slashes in &lt;code&gt;location&lt;/code&gt; and &lt;code&gt;proxy_pass&lt;/code&gt; decide whether the &lt;code&gt;/blog&lt;/code&gt; prefix is stripped:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="n"&gt;location&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;blog&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;proxy_pass&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//example-blog.hostingplatform.app/;   # /blog/post -&amp;gt; /post&lt;/span&gt;
    &lt;span class="n"&gt;proxy_set_header&lt;/span&gt; &lt;span class="nc"&gt;Host&lt;/span&gt; &lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;blog&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hostingplatform&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;proxy_set_header&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Forwarded&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Host&lt;/span&gt; &lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;proxy_set_header&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Forwarded&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Proto&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;proxy_ssl_server_name&lt;/span&gt; &lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="na"&gt;location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;blog&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;301&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;blog&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Test a few URLs with and without trailing slashes, with query strings, and with encoded characters, and confirm each one reaches the right origin path.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Rewrite redirects from the origin
&lt;/h3&gt;

&lt;p&gt;If the origin redirects &lt;code&gt;/old-post&lt;/code&gt; to &lt;code&gt;/new-post&lt;/code&gt;, it sends &lt;code&gt;Location: https://example-blog.hostingplatform.app/new-post&lt;/code&gt; (or &lt;code&gt;/new-post&lt;/code&gt;). Without rewriting, visitors and crawlers are sent off your domain, or to &lt;code&gt;example.com/new-post&lt;/code&gt; without the prefix. In Nginx:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="n"&gt;proxy_redirect&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//example-blog.hostingplatform.app/ /blog/;&lt;/span&gt;
&lt;span class="n"&gt;proxy_redirect&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;blog&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then check with &lt;code&gt;curl -sI https://example.com/blog/old-post&lt;/code&gt; that the &lt;code&gt;Location&lt;/code&gt; header stays on your domain and keeps the prefix.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Handle cookies and caching headers on purpose
&lt;/h3&gt;

&lt;p&gt;Cookies set by the origin may be scoped to its own domain or to &lt;code&gt;/&lt;/code&gt;. Decide whether the blog needs cookies at all. Make sure &lt;code&gt;Cache-Control&lt;/code&gt; from the origin passes through sensibly, and that your CDN doesn't cache personalised pages from either app under the other's rules.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Block the origin hostname from being indexed
&lt;/h3&gt;

&lt;p&gt;The origin is still publicly reachable. If search engines find it, you have a duplicate of your whole blog. Options, in order of preference: restrict the origin to requests from your proxy (an allowlist or a shared secret header), make the origin redirect to the proxied URL for any request that doesn't come from the proxy, or at minimum make its canonical tags point at the proxied URLs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The app behind the proxy
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. Configure the public base URL in the app
&lt;/h3&gt;

&lt;p&gt;Most blog engines and docs generators have a setting for the site's public URL and base path (&lt;code&gt;site_url&lt;/code&gt;, &lt;code&gt;baseURL&lt;/code&gt;, &lt;code&gt;basePath&lt;/code&gt;, &lt;code&gt;url&lt;/code&gt; plus &lt;code&gt;baseUrl&lt;/code&gt;). Set it to &lt;code&gt;https://example.com/blog/&lt;/code&gt;. This one setting usually fixes canonicals, sitemaps, feeds and Open Graph URLs together.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Check the canonical tags
&lt;/h3&gt;

&lt;p&gt;View source on a few proxied pages. The canonical should be &lt;code&gt;https://example.com/blog/my-post&lt;/code&gt;, not the origin hostname and not a URL without the prefix. A wrong canonical is the most damaging leak on this list, because it tells search engines the proxied page isn't the real one.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Check the sitemap and robots.txt
&lt;/h3&gt;

&lt;p&gt;The blog's sitemap must list proxied URLs, and should be referenced from the main site's &lt;code&gt;robots.txt&lt;/code&gt; (the one at &lt;code&gt;example.com/robots.txt&lt;/code&gt;; crawlers don't read &lt;code&gt;example.com/blog/robots.txt&lt;/code&gt;). If the origin serves its own &lt;code&gt;robots.txt&lt;/code&gt;, it only applies to the origin hostname.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A reverse-proxy docs move can hide indexing problems until traffic drops.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; checks for canonical, robots and indexing issues after the cutover.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The HTML
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. Find root-relative links and assets
&lt;/h3&gt;

&lt;p&gt;A link like &lt;code&gt;href="/tags/nginx"&lt;/code&gt; or &lt;code&gt;src="/assets/app.js"&lt;/code&gt; in the blog's HTML resolves to &lt;code&gt;example.com/tags/nginx&lt;/code&gt;, which is the main site, not the blog. Some engines handle this through the base path setting; hand-written templates often don't. Crawl a sample of proxied pages and list every root-relative URL that doesn't start with &lt;code&gt;/blog/&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. Be careful with the &lt;code&gt;&amp;lt;base&amp;gt;&lt;/code&gt; tag
&lt;/h3&gt;

&lt;p&gt;Some setups fix relative links with &lt;code&gt;&amp;lt;base href="/blog/"&amp;gt;&lt;/code&gt;. That changes how every relative URL on the page resolves, including fragment links like &lt;code&gt;href="#comments"&lt;/code&gt; (which then point to &lt;code&gt;/blog/#comments&lt;/code&gt;, not the current page) and scripts that build URLs. If you use it, test fragment links, forms and client-side routing explicitly. Fixing the templates is usually safer.&lt;/p&gt;

&lt;h3&gt;
  
  
  10. Check structured data and social tags
&lt;/h3&gt;

&lt;p&gt;JSON-LD &lt;code&gt;url&lt;/code&gt;, &lt;code&gt;@id&lt;/code&gt;, &lt;code&gt;mainEntityOfPage&lt;/code&gt;, and image URLs, plus &lt;code&gt;og:url&lt;/code&gt; and &lt;code&gt;og:image&lt;/code&gt;, are often built from a different config value than the canonical. Check them separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick leak check script
&lt;/h2&gt;

&lt;p&gt;Most of items 6 to 10 can be checked at once. Fetch a handful of proxied pages and search the HTML for the origin hostname and for root-relative URLs outside the prefix:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; p &lt;span class="k"&gt;in&lt;/span&gt; / /my-post/ /tags/nginx/&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;html&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;curl -s &lt;span class="s2"&gt;"https://example.com/blog$p"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"== $p"&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$html"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -o &lt;span class="s1"&gt;'hostingplatform\.app[^"]*'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort -u
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$html"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -oE &lt;span class="s1"&gt;'(href|src)="/[^"]*"'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -v &lt;span class="s1"&gt;'="/blog/'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort -u &lt;span class="o"&gt;|&lt;/span&gt; head
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any output is a leak worth fixing. Run it again after every upgrade of the blog engine, since template changes are a common way leaks come back.&lt;/p&gt;

&lt;h2&gt;
  
  
  After launch
&lt;/h2&gt;

&lt;h3&gt;
  
  
  11. Crawl both hostnames
&lt;/h3&gt;

&lt;p&gt;Crawl &lt;code&gt;example.com/blog/&lt;/code&gt; and confirm no links lead to the origin hostname or outside the prefix. Then search for the origin hostname in a search engine and in your logs' referrers, to see whether it has been found.&lt;/p&gt;

&lt;h3&gt;
  
  
  12. Watch Search Console for the new section
&lt;/h3&gt;

&lt;p&gt;Add a URL-prefix property for &lt;code&gt;https://example.com/blog/&lt;/code&gt;, or filter the domain property by that path, and watch indexing and the "Alternate page with proper canonical tag" and "Duplicate" statuses for the first few weeks. Leaks in canonicals and redirects show up there quickly.&lt;/p&gt;

&lt;p&gt;Leaked hostnames and canonicals pointing at the wrong place are exactly the kind of quiet issues we look for at &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;, because the pages look perfect to a human visitor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-fix-discovered-currently-not-indexed-in-google-search-console-a-developers-list-of-5731"&gt;How to fix 'Discovered – currently not indexed' in Google Search Console (a developer's list of causes)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/quick-tip-a-tiny-script-that-prints-the-title-canonical-and-robots-tags-of-any-url-so-you-stop-1500"&gt;Quick tip: a tiny script that prints the title, canonical and robots tags of any URL (so you stop opening view-source)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/canonical-tags-faq-10-questions-developers-ask-about-relcanonical-with-straight-answers-3hij"&gt;Canonical tags FAQ: 10 questions developers ask about rel=canonical (with straight answers)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/case-study-hypothetical-the-staging-site-that-got-indexed-and-started-outranking-production-41oi"&gt;Case study (hypothetical): the staging site that got indexed, and started outranking production&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you put a blog or docs site behind a /blog proxy? What leaked: the canonical, the redirects, or the sitemap?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>nginx</category>
      <category>webdev</category>
      <category>devops</category>
    </item>
    <item>
      <title>How to fix 'Discovered – currently not indexed' in Google Search Console (a developer's list of causes)</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Mon, 16 Nov 2026 11:51:51 +0000</pubDate>
      <link>https://dev.to/seotto/how-to-fix-discovered-currently-not-indexed-in-google-search-console-a-developers-list-of-5731</link>
      <guid>https://dev.to/seotto/how-to-fix-discovered-currently-not-indexed-in-google-search-console-a-developers-list-of-5731</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In the Pages report of Search Console, under "Why pages aren't indexed", there's a reason that confuses a lot of people: &lt;strong&gt;Discovered – currently not indexed&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It's easy to read it as "Google looked at these pages and rejected them". That's a different status ("Crawled – currently not indexed"). "Discovered" means something earlier in the pipeline: &lt;strong&gt;Google knows the URL exists, from a link or a sitemap, but hasn't crawled it yet.&lt;/strong&gt; The "last crawled" date for these URLs is usually empty.&lt;/p&gt;

&lt;p&gt;So the question isn't "what's wrong with these pages?" but "why hasn't Google got round to fetching them?" The answer is almost always one of a few things, and most of them are fixable from the code side.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: look at what's in the list
&lt;/h2&gt;

&lt;p&gt;Open the report and export the example URLs. Before changing anything, sort them into groups:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pages you actually want indexed&lt;/strong&gt;: articles, products, docs pages.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pages you don't care about&lt;/strong&gt;: filter combinations, internal search results, tag archives, paginated lists, tracking parameter variants, print versions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A mix of both&lt;/strong&gt;, which is the most common result.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the list is mostly the second group, the fix isn't to get those indexed. It's to stop generating and linking them, so Google spends its attention on the first group.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: check which of these causes applies
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Cause 1: too many URLs for the site's perceived value
&lt;/h3&gt;

&lt;p&gt;Google decides how much to crawl a site based on how important the site seems and how much its server can handle. A small site that suddenly exposes 200,000 URLs (an import of product variants, auto-generated location pages, an infinite calendar) will see most of them sit in "Discovered" for a long time. Google found them, but it's in no hurry.&lt;/p&gt;

&lt;p&gt;Fixes: reduce the number of crawlable low-value URLs. Stop linking parameter combinations, don't list them in sitemaps, canonicalise variants to a main page, and consider whether every auto-generated page deserves to exist at all. Fewer, better URLs get crawled faster.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cause 2: the pages are only in the sitemap, not linked
&lt;/h3&gt;

&lt;p&gt;A URL that appears in the sitemap but isn't linked from any other page on your site is weak evidence that the page matters. Sitemaps help discovery, but internal links are a much stronger signal of priority.&lt;/p&gt;

&lt;p&gt;Check: for a sample of the stuck URLs, search your own site's HTML for links to them. Orphan pages, reachable only through the sitemap, are very common after migrations and on sites with JavaScript-only navigation. Link to important pages from relevant category pages, hubs, or related content, using normal &lt;code&gt;&amp;lt;a href&amp;gt;&lt;/code&gt; links in the HTML.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cause 3: the server slowed down or errored when Google crawled
&lt;/h3&gt;

&lt;p&gt;If Google's crawlers see slow responses, timeouts or 5xx errors, they reduce the crawl rate to avoid overloading the site. Lots of URLs then queue up in "Discovered".&lt;/p&gt;

&lt;p&gt;Check the Crawl Stats report (Settings, Crawl stats): look at average response time, the share of responses by status code, and "host status" problems. Then check your own access logs for Googlebot requests that returned 5xx or took several seconds. A response time that rises sharply when crawling increases is a sign your server or database is the bottleneck.&lt;/p&gt;

&lt;p&gt;Fixes: cache rendered pages, make sure crawlers get the same cached pages users do, and fix whatever makes uncached pages slow. Return &lt;code&gt;503&lt;/code&gt; with &lt;code&gt;Retry-After&lt;/code&gt; during maintenance, never a slow &lt;code&gt;200&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cause 4: the content looks similar to pages already known
&lt;/h3&gt;

&lt;p&gt;When many URLs share a template and most of their text (thin product variants, near-identical location pages), Google may deprioritise crawling more of them. That's not a penalty; it's an expectation that the next page won't add much.&lt;/p&gt;

&lt;p&gt;Fixes: merge thin variants into stronger pages, or make each page genuinely different with real content.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cause 5: it's simply new
&lt;/h3&gt;

&lt;p&gt;A site or section launched recently can have pages in "Discovered" for days or weeks. If the URLs are good, linked and the server is fast, this usually resolves itself.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;“Discovered – currently not indexed” is easier to fix when you catch the cause early.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; watches for indexing blocks and deploy regressions that leave new URLs stuck in that state.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 3: what not to do
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Don't spam "Request indexing".&lt;/strong&gt; It's rate-limited, helps for a handful of important URLs, and does nothing for thousands of them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't resubmit the sitemap every day.&lt;/strong&gt; Google already knows about the URLs; that's what "Discovered" means.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't add more URLs to fix the problem.&lt;/strong&gt; "More content" in the form of more auto-generated pages makes cause 1 worse.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't block the low-value URLs in robots.txt as the only fix&lt;/strong&gt; if they're already linked everywhere. They'll stay discovered, uncrawled and sometimes still appear in results without a snippet. Remove the links and canonicalise or noindex them so Google can see that.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: measure whether it's working
&lt;/h2&gt;

&lt;p&gt;Track two numbers monthly: the count in "Discovered – currently not indexed", and, from your logs, the number of distinct URLs verified Googlebot requests per day in the sections you care about. If the second number rises and the first falls after you've cut low-value URLs and improved linking, you're on the right track. Expect weeks, not days.&lt;/p&gt;

&lt;p&gt;Separating the URLs you care about from the URLs you've accidentally created is the core of most crawl problems, and it's what we spend much of our time on at &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-find-orphan-pages-with-python-diff-your-sitemap-against-your-internal-links-45mg"&gt;How to find orphan pages with Python: diff your sitemap against your internal links&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/your-site-search-creates-infinite-urls-an-seo-checklist-for-internal-search-result-pages-2b3e"&gt;Your site search creates infinite URLs: an SEO checklist for internal search result pages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/pagination-and-infinite-scroll-an-seo-checklist-for-the-component-everyone-builds-in-an-afternoon-3g2c"&gt;Pagination and infinite scroll: an SEO checklist for the component everyone builds in an afternoon&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/out-of-stock-discontinued-or-gone-forever-an-seo-checklist-for-product-pages-developers-actually-1cn7"&gt;Out of stock, discontinued, or gone forever? An SEO checklist for product pages developers actually own&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/serving-your-blog-or-docs-at-blog-through-a-reverse-proxy-a-12-point-seo-checklist-f8a"&gt;Serving your blog or docs at /blog through a reverse proxy? A 12-point SEO checklist&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How big is your "Discovered – currently not indexed" list, and what turned out to be in it? Filters and tag archives, or real content stuck in the queue?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>beginners</category>
    </item>
    <item>
      <title>How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:14:19 +0000</pubDate>
      <link>https://dev.to/seotto/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-logs-in-python-1hfd</link>
      <guid>https://dev.to/seotto/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-logs-in-python-1hfd</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Opinions about AI crawlers are everywhere. Numbers from your own site are rarer, and they're the only ones that should drive a &lt;code&gt;robots.txt&lt;/code&gt; change.&lt;/p&gt;

&lt;p&gt;The questions worth answering first:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What share of all bot requests come from AI-related crawlers?&lt;/li&gt;
&lt;li&gt;How much bandwidth do they use?&lt;/li&gt;
&lt;li&gt;Which sections of the site do they hit?&lt;/li&gt;
&lt;li&gt;Are they training crawlers, search crawlers, or fetches triggered by a person asking an assistant to read your page?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Your access logs already have the answers. This post is a short Python script that classifies requests by user agent and reports exactly those numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  The categories
&lt;/h2&gt;

&lt;p&gt;User agent strings change, and operators add new ones, so treat this as a starting list and check each operator's documentation before relying on it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Search crawlers&lt;/strong&gt;: &lt;code&gt;Googlebot&lt;/code&gt;, &lt;code&gt;bingbot&lt;/code&gt;, &lt;code&gt;DuckDuckBot&lt;/code&gt;, &lt;code&gt;YandexBot&lt;/code&gt;, &lt;code&gt;Applebot&lt;/code&gt; (the plain one, used for Apple's search features).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI training&lt;/strong&gt;: &lt;code&gt;GPTBot&lt;/code&gt;, &lt;code&gt;ClaudeBot&lt;/code&gt;, &lt;code&gt;CCBot&lt;/code&gt;, &lt;code&gt;Bytespider&lt;/code&gt;, &lt;code&gt;meta-externalagent&lt;/code&gt;, &lt;code&gt;Amazonbot&lt;/code&gt;, &lt;code&gt;cohere-ai&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI search&lt;/strong&gt;: &lt;code&gt;OAI-SearchBot&lt;/code&gt;, &lt;code&gt;PerplexityBot&lt;/code&gt;, &lt;code&gt;Claude-SearchBot&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI user-triggered&lt;/strong&gt;: &lt;code&gt;ChatGPT-User&lt;/code&gt;, &lt;code&gt;Claude-User&lt;/code&gt;, &lt;code&gt;Perplexity-User&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;Google-Extended&lt;/code&gt; and &lt;code&gt;Applebot-Extended&lt;/code&gt; won't appear in logs: they're &lt;code&gt;robots.txt&lt;/code&gt; tokens, not separate crawlers. Google's AI products fetch with the normal Googlebot.&lt;/p&gt;

&lt;p&gt;Like Googlebot, all of these can be faked. A request claiming to be &lt;code&gt;GPTBot&lt;/code&gt; from a residential IP is probably a scraper. The bigger operators publish their IP ranges; for numbers you'll act on, check requests against those ranges, the same way you'd verify Googlebot with reverse DNS.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;gzip&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;

&lt;span class="no"&gt;CATEGORIES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"search"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;         &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Googlebot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"bingbot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"DuckDuckBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"YandexBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Applebot/"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"ai-training"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"GPTBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ClaudeBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"CCBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Bytespider"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"meta-externalagent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="s2"&gt;"Amazonbot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cohere-ai"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"ai-search"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;      &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"OAI-SearchBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"PerplexityBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Claude-SearchBot"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"ai-user"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ChatGPT-User"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Claude-User"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Perplexity-User"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="no"&gt;BOTLIKE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"bot|crawl|spider|slurp|fetch"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# combined log format: ... "GET /path HTTP/1.1" 200 1234 "referer" "user agent"&lt;/span&gt;
&lt;span class="no"&gt;LINE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s1"&gt;'"[A-Z]+ (\S+) [^"]*" (\d{3}) (\d+|-) "[^"]*" "([^"]*)"'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;needles&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;CATEGORIES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;needles&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="no"&gt;BOTLIKE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"other-bot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"other"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"human/other"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pattern&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pattern&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gzip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rt"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"replace"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;".gz"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"replace"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;

&lt;span class="n"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agents&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;paths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;statuses&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;LINE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ua&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="n"&gt;bytes_&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="n"&gt;section&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="n"&gt;statuses&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"xx"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="n"&gt;total_bot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"human/other"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{'category':14s} {'requests':&amp;gt;10s} {'% of bots':&amp;gt;10s} {'MB':&amp;gt;10s}  statuses"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;share&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"human/other"&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{100 * n / max(total_bot, 1):.1f}%"&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{cat:14s} {n:10d} {share:&amp;gt;10s} {bytes_[cat] / 1e6:10.1f}  {dict(statuses[cat])}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\nTop agents:"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"human/other"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{n:8d}  {cat:12s} {agent}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ai-training"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ai-search"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ai-user"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"\nTop sections for {cat}:"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{n:8d}  {s}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it over a month of logs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 ai_bots.py &lt;span class="s2"&gt;"/var/log/nginx/access.log*"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you're behind a CDN, run it on the CDN's logs instead. Many bot requests are served from the edge cache and never reach your origin, so origin logs undercount.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Logs tell you what happened; a recurring check tells you when it changes.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; flags indexing and crawler-access regressions so you don't have to re-run the script by hand.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Reading the numbers
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The share of bot traffic
&lt;/h3&gt;

&lt;p&gt;On many content sites, AI-related crawlers now make up a noticeable share of all bot requests, sometimes rivalling Googlebot. On a small site with little content, they may be a rounding error. Either way, this one number tells you whether the issue is worth more of your time.&lt;/p&gt;

&lt;h3&gt;
  
  
  Bandwidth versus requests
&lt;/h3&gt;

&lt;p&gt;Compare the MB column with the request count. A crawler with many requests but small transfers is mostly hitting cached or small pages. One with large transfers per request may be downloading images, PDFs or large HTML pages. If server cost is your concern, bandwidth is the number to watch.&lt;/p&gt;

&lt;h3&gt;
  
  
  Status codes
&lt;/h3&gt;

&lt;p&gt;A lot of &lt;code&gt;4xx&lt;/code&gt; for one agent usually means it's requesting URLs that don't exist: old links, guessed paths, or URLs from a stale dataset. A lot of &lt;code&gt;3xx&lt;/code&gt; means it's following your redirects. Neither is harmful, but both are wasted capacity on your side.&lt;/p&gt;

&lt;h3&gt;
  
  
  Sections
&lt;/h3&gt;

&lt;p&gt;If training crawlers spend most of their requests on &lt;code&gt;/blog&lt;/code&gt; and &lt;code&gt;/docs&lt;/code&gt;, that's your published content being collected. If they're in &lt;code&gt;/search&lt;/code&gt;, &lt;code&gt;/tag&lt;/code&gt; or parameter URLs, they're stuck in the same crawl traps that waste Googlebot's time, and fixing those helps both.&lt;/p&gt;

&lt;h3&gt;
  
  
  User-triggered fetches
&lt;/h3&gt;

&lt;p&gt;Requests from agents like &lt;code&gt;ChatGPT-User&lt;/code&gt; are people asking an assistant about your page. The top paths here show which content people bring to AI assistants, which is useful product and content feedback in its own right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Spot-check the claims with IP ranges
&lt;/h2&gt;

&lt;p&gt;To check whether a busy "AI crawler" is genuine, take its top IP addresses from the logs and compare them with the operator's published ranges. OpenAI, for example, publishes JSON files of IP ranges for its crawlers, and other large operators document theirs. Python's &lt;code&gt;ipaddress&lt;/code&gt; module makes the comparison a few lines: load the CIDR blocks, then test &lt;code&gt;ipaddress.ip_address(ip) in network&lt;/code&gt; for each. Anything claiming to be a known crawler from outside its operator's ranges belongs in the "other-bot" bucket, and is often the real source of the load people blame on AI.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turning it into a decision
&lt;/h2&gt;

&lt;p&gt;With these numbers you can decide per category: allow, block in &lt;code&gt;robots.txt&lt;/code&gt;, or rate limit at the CDN. Rerun the script a month after any change to confirm the crawlers you blocked actually stopped, and that search crawlers were unaffected. At &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; we like decisions like this to start from a log query rather than a headline.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/should-you-block-ai-crawlers-gptbot-claudebot-ccbot-in-robotstxt-lets-argue-about-it-18af"&gt;Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-read-your-server-logs-like-googlebots-diary-with-a-short-python-script-4o3d"&gt;How to read your server logs like Googlebot's diary (with a short Python script)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-tell-real-googlebot-from-fakes-in-your-access-logs-reverse-dns-and-ip-ranges-in-python-44n9"&gt;How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-long-until-googlebot-sees-your-change-measure-recrawl-latency-from-your-own-access-logs-275b"&gt;How long until Googlebot sees your change? Measure recrawl latency from your own access logs&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What share of your bot traffic turned out to be AI crawlers? And did any agent show up that you'd never heard of?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>python</category>
      <category>logs</category>
      <category>ai</category>
    </item>
    <item>
      <title>Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it</title>
      <dc:creator>seotto.com</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:13:43 +0000</pubDate>
      <link>https://dev.to/seotto/should-you-block-ai-crawlers-gptbot-claudebot-ccbot-in-robotstxt-lets-argue-about-it-18af</link>
      <guid>https://dev.to/seotto/should-you-block-ai-crawlers-gptbot-claudebot-ccbot-in-robotstxt-lets-argue-about-it-18af</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Automated technical SEO checks that catch indexing blocks and deploy regressions — &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A couple of years ago, &lt;code&gt;robots.txt&lt;/code&gt; was mostly about search engines. Now a typical site's logs include a whole list of AI-related crawlers: OpenAI's, Anthropic's, Common Crawl's, Perplexity's, Apple's, ByteDance's and others. Many sites added blanket blocks in a hurry. Others never touched their &lt;code&gt;robots.txt&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Should you block them? I think this is a genuinely open question, and the answer has changed as the bots themselves have changed. Let's argue about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, they're not all the same bot
&lt;/h2&gt;

&lt;p&gt;The big operators now publish several user agents with different purposes. Roughly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Training crawlers&lt;/strong&gt; collect pages to train models: for example OpenAI's &lt;code&gt;GPTBot&lt;/code&gt;, Anthropic's &lt;code&gt;ClaudeBot&lt;/code&gt;, Common Crawl's &lt;code&gt;CCBot&lt;/code&gt; (used by many model builders).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Search crawlers&lt;/strong&gt; index pages so an AI product can cite and link to them in answers: for example &lt;code&gt;OAI-SearchBot&lt;/code&gt;, &lt;code&gt;PerplexityBot&lt;/code&gt;, &lt;code&gt;Claude-SearchBot&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User-triggered fetchers&lt;/strong&gt; load a page because a person asked the assistant to read it right now: for example &lt;code&gt;ChatGPT-User&lt;/code&gt;, &lt;code&gt;Claude-User&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Control tokens&lt;/strong&gt; that aren't crawlers at all: &lt;code&gt;Google-Extended&lt;/code&gt; and &lt;code&gt;Applebot-Extended&lt;/code&gt; are names you can disallow in &lt;code&gt;robots.txt&lt;/code&gt; to opt out of your content being used for those companies' AI models, without affecting Google Search or Apple's search features.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Names and purposes change, so check each operator's current documentation before relying on a list. The point is that "block AI" now means several separate decisions.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for blocking
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Consent.&lt;/strong&gt; Your content was written for readers, and nobody asked whether it could be used to train a commercial model. Blocking training crawlers is the clearest signal you can give.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No traffic in return.&lt;/strong&gt; Training a model on your article doesn't send anyone to your site. If the model can then answer the question your article answers, it may reduce your traffic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Server cost.&lt;/strong&gt; Some AI crawlers are aggressive, request huge numbers of pages, and ignore crawl-delay hints. Smaller sites have seen real bandwidth and CPU costs from them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Leverage.&lt;/strong&gt; Publishers negotiating licensing deals have argued that blocking is the only bargaining position content owners have.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for allowing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;AI answers are a discovery channel now.&lt;/strong&gt; People ask assistants for recommendations, documentation and comparisons. Products that cite sources can only cite pages their search crawlers can read. Block the search agents and you're invisible there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Developer content wants to be found.&lt;/strong&gt; For documentation, open source projects and developer tools, being the source an assistant learned from, or cites, is often the whole point of publishing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Blocking doesn't remove what's already collected.&lt;/strong&gt; Pages crawled before you added the rule may already be in datasets. A new block only affects future crawling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;robots.txt&lt;/code&gt; is voluntary.&lt;/strong&gt; Well-behaved crawlers follow it. Badly behaved ones don't, and some scrapers simply pretend to be a browser. A block mostly affects the operators who were already playing fair.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Before you tighten robots.txt, it helps to see what is already blocked.&lt;/strong&gt; &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt; checks for indexing blocks and robots regressions, including rules you didn't mean to ship.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;For most sites, I'd make the decision per purpose, not per company:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;User-triggered fetchers: allow.&lt;/strong&gt; A person asked an assistant to read your page. That's closer to a browser visit than to crawling, and blocking it mainly frustrates your own readers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI search crawlers: allow, if being cited matters to you.&lt;/strong&gt; For most commercial and developer sites, it does. Check your logs and analytics for referrals from AI products before deciding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Training crawlers: a values decision.&lt;/strong&gt; There's no technical right answer. Blocking &lt;code&gt;GPTBot&lt;/code&gt;, &lt;code&gt;ClaudeBot&lt;/code&gt;, &lt;code&gt;CCBot&lt;/code&gt; and setting &lt;code&gt;Google-Extended&lt;/code&gt; and &lt;code&gt;Applebot-Extended&lt;/code&gt; to disallow is a coherent position; so is allowing them. Decide deliberately and write down why.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Aggressive crawlers: rate limit, don't just disallow.&lt;/strong&gt; If one bot is costing you real money, a rate limit at the CDN or server is more reliable than hoping it reads &lt;code&gt;robots.txt&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never block Googlebot or Bingbot by accident.&lt;/strong&gt; A blanket &lt;code&gt;User-agent: *&lt;/code&gt; disallow added in a hurry to stop AI bots also removes you from regular search.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;An example of the split approach:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;GPTBot&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ClaudeBot&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;CCBot&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Google&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Extended&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="nc"&gt;Allow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Search and user-triggered agents fall under &lt;code&gt;*&lt;/code&gt; and stay allowed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check what your robots.txt actually says today
&lt;/h2&gt;

&lt;p&gt;Before changing anything, fetch your live file and test a few agents against it. Python's standard library can do it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;RobotFileParser&lt;/span&gt;
&lt;span class="n"&gt;rp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;RobotFileParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"https://example.com/robots.txt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ua&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Googlebot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Bingbot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"GPTBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"OAI-SearchBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ChatGPT-User"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ClaudeBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"CCBot"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{ua:15s}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;can_fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"https://example.com/blog/some-post"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's a quick way to catch the worst outcome, which is blocking Googlebot while trying to block something else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Revisit it every few months
&lt;/h2&gt;

&lt;p&gt;Whatever you decide, put a reminder in the calendar. The list of crawlers, what each one is used for, and the products that cite sources are all changing quickly. A rule written for last year's bots may block a search agent that didn't exist then, or miss a new training crawler entirely. A quarterly look at your logs and at each operator's current documentation keeps the file honest.&lt;/p&gt;

&lt;h2&gt;
  
  
  The questions I can't settle
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Partial blocks.&lt;/strong&gt; Should you allow AI crawlers on marketing pages and docs but block them on paid or premium content?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;llms.txt and similar files.&lt;/strong&gt; Do they help, or are they another file nobody reads?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measuring the trade-off.&lt;/strong&gt; Is anyone seeing meaningful referral traffic from AI search products, enough to justify allowing their crawlers?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before deciding, look at what these bots actually do on your site. How much of your crawl traffic they make up, and which pages they hit, is a ten-minute log query. Measuring that kind of thing from real logs, rather than guessing, is what we prefer at &lt;a href="https://seotto.com" rel="noopener noreferrer"&gt;seotto.com&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/should-every-page-have-a-meta-description-lets-argue-about-it-3gkm"&gt;Should every page have a meta description? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-check-if-robotstxt-is-blocking-google-and-fix-it-without-breaking-anything-1ck5"&gt;How to check if robots.txt is blocking Google (and fix it without breaking anything)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-tell-real-googlebot-from-fakes-in-your-access-logs-reverse-dns-and-ip-ranges-in-python-44n9"&gt;How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/seotto/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-logs-in-python-1hfd"&gt;How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So what's in your robots.txt today: block everything AI, allow everything, or a split like this? And has anyone measured traffic from AI answers well enough to know whether allowing them paid off?&lt;/p&gt;

</description>
      <category>seo</category>
      <category>ai</category>
      <category>webdev</category>
      <category>discuss</category>
    </item>
  </channel>
</rss>
