DEV Community: 任宏涛

Nine SEO checks to run before you publish a page

任宏涛 — Wed, 23 Sep 2026 03:35:11 +0000

The problem with SEO checklists

Most of them list things you can see. The real problems are the ones you can't.

A missing canonical, a duplicated h1, a meta description that gets truncated — none of these break anything. The page renders, the content is good, and you find out weeks later in a report nobody reads closely.

So this is a list of checks for invisible failures. The order is deliberate: structural checks first, because fixing structure usually changes the text, and re-running the content checks after a rewrite wastes the first pass.

1. Does the page have exactly one `h1`?

Not zero, not two.

Zero usually means the title was styled with a class instead of a heading element. Two usually means a template printed the site name as an h1 and the page title as another one.

Both are invisible in a browser, and both make the page's main subject ambiguous.

Read the heading outline rather than an issue count. If the h2 list reads like a table of contents for the page, the structure is fine.

2. Do heading levels descend without skipping?

An h2 followed by an h4 tells a reader a level is missing.

Sometimes a level genuinely is missing and the outline is still correct — but more often the h4 was chosen for its smaller font size.

Choose levels for meaning and style them in CSS. Nothing in HTML requires an h2 to be visually larger than an h3.

3. Is the title tag specific to this page?

Open ten tabs of your site and read the titles.

If several are identical, those pages compete for the same query and none of them wins clearly. A title should describe what this page covers, not the category it belongs to.

4. Will the description survive truncation?

The widely quoted 155-character limit is an average, not a rule. The real constraint is rendered pixel width, and mobile is meaningfully narrower than desktop.

Put the sentence that must survive in the first ~120 characters and treat the rest as elaboration.

Preview it rather than counting — measure in pixels, not characters.

5. Does the canonical point at itself?

A page's canonical should be its own URL, in the same form the page is served at.

Trailing slashes are the common case. If /tools redirects to /tools/, the canonical must say /tools/ — otherwise you're telling Google the real page is a URL that redirects.

Check index pages specifically. They're the ones most often generated with a canonical that omits the trailing slash.

6. Is the page in the sitemap, in the right URL form?

Same trailing-slash question. A sitemap entry that redirects is a wasted crawl.

If your sitemap is generated, generate it from the same value as the canonical. Two independent string constructions will eventually disagree — and nothing will tell you when they do.

// Wrong: two places that must agree, with nothing enforcing it
const canonical = `${SITE}/blog/${slug}`;
sitemap.push(`${SITE}/blog/${slug}/`);   // ← drifts

// Right: one value, used twice
const url = `${SITE}/blog/${slug}/`;

7. Are the internal links real and useful?

Every new page should link to at least two existing pages, and at least one existing page should link to it.

A page nothing links to is a page that's hard to find. Internal links are the only navigation signal you fully control.

Use descriptive anchor text. "Read the guide" tells a reader nothing about where it goes.

8. Do images have dimensions and alt text?

Set explicit width and height on every image.

Without them the browser can't reserve space, and the page shifts as images load — the single most common cause of a bad layout-shift score.

<img src="/chart.png" width="800" height="420" alt="Response time by region">

Alt text describes the image for someone who can't see it. If the image is purely decorative, alt="" is correct — and better than a description nobody needs.

9. Does the structured data describe this page accurately?

If you mark up an Article, the headline should match the visible title and the dates should be real.

If you mark up an FAQPage, those questions must be visible on the page. Markup describing content that isn't there is a violation, not an optimisation.

What is deliberately not on this list

Keyword density. There's no target to hit, and writing to one produces worse text. Run a density check only to see whether the page is actually about the topic you intended — if the top phrases surprise you, that's a content problem, not a density problem.

Word count. Length is a consequence of covering the subject, not a goal. A page that answers the question in 400 words is finished. Padding it to 1200 makes it worse.

Meta keywords. Ignored for over a decade.

The full version of this, with the reasoning behind each item, is on SerpPrism. The tools mentioned above — heading analyzer, SERP preview, robots.txt tester — all run client-side, nothing is uploaded.

Curious whether anyone has a check I've missed. The one that keeps biting me is #5, because the failure only shows up on index pages.

I checked the Open Graph tags on 62 homepages. Here's what's actually broken.

任宏涛 — Wed, 23 Sep 2026 03:35:02 +0000

Open Graph tags don't affect rankings. They decide something narrower and easier to forget: whether your link renders as a card with an image, or as a bare grey link that people scroll past.

I got curious about what's actually in the wild, so I fetched the homepages of 78 well-known domains and parsed every <meta> tag whose property or name started with og: or twitter:. 62 came back usable.

Here's what I found, including one failure mode I had never considered.

Method, and the two counting rules that changed the numbers

78 domains, 62 usable homepages. Two rules mattered:

An empty or missing content attribute counts as absent. og:title="" is a tag that exists and says nothing. If you count it as present, you inflate your numbers and — as it turns out — hide the most interesting bug in the dataset.

No JavaScript was executed. This is server-rendered HTML only. That's usually fine for social tags (they're server-rendered because crawlers don't run JS), but it means every percentage below is a floor, not a rate.

Finding 1: Open Graph is near-universal. Twitter cards are not.

56 of 62 homepages (90%) had at least one Open Graph tag. But that headline hides how patchy individual fields are:

tag	count	%
`og:title`	54/62	87%
`og:description`	53/62	85%
`og:url`	49/62	79%
`og:image`	48/62	77%
`og:type`	48/62	77%
`og:site_name`	40/62	65%
`og:locale`	22/62	35%

So about one homepage in four has no shareable image.

The Twitter side is where the real gap is. Of the 62:

34 (55%) declared twitter:card = summary_large_image
14 (23%) = summary
2 (3%) = app
12 (19%) declared no twitter:card at all

That last group is the one to care about. X falls back to Open Graph for title, description and image — but it still needs twitter:card to know which layout to build. Without it you get a plain link, no matter how good your Open Graph tags are.

Finding 2: two sites ship an `og:image` tag with nothing in it

I expected this to be common. It wasn't: only 2 of the 34 sites declaring summary_large_image had no usable image. But the two fail differently, and one of them is genuinely sneaky.

netflix.com declares twitter:card = summary_large_image and twitter:site, and nothing else. No og:image, no og:title, no og:url. The card type promises a large image that was never supplied.

forbes.com does emit the tag:

<meta property="og:image" name="image" data-next-head=""/>

There is no content attribute at all.

This is the failure mode worth internalising. A missing tag fails loudly — any validator catches it. A present-but-empty tag passes every "is it there?" check and renders nothing. If your audit tool asks "does og:image exist?", Forbes returns yes.

Its twitter:image has the same shape, and its og:image:type is:

<meta property="og:image:type" content="image/jpeg,image/gif,image/png" data-next-head=""/>

A comma-separated list. Not a single MIME type. Not a valid value for that field.

You can spot this in one line:

curl -s https://example.com | grep -o '<meta[^>]*og:image[^>]*>'

If the tag shows up with no content=, you've found it.

I should be candid that I had this exactly backwards: until the day I collected this data, all 25 pages on my own site declared summary_large_image with no og:image. The pattern that turns out to be rare in the wild was on every page of my site.

Finding 3: only 23% declare image dimensions

Of the 48 homepages with a usable og:image, just 11 (23%) also declared og:image:width and og:image:height.

This sounds harmless. It isn't, and the reason is subtle: without dimensions the platform has to download the image before it can lay out the card. If that fetch is slow, rate-limited, or blocked — a CDN that dislikes the crawler's user agent, a region where the image host is slow — the first share goes out with no image and the second one works.

An intermittent bug like that is far harder to diagnose than a missing tag, because by the time you look, it works.

The 11 that do declare dimensions: figma.com, x.com, docker.com, developer.mozilla.org, w3.org, moz.com, screamingfrog.co.uk, yoast.com, searchenginejournal.com, harvard.edu, heroku.com.

Four of those — Moz, Screaming Frog, Yoast, Search Engine Journal — are SEO vendors or publications. All four SEO sites in the sample declare dimensions, against 23% overall. They're also the same four that showed up as @graph users when I ran the equivalent survey on JSON-LD. Consistent signal: the sites whose business is this metadata treat it better than average by a wide margin.

Only 5 of 48 (10%) declared og:image:alt. Alt text on a social card isn't decorative — it's what a screen reader announces.

Finding 4: one in four `og:title` disagrees with `<title>`

14 of the 54 homepages with both showed different text in each (26%).

Most are trivial: github.com drops a trailing "· GitHub" from the Open Graph version. Some aren't. kubernetes.io ships:

<title>Kubernetes</title>
<meta property="og:title" content="Production-Grade Container Orchestration">

The search result says one thing; the shared link says another. Neither is wrong — but it means the title you optimised for search isn't the one people see when the page is shared, and those are written for different purposes. If you only check one, check the one that appears in search.

Finding 5: the things I expected and didn't find

Across 48 usable images:

1 relative path (kubernetes.io, /images/kubernetes-open-graph.png)
0 http:// images
0 sites declaring more than one og:image

Relative og:image URLs are the classic warning — scrapers have no base URL to resolve against, so the tag gets dropped. It happened once in 48. Insecure images, not at all.

This is the third survey in a row where a widely repeated warning turned out to be rare at this sample size. That's not an argument for ignoring the warnings. It's an argument for checking the base rate before you spend a day on the thing you're worried about.

What to check on your own tags

Is the image tag carrying anything? View source, look for content=. Present-but-empty is the check no validator does for you.
Do you declare width and height? Two lines, kills the "first share has no image" failure mode.
Do you declare twitter:card? One in five sites skip it.
Does og:title match your title? If not, pick which one you meant.

The full dataset — including the raw HTML for all 62 domains, so you can check any claim against the markup rather than my summary of it — is on SerpPrism. There's also a preview tool that renders what your tags will actually produce.

If you find a row that's wrong, the markup settles it — and I'd genuinely like to know.

Your JSON-LD is duplicated in three places. Here's the fix.

任宏涛 — Wed, 23 Sep 2026 03:34:12 +0000

The pattern almost everyone starts with

You add an Organization block to the footer. An Article block to the article template. A BreadcrumbList to the breadcrumb component. Three separate <script type="application/ld+json"> blocks, each describing one thing.

It works — Google reads all of them — and it's where most sites stop.

The cost isn't visible immediately. It shows up the first time you rename the company, change the logo URL, or move domains. Now you have three places containing the same facts, and nothing enforces that they agree. One gets updated, the others don't, and your structured data now contradicts itself in a way no error message will tell you about.

What `@id` actually does

Every node in JSON-LD can carry an @id — a stable identifier. A URL is conventional.

Once a node has an @id, any other node can refer to it instead of repeating its properties.

So instead of the Article containing its own copy of the publisher's name and logo:

{
  "@type": "Article",
  "publisher": {
    "@type": "Organization",
    "name": "Acme",
    "logo": { "@type": "ImageObject", "url": "https://acme.com/logo.png" }
  }
}

…it contains a reference:

{
  "@type": "Article",
  "publisher": { "@id": "https://acme.com/#organization" }
}

The full description lives once, in a node other nodes point at. Change the logo in that one place and every reference is correct.

The shape that works

One @graph array in the page head, containing in order:

Organization
WebSite
WebPage (for the current URL)
whatever page-specific nodes apply — Article, FAQPage, HowTo, Product

The Organization and WebSite nodes are identical on every page — that's the point. They're declared once per page but defined consistently, and page-specific nodes reference them by @id rather than restating them.

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://acme.com/#organization",
      "name": "Acme",
      "url": "https://acme.com/"
    },
    {
      "@type": "WebSite",
      "@id": "https://acme.com/#website",
      "url": "https://acme.com/",
      "publisher": { "@id": "https://acme.com/#organization" }
    },
    {
      "@type": "WebPage",
      "@id": "https://acme.com/blog/post#webpage",
      "url": "https://acme.com/blog/post",
      "isPartOf": { "@id": "https://acme.com/#website" },
      "datePublished": "2026-03-14",
      "dateModified": "2026-08-02"
    },
    {
      "@type": "Article",
      "@id": "https://acme.com/blog/post#article",
      "headline": "…",
      "mainEntityOfPage": { "@id": "https://acme.com/blog/post#webpage" },
      "publisher": { "@id": "https://acme.com/#organization" }
    }
  ]
}

Four nodes, one block, every relationship explicit.

Three ways this goes wrong

1. Referencing an @id that doesn't exist on the page.

If the Article says its publisher is #organization but no node with that @id is present, the reference resolves to nothing. Google handles this by ignoring the property — so the symptom isn't an error, it's silently missing data. Always include the node you reference.

2. Reusing the same @id for different things.

Identifiers must be unique. Two nodes sharing an @id means you've declared two contradictory definitions of one entity, and the outcome depends on which one a consumer reads first.

3. Relative @id values.

Use absolute URLs. A relative reference like #organization is interpreted against the page URL, which means the same markup on two domains produces two different identifiers — and any cross-page relationship you were expressing breaks.

Dates deserve their own section

datePublished should never change after the page goes live.

dateModified should change when the content genuinely changes, and not otherwise.

Incrementing dateModified on every deploy is a common habit, usually adopted because someone believed freshness helps rankings. It doesn't — and the pattern is trivially detectable: a modified date that advances daily while the words stay identical tells any consumer the field is noise, which means it gets discounted on the pages where it was actually true.

If you use a build system, wire dateModified to a per-page content hash, not the build timestamp:

import crypto from "node:crypto";

const hash = crypto
  .createHash("sha256")
  .update(pageBody)
  .digest("hex")
  .slice(0, 12);

// store hash → date in a committed JSON file
// update the date only when the hash changes

The field then moves only when the content does.

Checking your markup

Validate with Google's Rich Results Test (for eligibility) and the Schema Markup Validator (for syntax).

Neither will tell you that your @id references are dangling — a dangling reference is not a syntax error.

The check that catches it takes a minute and is manual:

Extract every @id defined in the page
Extract every value used in a reference
Confirm every reference appears in the defined list

On a page with six nodes this is faster by eye than to automate.

I built a small generator for this shape that validates required properties as you type — SerpPrism's JSON-LD generator. It's client-side, nothing is uploaded. There's also a longer version of this post with a few more edge cases on the site itself.

Happy to hear how other people handle the dateModified problem — it's the one I've seen go wrong most often in production.

Why Google Won't Index Your Page: A Checklist

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

Two problems that look identical and are not

Almost every "my page is not indexed" complaint is one of two things, and they need opposite responses.

A code fault is something wrong with the URL or the page. It is fixable today, and once it is fixed the page becomes eligible. A trust gate is Google having fetched the page and decided not to spend index space on it yet. There is nothing to fix, because nothing is broken — and this is where most advice sends people in circles.

Order matters because the two produce the same symptom in Search Console, and the fix for one does nothing for the other. Re-requesting indexing on a page in a trust gate wastes your one lever. Rewriting the content of a page blocked by a noindex tag is worse than wasted, because the actual problem is still sitting there.

Step 1 — read the status code, not the page

A page that renders in your browser is not proof of anything. Ask the server instead:

curl -sS -o /dev/null -w "%{http_code}" https://example.com/page

What the answer tells you:

404 or 410 — the page is not indexed by design. Either the URL is wrong or it was deliberately removed. Nothing else in this guide applies.
5xx — the server is failing. Google backs off and retries later; it will not index a page it cannot fetch. Fix the server first.
301 or 302 — you are looking at a different URL from the one you think. Follow the redirect and diagnose the destination.
200 — good, but keep reading, because 200 is where the real traps live.

The trap that hides inside a 200 is the soft 404 : the server reports success and the content is an error page. Our robots.txt survey caught exactly this on two of 78 well-known domains — khanacademy.org and cdc.gov answered 200 with an HTML error page instead of a robots file. A monitor watching status codes sees nothing wrong and a browser shows a page. If your unindexed URL returns 200 and reads like an error, you have your answer.

Step 2 — is it blocked?

Two mechanisms can block indexing, and they are not the same mechanism:

robots.txt — a Disallow line stops the crawl. Note that it stops the fetch, not the indexing of a URL Google already knows about from links elsewhere.
A noindex meta tag or X-Robots-Tag header — this stops indexing even when the page is crawled normally.

One combination costs people weeks: a robots.txt Disallow and a noindex on the same page. The disallow prevents Google from fetching the page, so Google never sees the noindex, so the URL can still be indexed from links pointing at it. You then remove the noindex, observe no change, and conclude the tag was never the problem. The order to undo it is: remove the Disallow first, let Google fetch the page, confirm it reads the noindex, and only then remove that.

You can test the first of these with our robots.txt tester. For the header, one curl is enough: curl -sSI https://example.com/page | grep -i x-robots-tag. And note that a page with no robots meta at all is normal — index, follow is the default, so its absence is not a fault.

Step 3 — can it be discovered?

Google cannot index a URL it never finds. Three ways a page stays invisible:

It is not in your sitemap. Check that the URL appears in sitemap.xml — and that the URL listed there actually exists. A sitemap full of 404s is a signal of neglect rather than of coverage.
The sitemap is not declared. Listing it in robots.txt lets a crawler find it without being told twice.
Nothing links to it. This is the most common cause of a page that is technically perfect and never gets indexed. A page with no internal links is an orphan, and orphans depend entirely on the sitemap being read.

That last one is the same class of omission our canonical survey found on 11 of 55 famous homepages: the tag was simply absent. Nobody forgot on purpose. Omission is the default state of a site nobody audits.

Step 4 — is it a duplicate?

If the page is crawlable, discoverable and unblocked, the next question is whether Google treats it as a second copy of something else. The canonical tag is the instruction that decides this, and the mistake that matters is comparing it against the wrong URL.

In our canonical survey, 43 of 55 homepages redirected at least once before serving content. If you compare your canonical against the URL you typed rather than the URL you were finally served, you will conclude you have a cross-domain canonical problem when you do not. Compare against the final URL after redirects. Then check that the canonical target itself returns 200 — two of the 43 we tested pointed at URLs that did not, and nothing warns you when that happens.

Three more duplicates worth ruling out: the http and https versions of the page, the www and non-www versions, and any tracking parameters that create a second address for the same content.

Step 5 — is the content actually there?

If the page is a single-page app, the HTML Google receives may contain almost nothing. Curl the URL and read the raw source rather than the rendered page. If the body is an empty container and a bundle of script tags, that is what a crawler may be evaluating.

This is not hypothetical. Two of the homepages in our SERP snippet survey returned client-rendered shells with no title in the served HTML at all. Google can render JavaScript, but it does so on a second pass, and a page that needs rendering is a page that is slower and less certain to be evaluated.

Step 6 — now it is a trust gate, and there is nothing to fix

If every check above passed, the page is eligible and Google is choosing not to index it yet. Search Console describes this in two ways:

Discovered — currently not indexed : Google knows the URL exists and has not fetched it yet.
Crawled — currently not indexed : Google fetched it and judged it not worth index space at this point.

Neither is a defect, and neither is a bug you can patch. The usual reasons are thin content, no internal links, a brand-new domain with no authority, or content close enough to pages that already exist that Google does not need another copy of it.

What actually moves it: content clearly more useful than what is already indexed, internal links from pages that are themselves indexed, genuine external links or mentions, and time. What does not move it: requesting indexing repeatedly, resubmitting the sitemap repeatedly, or rewriting the page for the third time.

We went through this on our own site. A batch of pages sat in "Crawled — currently not indexed" while every technical check passed: 200 status, self-referencing canonical, index and follow, well over a thousand words each. There was nothing to fix. They drained on their own within days. The useful conclusion is not "wait and hope" — it is that when all five steps above pass, the correct action is to stop working on that page and go make the rest of the site better.

What "Request indexing" does, and does not do

It queues a fetch. That is the whole of it. It is worth using on a brand-new URL, or on a page you have just fixed, because it shortens the wait for the next crawl.

It is not worth using on a page that has already been crawled and judged. Requesting indexing does not override a quality decision, there is a practical limit on how often you can use it, and spending it on a trust gate burns the one lever you have for a genuine fault.

The order, in one table

#	Check	If it fails	Fixable by you
1	Status code	404, 410, 5xx, or a soft 404	Yes, today
2	robots.txt, noindex, X-Robots-Tag	Blocked from crawl or from index	Yes, today
3	Sitemap and internal links	Undiscoverable, or an orphan	Yes, today
4	Canonical and duplicates	Consolidated into another URL	Yes, today
5	Content present in the served HTML	Empty shell as seen by a crawler	Yes, today
6	All of the above passed	A trust gate	No — content, links and time

The point of working in this order is that you stop at the first failure, and you stop entirely at step 6. Most people start at step 6 and work backwards, which is why the advice to "just keep improving the content" gets applied to pages that were never crawlable in the first place.

What 59 Homepages Put in Their SERP Snippet

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

How this was measured

On 2026-09-25 we fetched https://<domain>/ for 78 hand-picked domains spanning news, commerce, SaaS, developer tooling, social, education, finance, government and SEO. 62 returned a usable page: 41 over a direct connection and 21 over a local proxy, because a number of these domains are unreachable from our network without one.

Three of the 62 were not homepages and were dropped. khanacademy.org answered 200 with a 3 KB Cloudflare interstitial whose entire title is "Client Challenge". paypal.com returned an 8 KB JavaScript shell with no <title> at all. archive.org returned a 1.9 KB shell. Counting them would have manufactured findings that do not exist — a 15-character title, two missing titles — so they are excluded and listed in the script's output instead.

One bug is worth naming, because it changed the headline. Our first pass read python.org as having no <title>. It has one. The site returns gzip unconditionally, and our fetch was not decompressing, so we were parsing gzip bytes as HTML. Once the fetch decompressed, the title appeared, and the count of homepages without a title went from two to zero. A measurement pipeline that fails quietly produces confident nonsense.

For each page we took the first <title> and the first <meta name="description">, decoded HTML entities, collapsed runs of whitespace and measured in Unicode code points. Skipping the decode step inflates every number: & counts as five characters instead of one.

Finding 1: every homepage has a title, and the median is 35 characters

59 of 59. Not one homepage in the sample ships without a <title>. The distribution is narrow at the bottom and long at the top: minimum 3 characters, 25th percentile 22, median 35, 75th percentile 54, 90th percentile 61, maximum 116, mean 36.9.

The familiar rule — keep your title under 60 characters — is true and almost empty. Only 7 of 59 pages (12%) exceed 60, and just 2 exceed 70. The real behaviour is at the opposite end: 21 of 59 (36%) are under 30 characters. Half the sample writes a title shorter than 35 characters.

If you have been treating 60 as a target to hit, you are aiming at a number almost nobody in this sample reaches.

Finding 2: the shorter the title, the bigger the brand

The shortest titles belong to the most famous names in the sample: w3.org (3 characters, "W3C"), forbes.com (6, "Forbes"), tumblr.com (6), twitch.tv (6), youtube.com (7), pinterest.com (9), wikipedia.org (9), wikimedia.org (9), kubernetes.io (10), developer.mozilla.org (12, "MDN Web Docs").

The longest belong to sites that still have to explain themselves: bbc.com at 116 characters, airbnb.com at 74, atlassian.com at 70, wise.com at 68, shopify.com at 66, github.com at 61.

That is not a style difference, it is an equity budget. "Forbes" is a complete title because the reader already knows what Forbes is; the word carries the meaning the other 54 characters would otherwise have to. A site nobody has heard of that ships its name alone has said nothing at all. The 60 characters are not a ceiling to stay under — they are space you have to fill until your name does the work by itself.

The practical corollary: copying the title format of a famous site means copying a luxury you have not earned yet.

Finding 3: nearly everyone writes a meta description, so that advice is stale

57 of 59 homepages (97%) ship a description. Only two do not: linkedin.com and cdc.gov.

This contradicts a piece of advice still repeated constantly — that most sites forget the description and let Google invent one. On this sample that is simply false. What the last decade of snippet rewriting actually did was make the tag universal rather than optional. The omission rate here is 3%, not the 50% the folklore implies.

Two caveats. linkedin.com served a 121 KB JavaScript shell and cdc.gov a 60 KB document, so a description injected after load would be invisible to this method; both are recorded as "none in the served HTML", which is also what a crawler parsing the initial document sees. And writing a description does not mean Google uses it — this measurement is only about what sites ship, not about which sentence ends up in the snippet.

Finding 4: descriptions cluster at 114 to 150, and the 160 guideline holds up

The length distribution: minimum 45, 25th percentile 114, median 131, 75th percentile 150, 90th percentile 184, maximum 559, mean 141.

That interquartile range — 114 to 150 — is remarkably tight for a number nobody enforces. Practitioners have converged on a working band just under the display limit without being told to.

12 of the 57 (21%) run past 160 characters. The extremes are instructive: kubernetes.io ships 559 characters, roughly three and a half times what will ever be shown; stanford.edu ships 364; trello.com ships 195. At the other end, four descriptions fall under 70 characters, leaving more than half of the available space empty.

The practical read: 120 to 155 characters is where this sample lives, and it is also where truncation is least likely.

Finding 5: pipe is the most common separator, but no separator is more common

Counting only separators with whitespace on both sides, so that the hyphen inside rust-lang is not mistaken for one: pipe appears in 16 titles (27%), hyphen in 11 (19%), em dash in 2, and colon, middle dot and slash once each.

But 27 of 59 (46%) use no separator at all. The pipe is the plurality, not the majority; nearly half the sample writes a name or a sentence and stops.

A separator is a formatting decision with no ranking consequence. What it does signal is how many ideas you are concatenating. One separator means two ideas, and in most of the titles above the second one is the weaker — the tagline, the region, the boilerplate. The titles with no separator have a single idea, and a single idea is usually the better title.

Finding 6: your title is not one thing

We fetched from a Chinese IP address, and several sites served a regional variant rather than the default. linkedin.com returned 领英企业服务. stripe.com returned "Stripe | 金融基础设施，托举营收增长". wise.com returned "Wise: The international account | Money without borders | Wise China". bloomberg.com returned "Bloomberg Europe".

None of those are mistakes. But they mean a title audit performed from one location gives you one slice of a title that changes with geography, and a crawler dispatched from a different location reads a different string. The same applies to the description.

If you localise titles per region, decide it deliberately and keep it consistent with your hreflang annotations and your canonical choice. If you do not intend to localise, check that your edge is not doing it for you — the way to find out is to fetch your own homepage from more than one country.

What to do with this

Write 50 to 60 characters, not 3. The median homepage title in this sample is 35 characters and 36% are under 30 — but those are household names. Unless yours is one, use the space.

Always write a description. 97% of the sample does. It is table stakes now, not an optional extra, and omitting it hands the sentence a searcher reads entirely to Google.

Target 120 to 155 characters. That is the measured interquartile band, and it sits just inside where truncation begins.

Do not copy a famous site's title. Its brevity is backed by recognition you have not earned yet.

Prefer one idea per title. 46% of the sample uses no separator. When you do need one, pipe is the safe default at 27% — but check whether the second half is pulling its weight.

Check what your edge actually serves. Fetch your homepage from two or three regions before concluding that your title is what you wrote.

To see the truncation before you ship it, the SERP preview tool on this site renders a title and description at Google's real desktop and mobile widths. The meta tag generator emits the title, description and canonical from a single input, so the values cannot drift apart from each other.

Limitations

59 homepages from 78 hand-picked, well-known domains — representative of famous sites, not of the web. A random sample would look different in both directions, and probably longer: small sites write longer titles than W3C does.

16 of the 78 domains never produced a readable page — 403s, bot checks, timeouts. That loss is not random; it skews toward large consumer and financial sites that fight crawlers, so the sample is biased toward sites that serve their homepage to anyone who asks.

We read the initial HTML only. A title or description injected by JavaScript after load is invisible here, and one of the two missing descriptions may be exactly that.

One fetch, from one IP address, on one day. Finding 6 is the direct consequence of that limitation rather than a separate discovery.

The brand-position analysis we planned did not survive checking and is not reported. Judging brand placement from the domain's first label fails on multi-word brands and on brands spelled differently from their domain: 9 of the 10 titles flagged as containing no brand did contain one — "the Guardian", "Node.js", "Rust", "Screaming Frog", "Search Engine Journal", "Square". We dropped the metric rather than publish a number we could not stand behind.

Reproduce it

The script and the shared domain list ship with this site. node scripts/survey-serp-snippet.mjs fetches and stores the raw HTML for every domain; --report re-derives every number from that stored HTML without touching the network; --sample=N prints raw title and description lines for manual checking; and --outliers lists every title over 60 characters, every description over 160, and every response excluded as a non-homepage. Our robots.txt, hreflang and canonical surveys cover the same domain list from different angles.

robots.txt Mistakes That Silently Deindex Pages

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

1. Disallowing a page you also marked noindex

This is the most damaging one and it looks reasonable. You want a page out of the index, so you add noindex — and you also disallow it so crawlers stop wasting time. But a crawler that cannot fetch the page can never read the noindex. The URL stays in the index indefinitely, usually with a note saying it was not crawled.

The fix is counterintuitive: to remove a page using noindex, you must allow crawling. Let Google fetch it, see the directive, and drop it. Only then disallow, if you still want to.

2. Blocking CSS and JavaScript

An older habit, and still common in older configs. Google renders pages, and rendering requires your stylesheets and scripts. Block /assets/ or /static/ and Google sees a page with no layout, which it may judge as broken or as a poor mobile experience.

Check your file for broad disallows on asset directories. If they exist, remove them — there is no upside to blocking Google from seeing your own stylesheet.

3. Assuming order decides priority

It does not. Within a user-agent group, the longest matching path wins regardless of line order. So Disallow: /admin/ followed by Allow: /admin/public/ leaves /admin/public/ crawlable. Reordering those lines changes nothing, which is why people conclude the file is being ignored when it is behaving exactly as specified.

4. Wildcard patterns that over-match

Disallow: /*.pdf$ blocks PDFs — and if written slightly wrong, blocks far more. A misplaced * or a missing $ turns a narrow rule into a site-wide block. Because robots.txt produces no warnings, this can run for months before anyone notices the traffic drop.

Test every rule containing a wildcard against a few real URLs before deploying it. That is what the tester on this site is for.

5. Forgetting that staging is one typo away from public

A staging host protected by robots.txt alone is not protected. Any crawler that ignores the file — and there are many — will index it. Duplicate staging content competing with production is a slow, confusing problem to untangle later.

Put real authentication in front of staging. Keep the robots.txt disallow as well, but understand it is politeness, not a lock.

A five-minute check that catches all of these

Fetch your robots.txt and paste it into a tester. Then run five URLs through it: your homepage, a category page, a deep article, one asset from your stylesheet directory, and a URL you believe is blocked. If any of the five gives a verdict you did not expect, you have found the bug — and you have found it before Google did.

Do this again after any deploy that touches routing, and any time someone new edits the file. The failure mode here is silence: nothing breaks visibly, nothing appears in an error log, and traffic from one section of the site quietly stops arriving. A monthly five-minute check is the entire maintenance cost.

Meta Description Length: What Gets Truncated

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

Where the 155-character rule comes from

It is a working average, not a limit. Someone measured a batch of results, found the cut usually landed near 155 characters, and the number spread because it is easy to remember. What Google actually does is render the description in a specific font at a specific size inside a container of a specific width, then drop whatever does not fit.

The practical consequence: character count and rendered width disagree constantly. A description of 150 narrow characters fits. A description of 138 characters with several capital M and W does not. If you have ever shipped a snippet that looked correct in your CMS and appeared chopped in search, this is why.

The two widths that matter

Desktop results render the description at roughly 14px in a container around 600px wide. Mobile is narrower — meaningfully narrower, not marginally. A description tuned exactly to the desktop limit will be cut on a phone, and most of your traffic is on a phone.

The safe approach is to aim at the mobile limit, not the desktop one. Put the part that must survive in the first 120 characters and treat everything after that as optional elaboration.

When Google ignores your description entirely

Often. If Google decides a sentence from your page body matches the query better than the description you wrote, it substitutes that sentence. This is normal behaviour and not a signal that anything is wrong.

You cannot force it to use your text. What you can do is remove the reason to substitute: write the opening sentence of the page so that it would work as a snippet on its own, and keep the meta description aligned with what the page actually says. When the two agree, there is nothing to gain by swapping.

What is not worth doing

Do not stuff the description with keyword variants. Descriptions are not a ranking factor; their entire job is to earn the click. A description written for a machine reads like it was written for a machine, and gets skipped.

Do not write one description and reuse it across a template. Duplicate descriptions are not penalised, but they make every page in a set look identical in the results, which costs you the clicks that a specific sentence would have earned.

A working method

Draft the description last, after the page is written. At that point you know what the page actually delivers, and you can say it in one sentence without guessing. Descriptions written before the page exists tend to describe the page you intended rather than the one you shipped.

Then put the sentence into a preview and check where it lands. If the important part is past the mobile cut, move it forward rather than trimming the tail — the end of a description is the cheapest thing to lose.

One more thing worth doing: read the first sentence of the page body next to the description. If they are far apart, Google has a reason to substitute one for the other. Aligning them is the closest thing to control you have over what gets shown.

What 61 Homepages Actually Ship in JSON-LD

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

How this was measured

On 2026-09-23 we fetched the homepage of 78 well-known domains and extracted every <script type="application/ld+json"> block from the response. 61 returned a usable page — 42 over a direct connection and 19 over a local proxy. Of those 61, 32 contained at least one JSON-LD block.

One caveat governs everything below, so it belongs at the top rather than the bottom: this measures what is present in the server-rendered HTML. A page that injects its structured data with JavaScript will look empty here even though a browser — and Google, which renders — would see it. Single-page applications are the obvious case. So when this article says a site "ships none", read it as "none in the initial HTML response", which is a different and weaker claim.

For the sites where we did find markup, that caveat disappears: everything about blocks, @graph, @id and node types is measured on markup that was actually present.

Finding 1: about half ship nothing in the HTML

29 of the 61 homepages contained no JSON-LD block at all. The list includes github.com, youtube.com, wikipedia.org, w3.org, mozilla.org, linkedin.com, x.com, spotify.com, npmjs.com, nodejs.org, kubernetes.io, mit.edu, stanford.edu, khanacademy.org, theguardian.com, slack.com, notion.so, trello.com, tumblr.com, vimeo.com, archive.org, wikimedia.org, digitalocean.com, squareup.com, uber.com, cdc.gov, go.dev, rust-lang.org and developer.mozilla.org.

Some of those are certainly JavaScript-rendered and will have markup in a browser. But wikipedia.org is not a JavaScript application, and neither is w3.org or github.com — those responses were 119KB, 51KB and 576KB of ordinary server-rendered HTML with no structured data anywhere in them.

The honest conclusion is narrower than "half the web has no structured data" and more useful than nothing: shipping JSON-LD is a choice that a large share of serious sites have not made, and it is not a precondition for ranking.

Finding 2: 14 sites still ship markup for a retired feature

This was the most surprising result. Google retired the sitelinks search box on 21 November 2024 — the search field that used to appear under a brand's result. The markup that powered it, a SearchAction on the WebSite node, now produces nothing.

Fourteen of the 32 sites with JSON-LD still carry it: cnn.com, forbes.com, walmart.com, atlassian.com, zoom.us, pinterest.com, docker.com, python.org, screamingfrog.co.uk, yoast.com, cloudflare.com, heroku.com, airbnb.com and who.int.

Two of those are SEO tool vendors. That is the interesting part: this is not a case of amateurs leaving stale markup around. It is a case of markup that was correct when it was written, that nothing broke when it stopped mattering, and that no tool flags because it is still valid schema.org. There is no error, no warning, and no rich result — just a few lines of JSON that no longer do anything.

It is worth being fair about this: leaving it in place costs nothing measurable. We are not suggesting the markup is harmful. The point is that it is invisible dead weight, and the only way to notice is to know the feature was retired — which is exactly the kind of thing that a site audit will never tell you.

If you are auditing your own markup, this is the check that pays: is any of it aimed at a feature that no longer exists? The schema generator on this site does not emit SearchAction for that reason.

Finding 3: a single block is the norm, and @graph is the minority

Of the 32 sites with markup, 24 used exactly one block, four used two, three used three, and one used four. So the "many small blocks" pattern that our @graph guide warns about is not actually the common shape — one block is.

Within that single block, however, the split matters. Only 13 of the 32 (41%) use a @graph array. The rest put a single node at the top level.

The 13: gitlab.com, atlassian.com, zoom.us, asana.com, docker.com, twitch.tv, moz.com, screamingfrog.co.uk, yoast.com, searchenginejournal.com, stripe.com, netlify.com and heroku.com.

Look at that list again. Four of the thirteen are SEO tool vendors or SEO publications — Moz, Screaming Frog, Yoast and Search Engine Journal. When the people who build the tooling converge on a pattern, that is usually worth more than a spec reading, and it is consistent with the argument in the @graph guide: one connected graph beats several disconnected blocks.

Finding 4: most markup is boilerplate

Twenty of the 32 sites ship only generic node types — Organization, WebSite, WebPage, ContactPoint, PostalAddress — with nothing specific to the page being viewed.

The type frequency across all 32 tells the same story. The most common types were PostalAddress (33 occurrences), Organization (25), Place (24), WebSite (20) and ImageObject (20). BreadcrumbList appeared 6 times. SoftwareApplication 6 times. VideoObject 6 times.

More addresses than web pages. For a homepage that is defensible — the homepage is where the organization describes itself. But it means that on most of these sites, the structured data says "here is a company" and never says "here is what this particular page is about".

Finding 5: the pattern at full size

stripe.com packs 54 nodes into a single @graph — by a wide margin the largest we found, and more than the next three sites combined.

That is the @graph pattern doing what it is for. Fifty-four connected nodes with @id references resolve to one coherent description of the site rather than fifty-four independent assertions that must agree with each other by hand.

What we did not find: almost no dangling references

Our @graph guide warns that referencing an @id which is never defined causes the property to be silently ignored. We expected to find this in the wild.

One site out of 32 — paypal.com, with a single reference to https://www.paypal.com/c2/home#website that no node defines.

Two other things we expected and did not find: zero JSON parse failures across every block we extracted, and no site shipping markup in more than four blocks. Invalid JSON-LD and block sprawl both turn out to be much rarer than the advice would suggest.

This is the second time in two surveys that a widely repeated warning did not appear at all in a sample of this size. That is not an argument for ignoring the warnings — a dangling reference is still a bug — but it is an argument for checking how often the thing you are worried about actually happens before you spend a week refactoring.

What to check on your own markup

Is any of it aimed at a retired feature? SearchAction is the one to search for today. Check the markup against the current list of supported rich results rather than against a tutorial you followed in 2023.

Does it say anything about this page? If every page on your site emits the same Organization and WebSite nodes and nothing else, the markup is describing your company and not your content.

Is it one connected graph or several disconnected blocks? Neither is an error. One is easier to keep consistent.

Does every @id you reference exist on the page? Extract the defined @id values and the referenced ones and compare the lists. It takes a minute by eye on a page with under ten nodes.

Limitations

The sample is 61 reachable homepages chosen by hand to span categories — representative of well-known sites, not of the web. Homepages only: article, product and FAQ markup lives on inner pages and is not measured here.

The server-rendered caveat is the big one. Any JavaScript-injected markup is invisible to this method, so the true rate of sites using structured data is certainly higher than 52%. The findings about @graph, node types and SearchAction are drawn only from markup we actually retrieved, so they are unaffected — but the headline number is a floor, not a rate.

Finally, this is a single-day snapshot of pages that change constantly.

Reproduce it

The extraction script and the domain list ship with this site and run in about two minutes. It writes the extracted JSON-LD for every domain, so each claim above can be checked against the markup itself rather than our summary of it.

If a row is wrong, the markup settles it. Corrections via the contact page — and if you know why the sitelinks search box markup is still so widespread two years after retirement, we would like to hear that too.

Redirect Chains in the Wild: 78 Domains

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

How this was measured

On 2026-09-30 we requested four entry URLs for each of the same 78 hand-picked domains our other surveys use — news, commerce, SaaS, developer tooling, social, streaming, reference, SEO, education, finance, cloud, local and government. The four entries were http://domain/, https://domain/, http://www.domain/ and https://www.domain/: 312 chain walks in total.

We did not use curl -L. We requested one hop at a time so each hop's status code and Location could be recorded separately. The reason is a measurement artefact: with -L and a header dump, a proxy's own CONNECT 200 is written into the header file alongside the real responses, and it is indistinguishable from a real 200. Walking the chain by hand avoids that entirely. Every hop was attempted directly first and then through a local proxy; 220 hops answered on the direct path, 114 needed the proxy, and one never answered at all. We recorded which path each hop used so this can be checked.

We used a survey user agent that identifies itself. 14 of the 78 domains answered with a bot wall rather than a page — those are reported as unreachable and excluded, not counted as defects.

Finding 1: 62 of 78 reach a page from the worst possible entry

http://domain/ is the entry old links arrive on, the one people type, the one a third party cites. 62 of 78 (79.5%) end at a 200. Sixteen do not, and nearly all of those are walls rather than failures: 11 answered 403 (nytimes.com, bloomberg.com, ebay.com, etsy.com, quora.com, stackoverflow.com, npmjs.com, searchengineland.com, coinbase.com, doordash.com, yelp.com), 2 answered 202 (amazon.com and booking.com, both challenge pages), one answered 406 (uber.com) and 2 answered 503 (bbc.com and nih.gov).

Four domains answered the bare HTTP entry without redirecting at all. archive.org returned 200 over plain HTTP. bbc.com and nih.gov returned 503 — they refuse HTTP outright, which is a defensible choice. searchengineland.com returned 403.

Finding 2: two hops is the most common answer, and 28 of them are avoidable

Hop counts from http://domain/: 32 take one hop (41.0%), 37 take two (47.4%) and 5 take three (6.4%). One hop is not the norm here; two is.

The shapes explain why. 28 domains go http://domain → https://domain → https://www.domain — the scheme upgrade and the host canonicalisation are two separate redirects. 16 collapse both into one hop (http://domain → https://www.domain). 15 do the scheme only and stop at the bare domain, which is their canonical form. So 28 of the 37 two-hop sites spend two round trips doing what the other 16 do in one.

The usual cause is two layers that do not know about each other: an edge or load balancer that upgrades HTTP to HTTPS, and an application that then adds www. Neither is wrong alone. Together they turn every first visit into two requests before any content arrives.

The five three-hop chains, verbatim:

theguardian.com: http://theguardian.com/ → https://theguardian.com/ → https://www.theguardian.com/ → /international.

notion.so: → https://notion.so/ → https://www.notion.so/ → https://www.notion.com/ — a domain migration stacked on top of the normalisation.

netflix.com: → https://netflix.com/ → https://www.netflix.com/ → https://www.netflix.com/hk-en/.

paypal.com: → https://paypal.com/ → https://www.paypal.com/ → https://www.paypal.com/cn/home.

cdc.gov: → https://cdc.gov/ → https://www.cdc.gov/ → /index.html. Three permanent redirects to land on a filename.

Finding 3: 14 sites use a temporary redirect for something permanent

Across all 312 walks we saw 301 ×245, 302 ×36, 308 ×16 and 307 ×14. 14 domains used 302 or 307 somewhere in the chain that ends at their own homepage: theguardian.com, cnn.com, notion.so, linkedin.com, nodejs.org, go.dev, developer.mozilla.org, netflix.com, mozilla.org, mit.edu, stripe.com, paypal.com, coinbase.com and doordash.com.

Four use it on the very first hop off HTTP: go.dev (302), mozilla.org (302), mit.edu (302) and doordash.com (302). mit.edu's first hop is http://mit.edu/ → http://web.mit.edu/, a temporary code on a host mapping that has been stable for years.

This matters because 301 and 308 tell a crawler to consolidate signals onto the target and update its index, while 302 and 307 tell it to keep the original. Reaching for the temporary pair during canonicalisation is a way of asking Google to remember the URL you just said you did not want.

Two details worth keeping. mozilla.org's first hop redirects to https://mozilla.org:443/ — an explicit default port, valid and mildly unusual. pinterest.com uses 308 twice, and vercel.com and khanacademy.org each use a single 308 to upgrade the scheme; 308 is 301's method-preserving twin and is the correct choice here.

Finding 4: six sites serve two reachable homepages

56 of 78 converge — every entry that resolves ends at the same URL. 7 do not , and one of those is our own artefact, so six are real:

bbc.com: https://bbc.com and https://www.bbc.com both answer 200 and neither redirects to the other. Plain HTTP returns 503, so at least the scheme is settled.

go.dev: https://go.dev and https://www.go.dev both answer 200.

developer.mozilla.org: the bare host redirects to /en-US/, while https://www.developer.mozilla.org answers 200 at the root.

mit.edu: http://mit.edu/ ends at https://web.mit.edu/, but http://www.mit.edu/ ends at https://www.mit.edu/. Which homepage you get depends on whether you typed www.

archive.org: all four entries answer 200. http://archive.org/ is served, and www redirects back to the bare host with a 302.

khanacademy.org: the bare domain upgrades to https://www.khanacademy.org, but http://www.khanacademy.org answers 200 in plain HTTP. Of the four combinations, www plus http is the one that was forgotten.

The seventh, paypal.com, landed on /cn/home on one path and /hk/home on the other. That is a geo redirect and an artefact of us exiting from two IP addresses, not a defect on their side.

Finding 5: eight homepages are not at /

Eight domains end their chain on a path other than the root: theguardian.com/international, nodejs.org/en, developer.mozilla.org/en-US/, netflix.com/hk-en/, stripe.com/cn, paypal.com/cn/home, squareup.com/us/en and cdc.gov/index.html.

Four of those are geography — we fetched from a Chinese IP address, and Stripe, PayPal, Netflix and Square each answered with the edition they serve from there. Those redirects are doing their job; the consequence is that the URL in your sitemap is not the URL a crawler somewhere else receives. One of our sitemap findings is the same effect seen from the other end.

Two are language negotiation (nodejs.org/en, MDN's /en-US/). One is an edition router (the Guardian). One, cdc.gov, redirects its root to /index.html — the homepage's address is a filename.

Seven also end on a different host from the one requested: cnn.com → edition.cnn.com, gitlab.com → about.gitlab.com, notion.so → www.notion.com, zoom.us → www.zoom.com, spotify.com → open.spotify.com, mit.edu → web.mit.edu and linkedin.com → www.linkedin.cn. Two are domain migrations (Notion, Zoom), one is geography (LinkedIn), and the rest are structural.

Finding 6: HSTS is on half the sample, preload on a third

39 of 78 send Strict-Transport-Security on their final HTTPS response. 25 of those include preload : Shopify, Atlassian, Slack, Notion, Figma, Asana, Trello, python.org, nodejs.org, go.dev, w3.org, Ahrefs, Semrush, Yoast, Search Engine Journal, Stanford, Coursera, Stripe, PayPal, Square, Vercel, Netlify, Airbnb, cdc.gov and who.int.

That leaves 39 sending no HSTS at all, which is why the scheme hop has to exist for them: without it, a browser's first request to a typed domain is still an HTTP request. With it — and especially with preload — the browser upgrades before the request leaves, and the first hop disappears for every visit after the first.

What to do with this

Count your own hops. Run curl -sIL http://yourdomain/ and count the HTTP/ lines. Two is the most common answer in this sample and one is achievable. If you see scheme and host handled separately, move both into a single rule at whichever layer is outermost.

Use 308 or 301 for canonicalisation, never 302 or 307. Fourteen sites here use the temporary pair on a redirect that will not change. 308 preserves the method and is the better default on modern stacks.

Choose www or bare and redirect every other form to it. The forgotten combination in this sample is almost always www plus http: Khan Academy normalises the bare domain to https plus www but serves http plus www as-is. Test all four entries, not two.

If you geo-redirect, test from more than one country. Four sites here served us an edition based on our IP. The redirect is correct, and the URL in your sitemap is still not the one most of the world sees. It is also part of why our hreflang survey found so much variation in what a homepage even is.

Decide whether a redirect or a canonical tag is the answer before reaching for either. They solve different problems: a redirect removes a URL, a canonical tag consolidates duplicates you have to keep serving. Our meta tag generator emits the tag for the second case — and if you are unsure which applies, the answer is usually that the redirect is the one you want.

Turn on HSTS last. It removes the first hop for repeat visitors and, once preloaded, for first visits too — but preload is deliberately hard to reverse, so only add it after every subdomain serves HTTPS.

Limitations

78 hand-picked well-known domains, not a random sample of the web. Categories are our own manual grouping, several contain five domains, and one site moves a rate by 20 points.

All requests came from one survey user agent on one day, over two egress IP addresses — one in China and one proxy exit. 14 domains answered with a bot wall (403, 406 or 202); those are excluded rather than counted as defects, and a browser-like user agent would have reached more of them. The geographic redirects in this data — Stripe's /cn, PayPal's /cn/home, Netflix's /hk-en/, Square's /us/en, LinkedIn's .cn — describe where we were, not what those sites do for everyone. PayPal appearing in the two-version list is an artefact of our two exits, not a finding about PayPal.

Channel is chosen per hop, so some chains switch egress partway through: 220 hops ran direct, 114 through the proxy. A geo-sensitive site can therefore show a hop that no single visitor would experience.

We counted hops, not latency. A three-hop chain inside one CDN is often cheaper than a single cross-origin redirect, and we did not measure TTFB or connection reuse. Hop count stands in for wasted round trips; it is not a measurement of how slow anything is.

We did not check what gets indexed. A redirect does not guarantee consolidation, and we did not read the canonical tag at the destination — several of these domains redirect to a URL that then declares something different in our canonical survey. Whether the homepage that ranks is the one at the end of the chain is a different question from the one answered here.

We followed up to eight hops and stopped on a loop or an unresolvable Location. No chain in this sample hit either limit.

Reproduce it

The script and the shared domain list ship with this site. node scripts/survey-redirects.mjs walks all four entries for all 78 domains and stores every hop; --report re-derives every number from stored data without touching the network; --evidence prints the status and Location of every hop for manual checking, which is how each chain quoted above was verified. Our robots.txt, canonical, hreflang, AI crawler and llms.txt surveys use the same domain list from other angles.

What 60 Sites Actually Do With hreflang

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

How this was measured

On 2026-09-24 we fetched the homepage of 78 high-traffic domains and looked for <link rel="alternate" hreflang="..."> tags in the initial HTML. Of the 78, 60 returned a usable page: 41 over a direct connection and 19 over a local proxy, because some domains are unreachable from our network without one. The other 18 were unreadable — four behind bot protection returning 403 or 418, the rest timing out or returning 404.

Extraction used a single regex over <link> tags, keeping only those whose rel contains alternate and whose hreflang is non-empty. A bare hreflang="" is treated as absent, because an empty annotation is the same as none and counting it would invent findings that are not there. The script and the domain list ship with this site, so every number below can be re-derived rather than taken on trust.

Finding 1: fewer than half use hreflang at all

28 of the 60 sites (46.7%) declare hreflang. The other 32 declare none. That sounds low until you look at who the 32 are: Amazon, LinkedIn, Reddit, GitHub, Wikipedia, the New York Times, Bloomberg — overwhelmingly monolingual or single-locale sites where hreflang would be meaningless. hreflang exists to tell a crawler "this page has equivalents in other languages or regions." A site with one language has nothing to point at, so its absence is correct, not a gap.

So the coverage number is not a score. It is a reminder that hreflang is a multilingual-site tool, and most of the web's famous names are not multilingual in the way that needs it.

Finding 2: when sites do it, they do it thoroughly

The 28 that declare hreflang are not dabbling. 24 of them ship six or more alternate links; the median site in the group declares 15. The outliers are startling: uber.com emits 173, stripe.com 89, mozilla.org 80, linkedin.com 77, shopify.com 71. The average across the 28 is 30.9 annotations per site.

Two things explain the spread. First, genuinely global products — ride-hailing, payments, browsers — really do serve dozens of locales, and each one is a link. Second, a few sites that are not obviously global still go deep: ahrefs.com and semrush.com, both SEO vendors, declare 14 each, which is consistent with the pattern we have seen across every survey on this site — the companies whose business is this metadata treat it more carefully than average.

The practical point for a normal site: you do not need 70 annotations. You need one correct set per localized URL, which for most operators is a handful. The big numbers belong to companies with genuinely big locale matrices.

Finding 3: the common miss is x-default

Of the 28 sites that declare hreflang, 9 (32.1%) have no x-default annotation. The list is not obscure: bbc.com, atlassian.com, notion.so, pinterest.com, docker.com, kubernetes.io, semrush.com, squareup.com, uber.com.

Google recommends x-default as the fallback for users whose language or region matches none of your specific alternates. Without it, the version an unmatched user sees is undefined, and because hreflang errors are silent, no validator will tell you it is missing. It is recommended, not required — a site without x-default is not penalised — but for sites serving this many locales, leaving it out is a real, if mild, gap. That a third of even the careful sites skip it tells you how easy it is to forget.

Finding 4: the format is cleaner than expected

We checked every hreflang value against BCP-47. Zero sites used a relative URL in the href. The values a naive validator might flag as non-standard — zh-Hans, zh-Hant-TW, sco, ast — are all valid: the Hans/Hant forms are script subtags, and sco and ast are legitimate ISO 639-3 language codes used by sites that genuinely serve Scots and Asturian. Nobody in the sample shipped a broken value like "english" or "en_en".

This is a useful contrast with our other surveys, which did find real defects. hreflang's format is mature enough that the mistakes are about relationships between annotations, not about the annotations themselves.

Finding 5: a sampling caveat, not a defect

Four sites — gitlab.com, notion.so, zoom.us, spotify.com — have their hreflang set living on a different domain or subdomain from the bare homepage we fetched: about.gitlab.com, notion.com, zoom.com, open.spotify.com. Because we sampled the bare homepage, that page shows no same-origin self-reference. In every case the bare domain is effectively an entry point that redirects to the host serving localized content, where the set is complete and reciprocal. We report it as a measurement artifact, not as a broken annotation.

The two errors that actually break hreflang

The format is fine and the coverage question is mostly irrelevant. The failures that matter are about relationships, and both are silent. The first is a missing return tag: if your English page points at the German page, the German page must point back, or Google treats the whole annotation as unconfirmed and ignores it. The second is a missing self-reference: every URL in the set must include a link to itself. A page that lists its siblings but not itself is, to a crawler, a page that never confirmed its own identity.

Both are invisible. The page renders, the links are present, and the annotation simply does not take effect — which is exactly the state you discover months later when a locale refuses to rank. The fix is to generate the full set, including x-default and the self-link, from one source, so the relationships cannot drift apart.

How to do it without the foot-guns

A correct set has one entry per locale plus x-default, every entry points at an absolute URL, the page lists itself, and every target links back. Writing that by hand for more than two locales is how the return-tag rule gets broken. Generating it — as the hreflang generator on this site does, from a single list of language and URL pairs — makes the reciprocal links and the self-reference impossible to forget, because the tool emits them for you.

If your site is monolingual, the right move is to declare nothing and spend the effort on content. hreflang is for sites that actually have equivalents to offer.

Limitations

60 pages from 78 hand-picked high-traffic domains — representative of well-known sites, not of the web. A random sample would be dominated by monolingual sites and would show lower usage. We parsed only the initial HTML of the homepage, so annotations delivered through HTTP headers, sitemap entries, or injected by client-side JavaScript on a single-page app are not counted. A site whose hreflang lives only in its sitemap would appear here as not using it. Where a specific site is named, the raw HTML we retrieved is the arbiter; corrections are welcome.

Reproduce it

The script and the shared domain list ship with this site and store the raw HTML for every domain, so every claim above can be checked against the source rather than against our summary of it.

What 55 Homepages Actually Do With Canonical

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

How this was measured

On 2026-09-24 we fetched https://<domain>/ for 78 well-known domains spanning news, commerce, SaaS, developer tooling, social, education, finance and government. 55 returned a usable page: 34 over a direct connection and 21 over a local proxy, because a number of these domains are unreachable from our network without one.

The other 23 could not be read. 14 answered with something other than 200 — mostly 403, one 401, one 202, one timeout — and 20 served a bot check or an interstitial instead of a page; the two groups overlap. One is worth naming: amazon.com answers 200 with a 2 KB page whose entire content is a bm-verify meta-refresh, not a homepage. Our first pass counted it as a real page and would have recorded Amazon as a site with no canonical. Re-reading the response is what caught it, and it is why every number below was checked against the raw HTML before it was written down.

For each page we took the <link rel="canonical"> from the initial HTML, resolved it against the URL we actually landed on after redirects, and compared the two after normalising: lowercased host, www stripped, trailing slash and index.html removed. We also captured the HTTP Link header, counted how many canonical tags each page ships, and recorded whether the tag sits before </head>. Then we made a second request to each canonical URL with redirects disabled, to see whether it points at a page that exists.

Finding 1: one homepage in five declares nothing

44 of the 55 pages (80%) declare a canonical. The other 11 declare none:

reddit.com, kubernetes.io, python.org, rust-lang.org, netflix.com, wikipedia.org, mozilla.org, w3.org, archive.org, wikimedia.org, airbnb.com

Two of those need an asterisk. reddit.com and archive.org returned client-rendered shells — 8.4 KB and 1.9 KB with no content in them — so a canonical injected by JavaScript would be invisible to a method that reads only the initial HTML. We cannot say they have no canonical; we can only say none is served with the document.

The other nine are server-rendered and full-sized, and the absence is real. Wikipedia, Mozilla, W3C and Python.org are not sites that forgot a tag through carelessness; for a homepage that is the most-linked URL on the domain, the duplicate-selection problem canonical solves is largely solved by the link graph anyway. airbnb.com is the interesting one: it ships <meta id="english-canonical-url" content="">, a canonical slot that arrived empty.

What the omission costs is narrow but real. Without it, /, /index.html, /?utm_source=newsletter and the http variant are four separate URLs that a crawler must choose between. It is one line of HTML to remove the question.

Finding 2: you have to follow the redirect first

43 of the 55 homepages (78%) redirected at least once before serving content, and 35 of the 44 that declare a canonical (80%) did. Some are ordinary www normalisation; several land on a different host entirely:

gitlab.com → about.gitlab.com, spotify.com → open.spotify.com, notion.so → notion.com, zoom.us → zoom.com, linkedin.com → www.linkedin.cn, cnn.com → edition.cnn.com.

This is the most common way to misread your own canonical. If you compare the tag against the URL you typed rather than the URL you were finally served, every one of those six looks like a cross-domain canonical pointing somewhere suspicious. They are not — they are self-referencing canonicals on the page that actually exists. When you audit, audit against the final URL after redirects, which is what a crawler sees.

Finding 3: six of 44 point anywhere other than themselves

38 of the 44 canonicals (86%) resolve to exactly the URL we were served. The six that do not are the whole story of this survey:

cnn.com — served edition.cnn.com, canonical https://www.cnn.com, which returns 200. This is geo consolidation done deliberately: the regional edition defers to the global URL.

linkedin.com — served www.linkedin.cn, canonical https://business.linkedin.com/zh-cn/zh-cn, 200. The same pattern on a different host.

tumblr.com — served /, canonical https://www.tumblr.com/explore/trending, 200. The logged-out homepage canonicalises to the trending feed rather than to itself.

stanford.edu — served /, canonical https://www.stanford.edu/home, which returns 404.

trello.com — served /, canonical https://www.trello.com/home, which returns 301.

mit.edu — served web.mit.edu, canonical https://tlecms.mit.edu/spotlight/zombie-cells: a different host, and a specific article about zombie cells. We could not reach that URL to check it — a direct request timed out and a proxied request failed the TLS handshake — so we report it as unverified rather than as broken. It has the shape of the classic template bug, where a homepage inherits a canonical slot filled from an article template, but we did not confirm it.

Finding 4: two canonical targets are not reachable pages

We managed to test 43 of the 44 canonical URLs. 41 returned 200 (95%). Two did not, and both belong to large, well-resourced sites.

Stanford's homepage tells a crawler that the preferred version of https://www.stanford.edu/ is a URL that answers 404. Trello's points at /home, which 301s — and in Trello's case the bare host redirects too, so the canonical points into a redirect chain rather than at a destination.

Neither is a syntax error, and nothing warns you. The tag is well-formed, the page renders, validators pass. The failure is silent in exactly the way canonical failures always are: a crawler that cannot resolve the canonical behaves as though you had not written one, which returns you to the duplicate-selection problem you wrote the tag to avoid. One command settles it for any URL you own:

curl -sS -o /dev/null -w "%{http_code}" https://example.com/your-canonical

The answer must be 200. Anything else is a canonical that does not do its job.

Finding 5: the mechanics are cleaner than the reputation suggests

Across 44 canonicals we found zero relative hrefs — every single one used an absolute URL. Zero http/https mismatches. Zero pages shipping two conflicting canonicals. Zero canonicals placed outside <head>.

Two near-misses are worth knowing. bbc.com ships the identical canonical tag twice, a harmless artefact of a framework rendering the head twice. moz.com is the only site in the sample to send a canonical in the HTTP Link header — and it also has the tag in its HTML, so the header is redundant rather than load-bearing.

Read alongside the two broken targets above, this is the useful shape of the result: nobody writes malformed canonicals any more. The failures are not in the syntax, they are in what the tag points at.

Finding 6: geo-personalisation is the real design decision

Seven of the 44 canonical URLs carry a language or region segment in the path. Four are region-personalised versions of what we asked for: stripe.com/cn, coinbase.com/en-ca, squareup.com/us/en, uber.com/ca/en. Because we fetched from a Chinese IP address, those are the variants we were served — and each of them self-canonicalises to the variant it served.

CNN made the opposite choice: it served us the regional edition and canonicalised back to the global URL. Both are defensible. What matters is that the choice is consistent, that it matches your hreflang annotations, and that one URL produces one canonical regardless of who asks. If your canonical varies with the visitor's IP, then a crawler geo-dispatched differently from your users will consolidate a different set of pages than the set your users actually see.

What to do with this

Emit an absolute, self-referencing canonical on every indexable URL. One in five famous homepages does not, and the ones that do are not doing anything clever.

Compare it against the final URL after redirects. 80% of the pages here redirected before serving content. Auditing against the URL you typed will manufacture cross-domain errors that do not exist.

Check that the canonical itself returns 200. Two of 43 did not. It is one curl command and no validator will do it for you.

Never point a canonical at a redirect. Point it at where the redirect lands.

Pick one geo policy and apply it everywhere. Consolidate globally like CNN, or self-canonicalise per locale like Stripe — but do not let the answer depend on who is asking.

Generate the tag rather than hand-writing it. Every failure above is a value problem, not a syntax problem, and values are what templates and generators get right. The meta tag generator on this site emits the canonical alongside the rest of the head from a single URL, so the value cannot drift away from the page it describes.

Limitations

55 pages from 78 hand-picked, well-known domains — representative of famous sites, not of the web. A random sample would look different in both directions.

The 23 unreadable pages are not a random loss. They are disproportionately large consumer and financial sites that fight crawlers — Amazon, GitHub, the New York Times, PayPal, Target, eBay — so the sample skews toward sites that serve their homepage to anyone who asks. That is a real bias, and it probably makes these numbers look tidier than the web as a whole.

We read only the initial HTML. A canonical delivered through JavaScript, a sitemap entry, or an HTTP header on a site that omits it from HTML is invisible here; two of the eleven "absent" pages were client-rendered shells where that caveat is live rather than theoretical.

One fetch, from one IP address, on one day. Because that address is in China we were served regional variants of several sites, which is what made Finding 6 visible — and it also means a different vantage point would produce a slightly different set of canonicals.

Reproduce it

The script and the shared domain list ship with this site. node scripts/survey-canonical.mjs fetches and stores the raw HTML for every domain; --report re-derives every number from that stored HTML without touching the network, and --evidence prints the raw <link> tag behind each verdict. Our first robots.txt survey covers the same domain list from a different angle.

Sitemaps in the Wild: 55 of 78 Sites

任宏涛 — Tue, 22 Sep 2026 00:00:00 +0000

How this was measured

On 2026-09-29 we requested https://<domain>/sitemap.xml for the same 78 hand-picked domains used by our other surveys — news, commerce, SaaS, developer tooling, social, streaming, reference, SEO, education, finance, cloud, local and government — then read each domain's robots.txt and followed any Sitemap: line it declared. Every request was attempted twice, once directly and once through a local proxy, because a number of these domains are unreachable from our network without one.

A file counted as a sitemap only if it returned 200 and the body was XML rather than HTML or empty. Five of them arrive as gzip files (.xml.gz), which curl --compressed does not decompress — that flag handles transport encoding, not a file that is itself a .gz. Those five had to be gunzipped before they could be read at all, and our first pass stored them as binary garbage.

Then the part that is not usually measured. From each of the 55 sitemaps we extracted five URLs at even intervals and requested them: 251 requests in total. Every 3xx was re-requested with a real GET before being counted, because the first pass used HEAD to save bandwidth and HEAD does not always answer the same way GET does.

Finding 1: 55 of 78 serve one, and 25 of those are not where you would look

55 of 78 domains (70.5%) serve a readable sitemap. But only 30 of those 55 answer at /sitemap.xml. The other 25 are reachable only by following the Sitemap: line in robots.txt. If you audit sitemap coverage by requesting /sitemap.xml, you undercount by 45%.

The declared locations are not variations on a theme. trello.com points at https://a594014.sitemaphosting7.com/4704043/sitemap_4704043.xml — a third-party sitemap hosting domain, not a Trello URL at all. netflix.com uses /sitemap/index, with no file extension. wise.com uses /sitemap. wikipedia.org points at a MediaWiki REST endpoint, /w/rest.php/site/v1/sitemap/0, which answered 403 to us. bbc.com declares https://www.bbc.com/afrique/sitemap.xml, its French edition. ebay.com declares /lst/AUCTION-0-index.xml.

The 23 that did not produce a file break down as follows. 12 return a clean 404 : etsy.com, linkedin.com, stackoverflow.com, python.org, rust-lang.org, go.dev, mozilla.org, w3.org, wikimedia.org, mit.edu, yelp.com and wikipedia.org. 4 return 200 with an HTML page : reddit.com (an 8 KB app shell), pinterest.com (1.36 MB of it), khanacademy.org (a 3 KB bot challenge) and cdc.gov, which redirects /sitemap.xml to /index.html and serves the homepage with a 200. 1 returns 200 with an empty body : x.com. 5 answer with an error : amazon.com 500, github.com 406, quora.com 403, npmjs.com 403, searchengineland.com 403. One, washingtonpost.com, was unreachable on both paths.

Finding 2: what robots.txt promises is not always what is there

74 of 78 domains serve a readable robots.txt; 55 of them declare a sitemap; 50 of those declarations resolve. Five do not:

washingtonpost.com declares a .xml.gz we could not reach. x.com declares /sitemap.xml, which is the empty 200 above. pinterest.com declares a seasonal-events file — seasonal_events_sitemap_20261129_first_sunday_of_advent_www.pinterest.com.xml — that returns 404, a generated reference left behind after the file it pointed at was retired. wikipedia.org declares the REST endpoint, 403 to us. searchengineland.com declares /sitemap_index.xml, also 403.

The failure runs the other way too. 5 sites serve a working sitemap and never declare it : shopify.com, gitlab.com, ahrefs.com, screamingfrog.co.uk and nih.gov. A crawler that only reads robots.txt misses those entirely.

Finding 3: most large sites ship an index, and 5 of them ship it gzipped

38 of the 55 are sitemapindex files pointing at children; 17 are flat urlset. The median index references 23 child files and is 2,584 bytes. The median flat urlset carries 617 URLs in 449 KB. The largest single file is gitlab.com at 19,009 URLs in 2.95 MB — inside the 50,000 URL / 50 MB limit, but one more product section away from needing to split.

Five serve their sitemap as a gzip file: nytimes.com, target.com, twitch.tv, vimeo.com and airbnb.com. Two omit the <?xml declaration entirely: theguardian.com and wise.com. Both still parse; neither needed to take the risk.

Four point at hosts other than their own. paypal.com declares 1,214 child sitemaps, all on www.paypalobjects.com, its CDN. notion.so lists 169 children on www.notion.com. nodejs.org includes three URLs on openjsf.org domains. youtube.com includes three on about.youtube and sibling hosts. who.int is the outlier: it declares its index over http:// and references children like http://www.who.int/SiteMaps/sitemap_static.xml.gz, while the site itself serves HTTPS.

Finding 4: lastmod is the field sites get wrong

18 of 55 sitemaps (33%) carry no lastmod at all. Three carry it inconsistently: asana.com on 611 of 3,487 entries, squareup.com on 5 of 16, and nih.gov on 2,688 of 2,689 — one entry short of complete.

Where the field exists, it is mostly date-only: 34,952 date-only values against 18,593 with a time. Date-only is valid and is what most of this sample chose; the ones carrying microseconds (cnn.com at 2026-09-28T23:24:32.775000+00:00) are the exception.

One file carries a value that is not a date. searchenginejournal.com lists -0001-11-30T00:00:00+00:00 alongside real entries like 2026-09-28T20:48:12+00:00. That is a zero-value timestamp leaking out of a templating layer — we cannot say which stack produced it, but a year of -0001 is not a date anyone set. It has the right shape, so a length check passes it, and it tells a crawler the page was last modified two thousand years ago.

Meanwhile 8 of 55 still emit changefreq and 6 still emit priority — fields Google has said publicly it ignores. The volumes are not small: nodejs.org writes 1,651 changefreq values, nih.gov writes 2,689 of them and 1,894 priority values, cloudflare.com writes 910 of each, shopify.com 557, theguardian.com 423, trello.com 112, stanford.edu 27, bbc.com 5.

Finding 5: one in eight sampled URLs does not answer 200

This is the number that matters, and it is the one nobody publishes. Of 251 URLs sampled from 55 sitemaps :

178 (70.9%) returned 200. 27 (10.8%) returned a redirect, confirmed by re-requesting with GET. 3 (1.2%) returned 404. 43 (17.1%) returned 401, 403 or 406 — and those are not sitemap faults. Six domains blocked every URL we tried: nytimes.com, reuters.com, bloomberg.com, ebay.com, coinbase.com and uber.com. Paywalls and bot walls, excluded rather than counted against them.

Excluding those six leaves 221 URLs: 178 (80.5%) returned 200, 27 (12.2%) redirected and 3 (1.4%) were dead. Put another way: 35 of the 55 sitemaps returned 200 for all five samples, and 12 of the 55 had at least one URL that redirects or 404s.

What the redirects are is more useful than how many there are:

walmart.com: 5 of 5. Every sampled /cp/ URL 301s to a /browse/ URL — /cp/blue-party-supplies/4514453 → /browse/party-occasions/blue-party-occasions/2637_1042319_1895000_4514453. That is a category URL migration the sitemap never absorbed.

stripe.com: 5 of 5. /legal → https://stripe.com/cn/legal. A geo redirect: we fetched from a Chinese IP address. The URL in the sitemap is not the URL a crawler in another country receives, and not the one that gets indexed.

zoom.us: 4 of 5. explore.zoom.us/es/customer_stories/eunis2020/ → www.zoom.com/es/customer-stories/all/. A domain migration, and one path changed shape on the way (/products/zoom-phone/ → /products/voip-phone/).

semrush.com: 4 of 5. /api-analytics/ → developer.semrush.com/api/v3/analytics/basic-docs/ — off the main domain entirely.

paypal.com: 2 of 5. /ae/webapps/mpp/mobile-apps?locale.x=ar_AE → /ae/digital-wallet?locale.x=ar_AE. The path was replaced; the query string survived.

cloudflare.com and stanford.edu: trailing slashes, in opposite directions. /case-studies/fossil 301s to /case-studies/fossil/; /health-medicine/ 308s to /health-medicine. Neither site agrees with the other about which form is canonical, and the sitemap disagrees with both.

The three genuine 404s: gitlab.com/gitlab-org/security-products/dependencies/golang/toml/-/issues, and two legacy release posts at nodejs.org/en/blog/release/v08.20 and /v2013.0.

Redirects outnumber dead URLs nine to one. That ratio matters because the two failures need different fixes. A 404 is a deletion — remove the entry. A redirect is a stale reference you have to trace to its target and then decide whether the sitemap or the URL is the thing that should change.

What to do with this

Audit the URLs, not the file. A sitemap that parses cleanly tells you nothing about whether its contents resolve. Sample ten URLs from yours and request them. Twelve of the 55 sitemaps here would have failed that test, and every one of them is a sitemap that "works".

Do not assume /sitemap.xml. Read the Sitemap: line in your robots.txt and request that URL. Twenty-five of the 55 sitemaps in this sample live somewhere else, and five of them are on paths with no .xml extension. Our robots.txt tester reads the file and shows you which directives a crawler would apply.

Regenerate after every URL migration. Walmart's /cp/ entries are what an un-regenerated sitemap looks like at scale: every sampled URL pointing at a structure that no longer exists. The redirect saves the user; it does not save the crawl budget.

If you geo-redirect, test from more than one country. Stripe's /cn/legal is a correct redirect doing its job, and it means the URL in the sitemap is not the URL that gets indexed from most of the world.

Pick a trailing-slash convention and make the sitemap match it. Cloudflare and Stanford chose opposite conventions, and both sitemaps disagree with their own site.

Either put lastmod on every entry or on none. A third of this sample ships none and is not visibly worse off. Three ship partial coverage, which is the version with no upside.

Drop changefreq and priority. Eight sites here write thousands of values Google ignores. If you serve .xml.gz, remember that the file is gzip, not just the transfer.

Limitations

78 hand-picked well-known domains, not a random sample of the web. Categories are our own manual grouping, several contain five domains, and one site moves a rate by 20 points.

All requests came from one IP address in China on one day, and that is directly visible in the data: 43 of 251 sampled URLs answered 401, 403 or 406, six domains blocked everything we tried, and Stripe redirected us to its /cn/ edition. Those are excluded rather than counted as defects. A different vantage point would find more 200s and fewer redirects.

Five URLs per sitemap, evenly spaced. A sitemap with one bad URL among 19,009 — GitLab's case — has roughly a 0.03% chance of showing it in our sample. These numbers describe the sample, not the full files, and the true per-sitemap defect rate is certainly higher than 12 of 55.

17 of the 38 child sitemaps we fetched were truncated at 400 KB before parsing, so their entry counts are lower bounds.

The 3xx count was confirmed with GET, but the initial pass used HEAD wherever it worked. Six URLs that looked like redirects turned out to be 403s on GET, and none turned out to be 200s — so the confirmed figure is lower than the raw one. If another host answers HEAD differently, this method over-counts redirects.

We did not follow redirect chains and did not check whether the redirect target is itself canonical, so "redirects" here includes both harmless normalisation and real migrations.

We read files discoverable from robots.txt or /sitemap.xml only. A sitemap submitted through Search Console and declared nowhere is invisible to us — and the five sites serving a sitemap with no declaration are proof that this configuration exists.

Reproduce it

The script and the shared domain list ship with this site. node scripts/survey-sitemap.mjs fetches and stores every response; --children pulls the first child of each index; --probe samples five URLs per sitemap; --recheck re-requests every 3xx with GET; --report re-derives every number from stored data without touching the network; --evidence prints the raw XML behind each verdict. Our robots.txt, canonical, hreflang, AI crawler and llms.txt surveys use the same domain list from different angles.

DEV Community: 任宏涛

Nine SEO checks to run before you publish a page

The problem with SEO checklists

1. Does the page have exactly one h1?

2. Do heading levels descend without skipping?

3. Is the title tag specific to this page?

4. Will the description survive truncation?

5. Does the canonical point at itself?

6. Is the page in the sitemap, in the right URL form?

7. Are the internal links real and useful?

8. Do images have dimensions and alt text?

9. Does the structured data describe this page accurately?

What is deliberately not on this list

I checked the Open Graph tags on 62 homepages. Here's what's actually broken.

Method, and the two counting rules that changed the numbers

Finding 1: Open Graph is near-universal. Twitter cards are not.

Finding 2: two sites ship an og:image tag with nothing in it

Finding 3: only 23% declare image dimensions

Finding 4: one in four og:title disagrees with <title>

Finding 5: the things I expected and didn't find

What to check on your own tags

Your JSON-LD is duplicated in three places. Here's the fix.

The pattern almost everyone starts with

What @id actually does

The shape that works

Three ways this goes wrong

Dates deserve their own section

Checking your markup

Why Google Won't Index Your Page: A Checklist

Two problems that look identical and are not

Step 1 — read the status code, not the page

Step 2 — is it blocked?

Step 3 — can it be discovered?

Step 4 — is it a duplicate?

Step 5 — is the content actually there?

Step 6 — now it is a trust gate, and there is nothing to fix

What "Request indexing" does, and does not do

The order, in one table

What 59 Homepages Put in Their SERP Snippet

How this was measured

Finding 1: every homepage has a title, and the median is 35 characters

Finding 2: the shorter the title, the bigger the brand

Finding 3: nearly everyone writes a meta description, so that advice is stale

Finding 4: descriptions cluster at 114 to 150, and the 160 guideline holds up

Finding 5: pipe is the most common separator, but no separator is more common

Finding 6: your title is not one thing

What to do with this

Limitations

Reproduce it

robots.txt Mistakes That Silently Deindex Pages

1. Disallowing a page you also marked noindex

2. Blocking CSS and JavaScript

3. Assuming order decides priority

4. Wildcard patterns that over-match

5. Forgetting that staging is one typo away from public

A five-minute check that catches all of these

Meta Description Length: What Gets Truncated

Where the 155-character rule comes from

The two widths that matter

When Google ignores your description entirely

What is not worth doing

A working method

What 61 Homepages Actually Ship in JSON-LD

How this was measured

Finding 1: about half ship nothing in the HTML

Finding 2: 14 sites still ship markup for a retired feature

Finding 3: a single block is the norm, and @graph is the minority

Finding 4: most markup is boilerplate

Finding 5: the pattern at full size

What we did not find: almost no dangling references

What to check on your own markup

Limitations

Reproduce it

Redirect Chains in the Wild: 78 Domains

How this was measured

Finding 1: 62 of 78 reach a page from the worst possible entry

Finding 2: two hops is the most common answer, and 28 of them are avoidable

Finding 3: 14 sites use a temporary redirect for something permanent

Finding 4: six sites serve two reachable homepages

Finding 5: eight homepages are not at /

1. Does the page have exactly one `h1`?

Finding 2: two sites ship an `og:image` tag with nothing in it

Finding 4: one in four `og:title` disagrees with `<title>`

What `@id` actually does