DEV Community: tong zhang

Chrome got zstd for the HTML and gzip for the CSS. Brotli was 279 KB smaller.

tong zhang — Fri, 13 Nov 2026 02:18:26 +0000

The page is 7.7 KB. The stylesheet it loads is 312 KB. Both came from the same host, one second apart, with the same request headers.

The HTML arrived as zstd. The CSS arrived as gzip — even though the same Accept-Encoding header listed brotli, and the same edge was willing to send brotli for the HTML. Asking for brotli alone on that stylesheet returns 33 KB.

So a browser that offers every encoding the server supports downloads 312 KB where 33 KB was available. That is one file, on one of 114 sites I run. I spent an afternoon measuring where else this happens.

What I actually measured

Two datasets, both taken today:

684 HTML responses — 114 sites × 6 Accept-Encoding variants.
492 assets — the same-origin CSS and JavaScript referenced by those 114 homepages, each fetched three ways (browser header, brotli only, identity).

The whole method is one idea: read the bytes, do not trust the label. No decompression library, no requests magic, no CDN analytics. len(body) after reading the raw stream is the number the user pays for.

import ssl, urllib.request

CTX = ssl.create_default_context()
CTX.check_hostname = False
CTX.verify_mode = ssl.CERT_NONE

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36")

def wire_bytes(url, accept_encoding):
    req = urllib.request.Request(url, headers={
        "User-Agent": UA,
        "Accept": "*/*",
        "Accept-Encoding": accept_encoding,
    })
    with urllib.request.urlopen(req, timeout=30, context=CTX) as r:
        body = r.read()
        return {
            "encoding": r.headers.get("content-encoding"),
            "wire": len(body),
            "vary": r.headers.get("vary"),
            "cache": r.headers.get("cf-cache-status"),
        }

Three details matter more than they look. read() with no argument, because a truncated read makes big files look identical. No brotli library installed, because I do not need to decode anything. And a browser User-Agent, because some of these hosts behave differently for unknown clients.

The HTML got zstd, all 114 of them

Chrome's real header is gzip, deflate, br, zstd. Fonts and assets offered the same set. For HTML, all 114 sites answered zstd:

`Accept-Encoding` sent	Encoding returned	Sites
`gzip, deflate, br, zstd`	zstd	114 / 114
`gzip, deflate, br`	br	114 / 114
`br`	br	114 / 114
`gzip`	gzip	114 / 114
`identity`	none (raw)	114 / 114
`br;q=0, gzip;q=0, deflate;q=0`	none (raw)	114 / 114

Two things to like here. identity is honoured everywhere, so the servers are not compression bullies. And refusing every coding with q=0 returns a plain 200 with raw bytes rather than a 406, which is the forgiving behaviour you want from a CDN.

The part I did not expect: zstd loses on size. I fetched every homepage three times and compared the wire bytes per encoding.

Encoding	Median bytes	Median share of raw	Total across 114 pages
raw (`identity`)	75,220	100%	9.70 MB
gzip	10,961	16.4%	—
br	10,855	15.6%	1.30 MB
zstd	11,710	16.4%	1.37 MB

Brotli beat zstd on 113 of 114 homepages. Median difference: 566 bytes, 6.3%. Across all 114 pages, zstd cost 72,493 bytes more than brotli would have — 636 bytes per page. Small, but it is a real tax paid for choosing the newer algorithm: at this compression level, the newest codec is the loosest one.

The exception is worth keeping: one homepage ships a 666,877-byte Next.js document with a lot of repeated JSON, and there zstd won by 5.8% (45,740 vs 48,554 bytes). Repetition-heavy payloads are exactly where zstd's window pays off. Brotli is not universally better; it is better on prose-shaped HTML.

The assets got gzip, 480 out of 492

This is where the numbers stop being a rounding error.

I pulled every same-origin stylesheet and script out of those 114 homepages, deduplicated by URL, and fetched each one three times. 480 of 492 came back gzip when the client offered brotli and zstd. Twelve came back zstd. Zero came back brotli.

Asset type	Files	Raw	Delivered to Chrome	With brotli	Saved
CSS	156	15.19 MB	3.58 MB	2.00 MB	1.58 MB
JavaScript	335	45.07 MB	13.16 MB	13.10 MB	0.06 MB
Total	492	60.31 MB	16.75 MB	15.11 MB	1.64 MB

Nine point eight percent of the bytes every visitor downloads, available for free, not taken.

The distribution is lopsided. Eight CSS files account for 80% of the entire saving, and almost all of it sits in one filename that recurs across the cluster:

$ curl -s -D - -o /dev/null -H 'Accept-Encoding: gzip, deflate, br, zstd' \
    https://spacexstockprediction.com/static/css/css2.css | grep -i content-encoding
content-encoding: gzip        # 312,302 bytes

$ curl -s -D - -o /dev/null -H 'Accept-Encoding: br' \
    https://spacexstockprediction.com/static/css/css2.css | grep -i content-encoding
content-encoding: br          # 33,328 bytes

Same URL, same cache entry, same minute. The difference is 279 KB per page load on that host, and the same pattern repeats on 15 sites in the cluster.

What is inside a 1.09 MB stylesheet

I fetched it with identity to see what was being compressed so poorly. It is 1,094,451 characters of @font-face rules for Noto Sans SC:

@font-face blocks : 1,010
unicode-range     : 1,010
url() references  : 1,010   (all self-hosted /static/font/*.woff2)
font-display      : swap

This is the Google Fonts CJK slicing pattern, copied in-house: a hundred-odd unicode-range subsets per weight, several weights. The page that loads it, a SpaceX explainer on the worst offender, contains 223 Han characters. The browser will only fetch the few subsets it needs for those 223 characters, so the fonts themselves are not the problem — the problem is that 1.09 MB of rules has to cross the wire and be parsed before the browser knows which subsets exist.

And this cluster has a language mismatch that makes it worse: there are Chinese-language guides served from an English-language template, which is exactly the setup that ends up shipping a full CJK font manifest to every visitor.

Cheap fix: that file does not change between deploys. Compress it once with brotli at build time, serve the .br variant with Content-Encoding: br and Vary: Accept-Encoding. 1.09 MB becomes 33 KB for every browser released this decade.

Where brotli does not help at all

I expected the JavaScript column to look like the CSS column. It does not:

brotli beat gzip on 138 of 335 JavaScript files.
On 196 of 335 it was larger. Median difference: −0.4%.
Total: 13.16 MB → 13.10 MB. Half a percent.

Whatever level this edge compresses at, it is not the level where brotli's context modelling starts to matter on minified JS. So the honest summary is not "turn on brotli". It is "turn on brotli where the file is repetitive text, and measure the rest" — CSS yes, minified JS no, HTML marginally.

If you want a second opinion on any of this, pick a page from the same template, for example a market page from the same cluster, and compare its content-encoding against curl -H 'Accept-Encoding: br'.

The negotiation table that explains it

Probing one cached CSS file with different header combinations, same URL, all cf-cache-status: HIT:

Header	Returned	Bytes
`gzip, deflate, br, zstd`	gzip	250,677
`br, gzip`	gzip	250,677
`gzip, br`	gzip	250,677
`gzip, deflate, br`	gzip	250,677
`zstd, br`	zstd	35,999
`br`	br	33,163

It is not header order. Any header that mentions gzip gets gzip on the asset path; drop gzip and the same file is served as zstd; ask for brotli alone and you get brotli. Meanwhile the HTML path, on the same zone, prefers zstd and never falls back to gzip while brotli is on the table.

I cannot see the CDN configuration from outside, so I am not going to claim a cause. What I can say is that the asset path and the document path negotiate differently, in the same zone, in the same second, and that the asset path's choice is the expensive one for CSS.

Two hygiene findings, one good, one not

All 492 assets vary their bytes by Accept-Encoding, and none of them send Vary. All 114 HTML responses do. If anything between the edge and the client caches by URL alone — a corporate proxy, a service worker with a careless cache key, a reverse proxy you forgot about — it can hand brotli bytes to a client that never advertised brotli, and that client will render garbage. The edge's own cache key handles it; nothing downstream is told to.

The other one is smaller but universal: 114 of 114 responses have no Content-Length and use Transfer-Encoding: chunked. Every homepage in the cluster streams without a declared size. That is not a bug, but it removes the browser's ability to show progress or reserve space, and it makes byte-count auditing harder than it should be.

What I am doing about it

Precompressing the CJK stylesheets at build time and serving brotli variants with Vary: Accept-Encoding. That is 1.58 MB of the 1.64 MB, and it needs no configuration change on the edge.
Re-checking per type instead of globally. CSS gains 44% of its bytes; minified JS gains 0.5%.
Adding wire-byte assertions to the deploy check, comparing len(body) for gzip versus br on the three largest text files. A regression here is invisible in every dashboard I have.

The uncomfortable part is that this cluster has been measuring performance for weeks — TTFB, cf-cache-status, 304 rates, image weight — and every one of those measurements looked fine, because none of them asked what encoding the bytes arrived in. The page was fast. It was also 279 KB heavier than it needed to be, on a file that has not changed since the last deploy.

21 of our 114 sites answer every missing URL with the homepage. Search Console calls that indexed.

tong zhang — Wed, 11 Nov 2026 02:03:53 +0000

I asked 114 of my own sites for a page that does not exist. Sixty-four times, the server said yes.

Not "yes, here's an error page." Yes, here is a 200, here are 53,861 bytes, and here is the homepage — the identical bytes, down to the SHA-256 hash. Same title tag. Same <h1> count (zero). Same everything.

Then I pulled Search Console's page-indexing export and found 133 URLs marked Submitted and indexed across the same hosts, and I started to understand how a site can look healthy on every dashboard and rank for nothing.

What "doesn't exist" means when you're the one asking

The cluster is 114 sites I run. Mostly small static or edge-rendered content sites, all behind Cloudflare, all with a sitemap, all with Search Console properties. The kind of setup where you trust the status code because you never had a reason not to.

A missing URL in HTTP has one correct answer: 404. Or 410 if you're being precise about intent. What it must not be is 200, because 200 is a promise. It tells every crawler, every cache, every link checker, and every downstream consumer that the resource is there.

So I generated paths that could not possibly exist — a random hex token, three shapes per site — and asked each host for them.

import uuid

def probe_paths(host):
    tok = uuid.uuid4().hex[:8]
    return [
        f"https://{host}/this-page-does-not-exist-{tok}/",
        f"https://{host}/blog/missing-post-{tok}-9f2c/",
        f"https://{host}/{tok}/nested/gone.html",
    ]

Three hundred forty-two requests total. No retries, no bot user agents, plain Chrome UA, no Accept-Encoding header (so I could read the bodies without a brotli decompressor).

64 of 342 came back 200

Result	Count	Share
404	276	80.7%
200	64	18.7%
TLS handshake failure	2	0.6%

Eighty percent behaved. That is the number that made me relax for about ten seconds, until I grouped by host instead of by request.

21 sites, one behaviour: the homepage, byte for byte

Twenty-one of the 114 hosts answered every nonexistent path with a 200. Not with a rendered error template — with their own front page.

The giveaway was the byte count. Compare the body length of a fake path against the body length of / on the same host, and the two numbers match exactly. Match exactly, on a page that includes a timestamped market table and a rotating ticker, is not a coincidence. It is a cache hit or a rewrite.

I hashed them to be sure.

import hashlib, ssl, urllib.request

CTX = ssl.create_default_context()
CTX.check_hostname = False
CTX.verify_mode = ssl.CERT_NONE

def body(url):
    req = urllib.request.Request(url, headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                      "AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/131.0.0.0 Safari/537.36",
        "Accept": "text/html,*/*",
    })
    with urllib.request.urlopen(req, timeout=25, context=CTX) as r:
        return r.status, r.read()

def identical(host):
    s1, home = body(f"https://{host}/")
    s2, fake = body(f"https://{host}/zz-{uuid.uuid4().hex[:8]}/")
    return hashlib.sha256(home).hexdigest() == hashlib.sha256(fake).hexdigest()

Seven hosts tested this way, seven identical:

Host	Homepage bytes	Fake-path bytes	SHA-256 match
nextgemcrypto.com	53,861	53,861	yes
spacexstockanalysis.com	8,749	8,749	yes
openaistocknews.com	10,528	10,528	yes
openaipreipo.com	11,745	11,745	yes
openaiipostock.com	10,060	10,060	yes
howtobuyspacexstock.com	8,090	8,090	yes
bullrunalpha.com	21,053	21,053	yes

Here is the part that matters: these are not broken sites. On the same hosts, real URLs render real documents. bullrunalpha.com is a 116-page site, and a sampled article there ships 25,023 bytes of distinct markup with its own title — a coin page that does return its own document, which is exactly what makes the fallback so easy to miss. spacexstockinsights.com behaves the same way: its secondary-market pricing page returns 15,040 bytes against a 13,298-byte homepage, close enough in size to look right in a log and far enough apart to be a genuinely different page.

The bug is not "the site is down." The bug is "any unknown path silently becomes the front page."

It isn't cloaking

Cloaking would be a different, more serious finding: a different response for Googlebot than for a browser. I tested for it, because 21 sites sharing an exact behaviour smells like a shared template, and shared templates sometimes carry shared hacks. The matrix ran against the 19 hosts where all three probe requests completed; two others dropped a TLS handshake mid-run, and I left them out rather than retry my way into a different sample.

Six request variants against all 19 hosts I deep-tested:

Request	soft-404 group	control group
Chrome UA, trailing slash	19 × 200	10 × 404
Googlebot UA	19 × 200	10 × 404
Bingbot UA	19 × 200	10 × 404
Chrome UA, no trailing slash	19 × 200	10 × 404
Chrome UA, `.html` suffix	19 × 200	10 × 404
Chrome UA, four segments deep	19 × 200	10 × 404

The control group is ten hosts drawn at random from the 92 that returned clean 404s. Every variant, every host, both directions. No UA sniffing, no cloaking, no exceptions.

That uniformity is worse than cloaking in one specific way: a cloaking hack at least suggests someone was paying attention.

The fallback survives every URL shape

The trailing-slash and .html rows above are the ones I expected to break. Static hosts usually treat /missing and /missing/ as different keys, and a rewrite that matches one often misses the other.

Not here. Whatever the mechanism, it matches at the routing layer, not the filesystem layer, and it matches greedily. Four-segment paths. Paths ending in .html. Paths with a hex token where a slug should be. All 200, all the homepage.

The one exception in the whole set is instructive. coinwikipro.com returns 200 for missing paths too, but it returns a 10,269-byte page titled 404 · 页面未找到 — someone did wire up an error page, and then served it with a 200 status and the homepage's <h1> and body underneath. It knows the page is missing. It tells you in the title. It just tells the HTTP layer something else.

Then I checked the sitemaps

A 200 on a made-up URL is embarrassing. A 200 on a URL you declared in your sitemap is expensive, because that URL was supposed to be a real page.

So for all 19 soft-404 hosts and the 10 controls, I pulled robots.txt, followed the Sitemap: directive, expanded one level of sitemap index, and sampled six declared page URLs per host.

Group	Sampled URLs	Returned homepage bytes	Share
soft-404 hosts	114	61	53.5%
control hosts	54	2	3.7%

Fifty-three percent of the URLs these sites declare as content are serving the front page. The control group — same cluster, same templates, same Cloudflare account — sits at 3.7%.

Five hosts scored six out of six: nextgemcrypto.com, openaiipostock.com, openaipreipo.com, openaistocknews.com, spacexstockanalysis.com. Between them they declare 4,868 URLs in their sitemaps.

Partial collapses are the harder ones. howtobuyspacexstock.com scored four out of six: /investor-resources/, /how-to-buy/ and /stock-price/ all returned the 8,090-byte homepage, while its guide to the routes that actually exist for buying SpaceX stock still ships 22,217 bytes of its own markup. Half the sitemap works, which is precisely why nobody would go looking.

4,868 declared URLs, one real page

spacexstockanalysis.com is the cleanest example. Its sitemap lists 17 URLs. Every one I requested came back as the same 8,749 bytes:

200  8749B  https://spacexstockanalysis.com/spacex-stock-symbol/
200  8749B  https://spacexstockanalysis.com/spacex-stock-price/
200  8749B  https://spacexstockanalysis.com/spacex-stock-history/
200  8749B  https://spacexstockanalysis.com/spacex-stock-secondary-market/
200  8749B  https://spacexstockanalysis.com/spacex-stock-starlink/

Seventeen URLs. One document. The sitemap says "here are seventeen things." The server says "here is the same thing, seventeen times."

That page is also where the canonical tag gets interesting. Its <link rel="canonical"> points at https://spacexstockanalysis.com/ — the homepage. Which is honest, in a way. But the same document also carries:

<meta name="robots" content="index, follow, max-snippet:-1,
      max-image-preview:large, max-video-preview:-1">

Index me. Follow me. But canonicalize me to a different URL. The page is arguing with itself in two adjacent <head> lines.

nextgemcrypto.com is worse, because it doesn't even offer a canonical escape hatch. Four thousand seven hundred twenty-seven URLs in the sitemap, zero <link rel="canonical"> anywhere in the 53,861-byte document that answers all of them.

Search Console calls this "indexed"

This is the part that changed how I read coverage reports.

Across the 19 soft-404 hosts, the page-indexing export GSC gives you (15 URLs sampled per property) contained 133 rows marked Submitted and indexed. I fetched 68 of them — five per host where available — and hashed each response against its host's homepage.

Checked	Byte-identical to homepage	Share
68 URLs	39	57.4%

Fifty-seven percent of the URLs Google called indexed were serving me the homepage, byte for byte, with an index, follow directive and no canonical pointing anywhere else.

openaistocknews.com is the extreme: 15 of 15 sampled URLs covered, and all five I fetched returned the same 10,528 bytes. GSC says fifteen pages indexed. The site has one page.

$ curl -s https://openaistocknews.com/openai-sec-filing | wc -c
10528
$ curl -s https://openaistocknews.com/nvidia-ai-stock  | wc -c
10528
$ curl -s https://openaistocknews.com/chatgpt-ipo-price | wc -c
10528

"Indexed" in Search Console means Google stored something for that URL. It does not mean the URL resolves to a distinct document, and it definitely does not mean the URL can appear in results. Once the canonical cluster collapses fourteen URLs into one homepage, you have coverage numbers and no rankings, and nothing in the UI connects those two facts.

The distribution it does show, if you go looking:

Coverage state	URLs
Submitted and indexed	133
URL is unknown to Google	82
Discovered – currently not indexed	48
Crawled – currently not indexed	13
Alternate page with proper canonical tag	5
Duplicate, Google chose different canonical	2
Page with redirect	1

Those last three rows are the collapse, itemised. Five pages Google decided were alternates of something else. Two where Google overrode the site's own canonical choice. And 133 where it stored a homepage under a URL that was supposed to be an article.

Four signals, four answers

For a single URL like spacexstockanalysis.com/spacex-stock-symbol/, the stack says four different things at once:

Layer	Signal	Claim
HTTP	`200 OK`	this resource exists
Sitemap	`<loc>` entry	this is a distinct page, please crawl it
`<meta name="robots">`	`index, follow`	please index this page
`<link rel="canonical">`	`https://…/`	this is actually a different page

A crawler has to pick one. Google's answer, broadly, is: fetch it, see the body is identical to another URL, follow the canonical, and mark the loser as an alternate. You keep your 200s, lose the URLs, and never see an error anywhere.

That is a hard failure mode to notice, because every surface reports success. The status code is success. The sitemap submitted cleanly. The coverage report is mostly green. Only the SERP disagrees, and the SERP doesn't have a dashboard.

Why this happens

The mechanism is almost always a rewrite rule that was written for a single-page app and never revisited when the site became something else.

Netlify and Cloudflare Pages both accept this in _redirects:

/*    /index.html    200

Fifteen characters of intent, and every URL on the host forever answers with the app shell. It is the correct rule for a client-routed SPA with no server-side rendering, where the shell genuinely is the response and JavaScript resolves the view. It is wrong the moment your pages are server-rendered, statically generated, or — as in spacexstockanalysis.com — supposed to be seventeen distinct documents.

A Next.js catch-all does the same thing without a config file:

// app/[...slug]/page.js  -- catches everything, renders the homepage
export default function CatchAll() {
  return <HomePage />;
}

Nobody writes that on purpose. It gets written as a placeholder, ships, and then quietly outlives its purpose by a year.

The fix is smaller than the bug

Three lines, depending on your host.

For a static host with a real 404 document, delete the catch-all line and ship the file:

/*    /index.html    200    <-- delete this, ship a 404.html at the root

For nginx, make the fallback terminal:

location / {
    try_files $uri $uri/ =404;
}

For Next.js, call notFound() instead of rendering something:

import { notFound } from "next/navigation";

export default async function Page({ params }) {
  const doc = await getDoc(params.slug);
  if (!doc) notFound();      // renders not-found.js with a real 404
  return <Article doc={doc} />;
}

Then verify it the way a crawler would, not the way your test suite would:

def audit(host, paths):
    _, home = body(f"https://{host}/")
    h = hashlib.sha256(home).hexdigest()
    bad = 0
    for p in paths:
        status, payload = body(f"https://{host}{p}")
        if status == 200 and hashlib.sha256(payload).hexdigest() == h:
            bad += 1
    return bad, len(paths)

If that returns anything but (0, n), you have a soft 404.

What I changed

I fixed the routing on the five six-out-of-six hosts first, because a sitemap that declares 4,868 URLs and resolves to one document is not a sitemap, it is a liability. Then I regenerated the sitemaps from the actual route table rather than from a slug generator, which is how nextgemcrypto.com ended up with 4,727 /keyword/ URLs that never had a page behind them.

The remaining hosts still have the fallback. I'm leaving them for a week to see whether the 404s change the coverage numbers — that's the part I don't have data for yet, and I'd rather report the before/after than guess.

Two things I'd check on any site this week, both cheap:

First, request a URL you know is missing and look at the status code, not the page. A 200 with a nice error message is still a 200.

Second, take five URLs from your sitemap and hash them against your homepage. If any two match, you have URLs that Google will crawl, index, and never rank — and no report anywhere will tell you.

The full probe took 68 seconds for 114 hosts. It found 21 sites answering every missing URL with the front page, 61 of 64 measured 200s that were byte-identical to a homepage, and a coverage export that marked 133 such URLs as indexed. The status code was never going to tell me any of that. I had to read the bytes.

Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.

tong zhang — Tue, 10 Nov 2026 02:17:46 +0000

114 sites, one robots.txt template, 31 banned user agents. I probed 7 of them and got three different answers.

The network I audit runs 114 sites behind the same edge. Every one of them serves a robots.txt. I had never read one end to end, because I assumed the file was boring. It is 2,756 characters at the median and half of it is a legal notice.

So I pulled all 114, parsed them, then did the thing robots.txt never gets: I tested whether the bans actually fire.

The file is half legal notice

All 114 returned 200. All 114 are text/plain. None of them accidentally served an HTML 404 page, which is where this kind of audit usually starts.

Here is the shape of a median file, by character count:

Section	Characters	Share
Comment block (EU DSM Article 4 reservation)	1,392	50.5%
`# BEGIN Cloudflare Managed content` marker	45	1.6%
User-agent groups and rules	1,230	44.6%
Trailing blank lines	89	3.2%

The comment block is not junk. It is a rights reservation under Article 4 of the EU Copyright in the Digital Single Market directive, explaining what search, ai-input, ai-train and use mean before the rules start. It is written for a lawyer, not for a crawler.

But the split matters for a different reason. When a file is 51% prose, nobody reads the other 49%. I know this because I wrote the deployment and I did not read it.

Thirty-one identical bans across 114 files

The union of banned user agents across all 114 files is 31. Each individual file contains exactly 31. Not 30, not 32 — the same 31 strings, in the same order, byte for byte, in all 114 files.

Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot,
Google-Extended, GPTBot, omgili, anthropic-ai, Claude-Web, cohere-ai,
MistralAI-Training, meta-externalagent, GoogleOther, Baiduspider,
PetalBot, AwarioRssBot, Google-CloudVertexBot, QualifiedBot, Cotoyogi,
ICC-Crawler, atlassian-bot, FishBot, BorderxBot, NavuBot,
SemrushBot-SWA, WARDBot, magpie-crawler, KimiBot, CitibotSiteCrawler

Each of those 31 gets the same two lines:

User-agent: Baiduspider
Disallow: /

That is a full-site ban. No path escaped, no rate limit, no partial allowance.

Read the list again. Roughly sixteen of those names are AI training or grounding crawlers, which is the stated intent. The rest are not. Baiduspider is Baidu's search crawler. PetalBot is Huawei's search crawler. GoogleOther is Google's miscellaneous fetcher, not Gemini training. atlassian-bot fetches link previews. SemrushBot-SWA is a site audit tool. ICC-Crawler is an academic index.

One list, two very different jobs, and the search crawlers are on the wrong side of it.

The wildcard group forbids nothing

Here is the part that surprised me more than the ban list. Every file has two User-agent: * groups — 113 of the 114 files, anyway. The first sits at the top of the managed block, right under the Content-Signal line. The second sits at the very bottom, after # END Cloudflare Managed content, where the site's own config gets appended.

Both of them contain exactly one directive:

User-agent: *
Allow: /

There is not a single Disallow in either wildcard group. Across all 114 files. So the effective policy for any crawler that is not one of the 31 named agents is: crawl everything, no restrictions, no crawl-delay.

The file is 2,756 characters long and imposes zero constraints on general crawling. It is a blocklist wearing the costume of a policy document.

Every file also carries the same content signal — search=yes,ai-train=no,use=reference — in all 114. So the machine-readable intent is consistent. The rules underneath it are not.

Two status codes, two different layers

Reading rules tells you what the operator intended. It does not tell you what the edge does. So I sent 14 user agents at 6 domains in the same minute, same URL, same headers apart from the UA string. The result was identical on all 6 domains.

User agent	Status	Response body
Chrome 131 (control)	200	77,104 bytes of HTML
Random crawler, not on any list (control)	200	full page
Googlebot	200	full page
Bingbot	200	full page
Applebot	200	full page
GoogleOther	200	full page
PetalBot	200	full page
GPTBot	403	`Your request was blocked.` (25 bytes)
ClaudeBot	403	`Your request was blocked.`
CCBot	403	`Your request was blocked.`
Bytespider	403	`Your request was blocked.`
Baiduspider	520	`error code: 520` (16 bytes)
AhrefsBot	520	`error code: 520`
SemrushBot	520	`error code: 520`

Two control probes matter here. The random crawler UA — a name that appears in no list anywhere — gets a 200 with the full page. So this is not a blanket "block all bots" rule that catches unknown agents by heuristics. It is name matching against a list.

And the two responses come from two different places:

403 is the edge deciding. The body is a 25-byte sentence written by the WAF rule. The request never reaches the origin.
520 is the edge reporting that the origin did not answer usefully. error code: 520 is Cloudflare's own body for "web server returned an unknown error". The request passed the edge and the origin returned nothing usable.

That distinction is the whole article. Same declared policy, two enforcement mechanisms, and they behave differently in ways that matter to a crawler.

Here is the probe, if you want to run it against your own sites:

import ssl, urllib.request, urllib.error

UA = {
    "chrome":    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                 "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "googlebot": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "baidu":     "Mozilla/5.0 (compatible; Baiduspider/2.0; "
                 "+http://www.baidu.com/search/spider.html)",
    "gptbot":    "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; "
                 "GPTBot/1.2; +https://openai.com/gptbot",
    "unknown":   "Mozilla/5.0 (compatible; ZzqRandomCrawler/1.0; +https://example.invalid/bot)",
}

ctx = ssl.create_default_context()

def probe(url, ua):
    req = urllib.request.Request(url, headers={"User-Agent": ua, "Accept": "text/html,*/*"})
    try:
        with urllib.request.urlopen(req, timeout=20, context=ctx) as r:
            body = r.read()
            return r.status, len(body), body[:40].decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        body = e.read()
        return e.code, len(body), body[:40].decode("utf-8", "replace")

for name, ua in UA.items():
    status, size, head = probe("https://example.com/", ua)
    print(f"{name:10} {status} {size:>7}B  {head!r}")

The unknown row is the one people skip. Without it you cannot tell a name-based blocklist from a generic bot heuristic, and those two have completely different consequences when you add a new crawler to your stack.

The bans that never fired

Of the 31 declared bans, I tested 7 by name. Five fired. Two did not.

PetalBot and GoogleOther are both named with Disallow: / in all 114 files. Both got a 200 and a full page. The declaration exists; nothing enforces it. If you are relying on robots.txt alone to keep either of them out, you are not keeping them out.

Then the reverse. AhrefsBot and SemrushBot are not in the list. The list has SemrushBot-SWA, which is the site-audit variant, but not the main SemrushBot. Neither name appears anywhere in the 2,756 characters. Both got a 520.

So the blocklist and the enforcement layer disagree in both directions. Things that were banned were let in. Things that were never mentioned were shut out. Whatever is generating the file and whatever is enforcing it at runtime are not reading from the same source, and nobody has reconciled them since the template was applied.

This is the same failure mode as a forecast that declares 70% confidence and delivers 62% — the declaration is not the behaviour, and the gap only shows up when you score outcomes instead of reading claims. I have been running that measurement on a set of forecasting experiments, and the method is identical: state the claim, then test it, then compare. A calibration experiment that scores stated confidence against realised outcomes makes the point faster than any audit report: the gap is usually in the places nobody instrumented.

A 520 is the wrong way to say no

If the goal is "do not crawl me", 403 is the correct answer. It is a 4xx: a permanent, explicit refusal. Crawlers treat it as an instruction and stop.

520 is a 5xx. To every crawler on the internet, a 5xx means "the server is broken, try again later". Google's documentation is explicit that persistent 5xx responses cause crawl rate reduction and, eventually, removal from the index. Baidu behaves the same way, with less patience.

So the sites that meant to block an AI crawler are instead telling Baidu that they are down.

I ran the Baiduspider probe three times against five domains, four seconds apart, to rule out a transient origin failure. Fifteen requests, fifteen 520s. Then I tried Baiduspider-render, the rendering variant: 520 as well.

Then I tried the sitemap. https://<domain>/sitemap-index.xml with the Baiduspider UA returned 520 too. Not the sitemap contents — the same 16-byte error line.

Whatever is producing that 520 is upstream of the routing, so it does not distinguish a homepage from a sitemap from a stylesheet. From Baidu's perspective the entire host is unreachable, and there is no diagnostic it can act on, because 520 carries no information about what went wrong.

The sites that got a clean 403 are in better shape than the ones that got a 520, and both of them are worse off than the ones that got a 200 when they should not have.

The sitemap was in perfect shape the whole time

Here is the part that stings. Of the 114 sites, 113 declare a Sitemap: line in robots.txt. I fetched all 113 and checked the root element, not just the status code — a 200 that returns HTML is a much more common failure than a 404.

113 out of 113 returned real XML. 83 were <urlset>, 30 were <sitemapindex>, all served as text/xml. Correct content type, correct root element, parseable, reachable with a normal user agent.

So the discovery chain was intact. The sitemap was well-formed, declared in robots.txt, and served correctly. The only thing standing between that sitemap and a search engine was a 520 that fires before routing.

That is the worst shape a technical problem can take: every component passes its own unit test, and the pipeline still delivers nothing. When I want to know what a system actually does rather than what its components claim, I go to the primary record — the way the Form D filings behind a funding number tell you more than any press summary of it. The status code is the primary record here. The robots.txt is the press release.

Checking your own network

Three checks, in order of how often they find something:

1. Does the ban fire? Take every agent in your robots.txt and probe your homepage with that exact UA string. Compare against a Chrome control and an unknown-crawler control. Three outcomes, three different bugs: 200 means the ban is decorative, 403 means it works, 5xx means it works in the worst possible way.

2. Is anything blocked that you never declared? The unknown-crawler control catches blanket rules. Then probe your analytics tools, your SEO tools, your uptime monitor. AhrefsBot and SemrushBot were being refused on 114 sites that never named them.

3. Is your wildcard group actually restrictive? Print it. If it is one Allow: / with no Disallow, your file is a blocklist and nothing else — and every new crawler that appears next quarter will walk straight in until someone adds it by hand.

from urllib.robotparser import RobotFileParser
import urllib.request

txt = urllib.request.urlopen(
    urllib.request.Request("https://example.com/robots.txt",
                           headers={"User-Agent": "Mozilla/5.0"}), timeout=20
).read().decode("utf-8", "replace")

rp = RobotFileParser()
rp.parse(txt.splitlines())

for agent in ("Googlebot", "Bingbot", "Baiduspider", "PetalBot", "GPTBot"):
    print(f"{agent:12} allowed={rp.can_fetch(agent, 'https://example.com/')}")

RobotFileParser alone gets you the declaration side in about ten lines. It will tell you PetalBot is banned. It will not tell you PetalBot walked in anyway, which is why the probe matters more than the parse.

What I am changing

Two fixes, in this order.

First the 520. Any refusal that returns 5xx for a named crawler should be a 403 or nothing at all. A 5xx is a lie about your own uptime, and search engines bill you for it in crawl budget before they bill you in rankings.

Second the list. Search crawlers and training crawlers should not share an allow/deny list, because they have opposite economics: one brings you traffic, the other takes your content. Baiduspider, PetalBot and GoogleOther come off the list. They do not train models. Keeping them in the same block as GPTBot costs index coverage and buys nothing.

The template itself is the harder problem. A managed block that is identical across 114 files is a good default and a bad policy — it encodes one vendor's opinion about which agents matter, and it ships that opinion to every property without asking whether the property agrees. The content layer already works this way: a coin page on one of the sites carries the same header, the same analytics script and the same caveats as the other 492 entries in that sitemap, because the template decided for all of them. Every site in the network inherited 31 decisions nobody made.

I have 114 sites and 3,534 declared bans across them. Five of the seven I tested were real. The other two were furniture, and the one that mattered most was answered with a 5xx.

Half the file was a legal notice. I should probably have read the other half sooner.

104 of 114 homepages ship no images at all. Their only image is a 270 KB social card.

tong zhang — Mon, 09 Nov 2026 02:17:19 +0000

I crawled the home pages of 114 sites that all belong to the same small publishing network — prediction-market and crypto-ticker properties, mostly Next.js and mostly Cloudflare. Every one answered HTTP 200. Between them, the HTML contains 727 <img> elements.

104 of those 114 pages contain none of them.

That number surprised me enough to re-run the crawl three different ways before I believed it: parsing <img> with a regex, checking for lazy-loading attributes and srcset, and then separately fetching every linked stylesheet to see whether the images were hiding in CSS instead. They mostly weren't. The median page is 60,452 characters of HTML with no image reference anywhere in it.

Here is the counting code, if you want to point it at your own site:

import re, urllib.parse

IMG_RE  = re.compile(r"<img\b([^>]*)>", re.I | re.S)
ATTR_RE = re.compile(r"""([a-zA-Z_:][-a-zA-Z0-9_:.]*)\s*=\s*("([^"]*)"|'([^']*)'|([^\s"'>]+))""")

def attrs_of(raw):
    d = {}
    for m in ATTR_RE.finditer(raw):
        v = m.group(3) if m.group(3) is not None else m.group(4) or m.group(5)
        d.setdefault(m.group(1).lower(), v)
    return d

def imgs(html, base):
    out = []
    for m in IMG_RE.finditer(html):
        a = attrs_of(m.group(1))
        src = a.get("src") or a.get("data-src") or ""
        if not src:
            continue
        out.append({
            "src": urllib.parse.urljoin(base, src),
            "w": a.get("width"), "h": a.get("height"),
            "loading": (a.get("loading") or "").lower(),
            "srcset": bool(a.get("srcset")),
        })
    return out

104 pages, zero images

Of the 114 pages, 104 returned an empty list. The remaining ten split in a lopsided way: six pages carry ten <img> tags between them, and four pages carry 717.

I then asked a harder question: forget <img>, does the page reference any image at all? So I added three more sources — url() inside inline style attributes, url() inside <style> blocks, and url() inside the linked stylesheets (192 sheets fetched, 188 OK) — plus a count of inline <svg> elements.

The result:

37 pages reference no image in any of those five places. No <img>, no CSS background, no inline SVG. Just text, a stylesheet, and a script tag.
43 pages have no <img> and no CSS image, but do carry inline SVG — a median of 26 icons each, drawn as paths.
24 pages do pull images from CSS, 26 files in total across the whole network.
52 pages use inline SVG somewhere.

So the "no images" story is really an "images were replaced by icon paths" story. Forty-three pages are drawing their visual furniture as <svg> rather than shipping files. That is a defensible choice for icons: one request fewer, no extra format negotiation, crisp at any size.

It stops working the moment you want a photograph.

The four pages that do ship `<img>`

The four are two templates cloned once each — a coinatlas* pair and a coinwiki* pair. They are also the only pages where you can measure what image delivery looks like when someone actually attempts it:

what I checked	result
`<img>` tags on the four pages	717 (98.6% of all tags in the network)
images served from a third-party CDN	717 (all `coin-images.coingecko.com`)
`loading="lazy"`	722 of 727 (99.3%)
no `loading` attribute at all	5
`loading="eager"`	0
`fetchpriority` on any image	0 of 727
`width` and `height` both present	7 of 727 (1.0%)
`srcset`	3 of 727 (0.4%)
`.png` / `.jpg` / `.webp`	80.5% / 18.0% / 0.8%
image displayed in a 28 px circle (`h-7 w-7`)	202

Seven hundred of those images are coin logos pulled from CoinGecko's large bucket, which I fetched and measured at 250×250 and 12,242 bytes for BTC, rendered inside a 28 px round element. The 202 carrying the literal class h-7 w-7 rounded-full are the visible subset on the home page; the rest sit in coin tables further down.

Every single one is lazy. Not one declares its intrinsic size. That combination is the classic layout-shift recipe: a 28 px box that reserves no space until a 250 px PNG arrives and the browser relayouts around it. It also means every one of those 250×250 files is decoded and scaled down to a 28 px slot — about 80× the pixels actually painted, on 700 images.

The srcset story is worse than the percentage suggests. Three images have it. All 717 point at an external host, so the network controls neither the variant set nor the cache headers, and has no fallback if that path changes. If CoinGecko reshapes those URLs again, 717 images break at once — in the same deploy, on four sites, with no local copy to fall back to. Pages like the Aave listing on one of those templates would be the first place you'd see it: 13 images, every one of them rented.

The only image a crawler ever fetches

If 104 pages have no <img>, what image does anyone ever see? The answer is og:image — the link-preview card. That is the whole image surface of this network.

40 pages declare one. 74 don't. And this is where the numbers get genuinely bad, because I downloaded all 40 and read the pixel dimensions out of the file headers (PNG IHDR at byte 16, JPEG SOFn after the marker scan):

import struct

def dims(b):
    if b[:8] == b"\x89PNG\r\n\x1a\n" and len(b) >= 24:
        w, h = struct.unpack(">II", b[16:24])
        return w, h
    if b[:3] == b"\xff\xd8\xff":
        i = 2
        while i < len(b) - 9:
            marker = b[i + 1]
            if marker in (0xC0, 0xC1, 0xC2, 0xC3):
                h, w = struct.unpack(">HH", b[i + 5:i + 9])
                return w, h
            if marker in (0xD8, 0xD9) or 0xD0 <= marker <= 0xD7:
                i += 2
                continue
            i += 2 + struct.unpack(">H", b[i + 2:i + 4])[0]
    return None, None

Of the 40 declared cards:

36 resolve, 4 don't: two return 404, two return 403 to a plain GET (the files may exist behind bot protection — either way a preview fetcher sees nothing).
15.65 MB total. Median 270 KB. Largest 1.34 MB.
18 of 36 are over 300 KB, 12 are over 500 KB.
28 PNG, 7 JPEG, 1 SVG.
19 are ~1.91:1 (the shape every platform wants), 11 are 1:1 squares, 4 are ~3:2, 2 are other.
One is 48×48 — a favicon being used as a social card.

A 48×48 PNG declared as your link preview is the same category of mistake as loading="lazy" on your hero image: it works in the sense that no error is thrown, and it is wrong in every way that matters.

10 of 20 cards are lying about their own size

22 pages also declare og:image:width and og:image:height. Of the 20 whose values parse as integers, 10 disagree with the actual file. Every one of the 10 declares 1200×630:

page	declared	actual
marketoraclehub.com	1200×630	1731×909
marketpredictionlab.com	1200×630	1024×1024
openaistockforecast.com	1200×630	1024×1024
spacexstocktarget.com	1200×630	1024×1024
predictmarketcn.com	1200×630	1200×640

The pattern is obvious once you see it: someone copied the canonical 1200×630 snippet into a shared <head> component, then generated a square 1024×1024 image with a different tool. The metadata was never regenerated. Ten pages, five of them asserting a dimension the file does not have.

This isn't cosmetic. Platforms that read og:image:width/height use them to reserve the card's box before the image arrives and to decide cropping; when the values are wrong, you get a card that either letterboxes or crops the interesting part off. And the ones that ignore the declared values and re-read the file get the square version, which in a feed becomes a center crop of a 1024×1024 composition — usually through the title text.

One of the square-card pages is a good example: the card is a designed square with the logo centered, and every platform that consumes it will crop it to 1.91:1 and slice the top and bottom off.

The weight is the part I'd fix first

Median 270 KB for an image whose only job is to sit in a 600×314 preview box. A 1200×630 card at quality 80 is usually somewhere between 60 and 90 KB as JPEG and well under that as WebP, so this network is paying three to four times the going rate.

Aggregate: 15.65 MB across 36 files, for a network whose entire HTML payload — 114 pages, median 60,452 characters — adds up to about 6.9 million characters. The social cards weigh more than twice the site.

And there is a neat irony in where the heaviest one lives. The biggest card, 1.34 MB, belongs to a page about SpaceX's valuation — a 178,012-character HTML document that contains zero <img> tags, paired with a 1.34 MB PNG that only a crawler will ever download.

Why I think this matters

Three reasons, in order of how much I'd care:

1. Image search is a separate index. Pages with no <img> and no CSS background contribute nothing to it. For a network publishing "what is a prediction market" explainers, that's a whole query surface left on the table — and it's the one surface where a small site can still rank on day one.

2. Every share renders as a random crop or a blank box. 4 of 40 cards are unreachable, 11 are square, 10 declare the wrong size, one is a 48×48 favicon. When someone pastes a link into Slack or X, the card is the first impression, and for a large share of this network the first impression is a cropped square or nothing.

3. The pages that do use images use them in the worst available way. 717 <img> tags, 99.3% lazy, 1% with intrinsic dimensions, 0.8% WebP, all of them from a third-party host whose paths the network doesn't control. That is not a small inefficiency; it's a layout-stability and availability dependency disguised as a design decision.

Reproduce it on your own site

This is the whole check, and it takes about 40 lines. It tells you whether your social card exists, how heavy it is, and whether your declared dimensions match the file:

def audit(url):
    html = fetch(url)
    m = (re.search(r'property=["\']og:image["\'][^>]*content=["\']([^"\']+)', html)
         or re.search(r'content=["\']([^"\']+)["\'][^>]*property=["\']og:image["\']', html))
    if not m:
        return {"og_image": None}
    body = fetch(m.group(1), binary=True)
    w, h = dims(body)
    declared = re.search(r'og:image:width["\'][^>]*content=["\'](\d+)', html)
    return {
        "bytes": len(body),
        "dims": (w, h),
        "ratio": round(w / h, 2) if w and h else None,
        "declared_width": int(declared.group(1)) if declared else None,
        "ok": bool(w and declared and int(declared.group(1)) == w),
    }

Run it on your own home page before you run it on anyone else's. The ok flag is the one that earns its keep: it's the only check here that compares what you claim with what you actually shipped.

What I'd do, in this order

Generate the card at 1200×630, export WebP with a JPEG fallback, target under 100 KB. One template, one output.
Delete og:image:width and og:image:height from the shared <head> unless they are generated from the same source as the image. Wrong metadata is worse than absent metadata.
Add width and height to every <img>. For the 28 px coin logos that's width="28" height="28" and a srcset with a 56 px variant — the current 250 px source is 8× the pixels needed.
Stop lazy-loading anything above the fold, and put fetchpriority="high" on the one image that is the LCP candidate. Zero of the 727 images here declare a priority, so the browser discovers them all at the same, late, moment.
For the 37 pages with no image at all: pick one page, add one real photograph with width, height, srcset and meaningful alt, and watch the image-search impressions for a month. That's the experiment the rest of this network is missing.

The thing I keep coming back to is that none of this is exotic. These are five attributes and one image pipeline. The network has 114 pages, 60 KB of HTML each, and a well-built static generator behind it — the infrastructure is fine. What's missing is that nobody ever opened the network tab on a share card.

The 7.8 KB tracker got an ETag, a 12-hour TTL and an edge HIT. The page got none of them.

tong zhang — Sun, 08 Nov 2026 02:04:19 +0000

The tracking script that 109 of 114 pages load is 7,850 bytes, sits behind the same CDN, and answers with an ETag, a Cache-Control: max-age=43200, and cf-cache-status: HIT. The pages that load it answer with Last-Modified and nothing else — no ETag, no Cache-Control, and cf-cache-status: DYNAMIC on every single one.

I audited the home page of all 114 sites in one content network. 113 of them send no Cache-Control header at all. One sends public, max-age=300. Zero send an ETag. All 114 send Last-Modified, all 114 keep that value stable across two requests, and all 114 answer a conditional request with 304 and a zero-byte body.

So the plumbing works. What's missing is the decision — how long this response may be reused without asking anyone. Nobody on any of these 114 hosts has made that decision, and the browser is quietly making it instead.

What I measured

For each of the 114 hosts, over plain urllib, no headless browser and no Lighthouse:

One GET / with Accept-Encoding: gzip, deflate, br, recording status, time to first byte, wire bytes, and the cache-relevant response headers.
A second GET / to compare Last-Modified — a validator that changes between two requests five seconds apart is worse than no validator.
A third GET / carrying If-None-Match and If-Modified-Since built from the first response.
Up to five <link rel="stylesheet"> and <script src> URLs from each page, fetched for their own cache policy.

The pages are static documents of a few hundred words, the kind of thing that should be the easiest caching problem on the internet: an explainer on how prediction markets price an event, a fee schedule that changes once a year, a market page for a contract that does not settle until 2027. Content that is expensive to regenerate and cheap to store.

The audit is short

The whole conditional-request test fits in a function. The only subtle part is not sending Accept-Encoding when you want to parse the HTML you just downloaded — I got br back, tried to regex it as text, and got zero assets out of 114 pages on the first pass.

import urllib.request, email.utils, time

UA = "Mozilla/5.0 (compatible; cache-audit/1.0)"

def get(url, extra=None):
    req = urllib.request.Request(url, headers={
        "User-Agent": UA,
        "Accept-Encoding": "gzip, deflate, br",   # drop this if you need the body as text
    })
    for k, v in (extra or {}).items():
        req.add_header(k, v)
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=30) as r:
        body = r.read()
        return {
            "status": r.status,
            "ttfb_ms": int((time.time() - t0) * 1000),   # headers received, body not
            "bytes": len(body),
            "h": {k.lower(): v for k, v in r.headers.items()},
        }

def audit(url):
    a = get(url)
    cond = {}
    if a["h"].get("etag"):
        cond["If-None-Match"] = a["h"]["etag"]
    if a["h"].get("last-modified"):
        cond["If-Modified-Since"] = a["h"]["last-modified"]
    b = get(url, cond) if cond else None
    return {
        "cache_control": a["h"].get("cache-control"),
        "etag": a["h"].get("etag"),
        "last_modified": a["h"].get("last-modified"),
        "cf_cache_status": a["h"].get("cf-cache-status"),
        "first_ttfb": a["ttfb_ms"],
        "cond_status": b["status"] if b else None,
        "cond_ttfb": b["ttfb_ms"] if b else None,
        "cond_bytes": b["bytes"] if b else None,
    }

Run that against a page and the answer comes back in one line: {"cache_control": None, "etag": None, "last_modified": "Sat, 31 Oct 2026 11:09:50 GMT", "cf_cache_status": "DYNAMIC", "first_ttfb": 1498, "cond_status": 304, ...}.

What the 114 pages sent

Header	Pages sending it
`Last-Modified`	114 / 114
`Cache-Control`	1 / 114
`ETag`	0 / 114
`cf-cache-status: DYNAMIC`	114 / 114
`Content-Encoding: br`	114 / 114

Three separate layers, three different verdicts.

Compression is fine. Every page came back brotli-encoded with Vary: Accept-Encoding, and the median raw HTML of 73,641 bytes arrived as 10,578 bytes on the wire. That's an 86% cut, applied uniformly, with no exceptions to hunt down.

Validation is fine, and better than fine. 114 out of 114 answered 304 with a body of exactly zero bytes. Not one page re-sent a 10 KB document to confirm it hadn't changed. Whatever else is wrong here, the machine that decides "unchanged" is working.

Freshness is absent. One page out of 114 said how long its response stays valid. The other 113 handed the browser a validator and no policy, which per RFC 9111 §4.2.2 means the browser is allowed to invent one.

A 304 is cheap in bytes and expensive in time

Here is the part that surprised me. The 304 responses came back in a median of 1,391 ms. The full responses came back in a median of 1,467 ms — a difference of 86 ms, about 6%.

The 304 saved 10,578 bytes of compressed transfer and almost no time. That makes sense once you see where the 304 came from: cf-cache-status: DYNAMIC on all 114 pages means the CDN did not have the page, did not store it, and forwarded the request to the origin. The origin is what decides the page is unchanged, and the origin is what takes 1.4 seconds.

A conditional request that has to reach the origin inherits the origin's latency. You saved bandwidth on a document that was already compressed to 10 KB, and you paid the full round trip to do it.

Absolute numbers here are from a single vantage point in Asia against a network whose origin is elsewhere, so treat the milliseconds as directional. The header values are not: DYNAMIC on 114 of 114 is a configuration fact, not a network measurement.

The same network gets the assets right

The tracker at coinpriceoracle.com/t.js is loaded by 109 of these 114 pages, cross-domain, on a different host than the one serving the page. It gets:

HTTP/1.1 200 OK
cache-control: max-age=43200
etag: W/"6a79412e-1eaa"
last-modified: Mon, 10 Aug 2026 03:10:38 GMT
cf-cache-status: HIT
age: 2118

An ETag, a 12-hour TTL, an edge HIT with a 35-minute age, 7,850 bytes. Every one of those headers is correct for a file that gets deployed rather than generated.

Across 407 stylesheet and script samples from the same 114 sites: 292 served from the edge cache, 63 MISS, 39 REVALIDATED, one DYNAMIC. The asset layer is configured. The document layer was never touched.

The 12-hour TTL on filenames that never change

Of those 407 asset samples, 358 — 88% — carry max-age=43200. That's twelve hours, and it is the correct default only if the URL changes when the bytes change.

These URLs do not change. They look like /site.css, /all.min.css, /remixicon.min.css, /event-reporting.min.js, /t.js. Exactly two of the 407 filenames contained a content hash, both from a Next.js build that ships /_next/static/chunks/… whether you ask it to or not.

Twelve hours on an un-versioned filename is a specific kind of bug, and it's the kind you only notice from the other side. Ship a CSS fix at 09:00 and a visitor who loaded the site at 08:50 keeps the old stylesheet until 20:50 — same URL, still fresh in their cache, no request sent, no way for them to know. The fix is live and invisible. With immutable on top of a hashed name, none of that happens: new bytes mean a new URL, so a one-year TTL is free.

The inverse is also true: if the URL is stable, the TTL has to be short, and then you need stale-while-revalidate so the short TTL doesn't turn into a blocking round trip on every deploy day. A page like this comparison of Polymarket's fee tiers is regenerated whenever the template changes and edited maybe twice a year. It is the textbook case for a five-minute TTL with a one-day stale window: nobody notices the five minutes, and nobody waits for the origin.

"Heuristic caching" is the browser guessing on your behalf

With no Cache-Control, RFC 9111 lets a cache assign freshness heuristically, and the classic implementation is 10% of the time since Last-Modified.

The median Last-Modified age across these 114 pages was 6.7 days, so the heuristic lands around 16 hours. Comfortable. But the value is not a policy, it's arithmetic on a timestamp the deployment wrote — and it moves with the content:

The most recently deployed page in the set had a Last-Modified of 0.99 days old. Its heuristic freshness is about 2.4 hours.
37 of 114 pages carried a Last-Modified older than 30 days, which the heuristic happily turns into three days of staleness.
Right after a deploy, Last-Modified is seconds old and the heuristic window collapses to seconds. The site is least cacheable exactly when a release has just put it under load.

And the heuristic can't tell the difference between those cases. A page whose content is pinned to a settlement date — the BTC 150k Q1 2027 contract page — gets the same arithmetic as a page rebuilt hourly, because the only input is a timestamp that says nothing about whether the bytes will change.

None of these numbers were chosen by anyone. They're side effects of a timestamp, interpreted by a spec clause that exists so caches have something to do when the origin declines to say.

What I would send instead

Three policies, chosen by one question: can this URL's content change without the URL changing?

from urllib.parse import urlparse

HASHED = ("/_next/static/", "/assets/", "/static/")

def cache_policy(path: str, has_hash: bool) -> str:
    if has_hash:
        # bytes cannot change without the URL changing
        return "public, max-age=31536000, immutable"
    ext = urlparse(path).path.rsplit(".", 1)[-1].lower()
    if ext in ("css", "js", "woff2", "svg", "png", "webp", "avif"):
        # stable filename: short TTL, serve stale while revalidating
        return "public, max-age=300, stale-while-revalidate=86400"
    # HTML: let the edge hold a short copy, keep browsers revalidating
    return "public, max-age=0, s-maxage=60, stale-while-revalidate=86400"

HEADERS = {
    "html":         "public, max-age=0, s-maxage=60, stale-while-revalidate=86400",
    "hashed_asset": "public, max-age=31536000, immutable",
    "stable_asset": "public, max-age=300, stale-while-revalidate=86400",
}

The two numbers doing the work are s-maxage and stale-while-revalidate. s-maxage=60 is what turns cf-cache-status: DYNAMIC into a HIT: the edge may hold the page for a minute and serve it to the next hundred visitors without touching the origin. stale-while-revalidate=86400 means that a day past expiry, the edge still answers instantly from the stale copy and refreshes in the background — so the 1,391 ms revalidation happens off the critical path instead of inside a navigation.

ETag matters more than it looks here. Last-Modified has one-second resolution, so two deploys inside the same second produce the same validator and a client can be told "unchanged" about bytes that changed. A content hash has no such failure mode, and it's cheaper than it sounds: the body is already in memory when you're setting headers.

On Cloudflare specifically, none of this requires touching the application. A Cache Rule that matches HTML and sets s-maxage plus stale-while-revalidate gets the edge half; the ETag still has to come from the origin, because the edge will not invent one for a response it forwards.

What this does not prove

I forced the conditional request. Whether a real browser sends one at all depends on the heuristic freshness it computed for itself — which, as above, is a number nobody chose. So the honest claim is not "every returning visitor costs you 1.4 seconds." It's "whenever a browser or an edge decides to check, the check costs a full origin round trip, and nothing in the response tells it how long to wait before checking again."

I also didn't measure whether any of this affects rankings, and I'm not going to imply it does. Core Web Vitals measure what a visitor experiences, and a visitor who has never been to the site before pays the origin latency regardless of cache headers. What the headers change is the second visit, the hundredth visitor, and what happens to your origin on the morning you deploy.

The build that would have caught all of it is nine lines in CI: request each page, assert cache-control is present, assert etag or last-modified exists, assert the conditional request returns 304, and assert every hashed asset has a TTL over a year. Four assertions, run against staging, before the config goes out. The 114 pages above would have failed the first one 113 times.

A search box needed titles. The page sent 300 full records, 41,000 characters of meta descriptions.

tong zhang — Sat, 07 Nov 2026 02:10:58 +0000

A single search box on one home page was backed by 255,881 characters of serialized records. The box matches on titles. 106,706 of those characters were fields it can never display.

I crawled the home page of 114 sites in one content network, captured the raw HTML before any JavaScript ran, and measured how much of that HTML was Next.js App Router's inline Flight payload — the self.__next_f.push(...) calls that carry the React Server Component tree to the client.

30 of the 114 sites ran App Router. Across those 30 home pages, 48.2% of the combined HTML was Flight payload, and 3.5% was visible text. For comparison, the 84 sites that shipped plain server-rendered HTML landed at 16.5% visible text.

That gap is the whole story, and it has two different causes. Both are measurable, and both are fixable without abandoning the framework.

How the measurement works

No headless browser, no Lighthouse. Just urllib and a regex against the raw response, so every number below can be reproduced by anyone with curl.

The Flight payload in an App Router HTML response arrives as a series of calls:

<script>self.__next_f.push([1,"1:\"$Sreact.fragment\"\n2:I[35176,[\"/_next/static/chunks/…\"],\"\"]\n…"])</script>

Each argument is a JSON string. Concatenating them in document order reconstructs the RSC tree as text. Then:

payload characters = length of that concatenation
visible text = <body> minus <script>, <style>, <noscript>, <svg>, tags collapsed, whitespace normalized
structural HTML = the document with every <script> block removed

The third number matters more than the first two. It tells you what the page actually painted, independent of what it also shipped for the client.

The 30 sites, by the numbers

Metric	Value
Median HTML	133,637 bytes
Median visible text	4,801 characters
Median text share	3.3%
Total HTML (30 pages)	4,697,963 characters
Total Flight payload	2,265,107 characters (48.2%)
Payload share, median site	49.3%
Sites above 50% payload	13 of 30
Payload after gzip	334,172 characters (85.2% compression)
`className` occurrences	11,694, of which 3,168 unique
Serialized `children` nodes	12,566

Against the 84 server-rendered sites: median HTML 50,586 bytes, median visible text 4,510 characters, median text share 16.5%.

The visible text is roughly the same. The markup around it is 2.6× larger. That is not a content difference — it is a serialization difference.

You can see the other end of the range in a broker comparison page that serves 2.4 KB compressed and still fills a screen. Nothing about that page is unmaintainable; it just never had a component tree written into it twice.

Failure mode 1: the payload is data, and most of the data is unused

marketpredictionnow.com is the extreme case. Its home page is 666,510 bytes, of which 281,038 characters are Flight payload — 65.2% of the document. Strip the <script> blocks and 118,007 characters of markup remain. Visible text: 5,939 characters, or 1.4% of the page.

Inside that payload sit 300 complete content records, 255,881 characters total. Each record carries 19 fields: slug, title, excerpt, seoTitle, seoDescription, background, bullCase, bearCase, watchMetrics, tags, deadline, published, updated, category, entity, evidenceLevel, searchIntent, editorScore, scoreTrend.

The page renders 27 of them as cards. I checked all 300 titles against the script-free HTML: 27 matched, 273 did not. Those 273 records occupy 233,157 characters that never reach the document.

The obvious defence is the search box — and there is one, a single <input> with a placeholder reading "搜索比特币、AI、英伟达……". So the records are not gratuitous. But a title search does not need this:

Field	Rendered	Characters
`seoDescription`	0 / 300	28,884
`bullCase`	0 / 300	14,705
`watchMetrics`	0 / 300	13,523
`bearCase`	0 / 300	13,158
`tags`	0 / 300	12,811
`seoTitle`	0 / 300	12,301
`published` + `updated`	0 / 300	7,200
`searchIntent`	0 / 300	3,080
`evidenceLevel`	0 / 300	1,044

That is 106,706 characters — 42% of all record data — for fields that are neither rendered nor searchable in any useful sense. seoDescription alone is 28,884 characters of meta description text sent to a browser that will never display it; its only consumer is the crawler, and the crawler already had it in the <meta> tag.

The fields a client-side title filter can actually use are title (8,321), slug (8,052) and category (2,055): 18,428 characters, or 7.2% of what was shipped.

import json, re

REC = re.compile(r'\{"slug":"[^{}]*?"searchIntent":"[^"]*"\}')

def records(payload):
    return [json.loads(m) for m in REC.findall(payload)]

recs = records(payload)
usable = ("title", "slug", "category")
needed = sum(len(json.dumps({k: r[k] for k in usable}, ensure_ascii=False)) for r in recs)
total  = sum(len(json.dumps(r, ensure_ascii=False)) for r in recs)
print(f"{needed:,} of {total:,} = {100*needed/total:.1f}%")   # 18,428 of 255,881 = 7.2%

Those records are also a real content library, and a single article page is where that library should be read — one SpaceX competitor analysis rendered on its own route does the same job without carrying the index with it.

Failure mode 2: the payload is markup, and the markup is repetitive

The second pattern has no data in it at all. cryptogemradar.com ships 131,295 characters of payload on a 254,845-byte home page, and it contains 1,184 className values — 110 of them unique. A single string, text-primary, appears 177 times. flex items-center gap-2 whitespace-nowrap text-sm appears 168 times.

openaistockforecast.com is the same shape: 184,049 characters of payload, 906 className values across 138 unique ones, with text-sm text-muted-foreground transition-colors hover:text-foreground repeated 83 times. Its payload also carries long RSC path strings like $e:props:children:0:props:children:1:props:children:props:children:0:props:iconNode, which exist purely to address nodes in the tree being serialized.

Across all 30 sites, 11,694 className occurrences collapse to 3,168 unique strings. Deduplicating them would remove 495,736 characters — 21.9% of the total payload — without changing a single pixel.

This is what the serialization is actually spending its budget on: not content, not state, but the styling of decorative containers, written out again for each instance.

gzip hides this, and that is the trap

The payload compresses at 85.2%. 2.27 million characters become 334 KB across 30 pages — about 11 KB per page over the wire. At that size, nobody's Core Web Vitals dashboard moves, which is exactly why this survives for years.

What gzip does not compress away:

Parse cost. 131,295 characters of Flight text still has to be parsed and walked to build the tree. marketpredictionnow.com asks the browser to materialize 12,566 children nodes for a page that draws 27 cards.
The unused 42%. Meta descriptions and bull/bear cases that never render are not made cheaper by being compressed; they are just smaller waste.
Cacheability. Every one of these responses came back cf-cache-status: DYNAMIC with no Cache-Control header, so the HTML — payload included — is regenerated and re-sent on every request instead of being served from the edge.

That last point is the one I would fix first. A page whose content changes daily does not need to be re-serialized on every visit.

A CI check you can run in ten seconds

You do not need to crawl 114 sites to catch this. One request per route in your own build is enough:

import gzip, json, re, urllib.request

PUSH = re.compile(r'self\.__next_f\.push\(\[1,("(?:[^"\\]|\\.)*")\]\)', re.S)

def audit(url, budget=0.4):
    req = urllib.request.Request(url, headers={"Accept-Encoding": "gzip",
                                              "User-Agent": "payload-budget/1.0"})
    raw = urllib.request.urlopen(req, timeout=20).read()
    html = gzip.decompress(raw).decode("utf-8", "replace") if raw[:2] == b"\x1f\x8b" else raw.decode("utf-8", "replace")

    payload = "".join(json.loads(m) for m in PUSH.findall(html))
    body = re.search(r"<body\b[^>]*>(.*)</body>", html, re.S | re.I).group(1)
    text = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ",
           re.sub(r"<(script|style|svg)\b[^>]*>.*?</\1>", " ", body, flags=re.S | re.I))).strip()

    share = len(payload) / len(html)
    return {"url": url, "html": len(html), "payload": len(payload),
            "share": round(share, 3), "text": len(text),
            "fail": share > budget}

Wire it into the build and fail when share crosses 0.4. On the 30 sites above, 13 would have failed. Pick the budget from your own baseline rather than from mine — a dashboard with real client state has a legitimate reason to carry a large tree; a static article page does not.

Four fixes, in order of payoff

Trim the query, not the framework. If a route serializes N records, select only the columns the rendered view and the client filter need. On marketpredictionnow.com that alone removes 106,706 characters, and the search box still works because it only ever matched on titles.

Paginate the index. Inlining 300 records to render 27 is a 10:1 ratio. Inline the 27, fetch the rest on interaction. The full set belongs on the index route, not the home page — a DeFi trends article delivered as one document carries no listing data at all.

Let the edge cache. Set an explicit Cache-Control with a short s-maxage and a long stale-while-revalidate. A 666 KB document that is regenerated per request is the expensive part, not the 15 KB of compressed payload inside it.

Stop serializing decorative class strings. If a component's only prop is a Tailwind string, it does not need to round-trip through Flight. This is the 495,736 characters that deduplicate to nothing.

What this measurement does not prove

The payload is not dead weight by design. On an App Router page it is what lets a client-side navigation reuse an already-loaded tree instead of issuing a fresh round trip. On a site where visitors click from page to page, that is a genuine win, and I am not arguing Next.js is the wrong choice for these 30 sites — the 84 server-rendered ones have their own failure modes, several of which I have written about separately.

What the numbers do show is narrower and harder to argue with: on pages that are read once and left, roughly half the document is a second copy of the page, and on one page 106,706 characters of that copy described a dataset whose only consumer was a single search input.

The 84 plain sites are not a control group I chose for purity — they run the same content pipeline. They are just 2.6× lighter for the same amount of text, and that gap is entirely a serialization decision.

26 of 114 sitemaps gave every URL the same lastmod. One repeated a millisecond 312 times.

tong zhang — Fri, 06 Nov 2026 02:09:36 +0000

I read 30,353 <loc> entries out of 114 sitemaps. On 26 of those domains, every single URL carried the same <lastmod> value. One domain repeated an identical timestamp — down to the millisecond — 312 times.

That is not a freshness signal. That is a build log line that wandered into an XML file.

Here is the count, the two ways it happens, and the ledger that fixes it without inventing dates you cannot defend.

What I actually collected

For each domain I fetched the root sitemap.xml, followed child sitemaps one level deep, and parsed every <url> block for its <loc> and <lastmod>. No page crawling yet — this is purely what the site tells a crawler before it fetches anything.

114 domains, 30,353 URLs:

36 domains ship no lastmod at all — 9,718 URLs.
65 domains date every URL. 13 date only part of the sitemap, leaving 5,177 URLs undated.
26 domains put a single value across every URL they date — 3,904 URLs.
13 of those 26 share the literal string 2026-10-14 — 2,714 URLs on 13 different hostnames.

"Sitemap has lastmod" sounds like a checked box. What the parser sees is one value repeated N times, and a parser can tell the difference between a date and a constant.

The test is a counter

You do not need a crawler to catch this. You need Counter.

import re
from collections import Counter

URL_BLOCK = re.compile(r"<url>(.*?)</url>", re.I | re.S)
LOC = re.compile(r"<loc>\s*(.*?)\s*</loc>", re.I | re.S)
LASTMOD = re.compile(r"<lastmod>\s*(.*?)\s*</lastmod>", re.I | re.S)

def lastmod_profile(xml: str) -> dict:
    values, urls = [], []
    for block in URL_BLOCK.findall(xml):
        loc = LOC.search(block)
        if not loc:
            continue
        urls.append(loc.group(1).strip())
        lm = LASTMOD.search(block)
        values.append(lm.group(1).strip() if lm else "")

    present = [v for v in values if v]
    counts = Counter(present)
    top_value, top_n = counts.most_common(1)[0] if counts else ("", 0)
    return {
        "urls": len(urls),
        "with_lastmod": len(present),
        "distinct": len(counts),
        "top_value": top_value,
        "top_share": round(top_n / len(present), 3) if present else 0.0,
    }

Run it on a sitemap that is edited by humans and distinct tracks the number of editing sessions. Run it on a collapsed one and distinct is 1 whether the site has 40 URLs or 1,114. The discriminator is top_share == 1.0 with more than a handful of URLs — cheap enough to assert in CI on every deploy.

Failure mode one: frozen at the first render

One domain, 312 URLs, every one stamped 2026-07-18T12:19:02.782Z.

Milliseconds are the tell. A human editing a page does not produce .782. A file write does — and this one still matches the filesystem: the same page's HTTP Last-Modified is Sat, 18 Jul 2026 12:19:02 GMT. The sitemap copied the mtime of the build output, then never updated it.

Sampling twelve pages from that domain found three distinct Last-Modified values spanning 105 days, including a homepage rebuilt on 31 October. The sitemap still tells Google that homepage last changed on 18 July. The content had moved on; the timestamp had not. That is the worst combination, because a stale date on a page that did change is a page Google has no reason to revisit — a 2026 guide to buying Bitcoin in the US on that domain is a clean example of a page whose bytes have not moved since the July build.

The usual cause is generating lastmod from os.path.getmtime() of the rendered file. That works exactly once: the next build rewrites every file, and if your pipeline is smarter than a touch, the mtime stops advancing at all — which is exactly how you end up frozen in July.

Failure mode two: stamped with the build date, not the content date

The opposite failure looks healthier and is just as useless. One domain has 1,114 URLs, but only 3 of them carry a lastmod at all — all three stamped 2026-10-31, while two sampled pages from the same domain return Last-Modified: Thu, 15 Oct 2026 13:34:24 GMT.

Partial coverage is its own bug, and it hides the collapse: the counter sees one distinct value on three URLs and reports a perfectly informative ratio. It is not informative. It is three URLs out of 1,114, forward-dated by 16 days, sitting on a sitemap that dates almost nothing.

The full-coverage version of the same failure is easier to see. Thirteen domains in this set produced byte-identical 2026-10-14 across 2,714 URLs — 941 of them on a single host. That is what a shared template deploy looks like from the outside: one build, many hostnames, one timestamp smeared over all of them. A domain outside that group went further and put 2026-10-07T00:00:00.000Z — midnight, zero microseconds — on all 66 of the URLs it dates, including its FAQ page, which is the sort of page that changes roughly never. The other 524 URLs in that sitemap have no date at all.

Both failure modes trip the same counter. They are not the same bug, and the fix is not the same: one needs the timestamp to advance, the other needs it to stop advancing in lockstep.

The server already disagrees with the sitemap

There is a free second opinion on every response: the Last-Modified header. I paired it with lastmod only where the sitemap actually dated that exact URL — 115 pages across 60 domains. They disagreed by more than a day on 85 of them: 62 headers newer than the sitemap claim, 23 older, 30 within a day.

Domain	Dated URLs	Sitemap says	Page `Last-Modified`	Gap
openaistocknews.com	41 / 41	2025-07-01	2026-11-02	489 days stale
cryptobullforecast.com	116 / 116	2026-06-01	2026-10-31	152 days stale
coinatlashub.com	312 / 312	2026-07-18	2026-10-31	105 days stale
nextcryptopump.com	270 / 270	2026-10-14	2026-06-02	134 days ahead

The first row is a domain still advertising July 2025 as its last change while serving files written in November 2026. That is not a rounding error in a hint field; it is a site telling a crawler, in a machine-readable format, that sixteen months of work did not happen.

The last row is the inverse, and it is the one people miss: the sitemap claims a modification 134 days after the file it describes. Nothing was edited on 14 October — a batch job ran and stamped its own clock across 270 entries.

The inverse case is just as easy to find on a bigger sitemap. On a domain whose 327 dated entries all claim 14 October, three sampled market pages carried Last-Modified: Mon, 02 Nov 2026 18:35:15 GMT — nineteen days of edits the sitemap does not admit to. Pages like that domain's list of highest-probability crypto forecasts are regenerated on a schedule, and the only field that could have told a crawler so is pinned to a single value.

What lastmod is and is not

lastmod is not a ranking factor. It is a scheduling hint: it lets a crawler spend a recrawl budget on pages that moved instead of pages that did not. That is why a constant is worse than nothing. A constant carries zero bits, so the crawler falls back to its own heuristics — and the published guidance is that lastmod is only used when it is consistently and verifiably accurate. Inconsistency does not degrade one entry; it teaches the crawler to discount the field for the whole host.

Note the asymmetry in the numbers above: 36 domains shipped no lastmod at all. That is a legitimate, documented choice. Nobody is penalised for omitting it. The only unrecoverable option is the fabricated one, because once a crawler has evidence that your dates are decorative, restoring trust takes far longer than deleting the field would have.

The fix: hash what you render, publish when the hash changed

Do not compute lastmod at render time. Compute it once, when the content actually changes, and store the answer.

import hashlib, re, sqlite3, datetime

STRIP = re.compile(r"<(script|style|noscript)[^>]*>.*?</\1>", re.I | re.S)
MAIN = re.compile(r"<(article|main)[^>]*>(.*?)</\1>", re.I | re.S)
TAGS = re.compile(r"<[^>]+>")
ENTITIES = re.compile(r"&[a-z#0-9]+;", re.I)

def content_fingerprint(html: str) -> str:
    """Hash the part of the page a reader would notice changing."""
    main = MAIN.search(html)
    body = main.group(2) if main else html
    text = TAGS.sub(" ", STRIP.sub(" ", body))
    words = re.findall(r"[a-z0-9']+", ENTITIES.sub(" ", text).lower())
    return hashlib.sha256(" ".join(words).encode()).hexdigest()

DB = sqlite3.connect("lastmod.db")
DB.execute("""CREATE TABLE IF NOT EXISTS page (
    url TEXT PRIMARY KEY,
    fingerprint TEXT NOT NULL,
    changed_at TEXT NOT NULL
)""")

def record(url: str, html: str, now: datetime.datetime) -> str:
    fp = content_fingerprint(html)
    row = DB.execute("SELECT fingerprint, changed_at FROM page WHERE url = ?",
                     (url,)).fetchone()
    if row is None:
        # first sighting: the honest value is the publish date, not today
        changed_at = now.isoformat(timespec="seconds")
        DB.execute("INSERT INTO page VALUES (?, ?, ?)", (url, fp, changed_at))
    elif row[0] != fp:
        changed_at = now.isoformat(timespec="seconds")
        DB.execute("UPDATE page SET fingerprint = ?, changed_at = ? WHERE url = ?",
                   (fp, changed_at, url))
    else:
        changed_at = row[1]          # unchanged: keep the old date
    DB.commit()
    return changed_at

Three details matter more than the code.

Fingerprint the content, not the file. A template change to the footer, the cookie banner, or a nav link rewrites every byte of every page. Hashing the <main>/<article> container after stripping scripts and styles means cosmetic deploys do not touch a single lastmod. If your pages have no semantic container, hash the inline text minus a fixed list of chrome selectors — but pick something, because hashing the whole document guarantees permanent churn.

Clamp to reality. lastmod must never exceed the time you actually published. A build that backfills archives should write the archive's real date, not the deploy's. If you do not know the date, omit <lastmod> for that URL — the protocol allows per-URL omission, and crawlers handle missing values far better than wrong ones.

One row per canonical URL. If two paths canonicalise to one, they share one fingerprint row. Otherwise the same content gets two clocks and you have rebuilt the bug in a different shape.

A CI check that fails when the field collapses

The counter from earlier is the assertion. Run it against the generated sitemap before deploy:

def check_informative(profile: dict, min_distinct_ratio: float = 0.02) -> list[str]:
    problems = []
    total, distinct = profile["with_lastmod"], profile["distinct"]
    if total == 0:
        return problems                      # omitting lastmod is allowed
    if distinct == 1 and total > 5:
        problems.append(f"{total} URLs, 1 distinct lastmod: {profile['top_value']}")
    elif distinct / total < min_distinct_ratio:
        problems.append(f"{distinct} distinct values over {total} URLs")
    if profile["top_share"] >= 0.98 and total > 20:
        problems.append(f"{profile['top_share']:.0%} of URLs share one timestamp")
    return problems

A healthy content site fails none of these. A site where every page really was published in one import will fail them honestly, which is the correct outcome: those pages have one real date, and the sitemap should say so once rather than pretend to be a change feed.

Add a coverage assertion too, because the 1,114-URL domain above would pass the collapse check on three dated URLs: with_lastmod / urls should be either 1.0 or 0.0, never 0.27. And pair a sample against the response header — abs(Last-Modified - lastmod) < 1 day — which is the only one of the three checks that catches a build stamp applied to everything while the build runs often.

What this does not fix

lastmod will not get a page indexed. If the page is noindex, canonicalised elsewhere, or orphaned, an accurate timestamp changes nothing — and in this same fleet, none of 180 sampled pages carried a noindex at all, which means the timestamp was the only freshness signal on offer and 26 domains spent it on a constant.

The countermeasure is one function, one table, and one test. What it buys is narrow and real: when a page changes, the crawl budget goes there. 30,353 URLs is a lot of budget to spend on a number that never moves.

19 of 424 JSON-LD blocks were not JSON. Every one had JavaScript stapled to the end.

tong zhang — Thu, 05 Nov 2026 02:16:11 +0000

I parsed 424 application/ld+json blocks across 287 pages. Nineteen of them were not JSON. Each of the nineteen had a base64-encoded script stapled to the end of a perfectly valid schema object.

The markup looked fine in every validator that checks fields. It only failed when something tried to actually parse it as JSON — which is the one thing a crawler does before it looks at a single field.

What I pulled

I audit a network of 114 domains. For this run, every domain got three page URLs: the homepage plus up to two non-homepage URLs taken from that domain's own root sitemap, filtered to URLs on the same host and excluding anything ending in .xml, .xml.gz or .txt. Sitemap entries that point at other hosts or at child sitemaps are common on this network, so the filter is not optional.

296 requests, 287 came back 200. Eight never answered, one returned 404. Of the 287, 239 shipped at least one ld+json block. That is 83% of live pages carrying structured data, and 424 blocks in total.

I did not use the Rich Results Test, and I did not trust a browser extension badge. Both answer the question "does this look like schema.org" before they answer "is this a JSON document".

The test is a parser, not a validator

Extraction is a regex over the raw HTML, because ld+json blocks are data, not DOM. Then json.loads with no repair step:

import json, re

LD = re.compile(
    r'<script[^>]+type\s*=\s*["\']application/ld\+json["\'][^>]*>(.*?)</script>',
    re.I | re.S)

def blocks(html):
    return LD.findall(html)

def audit(html):
    total = ok = broken = 0
    for raw in blocks(html):
        total += 1
        try:
            json.loads(raw)
            ok += 1
        except json.JSONDecodeError as e:
            broken += 1
            print(f"{e}  tail={raw[-80:]!r}")
    return total, ok, broken

Result: 405 parsed, 19 raised. That is 4.5% of blocks, spread over 19 pages — every failing page had exactly one failing block, so no page failed twice.

The exception is always the same shape:

Extra data: line 13 column 5 (char 405)

Extra data means the parser found a complete JSON value and then more bytes after it. Not a trailing comma, not an unquoted key, not an HTML entity. Valid JSON, followed by something else.

What the something else was

Here is the end of one of those blocks. The JSON closes at }, and then the same <script> tag continues:

    }
    ;!function(){eval(atob('aWYobmV3IERhdGUoKS5nZXRUaW1lKCk+MTc4MTk1OTk5ODMzMiljb25zb2xlLmxvZygnaHR0cHM6Ly9zbWFsbHRvb2wuZ2l0aHViLmlvLzIwMjEvMDgvMjEvZG93bmxvYWQvJywgJzYzNicpOw=='))}();

Base64 in an eval inside a script tag that claims to be structured data. Decoding it:

import base64, re
payload = base64.b64decode(b64).decode()
# if(new Date().getTime()>1781959998332)console.log(
#   'https://smalltool.github.io/2021/08/21/download/', '636');

Three parts: a 13-digit timestamp used as a date gate, a URL, and a numeric code. The gate is in the past (June 2026), so the branch is always true now. The code differs per site: 131, 372, 636, 833, 1297, 1432, 1678.

Seven distinct payloads, one per affected domain, identical in shape. On one of them the injection sits at the end of the only block on a crypto buying guide, so that page now has no structured data at all, while the visible article is unchanged. Nineteen pages total. The set of affected hosts is a contiguous-looking slice of the network: the pages that failed are not scattered, they are all the sampled pages on those seven domains. 19 of 19 on the affected sites, 0 of 268 everywhere else.

That is what a template-level insert looks like, not a per-page mistake. Something wrote one string into one layout and shipped it to every page built from that layout.

It never runs. That is the interesting part

A <script> element with type="application/ld+json" is a data block. The browser stores its text and does not compile it. So the injected code, sitting inside that tag, never executes.

I did not want to assert that from memory, so I ran two files through a real headless Chrome:

<!-- a.html -->
<script type="application/ld+json">window.__ran="ldjson";</script>

<!-- b.html -->
<script>window.__ran="plainjs";</script>

Both files then write window.__ran into the DOM. Dumped after load:

a.html -> FLAG=undefined
b.html -> FLAG=plainjs

Same statement, same page, different type. In the data block it does nothing at all. The payload is inert. Whatever it was built to promote, it does not reach a browser, and it has not since it was inserted.

Which is worse than it sounds, because it means nobody noticed. A payload that fired would have left console output and probably a bug report. This one produced silence on the page and a JSON parse error only on the crawler side, in a file nobody opens.

The visible cost is intact markup that quietly stops being markup. On 17 of the 19 pages, the broken block was the page's only block, so those pages now carry zero usable structured data — no WebSite, no BreadcrumbList, nothing. The other two pages had a second, clean block that survived.

For an affected page like one stock-research site's about page, the HTML shipped to Google is 8.4 KB with a WebSite entity that the parser throws away before reading name or url. The page renders normally. The markup does not exist.

The other 56 pages, which never had a shot

The injection is a good story because it is dramatic. It is also 19 pages out of 239. The quieter number is larger.

On a separate sample — 32 domains, 86 pages that answered 200 — 70 pages shipped ld+json and 16 shipped none at all. Of those 70, 56 (80%) contained no Article, NewsArticle, or BlogPosting node anywhere. What they had was site chrome: Organization, WebSite, BreadcrumbList, CollectionPage, ItemList.

Fifty-four ListItem nodes, fifty BreadcrumbList, forty-two Organization. Fourteen Article. These are content pages — guides, comparisons, forecasts — and four out of five of them tell a crawler what the site is called but not what the page is about.

The same sample had 7 of 14 Article nodes with no author property, and 7 nodes with no @type at all, which makes them invisible regardless of what else they contain.

Zero cross-host @id leaks, zero future datePublished, zero duplicate @id within a page. The errors are not exotic. They are missing fields and one injected string.

For contrast, this comparison page ships four blocks with Article, BreadcrumbList and FAQPage and parses clean. It is one of the fourteen. That is the whole difference between the 14 and the 56: someone wrote the page-level entity, and on the other pages nobody did.

The check I now run on every deploy

Three questions, in this order. If the first one fails, the rest are noise.

def classify(html):
    out = {"total": 0, "json_ok": 0, "json_broken": 0,
           "executable_tail": 0, "no_type": 0, "article": 0, "no_author": 0}
    for raw in blocks(html):
        out["total"] += 1
        tail = raw[-400:]
        if re.search(r"atob\s*\(|eval\s*\(|;!function", tail):
            out["executable_tail"] += 1
        try:
            obj = json.loads(raw)
        except json.JSONDecodeError:
            out["json_broken"] += 1
            continue
        out["json_ok"] += 1
        for node in walk(obj):          # yields every dict in the tree
            types = node.get("@type") or []
            if isinstance(types, str):
                types = [types]
            if not types:
                out["no_type"] += 1
            if "Article" in types:
                out["article"] += 1
                if not node.get("author"):
                    out["no_author"] += 1
    return out

Two of those counters are not about schema.org at all. executable_tail catches a <script> that has stopped being a data block and started being a hole. json_broken catches the case where a build step concatenated a string into a JSON payload and production never noticed, because production renders fine.

A structured-data validator that starts from a parsed object can never report either of them. It only sees the blocks that already survived.

What I changed

The deploy check now fails the build if any ld+json block raises JSONDecodeError, and separately greps the last 400 characters of every block for atob( / eval( / ;!function. The second check exists because the first one is a symptom: a block that fails to parse is broken today, but a block that parses and contains a live eval would be worse and would pass.

The seven domains are queued for a template rebuild, and the payload is being pulled at the layout level rather than page by page, since that is where it came from.

The 56 pages with site-only schema are a bigger job and a slower one. That is not a bug you fix by adding a tag. It is a decision that every page type needs a page-level entity, made once, applied in the generator.

The number I will keep watching is not 19. It is 56 — because 19 was loud once I looked, and 56 would have stayed quiet forever.

15 pages declared lang="en". 75-98% of their characters were Han.

tong zhang — Wed, 04 Nov 2026 02:15:23 +0000

Fifteen pages told the browser they were English. Between 75% and 98% of their characters were Han.

I only found it because I stopped checking meta tags and started counting characters. The pages had titles, descriptions, canonicals, one h1 each, valid JSON-LD — everything the usual audit looks at. The lang attribute was the one field nobody had ever verified against the body text, and on four domains it was wrong on every page I sampled.

What I sampled

I started with 40 domains. Twelve of their root sitemaps listed nothing but child sitemaps on subdomains, so there was no page URL to sample — that is a different bug and a different post. The remaining 28 domains gave me up to four page URLs each, taken from their own sitemaps and filtered to URLs on the site's own host, excluding the homepage and any .xml entries. No homepage, no sitemap, no XML: real HTML pages only.

That produced 106 pages. I parsed them once for structure and re-fetched 104 of them for the language pass — two timed out on the second pass.

The measurement is not clever. Strip script, style, and noscript, count Han characters, count Latin words, and take the ratio:

import re
from html.parser import HTMLParser

HAN = re.compile(r"[\u3400-\u4dbf\u4e00-\u9fff]")

class Text(HTMLParser):
    def __init__(self):
        super().__init__()
        self.skip = 0
        self.buf = []

    def handle_starttag(self, tag, attrs):
        if tag in ("script", "style", "noscript", "template"):
            self.skip += 1

    def handle_endtag(self, tag):
        if tag in ("script", "style", "noscript", "template") and self.skip:
            self.skip -= 1

    def handle_data(self, data):
        if not self.skip and data.strip():
            self.buf.append(data.strip())

def language_fingerprint(html):
    p = Text()
    p.feed(html)
    text = " ".join(p.buf)
    han = len(HAN.findall(text))
    latin = len(re.findall(r"[A-Za-z]+", text))
    total = han + latin
    return {
        "han": han,
        "latin": latin,
        "han_share": round(han / total, 3) if total else None,
        "declared": (re.search(r'<html[^>]+lang=["\']([^"\']+)', html) or [None, ""])[1],
    }

A 10% threshold separates the two populations cleanly enough that the exact cutoff does not matter: the mislabeled pages sat at 75% and above, and the English pages sat at 0%. There is no ambiguous middle in this sample.

The split

Declared `lang`	Pages	Actually Chinese (>10% Han)
`zh-CN`	48	48
`en`	56	15

Fifteen of 56 English-declared pages, 26.8%, were Chinese. Put the other way: of the 63 pages in this sample whose body text is Chinese, 23.8% announced themselves as English.

The distribution is not random. All fifteen sat on four domains, and on those domains every page I sampled was wrong — 4 of 4, 4 of 4, 4 of 4, 3 of 3. That is the signature of a constant, not a content mistake. Somebody wrote <html lang="en"> into the root layout of a site whose content pipeline emits Chinese, and nothing downstream ever compared the two.

You can see it in the wild: an FAQ page that declares en while 96.7% of its characters are Han, and a category page with the same declaration and 94.3% Chinese text. Both render fine, both pass a head-tag audit, both would fail this one.

What a wrong `lang` actually breaks

It does not break rendering, which is why it survives. It breaks the parts of the stack that read the attribute instead of the pixels:

Screen readers choose a voice from lang. An English voice reading Han characters either spells them out or skips them, depending on the engine. This is the one failure mode with a real user on the other end.
Translation prompts. Chrome keys its offer to translate off the declared language first. A page that says en rarely gets the offer, even when every word is Chinese.
lang-aware CSS. :lang(zh) rules never match, and quote characters, hyphenation, and text-transform follow English rules on Chinese text.
Font fallback. The browser resolves fonts per script, but the declared language steers the choice within a script — and it decides whether CJK glyphs come from the page's intended font stack or from whatever the OS falls back to.
Search. Google has said for years that it infers page language primarily from the visible content, so I am not going to claim a ranking delta from 104 pages. What I can say is that the stack had no second opinion to offer: 28 of 100 pages carried any hreflang alternate, zero of 100 sent a Content-Language header, and all fifteen mislabeled pages carried zero hreflang. Nothing in the markup disagreed with the wrong tag.

The control case is boring, which is the point: the same FAQ template on another domain in the same network declares en and is 0% Han. Same generator, correct locale, because that site's config was set correctly.

The check, as a CI gate

I turned this into a script that crawls a sample and exits non-zero. It is thirty lines and it would have caught all fifteen:

import sys, urllib.request
from language_fingerprint import language_fingerprint  # the function above

URLS = [...]  # one page per template, not every page

def check(url, expect):
    req = urllib.request.Request(url, headers={"User-Agent": "lang-guard/1.0"})
    with urllib.request.urlopen(req, timeout=20) as r:
        fp = language_fingerprint(r.read().decode("utf-8", "replace"))
    declared = fp["declared"].split("-")[0].lower()
    share = fp["han_share"] or 0
    implied = "zh" if share > 0.10 else "en"
    ok = declared == implied
    print(f"{'ok  ' if ok else 'FAIL'} {url} declared={declared} "
          f"implied={implied} han={share}")
    return ok

failed = [u for u, e in URLS if not check(u, e)]
sys.exit(1 if failed else 0)

Run it on one representative page per template — not per URL. The failure I found was template-level, so a four-URL sample per site was enough to catch it on four sites out of 28.

The fix is one line, the source of truth is the hard part

<html lang="en"> in a root layout is a hardcoded default pretending to be a decision. The page's language belongs to the content, or to the locale segment of the route, and the layout should read it:

// __root.tsx — one locale, resolved per request, never a literal
export function Root({ locale, children }: { locale: "en" | "zh"; children: ReactNode }) {
  return (
    <html lang={locale}>
      <body>{children}</body>
    </html>
  );
}

If articles carry their own language column, use that instead and let the route default be a fallback that logs when it fires. A fallback that fires silently is how you end up with a site whose Chinese content announces itself as English for as long as nobody counts characters.

Two more things the same crawl turned up

2,777 leaked source annotations. Seven pages across two domains shipped attributes like data-tsd-source="/src/components/Layout.tsx:10:9" — 2,777 of them in total. That is TanStack Start's dev-mode source annotation surviving into production HTML. It costs users nothing to download and it hands anyone who views source your component tree, file names, and line numbers. If you see it, your production build is running with the dev transform on.

Zero Content-Language headers. Not one of the 100 pages that answered sent the header. That is normal for most static stacks and not a bug on its own — but combined with a wrong lang, it means the response carried no language signal that was correct.

What this does not prove

Four pages per domain, one crawl, one afternoon. The Han-ratio heuristic ignores mixed-script pages, and I dropped four URLs that failed on re-fetch rather than retrying them. I have no before-and-after data on indexing or rankings, because I changed nothing — this was a measurement pass, and the fix is not shipped yet.

What it does establish is narrower and more useful: a head-tag audit says nothing about whether your lang attribute is true, and on a 28-site sample, 15 pages were lying about it in the same direction, from the same root layout, on four sites that had never checked.

My sitemap listed 1,926 entries. Every one was a subdomain.

tong zhang — Tue, 03 Nov 2026 02:10:15 +0000

I expected 23 of my 119 sites to be underperformers. I did not expect Google to have zero of their real pages in the index.

I sampled 15 URLs per property through the Search Console URL Inspection API and grouped the results. On 91 properties the sample came back with apex URLs, and 713 of 1,346 sampled URLs were Submitted and indexed: a 53.0% hit rate. On 23 properties the sample returned no apex URLs at all. Every URL Google knew about for those domains, 2,104 of them, sat on a subdomain.

Those subdomains were mine. I just never meant to publish them.

The measurement

The check is small enough to run in a loop. For each property, ask the inspection API for a sample of URLs and bucket each result by whether its host matches the property:

import json, urllib.request

def inspect_sample(domain, token, limit=15):
    url = ("https://searchconsole.googleapis.com/v1/urlInspection/index:inspect")
    body = json.dumps({"inspectionUrl": f"https://{domain}/",
                       "siteUrl": f"sc-domain:{domain}"}).encode()
    req = urllib.request.Request(url, data=body, headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)

def bucket(domain, rows):
    same = [r for r in rows if r["url"].split("/")[2] == domain]
    foreign = [r for r in rows if r["url"].split("/")[2] != domain]
    return {"same_host": len(same), "foreign_host": len(foreign),
            "indexed": sum(1 for r in same if r["verdict"] == "PASS")}

The split is blunt: 91 properties had a mix leaning toward apex URLs. Twenty-three had same_host: 0 and a foreign_host count in the dozens or low hundreds. Two more were lopsided rather than empty, with 62 apex URLs against 250 subdomain URLs each.

That last group is the tell. A property with 62 apex URLs and 250 subdomain URLs is not a property with a crawling problem. It is a property where something else is competing for the same index space, and winning.

The pages that lost that competition are the ones I actually maintain. These are not thin domains with nothing to index: they carry the same kind of work as the properties that sampled cleanly. On a healthy property that work does show up in the sample. predictionguidecn.com returned 183 apex URLs with 11 of its 15 sampled pages Submitted and indexed; eventmarketcn.com returned 145 apex URLs with 10 of 15 indexed. Content of that shape includes things like the encyclopedia entry on how liquidity is priced in prediction markets, the comparison of how Polymarket and Kalshi differ on fees and settlement, and a year of crypto market trend archives.

On the 23 properties, the same sampling loop came back empty. Same content pipeline, same templates, no apex URLs in the index at all.

The subdomains were alive, and they claimed themselves

Before assuming a DNS wildcard was pointing at a parking page, I fetched a few. 21 of the 23 responded 200. All 21 declared index, follow in their robots meta. And all 21 pointed their canonical at themselves:

https://a7a5.altcoinfundingdesk.com/
  status    : 200
  canonical : https://a7a5.altcoinfundingdesk.com/
  robots    : index,follow,max-image-preview:large
  title     : A7A5 资金费率事件怎么复盘？

https://altcoinfundingdesk.com/
  status    : 200
  canonical : https://altcoinfundingdesk.com/
  robots    : index

Both are correct by their own rules. Neither knows the other exists. That is the whole failure: two URLs, same publishing system, two independent canonical claims, no cross-reference. Google has to pick one, and on 23 properties it picked the subdomain.

Twenty-one out of twenty-one is not a random sample. When every single host self-canonicalizes, the canonical tag is not expressing an editorial decision. It is expressing whatever request.headers.host happened to be.

Why Google went looking there

A canonical tag does not create discovery. Something has to link to those subdomains first, or submit them.

The apex sitemap.xml was doing it.

I fetched the sitemap from eight of the affected domains and counted <loc> entries of each shape. 1,926 entries in total. Every one pointed at a subdomain sitemap. Zero pointed at a page on the domain that served the file.

cryptochartshub.com       loc=256    subdomain=256    apex=0
altcoinfundingdesk.com    loc=118    subdomain=118    apex=0
dexguidehub.com           loc=278    subdomain=278    apex=0
tokenmarginlab.com        loc=256    subdomain=256    apex=0
perpfundingwatch.com      loc=256    subdomain=256    apex=0
coinwikicenter.com        loc=256    subdomain=256    apex=0
web3academyhub.com        loc=256    subdomain=256    apex=0
exchangeguidehub.com      loc=250    subdomain=250    apex=0
--------------------------------------------------------------
total                     loc=1926   subdomain=1926   apex=0

100.0% subdomain, 0.0% apex. The file named sitemap.xml on cryptochartshub.com was a sitemap index listing 256 per-token microsites, and not one entry for cryptochartshub.com itself.

So the sequence was not mysterious. The sitemap told Google "here are 256 sites", Google crawled them, each one said index, follow and claimed itself as canonical, and the apex pages were never submitted at all. No contradiction existed for Google to resolve, because nothing ever told it the apex mattered.

One config value doing two jobs

The generator reads a content registry: one record per token, one per site. The registry is the source of truth for what to build. Somewhere along the way it also became the source of truth for what to submit.

The sitemap builder walked the registry and emitted every host it found:

def build_sitemap_index(registry):
    # one loc per host the content registry mentions
    locs = [f"https://{entry['host']}/sitemap.xml" for entry in registry]
    return render_index(locs)

And the canonical helper did the same thing one layer down, reading the host off the request instead of off the config:

canonical = f"https://{request.headers['host']}{request.path}"

Two lines, same assumption: that "the host this code is running on" and "the host this content belongs to" are the same value. They are the same value in production with one deployment and one domain. They stop being the same value the moment a registry has more than one host in it, which is exactly the situation the registry exists to describe.

The fix is not a canonical tag on the subdomains. It is making the canonical host an explicit configuration value and deriving both the sitemap and the canonical tag from that value alone:

PRIMARY_HOST = config["primary_host"]          # e.g. "cryptochartshub.com"

def canonical_for(path: str) -> str:
    return f"https://{PRIMARY_HOST}{path}"

def build_sitemap_index(site):
    # only the site's own sitemaps, never the registry's other hosts
    return render_index([f"https://{PRIMARY_HOST}/sitemap-{n}.xml"
                         for n in site.sitemap_shards])

def guard_host(request):
    if request.headers["host"] != PRIMARY_HOST:
        return redirect(f"https://{PRIMARY_HOST}{request.path}", status=301)

Three changes, and they have to ship together. Fixing the sitemap alone leaves the subdomains reachable and self-canonical. Fixing the canonical alone leaves the sitemap advertising 256 hosts. The redirect is what actually retires the duplicates, because a 301 from the subdomain to the apex is the only signal that says "these were never separate things".

How to check your own site in two minutes

You do not need Search Console to catch this. The sitemap check is a single request per domain, and it finds the problem before the index does:

import re, urllib.request, urllib.parse

def audit_sitemap(domain):
    req = urllib.request.Request(
        f"https://{domain}/sitemap.xml",
        headers={"User-Agent": "Mozilla/5.0 (compatible; sitemap-audit/1.0)"})
    with urllib.request.urlopen(req, timeout=20) as r:
        xml = r.read().decode("utf-8", "ignore")

    locs = re.findall(r"<loc>([^<]+)</loc>", xml)
    hosts = [urllib.parse.urlsplit(u).netloc for u in locs]
    apex = sum(1 for h in hosts if h == domain)
    sub  = sum(1 for h in hosts if h.endswith("." + domain))
    other = len(hosts) - apex - sub
    return {"domain": domain, "locs": len(locs),
            "apex": apex, "subdomain": sub, "foreign": other}

for d in ["cryptochartshub.com", "dexguidehub.com", "web3academyhub.com"]:
    print(audit_sitemap(d))

Run it against your own domain. If apex is 0 and locs is not, you have the same bug, and it has been running for as long as the registry has had more than one host in it. If foreign is non-zero, someone else's host is in your sitemap, which is a different and more urgent problem.

What I would do differently

The index damage is not repaired by fixing the code, and I am not going to pretend otherwise. I have not finished the rollout, so I have no recovery number to report. Consolidating duplicate hosts takes weeks, not days. The honest measurement window is four to six weeks, and the metric to watch is coverageState in the URL Inspection response, not raw impressions, which bounce around for unrelated reasons.

Two things I would keep from this:

Treat the host as a config value, not a runtime fact. Any string built from request.headers.host that ends up in a canonical, an og:url, a sitemap, or a redirect target is a bug waiting for a second deployment target. There is no version of this that stays correct when the same code serves two hostnames.

Audit the shape of your sitemap, not just its size. Every SEO check I had counted URLs and checked status codes. All of them passed, on a sitemap where 100% of the entries belonged to somebody else. A sitemap index that is 100% subdomains looks perfectly healthy to any tool that only counts <loc> elements and fetches them.

The 2,104 subdomain URLs were not spam, not a hijack, and not an attack. They were my own build output, deployed exactly where the config told it to go, competing with the pages I wanted indexed. That is the part that took longest to accept: nothing was broken in the sense of returning an error. It was all working, and pointing at the wrong host.

38.8% to 58.5% indexed in 15 days. Five sites out of 45 did half of it.

tong zhang — Mon, 02 Nov 2026 02:32:56 +0000

Submitted-and-indexed went from 164 to 247 on the same 524 URLs. That is 38.8% → 58.5% in fifteen days, the best number this fleet has produced all year.

Then I grouped the same rows by site. Nineteen of 45 sites moved up, 24 did not move at all, and two went down. The five biggest movers produced 43 of the 85 newly indexed URLs — 50.6% of the gain — while eight sites still had zero of ten sampled URLs indexed.

The aggregate was real. It was also a story about five sites that the other forty were invited to.

This post covers what I measured, the two hypotheses that failed, and the instrumentation bug that had been hiding a fifth of the network from every report I ran.

The sample, and what it is not

Forty-five sites, ten sitemap URLs each, inspected through the Search Console URL Inspection API on 2026-10-15 and again on 2026-10-30. Same 524 URLs both times — intersection 524, zero added, zero dropped. A fixed panel, so the movement below is genuine URL-level state change rather than sampling churn.

What it is not: a census. The fleet serves 67,543 HTML pages across 119 sites, so 524 URLs is 0.8% of it, and the panel is capped at ten per site on sites with a median of 243 pages. Every number after this point is a sample statistic. I keep saying that because the last time I forgot, I shipped a slide that said "1.5% redirect rate" about two broken domains.

The pages in the panel are mostly long-form explainers — the kind of thing a site keeps under /learn/: how to read a prediction-market price as a signal, Kelly sizing for binary markets, macro-indicator glossaries. Boring, stable, cheap to produce. That matters later.

A fifth of the panel could never be measured

102 of 524 inspections came back unusable. Not "no data" — an actual error, API_ERROR_403, with the message every Search Console user eventually meets:

"You do not own this site, or the inspected URL is not part of this property."

The URL is fine. The property is wrong. Twenty-eight of the 119 sites in this fleet are multi-root: one apex plus up to 280 subdomains (aave.coinatlaslab.com, apenft.altcoinfundingdesk.com, and 5,700 more across the network). The Search Console property is registered on the apex. Pages served from aave.coinatlaslab.com are therefore outside every property I own, which is technically correct and operationally useless: the sitemap lists them, Google may well index them, and no coverage report will ever show either fact.

Of the 112 subdomain URLs in the panel, 101 returned 403 — 90%. They span 16 domains. The one non-403 error was a 500 on a retry-after-three-attempts host.

The obvious fix is a property per subdomain, and it does not scale: 5,739 roots would need 5,739 properties. The cheaper fix is to stop putting subdomain URLs in an apex sitemap you cannot report on, or to front them with a canonical to an apex URL you can. Either way, the instrumentation has to check host ownership before it trusts a coverage state:

from urllib.parse import urlparse

def owner(url, prop):
    """apex | sub — sub means the property cannot inspect this URL at all."""
    return "apex" if urlparse(url).netloc == urlparse(prop).netloc else "sub"

def inspect(url, prop, token):
    body = json.dumps({"inspectionUrl": url, "siteUrl": prop,
                       "languageCode": "en-US"}).encode()
    req = urllib.request.Request(
        "https://searchconsole.googleapis.com/v1/urlInspection/index:inspect",
        data=body, method="POST",
        headers={"Authorization": "Bearer " + token,
                 "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=45) as r:
            res = json.load(r)["inspectionResult"]["indexStatusResult"]
            return res.get("coverageState", "UNKNOWN")
    except urllib.error.HTTPError as e:
        # 403 = not my property. Do not fold this into "not indexed".
        return "API_ERROR_%d" % e.code

The important line is the comment. If you count a 403 as a coverage state, your index rate silently improves whenever you sample more subdomains, because the failures vanish from the denominator. Mine did exactly that for three weeks before I looked.

What actually moved

Denominator 423 → 422, so the delta is not an artifact of dropped rows:

coverageState	10-15	10-30	change
Submitted and indexed	164 (38.8%)	247 (58.5%)	+19.8 pt
Discovered – not indexed	160 (37.8%)	82 (19.4%)	−18.4 pt
URL unknown to Google	94 (22.2%)	84 (19.9%)	−2.3 pt
Crawled – not indexed	1	0	−0.2 pt
Soft 404	0	2	+0.5 pt
Page with redirect	0	2	+0.5 pt
Duplicate, no user canonical	0	3	+0.7 pt

150 of 524 URLs changed state. The forward bucket is the one you want: 67 went Discovered → Indexed, 17 went Unknown → Indexed, 1 came back from Crawled. The backward bucket is the one the dashboard hides: 33 URLs went Discovered → Unknown, meaning Google dropped even the queued-crawl record it previously had. Add the new soft 404s, redirects and duplicates and 42 of the 150 transitions were regressions.

A network-level "+20 points" and "42 URLs quietly lost their place in the crawl queue" are the same dataset. Only one of them fits on a chart.

Five sites did half of it

from collections import Counter

def per_site(rows, prop_key="site"):
    n, idx = Counter(), Counter()
    for r in rows:
        n[r[prop_key]] += 1
        idx[r[prop_key]] += r["coverage"] == "Submitted and indexed"
    return {s: (idx[s], n[s]) for s in n}

base, cur = per_site(rows_1015), per_site(rows_1030)
delta = {s: cur[s][0] - base[s][0] for s in cur}

gain = sum(d for d in delta.values() if d > 0)          # 85
top5 = sum(sorted((d for d in delta.values() if d > 0), reverse=True)[:5])
print(top5 / gain)                                     # 0.506
print(sum(1 for d in delta.values() if d == 0))        # 24 sites, unmoved

Nineteen sites up, 24 flat, 2 down. Gross gain 85, gross loss 2, net 83. Half the gain sits in five domains: predictmarketcn.com 0→10, eventmarketcn.com 0→9, predictionlabcn.com 0→9, marketforecastcenter.com 1→9, marketpredictionnow.com 0→7.

Now the part I cannot explain well. Those five sites share a template, a deploy pipeline, and a sitemap generator with the eight sites that stayed at zero. markettrendlab.com publishes 1,111 sitemap URLs — a per-coin technical and fundamental analysis archive — and indexed 0 of 10 sampled pages in both rounds. coinatlaslab.com has 1,612 pages and 0 of 10. predictioncn.com, predictionwikicn.com, marketoraclecenter.com, eventmarkethub.com, coinwikipro.com: also 0 of 10.

Same code, same host, same day. If your model of indexing is "publish, ping sitemap, wait", this table is telling you the model is missing a variable. I have not isolated it yet, and I am not going to pretend a fifteen-day window with one measurement per URL did.

The hypothesis that failed

The fleet-wide crawl I ran the same morning gave me a clean suspect: 915 of 67,543 pages (1.35%) had zero internal inlinks. 109 of 119 sites had at least one orphan, 16 sites were 100% orphaned, and orphan pages are exactly the ones that sit in Discovered forever. It should have predicted the index rate.

It did not. On the 45 sites in the panel, the correlation between orphan ratio and indexed ratio is r = 0.034 — indistinguishable from noise.

site	orphan ratio	indexed
marketoracledaily.com	0.00%	1/10
coinatlaslab.com	0.00%	0/10
polymarkettutorials.com	4.84%	0/10
predictionwikicn.com	7.03%	0/10

The two worst orphan ratios in the fleet have the same index rate as two sites with none. A 1.35% orphan rate is simply too small to move a ten-URL sample: you would need the orphans to be sampled, and at ten per site they mostly are not. I fixed nothing here, and the honest conclusion is that the orphan census was measuring the wrong population for this question — it is a link-graph health metric, not an indexing predictor.

The second experiment, which is less flattering

On 2026-10-16 I shipped server-rendered static output for eight single-page-app sites, so that crawlers would get real HTML instead of an empty <div id="root">. Fourteen days later I sampled 15 routes per site — 120 URLs, deterministic seed, drawn from the manifest of routes that were actually static-rendered:

result	count	share
Submitted and indexed	36	30.0%
URL unknown to Google	47	39.2%
Discovered – not indexed	25	20.8%
Duplicate / canonical conflicts	10	8.3%
Other (redirect, alternate canonical)	2	1.7%

Per site: coinpriceoracle.com 13/15, coinforecastpro.com 8/15, then bullrunalpha.com, cryptobullforecast.com, nextaltcoinhub.com at 5/15, and three sites at 0/15.

Thirty percent after two weeks is not a failure — new routes need discovery time, and the pre-change baseline for these same routes was not measured, which is the mistake I would undo first. But 39% of freshly rendered URLs are still unknown to Google, not "discovered, waiting". Google has no record of them at all. That is a discovery problem, not a quality problem, and no amount of on-page work fixes it.

What I changed in the instrumentation

Four rules, all of which cost me a wrong report first:

Report per-site distribution, never the fleet average. 19 up / 24 flat / 2 down next to +19.8 pt is the whole story; either one alone is misinformation.
Separate transport and permission errors from coverage states. A 403 is missing instrumentation, not a bad page. Keep it out of the denominator and print it as its own line.
Snapshot the same URL panel. 524 URLs, identical both runs, made a 150-row transition matrix possible. Random sampling each time would have produced a plausible number and no explanation.
Measure the baseline before shipping the fix. I did this for the panel and did not do it for the static-rendering rollout, so the most interesting change in the quarter has no before-row.

The fleet now sits at 58.5% indexed on the panel, with a fifth of its URLs structurally unmeasurable and a discovery gap on the newest pages that no dashboard will show me. Next window I am going after the 33 URLs that fell out of the crawl queue — those are the ones with an answer in them.

A 1.5% redirect rate looked fine. One site's sitemap was 92% redirects.

tong zhang — Thu, 29 Oct 2026 02:09:25 +0000

I probed 1,089 sitemap URLs across 119 sites with redirects switched off. 1,062 came back 200. Sixteen came back 301. That is a 1.5% redirect rate, which reads like a rounding error — the kind of number you put in a slide and move on from.

Then I grouped the same results by domain. Two sites accounted for all sixteen: one at 11 of 12 (92%), one at 5 of 12 (42%). The other 117 sites contributed zero. The aggregate number was not a measurement of the fleet. It was 1,062 green lights diluting two red ones.

That is the bug this post is about, and it is not a redirect bug. The redirects were correct and intentional. The bug was that my sitemap validator — and probably yours — had been reporting on an XML file for months and calling it a verdict on the server.

What the validator actually checks

Before touching HTTP, I ran the string layer over all 119 sitemaps: 31,838 URLs, checking for duplicate <loc> entries, trailing-slash duplicates, illegal percent-encoding, and cross-host URLs.

check	result
exact duplicates	0
trailing-slash duplicate pairs	0
illegal encoding	0
cross-host URLs	4,485 in 25 sites

Format-wise, the fleet was clean. Zero duplicates, zero encoding faults, and the sitemap parsed as valid XML in every case. If your CI asserts on sitemap shape — and mine did — this run passed.

Four thousand four hundred and eighty-five URLs were pointing at hosts that are not the site that submitted them, which is its own problem, but the format was flawless. A document can be perfectly well-formed and still describe URLs that the server refuses to serve. Nothing in the XML tells you which.

Probe the server, not the file

The probe has to do one thing the validator does not: refuse to follow redirects. curl without -L, requests with allow_redirects=False, or in bare stdlib, a redirect handler that returns None:

import concurrent.futures as cf
import urllib.request, urllib.error

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/141.0"

class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *args, **kwargs):
        return None                      # hand the 3xx back instead of chasing it

OPENER = urllib.request.build_opener(NoRedirect)

def probe(url, timeout=20):
    """Return (status, Location). status is None on transport failure."""
    req = urllib.request.Request(url, headers={"User-Agent": UA,
                                               "Range": "bytes=0-0"})
    try:
        with OPENER.open(req, timeout=timeout) as r:
            return r.status, r.headers.get("Location")
    except urllib.error.HTTPError as e:   # 3xx and 4xx arrive here
        return e.code, e.headers.get("Location")
    except Exception as e:
        return None, type(e).__name__     # do NOT count these as failures yet

Range: bytes=0-0 keeps the transfer at a byte; you are asking for a status line, not a document.

That last line matters more than it looks. On this run, 11 URLs came back as transport errors, and every single one returned 200 when retried with curl. Python's TLS stack was throwing record layer failure on hosts that were perfectly healthy — confirmed by those hosts having normal impressions in Search Console the same week, and by the second opinion:

import subprocess

def probe_curl(url, timeout=20):
    out = subprocess.run(
        ["curl", "-sS", "-o", "/dev/null", "-w", "%{http_code}",
         "-m", str(timeout), "-A", UA, "--max-redirs", "0", url],
        capture_output=True, text=True).stdout.strip()
    return int(out) if out.isdigit() else None

Never let a language's HTTP client be the final word on whether a host is up. A None means "my client failed", not "their server failed". Those are different findings and they go in different columns.

The aggregate that hid everything

The number that matters is not the fleet-wide rate. It is the per-site rate, with a hard threshold, because trailing-slash breakage is not distributed — it is all-or-nothing per deploy configuration. A site either normalizes to slash or it does not.

def audit(locs_by_site, workers=16):
    stats = {}
    with cf.ThreadPoolExecutor(max_workers=workers) as ex:
        jobs = {ex.submit(probe, u): (dom, u)
                for dom, urls in locs_by_site.items() for u in urls}
        for fut in cf.as_completed(jobs):
            dom, url = jobs[fut]
            code, loc = fut.result()
            s = stats.setdefault(dom, {"n": 0, "ok": 0,
                                       "slash": 0, "moved": 0, "error": 0})
            s["n"] += 1
            if code == 200:
                s["ok"] += 1
            elif code is None:
                s["error"] += 1
            elif code in (301, 302, 307, 308):
                # a redirect that only differs by the trailing slash is a
                # canonicalisation rule firing, not a URL migration
                if loc and loc.rstrip("/") == url.rstrip("/") and loc != url:
                    s["slash"] += 1
                else:
                    s["moved"] += 1
    return stats

def gate(stats, threshold=0.05, min_sample=8):
    bad = []
    for dom, s in stats.items():
        if s["n"] < min_sample:
            continue
        rate = s["slash"] / s["n"]
        if rate > threshold:
            bad.append((dom, s["slash"], s["n"], round(rate * 100, 1)))
    return sorted(bad, key=lambda r: -r[3])

That loc.rstrip("/") == url.rstrip("/") comparison is the whole diagnosis in one line. It separates two findings that look identical in a status code column:

slash — the URL is alive, but under a different spelling. Your sitemap and your host disagree about canonical form. Every crawl pays a redirect hop and Google records Page with redirect.
moved — the URL genuinely relocated. Real migration, different fix.

My first pass conflated them and reported "16 redirects" as one category. It is two.

Five policies behind eleven identical-looking sitemaps

The interesting result came from narrowing to the 11 sites whose sitemap contains only URLs without a trailing slash. If trailing slashes were a single convention, these 11 should behave alike. I fetched the first five URLs from each sitemap, hit them verbatim, then hit the opposite-slash spelling of each:

site	sitemap form	as-listed	opposite slash	reading
coinwikipro.com	no slash	200	200	both spellings served — duplicate risk
openaiipovaluation.com	no slash	200	200	both spellings served
openaipreipo.com	no slash	200	200	both spellings served
openaistocknews.com	no slash	200	200	both spellings served
cryptobullforecast.com	no slash	301	200	canonicalises to slash
forecastmarketcn.com	no slash	200	301	canonicalises to no-slash
marketsignalscenter.com	slash	200	301	canonicalises to slash
marketpredictionworld.com	no slash	200	403	opposite spelling blocked
polymarkettutorials.com	no slash	200	404	static files, no directory index
yesno-market.com	no slash	200	404	static files, no directory index
openaiipostock.com	no slash	200	200 ×4, 301 ×1	mixed within one sitemap

Five distinct policies across eleven sitemaps that are, at the XML level, byte-for-byte interchangeable in shape. A generated comparison page that canonicalises to the slashless form and a static hand-written page that has no directory index at all look the same to every sitemap linter ever written, and disagree about what /foo means.

Only one of the five policies is a defect today. And that framing is the trap.

The sites that pass are the ones to worry about

Four of those sites return 200 for both spellings. That is not a correct configuration — it is a tolerant one. Two URLs serve identical content, and whether Google consolidates them depends entirely on a <link rel="canonical"> you may or may not have emitted. Nothing in the sitemap expresses this, and nothing in the response code flags it.

The single redirecting site in the fleet is the one I'd look at as a fixed target: its category pages are listed in the sitemap in their non-canonical form, so a crawler fetches a URL whose only job is to hand back a 301 pointing at the slash version. Eleven of twelve sampled URLs did exactly that.

At the other end, marketpredictionworld.com answers the opposite slash with 403 rather than a redirect — five out of five. Not a 301, not a 404. Something in front of that origin treats the alternate spelling as forbidden, which means a single wrong link anywhere on the site produces a status code that reads like a block rather than a typo.

So here is the uncomfortable part. Nine of those eleven sites pass my gate right now. Their safety comes from server leniency, not from their sitemap being right. Change hosting, flip one framework config value, migrate the origin — and those nine flip to 301 together, on a day when nobody touched a page. The two sites I can name today are the ones that already flipped.

Two independent observers, one root cause

Redirects alone would be arguable — maybe Google tolerates them. It does not, and I did not have to guess: Search Console's sitemap report for that same domain went from 2 warnings to 3 in the same week the probe found 11 of 12 URLs redirecting. A server-side probe and a Google-side report, arrived at independently, naming the same file.

That coincidence is worth more than either observation. When a status probe and a Search Console warning agree on the same URL set, you have a root cause. When only one of them fires, you have a hypothesis.

The rest of the fleet-wide sitemap errors told the same story from the other direction: five sites were reporting errors, and all five were the sites whose sitemaps were 100% cross-host URLs. The Google-side error and the XML-side pollution were the same fact recorded by two systems.

The gate

What I run now, in order, on every deploy:

Parse, then probe. Validate the XML, then request every URL in it — or a sample of at least 12 per site — with redirects disabled. Format checks and liveness checks are not substitutes for each other.
Probe the URL as listed. Do not normalize first. My own first attempt at a control case rewrote the slash before requesting it, which measured a URL that was never in the sitemap and produced a convincing false alarm. The sitemap string is the input to the test.
Classify redirects by target. Location equal to the source modulo trailing slash is a canonicalisation bug. Anything else is a move. Different fixes.
Threshold per site, never fleet-wide. Any site above 5% slash-redirects on ≥8 samples fails the build. A 1.5% fleet average is compatible with one site being at 92%, and the average is what gets reported.
Retry transport errors out of band. Curl before believing that a host is down.
Treat 403 and 404 on the alternate spelling as intent. They tell you which policy the origin actually implements, which is more reliable than reading the framework's config.

None of this is exotic. It is about thirty lines and a threshold. What made it worth writing down is that for months the check I had was the parse, and the parse was green the entire time — on a fleet where one site had 92% of its URLs answering with a redirect that cost a crawl hop each, and nine more were one config change away from joining it.

DEV Community: tong zhang

Chrome got zstd for the HTML and gzip for the CSS. Brotli was 279 KB smaller.

What I actually measured

The HTML got zstd, all 114 of them

The assets got gzip, 480 out of 492

What is inside a 1.09 MB stylesheet

Where brotli does not help at all

The negotiation table that explains it

Two hygiene findings, one good, one not

What I am doing about it

21 of our 114 sites answer every missing URL with the homepage. Search Console calls that indexed.

What "doesn't exist" means when you're the one asking

64 of 342 came back 200

21 sites, one behaviour: the homepage, byte for byte

It isn't cloaking

The fallback survives every URL shape

Then I checked the sitemaps

4,868 declared URLs, one real page

Search Console calls this "indexed"

Four signals, four answers

Why this happens

The fix is smaller than the bug

What I changed

Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.

The file is half legal notice

Thirty-one identical bans across 114 files

The wildcard group forbids nothing

Two status codes, two different layers

The bans that never fired

A 520 is the wrong way to say no

The sitemap was in perfect shape the whole time

Checking your own network

What I am changing

104 of 114 homepages ship no images at all. Their only image is a 270 KB social card.

104 pages, zero images

The four pages that do ship <img>

The only image a crawler ever fetches

10 of 20 cards are lying about their own size

The weight is the part I'd fix first

Why I think this matters

Reproduce it on your own site

What I'd do, in this order

The 7.8 KB tracker got an ETag, a 12-hour TTL and an edge HIT. The page got none of them.

What I measured

The audit is short

What the 114 pages sent

A 304 is cheap in bytes and expensive in time

The same network gets the assets right

The 12-hour TTL on filenames that never change

"Heuristic caching" is the browser guessing on your behalf

What I would send instead

What this does not prove

A search box needed titles. The page sent 300 full records, 41,000 characters of meta descriptions.

How the measurement works

The 30 sites, by the numbers

Failure mode 1: the payload is data, and most of the data is unused

Failure mode 2: the payload is markup, and the markup is repetitive

gzip hides this, and that is the trap

A CI check you can run in ten seconds

Four fixes, in order of payoff

What this measurement does not prove

26 of 114 sitemaps gave every URL the same lastmod. One repeated a millisecond 312 times.

What I actually collected

The test is a counter

Failure mode one: frozen at the first render

Failure mode two: stamped with the build date, not the content date

The server already disagrees with the sitemap

What lastmod is and is not

The fix: hash what you render, publish when the hash changed

A CI check that fails when the field collapses

What this does not fix

19 of 424 JSON-LD blocks were not JSON. Every one had JavaScript stapled to the end.

What I pulled

The test is a parser, not a validator

What the something else was

It never runs. That is the interesting part

The other 56 pages, which never had a shot

The check I now run on every deploy

What I changed

15 pages declared lang="en". 75-98% of their characters were Han.

The four pages that do ship `<img>`

What a wrong `lang` actually breaks