DEV Community: tong zhang

Static HTML passed on all 46 pages. Rendered in Chrome, 8 were broken

tong zhang — Fri, 23 Oct 2026 02:05:54 +0000

Static HTML passed on all 46 pages. Rendered in Chrome, eight of them were broken.

The audit was routine: 46 new pages across four sites, all generated from templates, all with correct <title>, <h1>, canonical and structured data. A static checker would return 46/46 green. Then I loaded them in headless Chrome and measured the box model, and eight of them had a layout that no HTML validator could see.

Why static checks cannot find layout bugs

A static check reads the document. A layout bug lives in the computed box model, which only exists after CSS cascade, font loading and JS hydration. Three of the eight had valid markup and a completely broken visual result:

A sidebar that was a grid child of the article container instead of a sibling, so it got placed inside the content column and squeezed the text to 40% width.
A page with no <main> element at all, because the template's hero section had been written before the wrapper and the wrapper was never added.
A page where an absolutely positioned element was 320px wide on a 390px viewport, forcing a horizontal scrollbar.

None of these throw, none fail validation, and all of them return 200.

The four measurements I run

const { chromium } = require("playwright-core");

const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 390, height: 844 } });
await page.goto(url, { waitUntil: "networkidle" });

const m = await page.evaluate(() => {
  const main = document.querySelector("main");
  const rail = document.querySelector("aside");
  const overflow = document.documentElement.scrollWidth - window.innerWidth;

  // 两个元素的重叠面积
  let overlap = 0;
  if (main && rail) {
    const a = main.getBoundingClientRect();
    const b = rail.getBoundingClientRect();
    const w = Math.min(a.right, b.right) - Math.max(a.left, b.left);
    const h = Math.min(a.bottom, b.bottom) - Math.max(a.top, b.top);
    overlap = w > 0 && h > 0 ? Math.round(w * h) : 0;
  }

  return {
    mainWidth: main ? Math.round(main.getBoundingClientRect().width) : 0,
    mainCount: document.querySelectorAll("main").length,
    railIsSibling: rail ? rail.parentElement === main?.parentElement : null,
    overlap,
    overflow,
    ctaVisible: !!document.querySelector("a[href*='t.me']")?.offsetParent,
    bodyText: document.body.innerText.length,
  };
});

Assertions, not prints:

measurement	fail condition	what it catches
`mainCount`	`!== 1`	missing or duplicated content landmark
`mainWidth` (390px viewport)	`< 300`	content squeezed by a misplaced grid child
`railIsSibling`	`false`	sidebar nested inside the article column
`overlap`	`> 0`	text running underneath a floated rail
`overflow`	`> 0`	horizontal scrollbar
`bodyText`	`< 800`	page rendered a shell instead of content

The width check on a mobile viewport is the one that earned its keep. At 1280px the squeezed column measured 512px and looked plausible in a screenshot. At 390px it measured 118px, which is obviously wrong the moment a threshold exists.

The bug behind the bug: hydrated content that disappears

Two of the sites are server-rendered app frameworks, and there the layout was fine while the content was the problem.

I had injected extra HTML into the static output — an internal-linking block, some extra structured data. It was present in the served HTML. Then the app hydrated, and the block vanished.

That is not a rendering bug, it is the framework doing what it is told: the client render owns the tree, and any DOM node that the client component tree does not know about gets discarded on hydration. Static injection into a hydrating app is fundamentally unstable.

The fix was to remove the hydration path from those pages entirely: strip type="module" scripts, modulepreload hints and the framework's payload scripts, keep the CSS, the analytics tag and the JSON-LD. The pages become fully static and the injected content stays. The cost is real — client-side navigation from those pages stops working — but for content pages that nobody navigates from, that trade was worth it.

The nastiest one: a streaming payload with a length prefix

If you hand-edit text inside a React Server Components payload (the self.__next_f.push([1, "..."]) blocks, or the flight stream), you can produce a white page with no error message.

Those payloads are length-prefixed:

3:TL5b,{"title":"Bitcoin forecast"}

L5b is a hex byte count of what follows. Change the text, forget to recompute the prefix, and the parser reads the wrong number of bytes, splits the record in half, and the client throws — React #412 territory, which surfaces as a blank page rather than a readable error.

The safe way to edit a text chunk:

import json

def rewrite_tchunk(record: str, new_text: str) -> str:
    # record = '3:TL5b,{"title":"..."}'  -> 首段 id + 长度前缀
    head, rest = record.split(":T", 1)
    old_len, body = rest.split(",", 1)
    payload = json.loads(body)          # 用真正的 JSON 解析拿到对象
    payload["title"] = new_text
    body = json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
    return "%s:T%x,%s" % (head, len(body.encode("utf-8")), body)

Two details that matter:

The prefix counts UTF-8 bytes, not characters. Chinese text will silently corrupt if you use len(str).
The record boundary is found by json.raw_decode() at the top level, not by regex. Splitting the stream on " produces garbage the moment a string contains an escaped quote.

I now run this as a post-generation assertion: every T chunk must parse, and the declared length must equal the byte length of the following segment. It fails loudly at build time instead of silently in a browser.

Two examples

The same audit, two page shapes:

A static page with a sibling rail — plain HTML, one main, aside as a sibling. Nothing can disappear on it.
A route from a hydrating app — server-rendered, hydration path removed so injected content survives. Same URL, different failure modes.

What I changed

The static checker still runs first — it is fast and catches typos, missing canonicals and duplicate analytics tags. But it is no longer allowed to be the gate on its own. Rendered measurements run second, on two viewports, with the table above as hard failures and a non-zero exit code.

Everything else — screenshots, read time, word count — is a report, not a check. If nothing can fail, nothing is being tested.

6 of 40 sites were pointing Google at a sitemap URL that returns HTML

tong zhang — Thu, 22 Oct 2026 05:30:11 +0000

A robots.txt Sitemap line that resolves to 200 is not the same thing as a readable sitemap. I found out the expensive way, on six sites at once, and the bug had been live for months.

The failure mode: 200 + text/html

$ curl -sI https://example.com/sitemap.xml.gz
HTTP/2 200
content-type: text/html; charset=utf-8

That is the entire bug, and it is completely invisible to normal monitoring.

The chain is boring and extremely common:

The site was originally built with a generator that gzipped its sitemap, so robots.txt got Sitemap: https://example.com/sitemap.xml.gz.
The build changed (Hugo out, a SPA build in). The .gz file stopped being produced.
The web server has the standard single-page-app fallback:

location / {
    try_files $uri $uri/ /index.html;
}

A request for a file that no longer exists therefore returns 200 OK with the app shell as HTML.

The crawler asks for a sitemap, gets a web page, and gives up. Silently. No 404 in the access log to alert on, no error in the build, and the submit button in Search Console still says "Success".

Six of the forty sites I look after had this exact robots.txt line. Every one of them returned a green 200.

Why status-code monitoring never catches it

A synthetic check that asserts status == 200 passes.
An uptime monitor passes.
A "sitemap submitted successfully" message in Search Console passes — submission is just a queue write.
Even a page-level error tracker passes, because nothing errored.

The only thing that noticed was Search Console URL Inspection, which reported entire sites as unknown to Google. That looks like an authority problem, so the first instinct is to buy links. That would have been money spent on the wrong problem: the discovery channel itself was broken.

Check 1 — parse the sitemap, don't fetch it

Never assert on the status code alone. Assert that the bytes parse as XML.

import re, urllib.request, xml.etree.ElementTree as ET

UA = {"User-Agent": "Mozilla/5.0 (compatible; sitemap-audit/1.0)"}

def sitemaps_from_robots(host):
    req = urllib.request.Request(f"https://{host}/robots.txt", headers=UA)
    txt = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "replace")
    return re.findall(r"(?im)^\s*sitemap:\s*(\S+)", txt)

def assert_real_sitemap(url):
    r = urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30)
    body = r.read()
    ctype = r.headers.get("content-type", "")
    if "xml" not in ctype:
        raise AssertionError(f"{url}: wrong content-type {ctype!r} ({len(body)} bytes)")
    try:
        root = ET.fromstring(body)
    except ET.ParseError as e:
        raise AssertionError(f"{url}: not XML — {e}; first bytes {body[:80]!r}")
    return root

Two details matter here:

content-type is a cheap early exit, but it is not sufficient. I have seen application/xml served for an error page.
ET.fromstring is the real assertion. "The string starts with <" passes for <!DOCTYPE html>, which is precisely the case we are trying to catch.

Six sites, six failures, thirty seconds of runtime. That one function was worth more than every link-building experiment I ran that quarter.

Check 2 — a sitemap full of soft 404s

Fixing the .gz path made the sitemap readable, and immediately exposed a second layer: the SPA fallback means any URL returns 200. So a sitemap can list thousands of URLs that are not files, do not correspond to a route, and render the generic shell.

Mine had 385 entries. 188 of them were fake.

The tell is that a soft 404 renders identical content to the fallback. So compare a fingerprint, not a status:

import hashlib, re

def fingerprint(html):
    m = re.search(r"<title[^>]*>(.*?)</title>", html, re.S | re.I)
    return hashlib.sha1((m.group(1) if m else "").encode()).hexdigest()

shell = fingerprint(open("index.html").read())

for loc in sitemap_locs:
    body = urllib.request.urlopen(urllib.request.Request(loc, headers=UA)).read().decode("utf-8", "replace")
    if fingerprint(body) == shell:
        print("SOFT-404:", loc)

For a React/Vue SPA this catches the majority of cases for free. For a pre-rendered site it is stricter: each route should have its own <title>, so equal titles means the route was not actually pre-rendered.

Result after pruning: 385 → 197 entries, all of which resolve to a distinct page.

Check 3 — wherever `canonical` points, that URL must return 200

This one I consider the most dangerous class of bug, because it does not look like a bug at all.

A page declared <link rel="canonical" href="https://example.com/en/">. Requests to /en/ returned:

HTTP/2 403

The directory existed. It contained only subdirectories. There was no index.html, and autoindex was off, so nginx answered 403. The homepage was telling Google: the authoritative version of me is that other URL, and that URL refused to serve anything.

In effect the site had been pointing its entire weight signal at a dead address, and the language entry point was closed to users as well.

Static HTML checks will not find it, because the canonical tag is injected by JavaScript. You need a rendered DOM and then a second request:

const { chromium } = require("playwright-core");

for (const url of pages) {
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: "networkidle" });
  const canonical = await page.evaluate(
    () => document.querySelector('link[rel="canonical"]')?.href || null
  );
  if (!canonical) { console.log("NO CANONICAL", url); continue; }

  const res = await page.request.get(canonical);
  if (res.status() !== 200) {
    console.log("DEAD CANONICAL", url, "->", canonical, res.status());
  }
}

That check takes about two seconds per page and is the single highest-value rule I added. It has since caught the same mistake on two other sites, both of which had a healthy-looking <link rel="canonical"> pointing at a 403.

Check 4 — probe language directories as their own URLs

Directory roots are not covered by sitemaps that only list article URLs. After finding one 403, I scanned 119 domains × 12 language prefixes (1,428 URLs) and found exactly one affected site — but that one was affected at both /en/ and /zh/, and it was the site whose root canonical pointed there.

If your site is example.com plus example.com/en/, treat those as four independent things to verify: status code, rendered <html lang>, <title>, and self-referencing canonical. A directory that renders correctly at / proves nothing about /en/.

How the two page types differ in practice

The audit covered both fully static pages and pre-rendered SPA routes, and the failure distribution was lopsided: static pages essentially never soft-404'd, because they are files. SPA routes failed whenever the pre-render step silently skipped a route and the fallback served the shell.

Two examples from that audit, if you want to compare the HTML:

A statically pre-rendered article — one file, one title, no hydration step that can drop content.
A route inside an SPA shell — pre-rendered HTML that then hydrates. This is the shape that produces soft 404s when the pre-render step misses a URL, and the shape where a static-only audit will happily report 200.

The lesson was not "SPAs are bad". It was that 200 means nothing in an app that returns index.html for everything. Every check has to assert on content, not on transport.

The pipeline I ended up with

Four layers, each one cheap enough to run on every deploy:

Static — every generated page: HTTP status, exactly one analytics tag, self-referencing canonical, title/H1 present, no placeholder strings.
Rendered — real headless Chrome: main content width, horizontal overflow, CTA visibility, and the canonical target returns 200.
Server-side — sitemap loc set vs. files on disk, lastmod not stale, no orphan pages, robots.txt Sitemap lines point at real XML.
Fleet — resolve robots.txt → sitemap for every domain, parse as XML, record the ones that need waiting on a CDN cache to expire.

The rule I keep coming back to: if a check only prints a value, it is not a check. It needs a threshold and a non-zero exit code, or it will pass forever while the site is invisible.

If you take one thing from this: go curl your robots.txt, pull the Sitemap line, and check the content-type. Six out of forty had it wrong, and none of them knew.