DEV Community

TBDS
TBDS

Posted on

One function that makes canonical, hreflang and sitemap unable to disagree

Multilingual static sites break in a boring, expensive way: the canonical tag says one URL, the hreflang alternates say a slightly different one, and the sitemap says a third. Google resolves the contradiction by ignoring your signals and picking whichever version it likes. You do not get an error. You get flat traffic and a "Duplicate, Google chose a different canonical" line in Search Console months later.

This is a write-up of a static generator — one file, no dependencies, no client JavaScript — built around two constraints:

  1. Every URL the site emits comes out of a single function.
  2. A page that does not clear a content-length threshold is not allowed to be indexable, and the build enforces it.

Why the usual setup fails

The usual setup is not careless. It is that URL construction is distributed. The canonical tag is written in a layout template. The hreflang block is generated in an i18n helper. The sitemap is emitted by a plugin. The nav links are written by hand. Four independent pieces of code each build "the URL for page X in language Y."

Each will be correct in isolation and they will still drift, because the interesting cases are the edges: trailing slash or not, does the default language live at / or /en/, is x-default a separate entity, does a page that only exists in some languages still emit alternates for all of them. Every one of those questions has to be answered identically in four places, and nothing checks that they were.

Client-side i18n makes it worse. If the language switcher swaps strings at runtime, then for a crawler every translation except the one baked into the HTML does not exist. You have one page pretending to be many.

What actually works: one exit point

The generator has exactly one URL constructor and one path constructor:

const ORIGIN = 'https://kibo.douhouse.com';

/** The single source of truth for every URL the site emits. */
const urlFor = (lang, slug) => `${ORIGIN}${pathFor(lang, slug)}`;

/** Root-relative form of the same URL, for links and assets inside a page:
 *  a page at /ja/ must not resolve ./styles.css against its own directory. */
const pathFor = (lang, slug) => {
  const base = lang === DEFAULT_LANG ? '/' : `/${lang}/`;
  return slug ? `${base}${slug}/` : base;
};
Enter fullscreen mode Exit fullscreen mode

Twelve lines, and they answer every edge case once. Default language at root. Trailing slash always — chosen because that is the form the hosting platform serves with a 200 and redirects to; a canonical pointing at a URL that 301s is a self-inflicted wound.

Crucially, write() uses the same pathFor to decide where the file goes on disk:

const write = (lang, slug, html) => {
  const dir = path.join(DIST, pathFor(lang, slug));
  fs.mkdirSync(dir, { recursive: true });
  fs.writeFileSync(path.join(dir, 'index.html'), html);
};
Enter fullscreen mode Exit fullscreen mode

So the URL structure and the directory structure cannot diverge either. There is no mapping layer to get wrong.

Head generation is also single-source

One head builder serves every page type. It takes langs — the set of languages this exact page exists in — and derives everything from it:

function headFor({ lang, slug, title, desc, langs, jsonld, noindex }) {
  const m = LANG_META[lang];
  const self = urlFor(lang, slug);
  const xdefault = langs.includes(DEFAULT_LANG) ? DEFAULT_LANG : langs[0];

  const alternates = langs.map(
    (l) => `  <link rel="alternate" hreflang="${LANG_META[l].html}" href="${urlFor(l, slug)}">`
  ).join('\n');

  return `  <meta charset="utf-8">
  ...
${noindex ? '  <meta name="robots" content="noindex,follow">\n' : ''}  <link rel="canonical" href="${self}">
${alternates}
  <link rel="alternate" hreflang="x-default" href="${urlFor(xdefault, slug)}">
  ...
  <meta property="og:url" content="${self}">`;
}
Enter fullscreen mode Exit fullscreen mode

Three properties fall out for free:

  • Canonical and og:url are literally the same variable. They cannot disagree.
  • hreflang is mutually referencing by construction. Every language of a page renders alternates from the same langs array, so page A links B and B links A automatically. The most common hreflang error — non-reciprocal annotation — is unrepresentable.
  • A page that exists in only some languages emits alternates only for those. Because langs is computed per page: GUIDE_LANGS.filter((l) => GUIDES[l]?.[slug]).

The sitemap uses urlFor too, so it is the same string a fourth time:

const entry = (lang, slug, langs, priority) => {
  const alts = langs.map(
    (l) => `    <xhtml:link rel="alternate" hreflang="${LANG_META[l].html}" href="${urlFor(l, slug)}"/>`
  ).join('\n');
  const xd = langs.includes(DEFAULT_LANG) ? DEFAULT_LANG : langs[0];
  return `  <url>
    <loc>${urlFor(lang, slug)}</loc>
${alts}
    <xhtml:link rel="alternate" hreflang="x-default" href="${urlFor(xd, slug)}"/>
Enter fullscreen mode Exit fullscreen mode

The thin-content gate

The second constraint is harder, because it is a rule about content, and content is written by humans in a hurry. A localized long-tail page that ends up as a heading plus two sentences is worse than not shipping it: it drags the whole directory's quality signal down.

So the build measures every page it just rendered and refuses to make short ones indexable.

function bodyWords(html, lang) {
  const main = html.match(/<main[^>]*>([\s\S]*?)<\/main>/i);
  if (!main) return 0;
  const text = main[1]
    .replace(/<(script|style)[\s\S]*?<\/\1>/gi, ' ')
    .replace(/<[^>]+>/g, ' ')
    .replace(/&[a-z]+;|&#\d+;/gi, ' ')
    .replace(/\s+/g, ' ')
    .trim();
  if (/^(ja|ko|zh)/.test(lang)) {
    const cjk = (text.match(/[぀-ヿ㐀-䶿一-鿿가-힯]/g) || []).length;
    return Math.round(cjk / 1.8);
  }
  return text ? text.split(' ').length : 0;
}
Enter fullscreen mode Exit fullscreen mode

Two details matter here. It counts <main> only, so boilerplate nav, footer and CTA cannot inflate a thin page into passing. And CJK has no spaces — splitting on whitespace would count an entire Japanese page as one gigantic "word" — so codepoints are converted to a rough English-word equivalent instead.

The gate itself is not a single pass, and this is the part I did not expect when I started:

function runGate() {
  const all = [];
  for (const lang of GUIDE_LANGS) {
    for (const slug of GUIDE_SLUGS) if (GUIDES[lang]?.[slug]) all.push([lang, slug]);
  }

  let indexable = new Set(all.map(([l, s]) => `${l}/${s}`));
  let counts = new Map();

  for (let pass = 0; pass < 10; pass++) {
    const next = new Set();
    counts = new Map();
    for (const [lang, slug] of all) {
      const key = `${lang}/${slug}`;
      const n = bodyWords(guidePage(lang, slug, indexable, false), lang);
      counts.set(key, n);
      if (n >= MIN_WORDS) next.add(key);
    }
    const stable = next.size === indexable.size && [...next].every((k) => indexable.has(k));
    indexable = next;
    if (stable) break;
  }

  return { all, indexable, counts };
}
Enter fullscreen mode Exit fullscreen mode

Delisting a page removes it from other pages' related-links lists. That shortens those pages. Which could, in principle, push a borderline page under the threshold. So the gate iterates to a fixpoint rather than deciding in one pass.

Delisting is not deletion. The page is still written, still reachable, still linked — it just gets noindex,follow and drops out of the sitemap. Related-link rendering degrades the entry to plain text rather than removing it:

return indexable.has(`${lang}/${s}`)
  ? `      <li><a href="${pathFor(lang, s)}">${t}</a></li>`
  : `      <li>${t}</li>`;
Enter fullscreen mode Exit fullscreen mode

And the build reports what it did, so a regression is visible in the log rather than in Search Console eight weeks later:

words: shortest 512, longest 1104
gate: N pages checked, 0 blocked (threshold 450)
Enter fullscreen mode Exit fullscreen mode

Costs and boundaries

The threshold is a proxy, not a measure of quality. Word count does not detect a padded page, a machine-translated page, or a page that says nothing at length. It only catches the specific failure of obviously too short. A page can clear 450 words and still deserve to be blocked; nothing here will tell you.

Rendering during the gate means pages are rendered more than once. runGate renders every page per pass, then the writer renders them again. For a site of this size that is milliseconds and completely irrelevant; if page generation ever became expensive, this loop would need memoizing per indexable set.

The fixpoint has an escape hatch that fails silently. pass < 10. If the set ever oscillates rather than converging — page A's links pushing B over the line while B's absence pushes A under — the loop exits after ten passes with whatever it happens to be holding. Nothing warns you. A console.warn on non-convergence is the obvious missing line.

bodyWords parses HTML with regex. It is correct for this generator because the generator itself emits the <main> tags, so the input is not arbitrary HTML. Point it at hand-authored markup with a nested <main> or a <main> mentioned in a comment and it will misbehave. This is acceptable only because producer and consumer are the same file.

The CJK divisor is a guess. cjk / 1.8 is a heuristic for "characters per equivalent English word." It is not derived from anything. It means the effective threshold is not the same across languages, which is a real inconsistency, just a small one.

Single origin is baked in. ORIGIN is a constant. Preview deployments on a different hostname emit production canonicals — which is arguably the safe failure (previews never compete for indexing) but it does mean the preview's own links are cross-origin absolute in the sitemap.

Zero dependencies is a real constraint, not a badge. No Markdown, no templating language, no image pipeline. Content lives as JavaScript objects and pages are built with template literals. That is genuinely fine at this scale and would be miserable at a hundred times the volume; the honest boundary is "small marketing site with a fixed set of page types."

This generator builds the Kibo site.

Top comments (0)