How We Structured Canonicals, Hreflang, and Sitemaps for a Multilingual AI Directory

Disclosure: I work on Toolso.AI. This article was prepared with AI assistance and checked against repository code and primary documentation. The code examples are reduced teaching fixtures, not production source or evidence of a complete live-site audit.

A multilingual directory has more URL states than its language menu suggests. A tool may be published in English, translated into Chinese, and missing a usable French description. The router can still render a French path. That does not automatically make it a French document worth advertising to search engines.

Our useful starting point was a contract: a published document, its canonical, its language alternatives, and its sitemap entry should agree about which page exists. This article explains that contract in a Next.js App Router application using next-intl. It separates the implementation we inspected from an extension we have not implemented: sitemap sharding.

1. Separate route keys from language annotations

The application supports ten route keys: en, zh, tw, ko, ja, fr, de, ru, es, and pt. English is the default and is unprefixed. Other languages use a path prefix. For example, the Raphael tool has an English route at /tool/raphael and a Simplified Chinese route at /zh/tool/raphael.

A route key is an internal identifier. It is not necessarily the value to emit in an HTML language annotation. Our configuration maps zh to zh-CN, tw to zh-TW, es to es-ES, and pt to pt-BR. The remaining keys map directly. In particular, emitting tw as a language code would confuse a routing convention with a language designation.

Keep this mapping explicit and review it alongside the actual editorial audience. A regional code describes targeting; it does not prove that a translation is culturally or linguistically appropriate. Adding a route must not silently add an unavailable translation to every document's alternate set.

2. Give URL construction one owner

We use one absolute URL builder underneath canonical, hreflang, and sitemap helpers. The following fixture illustrates the shape. It deliberately accepts only a small set of locales and plain internal paths; a real router must also handle its own encoded parameters and localized pathnames.

const origin = 'https://example.com';
const languageTags = {en: 'en', zh: 'zh-CN', tw: 'zh-TW'};

function documentUrl(path, locale) {
  if (!Object.hasOwn(languageTags, locale)) throw new Error('locale');
  if (!path.startsWith('/') || /[?#]/.test(path)) {
    throw new Error('plain internal path required');
  }
  const prefix = locale === 'en' ? '' : `/${locale}`;
  return origin + prefix + (path === '/' ? '' : path);
}

The production helper delegates locale path generation to the configured router. The important property is shared ownership, not copying this fixture into an application. Three separately maintained string templates can disagree about the default prefix, trailing slash, or hostname while each looks plausible in isolation.

Choose a stable homepage spelling and consistently emit it. Keep tracking parameters out of document identity. Pagination parameters require a separate policy: if page two has distinct list content and is indexable, canonicalizing every page to page one can erase that distinction. Our metadata helper can retain an approved canonical query and apply it consistently to language alternatives.

3. Build a language cluster from eligible content

Canonical answers which URL represents a document; hreflang describes available language or regional variants. A fully translated Chinese page should normally identify its own Chinese canonical, rather than declaring the English version canonical merely because English was written first.

Google's localized-page guidance requires fully qualified alternate URLs, self references, and reciprocal annotations. We derive the eligible language set from usable content for that particular record, then reuse the same set for every member of its cluster.

function languageCluster(path, available) {
  const eligible = Object.keys(languageTags)
    .filter(locale => available.includes(locale));
  const languages = Object.fromEntries(eligible.map(locale => [
    languageTags[locale], documentUrl(path, locale)
  ]));
  if (eligible.includes('en')) languages['x-default'] = documentUrl(path, 'en');
  return languages;
}

Here, x-default points to English only when English belongs to the eligible set. It is a deliberate fallback choice, not an instruction to label English as every missing language. Google also supports a language selector as the fallback destination; choose whichever accurately describes the site.

We disabled next-intl's automatic alternate response links and generate annotations from our content-aware helpers. That avoids maintaining a router-generated set of all configured languages alongside a narrower content-derived set. HTML and sitemap annotations can coexist, but Google does not require multiple implementations. If you emit both, agreement becomes an additional invariant to test.

4. Treat missing translations as an editorial state

A translated navigation bar around an English description is not equivalent to a translated tool description. In our tool-detail metadata branch, a missing translation can display the English fallback while marking that fallback route noindex, identifying the English original as canonical, and suppressing language alternates for the fallback page.

This is an explicit decision to keep an untranslated fallback out of Search. It is not a general recipe for consolidating duplicate URLs. Google advises using canonical annotations rather than noindex for canonical selection. Genuine translated pages remain separate language documents with their own canonicals.

Translation-query unavailability is another state: the application cannot trust the current language set. Our metadata branch suppresses alternates there too. Do not collapse missing translations, transient data failures, archived tools, and genuine translated pages into a single boolean called “duplicate.” Their content, retry behavior, and indexing policy differ.

The sitemap uses the same content eligibility standard: a nonempty translated name plus usable description content. A database row alone is insufficient. A record with whitespace or invalid Markdown must not create an alternate that leads to an empty or fallback page.

5. Keep batching and sitemap sharding separate

The current implementation builds a single sitemap response. Translation rows are read in cursor batches of 100, filtered for valid content, and reduced to the language sets needed for URL generation. This avoids retaining every long description at once. It does not split the sitemap into multiple files, and the final URL array still consumes memory.

One subtle cursor rule matters: advance using the last raw row fetched, even when every row in that batch fails validation. Advancing only from accepted rows can stall a scan on an invalid batch. Use a stable ordering and test an entirely rejected batch, not just the happy path.

The next scaling step would be sitemap sharding. Google limits each sitemap to 50,000 URLs or 50 MB uncompressed. Alternate annotations also add bytes, so URL count alone is an incomplete capacity check. We have not measured a production limit or implemented the following example; these are hypothetical routes:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemaps/tools-0001.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemaps/blog-0001.xml</loc></sitemap>
</sitemapindex>

For a future implementation, define deterministic membership using stable identifiers or recorded boundaries. Splitting on a mutable offset can move URLs between files as records change. Check shard size after serialization, include every eligible URL once, and validate the index references. Language clusters may span files; each member must still advertise the same eligible alternates.

Use meaningful content timestamps for lastmod. Our generator prefers editorial SEO timestamps and clamps future values. Rebuilding the sitemap should not pretend every document changed. Accurate timestamps describe significant updates; they are not a mechanism to force crawling.

6. Test disagreements, not just tag presence

A page can contain canonical and alternate tags while still contradicting its sitemap. These are failure cases worth keeping as fixtures:

• An unprefixed English page emits an /en/ canonical that redirects.
• A tw path produces hreflang="tw" instead of zh-TW.
• English advertises Chinese, but Chinese does not link back.
• A missing French translation is advertised because French is configured globally.
• A fallback page is excluded from HTML alternates but remains in the sitemap.
• An empty translation batch prevents a cursor from advancing.
• Every deploy changes lastmod despite unchanged content.

The local examples accompanying this draft check URL normalization and cluster membership, including the absence of x-default when English is unavailable. Those tests validate the teaching fixtures. They do not execute the production data layer or establish that every deployed page behaves correctly.

For integration coverage, construct a tiny content matrix before touching the crawler: one complete translation, one blank description, one archived record, and one temporary lookup failure. Write the expected document set first. Compare metadata and sitemap output against that independent expectation, rather than deriving the test result from the same helper being tested. Then remove a translation and verify that both outputs lose the same member. This catches stale availability caches that a static happy-path snapshot can miss.

7. Monitor separate layers of evidence

Use a repeatable post-deployment checklist:

1. Sample default, translated, missing-translation, archived, and paginated routes.
2. Inspect status codes and redirect destinations before interpreting metadata.
3. Compare the canonical against the intended URL, including query policy.
4. Verify alternate language codes, self references, and reciprocal destinations.
5. Compare eligible sitemap entries with actual page metadata and robots directives.
6. Check headers as well as HTML for conflicting indexing instructions.
7. Inspect significant-update timestamps and serialized sitemap size.
8. Review Search Console separately for crawl, indexing, and selected-canonical evidence.

The repository contains a post-deployment sitemap audit script, but we did not run a full production crawl for this article. Historical page samples are snapshots, not continuing guarantees. Source inspection explains intended behavior; response inspection checks deployment; Search Console reports another layer. None of these alone demonstrates ranking, referral traffic, or revenue improvement.

The reusable lesson is to model publication and translation eligibility before generating tags. Once the eligible document set and URL contract have clear owners, metadata disagreements become testable data problems. The broader project context is in the original implementation notes. Which boundary case would you add to this checklist?