DEV Community: Aulvem

Generating a multilingual llms.txt in Astro

Aulvem — Sun, 14 Jun 2026 05:52:38 +0000

llms.txt is a Markdown index for LLMs, placed at the site root. Where sitemap.xml is a machine-readable list of URLs, llms.txt describes — with one-line notes — what the site is and where to start reading.

In Astro you can generate it from Content Collections as an API route, so the post list never has to be hand-maintained. This post is the minimum setup for a bilingual (EN/JA) site: emit /llms.txt, /ja/llms.txt and /llms-full.txt from one renderer.

Up front: how much llms.txt actually helps AI-search traffic isn't a settled or measured thing yet. This is only about the implementation.

The minimal route

Astro's file-based API routes return text when you drop a .txt.ts file under src/pages/. Return a text/plain Response from a GET handler.

// src/pages/llms.txt.ts
import type { APIContext } from "astro";
import { renderLlmsTxt } from "../lib/llmsTxt";

export async function GET(_context: APIContext) {
  const body = await renderLlmsTxt({ docLang: "en" });
  return new Response(body, {
    status: 200,
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "public, max-age=3600",
    },
  });
}

The .txt.ts extension builds to the URL /llms.txt. Keep the assembly logic in src/lib/llmsTxt.ts and leave the route thin, so a per-language endpoint can reuse it.

Build it from Content Collections

Get the post list with getCollection and lay it out on the fly. A hand-kept list goes stale — add a post, forget the index, and llms.txt drifts from the content.

// src/lib/llmsTxt.ts (excerpt)
export async function renderLlmsTxt(opts: LlmsTxtOptions): Promise<string> {
  const blog = await getCollection("blog", ({ data }) => !data.draft);
  blog.sort((a, b) => b.data.pubDate.getTime() - a.data.pubDate.getTime());
  // ...assemble sections and return join("\n")
}

Don't drop the ({ data }) => !data.draft filter. Skip it and a half-written draft lands in llms.txt, advertising a URL you haven't published. Reuse the same exclusion sitemap and RSS use.

Split by language on two axes

This is the part that matters for a multilingual site. Give the renderer two axes:

filterLang: which language's posts to include
docLang: which language the headings and notes are written in

Separating them lets one renderer emit three endpoints.

// src/pages/llms.txt.ts        → English headings, posts from all languages
renderLlmsTxt({ docLang: "en" });

// src/pages/ja/llms.txt.ts     → Japanese headings, Japanese posts only
renderLlmsTxt({ filterLang: "ja", docLang: "ja" });

The English /llms.txt leaves filterLang unset on purpose — it's the whole-site entry point, so it surfaces posts in either language. The Japanese /ja/llms.txt closes to the Japanese surface with filterLang: "ja".

Narrow only the featured list to docLang

One design call. The English version can surface both languages, but the Featured section alone narrows to docLang.

const featuredSource = filteredBlog.filter(
  (p) => entryLangLocal(p.id) === opts.docLang,
);

Listing both halves of a translation pair in the featured slots spends two slots on one piece of content and halves the unique signal in a bounded list. So a limited list (featured) narrows by language; a full dump with loose size limits (llms-full.txt) carries both.

Gotchas

Put the draft exclusion in the shared renderer: so all three endpoints pass the same filter. Let one route through raw and drafts leak
Don't conflate filterLang and docLang: "which posts" and "what language to write in" are different axes. Leaving filterLang unset on the English version is deliberate — don't read it later as a bug and add a filter

Wrapping up

Generate llms.txt from Content Collections as an Astro API route and the manual upkeep goes away. Split on filterLang and docLang and one renderer emits all three files.

The language cross-references in llms-full.txt, how it sits next to robots.txt, and how I word the usage/citation section are on the Aulvem site → Generating llms.txt and llms-full.txt in Astro for a Bilingual Site

Reciprocal hreflang in Astro, only when both languages exist

Aulvem — Sat, 06 Jun 2026 10:12:16 +0000

Astro 5's i18n routing handles URL generation and locale detection, but it doesn't emit <link rel="alternate" hreflang> for you.

The naive fix — emit one alternate per supported locale on every page — quietly breaks when an entry only exists in one language. The alternate points at a 404, Google flags the reciprocal mismatch, and the hreflang is ignored. Worst case, it weakens the site's overall multilingual signal.

This post walks through the minimum override: walk Content Collections, detect which languages have the entry, then emit <link rel="alternate"> conditionally from Seo.astro.

What the override looks like

Three pieces:

Helper functions in src/lib/posts.ts that ask the collection "which languages have this slug?"
Seo.astro takes an availableLangs prop and emits <link rel="alternate"> only for those languages
Layouts call the helper and pass the result to Seo.astro

Step 1: existence helpers

import { getCollection } from "astro:content";
import type { Lang } from "../i18n";

const LANGS: Lang[] = ["en", "ja"];

export async function blogAvailableLangs(slugWithoutLang: string): Promise<Lang[]> {
  const all = await getCollection("blog", ({ data }) => !data.draft);
  return LANGS.filter((lang) =>
    all.some((e) => e.id === `${lang}/${slugWithoutLang}`),
  );
}

Two details to get right:

Drop draft: true in the filter: treating a draft-only language as "available" emits a hreflang pointing at a URL the published build won't serve
Pass the slug without the language prefix: Content Collections IDs are ja/2026-05-30-foo. entrySlug strips the prefix before the lookup

For paginated category pages, the equivalent helper checks whether page N exists per language with Math.ceil(inCat.length / pageSize).

Step 2: conditional emission in `Seo.astro`

---
interface Props {
  availableLangs?: Lang[];
  lang?: Lang;
  // ...
}
const { availableLangs = ["en", "ja"], lang: langProp } = Astro.props;
const lang: Lang = langProp ?? getLangFromUrl(Astro.url);

const currentPath = Astro.url.pathname;
const otherLangPath = altPathForOtherLang(currentPath, lang);
const enPath = lang === "en" ? currentPath : otherLangPath;
const jaPath = lang === "ja" ? currentPath : otherLangPath;
const enHref = new URL(enPath, Astro.site).toString();
const jaHref = new URL(jaPath, Astro.site).toString();
const hasEnVersion = availableLangs.includes("en");
const hasJaVersion = availableLangs.includes("ja");
// x-default points to EN when available; otherwise the only available lang.
const xDefaultHref = hasEnVersion ? enHref : jaHref;
---
{hasEnVersion && <link rel="alternate" hreflang="en" href={enHref} />}
{hasJaVersion && <link rel="alternate" hreflang="ja" href={jaHref} />}
<link rel="alternate" hreflang="x-default" href={xDefaultHref} />

availableLangs = ["en", "ja"] as the default makes top-level pages (/, /about/, /blog/) work without explicit configuration — those are always bilingual. Per-entry layouts must override.

x-default is emitted even for single-language posts, pointing at the only available language. Google's documentation recommends declaring at least one supported locale explicitly.

Step 3: wiring layouts

---
import { entrySlug, blogAvailableLangs } from "../lib/posts";

const slug = entrySlug(entry); // "ja/2026-05-30-foo" → "2026-05-30-foo"
const availableLangs = await blogAvailableLangs(slug);
---
<Seo
  title={data.title}
  description={data.description}
  lang={lang}
  availableLangs={availableLangs}
  // ...
/>

entrySlug strips the language prefix before the helper lookup. Forgetting this turns every comparison into a miss and the helper returns [] — no <link rel="alternate"> ends up in the output, and the build still passes. That's a silent regression worth watching for.

Pitfalls

The default value bites if you forget to override: availableLangs = ["en", "ja"] is the default. A per-entry layout that doesn't pass an override emits alternates for both languages even when only one has an entry — the exact failure mode this whole setup was meant to fix
entrySlug placement: stripping the prefix is mandatory. Without it, every helper call returns [] silently
Paginated routes: /blog/build/3/ can exist in one language and not the other if the post counts diverge. categoryPageAvailableLangs(category, pageNum, pageSize) handles this
x-default for single-language posts: emit it anyway. Google recommends declaring at least one supported locale explicitly

The longer write-up — why routing stays on the official integration, how this interacts with @astrojs/sitemap's hreflang output, and the operational failure modes — is on the Aulvem site → Emitting reciprocal hreflang only when both languages exist — Aulvem's i18n customisation

Catching content rule violations at build time with Astro Content Collections + Zod

Aulvem — Fri, 29 May 2026 15:04:43 +0000

If you run a Markdown-based blog long enough, the frontmatter starts accumulating rules. "A reviews post must carry an ad disclosure." "FAQ questions in JSON-LD must also appear in the body." Eventually a README check-list isn't enough — you forget.

Astro Content Collections plus Zod lets you push most of those rules into build failures. .refine() couples two fields, nested z.object types your structured data, and a violation gets caught at astro build.

This post is the code-first version of the setup I use on aulvem.com. The longer version with operational notes is linked at the end.

Minimal setup: `defineCollection` + `z.object`

// src/content.config.ts
import { defineCollection, z } from "astro:content";
import { glob } from "astro/loaders";

const blog = defineCollection({
  loader: glob({
    pattern: "**/[^_]*.{md,mdx}",
    base: "./src/content/blog",
  }),
  schema: z.object({
    title: z.string(),
    description: z.string(),
    pubDate: z.coerce.date(),
    category: z.enum(["build", "reviews"]),
    tags: z.array(z.string()).default([]),
    draft: z.boolean().default(false),
    affiliate: z.boolean().default(false),
  }),
});

export const collections = { blog };

Four moves cover most of the surface:

z.enum pins the category to a fixed set — typos break the build
z.coerce.date reads 2026-05-23 as a Date
.default(false) makes the field omissible at the YAML side
z.array(z.string()) and other composites work as-is

This is straight out of the Astro 5 docs. The interesting work starts with .refine().

`.refine()` for "two fields must move together"

When two fields are coupled — change one, the other must follow — .refine() at the end of the schema is the right shape. Aulvem's case: category: reviews posts must have affiliate: true so the disclosure banner and rel="sponsored" injection both kick in.

const blog = defineCollection({
  loader: glob({ /* ... */ }),
  schema: z
    .object({
      title: z.string(),
      category: z.enum(["build", "reviews"]),
      affiliate: z.boolean().default(false),
      // ...
    })
    .refine((data) => (data.category === "reviews") === data.affiliate, {
      message: "affiliate must be true iff category is 'reviews'",
      path: ["affiliate"],
    }),
});

(A === B) === affiliate reads as "these two are always equal" — same logic as XOR, easier to scan months later.

Build error from a reviews post that forgot affiliate: true:

[ContentEntryInvalidError] Content config error in `blog → 2026-05-...`:
affiliate must be true iff category is 'reviews'
  at affiliate

message lands in the output verbatim, so it's worth writing it as instructions for future-you.

`.refine` vs `.superRefine`

When you need more than one independent constraint on an object — or per-field error messages — .superRefine is easier:

.superRefine((data, ctx) => {
  if (data.category === "reviews" && !data.affiliate) {
    ctx.addIssue({
      code: z.ZodIssueCode.custom,
      message: "reviews posts must set affiliate: true",
      path: ["affiliate"],
    });
  }
  if (data.draft && data.updatedDate) {
    ctx.addIssue({
      code: z.ZodIssueCode.custom,
      message: "draft posts should not carry updatedDate",
      path: ["updatedDate"],
    });
  }
})

For a single relationship between two fields, .refine() stays lighter.

Typed structured data in frontmatter

HowTo and FAQPage JSON-LD blocks pull their data from frontmatter rather than from parsed body text. The reasons:

Frontmatter is what Zod validates, so the shape is enforced for free
A heading rename doesn't quietly break JSON-LD
The JSON-LD generator can trust frontmatter without re-parsing MDX

Schema:

howto: z
  .object({
    name: z.string().optional(),
    description: z.string().optional(),
    totalTime: z.string().optional(),
    steps: z.array(
      z.object({
        name: z.string(),
        text: z.string(),
        image: z.string().optional(),
      }),
    ),
  })
  .optional(),
faq: z
  .array(
    z.object({
      question: z.string(),
      answer: z.string(),
    }),
  )
  .optional(),

YAML side:

---
title: "Astro Content Collections tips"
faq:
  - question: "When do you reach for .superRefine over .refine?"
    answer: "When one object needs more than one independent constraint..."
  - question: "What breaks when the schema changes?"
    answer: "Every existing post — by design..."
---

A howto with zero steps, or a faq entry missing answer, fails the build.

What Zod can't reach

Zod only inspects frontmatter — the body MDX is outside its scope.

Google's quality guidelines flag JSON-LD without body counterparts as structured-data mismatch and pull the rich-result eligibility. A post with frontmatter FAQs that never appear in the body passes the schema and silently disqualifies itself.

The fix is a separate layer. A small grep-based validator covers it:

import { readFile } from "node:fs/promises";
import { parse as parseYaml } from "yaml";

const raw = await readFile(path, "utf8");
const m = /^---\r?\n([\s\S]*?)\r?\n---\r?\n([\s\S]*)$/.exec(raw);
if (!m) process.exit(0);

const data = parseYaml(m[1]);
const body = m[2].replace(/\s+/g, " ").toLowerCase();

const mismatches = [];
if (Array.isArray(data.faq)) {
  for (const [i, q] of data.faq.entries()) {
    const needle = q.question.replace(/\s+/g, " ").toLowerCase();
    if (!body.includes(needle)) {
      mismatches.push(`faq[${i}].question not in body: "${q.question}"`);
    }
  }
}

if (mismatches.length) {
  for (const e of mismatches) console.error(e);
  process.exit(1);
}

It's substring presence only. The script doesn't catch a wrong answer under the right question — that's a review-time concern.

The three-layer split

Once you split rules across three layers, "where should this rule live?" becomes answerable:

Layer	Fires at	Catches	Misses
Zod schema	`astro build`	types, enums, required/optional, field relations	meaning, body parity
Lint script	pre-commit, CI	banned phrases, substring parity	meaning
Review	pre-publish	meaning, judgment calls	not automatable

Rule of thumb: if a higher layer can catch it, don't push it down.

The full operational notes — the failure modes I keep an eye on, the disclosure-strength judgments, the decision history of why some rules stay in review — live on Aulvem → Pushing operational rules into Astro Content Collections with Zod

Reading sitemap lastmod from MDX frontmatter in Astro

Aulvem — Fri, 29 May 2026 15:04:00 +0000

@astrojs/sitemap won't read updatedDate from MDX frontmatter on its own.

Left at the defaults, every entry in the generated sitemap.xml ends up with lastmod set to the build time. Search engines and AI search treat that as "everything was updated, all the time", which is worse than not setting a freshness signal at all.

This post walks through the minimum implementation: walk MDX inside astro.config.mjs, build a path-to-date map, and feed it into serialize on @astrojs/sitemap. Paginated noindex pages get filtered out in the same pass.

What we're building

Three steps inside astro.config.mjs:

Build a lastmod map: read every blog MDX with fs.readdir, extract updatedDate ?? pubDate, key by URL path
Feed it into the sitemap: use serialize on @astrojs/sitemap to look up each URL and set item.lastmod
Drop noindex paginated pages: use filter to skip /blog/<cat>/<N>/ so the sitemap doesn't contradict the meta robots tag

Step 1: build the lastmod map

Put an async function at the top of astro.config.mjs and await it into a constant:

import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";

async function buildBlogLastmodMap() {
  const map = new Map();
  for (const lang of ["en", "ja"]) {
    const dir = join(process.cwd(), "src", "content", "blog", lang);
    let files = [];
    try {
      files = await readdir(dir);
    } catch {
      continue;
    }
    for (const file of files) {
      if (!file.endsWith(".mdx") && !file.endsWith(".md")) continue;
      const slug = file.replace(/\.(mdx|md)$/, "");
      const raw = await readFile(join(dir, file), "utf8");
      const fm = /^---\n([\s\S]*?)\n---/.exec(raw);
      if (!fm) continue;
      const front = fm[1];
      if (/^draft:\s*true/m.test(front)) continue; // drop drafts
      const updated = /^updatedDate:\s*(\S+)/m.exec(front);
      const pub = /^pubDate:\s*(\S+)/m.exec(front);
      const dateStr = (updated && updated[1]) || (pub && pub[1]);
      if (!dateStr) continue;
      const d = new Date(dateStr);
      if (Number.isNaN(d.getTime())) continue;
      const path = lang === "ja" ? `/ja/blog/${slug}/` : `/blog/${slug}/`;
      map.set(path, d.toISOString());
    }
  }
  return map;
}

const blogLastmod = await buildBlogLastmodMap();

Why not getCollection("blog")? Because astro.config.mjs is evaluated before the content loader is initialised — the Content Collections API isn't available yet.

The only fields the map needs are updatedDate and pubDate, so a light regex covers it. No YAML parser dependency for two fields.

Step 2: feed it into `serialize`

@astrojs/sitemap exposes a serialize hook that lets you rewrite each emitted URL entry:

import sitemap from "@astrojs/sitemap";

export default defineConfig({
  // ...
  integrations: [
    sitemap({
      i18n: {
        defaultLocale: "en",
        locales: { en: "en", ja: "ja" },
      },
      serialize(item) {
        const url = new URL(item.url);
        // Strip /ja/ for the branch decision so both EN and JA hit
        // the same changefreq / priority. lastmod lookup uses the
        // original pathname because the map keys keep /ja/.
        const pathname = url.pathname.replace(/^\/ja\//, "/").replace(/^\/ja$/, "/");
        if (pathname === "/") {
          item.changefreq = "daily";
          item.priority = 1.0;
        } else if (pathname.startsWith("/blog/")) {
          item.changefreq = "monthly";
          item.priority = 0.7;
          const lastmod = blogLastmod.get(url.pathname);
          if (lastmod) item.lastmod = lastmod;
        } else {
          item.changefreq = "monthly";
          item.priority = 0.5;
        }
        return item;
      },
    }),
  ],
});

changefreq and priority are set in the same hook so each path category stays consistent. priority is officially "ignored" by Google these days, but Bing and the AI crawlers still read it, so keeping it consistent is the cheap default.

Step 3: filter paginated `noindex` pages

If your blog category pages return <meta name="robots" content="noindex, follow"> from page 2 onwards (only page 1 is index-eligible), shipping page 2+ URLs in the sitemap is a contradiction.

"Listed in sitemap" reads as "please index this". "Meta robots: noindex" reads as "don't index this". Both at once is treated as a quality smell by Google and Bing.

sitemap({
  filter: (page) => {
    if (page.endsWith("/404/") || page.endsWith("/404")) return false;
    // Paginated category pages (/blog/build/2/, /blog/reviews/3/ ...)
    // are noindex — drop them so the sitemap doesn't conflict with
    // the meta robots tag.
    if (/\/blog\/(build|reviews)\/\d+\/?$/.test(new URL(page).pathname)) return false;
    return true;
  },
  // ...
});

The noindex decision and the sitemap filter are two halves of one change.

Pitfalls

A short list of things that almost broke:

Top-level await: works because astro.config.mjs is evaluated as ESM. .cjs configs won't accept it
draft: true filtering: skipping drafts during map construction is necessary, otherwise draft URLs leak into the sitemap
Regex tightness: /^updatedDate:\s*(\S+)/m reads updatedDate: 2026-05-25. Quoted strings still parse because \S+ captures "2026-05-25" whole and new Date() handles the quotes, but multi-line YAML values won't survive
Language-folder merge: en and ja are walked separately and joined into one map. Keys stay distinct (/blog/<slug>/ vs /ja/blog/<slug>/) so lookups during serialize resolve
updatedDate policy: the implementation only works if updatedDate is updated honestly. Bumping it for trivial edits poisons the signal — pair this with a "only update on substantive revision" rule

The longer write-up on the Aulvem site covers the updatedDate policy, alternatives I rejected, and the threshold for moving this code back into the official integration → Reading sitemap lastmod from MDX frontmatter — customising Astro's sitemap integration

How I run a small blog on Astro 5 + Content Collections

Aulvem — Mon, 25 May 2026 14:16:46 +0000

I run a small blog (aulvem.com) on Astro 5 + MDX + Content Collections, hosted statically on Cloudflare Pages. The interesting part isn't the stack — it's the operational rules I lean on the schema to enforce so that a writer (me) can't ship something half-broken.

This post is a short tour of that setup: which packages I keep, which dependencies I deliberately don't add, and the three build-time checks that hold the writing flow together.

The stack, in eight runtime packages

// package.json (runtime deps, abridged)
{
  "astro": "^5",
  "@astrojs/mdx": "^4",
  "@astrojs/sitemap": "^3",
  "@astrojs/rss": "^4",
  "@astrojs/tailwind": "^6",
  "rehype-external-links": "^3",
  "rehype-mermaid": "^3",
  "tailwindcss": "^3.4"
}

Dev deps: pagefind (full-text search), sharp (local image processing), playwright (build-time SVG render for mermaid), typescript, @types/node. No React. No Vue. No Vite plugins.

The rule I started with: don't add a dependency on the hope it'll be useful later. Anything that doesn't have a written-down use case stays out.

Three build-time rules

1. `category: reviews` ⇔ `affiliate: true`

A post in the reviews category is, by ad-network rules, advertising. It has to carry a disclosure banner and rel="sponsored" on outbound links. Both of those are injected by rehype plugins gated on affiliate: true. So the worst-case failure mode is publishing a review post with affiliate left at its default — disclosure missing, sponsored rel missing.

The Zod schema couples them with one .refine():

.refine((data) => (data.category === "reviews") === data.affiliate, {
  message: "affiliate must be true iff category is 'reviews'",
  path: ["affiliate"],
})

Both sides compared with ===. Change only one and the build fails.

2. Typed structured data in frontmatter

HowTo and FAQPage JSON-LD blocks pull from frontmatter rather than from parsed body text. Reasons:

Heading renames don't quietly break JSON-LD
Zod validates the shape, so missing answer fields are caught at build
The JSON-LD generator can trust frontmatter without touching MDX

faq:
  - question: "Why Astro 5?"
    answer: "Markdown-centric, static-only output, schema-typed frontmatter, small core deps. Astro 5 fits all four."

3. `lastmod` from frontmatter

Astro's official sitemap integration doesn't read updatedDate from MDX frontmatter. Default lastmod is the build time, which broadcasts "every post updated on every build" to search engines and AI search.

I walk the collection at build time and pipe updatedDate ?? pubDate into the sitemap entry as lastmod. Paginated noindex pages get dropped from the sitemap in the same pass — submitting them through the sitemap is a contradictory signal otherwise.

Single source of truth for operational flow

Adding a post, retiring a post, updating product pages — all of these are anchored in one doc (docs/content-flow.md), and the scaffolding scripts pull from it. Same rule every time. That absorbs most of the "the approach drifts run to run" variance, which is the failure mode that gets me when I haven't touched the project in a few weeks.

The full version with the decision history, what I dropped, and where Zod can't reach lives on Aulvem → How this blog is built — Aulvem on Astro 5 and Content Collections

DEV Community: Aulvem

Generating a multilingual llms.txt in Astro

The minimal route

Build it from Content Collections

Split by language on two axes

Narrow only the featured list to docLang

Gotchas

Wrapping up

Reciprocal hreflang in Astro, only when both languages exist

What the override looks like

Step 1: existence helpers

Step 2: conditional emission in Seo.astro

Step 3: wiring layouts

Pitfalls

Catching content rule violations at build time with Astro Content Collections + Zod

Minimal setup: defineCollection + z.object

.refine() for "two fields must move together"

.refine vs .superRefine

Typed structured data in frontmatter

What Zod can't reach

The three-layer split

Reading sitemap lastmod from MDX frontmatter in Astro

What we're building

Step 1: build the lastmod map

Step 2: feed it into serialize

Step 3: filter paginated noindex pages

Pitfalls

How I run a small blog on Astro 5 + Content Collections

The stack, in eight runtime packages

Three build-time rules

1. category: reviews ⇔ affiliate: true

2. Typed structured data in frontmatter

3. lastmod from frontmatter

Single source of truth for operational flow

Step 2: conditional emission in `Seo.astro`

Minimal setup: `defineCollection` + `z.object`

`.refine()` for "two fields must move together"

`.refine` vs `.superRefine`

Step 2: feed it into `serialize`

Step 3: filter paginated `noindex` pages

1. `category: reviews` ⇔ `affiliate: true`

3. `lastmod` from frontmatter