DEV Community: FADI MAMAR

I Built a Dynamic llms.txt for Next.js. Then Google Said Don't Bother.

FADI MAMAR — Thu, 21 May 2026 02:32:17 +0000

Two weeks ago I shipped a dynamic llms.txt and llms-full.txt for a Next.js 16 site I run. Hourly revalidation, pulls from the same content sources as the sitemap, auto-categorizes by URL pattern, returns proper text/plain. Took about 40 minutes to build.

Last week Google published their AI Search optimization guide.

The relevant line:

Don't create llms.txt files and other "special" markup.

So now I have a dynamic llms.txt that the search engine I most care about explicitly says not to bother with.

I'm keeping it anyway. Here's why, and here's the code in case you want to do the same.

What llms.txt is supposed to do

The llms.txt spec was proposed in 2024 (by Jeremy Howard at Answer.AI) as a way for sites to give AI assistants a condensed, machine-readable summary of the site's content. Two files:

/llms.txt — short, table-of-contents style. Top-level pages, brief descriptions.
/llms-full.txt — long-form. The actual content for AI ingestion.

The pitch: LLMs that crawl your site to answer user questions get a curated index instead of having to crawl every page.

In practice, it never became a real standard. Anthropic, OpenAI, and Perplexity have never officially committed to reading these files. Google has now publicly said it doesn't either.

But several smaller AI engines and developer-focused indexers do consume llms.txt when present. And the cost to maintain a properly designed one is approximately zero. So the cost/benefit math is interesting.

My implementation in Next.js 16 App Router

Three files. Total: about 90 lines.

Step 1: a renderer that pulls content from your CMS

I keep mine in lib/llms.ts. It reads from Sanity, but the same pattern works with anything: Markdown files, MDX, a database, a JSON config.

// lib/llms.ts
import { getAllBlogPosts, getAllResources } from '@/lib/sanity'

const SITE_URL = 'https://example.com'

function resourceCategory(slug: string): 'permit' | 'comparison' | 'glossary' | 'general' {
  if (slug.startsWith('building-permit-guide-')) return 'permit'
  if (slug.includes('-vs-')) return 'comparison'
  if (slug.startsWith('what-is-')) return 'glossary'
  return 'general'
}

export async function renderLlmsTxt(): Promise<string> {
  const posts = await getAllBlogPosts()
  const resources = await getAllResources()

  const lines: string[] = []
  lines.push('# Example Company')
  lines.push('')
  lines.push('> Toronto-based contractor specializing in framing, drywall, and insulation.')
  lines.push('')
  lines.push('## Core Pages')
  lines.push('- [Home](https://example.com)')
  lines.push('- [Services](https://example.com/services)')
  lines.push('- [About](https://example.com/about)')
  lines.push('- [Contact](https://example.com/contact-us)')
  lines.push('')

  const permits = resources.filter(r => resourceCategory(r.slug.current) === 'permit')
  const comparisons = resources.filter(r => resourceCategory(r.slug.current) === 'comparison')
  const glossary = resources.filter(r => resourceCategory(r.slug.current) === 'glossary')

  if (permits.length) {
    lines.push('## Building Permit Guides')
    for (const r of permits) {
      lines.push(`- [${r.title}](${SITE_URL}/resources/${r.slug.current})`)
    }
    lines.push('')
  }

  if (comparisons.length) {
    lines.push('## Comparison Pages')
    for (const r of comparisons) {
      lines.push(`- [${r.title}](${SITE_URL}/resources/${r.slug.current})`)
    }
    lines.push('')
  }

  if (glossary.length) {
    lines.push('## Glossary')
    for (const r of glossary) {
      lines.push(`- [${r.title}](${SITE_URL}/resources/${r.slug.current})`)
    }
    lines.push('')
  }

  if (posts.length) {
    lines.push('## Blog')
    for (const p of posts) {
      lines.push(`- [${p.title}](${SITE_URL}/blog/${p.slug.current})`)
    }
  }

  return lines.join('\n')
}

export async function renderLlmsFullTxt(): Promise<string> {
  // Same shape as renderLlmsTxt, but inline the actual body content per
  // resource/post. Pull body from your CMS, convert to plaintext, append
  // under each section header.
  // omitted for brevity
}

The category split is my preference. The spec is loose. What matters is that a machine reading the file gets a clean hierarchical view of your site.

Step 2: the route handlers

Next.js 16 App Router treats files inside /app as routes. Drop a route.ts inside any folder named for your URL path:

// app/llms.txt/route.ts
import { renderLlmsTxt } from '@/lib/llms'

export const revalidate = 3600 // 1 hour cache

export async function GET() {
  const body = await renderLlmsTxt()
  return new Response(body, {
    headers: { 'content-type': 'text/plain; charset=utf-8' },
  })
}

// app/llms-full.txt/route.ts
import { renderLlmsFullTxt } from '@/lib/llms'

export const revalidate = 3600

export async function GET() {
  const body = await renderLlmsFullTxt()
  return new Response(body, {
    headers: { 'content-type': 'text/plain; charset=utf-8' },
  })
}

The revalidate = 3600 line is the key. It tells Vercel/Next.js to serve a cached version for an hour, then regenerate on the next request. If you publish 10 blog posts a day, the file picks them up within 60 minutes. If you publish once a week, it picks them up within an hour of publishing. No build step required.

Step 3: deploy and verify

curl https://example.com/llms.txt

Should return your generated markdown-flavored text in seconds.

That's it. Total code: about 90 lines. Total time: 40 minutes if you've used Next.js before.

Then Google said don't bother

Google's AI optimization guide (published this month) is explicit:

Don't create llms.txt files and other "special" markup.

The reasoning, per Google: their AI features (AI Overviews, AI Mode) ingest the same content humans see. They crawl pages, parse the rendered HTML, and synthesize. They don't read llms.txt. Creating one is "unnecessary effort."

That's an honest position. Google isn't penalizing you for having an llms.txt. They're telling you it's wasted work from their perspective.

But Google isn't the only AI engine.

Why I'm keeping mine anyway

Four reasons.

1. The maintenance cost is zero

Because the file is dynamic and pulls from the same content source as the rest of the site, it updates itself. There's no separate workflow to maintain, no monthly chore, no risk of going stale. The code sat in the repo for two weeks with zero attention required. The next time I add a blog post or resource page, the file picks it up within an hour.

If maintenance cost were measured in minutes per month, llms.txt would be at zero. There's nothing to abandon.

2. Not every AI engine has spoken

Perplexity, ChatGPT, Claude, and several smaller AI search products have not officially said whether they read llms.txt. Some open-source RAG indexers do. Some agent frameworks default to checking for it.

If even one of these tools reads my llms.txt and produces a citation that a human user clicks, the file paid for itself. The opportunity cost of having it is approximately the bytes Vercel serves on the rare requests that hit the endpoint.

3. It functions as a structured fallback

The llms.txt is one of the cleanest, most human-readable summaries of my site that exists anywhere. When someone asks me for an overview of my site's content, I can paste the llms.txt into a chat and have an instant brief. It's a free side effect of building it.

4. Building it was a useful exercise

Going through the exercise of categorizing every page, deciding what's important, and giving each section a clean summary is exactly the kind of content audit most sites should do anyway. The output (the llms.txt file) is the artifact. The thinking is the value.

When to skip it

If your site is purely commerce, your content is highly dynamic (millions of product pages), or your content lives mostly behind login, llms.txt has less value. AI engines aren't going to ingest your product catalog from a flat text file.

If you're a content site, a docs site, a portfolio, an editorial publication, or anything where the page-level content matters and the URL structure has meaning, llms.txt is worth the 40 minutes.

Working example

You can see the live output at:

Both regenerate every hour. Both pull from the same Sanity CMS that drives the rest of the site. Total runtime memory cost: indistinguishable from zero. Pageviews: low but non-zero.

The bigger point

Google publishing guidance that says "don't bother with X" is not the same as Google penalizing X. Sometimes Google is telling you the optimal allocation of your effort. Sometimes you should agree. Sometimes you should keep doing X because the cost is zero and the upside, while small, is nonzero.

llms.txt is in the second bucket. Build it once, let it run, ignore the file for six months, see what happens.

If in 18 months Google publishes "we now use llms.txt to inform AI Overviews," you'll be glad you left it running. If they don't, the file cost you 40 minutes and approximately zero ongoing attention.

The optimal play with low-cost, low-confidence bets is to make a lot of them and let outcomes resolve themselves.

Build the llms.txt.

The Discord.js gotchas that cost me a week each (so they don't have to cost you one)

FADI MAMAR — Thu, 21 May 2026 02:15:25 +0000

I run an AI Discord bot builder that's deployed 500+ live bots over the last year. The service takes a plain-English description, generates discord.js code, and ships the container to production in under 30 seconds.

That sounds clean. It is not clean.

Most of what I learned came from logs at 3am after someone's bot stopped responding to messages, or a community went silent because of an undocumented Discord behavior. Here's the list I wish I'd had on day one.

Missing Permissions (50013) is usually about role position, not permissions

The first time I hit this I spent two hours auditing the bot's permission integer. It was correct. The bot still couldn't kick anyone.

The thing nobody puts in big letters: discord.js permissions are gated by role hierarchy. A bot can have the KickMembers permission and still fail to kick anyone whose top role sits above the bot's top role.

client.on('interactionCreate', async (interaction) => {
if (!interaction.isChatInputCommand()) return;
if (interaction.commandName !== 'kick') return;

const target = interaction.options.getMember('user');
const botMember = interaction.guild.members.me;

if (target.roles.highest.position >= botMember.roles.highest.position) {
return interaction.reply({
content: "I can't kick that user — their role is at or above mine.",
ephemeral: true,
});
}

await target.kick(interaction.options.getString('reason') ?? 'No reason');
await interaction.reply(`Kicked ${target.user.tag}.`);

});

Fix in Discord: drag the bot's role above the roles it needs to manage in Server Settings → Roles. This is the single most common reason a "working" bot doesn't work.

Unknown Interaction (10062) is a 3-second race

Slash commands give you exactly 3 seconds to respond. If you call any awaited DB query, third-party API, or LLM before.reply(), you'll lose that race.

Wrong:
const result = await callLLM(prompt); // 2.4s
await interaction.reply(result); // 10062

Right:
await interaction.deferReply(); // buys you 15 minutes
const result = await callLLM(prompt);
await interaction.editReply(result);

The deferred reply pattern is one of those things you need to internalize once and then never get bit by again.

Intent flags are silent killers

This one is brutal because there's no error. Your bot connects, shows as online, registers commands, and never reads a message.

Two parts:

In the Discord Developer Portal, toggle "Message Content Intent" on.
In your client config, list it:

import { Client, GatewayIntentBits } from 'discord.js';

const client = new Client({
intents: [
GatewayIntentBits.Guilds,
GatewayIntentBits.GuildMessages,
GatewayIntentBits.MessageContent, // ← without this, message.content is empty
GatewayIntentBits.GuildMembers,
],
});

If you skip either step, message.content arrives as an empty string and your prefix commands look broken for no apparent
reason.

Reactions and edits on old messages are partial

If someone reacts to a message your bot didn't observe being created (anything older than the bot's session), the event payload is partial. Touching .content returns null until you fetch.

client.on('messageReactionAdd', async (reaction, user) => {
if (reaction.partial) {
try { await reaction.fetch(); } catch { return; }
}
if (reaction.message.partial) {
try { await reaction.message.fetch(); } catch { return; }
}
// safe to use reaction.message.content now
});

Add the Partials config too:

import { Client, Partials } from 'discord.js';
new Client({
intents: [...],
partials: [Partials.Message, Partials.Channel, Partials.Reaction],
});

Ephemeral replies don't behave like normal messages

You can't fetch them with channel.messages.fetch(). You can't add reactions. If you want to update one later, you need the original interaction token, not a message ID:

await interaction.reply({ content: 'Loading…', ephemeral: true });
// 10 seconds later
await interaction.editReply({ content: 'Done.' });

If you serialize the interaction across processes (queue, webhook handler, etc.), serialize the token, not the message
reference.

Process crashes leave gateway sessions hanging

If your bot crashes hard without closing the websocket, Discord keeps that session alive on their end for a few minutes.
New deploys then count against the 1000/day session start limit, and on cold restarts you'll see a brief window where two instances of your bot post duplicate messages.

async function shutdown(signal) {
console.log([bot] ${signal} received, closing gateway);
try { await client.destroy(); } catch {}
process.exit(0);
}
process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

process.on('unhandledRejection', (err) => {
console.error('[bot] unhandled rejection:', err);
// don't crash — log and continue
});
process.on('uncaughtException', (err) => {
console.error('[bot] uncaught:', err);
// do crash, but cleanly
shutdown('uncaughtException');
});

If you're running on a container platform that sends SIGTERM before forcibly killing, this single block saves you a
class of bugs you'd never spot.

Rate limits aren't always 429s

The REST client emits a rateLimited event for the soft cases (Discord's per-route bucket warnings). If you log it from
day one, you get a free monitoring signal:

client.rest.on('rateLimited', (info) => {
console.warn('[ratelimit]', {
route: info.route,
method: info.method,
timeToReset: info.timeToReset,
limit: info.limit,
});
});

In production these almost always trace back to one bot in a 10k-member server doing something dumb in a loop. They Never show up in your error monitoring otherwise.

The biggest one: most production failures aren't bugs in your code

When you run a few hundred bots, the failure distribution shifts. Probably 30% of "the bot is broken" tickets I've seen trace back to:

The server admin moved roles around
The user revoked the bot's invite
The bot was rate-limited because someone scripted 1000 reactions
Discord shipped a behavior change
A channel was deleted that the bot was scheduled to post in

Your bot needs to log enough that you can prove which of these happened quickly. A single structured log per important event (action attempted, guild ID, user ID, outcome, latency) gets you 80% of the way there.

I learned most of this the hard way running vibebot.gg, where users describe a Discord bot in English and the system generates and deploys discord.js code for them. The interesting failure mode in that setup is that the LLM-generated code has to handle all eight of these gotchas correctly the first time, because the user never sees the code. The patterns above are exactly what we now bake into every generated bot at the template level.

If you're building a Discord bot from scratch, copy the snippets above directly. They're worth a week of debugging each.