DEV Community: Mudassir Khan

llms.txt vs robots.txt vs ai.txt: The Developer's Cheat Sheet

Mudassir Khan — Sat, 23 May 2026 12:40:02 +0000

llms.txt vs robots.txt vs ai.txt: The Developer's Cheat Sheet

Every Next.js developer building a public site in the last 18 months has hit the same wall: you Google "how to control what AI crawlers read" and get three different answers pointing at three different files — robots.txt, llms.txt, and ai.txt. They are not the same thing. They do not talk to the same audience. And using the wrong one (or none at all) means AI search engines are either ignoring your content entirely or indexing pages you never intended them to.

This is the one-stop breakdown I wish existed when I was figuring this out.

The three files at a glance

	`robots.txt`	`llms.txt`	`ai.txt`
Proposed by	Martijn Koster (1994)	Anthropic + community	AI-txt.com initiative
Primary audience	Web crawlers (Googlebot, Bingbot, etc.)	LLM training & AI search crawlers	AI assistants (ChatGPT, Claude, Gemini)
Format	Key-value directives	Markdown / structured text	Key-value + JSON blocks
Spec status	RFC standard, universally supported	Emerging, growing adoption	Early proposal, limited adoption
Enforced by	All major search engines	Anthropic, Perplexity, some others	No major enforcer yet
Location	`yourdomain.com/robots.txt`	`yourdomain.com/llms.txt`	`yourdomain.com/ai.txt`

The short mental model: robots.txt is for Googlebot. llms.txt is for ClaudeBot and GPTBot when they are building knowledge, not just indexing. ai.txt is a newer proposal that tries to cover AI assistants reading your site in real time. Use all three if you want complete coverage — but robots.txt + llms.txt is where you get 90% of the value today.

robots.txt — the original gatekeeper

robots.txt has been around since 1994. Every crawler on the internet — Googlebot, Bingbot, DuckDuckBot, GPTBot, ClaudeBot, PerplexityBot — checks it before crawling. If you block GPTBot in robots.txt, it will not crawl your site for training data or AI-search indexing.

Basic syntax:

User-agent: *
Disallow: /admin/
Disallow: /private/

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /blog/
Disallow: /

User-agent: * applies to all crawlers. Named user-agents override * for that specific bot. Allow and Disallow are path-based — no wildcards by default in the original spec, though most modern crawlers support them.

In Next.js App Router — generate it dynamically from app/robots.ts:

// app/robots.ts
import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin/', '/api/private/'],
      },
      {
        userAgent: 'GPTBot',
        allow: ['/blog/', '/services/'],
        disallow: ['/'],
      },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Next.js renders this as text/plain at /robots.txt automatically. No separate file needed.

What robots.txt does NOT do: it does not stop a crawler from reading pages it already knows about from other sources (backlinks, sitemaps). It only stops it from actively crawling those paths. If GPTBot found your /admin/ page linked from a public page, it may already have cached it.

llms.txt — built for the AI era

llms.txt was proposed by Answer.AI and picked up by Anthropic, Perplexity, and others as a structured way to tell LLMs what your site is actually about — not just what they can crawl, but what context they should carry when reasoning about your content.

Unlike robots.txt which is access control, llms.txt is documentation. Think of it as a README for your site aimed at language models.

Basic structure:

# YourSite

> One-line description of what this site is and who it's for.

A few sentences of context. What does this site do? Who is the author?
What should an LLM understand before citing any page from this domain?

## Blog

- [Post title one](https://yourdomain.com/blog/post-one): One-line summary.
- [Post title two](https://yourdomain.com/blog/post-two): One-line summary.

## Services

- [Service name](https://yourdomain.com/services/name): What this service does in one line.

## Contact

- Author: Your Name
- Email: you@yourdomain.com
- LinkedIn: linkedin.com/in/yourhandle

The format is intentionally plain Markdown. No special parser needed — any LLM can read it. The /llms.txt path is the convention; some sites also serve /llms-full.txt with deeper content for models that want more context.

In Next.js — generate dynamically from app/llms.txt/route.ts:

// app/llms.txt/route.ts
import { blogPosts } from '@/data/blog-posts'

export async function GET() {
  const lines = [
    '# YourSite',
    '',
    '> AI-search-ready Next.js development and SEO consulting.',
    '',
    'This site covers AI engineering, GEO/AEO, and production Next.js patterns.',
    '',
    '## Blog',
    '',
    ...blogPosts.map(p => `- [${p.title}](https://yourdomain.com/blog/${p.slug}): ${p.excerpt}`),
    '',
    '## Services',
    '',
    '- [AI-Search Consulting](https://yourdomain.com/services): End-to-end GEO and AEO for Next.js sites.',
  ]

  return new Response(lines.join('\n'), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

This keeps llms.txt in sync with your actual content automatically — no manual updates.

Who reads llms.txt today: ClaudeBot (Anthropic's crawler), PerplexityBot, some versions of GPTBot. Adoption is growing fast. If you are publishing content you want AI search engines to cite accurately, this file is non-negotiable.

ai.txt — the wildcard

ai.txt is a newer proposal from a different working group. Where llms.txt focuses on what an LLM should know about your site, ai.txt focuses on granting or denying permission for AI assistants to use your content in responses.

Basic syntax:

# ai.txt
Version: 1.0

[permissions]
allow: true
commercial-use: false
training: false
real-time-access: true

[attribution]
require: true
format: "Source: {title} ({url})"

[contact]
email: you@yourdomain.com

Honest assessment: ai.txt has minimal enforcer support right now. No major AI company officially reads it. The spec is still evolving. That said, if the initiative gains traction (similar to how robots.txt went from informal convention to de-facto standard), having it early costs nothing and signals intent.

For most developers today: add it, keep it simple, and do not spend more than 10 minutes on it.

How a crawler actually decides what to read

The flow for a modern AI crawler like GPTBot or ClaudeBot hitting your domain:

Fetch robots.txt — am I allowed to crawl this path?
If allowed, fetch the page HTML
Fetch llms.txt (periodically, not per-request) — what is this site actually about?
Check ai.txt if the implementation supports it
Index the content with the context from steps 2–4 combined

The key insight: robots.txt is checked per crawl request. llms.txt is fetched periodically and cached — it shapes how the model understands your whole site over time, not just whether it can read one page.

Putting it all together in Next.js App Router

Here is the complete implementation for a Next.js 15 App Router site:

app/
├── robots.ts          ← generates /robots.txt
├── sitemap.ts         ← generates /sitemap.xml
├── llms.txt/
│   └── route.ts       ← generates /llms.txt (dynamic, always in sync)
public/
└── ai.txt             ← static file, update manually

robots.ts (full version with AI crawler rules):

// app/robots.ts
import { MetadataRoute } from 'next'

const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://yourdomain.com'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // Default: allow everything
      { userAgent: '*', allow: '/' },

      // GPTBot: allow blog and services, block everything else
      {
        userAgent: 'GPTBot',
        allow: ['/blog/', '/services/', '/tools/'],
        disallow: ['/admin/', '/api/'],
      },

      // ClaudeBot: same rules
      {
        userAgent: 'ClaudeBot',
        allow: ['/blog/', '/services/', '/tools/'],
        disallow: ['/admin/', '/api/'],
      },

      // PerplexityBot: full access (it drives meaningful referral traffic)
      { userAgent: 'PerplexityBot', allow: '/' },

      // Google-Extended (used for Gemini training): restrict to blog only
      {
        userAgent: 'Google-Extended',
        allow: ['/blog/'],
        disallow: ['/'],
      },
    ],
    sitemap: `${BASE_URL}/sitemap.xml`,
  }
}

llms.txt/route.ts (dynamic, pulls from your data layer):

// app/llms.txt/route.ts
import { blogPosts } from '@/data/blog-posts'
import { servicePosts } from '@/data/services'

const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://yourdomain.com'

export async function GET() {
  const content = `# YourSite

> [One-sentence description of your site and its purpose.]

[Two or three sentences giving an LLM the context it needs to cite your site accurately.
What topics do you cover? Who is the author? What makes this site's perspective unique?]

## Blog

${blogPosts.map(p => `- [${p.title}](${BASE_URL}/blog/${p.slug}): ${p.excerpt}`).join('\n')}

## Services

${servicePosts.map(s => `- [${s.title}](${BASE_URL}/services/${s.slug}): ${s.summary}`).join('\n')}

## Author

- Name: Your Name
- Site: ${BASE_URL}
- Expertise: [Your primary expertise areas]
`

  return new Response(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=3600, stale-while-revalidate=86400',
    },
  })
}

public/ai.txt (static, update when the spec stabilizes):

# ai.txt
Version: 1.0

[permissions]
allow: true
commercial-use: false
training: false
real-time-access: true

[attribution]
require: true

[contact]
site: https://yourdomain.com

Which file do you actually need?

Start here:

Building a public Next.js site? → Add robots.txt first. Always.
Want AI search engines (ChatGPT, Perplexity, Claude) to cite your content accurately? → Add llms.txt. This is the highest-leverage file for AI-search visibility right now.
Want to future-proof against the ai.txt spec gaining enforcement? → Add a simple public/ai.txt. It costs 10 lines.
Want to block AI crawlers from training on your content? → Set Disallow: / for GPTBot, ClaudeBot, and Google-Extended in robots.txt. This is the only file that actually enforces it today.

The one mistake I see most often: developers add robots.txt but skip llms.txt, then wonder why ChatGPT gives wrong answers about their site even though Googlebot indexes it fine. Googlebot and GPTBot-for-knowledge are completely different crawlers with different purposes.

Three things to verify right now

Open your terminal and check these three URLs on your live site:

curl https://yourdomain.com/robots.txt
curl https://yourdomain.com/llms.txt
curl https://yourdomain.com/ai.txt

If any returns a 404 or HTML error page, that file is missing. For robots.txt, a missing file means all crawlers assume full access — usually fine for public sites, but you lose granular control. For llms.txt, missing means LLMs are forming their understanding of your site from raw page HTML with no structured context — which almost always leads to inaccurate citations.

If you want a deeper look at how AI crawlers read Next.js sites specifically — what RSC payloads they fetch, how streaming affects what they see, and which metadata fields they actually use — I have a longer writeup on the AI-search architecture patterns I use in production that goes further than this cheat sheet.

And if you want this wired up on your own site end-to-end, that is exactly the kind of work I take on.

If your own llms.txt or robots.txt setup looks different from what I showed here — especially if you are on an older Next.js version or using the Pages Router — drop it in the comments. Curious what variations people are running in production.

CLaRa: Fixing RAG’s Broken Retrieval–Generation Pipeline With Shared-Space Learning

Mudassir Khan — Tue, 09 Dec 2025 11:00:13 +0000

Retrieval-Augmented Generation (RAG) has become the default solution for grounding LLM outputs in external knowledge. But the classical RAG setup still carries a major architectural flaw: the retriever and generator learn in isolation. This separation quietly sabotages accuracy, increases hallucinations, and prevents genuine end-to-end optimization.

CLaRa (Closed-Loop Retrieval and Augmentation) introduces a fundamentally different approach — one that actually allows the retriever to learn from what the generator gets wrong.

Let’s break down why that matters.

The Core Problem: RAG Is Optimizing Two Brains That Never Talk

Traditional RAG pipelines train two components separately:

Retriever → picks documents using similarity search (dense or sparse).

Generator (LLM) → takes raw text and tries to answer.

The failure point?
There is no gradient flow between these two components.

The retriever has no idea whether the documents it selected actually helped the generator produce the correct answer. It only optimizes for similarity—not usefulness.

This leads to:

"Close but wrong" retrieved documents

Irrelevant context passed to the LLM

Weak factual grounding because retrieval can't learn from generation errors

RAG keeps trying harder at the wrong task.

CLaRa’s Fix: A Shared Continuous Representation Space

CLaRa solves the broken gradient issue by mapping both queries and documents into a shared representation space.

This changes everything.

How the shared space helps:

Document embeddings and query embeddings coexist in the same vector space

The generator’s final answer loss backpropagates through the retriever

Retriever learns what actually helps answer a query

Retrieval stops being a similarity contest and becomes a relevance optimization loop

This feedback loop is the missing piece in traditional RAG.

The result:
Your retriever becomes intelligent — not just associative.

3.Document Compression: Retrieval Without Text Bloat

One of CLaRa’s most practical innovations is how it handles documents:

It never retrieves raw text. It retrieves compressed memory tokens.

These are compact, dense vector representations that summarize meaning, not wording.

How it works:

Document → compressed memory tokens (embeddings)

Retriever fetches tokens instead of full text

Generator consumes tokens directly

Why this matters:

Context length shrinks dramatically

You can process more documents without hitting LLM token limits

Computation cost drops

Throughput increases

This isn’t just more accurate — it’s more efficient.

SCP: Training the Compressor to Capture Meaning, Not Noise

CLaRa doesn’t trust standard compression to produce semantically meaningful vectors (and rightly so).
So it introduces Salient Compressor Pre-training (SCP).

Goal of SCP:

Make compressed representations focus on meaning, not superficial text features.

How SCP trains the compressor:

The system uses synthetic data generated by an LLM:

Simple QA pairs

Complex QA tasks

Paraphrased document sets

The compressor is trained to:

Generate embeddings that can answer these questions

Reconstruct paraphrased meaning (not the exact text)

This forces the vectors to internalize the semantic core of the document.

By the time end-to-end training starts, the compressor already knows how to distill content into high-information embeddings.

Why CLaRa Matters ?

CLaRa isn't just a tweak — it’s a structural correction to how RAG should work:

Retriever learns from generator errors

Vector-based compressed memory beats raw-text retrieval

End-to-end gradients reconnect the entire pipeline

Accuracy improves without inflating compute

Embeddings become meaning-first, not token-first

This is the kind of architecture shift that will define the next generation of knowledge-augmented LLM systems.

The Rise of Prompt-Driven Development: Why the Future of Software May Be Written in Prompts

Mudassir Khan — Wed, 24 Sep 2025 03:36:03 +0000

f someone told you a few years ago that software engineers would spend more time writing prompts than code, you probably would have laughed. Yet, here we are, standing right on the edge of that reality.

Think about it. The last decade was all about cloud computing, DevOps, and agile transformations. Today, the shift is happening all over again—this time towards Prompt-Driven Development (PDD) and Governed Prompt Software (GPS) Engineering.

📖 A Little Story: From Code to Prompts

Imagine you're building a chatbot for a healthcare startup.

The Old Way: You'd spend weeks coding dialogue trees, managing APIs, and writing endless if/else statements.

The New Way: You simply design structured prompts: "If a patient reports chest pain, escalate immediately to a human doctor and provide emergency guidelines."

The code is still there, of course. But most of the intelligence now comes from how you craft prompts, set rules, and govern the AI's behavior.

It feels less like writing code… and more like writing instructions for a very smart but unpredictable intern. 😅

🚀 What is Prompt-Driven Development (PDD)?

PDD treats prompts the same way traditional software treats code. Instead of just focusing on syntax and logic, you are now responsible for designing prompt workflows, testing them, and documenting your design choices.

Some of its main building blocks include:

Prompt Requests (PRs): Think of them as reusable functions, but for prompts.

Architectural Decision Records (ADRs): Why did you phrase the prompt this way? Why was this workflow better than another?

Prompt History Records (PHRs): A changelog of how your prompts evolve over time, just like version control for code.

Testing & Validation: Running prompts against test cases (almost like TDD, but for language models).

🛡️ And What About GPS (Governed Prompt Software)?

If PDD is about building the system, GPS is about keeping it safe.

AI is incredibly powerful, but it can also be unpredictable. GPS Engineering ensures that your prompts don't just "work," but that they also follow governance rules:

Prevent biased or harmful outputs.
Ensure compliance with safety standards.
Maintain accountability (who wrote this prompt, and why?).

Think of it as DevSecOps for prompts—a safety layer that ensures your AI systems are trustworthy.

💡 Why Does All This Matter?

Let's look at a few real-world scenarios:

Finance: Imagine tokenizing ETFs (like what BlackRock is experimenting with). Instead of traders managing everything manually, prompts can govern transactions, risk checks, and reporting in real time, 24/7.

Healthcare: A digital nurse agent could run entirely on prompt workflows, escalating only when human intervention is truly required.

Education: Personalized tutors, driven by carefully crafted prompts, could adapt teaching styles to each student in ways no static curriculum ever could.

In all of these cases, the prompts themselves become the real intellectual property (IP).

⚡ The Big Shift: From Coder to Prompt Architect

Just as the rise of DevOps created new roles like SREs and Platform Engineers, this new era is already giving birth to new roles:

Prompt Architects

AI Workflow Engineers

Governance Leads for AI Systems

The engineers of tomorrow may not just write Python or Java—they will design conversational logic, governance frameworks, and agent orchestration strategies.

⚠️ A Reality Check

We are still in the very early innings of this journey. Every company has its own approach, and the standards are still evolving. But remember: agile, DevOps, and cloud computing all started this way—as small experiments that eventually reshaped the entire industry.

Prompt-Driven Development and GPS Engineering could be the next major transformation.

🧭 Final Thoughts

The future of software might not be written line by line in code, but designed prompt by prompt, governed with care, and orchestrated at scale.

The real question is: are you ready to evolve from just a developer into a prompt architect?

Because the next generation of apps won't just be coded. They'll be prompted, governed, and trusted.

👉 What do you think? Let me know in the comments! Will prompts ever truly replace coding workflows, or will they simply add another powerful layer to the software stack?

I Had a Fight With My Toaster. It Made Me Realize Everything About the Future of AI.

Mudassir Khan — Tue, 23 Sep 2025 06:43:40 +0000

It was 7 AM. The coffee was brewing, the sun was streaming in, and I was arguing with a toaster.

You had one job, I mumbled, scraping the blackened edges of my toast into the sink. The toaster, a "smart" one, was supposed to know my preferences. Yet, here we were. This little gadget, a tiny island of supposed intelligence, was completely disconnected from the rest of my morning. It didn't know I was running late. It didn't know the coffee machine had just finished a dark roast, which pairs terribly with burnt bread. It just knew toast.

This frustratingly common experience is a symptom of a larger problem. We've built millions of "smart" things, but they're not wise. They're isolated, they follow rigid rules, and they don't talk to each other.

But what if they did? What if we're on the verge of a world that isn't just smart, but truly alive? I recently stumbled upon a vision for the future that gave this idea a name: Agentia World.

From Rigid Commands to Living Conversations
Before we dive in, let's talk about how things work now. Almost every digital interaction you have is governed by APIs (Application Programming Interfaces). Think of an API as a strict restaurant menu. You can order item #3 with a side of #B, and the kitchen knows exactly what to do. It's efficient, but rigid. You can't say, "I'm feeling something light and spicy today." The system doesn't understand intent.

Agentia World imagines a future that scraps this menu. Instead, our devices—our "agents"—will have intelligent dialogues.

Imagine your car agent talking to your home agent. It wouldn't send a rigid API call like home.gate.open(). It would have a conversation:

Car Agent: "Hey, I'm about five minutes away with the owner. It's been a long day."

Home Agent: "Understood. Preparing for arrival. I'll open the garage, turn on the hallway lights, and start the evening focus playlist on the speakers."

This isn't just a command; it's a collaborative exchange based on understanding the goal. The home doesn't just obey; it anticipates.

A World Where Everything is an Agent
This is where the vision gets really wild. It proposes that everything becomes an AI agent. Not just your phone and your car, but the mundane, everyday objects.

Your coffee machine becomes a personal barista, checking your calendar for early meetings and your health tracker for your caffeine limits.

Your houseplants have agents that negotiate with the window blinds for the perfect amount of sunlight.

An entire city becomes a macro-agent. The traffic light agents talk to public transport agents, which talk to the power grid agents, all working in a seamless, living network to eliminate traffic jams and save energy.

This creates a living network that's constantly learning and adapting. It's a system that's both digital (the AI making decisions) and physical (the agents controlling real-world objects).

So, What Does This Mean for Us?
For the average person, it means a world that simply works. A world where the technology disappears into the background, seamlessly orchestrating our lives for the better. The constant micro-management of our digital lives fades away.

For us developers, it represents the next frontier. We'll move from building isolated apps and services to designing collaborative agents that can negotiate, learn, and act within a massive, decentralized ecosystem. The challenge will shift from writing rigid code to teaching intelligent systems.

The fight with my toaster was a reminder of how far we still have to go. We don't just need smarter devices; we need a wiser world. And that's the promise of a future where everything, from our toasters to our cities, is part of one intelligent, living conversation.

From Full-Stack to AI + Web3: Why I’m Re-Entering Tech in 2025

Mudassir Khan — Sat, 06 Sep 2025 13:29:05 +0000

Hey DEV! I’m Mudassir Khan—a full-stack builder getting back to hands-on coding with a sharper focus on AI automation and blockchain. I’ve run products and client work for a while; now I’m doubling down on building in public, sharing what actually works, and collaborating with the community.

TL;DR

Full-stack with Next.js, React, Node.js, TypeScript
Automations with Python, n8n, CrewAI, a bit of PyTorch
Exploring Web3 (Solidity + practical dApps)
Open to open-source & startup collaborations (remote)

Why I’m back to building (and sharing)

I missed the craft: shipping features, deleting code that doesn’t pull its weight, and watching users find value. I’m here to write short, practical posts—the kind you can skim in 3–5 minutes and apply the same day.

What you can expect here

1) Small, repeatable wins

Short tutorials on things like:

turning messy scripts into reliable n8n flows
using CrewAI patterns for agent hand-offs
cutting build times, faster previews, cleaner DX

2) Open-source first

I’ll publish small utilities and examples. If a snippet helps you, PRs are welcome. If you want a maintainer Buddy, ping me.

3) Honest notes from the trenches

What failed, what shipped, and what I’d do differently. No buzzword bingo—just notes that save you time.

Tech I use (and enjoy)

Frontend: React, Next.js, Tailwind
Backend: Node.js/Express, TypeScript, Python
Data: SQL, MongoDB
AI/Automation: n8n, CrewAI, PyTorch (light), agents & orchestration
DevOps: Docker
Web3: Solidity basics, dApp prototypes

Recent tiny win: a CLI to batch-convert assets to modern formats and auto-rewrite imports in Next.js repos—helped cut bundle size and image payloads.

2025 focus

AI agents for real developer workflows (not demos)
Developer utilities that reduce toil
Web3 experiments where decentralization makes sense
Writing and shipping in public, consistently

Let’s collaborate

Building an OSS tool and need a teammate?
Startup founder who needs a pragmatic full-stack dev?
Have an idea around AI × automation × Web3?

Comment, DM, or drop your GitHub—happy to jam on ideas, review PRs, or pair on a weekend hack.

Thanks for reading. If any of this resonates, hit Follow—the next post will be a 5-minute automation guide you can copy, tweak, and ship. 🚀

— Mudassir Khan