DEV Community: CiteScore

How to check if ChatGPT and Perplexity can cite your site: a 10-step checklist

CiteScore — Sun, 11 Oct 2026 19:43:41 +0000

AI assistants increasingly answer questions with links to sources. If you want your pages cited, they need to be reachable, readable, and easy to attribute. This checklist covers the checks you can run yourself, roughly in the order you should do them. None of them guarantees a citation. They remove common blockers and give you a way to measure what happens.

1. Check crawler access in robots.txt

Open https://yourdomain.com/robots.txt and look for rules that block AI crawlers. OpenAI uses separate crawlers: GPTBot (training), OAI-SearchBot (search in ChatGPT), and ChatGPT-User (fetches when a user asks ChatGPT to open a page). Perplexity uses PerplexityBot for indexing and Perplexity-User for user-requested fetches. Crawler names change, so confirm them against each vendor's current documentation. A blanket Disallow: / under User-agent: * blocks everything. Blocking GPTBot does not block OAI-SearchBot, so review each agent you care about.

2. Run a server-rendered HTML check

Not every fetcher runs JavaScript reliably. Run curl -s https://yourdomain.com/your-page and confirm that the title, main heading, and key paragraphs appear in the raw HTML. If the page is an empty shell that a script fills in later, move the important content into the server response or prerender it.

3. Add an llms.txt file

llms.txt is a proposed convention: a Markdown file at /llms.txt that gives language models a short map of your site. It starts with a one-line summary, then links to your most important pages with brief descriptions. It is not a formal standard, and not every AI system reads it. It takes little time to write, so treat it as a helpful index rather than a ranking lever. Keep the list to your core pages.

4. Check your sitemap

Make sure /sitemap.xml exists, lists only canonical URLs that return a 200 status, and includes accurate lastmod dates. Submit it in Google Search Console and Bing Webmaster Tools. Some AI search tools draw on search indexes, so checking Bing is worth the few minutes.

5. Add structured data

Add JSON-LD to the page head that describes what the page is. For articles, use Article with headline, author, datePublished, and dateModified. For the site, use Organization with name, url, logo, and sameAs links to your official profiles. Validate the markup with Google's Rich Results Test or the schema.org validator. Schema does not guarantee citations, but it removes ambiguity about who wrote the page and when it was updated.

6. Write answer-first headings

AI systems often lift a passage that directly answers a question. Make each H2 a plain question or task, and put the answer in the first one or two sentences beneath it. Add detail after that. Use "Does robots.txt block ChatGPT?" rather than "A few thoughts on crawlers." Keep paragraphs short and specific, with concrete names, numbers, and dates.

7. Publish a clear about and author page

Models and readers both need to know who is behind a page. Create an About page that says what your site does, who runs it, and how to contact you. Give articles a byline that links to an author page with a real role, relevant past work, and links to profiles you control. Keep claims verifiable and avoid vague superlatives.

8. Use consistent brand naming

Pick one official name and use it everywhere: the site title, page titles, the name field in your schema, social profiles, and the About page. If your brand appears under three spellings, a model may treat them as three different entities. Match capitalization and domain across the web so your pages and mentions point to the same thing.

9. Test real prompts in ChatGPT and Perplexity

Write 10 to 20 questions your ideal reader would actually ask, such as "how do I check if my site blocks AI crawlers?" Run each one in ChatGPT with search turned on, and in Perplexity. Record whether your domain is cited, whether a competitor is, and what the answer says about your topic. Repeat each prompt a few times on different days, because answers vary. Keep a simple spreadsheet with the date, prompt, tool, whether you were cited, and the URL cited. One run tells you little. A trend over several weeks tells you something.

10. Track referrals from chat.openai.com and perplexity.ai in analytics

In your analytics tool, look at referral traffic from AI sources. ChatGPT clicks have shown up with referrers such as chat.openai.com and chatgpt.com, so check both. Perplexity usually appears as perplexity.ai. Create a channel group or segment for these sources so you can see which landing pages they reach and whether those visitors convert. Many AI clicks arrive with no referrer at all, so this number undercounts. Compare it with the citations you logged in step 9.

Wrap-up

Work through the list once, fix the blockers, and then re-run your prompt tests in a month. If you would rather start with an automated pass over some of these checks, there is a free check at citescore.vercel.app/free-check. The manual steps above work without it.

FAQPage JSON-LD in 2026: still worth adding for AI answers?

CiteScore — Sat, 10 Oct 2026 22:09:32 +0000

FAQ markup used to be one of the easiest structured data wins. Its search benefit has mostly gone, but the underlying idea still helps: a page that states its questions and answers explicitly is easier for machines to parse. Here is where things stand and how to add it correctly.

What FAQPage schema is

FAQPage is a schema.org type for pages whose main content is a list of questions, each with an answer. You describe it in JSON-LD, a script block that sits alongside the HTML. Each item is a Question with a name (the question text) and an acceptedAnswer of type Answer with a text property.

What changed in Google Search

FAQ rich results once let a search listing show expandable questions under the link. Google narrowed that in 2023, limiting FAQ rich results to well-known government and health websites. The feature has shrunk further since then, and for most sites it no longer produces a visible result in search. If your goal is a rich result, FAQPage markup is unlikely to deliver one.

Why structured Q&A still helps AI answers

Answer engines and retrieval systems split pages into passages and try to match a question to the passage that answers it. A clear question as a heading, followed by a direct two- or three-sentence answer, gives that process an obvious unit to extract. FAQPage markup repeats that structure in machine-readable form, so a parser does not have to guess where one question ends and the next begins.

Be clear about the limits. Markup does not guarantee citation, and no AI provider has said it is required. Its value is that it describes content you already publish. The visible text does the real work.

A complete JSON-LD example

This example marks up three questions about FAQPage itself. In a real page, each question and answer should also appear as visible text in an FAQ section.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is FAQPage schema?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "FAQPage is a schema.org type that describes a page whose main content is a list of questions, each with an answer. It is written as JSON-LD."
      }
    },
    {
      "@type": "Question",
      "name": "Does FAQPage markup still produce rich results in Google?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Not for most sites. Google first limited FAQ rich results to well-known government and health websites in 2023, and the feature has since been reduced further."
      }
    },
    {
      "@type": "Question",
      "name": "Should I add FAQPage markup to every page?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No. Add it only to pages that show a real set of questions and answers, and make sure every item in the markup is visible on that page."
      }
    }
  ]
}
</script>

Rules

Every question and answer must be visible on the page. Do not put text in the markup that readers cannot see. Collapsed accordions are fine as long as users can open them.
Use questions readers actually ask. Do not write questions just to repeat keywords.
Keep answers factual and specific. Two to four sentences is usually enough.
No marketing fluff. Avoid "best-in-class" language and sales calls to action inside answers.
Mark up only the Q&A that exists. Do not copy the same FAQ onto every page of a site.

How to validate

Check the JSON parses. Paste the block into a JSON linter. A single trailing comma breaks the whole block.
Check the schema. Run the page through validator.schema.org to confirm the types and properties are correct.
Check eligibility. Google's Rich Results Test will show whether the markup is detected. A "valid" result does not mean a visible rich result will appear for most sites.
Compare against the page. Open the live page and check that every question and answer in the JSON matches visible text word for word.
Check the rendered HTML. If the block is injected by JavaScript, confirm it is present in the HTML a crawler receives, not only after the page runs.

Common mistakes

Marking up text that is not on the page.
Answers that are mostly links, ads, or a pitch.
Invalid JSON, such as trailing commas or unescaped quotes inside the answer text.
Several FAQPage blocks on one page, or FAQPage markup on a page with no FAQ.
Treating the markup as a ranking boost rather than a description of content.
Injecting the block with JavaScript after load, so some crawlers never see it.

Where to start

If you want a starting block for your own questions, the free generator at https://citescore.vercel.app/faq-schema-generator builds the JSON-LD from a list of Q&A pairs. Review its output against the rules above before you publish it.

robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained

CiteScore — Sat, 10 Oct 2026 20:43:44 +0000

AI companies now run their own crawlers, and most identify themselves with a user-agent name you can put in robots.txt. This guide explains what the main ones do, how training crawlers differ from search and answer crawlers, and gives copy-paste rules for three common policies. Confirm details against each vendor's current documentation before you deploy.

First, a limit: robots.txt is a request, not access control. Well-behaved crawlers honor it. Scrapers that ignore it are not stopped by it, and a blocked URL can still appear in search results if other sites link to it.

The crawlers

GPTBot is OpenAI's crawler. OpenAI documents it as collecting web content that may be used to train its models.

OAI-SearchBot is OpenAI's crawler for ChatGPT search. It indexes pages so ChatGPT can show them in search results and cite them. It is separate from GPTBot, so you can allow one and block the other.

ChatGPT-User runs when a person asks ChatGPT to open a page during a conversation. It is user-triggered rather than a bulk crawl, and OpenAI notes that robots.txt rules may not apply to these requests.

ClaudeBot is Anthropic's crawler, used to collect content that may be used for training. Anthropic also documents separate agents for search and user-requested fetching, so check its current list before writing rules for them.

PerplexityBot indexes pages for Perplexity's answers and citations. Perplexity says it respects robots.txt. Its user-triggered fetcher, Perplexity-User, behaves like ChatGPT-User.

Google-Extended is not a crawler. It is a robots.txt token that controls whether content Google has already crawled can be used to train Gemini and for certain grounding features. Googlebot still crawls your site for Search whether or not you block this token, so blocking Google-Extended does not remove you from Google Search.

Training crawlers vs. search and answer crawlers

The useful split is purpose.

Training crawlers collect content to build future model versions. GPTBot, ClaudeBot, and the Google-Extended token belong here. Blocking them signals that you don't want future training use. It does not undo training that has already happened.
Search and answer crawlers fetch pages so an assistant can find and cite them. OAI-SearchBot and PerplexityBot belong here. Blocking them makes it less likely your pages appear as sources.
User-triggered fetchers such as ChatGPT-User request one page because someone asked for it.

If you want citations and referral traffic from AI answers, allow the search crawlers. If your objection is only to training, block the training agents and keep the search agents open.

Example rules

Allow everything. This is the default, written out explicitly:

User-agent: *
Allow: /

Block training, allow search and answers:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Block all of these AI crawlers:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

A crawler obeys the group that names it and ignores the * group. So the bots you block stay blocked even though * allows everything. The * group applies to crawlers you did not name, such as Googlebot and Bingbot.

Common mistakes

Blocking everything with User-agent: * and Disallow: /. That hides your site from Google and Bing too. For AI-only policies, name the bots.
Thinking Google-Extended controls Search. It only affects AI training and grounding use. Blocking Googlebot is what removes you from Search.
Putting the file in the wrong place. It must live at https://example.com/robots.txt. Each subdomain, such as blog.example.com, needs its own file.
Getting path case wrong. /Blog/ and /blog/ are different paths. Bot names are case-insensitive, but copy the vendor's spelling anyway.
Assuming it is enforced. Robots.txt is voluntary. If you need hard blocking, use server, CDN, or WAF rules. A bot-protection rule that returns 403 blocks crawlers regardless of robots.txt.
Using it to remove pages. Blocking stops crawling, so a search engine cannot see a noindex tag on a blocked page. To keep a page out of an index, allow crawling and use noindex, or remove the page.
Forgetting to recheck. Crawler names change, and a CMS or CDN deploy can overwrite your file.

How to test

Confirm the file is live. Open https://example.com/robots.txt. It should return 200 and plain text. An HTML page means your CMS is serving a fallback, and a 404 means no rules exist at all.
Check the rules with a parser. Python's standard library handles simple files:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
for bot in ["GPTBot", "OAI-SearchBot", "PerplexityBot"]:
    print(bot, rp.can_fetch(bot, "https://example.com/blog/post"))

Its wildcard handling differs from Google's, so verify key paths with a second parser.

Check your logs. Search access logs for the user-agent strings. Allowed bots should reach your pages, and blocked bots should request only robots.txt. User-agent strings can be spoofed, so verify important bots by IP where the vendor publishes ranges.
Retest after every deploy. Confirm the live file still matches your intent.

Wrap-up

Decide your policy first: allow all, block training only, or block everything. Write named groups for the bots you care about, host the file at the root, and verify it with logs and a parser.

If you would rather generate these groups than type them, the free generator at https://citescore.vercel.app/robots-txt-ai-crawler-generator builds them.

How to write an llms.txt file (with a free generator)

CiteScore — Sat, 10 Oct 2026 17:37:05 +0000

Large language models increasingly read websites to answer questions, but a typical page is full of navigation menus, scripts, cookie banners, and footers. An llms.txt file is a short Markdown file that gives those models a clean map of your site: what it is, and which pages matter most. This guide covers what the file is, where it goes, how to format it, what to avoid, and how it differs from robots.txt.

What llms.txt is

llms.txt is a proposed convention, described at llmstxt.org since 2024. It is not an official standard from a standards body, and no AI provider has promised to read it in every case. Treat it as low-cost documentation for tools that choose to use it. Its main appeal is that it is easy to write and easy to maintain.

Where it goes

Place the file at the root of your domain, so it is available at https://example.com/llms.txt. A file in a subdirectory such as /docs/llms.txt will not be found by convention. Serve it as plain text with a 200 status code, and make sure it is not behind a login or a redirect chain.

The Markdown format

The format is deliberately simple. It has four parts, in this order:

An H1 with the name of the site or project. This is the only required element.
A blockquote summary. One or two sentences that say what the site is and who it is for.
Optional paragraphs of context. Plain text that explains anything the links cannot, such as scope or how the documentation is organized.
H2 sections containing link lists. Each entry is a Markdown link, a colon, and a short description.

The skeleton looks like this:

# Site Name

> One or two sentences that say what the site is and who it is for.

Optional context paragraph.

## Section name

- [Page title](https://example.com/page): Short description of what the page covers.

A complete example

Here is a file for a fictional note-taking app called Quillpad:

# Quillpad

> Quillpad is a note-taking app for teams that stores notes as plain Markdown and syncs them across devices. Setup guides, API docs, and pricing are linked below.

Quillpad saves every note as a standard .md file. The API is versioned, and the current version is v2.

## Documentation

- [Getting started](https://example.com/docs/getting-started): Install Quillpad and create your first workspace.
- [Markdown syntax](https://example.com/docs/markdown): Supported formatting, including tables and task lists.
- [API reference](https://example.com/docs/api): Endpoints, authentication, and rate limits for v2.

## Product

- [Pricing](https://example.com/pricing): Plans, seat limits, and what each tier includes.
- [Security](https://example.com/security): Encryption, data residency, and audit reports.

## Optional

- [Changelog](https://example.com/changelog): Release notes by date.
- [Careers](https://example.com/careers): Open roles.

The Optional section has a specific meaning in the convention. It marks secondary content that a tool can skip when it has limited context space.

Common mistakes

A missing or wrong H1. The H1 anchors the file. Without it, the file does not follow the format.
Links with no descriptions. A bare list of URLs tells a model very little. Each link needs a sentence about what is there.
Serving HTML instead of Markdown. A catch-all route can return your HTML error page at /llms.txt. Check the response with curl or your browser's network tab.
Dead or redirecting links. Use final URLs, and check them whenever you restructure the site.
Listing everything. A 2,000-line file is less useful than a focused 30-line one. Prioritize the pages you most want people and tools to find.
Marketing copy in the summary. The blockquote should describe the site, not sell it. Phrases like "the best" or "revolutionary" add nothing.
Letting it go stale. A file that points to removed pages is worse than no file at all.
Blocking it. Do not disallow /llms.txt in robots.txt, and do not require authentication to read it.

How it differs from robots.txt

These two files do different jobs, and most sites benefit from having both.

robots.txt tells crawlers what they may request. It holds allow and disallow rules for each user agent and follows the Robots Exclusion Protocol (RFC 9309). It controls crawling, and compliance is voluntary, though major crawlers generally honor it.

llms.txt does not restrict access. It is a curated map that points a model to the content you consider most useful and briefly explains each part. If you want to keep a crawler away from pages, use robots.txt or real access controls. If you want to point a model toward your best documentation, write an llms.txt.

A short checklist

Put the file at /llms.txt on your root domain.
Start with an H1 and a one- or two-sentence blockquote.
Use H2 sections with full URLs and one-line descriptions.
Move secondary material under "Optional."
Confirm that every link returns a 200 status.
Review the file whenever you make major changes.

If you would rather not write the file by hand, the free llms.txt generator can produce a starting draft. Edit it, then check the result against the rules above before you publish.

Which AI crawlers should you allow in robots.txt? (GPTBot, PerplexityBot, Google-Extended and more)

CiteScore — Sat, 10 Oct 2026 14:44:15 +0000

AI assistants find and cite websites in two different ways. Some crawlers collect pages to train future models. Others fetch pages so an assistant can search the web and link to your site in an answer. Your robots.txt file can treat these differently, but only if you know what each user-agent does.

One caveat first: robots.txt is a request, not a lock. Reputable crawlers honor it. Anything else needs server-level or firewall blocking.

The major AI crawlers

User-agent	Owner	What it does
GPTBot	OpenAI	Collects content for training OpenAI models
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT search
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user asks for it
ClaudeBot	Anthropic	Collects content for training Anthropic models
Claude-SearchBot	Anthropic	Indexes pages for Claude's search features
Claude-User	Anthropic	Fetches a page when a Claude user asks for it
PerplexityBot	Perplexity	Indexes pages for Perplexity answers and citations
Perplexity-User	Perplexity	Fetches a page when a Perplexity user asks for it
CCBot	Common Crawl	Builds a public web archive that many AI teams use for training
Google-Extended	Google	A control token, not a crawler. Governs use of content for Gemini training and grounding
Applebot-Extended	Apple	A control token. Governs use of Applebot-crawled content for Apple's AI training

Google-Extended and Applebot-Extended are not separate crawlers. They are opt-out tokens that apply to content already crawled by Googlebot or Applebot. Blocking them does not remove you from Google Search, and AI Overviews follow Googlebot and standard directives such as nosnippet.

A recommended robots.txt block

If you have no AI rules today, your site is already open to all of these bots. The block below opts out of training while staying visible in AI search:

# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Search and answer crawlers: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

Keep your existing rules for Googlebot and * in their own groups.

Training vs. answer crawlers: the trade-off

Training crawlers feed future model versions. Blocking them signals that you do not want your content used for future training. It does not remove anything a model has already learned, and it is hard to measure its effect on your traffic.

Answer crawlers decide whether you appear in live answers. If you block PerplexityBot or OAI-SearchBot, your pages are less likely to be cited when people ask a relevant question. For most sites that want visibility, that is the bigger cost.

The block above is a reasonable default. Adjust it if you have licensing concerns, paywalled content or private material. Vendors also note that user-triggered fetchers such as ChatGPT-User may not apply robots.txt rules, so treat those rules as best effort.

How to verify your rules

Open https://yoursite.com/robots.txt in a browser. It should return 200 and plain text. A 404 means no rules, so everything is allowed. A 5xx error can stop crawlers entirely.
Test the rules with Python's standard library:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://yoursite.com/robots.txt")
rp.read()
for bot in ["GPTBot", "OAI-SearchBot", "PerplexityBot"]:
    print(bot, rp.can_fetch(bot, "https://yoursite.com/blog/post"))

Search your server logs or analytics for these user-agent strings. Confirm that the bots you allowed reach your pages, and that the bots you blocked request only robots.txt.
Check your CDN or WAF. A bot-protection rule that returns 403 blocks crawlers regardless of robots.txt, and it can block the bots you meant to allow.

Python's parser does not handle every wildcard the way Google does, so test important paths with more than one tool.

Common mistakes

Blocking Googlebot by accident. Google-Extended is not Googlebot. Blocking Googlebot removes you from Google Search.
Assuming a * rule covers AI bots. A crawler follows the most specific group that matches its name and ignores *. If GPTBot has its own group, that group must contain every rule you want for GPTBot.
Using robots.txt for privacy. The file is public and it is only a request. Protect private pages with authentication.
Mixing up Disallow and Allow. Disallow: / blocks the whole site, while an empty Disallow: allows everything. A misplaced slash can hide your entire site from search.
Ignoring the CDN. Bot-protection rules can override what your robots.txt says.
Expecting instant results. Crawlers cache robots.txt, and removal from AI answers takes time.

Test your robots.txt AI-crawler rules for free at https://citescore.vercel.app/free-check

How to write an llms.txt file (with a copy-paste template)

CiteScore — Sat, 10 Oct 2026 14:42:06 +0000

How to write an llms.txt file (with a copy-paste template)

If you want AI assistants and large language models to understand your site accurately, you can give them a short, curated map. That map is an llms.txt file. This guide explains what it is, where it goes, how to format it, and how to test it before you rely on it.

What llms.txt is

llms.txt is a proposed standard, introduced by Jeremy Howard in 2024 and documented at llmstxt.org. It gives language models a plain Markdown file that summarises your website: what it is, what it offers, and which pages matter most.

It works alongside two files you probably already have. robots.txt tells crawlers what they may access. A sitemap.xml lists every URL. llms.txt does something different: it selects the most important content and explains it in a format a model can read within a limited context window.

Be realistic about adoption. The standard is still informal, and not every AI company has confirmed that it reads llms.txt. Treat it as low-cost documentation that helps the tools that do use it, not as a guaranteed ranking lever.

Where it goes

The file must live at the root of your domain:

https://example.com/llms.txt

Serve it with a 200 status as plain text or Markdown. It should not redirect to a login page or depend on JavaScript to render. If you have subdomains such as docs.example.com and want them covered, publish a separate llms.txt on each one.

The format

The structure is deliberately simple. Include these parts, in this order:

An H1 title. The name of the project or site. This is the only required element.
A blockquote summary. One or two sentences that explain what the site is.
Optional paragraphs. Extra context in plain prose, such as who the site is for or what is out of scope.
H2 sections. Groups of links, each written as a Markdown list item: - [Link title](URL): short note.
An Optional section. Links a model can skip when context is tight.

Use absolute URLs and point them at clean, public pages that load directly.

Full copy-paste template

Replace the placeholders and delete any section you do not need:

# Example Co

> Example Co makes invoicing software for freelancers. The product covers quotes, recurring billing and tax exports for the US and UK.

Pricing changes from time to time, so check the pricing page for current numbers. Documentation covers the public REST API and the Zapier integration.

## Docs

- [Getting started](https://example.com/docs/getting-started): Create an account and send your first invoice
- [API reference](https://example.com/docs/api): Endpoints, authentication and rate limits
- [Webhooks](https://example.com/docs/webhooks): Event types and payload examples

## Product

- [Features](https://example.com/features): Full list of invoicing and tax features
- [Pricing](https://example.com/pricing): Plans and per-seat costs

## Optional

- [Blog](https://example.com/blog): Product updates and tutorials
- [Careers](https://example.com/careers): Open roles

Common mistakes

Putting the file in the wrong place. /docs/llms.txt is not the root. Crawlers look for /llms.txt.
Dead or redirected links. Every URL should return 200. A list full of 404s signals an unmaintained file.
Listing everything. Pasting 300 URLs defeats the purpose. Pick the 10 to 30 pages that answer the most common questions.
Vague descriptions. "Our amazing solutions" tells a model nothing. Say what the page actually contains.
Marketing copy in the summary. Keep the blockquote factual. Plain statements are easier to use than slogans.
Forgetting the H1. Without a title the file does not follow the format, and parsers may fail on it.
Blocking the file. If a firewall, CDN rule or robots.txt rule blocks /llms.txt, nothing else you write matters.
Letting it go stale. Update the file when you launch, rename or remove a major page.

How to test it

Check the status and redirects. Run curl -I https://example.com/llms.txt and confirm a 200 response with no redirect chain.
Check the content type. It should be text/plain or text/markdown, not text/html.
Read it as a stranger would. Open the file and ask whether someone who has never seen the site can tell what it offers from the first 20 lines.
Check every link. Extract the URLs from the file and request each one with curl. Flag anything that is not a 200.
Confirm the structure. The first line should be a single H1, followed by a blockquote, then H2 sections that contain list items.
Check robots.txt. Make sure the pages you link are not disallowed for the AI user agents you want to reach.

Repeat these checks after every deploy that touches the file.

Summary

A good llms.txt is short, accurate and maintained. Start with ten strong links, a factual summary and a clean root URL, then expand as your documentation grows.

You can check your site's AI-crawler setup for free at https://citescore.vercel.app/free-check