I shipped llms.txt, JSON-LD and AI crawler allowances. Here's what each one actually does.

Liam Hwang — Sat, 12 Sep 2026 00:27:53 +0000

I am building a site whose entire job is to be quoted by AI assistants. It is the marketing site for SearchD, an answer engine optimization practice, which makes the site both the product and the test case.

Three things get recommended everywhere for this: let the AI crawlers in, publish an llms.txt, ship JSON-LD. I implemented all three. Then I went looking for evidence that each one does what the advice says it does.

Only one of the three can fail silently. The other two turned out to do a different job from the one the advice promises.

1. Can the crawlers actually reach you?

A robots.txt that allows GPTBot proves nothing. The file is a request to a well-behaved crawler; it has no effect on the layer in front of it. If your CDN has a bot-management rule, the crawler can be challenged or blocked at the edge and never get as far as reading your robots file.

The only way to know is to ask as each crawler and look at the status code:

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot \
          Google-Extended Applebot-Extended CCBot Amazonbot Bingbot Googlebot; do
  printf "%-20s %s\n" "$ua" "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/)"
done

Run against my own site on 11 September 2026:

GPTBot               200
OAI-SearchBot        200
ChatGPT-User         200
ClaudeBot            200
Claude-User          200
PerplexityBot        200
Google-Extended      200
Applebot-Extended    200
CCBot                200
Amazonbot            200
Bingbot              200
Googlebot            200

A 403 or a 503 here outranks any amount of markup work, because it means nothing downstream of it can matter. Passing the check does not get you cited. It removes a way of being invisible that you would otherwise never see.

The status code leaves two things open. It says nothing about whether the response body is a JavaScript shell, so fetch one and read it: a 200 carrying an empty <div id="root"> is a 200 that says nothing. It also says nothing about whether the crawler that matters is on your list. The names change. Claude-SearchBot and Perplexity-User were both added to mine after the first pass.

2. llms.txt: generate it, but do not count on it

llms.txt is a markdown index of your site at /llms.txt, meant to give a model a clean map instead of making it parse your nav. Generating it is cheap if your content is already in a collection. This is an Astro endpoint, about forty lines:

// src/pages/llms.txt.ts
import type { APIRoute } from 'astro';
import { getCollection } from 'astro:content';
import { SITE } from '../config';

const line = (title: string, path: string, note: string) =>
  `- [${title}](${SITE.domain}${path}): ${note}`;

export const GET: APIRoute = async () => {
  const answers = await getCollection('answers');
  const glossary = await getCollection('glossary');

  const body = `# ${SITE.name}

> ${SITE.description}

## Answers
${answers.map((a) => line(a.data.question, `/answers/${a.id}`, a.data.description)).join('\n')}

## Glossary
${glossary.map((t) => line(t.data.term, `/glossary/${t.id}`, t.data.definition)).join('\n')}
`;

  return new Response(body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } });
};

Google states plainly that Search ignores these files, and that having one neither helps nor harms. No major engine has published a commitment to read it.

I keep mine. It is generated, so it costs nothing to maintain, and it makes a useful index when I point an agent at my own site. Nobody has shown it does harm either. None of that amounts to a model reading it, and if you are writing a plan for someone else, llms.txt does not belong in the column marked citations.

3. Schema markup did not lift citations

This is the one I expected to defend and could not.

Ahrefs ran a controlled study across 1,885 pages (May 2026) and after matched controls found no citation lift from schema markup: +2.4% on AI Mode, +2.2% on ChatGPT, and −4.6% on AI Overviews. Those sit inside the noise. If you are adding schema to get quoted more often, the published evidence does not support you.

Schema still earns its place, for a different job. It is how you tell a machine that the "Liam Hwang" on the byline, the one on the author page and the one in the organization's founder field are one person rather than three. Get that wrong and you have manufactured the entity confusion you were trying to prevent.

The rule that makes it work is one @graph per page, with @id values that are stable URIs rather than page-local strings:

// src/lib/schema.ts
const id = (path: string) => `${SITE.domain}${path}`;

export const ORG_ID = id('/#organization');
export const authorPath = (slug: string) => `/authors/${slug}`;
export const authorId = (slug: string) => id(`${authorPath(slug)}#person`);

export const person = (a: Author) => ({
  '@type': 'Person' as const,
  '@id': authorId(a.slug),          // the same URI on every page that names them
  name: a.name,
  url: id(authorPath(a.slug)),
  // Writing for the site is not an employment claim.
  ...(a.slug === FOUNDER.slug ? { worksFor: { '@id': ORG_ID } } : {}),
});

A person's canonical URI is their author page, including the founder's. An id derived from a role (/about#founder) moves the day the role does, and it makes the founder a different kind of thing from everyone else who writes there.

Every page then references the node instead of redeclaring it (author: { '@id': authorId(slug) }, founder: { '@id': authorId(FOUNDER.slug) }), and the layout emits one <script type="application/ld+json"> carrying the whole graph.

Getting this wrong produces two bugs. Emitting a second Person node for someone who already has one is the split you were trying to prevent, so the helper that adds a post's author filters out the founder, whose node the layout already includes. Describing something in schema that is not visible on the page has the same shape: the markup and the rendered page have to agree, or you have told two stories about one object.

Check indexing before you check markup

All three are shipped on my site. None of them is the reason a page gets retrieved and quoted.

Crawler access, llms.txt and JSON-LD are upstream plumbing, and the binding constraint sits earlier than any of them. A page that is not indexed cannot be retrieved by anything that searches, however clean its markup is. Google Search Console reports that in one screen, as a count of submitted URLs against indexed ones, and on a young domain the gap between those two numbers will tell you more than any markup audit.

So the order is: run the crawler check, because it is the only one of the three that can cost you everything without saying so. Then read your index coverage, and keep reading it. Do the other two because they are cheap and correct, and file them under plumbing rather than citations.

DEV Community: Liam Hwang

I shipped llms.txt, JSON-LD and AI crawler allowances. Here's what each one actually does.

1. Can the crawlers actually reach you?

2. llms.txt: generate it, but do not count on it

3. Schema markup did not lift citations

Check indexing before you check markup