DEV Community: Furqan Khalid

Should You Serve Markdown to AI Agents? llms.txt, Accept: text/markdown and What Google Actually Said

Furqan Khalid — Tue, 27 Oct 2026 06:00:00 +0000

There's a 2026 argument that runs roughly like this:

"LLMs prefer Markdown. HTML is 80% noise. So serve Markdown to AI bots and they'll understand you better and cite you more."

Half of that is true. This post separates the half that is from the half that isn't, then shows how to do it safely if you decide to.

The three things people mean by "Markdown for AI"

Approach	What it is	Who reads it
`/llms.txt`	A curated Markdown index of your site at the root (llmstxt.org)	Mostly coding assistants and tools a user points at your docs
`.md` twins	Every page also available as Markdown, e.g. `/docs/auth.md`	Same as above, plus agents that know the convention
Content negotiation	Same URL, Markdown returned when the request sends `Accept: text/markdown`	Agents that ask for it; Cloudflare's "Markdown for Agents" does this at the edge

What the evidence says

On llms.txt for search visibility: not much.

Google's Gary Illyes said in July 2025 that Google doesn't support llms.txt and has no plans to.
Google's May 2026 guide to AI features in Search goes further. It says optimizing for AI features "is optimizing for the search experience, and thus still SEO," and lists llms.txt and special machine-readable formats among the things you don't need for Google (Google's AI optimization guide).
Log studies published in 2026 found the big AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) rarely request /llms.txt and mostly crawl HTML. An SE Ranking study of ~300k domains reported roughly 10% adoption and no measurable link between having the file and being cited.

On Markdown for agents: real, but a different job.

Cloudflare launched Markdown for Agents in 2026. If a request includes text/markdown in its Accept header, Cloudflare fetches your HTML, converts it, and returns Markdown with an x-markdown-tokens header estimating the size (Cloudflare docs). Their own example blog post went from about 16,000 tokens as HTML to about 3,000 as Markdown.
Google's John Mueller was blunt about serving Markdown to bots for SEO, reportedly calling it "a stupid idea" (Search Engine Journal).

Both of these can be true at once. Search crawlers are built to parse HTML and don't need help. Agents and coding assistants working within a context window benefit a lot from getting 3k tokens instead of 16k.

So when is it worth doing?

Worth it:

Developer docs, API references, SDK guides. Developers paste your docs URL into Cursor, Claude Code or ChatGPT all day. A clean Markdown version means fewer tokens, less nav junk and better answers about your product. Stripe, Anthropic, Cloudflare and Vercel all publish llms.txt or Markdown docs for this reason.
Agent-heavy flows, such as product catalogues or knowledge bases that agents query on users' behalf.

Not worth it (on its own):

A marketing site hoping Markdown will rank it higher in Google or get it cited more by ChatGPT. There's no evidence it does. Spend the time on content and crawlability.

How to do it without getting into trouble

The risk is cloaking: showing bots different content from users. Google's spam policies forbid that, and since May 2026 they explicitly apply to AI Overviews and AI Mode as well. Markdown is fine if it's the same content in a different format. It stops being fine the moment the Markdown version has extra claims, keywords or "notes for AI" that the HTML doesn't.

Rules I'd follow:

Generate Markdown from the same source as the HTML (your MDX, CMS, or the rendered HTML). Never maintain it by hand.
Send Vary: Accept so caches don't serve Markdown to browsers or HTML to agents.
Point to it with rel="alternate" so it's discoverable rather than hidden.
Keep the HTML canonical. Add Link: <https://example.com/page>; rel="canonical" on Markdown responses, or add X-Robots-Tag: noindex to keep .md URLs out of search results.
Don't sniff user agents. Respond to what the client asks for (Accept), not to who you think it is.

Implementation: Next.js content negotiation

Assume your docs are MDX files. We'll serve the raw Markdown (front matter stripped) at /docs/foo.md and also when someone requests /docs/foo with Accept: text/markdown.

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";

export function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  if (!pathname.startsWith("/docs")) return NextResponse.next();

  const accept = req.headers.get("accept") ?? "";
  const wantsMarkdown =
    pathname.endsWith(".md") ||
    // Only when Markdown is explicitly listed. Browsers send text/html and */*.
    /(^|,)\s*text\/markdown/i.test(accept);

  if (!wantsMarkdown) return NextResponse.next();

  const slug = pathname.replace(/^\/docs\/?/, "").replace(/\.md$/, "") || "index";
  const url = req.nextUrl.clone();
  url.pathname = `/api/md/${slug}`;
  return NextResponse.rewrite(url);
}

export const config = { matcher: ["/docs/:path*"] };

// app/api/md/[...slug]/route.ts
import { readFile } from "node:fs/promises";
import path from "node:path";
import matter from "gray-matter";

const DOCS_DIR = path.join(process.cwd(), "content/docs");

export async function GET(_req: Request, { params }: { params: Promise<{ slug: string[] }> }) {
  const { slug } = await params;
  const rel = slug.join("/");

  // Basic path-traversal guard
  const file = path.normalize(path.join(DOCS_DIR, `${rel}.mdx`));
  if (!file.startsWith(DOCS_DIR)) return new Response("Not found", { status: 404 });

  let source: string;
  try {
    source = await readFile(file, "utf8");
  } catch {
    return new Response("Not found", { status: 404 });
  }

  const { content, data } = matter(source);
  const body = `# ${data.title ?? rel}\n\n${stripJsx(content)}`;
  const canonical = `https://example.com/docs/${rel === "index" ? "" : rel}`;

  return new Response(body, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Vary": "Accept",
      "Link": `<${canonical}>; rel="canonical"`,
      "Cache-Control": "public, max-age=3600",
    },
  });
}

// Remove MDX component tags but keep their text content.
function stripJsx(md: string) {
  return md
    .replace(/^import .*$/gm, "")
    .replace(/<([A-Z][A-Za-z0-9]*)[^>]*\/>/g, "")
    .replace(/<\/?[A-Z][A-Za-z0-9]*[^>]*>/g, "");
}

And in the HTML page's <head>:

// app/docs/[...slug]/page.tsx
export async function generateMetadata({ params }) {
  const { slug } = await params;
  return {
    alternates: {
      types: { "text/markdown": `/docs/${slug.join("/")}.md` },
    },
  };
}

Test it:

curl -s https://example.com/docs/auth | head -c 200                           # HTML
curl -s -H "Accept: text/markdown" https://example.com/docs/auth | head -20  # Markdown
curl -sI -H "Accept: text/markdown" https://example.com/docs/auth | grep -i -E "content-type|vary|link"

On Cloudflare? If you're on a plan that includes Markdown for Agents, you can switch it on in the dashboard (AI Crawl Control) instead of writing this. Writing it yourself only pays off if you want cleaner output from your source files than an HTML-to-Markdown conversion gives you.

And llms.txt?

If you run a docs site, a short llms.txt is cheap and helpful for coding assistants. Generate it at build time from your nav config so it never drifts out of sync:

# Acme API

> Acme is a payments API for marketplaces. These docs cover authentication, payouts and webhooks.

## Core docs
- [Authentication](https://example.com/docs/auth.md): API keys, OAuth, key rotation
- [Payouts](https://example.com/docs/payouts.md): schedules, currencies, limits
- [Webhooks](https://example.com/docs/webhooks.md): events, retries, signature verification

## Optional
- [Changelog](https://example.com/changelog.md)

Just don't sell it to your boss as an SEO project.

Conclusion

For Google rankings and AI Overviews: Markdown does nothing. HTML is fine.
For agents and coding assistants using your docs: Markdown helps a lot. Do it.
To stay out of trouble: same content, Vary: Accept, canonical to the HTML, no bot-only text.

Next in the series: Hidden prompts in your HTML are now spam.

ChatGPT Can't Cite What Bing Hasn't Indexed: Setting Up IndexNow in Next.js

Furqan Khalid — Mon, 26 Oct 2026 06:00:00 +0000

Most developers treat Bing as a rounding error. It's a small share of classic search, so "submit the sitemap once and forget it" felt fine.

In 2026 that's a mistake. ChatGPT search and Microsoft Copilot both draw on Bing's index to find pages to cite. If Bing hasn't crawled your new docs page, or still has last month's version of your pricing page, those assistants are working from stale or missing information.

Bing supports a simple fix: IndexNow, a protocol that lets you push "this URL changed" instead of waiting for a crawler to notice. It takes about 30 minutes to wire into a Next.js deploy.

What IndexNow is (and isn't)

Is: an open protocol (indexnow.org). You POST a list of changed URLs, and participating search engines (Bing, Yandex, Seznam, Naver and others) share the notification with each other.
Isn't: a ranking boost, or a way into Google. Google doesn't use IndexNow, so keep your sitemap and Search Console set up for Google.
Isn't: a guarantee. It tells Bing to look. Bing still decides whether and when to index.

On the AI side, be precise about what you're getting. Faster Bing indexing makes it possible for Bing-grounded assistants to find your update sooner. No vendor has published proof that IndexNow by itself makes ChatGPT cite a page faster. It removes one delay. It doesn't create demand.

![Deploy pipeline from content change to IndexNow ping to Bing index to AI answers, with notes on Google, lastmod and Bing's AI Performance report]

Step 1: Create and host your key

Generate a key (8–128 characters, letters, numbers and dashes):

node -e "console.log(crypto.randomUUID().replaceAll('-', ''))"
# e.g. 4f9c2b7e1a0d4c6b8e3f5a7c9d1b2e4f

Host it as a text file at your site root whose name and content are both the key. In Next.js, the simplest way is the public/ folder:

echo -n "4f9c2b7e1a0d4c6b8e3f5a7c9d1b2e4f" > public/4f9c2b7e1a0d4c6b8e3f5a7c9d1b2e4f.txt

Check it's live:

curl https://example.com/4f9c2b7e1a0d4c6b8e3f5a7c9d1b2e4f.txt

The key isn't a secret (it's public by design). It proves you control the host.

Step 2: A small submit helper

// lib/indexnow.ts
const KEY = process.env.INDEXNOW_KEY!;
const HOST = process.env.SITE_HOST!; // e.g. "example.com"

export async function submitToIndexNow(urls: string[]) {
  const urlList = [...new Set(urls)].filter(u => new URL(u).host === HOST);
  if (urlList.length === 0) return { skipped: true };

  // Max 10,000 URLs per request
  for (let i = 0; i < urlList.length; i += 10_000) {
    const res = await fetch("https://api.indexnow.org/indexnow", {
      method: "POST",
      headers: { "Content-Type": "application/json; charset=utf-8" },
      body: JSON.stringify({
        host: HOST,
        key: KEY,
        keyLocation: `https://${HOST}/${KEY}.txt`,
        urlList: urlList.slice(i, i + 10_000),
      }),
    });
    // 200 = OK, 202 = accepted (key validation pending). Anything else: log it.
    if (![200, 202].includes(res.status)) {
      throw new Error(`IndexNow ${res.status}: ${await res.text()}`);
    }
  }
  return { submitted: urlList.length };
}

Option A: Call it when content changes (CMS webhook)

If your content lives in a headless CMS, submit on publish:

// app/api/cms-webhook/route.ts
import { revalidatePath } from "next/cache";
import { submitToIndexNow } from "@/lib/indexnow";

export async function POST(req: Request) {
  if (req.headers.get("x-webhook-secret") !== process.env.CMS_WEBHOOK_SECRET) {
    return new Response("Unauthorized", { status: 401 });
  }
  const { slug } = await req.json();
  const path = `/blog/${slug}`;

  revalidatePath(path);
  await submitToIndexNow([`https://${process.env.SITE_HOST}${path}`]);

  return Response.json({ ok: true });
}

Revalidate first, then ping. If you ping before the new version is live, Bing may fetch the old one.

Option B: Submit changed pages on deploy (Markdown/MDX in git)

If your content is files in the repo, let git tell you what changed:

// scripts/indexnow-changed.mjs
import { execSync } from "node:child_process";

const HOST = process.env.SITE_HOST;
const KEY = process.env.INDEXNOW_KEY;
const base = process.env.BASE_SHA || "HEAD~1";

const changed = execSync(`git diff --name-only ${base} HEAD -- content/`, { encoding: "utf8" })
  .split("\n")
  .filter(f => /\.(md|mdx)$/.test(f));

// Map file paths to URLs. Adjust to your routing.
// content/blog/my-post.mdx -> /blog/my-post
const urls = changed.map(f =>
  `https://${HOST}/` + f.replace(/^content\//, "").replace(/\/index\.mdx?$|\.mdx?$/, "")
);

if (!urls.length) {
  console.log("No content changes.");
  process.exit(0);
}

const res = await fetch("https://api.indexnow.org/indexnow", {
  method: "POST",
  headers: { "Content-Type": "application/json; charset=utf-8" },
  body: JSON.stringify({ host: HOST, key: KEY, keyLocation: `https://${HOST}/${KEY}.txt`, urlList: urls }),
});
console.log(res.status, urls);
if (![200, 202].includes(res.status)) process.exit(1);

# .github/workflows/indexnow.yml
name: IndexNow
on:
  deployment_status:   # fires when Vercel/Netlify report a deploy
jobs:
  ping:
    if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Production'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 2 }
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: node scripts/indexnow-changed.mjs
        env:
          SITE_HOST: example.com
          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}

Waiting for deployment_status: success matters for the same reason as before: only ping once the new content is actually live.

Step 3: Make `lastmod` honest

Your sitemap's <lastmod> is the other freshness signal. Bing and Google both say they use it when it's accurate, and they learn to ignore sites that set every page to "today" on every build.

Use the last git commit that touched the content file:

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { execSync } from "node:child_process";
import { getAllPosts } from "@/lib/posts"; // returns { slug, file }

function gitLastModified(file: string): Date | undefined {
  try {
    const iso = execSync(`git log -1 --format=%cI -- "${file}"`, { encoding: "utf8" }).trim();
    return iso ? new Date(iso) : undefined;
  } catch {
    return undefined;
  }
}

export default function sitemap(): MetadataRoute.Sitemap {
  return getAllPosts().map(({ slug, file }) => ({
    url: `https://example.com/blog/${slug}`,
    lastModified: gitLastModified(file),
  }));
}

Vercel and many CI systems do shallow clones, which make git log wrong for older files. Fetch full history in the build (VERCEL_DEEP_CLONE=true on Vercel, fetch-depth: 0 in Actions), or compute dates at commit time and store them in front matter.

Keep dateModified in your Article JSON-LD consistent with this date, and only bump it for meaningful changes. Fixing a typo isn't an update.

Step 4: Verify in Bing Webmaster Tools

Verify your site in Bing Webmaster Tools (you can import from Google Search Console in a couple of clicks).
IndexNow report: confirms Bing is receiving your submissions and lists any errors.
URL Inspection: check that a specific page is indexed and see which version Bing has.
AI Performance report: in public preview since February 2026, this shows how often your pages are cited in Copilot and Bing's AI answers, and the grounding queries behind those citations. It's CSV export only, sampled, and has no click data, but it's the most direct first-party view of AI citations you can get from any search engine right now.

The grounding queries are the hidden gem: they show the short background searches Copilot actually ran. Compare them with your page headings. If the phrasing doesn't match, you've found a content gap (Part 4 of this series shows how to look for these systematically).

Bing-indexed is step one, not the finish line

Getting indexed only makes you eligible to be cited. To find out whether ChatGPT, Copilot and Perplexity actually cite your pages, and which pages, you have to check the answers themselves. Vista AI's citation tracking does that across engines. Even without a tool, a weekly spreadsheet of 10 important prompts and the URLs each assistant cites will tell you whether your IndexNow work is paying off.

Quick checklist

[ ] Key file live at /{key}.txt
[ ] Ping after publish/deploy, not before
[ ] Sitemap lastmod comes from real content changes
[ ] Site verified in Bing Webmaster Tools, IndexNow report clean
[ ] AI Performance report exported monthly

Next in the series: Should you serve Markdown to AI agents?

Tracking ChatGPT, Perplexity and Claude Referrals Properly: GA4 Regex, Server Logs and the 'Dark AI' Problem

Furqan Khalid — Fri, 23 Oct 2026 06:30:00 +0000

"How much traffic do we get from ChatGPT?" sounds like a one-line question. In practice, the honest answer is "at least X, and probably more."

AI assistants do send visitors: people click the citations in ChatGPT search, Perplexity, Claude, Gemini and Copilot. But that traffic reaches you in at least four different shapes, and default analytics setups only catch some of them.

This post gives you:

A classifier that recognises AI referrals from the referrer and UTM tags
A GA4 custom channel group regex
Express/Next.js middleware to log AI visits server-side
SQL to report on them, plus a way to estimate the part you can't see

The four shapes of AI traffic

Shape	What you see	Example
Referrer present	`document.referrer` / `Referer` header from an AI domain	`https://www.perplexity.ai/`
UTM tagged	AI product adds a UTM to outbound links	`?utm_source=chatgpt.com`
No referrer	Desktop apps, mobile apps, copy-pasted links, strict referrer policies	Lands as Direct
Inside Google	Clicks from AI Overviews / AI Mode	Lands as Organic Search (google)

Shapes 1 and 2 are trackable. Shape 3 is the "dark AI" traffic. Shape 4 is AI-driven, but Google reports it as part of normal Search (Search Console's generative AI performance report, rolled out to all sites in 2026, is the place to look for that).

![Four shapes of AI traffic: referrer present, UTM tagged, no referrer and inside Google, with how trackable each one is]

Image: diagram by the author.

1. A classifier you can reuse everywhere

Keep the rules in one module so the browser, the server and your SQL all agree.

// ai-referrers.js
export const AI_SOURCES = [
  { name: "ChatGPT",    hosts: ["chatgpt.com", "chat.openai.com"],          utm: ["chatgpt.com", "chatgpt"] },
  { name: "Perplexity", hosts: ["perplexity.ai", "www.perplexity.ai"],     utm: ["perplexity", "perplexity.ai"] },
  { name: "Claude",     hosts: ["claude.ai"],                              utm: ["claude.ai", "claude"] },
  { name: "Gemini",     hosts: ["gemini.google.com", "bard.google.com"],   utm: ["gemini", "gemini.google.com"] },
  { name: "Copilot",    hosts: ["copilot.microsoft.com", "copilot.cloud.microsoft"], utm: ["copilot"] },
  { name: "DeepSeek",   hosts: ["chat.deepseek.com"],                      utm: ["deepseek"] },
  { name: "Meta AI",    hosts: ["meta.ai", "www.meta.ai"],                 utm: ["meta.ai"] },
  { name: "Grok",       hosts: ["grok.com"],                               utm: ["grok"] },
];

export function classifyAI({ referrer = "", url = "" }) {
  let refHost = "";
  try { refHost = new URL(referrer).hostname.toLowerCase(); } catch {}

  let utmSource = "";
  try { utmSource = (new URL(url).searchParams.get("utm_source") || "").toLowerCase(); } catch {}

  for (const src of AI_SOURCES) {
    if (src.hosts.some(h => refHost === h || refHost.endsWith("." + h))) {
      return { source: src.name, method: "referrer" };
    }
    if (src.utm.includes(utmSource)) {
      return { source: src.name, method: "utm" };
    }
  }
  return null;
}

classifyAI({ referrer: "https://www.perplexity.ai/", url: "https://example.com/docs" });
// → { source: "Perplexity", method: "referrer" }

classifyAI({ referrer: "", url: "https://example.com/pricing?utm_source=chatgpt.com" });
// → { source: "ChatGPT", method: "utm" }

Domains change. Gemini used to be Bard, Copilot has moved hosts more than once, and new assistants keep launching. Review this list every quarter and treat it as config, not code.

2. GA4: a custom channel group

Some 2026 reports say GA4 added a built-in AI assistant channel. I couldn't find a Google document listing exactly which sources it covers, so I still recommend your own channel group, where you control the rules.

Admin → Data display → Channel groups → Create new channel group → Add new channel named AI Assistants, with the condition:

Session source · matches regex ·

^(.*\.)?(chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|bard\.google\.com|copilot\.microsoft\.com|copilot\.cloud\.microsoft|chat\.deepseek\.com|meta\.ai|grok\.com)$|^(chatgpt|perplexity|claude|gemini|copilot|deepseek|grok)$

Two settings people get wrong:

Don't add a medium = referral condition. UTM-tagged ChatGPT visits often arrive as chatgpt.com / (none) or (not set). Restricting the medium drops them.
Move the channel above "Referral" and "Organic Search". GA4 evaluates channels top to bottom and the first match wins.

Channel groups apply to historical data in reports, which is handy for checking how this year compares with last.

3. Log AI visits server-side

GA4 misses visitors who block analytics, and you can't easily join it with your own data (signups, orders). A tiny middleware fixes both.

Express

// ai-log.js
import { classifyAI } from "./ai-referrers.js";

export function aiReferralLogger(db) {
  return (req, res, next) => {
    if (req.method !== "GET" || !req.accepts("html")) return next();

    const hit = classifyAI({
      referrer: req.get("referer") || "",
      url: `${req.protocol}://${req.get("host")}${req.originalUrl}`,
    });

    if (hit) {
      db.query(
        `INSERT INTO ai_visits (ts, source, method, path, user_agent)
         VALUES (now(), $1, $2, $3, $4)`,
        [hit.source, hit.method, req.path, req.get("user-agent") || ""]
      ).catch(() => {}); // never block the request on analytics
    }
    next();
  };
}

Next.js (App Router)

Middleware can't easily write to a database at the edge, so forward the hit to a route handler or your log pipeline:

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { classifyAI } from "./lib/ai-referrers";

export function middleware(req: NextRequest) {
  const hit = classifyAI({ referrer: req.headers.get("referer") ?? "", url: req.url });
  if (hit) {
    // Structured log line; ship it with your log drain.
    console.log(JSON.stringify({ type: "ai_visit", ...hit, path: req.nextUrl.pathname, ts: Date.now() }));
  }
  return NextResponse.next();
}

export const config = { matcher: ["/((?!_next|api|.*\\..*).*)"] };

Note the referrer only shows the origin (https://chatgpt.com/), not the conversation, and that's how it should be. You'll never know which prompt sent the visit, so don't try to fingerprint it.

4. Report on it

-- Weekly AI visits by source
SELECT date_trunc('week', ts) AS week, source, count(*) AS visits
FROM ai_visits
GROUP BY 1, 2
ORDER BY 1 DESC, 3 DESC;

-- Which pages do AI assistants send people to?
SELECT path, source, count(*) AS visits
FROM ai_visits
WHERE ts > now() - interval '30 days'
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT 25;

The second query is the useful one. AI assistants tend to send people to a different set of pages from Google: deep docs pages, comparison pages and pricing are common. That tells you which content is getting cited.

5. Estimating the dark part

You can't classify visits with no referrer, but you can spot when Direct traffic behaves like AI traffic:

Deep-page Direct landings. Real "direct" visits (typed URL, bookmark) mostly land on the homepage. A spike in Direct visits to /docs/configuring-webhooks usually means someone shared or cited the link, and AI apps are a common source.
Correlate with known AI visits. If a page's referrer-tagged ChatGPT visits double and its deep-page Direct visits rise in the same week, some of that Direct is probably AI too.
Ask. A "How did you hear about us?" field on signup with an "AI assistant (ChatGPT, Claude…)" option is low-tech and surprisingly informative.

-- Deep-page "direct" sessions: a rough proxy for unattributed shares/citations
SELECT landing_path, count(*) AS sessions
FROM sessions
WHERE channel = 'Direct'
  AND landing_path NOT IN ('/', '/login', '/app')
  AND ts > now() - interval '30 days'
GROUP BY 1
ORDER BY 2 DESC
LIMIT 25;

Report AI traffic as "tracked AI visits (floor)" and leave it at that. Inventing a multiplier for the dark part won't survive the first question in a meeting.

Traffic is the lagging indicator

Referral tracking tells you what happened after an assistant cited you. It doesn't tell you how often you were mentioned without a click, or how often a competitor was recommended instead. Those are the leading indicators. We track them in Vista AI's brand visibility dashboard, which samples real prompts across the major assistants. Whatever you use, put both views next to each other: visibility going up while referrals stay flat usually means you're being mentioned but not linked.

Did I miss an AI source that sends you traffic? Post the referrer host in the comments and I'll add it to the list.

Next in the series: ChatGPT can't cite what Bing hasn't indexed: IndexNow in Next.js.

Simulate Google's Query Fan-Out to Find Your Content Gaps (Python)

Furqan Khalid — Thu, 22 Oct 2026 06:00:00 +0000

When someone asks Google's AI Mode "what's the best way to host a Next.js app for a small team?", Google doesn't run one search. It splits the question into sub-questions and runs those in parallel. Google calls this query fan-out. Its docs say AI Overviews and AI Mode "may use a 'query fan-out' technique, issuing multiple related searches across subtopics and data sources, to develop a response" (Search Central).

Microsoft lets you see this happening. Bing Webmaster Tools' AI Performance report lists grounding queries, the background queries Copilot generated before citing your pages. They're shorter and more specific than what users type.

That changes what "covering a topic" means. Your page isn't competing for one query. It's competing for 5–15 machine-generated ones, and every sub-question you skip is a retrieval you hand to someone else.

You can't see Google's real fan-out. But you can approximate it closely enough to find gaps. Here's a script that does exactly that.

What we're building

seed question ──► LLM generates likely sub-queries
                             │
your page ──► split into passages ──► embed
                             │
              for each sub-query: best-matching passage + score
                             │
                     coverage report (gaps first)

Two parts:

Fan-out: an LLM writes the sub-queries a search system would plausibly issue.
Coverage: an embedding model scores how well each sub-query is answered by some passage on your page.

Image: diagram by the author.

Setup

pip install anthropic pydantic sentence-transformers requests beautifulsoup4
export ANTHROPIC_API_KEY=...

I'm using Claude for generation and a small open-source embedding model for scoring. Both are swappable.

Step 1: Generate the fan-out

Structured output keeps this clean: we ask for a typed list and get a validated Python object back, with no regex over model text.

# fanout.py
from typing import Literal
import anthropic
from pydantic import BaseModel

class SubQuery(BaseModel):
    query: str
    intent: Literal["definition", "comparison", "how-to", "pricing", "constraint", "alternative", "follow-up"]

class FanOut(BaseModel):
    sub_queries: list[SubQuery]

client = anthropic.Anthropic()

PROMPT = """You are simulating the query fan-out step of an AI search engine.
Given a user's question, list the 12 short web searches the engine would most
likely run in parallel to build a complete, trustworthy answer.

Cover different facets: definitions, comparisons, pricing, constraints
(size, budget, region, stack), how-to steps, alternatives, and the follow-up a
careful user would ask next. Write each one the way a search engine would
(short, keyword-like), not as a full sentence.

User question: {question}"""

def fan_out(question: str) -> list[SubQuery]:
    response = client.messages.parse(
        model="claude-opus-5-5",
        max_tokens=16000,
        output_config={"effort": "low"},  # this is a simple task; keep it cheap
        messages=[{"role": "user", "content": PROMPT.format(question=question)}],
        output_format=FanOut,
    )
    return response.parsed_output.sub_queries

Run it on a real question:

for sq in fan_out("best way to host a Next.js app for a small team"):
    print(f"[{sq.intent:11}] {sq.query}")

[comparison ] vercel vs netlify vs cloudflare pages nextjs
[pricing    ] vercel pro pricing per seat 2026
[constraint ] nextjs self hosting docker small team
[how-to     ] deploy nextjs app router to cloudflare
[constraint ] nextjs ISR support hosting providers
[alternative] nextjs hosting without vercel
[follow-up  ] nextjs hosting cold starts serverless
...

Generated output varies between runs. That's fine. Run it 3–5 times and merge the results, because real fan-out isn't deterministic either.

Step 2: Split your page into passages

AI systems retrieve passages, not whole pages. We'll split by headings and paragraphs, keeping each heading attached to its text so the passage still makes sense on its own.

# passages.py
import requests
from bs4 import BeautifulSoup

def page_passages(url: str, min_words: int = 25) -> list[str]:
    html = requests.get(url, timeout=20, headers={"User-Agent": "fanout-audit/1.0"}).text
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
        tag.decompose()

    root = soup.find("main") or soup.find("article") or soup.body
    passages, heading = [], ""
    for el in root.find_all(["h1", "h2", "h3", "p", "li", "td"]):
        text = " ".join(el.get_text(" ", strip=True).split())
        if el.name in ("h1", "h2", "h3"):
            heading = text
            continue
        if len(text.split()) >= min_words:
            passages.append(f"{heading}: {text}" if heading else text)
    return passages

Note that this reads the raw HTML, the way many AI fetchers do. If your content only appears after JavaScript runs, this returns almost nothing, and that's a finding in its own right (see Part 9).

Step 3: Score coverage with embeddings

# coverage.py
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

def coverage(sub_queries, passages):
    q_emb = model.encode([sq.query for sq in sub_queries], normalize_embeddings=True)
    p_emb = model.encode(passages, normalize_embeddings=True)
    sims = util.cos_sim(q_emb, p_emb)  # [n_queries, n_passages]

    rows = []
    for i, sq in enumerate(sub_queries):
        best = int(sims[i].argmax())
        rows.append({
            "query": sq.query,
            "intent": sq.intent,
            "score": float(sims[i][best]),
            "best_passage": passages[best][:110] + "…",
        })
    return sorted(rows, key=lambda r: r["score"])  # gaps first

Step 4: Put it together

# main.py
import sys
from fanout import fan_out
from passages import page_passages
from coverage import coverage

question, url = sys.argv[1], sys.argv[2]

# Merge a few runs to smooth out randomness.
seen, subs = set(), []
for _ in range(3):
    for sq in fan_out(question):
        key = sq.query.lower()
        if key not in seen:
            seen.add(key)
            subs.append(sq)

passages = page_passages(url)
print(f"{len(subs)} sub-queries · {len(passages)} passages\n")

for r in coverage(subs, passages):
    flag = "GAP " if r["score"] < 0.55 else "weak" if r["score"] < 0.68 else "ok  "
    print(f"{flag} {r['score']:.2f}  [{r['intent']}] {r['query']}")
    if flag != "ok  ":
        print(f"           closest: {r['best_passage']}")

python main.py "best way to host a Next.js app for a small team" https://your-site.com/blog/nextjs-hosting

(Illustrative output. Your numbers will differ.)

33 sub-queries · 41 passages

GAP  0.41  [pricing] vercel pro pricing per seat 2026
           closest: Why we moved: Our bill grew faster than our traffic…
GAP  0.47  [constraint] nextjs ISR support hosting providers
           closest: Deploying to Cloudflare: The adapter handles most…
weak 0.61  [follow-up] nextjs hosting cold starts serverless
...
ok   0.79  [comparison] vercel vs netlify vs cloudflare pages nextjs

Reading the results

The thresholds are rough. 0.55 and 0.68 work reasonably well for bge-small on English technical content. Calibrate them by checking a few passages yourself: find the score below which the "closest passage" clearly doesn't answer the question.
Gaps aren't automatically to-dos. "Vercel pricing per seat" may not belong on your page. But if your page is "the best way to host Next.js for a small team," pricing per seat is exactly what that reader needs. Add a short, specific passage with real numbers and a date.
Fix gaps with self-contained passages. Lead with the answer, name the entity ("Vercel Pro costs…", not "It costs…"), and include the number or constraint. A passage that makes sense without the paragraph above it can be lifted into an answer.
Don't spin up a page per sub-query. That's the doorway-page pattern, and Google's spam policies now explicitly cover AI features too. One strong page that answers the cluster beats twelve thin ones.

Caveats (please read)

This approximates fan-out. Google hasn't published how many sub-queries it issues or how it picks them. Use the output as a gap-finding tool, not as a model of Google.
Embedding similarity measures topical match, not correctness or quality. A passage can score 0.8 and still be out of date.
Coverage is necessary but not sufficient. Pages also need to be indexed, crawlable, and trusted enough to be picked over competitors.

If you want a reality check, compare your generated sub-queries with Bing's real grounding queries (Bing Webmaster Tools → AI Performance, CSV export). Matching phrasing there is the closest thing to ground truth you'll get today.

Where this fits

Coverage tells you whether your page can answer the sub-questions. It doesn't tell you whether AI engines actually do pick you. For that you need to run the real prompts across ChatGPT, Perplexity, Gemini and others over time and track who gets mentioned. We built Vista AI's AI rank tracker for that part. The script above is the free, local half of the loop: find gaps, fix them, then measure.

The full script is ~120 lines. If you adapt it (different embedding model, multilingual, or batch mode over a sitemap), share it in the comments.

Next in the series: Track ChatGPT, Perplexity and Claude referrals properly.

Your Next Visitor Is an AI Agent: Semantic HTML Is SEO Again

Furqan Khalid — Wed, 21 Oct 2026 06:00:00 +0000

For twenty years, "write semantic HTML" was advice people agreed with and then ignored. A <div onclick> worked fine for mouse users, Googlebot didn't care much, and the screen-reader users it failed were easy to forget.

That changed in 2026. Agentic browsers such as ChatGPT Atlas, Perplexity Comet and Chrome's auto-browse now act on pages for users: comparing products, filling forms, booking things. They have to decide what each element on your page is and what it does. The cheapest, most reliable signal they have is the same one a screen reader uses: the accessibility tree.

So the boring advice is now traffic advice. If an agent can't tell your "Book now" div is a button, it books somewhere else.

How agents see a page

Browser agents mix three inputs:

The accessibility tree: roles (button, link, textbox), accessible names ("Search", "Party size") and states (checked, disabled, expanded).
The DOM and text: for content and context.
Screenshots: a vision model looks at the rendered page, which is slower and more expensive and is usually the fallback.

Every vendor balances these differently, and none of them publish exact details. But the accessibility tree is the compact, structured option, which is why agent frameworks lean on it. If your tree is clean, agents work faster and make fewer mistakes. If it's a pile of unnamed generic nodes, they fall back to guessing from pixels.

Image: diagram by the author.

See your page the way an agent does

Playwright can print a page's accessibility tree as a YAML-like "ARIA snapshot". It's the fastest way I know to understand what an agent is working with.

npm i -D playwright
npx playwright install chromium

// agent-view.mjs
import { chromium } from "playwright";

const url = process.argv[2] ?? "https://example.com";
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: "networkidle" });

// The accessibility tree for the main content (fall back to body).
const root = (await page.locator("main").count()) ? page.locator("main") : page.locator("body");
console.log(await root.ariaSnapshot());

await browser.close();

node agent-view.mjs https://your-site.com/booking

Here's a (real-world-style) before and after from a booking widget:

# BEFORE
- generic:
  - generic: Book a table
  - generic: "2"
  - generic: Tonight
  - generic: Continue

# AFTER
- heading "Book a table" [level=2]
- combobox "Party size": "2"
- textbox "Date": 2026-10-14
- button "Find available times"

Same visual design. The first version gives an agent four meaningless boxes. The second tells it exactly what to do.

The 8 fixes that matter most

1. Use real buttons and links

<!-- ❌ invisible as an action -->
<div class="btn" onclick="checkout()">Checkout</div>

<!-- ✅ -->
<button type="button" onclick="checkout()">Checkout</button>

Rule of thumb: navigates to a URL → <a href>. Does something on this page → <button>.

2. Label every input

<!-- ❌ placeholder isn't a reliable label -->
<input type="email" placeholder="Email">

<!-- ✅ -->
<label for="email">Work email</label>
<input id="email" type="email" name="email" autocomplete="email" required>

autocomplete tokens are a bonus: they tell any agent precisely what a field expects.

3. Give icon buttons names

<button aria-label="Add to wishlist"><svg aria-hidden="true">…</svg></button>

An unnamed icon button shows up as button with no name. An agent can't pick it with any confidence.

4. Use native form controls (or full ARIA patterns)

Custom dropdowns built from <div>s are the most common thing agents fail on. Use <select> where you can. If you must build a custom combobox, follow the ARIA Authoring Practices pattern completely, states included.

5. Landmarks and headings

One <main>, a <nav>, sensible <h1>–<h3> order. Agents use these to jump straight to the content, just like screen-reader users do.

6. Put errors where they can be read

<input id="zip" aria-describedby="zip-error" aria-invalid="true">
<p id="zip-error">Enter a 5-digit ZIP code.</p>

A red border alone means nothing to an agent. A linked error message tells it how to fix the input and try again.

7. Don't hide key actions behind hover

Hover-only menus and "appears on mouseover" buttons are hard for agents and touch users alike. Make primary actions visible and focusable.

8. Avoid layout shifts after load

Agents act on a snapshot. If your cookie banner or promo bar pushes the button down 80px after the snapshot, the click lands on the wrong element. Reserving space for late content (which you should do for CLS anyway) fixes this too.

Put it in CI

Make the agent view a test, so nobody quietly turns your checkout button back into a div:

// tests/agent-ready.spec.ts
import { test, expect } from "@playwright/test";

test("booking flow is operable from the accessibility tree", async ({ page }) => {
  await page.goto("/booking");

  // These locators only pass if role + accessible name are correct.
  await page.getByRole("combobox", { name: "Party size" }).selectOption("4");
  await page.getByRole("textbox", { name: "Date" }).fill("2026-10-20");
  await page.getByRole("button", { name: "Find available times" }).click();

  await expect(page.getByRole("heading", { name: /available times/i })).toBeVisible();
});

If a test written with getByRole can complete your key flow, an agent has a good chance too. Add @axe-core/playwright for broader accessibility checks.

How this connects to search

Agents don't replace search. They usually start with it: they search, open the top results, then act on the pages. So the pipeline looks like this:

Get found: classic SEO and AI visibility (the rest of this series).
Get understood: content in the HTML, clear headings and structured data.
Get used: an accessibility tree an agent can operate (this post) and optionally WebMCP tools (Part 2).

Most teams only think about step 1. Step 3 is where you lose the conversion after the agent already picked you.

And there's a nice side effect: every fix in this post makes your site better for people using assistive technology. That alone was always reason enough.

Try it: run agent-view.mjs on your highest-value page (signup, checkout, booking, contact) and paste the snapshot into the comments. I'm curious how many generic nodes people find.

Next in the series: Simulate query fan-out to find your content gaps (Python).

WebMCP: Make Your Site Callable by AI Agents (With Working Code)

Furqan Khalid — Tue, 20 Oct 2026 06:00:00 +0000

Right now, when an AI agent books a table on your site, it does it the hard way. It reads the page, guesses which input is "party size", types into it, finds something that looks like a submit button, clicks, and hopes.

WebMCP is a proposal to skip the guessing. Your page tells the browser: "here are the actions I support, here's the JSON Schema for each one, call this function." The agent calls the function. No DOM archaeology.

Chrome's team announced an early preview in February 2026, the spec is being developed at the W3C with Microsoft involved, and Chrome's docs were last updated in May 2026 (Chrome for Developers). It's still experimental. That makes now a good time to prototype it and a bad time to bet a launch on it.

⚠️ Version note (October 2026): early demos used navigator.modelContext. Chrome's current docs use document.modelContext. Method names and options have shifted between Chrome versions. Check the current docs before copying anything, including from this post.

The mental model

If you've used MCP (Model Context Protocol) on the server side, WebMCP is the same idea moved into the page:

	Server MCP	WebMCP
Where tools live	A separate MCP server process	JavaScript in your page
Who calls them	An AI app connected to your server	An agent running in the user's browser
Auth	You build it (OAuth, API keys)	The user's existing session and cookies
Best for	Back-end integrations	Actions a logged-in user takes on your site

The second-to-last row is the important one. A WebMCP tool runs as the user, in their tab. You don't need a new auth system. You also inherit every security problem that comes with that, which we'll get to.

Image: diagram by the author.

The API surface

From Chrome's imperative API docs, document.modelContext gives you:

registerTool(tool, options?): add a tool. Options include an AbortSignal (to unregister) and exposedTo (origins allowed to see it).
getTools() and executeTool(): used by the agent side, and handy for testing.
A toolchange event when the available tools change.

A tool definition has:

name: a stable identifier like search_products
description: plain-language text the model reads to decide when to call it
inputSchema: JSON Schema for the arguments
annotations (optional): hints such as readOnlyHint and consequentialHint
execute(input, { signal }): your function

There's also a declarative flavour that annotates existing <form> elements so you don't write JS at all. This post sticks to the imperative API because it's easier to reason about.

Example: a store search and add-to-cart

Here's a realistic pair of tools for an e-commerce page: one read-only, one that changes state.

// webmcp-tools.js
// Progressive enhancement: do nothing if the browser doesn't support WebMCP.
const mc = document.modelContext ?? navigator.modelContext; // older previews used navigator

if (mc?.registerTool) {
  const controller = new AbortController();

  mc.registerTool(
    {
      name: "search_products",
      description:
        "Search this store's catalogue. Returns up to 10 products with id, name, price in USD and stock status.",
      inputSchema: {
        type: "object",
        properties: {
          query: { type: "string", description: "Free-text search, e.g. 'trail running shoes'" },
          maxPrice: { type: "number", description: "Optional upper price limit in USD" },
        },
        required: ["query"],
      },
      annotations: { readOnlyHint: true },
      async execute({ query, maxPrice }, { signal }) {
        const params = new URLSearchParams({ q: query });
        if (maxPrice) params.set("max_price", String(maxPrice));

        const res = await fetch(`/api/search?${params}`, { signal });
        if (!res.ok) return `Search failed with status ${res.status}.`;

        const items = await res.json();
        return JSON.stringify(
          items.slice(0, 10).map(({ id, name, price, inStock }) => ({ id, name, price, inStock }))
        );
      },
    },
    { signal: controller.signal }
  );

  mc.registerTool(
    {
      name: "add_to_cart",
      description:
        "Add a product to the signed-in user's cart. Use a product id returned by search_products.",
      inputSchema: {
        type: "object",
        properties: {
          productId: { type: "string" },
          quantity: { type: "integer", minimum: 1, maximum: 10 },
        },
        required: ["productId"],
      },
      annotations: { consequentialHint: true },
      async execute({ productId, quantity = 1 }, { signal }) {
        const res = await fetch("/api/cart", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ productId, quantity }),
          signal,
        });
        if (!res.ok) return `Could not add item (status ${res.status}).`;

        // Keep the visible UI in sync so the human sees what the agent did.
        document.dispatchEvent(new CustomEvent("cart:updated"));
        return `Added ${quantity} × ${productId} to the cart.`;
      },
    },
    { signal: controller.signal }
  );

  // Unregister when the SPA route changes, e.g. controller.abort();
}

A few choices worth copying:

Feature-detect and exit quietly. Most browsers won't have this API for a while. The page must work without it.
Reuse your existing API routes. The tool is a thin wrapper. Your server still validates everything, because the server can't tell whether a human or an agent clicked.
Return small, structured results. Agents pay for every token you return. Don't send the whole product object.
Update the UI. If the agent changes state, the human watching the tab should see it.

Testing it locally

Chrome exposes WebMCP behind an origin trial and/or flags, depending on version. Check the Chrome Status entry linked from the docs for your channel. Once it's on, you can poke your tools from DevTools:

const tools = await document.modelContext.getTools();
console.table(tools.map(t => ({ name: t.name, description: t.description })));

const result = await document.modelContext.executeTool(
  tools.find(t => t.name === "search_products"),
  { query: "trail shoes", maxPrice: 120 }
);
console.log(result);

Security: treat every tool like a public API endpoint

WebMCP doesn't create new permissions. It makes existing ones easier for an automated caller to use. So:

Server-side authorization stays the source of truth. Never put "is this user allowed?" logic only in execute.
Mark consequential tools. Anything that spends money, sends messages, or deletes data should carry consequentialHint so the agent and browser can ask the user to confirm. Don't count on that, though. Add your own confirmation step for irreversible actions.
Don't trust strings that come back in. The agent's arguments are model output, which may have been steered by a prompt injection on another page. Validate against your schema on the server.
Limit cross-origin exposure. Only use exposedTo for origins you control.
Rate-limit. An agent can call add_to_cart 500 times faster than a human can click.

The W3C repo has open issues about tool-registration security (for example, an early provideContext() call that could clear and replace another script's tools). Third-party scripts on your page can register tools too. Audit what your tag manager loads.

So is this SEO?

Not directly. WebMCP doesn't affect crawling, indexing or ranking, and Google hasn't said anything to suggest it will. Discovery still happens the old way: an agent has to find your page through search or a link before it can call your tools. Any blog post claiming "WebMCP is a ranking factor" is guessing.

Where it does matter is the step after discovery. If an agent is choosing between two stores to complete a task, the one with a clean, callable search_products is the one that finishes the job, and probably the one that gets used again. That's conversion optimization for a new kind of user.

What I'd do this quarter

Fix the basics first: semantic forms and buttons (next post in this series). They help every agent today, with or without WebMCP.
Pick one or two read-only tools (search, availability, store finder) and prototype them behind a feature flag.
Log tool calls separately from normal API calls so you can see whether any agent actually uses them.
Wait on write tools until the spec and the browser confirmation UX settle down.

Next in the series: Your next visitor is an agent: semantic HTML is SEO again.

SEO in Late 2026: What Actually Changed for Developers (and What Didn't)

Furqan Khalid — Mon, 19 Oct 2026 06:00:00 +0000

Every few months somebody posts "SEO is dead" and somebody else replies "SEO is just GEO now." Neither is useful if you're the person who has to change the code.

So this is the developer version. These are the changes from 2026 that touch your HTML, server, analytics or build pipeline, plus the things that didn't change, which matter just as much because they stop you wasting a sprint.

This is the first post in a 10-part series. Each of the later posts takes one item from this list and goes deep, with code.

Change	What it means for you	Deep dive
AI answers do the searching for users (query fan-out)	One page has to answer the sub-questions too, not just the head query	Part 4
Agents now browse and click for users	Semantic HTML and the accessibility tree are your agent API	Parts 2, 3
ChatGPT and Copilot lean on Bing	Bing indexing speed is now an AI-visibility problem	Part 6
AI referrals are real but badly attributed	You need your own tracking, not just GA4 defaults	Part 5
Markdown for agents went mainstream (Cloudflare)	Useful for docs and agents. Not a ranking trick	Part 7
Google's spam policies explicitly cover AI answers	Hidden "instructions for LLMs" are now spam	Part 8
Client-side-only rendering hurts more than ever	Many AI fetchers don't run your JavaScript	Part 9
AI assistants recommend tools by consensus	Third-party mentions beat your own landing page	Part 10

Image: diagram by the author.

1. Search engines now search for the user

Google's documentation says that AI Overviews and AI Mode "may use a 'query fan-out' technique, issuing multiple related searches across subtopics and data sources, to develop a response" (Google Search Central: AI features).

Microsoft is even more explicit. Bing Webmaster Tools' AI Performance report, which entered public preview in February 2026, shows grounding queries: the short, background queries Copilot generates to fetch pages, which are not what the user typed (Search Influence's analysis of the report).

What it means in code terms: your page isn't matched against one query any more. It's matched against a handful of machine-generated sub-queries. A page that answers the head term but skips the obvious follow-ups ("pricing", "vs X", "does it work with Y") loses those retrievals to someone else's page.

What didn't change: to be used in an AI Overview or AI Mode answer, a page still has to be indexed and eligible to show a snippet. Google says there are no extra technical requirements (same doc).

2. Your next visitor might be an agent, not a person

Agentic browsers (ChatGPT Atlas, Perplexity Comet, Chrome's auto-browse) shipped to real users in 2025–2026. They read pages, fill forms and click buttons for the user.

On top of that, Chrome shipped an early preview of WebMCP in February 2026: a browser API that lets a page register "tools" an agent can call directly instead of poking at the DOM (Chrome for Developers: WebMCP imperative API).

What it means: two layers of "agent readiness".

Layer 1 (do it now): semantic HTML, labelled form fields, real <button>s. Agents mostly read the accessibility tree, so the work you'd do for screen readers is the same work. → Part 3
Layer 2 (experiment): expose your key actions as WebMCP tools. → Part 2

3. Bing quietly became an AI-visibility dependency

ChatGPT's search and Microsoft Copilot both draw on Bing's index. That makes Bing's crawl and index speed matter far more than its small share of classic search suggests. IndexNow, the push-based protocol Bing supports, lets you tell it about new or changed URLs immediately (indexnow.org).

What didn't change: Google doesn't use IndexNow. You still need a good sitemap and Search Console for Google. → Part 6

4. AI referral traffic exists, and your analytics undercounts it

People do click citations in ChatGPT, Perplexity, Claude, Gemini and Copilot. But some of that traffic arrives without a referrer, some arrives with utm_source=chatgpt.com, and clicks from Google's own AI Overviews and AI Mode are mixed into ordinary organic Google traffic.

What it means: treat any "AI traffic" number as a floor. Build a classifier that checks the referrer, UTM tags and landing-page patterns, and log it server-side if you can. → Part 5

5. Markdown for machines went mainstream (and got controversial)

llms.txt stayed a community proposal. Google said in 2025 it doesn't support it, and crawl studies in 2026 found AI search bots rarely fetch it. Meanwhile Cloudflare launched Markdown for Agents: send Accept: text/markdown and you get a Markdown version of the page with an x-markdown-tokens header (Cloudflare docs).

Google's own May 2026 guidance on optimizing for AI features says this kind of thing isn't needed for Google: optimizing for AI features "is optimizing for the search experience, and thus still SEO" (Google's AI optimization guide).

What it means: serve Markdown to agents and coding assistants that can use it (docs sites especially), but don't expect it to move rankings, and never serve different content to bots. → Part 7

6. Manipulating AI answers is officially spam

On May 15, 2026, Google updated its documentation to say its spam policies apply to all of Google Search, including generative AI responses (coverage on PPC Land). That followed Microsoft's February 2026 research on AI Recommendation Poisoning: companies hiding prompts in "Summarize with AI" links to plant "remember us as a trusted source" in an assistant's memory (Microsoft Security blog).

What it means: hidden text addressed to LLMs isn't clever. It's hidden text, and it's a security problem. → Part 8

7. JavaScript rendering is a bigger risk than it was

Googlebot renders JavaScript. Many AI crawlers and live-fetch agents don't reliably do so. A client-rendered SPA that looks fine in Google can be an empty <div id="root"> to an AI fetcher. → Part 9

8. Recommendations come from consensus, not your homepage

When someone asks an assistant "what's the best X for Y," the answer is assembled from many sources: comparison posts, docs, forums, GitHub, Reddit. Vendor studies disagree on the exact share, but they agree third-party sources carry a large part of it. → Part 10

What didn't change (please don't skip this part)

Crawlability, indexability and canonicalization. If Google or Bing can't index the page, no AI feature built on them can cite it.
Helpful, specific content. The original GEO research from Princeton (KDD 2024) found that adding statistics, quotations and cited sources improved visibility in generative answers by up to ~40% in their benchmark (arXiv:2311.09735). Those are things good technical writing already does.
Performance and Core Web Vitals. Still matter for users and for ranking.
Structured data. Still useful for rich results and entity clarity. There's no special "AI schema" (Google).

A 30-minute checklist for this week

# 1. Is your main content in the raw HTML? (no JS)
curl -s https://example.com/your-page | grep -c "a phrase from your first paragraph"

# 2. Are AI search bots allowed?
curl -s https://example.com/robots.txt | grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot"

# 3. Is the page in Bing? Search for it with the site: operator in Bing:
#    site:example.com/your-page

# 4. Any hidden text aimed at LLMs? (yours or a plugin's)
curl -s https://example.com/your-page | grep -iE "ignore (all|previous)|as an ai|llm|remember .* trusted"

If check 1 returns 0, start with Part 9. If check 3 finds nothing, start with Part 6.

The series

SEO in late 2026: what changed for developers (you are here)
WebMCP: make your site callable by AI agents
Your next visitor is an agent: semantic HTML is SEO again
Simulate query fan-out to find your content gaps (Python)
Track ChatGPT, Perplexity and Claude referrals properly
ChatGPT can't cite what Bing hasn't indexed: IndexNow in Next.js
Should you serve Markdown to AI agents?
Hidden prompts in your HTML are now spam
Is your React app invisible to AI crawlers?
How AI assistants decide which dev tool to recommend

What change hit your site hardest this year? I'm collecting real examples for the later posts. Drop yours in the comments.

Tracking Traffic From ChatGPT, Perplexity and Gemini in GA4 and BigQuery

Furqan Khalid — Fri, 16 Oct 2026 06:00:00 +0000

AI visibility metrics tell you how often assistants mention and cite you. Sooner or later someone asks the follow-up: does any of that turn into traffic?

Part of the answer is already in your analytics, mixed into "Referral" alongside every other website. This post separates it out:

a custom channel group in GA4 for AI assistants,
a BigQuery query over the GA4 export that pulls AI-referred sessions and landing pages,
a join against citation data to compare "cited" with "clicked".

It also covers the large share of AI-influenced traffic you can't see this way, because that limitation should shape how you report the numbers.

How AI traffic shows up

When a user clicks a source link inside an AI answer, the visit arrives in one of three ways:

Signal	Example	Notes
Referrer header	`https://chatgpt.com/`, `https://www.perplexity.ai/`	Most common for web clients.
UTM parameter	`?utm_source=chatgpt.com`	ChatGPT adds this to many outbound links.
Nothing	(direct)	Mobile apps, privacy settings and copy-pasted URLs often strip the referrer.

Referrer hosts to match (keep this list in one place, it changes):

chatgpt.com, chat.openai.com
perplexity.ai, www.perplexity.ai
gemini.google.com
copilot.microsoft.com
claude.ai
chat.deepseek.com

Google's AI Overviews and AI Mode clicks generally come through as ordinary Google organic traffic, so they won't appear here.

Step 1: A custom channel group in GA4

In Admin → Data display → Channel groups, copy the default group and add a channel called AI assistants with this condition:

Source matches regex:

^(chatgpt\.com|chat\.openai\.com|(www\.)?perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai|chat\.deepseek\.com)$

Then drag the new channel above "Referral". Channels are evaluated in order, and the first match wins, so if Referral sits above it, AI traffic is still caught by Referral.

Because the rule matches source, it catches both referrer-based and utm_source=chatgpt.com visits. Custom channel groups also apply retroactively in GA4 reports, so you get history from day one.

Step 2: Raw AI sessions from the BigQuery export

The UI is fine for trends. For landing pages, joins and custom logic, use the BigQuery export. This query finds the first page_view of each session that came from an AI assistant, using either the referrer or the UTM source:

DECLARE ai_re STRING DEFAULT
  r'(chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai|chat\.deepseek\.com)';

WITH pv AS (
  SELECT
    PARSE_DATE('%Y%m%d', event_date)                                   AS day,
    user_pseudo_id,
    (SELECT value.int_value    FROM UNNEST(event_params) WHERE key = 'ga_session_id') AS session_id,
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'page_location') AS page_location,
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'page_referrer') AS page_referrer,
    collected_traffic_source.manual_source                              AS utm_source,
    event_timestamp
  FROM `my-project.analytics_123456789.events_*`
  WHERE _TABLE_SUFFIX BETWEEN FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 28 DAY))
                          AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
    AND event_name = 'page_view'
),
landing AS (
  SELECT *,
    ROW_NUMBER() OVER (PARTITION BY user_pseudo_id, session_id ORDER BY event_timestamp) AS rn
  FROM pv
)
SELECT
  day,
  COALESCE(REGEXP_EXTRACT(page_referrer, ai_re), REGEXP_EXTRACT(utm_source, ai_re)) AS assistant,
  REGEXP_REPLACE(REGEXP_EXTRACT(page_location, r'^https?://[^/]+([^?#]*)'), r'/$', '') AS landing_path,
  COUNT(*) AS sessions
FROM landing
WHERE rn = 1
  AND (REGEXP_CONTAINS(IFNULL(page_referrer, ''), ai_re)
       OR REGEXP_CONTAINS(IFNULL(utm_source, ''), ai_re))
GROUP BY day, assistant, landing_path
ORDER BY sessions DESC;

Notes:

collected_traffic_source exists in exports from mid-2023 onward. For older data, rely on the referrer only.
REGEXP_EXTRACT returns the matched host, so www.perplexity.ai and perplexity.ai both come out as perplexity.ai.
Filtering on _TABLE_SUFFIX keeps the scan, and the bill, to the date range you need.
If you've also enabled the intraday export, events_* matches events_intraday_* tables as well, so the most recent day can be double-counted. Use events_2* to match only the daily tables.

Step 3: Join "cited" with "clicked"

If you're collecting citations (see the earlier posts in this series), you have, per page, how often AI answers cited it. Now you have, per page, how many sessions AI assistants sent. Put them side by side:

WITH cited AS (
  SELECT landing_path, SUM(citations) AS citations
  FROM `my-project.ai_visibility.page_citations_28d`   -- your citation data, by path
  GROUP BY landing_path
),
clicked AS (
  SELECT landing_path, SUM(sessions) AS ai_sessions
  FROM `my-project.ai_visibility.ai_sessions_28d`      -- the Step 2 result, saved
  GROUP BY landing_path
)
SELECT
  COALESCE(c.landing_path, k.landing_path) AS landing_path,
  IFNULL(c.citations, 0)    AS citations,
  IFNULL(k.ai_sessions, 0)  AS ai_sessions,
  SAFE_DIVIDE(k.ai_sessions, c.citations) AS sessions_per_citation
FROM cited c
FULL OUTER JOIN clicked k USING (landing_path)
ORDER BY citations DESC;

The four quadrants are the actionable part:

High citations, high sessions: working. Protect these pages and keep them current.
High citations, low sessions: the AI uses your page as a source but users don't need to click, because the answer is already complete. Common for definitions and simple facts. Consider whether the page offers a reason to visit (a tool, a template, deeper data).
Low citations, high sessions: traffic you can't explain from your tracked prompts. Your prompt set is missing questions people actually ask. Add them.
Low, low: most pages. Fine, unless they're pages you need to win.

What this can't measure

Be upfront with stakeholders about three blind spots:

Referrer loss. App clicks and privacy settings drop referrers, and those visits land in Direct. Measured AI traffic is a floor, not a total.
Zero-click influence. Many users read the AI's recommendation, then search your brand name on Google or type your URL. That shows up as branded search or direct traffic, credited to anything but AI. A rise in branded search alongside rising AI visibility is circumstantial evidence. Treat it as that.
AI Overviews and AI Mode are folded into Google organic in GA4.

This is why referral traffic alone underrates AI search, and why it needs to sit next to visibility metrics rather than replace them.

One view for both

The useful report puts visibility (mentions, citations, share of voice) and outcome (AI sessions, conversions) in front of the same people, at the same cadence.

Vista AI's Brand Visibility Dashboard covers the visibility side: visibility by engine and region, share of voice against competitors, sentiment, per-query rankings and 30/90/365-day trends, with weekly email reports. It also supports multiple brands with separate query sets, which helps agencies report per client. Pair it with the GA4 channel above and you have both halves: how often AI recommends you, and how often that recommendation becomes a visit.

The SQL Behind an AI Visibility Dashboard: Five Metrics and Their Traps

Furqan Khalid — Thu, 15 Oct 2026 06:00:00 +0000

Once you're collecting LLM answers daily and scoring them for brand mentions, someone will ask for a dashboard. That's when a pile of good data turns into misleading charts.

The traps are specific to this kind of data. Answers are non-deterministic, sample sizes per prompt are small, engines differ wildly, and the competitor set changes. This post writes the five metrics worth putting on a dashboard as SQL (SQLite dialect, easy to port), with the guards that keep each one honest.

The schema is the one from earlier in this series: prompt, brand, run (one API call) and score (one row per run × brand).

A base view first

Every metric starts from the same join. Put it in a view so the dashboards can't drift apart:

CREATE VIEW v_scored AS
SELECT r.run_date,
       r.engine,
       r.prompt_id,
       p.intent_stage,
       p.topic,
       b.name   AS brand,
       b.is_ours,
       s.mentioned,
       s.position,
       s.cited,
       s.sentiment
FROM run r
JOIN prompt p ON p.id = r.prompt_id AND p.active = 1
JOIN score  s ON s.run_id = r.id AND s.scorer_version = 'v2'
JOIN brand  b ON b.id = s.brand_id;

Pinning scorer_version in one place means a scoring upgrade is one line, not a hunt through twenty chart queries.

Metric 1: Visibility rate (with a rolling window)

Share of answers that mention you. Daily values jump around because each prompt only gets a handful of samples per day, so chart a 7-day rolling rate and show the daily points faintly behind it.

WITH daily AS (
  SELECT run_date, engine,
         SUM(mentioned) AS k,
         COUNT(*)       AS n
  FROM v_scored
  WHERE is_ours = 1
  GROUP BY run_date, engine
)
SELECT run_date, engine, n,
       1.0 * SUM(k) OVER w / SUM(n) OVER w AS visibility_7d
FROM daily
WINDOW w AS (PARTITION BY engine ORDER BY run_date
             ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)
ORDER BY engine, run_date;

Trap: averaging daily percentages (AVG(k*1.0/n)) weights a 4-sample day the same as a 40-sample day. Sum the numerators and denominators across the window, as above.

Trap: ROWS BETWEEN 6 PRECEDING assumes one row per day. If the collector skipped a day, the window silently stretches to 8 days. If gaps matter, join against a calendar table first.

Metric 2: Share of voice

Your share of all brand mentions, per engine. The denominator is mentions of tracked brands, so it only means something with a fixed competitor set:

SELECT engine,
       brand,
       SUM(mentioned) AS mentions,
       ROUND(1.0 * SUM(mentioned) /
             SUM(SUM(mentioned)) OVER (PARTITION BY engine), 3) AS sov
FROM v_scored
WHERE run_date >= date('now', '-28 days')
GROUP BY engine, brand
ORDER BY engine, sov DESC;

SUM(SUM(...)) OVER (...) is a window over the grouped result: total mentions per engine, computed after the GROUP BY.

Trap: adding a competitor mid-period drops everyone's SOV. Annotate the chart when the brand set changes, or recompute history with the new set.

Metric 3: Average position, when listed

Position only exists when the answer is a list and you're in it. Report it alongside the count it's based on, never alone:

SELECT engine,
       COUNT(position)                 AS listed_n,
       ROUND(AVG(position), 2)         AS avg_position,
       ROUND(1.0 * SUM(position = 1) / COUNT(position), 3) AS top1_rate
FROM v_scored
WHERE is_ours = 1
  AND run_date >= date('now', '-28 days')
GROUP BY engine
HAVING COUNT(position) >= 20;

Trap: average position can improve while visibility falls. If you drop out of the answers where you were listed fifth, the remaining average goes up. That's why this metric never goes on a dashboard without metric 1 next to it.

HAVING COUNT(position) >= 20 hides engines with too few listed answers to say anything. Pick your own threshold, but have one.

Metric 4: Citation rate vs. mention rate

Two separate lines, same chart. The gap between them is the useful part:

SELECT run_date,
       ROUND(AVG(mentioned), 3) AS mention_rate,
       ROUND(AVG(cited), 3)     AS citation_rate,
       ROUND(1.0 * SUM(mentioned AND NOT cited) / NULLIF(SUM(mentioned), 0), 3)
                                AS mentioned_not_cited
FROM v_scored
WHERE is_ours = 1
GROUP BY run_date
ORDER BY run_date;

A high mentioned_not_cited share means the engines know your brand but source their claims about you elsewhere, usually review sites, forums or competitor comparison pages. That's worth knowing, because those third-party pages are then what shapes the description of you.

Metric 5: Sentiment mix

If your scorer classifies how each mention is framed (positive recommendation, neutral listing, critical), show the mix as a stacked share, not a single "sentiment score":

SELECT strftime('%Y-%W', run_date) AS week,
       ROUND(AVG(sentiment = 'positive'), 3) AS positive,
       ROUND(AVG(sentiment = 'neutral'),  3) AS neutral,
       ROUND(AVG(sentiment = 'critical'), 3) AS critical,
       COUNT(*) AS n
FROM v_scored
WHERE is_ours = 1 AND mentioned = 1
GROUP BY week
ORDER BY week;

(In SQLite, a comparison returns 0 or 1, so AVG(x = 'y') is a proportion. In Postgres, write AVG((x = 'y')::int).)

Trap: a net score (positive minus critical) hides a shift from neutral to critical when positive stays flat. The stacked view shows it. Also, audit the classifier: sample 50 labelled mentions a month and check them by hand. LLM-based sentiment labels drift when the labelling model changes.

Dashboard layout that holds up in a meeting

Five metrics, laid out so that each one is read next to the one that keeps it honest:

Top row: visibility rate (7-day) by engine, with n on hover.
Next to it: share of voice by engine, with a marker wherever the brand set changed.
Second row: average position + top-1 rate, beside listed_n.
Mention vs. citation lines on one chart.
Sentiment as stacked weekly bars.
A drill-through to raw answers behind every number. When someone asks "what exactly did it say?", the answer should be one click away.

That last one is the most-used feature of any dashboard like this, and it's only possible because the collector stores raw answers (see the schema post).

When to stop building

The SQL above is the easy part. A production dashboard also needs regional breakdowns (answers vary by country), multi-brand separation for agencies, scheduled email reports, alerting on sudden drops and an interface non-technical stakeholders will actually open.

Vista AI's Brand Visibility Dashboard packages those: visibility overall and by engine, share of voice against competitors, sentiment split into positive, neutral and critical, per-query rankings, 30-day, 90-day and yearly trends, regional breakdowns, weekly email reports, and a raw data stream that keeps the full AI responses behind each number.

Build or buy, keep the rule that runs through every query above: every rate ships with its denominator.

Writing Pages That Survive RAG Chunking

Furqan Khalid — Wed, 14 Oct 2026 06:00:00 +0000

When Perplexity, ChatGPT search or Google's AI features answer a question, they don't read your page like a person. A retrieval pipeline splits it into passages, ranks those passages against the query, and hands a few of them to the model. The model writes its answer from those passages.

So the unit that wins or loses citations is the passage, not the page. A page can be excellent overall and still lose if its best answer is split across three chunks, or starts with "As mentioned above, it…".

Nobody outside these companies knows their exact chunking. But the general mechanics are well understood from open-source RAG systems, and you can simulate them well enough to find weak passages. That's what this post does.

How a typical retriever sees your page

A common pipeline:

Fetch HTML, strip navigation and boilerplate.
Split by structure (headings), then by size (a few hundred tokens), with some overlap.
Embed each chunk, and often score it with keyword search (BM25) as well.
Rerank the top candidates against the query.
Pass the top few chunks to the model with their source URLs.

Steps 2 and 4 are where page structure matters. A chunk that contains a heading phrased like the question, followed immediately by a direct answer, is easy to match and easy to quote.

Simulate the chunking

This splits a page at H2/H3 boundaries and then caps chunk size, which is roughly what many heading-aware splitters do:

import re, requests
from bs4 import BeautifulSoup

MAX_WORDS = 220

def chunks(url: str) -> list[dict]:
    soup = BeautifulSoup(requests.get(url, timeout=20).text, "html.parser")
    for t in soup(["script", "style", "nav", "footer", "header", "aside", "noscript"]):
        t.decompose()
    main = soup.find("main") or soup.find("article") or soup.body

    out, heading, buf = [], "(intro)", []
    def flush():
        words = " ".join(buf).split()
        for i in range(0, len(words), MAX_WORDS):
            out.append({"heading": heading, "text": " ".join(words[i:i + MAX_WORDS])})
        buf.clear()

    for el in main.find_all(["h1", "h2", "h3", "p", "li", "td"]):
        if el.name in ("h1", "h2", "h3"):
            flush()
            heading = el.get_text(" ", strip=True)
        else:
            buf.append(el.get_text(" ", strip=True))
    flush()
    return out

Print the chunks for one of your key pages and read them in isolation, as if each one was the only thing the model got. That exercise alone tends to be eye-opening.

Lint the chunks

Now automate the reading. Each check below flags a passage that's likely to lose to a cleaner one:

DANGLING = re.compile(r"^(this|that|these|those|it|they|as (mentioned|noted|discussed)|see above|the above)\b", re.I)
VAGUE_HEADINGS = {"overview", "introduction", "details", "more", "getting started", "our approach", "why us"}

def lint(c: dict) -> list[str]:
    issues = []
    words = c["text"].split()
    first = " ".join(words[:40])

    if DANGLING.match(c["text"]):
        issues.append("starts with a back-reference; the chunk doesn't stand alone")
    if c["heading"].lower().strip(" :") in VAGUE_HEADINGS:
        issues.append(f"vague heading '{c['heading']}' carries no query signal")
    if len(words) < 40:
        issues.append("very short; may not hold a complete answer")
    if not re.search(r"\d", c["text"]):
        issues.append("no concrete numbers")
    if c["heading"].endswith("?") and len(first.split(".")[0].split()) > 35:
        issues.append("question heading, but the first sentence is long; answer may be buried")
    return issues

for c in chunks("https://example.com/pricing"):
    if problems := lint(c):
        print(f"\n## {c['heading']}\n   {c['text'][:120]}…")
        for p in problems:
            print("   -", p)

These are heuristics. "No numbers" isn't a defect on every chunk. Use the output as a reading list, not a score.

The fixes, as patterns

Self-contained answers. Repeat the subject instead of using a pronoun. The first sentence should make sense if it's the only thing quoted.

- ## Pricing
- It starts at $49, which includes everything above.
+ ## How much does Acme Analytics cost?
+ Acme Analytics costs $49 per month for up to 5 seats on the Starter plan,
+ and $199 per month for up to 25 seats on Growth. Both include unlimited events.

Answer first, then context. Lead each section with one or two sentences that directly answer its heading, then expand. A retriever ranks the chunk on its full content, but the model quotes what's easiest to lift.

Headings as queries. "How does Acme compare to Mixpanel?" matches a comparison query. "The Acme difference" doesn't.

Tables for comparisons. A comparison table chunks cleanly, and models quote it accurately. A comparison buried in four paragraphs of prose does neither.

Keep sections a sensible size. A 1,500-word section under one H2 gets split mid-thought. Break it with H3s at natural boundaries.

Pin down facts with JSON-LD

Prose is ambiguous. Structured data states facts explicitly: who made this, what it costs, when it was updated. Search engines use it heavily, and pages with clean, consistent facts are less likely to be misquoted.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Acme Analytics",
  "applicationCategory": "BusinessApplication",
  "operatingSystem": "Web",
  "offers": [
    { "@type": "Offer", "name": "Starter", "price": "49", "priceCurrency": "USD" },
    { "@type": "Offer", "name": "Growth",  "price": "199", "priceCurrency": "USD" }
  ],
  "publisher": { "@type": "Organization", "name": "Acme Inc.", "url": "https://acmeanalytics.com" }
}
</script>

Rules that matter more than which types you pick:

Match the visible page. If JSON-LD says $49 and the page says $59, you've created a contradiction for the model to resolve, and it may resolve it wrong.
Validate it. Run it through the Schema.org validator or Google's Rich Results Test. Invalid JSON is silently ignored.
Render it server-side. JSON-LD injected by client-side JavaScript has the same visibility problem as any other client-rendered content.

A quick check that it's in the raw HTML and parses:

curl -s https://example.com/pricing \
  | grep -oP '(?s)<script type="application/ld\+json">\K.*?(?=</script>)' \
  | python -m json.tool > /dev/null && echo "JSON-LD OK" || echo "JSON-LD missing or invalid"

(That grep handles the common case of one block on one line. For multi-line or multiple blocks, use the BeautifulSoup extractor from the competitor-diff post.)

What about llms.txt?

llms.txt is a proposed convention: a Markdown file at your site root that summarises the site and links to LLM-friendly versions of key pages. It's cheap to add and can help tools that read it, such as some coding assistants and agents working with documentation. Google has said it doesn't use it for Search, and none of the major AI search engines have confirmed relying on it for ranking or citation. Add it if it's easy, but don't treat it as a substitute for the passage-level work above.

Measuring whether it worked

Structural rewrites are hypotheses. Before you rewrite, record the page's baseline, ship the change, and watch citation rate for the prompts that page targets over the next few weeks. Rewrite one page cluster at a time so you can attribute the change.

For the baseline and the after-check, Vista AI's Page Audit scores a page on AI Visibility, AI Understanding and AI Citation (each 0–100), including content extraction (how easily passages can be lifted for RAG), schema and citation potential, with a prioritised fix list and competitor comparison. Running it before and after a rewrite like this is a quick way to see whether the changes did what you intended.

The short version: write every section as if it's the only passage the model will see, because often it is.

See Your Site the Way an AI Crawler Does: robots.txt, Status Codes and No JavaScript

Furqan Khalid — Tue, 13 Oct 2026 18:00:00 +0000

When a page ranks on Google but never shows up in ChatGPT or Perplexity answers, the cause is often technical rather than editorial. Somewhere between the AI crawler and your content, something says no, or returns an empty shell.

Google has spent two decades learning to cope with messy sites, and it renders JavaScript. Many AI crawlers are less forgiving. This post builds a small audit script for the three most common technical blockers:

robots.txt disallows AI user agents, sometimes without anyone deciding to.
The CDN or WAF blocks or challenges them before robots.txt even matters.
The content doesn't exist without JavaScript, so a non-rendering fetcher gets an empty page.

Check 1: robots.txt per AI user agent

Python's standard library can evaluate robots.txt rules per user agent:

# ai_audit.py
from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin, urlparse

AI_AGENTS = [
    # training
    "GPTBot", "ClaudeBot", "CCBot", "Google-Extended", "Applebot-Extended",
    # search indexing
    "OAI-SearchBot", "Claude-SearchBot", "PerplexityBot",
    # user-triggered fetches
    "ChatGPT-User", "Claude-User", "Perplexity-User",
]

def robots_report(page_url: str) -> dict[str, bool]:
    root = f"{urlparse(page_url).scheme}://{urlparse(page_url).netloc}"
    rp = RobotFileParser(urljoin(root, "/robots.txt"))
    rp.read()
    return {ua: rp.can_fetch(ua, page_url) for ua in AI_AGENTS}

Two caveats:

RobotFileParser follows the original robots.txt convention and doesn't implement every detail of RFC 9309 (such as longest-match precedence between Allow and Disallow). For edge cases, confirm with a spec-compliant parser like Google's open-source robotstxt library.
Some vendors document that user-triggered agents may not follow robots.txt in the same way as crawlers, because a human requested the page. Read each vendor's bot docs for the current policy rather than assuming.

Decide on purpose. Training, search and user-triggered agents are separate so you can, for example, block training while allowing the agents that fetch pages to answer live questions:

# robots.txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Google-Extended controls whether Google may use your content for Gemini model training and grounding. It does not affect Google Search or AI Overviews, which use regular Googlebot.

Check 2: What does the edge actually return?

robots.txt is advisory. Your CDN or WAF decides what really gets served, and several bot-protection products now block or challenge AI crawlers by default.

Request the page with each bot's user agent and look at the status code and body size:

import requests

UA_STRINGS = {
    "browser": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126 Safari/537.36",
    "GPTBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
    "ClaudeBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)",
    "PerplexityBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
}

def edge_report(url: str) -> dict:
    out = {}
    for name, ua in UA_STRINGS.items():
        try:
            r = requests.get(url, headers={"User-Agent": ua}, timeout=20, allow_redirects=True)
            challenged = any(s in r.text[:5000].lower() for s in
                             ("just a moment", "cf-challenge", "captcha", "access denied"))
            out[name] = {"status": r.status_code, "bytes": len(r.content), "challenge": challenged}
        except requests.RequestException as e:
            out[name] = {"error": str(e)}
    return out

Read the result relative to the browser row. If browser gets 200 with 80 KB and GPTBot gets 403, or a 6 KB challenge page, the edge is blocking it.

Important limitation: this test sends a spoofed user agent from your IP. Some CDNs verify bots by IP, so they may block your fake GPTBot while letting the real one through. Equally, an allowlist could pass your test and still block the real crawler. Treat a block here as a strong hint, then confirm with:

your CDN's bot or firewall event logs filtered by those user agents, and
your origin access logs (the previous post in this series covers that), looking for real, IP-verified hits and their status codes.

Check 3: Does the content exist without JavaScript?

This is the most common silent failure on modern frontends. If the main content is fetched and rendered client-side, a crawler that doesn't execute JS sees a <div id="root"></div> and a bundle.

Compare what's in the raw HTML with what a browser renders:

from bs4 import BeautifulSoup

def visible_text(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for t in soup(["script", "style", "noscript", "template", "svg"]):
        t.decompose()
    return " ".join(soup.get_text(" ").split())

def raw_html_report(url: str, must_contain: list[str]) -> dict:
    html = requests.get(url, headers={"User-Agent": UA_STRINGS["GPTBot"]}, timeout=20).text
    text = visible_text(html)
    return {
        "raw_words": len(text.split()),
        "missing_phrases": [p for p in must_contain if p.lower() not in text.lower()],
    }

must_contain should be a few sentences you know are in the page's main content: the price, the product description, the answer to the page's main question. If they're missing from raw HTML, AI fetchers that don't render JS can't see them.

For a rendered comparison, add Playwright:

from playwright.sync_api import sync_playwright

def rendered_words(url: str) -> int:
    with sync_playwright() as p:
        b = p.chromium.launch()
        page = b.new_page()
        page.goto(url, wait_until="networkidle")
        n = len(visible_text(page.content()).split())
        b.close()
        return n

A raw/rendered word ratio well below ~0.8 on a content page means a meaningful share of it depends on JavaScript.

Fixing it

Next.js / Nuxt / SvelteKit / Astro: make sure content pages use SSR or static generation, not a client-only fetch in useEffect / onMounted. In the Next.js App Router, Server Components render on the server by default. The usual mistake is a "use client" page that fetches its content after mount.
SPA with no SSR option: prerender content routes at build time.
Tabs, accordions, "read more": keep the text in the DOM and hide it with CSS rather than loading it on click.

Putting it together

import json, sys

if __name__ == "__main__":
    url, *phrases = sys.argv[1:]
    print(json.dumps({
        "robots": robots_report(url),
        "edge": edge_report(url),
        "raw_html": raw_html_report(url, phrases),
    }, indent=2))

python ai_audit.py https://example.com/pricing "per seat per month" "Starter plan"

Run it across your top 20 pages by revenue, not your homepage. Pricing, docs, comparison and integration pages are where AI answers pull from most, and they're the pages most likely to be built as client-rendered app routes.

Beyond the three blockers

Passing these checks means AI crawlers can read your page. It says nothing about whether they'll use it, which depends on structure, schema, trust signals and whether the page says anything worth quoting. The next post covers that half.

For both halves at once, Vista AI's Page Audit checks a page across seven areas: traditional SEO signals, AI crawlability and rendering, schema, AI content extraction (RAG), E-E-A-T signals, link quality and citation potential. It returns 0–100 scores for AI Visibility, AI Understanding and AI Citation, a prioritised fix list, and side-by-side scoring of competitor pages.

If you just want the quick version this week: check robots.txt, check your CDN's bot settings, and view-source on your five most important pages.

Why Does AI Cite Their Page and Not Yours? Diff Them With a Script

Furqan Khalid — Mon, 12 Oct 2026 18:05:00 +0000

You've measured share of voice and found a prompt cluster where a competitor keeps getting recommended and cited, and you don't. The obvious next question is what's different about their page?

Most people answer that by eyeballing two browser tabs. This post does it with a script: take the URLs AI engines cited for a prompt, fetch them the way a non-JavaScript crawler would, extract the features that tend to matter for extraction and citation, and print a diff against your page.

It won't explain everything. Brand reputation and third-party mentions matter too, and they don't show up in page HTML. But it reliably catches the boring structural gaps, which are the fastest ones to fix.

Step 1: Get the cited URLs

Start from stored citations (see the earlier post on parsing them) for the prompts where you lose:

import sqlite3, json
from collections import Counter

conn = sqlite3.connect("visibility.db")
rows = conn.execute("""
    SELECT r.sources FROM run r
    JOIN prompt p ON p.id = r.prompt_id
    WHERE p.topic = ? AND r.run_date >= date('now','-28 days')
""", ("analytics-pricing",)).fetchall()

counts = Counter(u for (s,) in rows for u in json.loads(s))
competitor_urls = [u for u, n in counts.most_common(10) if "acmeanalytics.com" not in u]

Pick the top three to five cited URLs. Pages cited across many runs and engines are the signal. A URL cited once is noise.

Step 2: Fetch like a simple crawler

Fetch raw HTML with no JavaScript rendering. If a page's content needs JS to appear, that's itself a finding:

import requests

UA = "Mozilla/5.0 (compatible; page-diff/0.1; +https://example.com/bot)"

def fetch(url: str) -> str:
    r = requests.get(url, headers={"User-Agent": UA}, timeout=20)
    r.raise_for_status()
    return r.text

Use an honest user agent and keep the volume tiny. You're reading a handful of public pages, not crawling a site.

Step 3: Extract features that matter for AI extraction

Retrieval systems split pages into passages, embed them and pull the best-matching chunks into the model's context. So the features worth measuring are the ones that affect whether a clean, self-contained answer passage exists and whether the facts are machine-readable.

import re, json
from bs4 import BeautifulSoup

def features(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    ld_types = []
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            ld_types.append("INVALID")
            continue
        items = data if isinstance(data, list) else data.get("@graph", [data])
        for it in items:
            t = it.get("@type")
            ld_types.extend(t if isinstance(t, list) else [t])

    for t in soup(["script", "style", "noscript", "nav", "footer", "header"]):
        t.decompose()
    main = soup.find("main") or soup.find("article") or soup.body or soup
    text = main.get_text(" ", strip=True)
    words = len(text.split())

    headings = [h.get_text(" ", strip=True) for h in main.find_all(["h2", "h3"])]
    question_heads = sum(1 for h in headings if h.endswith("?") or
                         re.match(r"(?i)^(how|what|why|when|which|is|can|does|should)\b", h))

    # words before the first paragraph under each H2: proxy for "answer first"
    first_para_lens = []
    for h in main.find_all("h2"):
        p = h.find_next("p")
        if p:
            first_para_lens.append(len(p.get_text(" ", strip=True).split()))

    return {
        "words": words,
        "h2_h3": len(headings),
        "question_headings": question_heads,
        "median_first_para_words": sorted(first_para_lens)[len(first_para_lens)//2] if first_para_lens else None,
        "tables": len(main.find_all("table")),
        "lists": len(main.find_all(["ul", "ol"])),
        "numbers": len(re.findall(r"\b\d[\d,.]*\s?(%|\$|ms|x|users|customers)?", text)),
        "jsonld_types": sorted(set(filter(None, ld_types))),
        "has_author": bool(soup.select_one('[rel=author], .author, [itemprop=author]')) or "Person" in ld_types,
        "has_date": bool(soup.find("time")) or bool(re.search(r'"date(Published|Modified)"', html)),
        "outbound_links": len([a for a in main.find_all("a", href=True) if a["href"].startswith("http")]),
    }

Each feature is a proxy, and some are crude (numbers will count list markers in some layouts). That's acceptable here. We're comparing pages against each other, not scoring them in absolute terms.

Step 4: Diff

import pandas as pd

ours = "https://acmeanalytics.com/pricing"
table = {url: features(fetch(url)) for url in [ours, *competitor_urls[:4]]}
df = pd.DataFrame(table).T
df.index = ["OURS"] + [f"C{i}" for i in range(1, len(df))]
print(df.to_string())

      words h2_h3 question_headings median_first_para_words tables lists numbers jsonld_types                     has_author has_date
OURS    410     4                 0                      62      0     2      11 [Organization]                   False      False
C1     1840    14                 9                      28      2     6      74 [FAQPage, Organization, Product]  True       True
C2     2210    17                 6                      31      1     8      91 [Article, BreadcrumbList]        True       True
C3     1390    11                 7                      24      3     4      58 [Product, SoftwareApplication]   False      True

(Illustrative output.) A diff like this tends to make the problem obvious:

Depth: the cited pages are 3–5× longer, and that isn't padding. They have 11–17 sections.
Question headings: competitors title sections the way people ask. The other page has none.
Answer first: the competitors open each section with a short (~25–30 word) direct answer. The other page opens with ~60 words of preamble.
Specifics: many more concrete numbers, plus comparison tables.
Machine-readable facts: Product / FAQPage schema, an author and a date.

Step 5: Check what's cited, not just what exists

Features tell you what a page has. The cited passages tell you which part the engine used. If you stored cited_text with your citations (here, in a citation table of run_id, url, cited_text rows produced by the normaliser from the citations post), line them up against the competitor page:

for url in competitor_urls[:3]:
    print("\n", url)
    for (s,) in conn.execute(
        "SELECT cited_text FROM citation WHERE url = ? AND cited_text IS NOT NULL LIMIT 5", (url,)):
        print("  -", s[:160])

You'll usually find the engine quoting the same two or three passages: a pricing sentence with a concrete number, a one-line definition, a row of a comparison table. Those are the passages to match or beat, with your own facts.

What the diff can't see

Be honest about what's out of scope:

Off-site reputation. Reviews, forum threads, publications and lists where competitors are mentioned. In model-knowledge-heavy engines, these often matter more than on-page features.
JavaScript-rendered content. If your page renders client-side, the raw fetch shows less than a browser does. That's informative (many AI fetchers don't render JS), but compare against a rendered version before drawing conclusions.
Freshness and authority signals that the engines don't disclose.

So treat the output as a list of hypotheses. Ship the fixes, then check whether citation rate for that prompt cluster moves over the following weeks.

Automating the loop

Running this by hand works for one prompt cluster. Across dozens of clusters and several competitors, you want it running continuously.

Vista AI's Competitor Analysis handles the tracking side: it lists which competitor pages get cited most, where competitors beat you per engine, and surfaces content gaps and quick wins. For the page-level side, Vista's Page Audit scores your page and competitor pages side by side on AI visibility, understanding and citation potential, which is a more complete version of the feature diff above.

Whatever you use, the loop is the same: find the prompts you lose → read what gets cited instead → close the specific gap → re-measure.

DEV Community: Furqan Khalid

Should You Serve Markdown to AI Agents? llms.txt, Accept: text/markdown and What Google Actually Said

The three things people mean by "Markdown for AI"

What the evidence says

So when is it worth doing?

How to do it without getting into trouble

Implementation: Next.js content negotiation

And llms.txt?

Conclusion

ChatGPT Can't Cite What Bing Hasn't Indexed: Setting Up IndexNow in Next.js

What IndexNow is (and isn't)

Step 1: Create and host your key

Step 2: A small submit helper

Option A: Call it when content changes (CMS webhook)

Option B: Submit changed pages on deploy (Markdown/MDX in git)

Step 3: Make lastmod honest

Step 4: Verify in Bing Webmaster Tools

Bing-indexed is step one, not the finish line

Tracking ChatGPT, Perplexity and Claude Referrals Properly: GA4 Regex, Server Logs and the 'Dark AI' Problem

The four shapes of AI traffic

1. A classifier you can reuse everywhere

2. GA4: a custom channel group

3. Log AI visits server-side

Express

Next.js (App Router)

4. Report on it

5. Estimating the dark part

Traffic is the lagging indicator

Simulate Google's Query Fan-Out to Find Your Content Gaps (Python)

What we're building

Setup

Step 1: Generate the fan-out

Step 2: Split your page into passages

Step 3: Score coverage with embeddings

Step 4: Put it together

Reading the results

Caveats (please read)

Where this fits

Your Next Visitor Is an AI Agent: Semantic HTML Is SEO Again

How agents see a page

See your page the way an agent does

The 8 fixes that matter most

1. Use real buttons and links

2. Label every input

3. Give icon buttons names

4. Use native form controls (or full ARIA patterns)

5. Landmarks and headings

6. Put errors where they can be read

7. Don't hide key actions behind hover

8. Avoid layout shifts after load

Put it in CI

How this connects to search

WebMCP: Make Your Site Callable by AI Agents (With Working Code)

The mental model

The API surface

Example: a store search and add-to-cart

Testing it locally

Security: treat every tool like a public API endpoint

So is this SEO?

What I'd do this quarter

SEO in Late 2026: What Actually Changed for Developers (and What Didn't)

1. Search engines now search for the user

2. Your next visitor might be an agent, not a person

3. Bing quietly became an AI-visibility dependency

4. AI referral traffic exists, and your analytics undercounts it

5. Markdown for machines went mainstream (and got controversial)

6. Manipulating AI answers is officially spam

7. JavaScript rendering is a bigger risk than it was

8. Recommendations come from consensus, not your homepage

What didn't change (please don't skip this part)

A 30-minute checklist for this week

The series

Tracking Traffic From ChatGPT, Perplexity and Gemini in GA4 and BigQuery

How AI traffic shows up

Step 1: A custom channel group in GA4

Step 2: Raw AI sessions from the BigQuery export

Step 3: Join "cited" with "clicked"

What this can't measure

One view for both

The SQL Behind an AI Visibility Dashboard: Five Metrics and Their Traps

Step 3: Make `lastmod` honest