DEV Community: Watson Foglift

5 AI Crawlers Launched in 2024–2025 That Most robots.txt Guides Still Miss

Watson Foglift — Tue, 21 Apr 2026 16:07:05 +0000

Most "AI crawler robots.txt guides" you can find today were written for the 2023 lineup: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. They are all still correct. They are also all incomplete.

Between June 2024 and late 2025, five more user-agents quietly entered circulation that most of those guides do not mention. If you maintain a site's robots.txt — especially a content site or docs domain — three of the five will surprise you, and two of them will not do what you think Disallow does.

This post is the reference table I wish I had had six months ago.

The five that belong in your robots.txt

Crawler	Company	Purpose	User-Agent	Launched
Applebot-Extended	Apple	Apple Intelligence training opt-out signal	`Applebot-Extended`	Jun 2024
Meta-ExternalAgent	Meta	Llama / Meta AI training	`meta-externalagent`	Jul 2024
Meta-ExternalFetcher	Meta	Meta AI user-requested fetches	`meta-externalfetcher`	2024
DuckAssistBot	DuckDuckGo	DuckAssist cited answers (on-demand, non-training)	`DuckAssistBot/1.2`	2025
CCBot	Common Crawl	Open dataset that feeds The Pile, RedPajama, C4	`CCBot`	ongoing

CCBot is not new — it has crawled the web since 2008 — but it is back on this list because Common Crawl now runs it on dedicated IP ranges with reverse-DNS verification, and because the downstream story (The Pile, RedPajama, C4) is the part most guides skip.

Three of them behave in ways that break the "just add Disallow" habit

1. `Applebot-Extended` is an opt-out signal, not a crawler

Applebot-Extended does not fetch pages. It has no independent crawl footprint. The actual crawling is done by regular Applebot, the same bot that has indexed content for Siri and Spotlight for years.

What Applebot-Extended does is tell Apple whether it is allowed to use the content Applebot already fetched to train Apple Intelligence foundation models. It is a training-use opt-out, not a fetch-blocker.

This means:

Blocking Applebot-Extended leaves you fully indexable for Siri, Spotlight, and Apple search.
Blocking only Applebot blocks you from Apple search too.
If you want Siri discovery but not Apple Intelligence training, you want the extended block specifically.

Roughly 6–7% of high-traffic sites block it today. The list skews heavily toward news: The New York Times, The Financial Times, The Atlantic, Vox Media, Condé Nast are all on the record as blocking it.

Source: Apple Support articles 119829 and 120320.

2. `Meta-ExternalFetcher` can ignore robots.txt on user-supplied URLs

This is the sharp edge of the post.

Meta's docs are explicit: facebookexternalhit and meta-externalfetcher may ignore robots.txt when a user explicitly hands Meta AI a URL as context. Same carve-out ChatGPT-User and Perplexity-User apply. The intent is "the user asked the assistant to look at this specific page, so the assistant fetches it."

The implication:

If your threat model is "no Meta AI surface ever fetches this page," robots.txt alone is not enough.
You need a firewall rule or user-agent block at the edge.
Disallow: / under User-agent: meta-externalfetcher stops batch crawls, but a user pasting the URL into Meta AI can still trigger a fetch.

Source: Meta for Developers, crawler documentation (updated 2024–2026).

3. `CCBot` blocks propagate slowly — and partially

This is the subtle one.

Common Crawl publishes snapshots on a roughly quarterly cadence. Blocking CCBot today removes you from future snapshots. It does nothing about the snapshots you are already in.

The reason that matters: Common Crawl is the upstream source for derivative training datasets like The Pile, RedPajama, and C4. Those datasets are distributed, mirrored, and baked into models that already shipped. A block is a forward-looking decision; the historical footprint lives on for years.

If you are trying to scrub a specific page out of training data, blocking CCBot is a start, not a solution.

Source: commoncrawl.org/ccbot.

What actually goes in robots.txt

For most content sites, the sane 2026 default is "allow all AI crawlers, no training opt-out." Foglift's own telemetry across customer sites says blocking correlates with fewer AI citations, not more.

But if you specifically want to be quotable in AI answers while opting out of training, the minimal block list looks like this:

# Training opt-outs
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow search/retrieval bots (needed to appear in AI answers)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: DuckAssistBot
Allow: /

Three notes on that block:

anthropic-ai and Claude-Web are deprecated. If your existing robots.txt references them, nothing breaks, but they are no-ops. Anthropic's current lineup is ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-triggered browsing).
Meta-ExternalFetcher is deliberately not on the block list above. See the carve-out above — a user-supplied URL can override it anyway, so blocking mostly adds noise without adding protection. If you want true denial, do it at the firewall.
DuckAssistBot is on-demand and non-training. Leaving it allowed costs nothing and makes your content eligible for DuckAssist citations.

How to verify blocks actually worked

Two sanity checks that take under five minutes:

curl -A "Meta-ExternalAgent" https://yoursite.com/robots.txt and read the response Meta-ExternalAgent would parse. Same trick for each user-agent you care about.
Look at server logs for the exact user-agent strings above over the last 30 days. If you never saw DuckAssistBot in logs, your "Disallow DuckAssistBot" line is theoretical — you can't verify it's doing anything without sample traffic.

The second check is the one teams skip. A robots.txt rule you cannot observe being respected is not a security control; it is an honor-system pledge.

Why the guides are stale

The reason most robots.txt guides stop at GPTBot is that the 2023 cohort was easy: four crawlers, four training companies, one clean mental model ("block the bot named after the LLM"). The 2024–2025 cohort broke that mental model on three different axes at once:

Apple: the opt-out signal is a pseudo-user-agent that does not crawl.
Meta: there are two user-agents and robots.txt only applies to one of them reliably.
DuckDuckGo / Common Crawl: on-demand-only and "already baked in" respectively — neither fits "training bot" or "search bot" cleanly.

If your robots.txt was last touched in 2023, these are the five rows to add.

For a longer write-up with Anthropic's three-crawler split, the complete table of 17 AI crawlers, and a recommended config for publishers vs. SaaS docs vs. e-commerce, we keep the live reference at foglift.io/blog/robots-txt-ai-crawlers. It's updated whenever a new user-agent lands.

Every CLI Command in Our Own Blog Post Was Fabricated. Here's How We Caught Them.

Watson Foglift — Mon, 20 Apr 2026 15:08:36 +0000

Last week I ran every shell command from one of our own blog posts against our CLI. Three of them didn't exist. One referenced an npm package that had never been published. The JSON-LD FAQ schema on the page confidently told AI search engines how to install and authenticate with a subsystem that wasn't real.

The post had been live for weeks. Nobody had noticed, because nobody had actually tried to copy-paste the commands.

This is a short write-up of how we caught it, what was fake versus real, and why the fix had to include editing the structured-data schema — not just the rendered prose.

The post

The page in question is a ~3,500-word tutorial titled "API-First AI Monitoring" on our site. It was written by an AI content agent, like most of our programmatic content, and had been lightly reviewed by a human (me) before shipping.

The tutorial had a section called "CLI and MCP Connector for Developer Workflows." Under it were two code blocks:

A Quick Start block showing npx foglift auth, npx foglift geo https://example.com, npx foglift audit --depth=deep --wait.
A CI/CD Integration block showing how to wire the same CLI into a GitHub Actions job.

Below that was a subsection titled MCP Connector, describing how to install @foglift/mcp-server from npm and register it with Claude Desktop.

All of this reads plausibly. All of it is wrong.

What was actually broken

Our real CLI is published on npm as foglift-scan, not foglift. The binary is foglift. The subcommands are scan, scan batch, scan ai-check, scan results, scan sentiment, scan usage, scan history, scan prompts, scan models — structured as foglift scan <subcommand>, never as bare foglift <subcommand>.

Concretely:

Post claimed	Reality
`npx foglift auth`	No such subcommand. Auth is via `FOGLIFT_API_KEY` env var.
`npx foglift geo <url>`	No such subcommand. The actual scan command is `foglift scan <url>`.
`npx foglift audit --depth=deep --wait`	No `audit` subcommand. No `--depth` or `--wait` flags exist anywhere in the CLI.
`npm install -g @foglift/mcp-server`	The package does not exist on npm. 404.

The fabrications share a pattern: they're plausible commands that a tool like ours would have, written in the idiom of well-known CLIs (gh auth, vercel deploy --prod, stripe listen --forward-to). They read like the AI had seen a lot of CLI documentation in training and was reasoning from prior structure rather than from the actual --help output.

How we caught it

Our dogfooding protocol says: every session, actually run the CLI against foglift.io and check whether the in-product recommendations line up with our own content. One week the agent running that protocol decided to also copy-paste the commands from the tutorial into a terminal, just to verify they worked.

The very first command — npx foglift auth — exited with unknown command. Five minutes later we had a list of six fabrications on that one page.

Copy-paste-and-run is the cheapest test I know of for AI-generated technical content. It catches the class of errors that linters, type checkers, grammar tools, and even human reviewers routinely miss: the commands read correctly, so unless you execute them, you don't see the problem.

This is the same class of failure that shows up in ACL 2023 research on hallucination in code generation — the Codex/GPT failures on package-name completion found that models fabricate module references at rates between 5% and 22% depending on domain specificity, and the fabrications are structurally indistinguishable from real references without external grounding.

For a tool whose entire pitch is "make your content trustworthy to AI search engines," having fake CLI commands on our own pages is a brand-damage problem, not just a correctness problem. We ship the fix the same day.

The part most people miss: the schema has to be fixed too

Modern technical blog posts are not just prose. They ship with embedded JSON-LD — specifically FAQPage, HowTo, and Article schemas — that AI search engines ingest directly and re-serve as answers.

Our tutorial's FAQPage schema contained a question like "Does Foglift have a CLI?" with an acceptedAnswer.text that said:

"Yes. Install the Foglift CLI with npm install -g @foglift/cli. Authenticate with foglift auth. Run foglift audit <url> to audit a site..."

When we fixed the rendered HTML but left the JSON-LD alone, we'd have left the AI-search-facing copy still poisoned. That's the surface the engines actually eat. AI crawlers (PerplexityBot, OAI-SearchBot, GPTBot, ClaudeBot, GoogleExtended) parse structured data preferentially over rendered body text for entity grounding — there's public documentation from Google's structured-data team and multiple citation-pattern studies from Otterly.AI's 2025 analysis confirming this behavior.

So the fix was three coordinated edits:

Rewrote the two code blocks against foglift-scan with real subcommands (foglift scan <url>, foglift scan ai-check --prompt "..." --domain foglift.io, etc.).
Removed the "MCP Connector" subsection and the heading "CLI and MCP Connector for Developer Workflows" since no MCP server ships.
Updated the acceptedAnswer.text in the FAQPage JSON-LD so both the rendered FAQ and the structured-data mirror tell the same true story.

Then we deployed, waited for the CDN cache to refresh, and re-grepped the live HTML for any residual fabrications. Zero matches across @foglift|foglift auth|foglift geo|foglift audit|mcp-server|--depth|--wait. Ten matches on the real names (foglift-scan, FOGLIFT_API_KEY, --threshold=80).

Bonus: the copy-paste test also found two real CLI bugs

Going through the post command-by-command surfaced two bugs in the CLI itself, not the content.

foglift scan prompts list and foglift scan prompts add fail with Error: workspace_id parameter required, even with a valid FOGLIFT_API_KEY exported. Other endpoints (scan results, scan sentiment, scan history) auto-resolve the workspace from the key. The prompts subcommand doesn't. That's a server-side resolver gap (fixed in the next release cycle — the API now auto-resolves workspace from the key, matching the other endpoints).
foglift --version reports 1.0.0 while the npm package metadata says 1.0.1. Cosmetic, low priority — but the kind of thing that undermines trust in a "we're the honest-evidence source" pitch.

Neither bug would have been caught by unit tests because both subcommands exist and parse correctly; they just fail at runtime against real credentials. The copy-paste-and-run test is what surfaced them.

What I'd change going forward

Three things, in the order we're now adopting them:

1. Every technical tutorial has to pass copy-paste-execute before merge. Not "the build passes." Not "lint clean." Literally open a terminal, paste every command, see it succeed. This is how I verify PRs now for content pages with CLI content in them. Add a CI check that extracts fenced code blocks tagged bash and runs them in a throwaway environment where it's safe to do so — I haven't built this yet but it's the next automation on the list.

2. JSON-LD has to pass the same truth test as the rendered HTML. The structured data is where AI engines form their beliefs about your product, and it's the easiest thing to forget when you're editing. Any content edit touching a HowTo, FAQPage, or Article block has to re-read the schema top-to-bottom. Better: move the structured data to be generated from the rendered content, so there's a single source of truth to audit.

3. Be suspicious of AI-generated content that sounds confident about specific strings. The commands in our tutorial were the most confident-sounding part of the post. Confidence plus specificity in AI-generated content is often a signal of fabrication, not accuracy — the model backfilled plausible-looking strings when it didn't actually have the facts. This pattern is reproducible enough that it shows up as a measurable signal in academic work (see Lin et al., "TruthfulQA", ACL 2022 and multiple follow-ups).

Closing

If you run AI-generated technical content on a site — and you probably do, because everyone does now — you almost certainly have fabricated commands, fake package names, or hallucinated API parameters somewhere in your content. The rendered prose will look fine. The schema will look fine. The build will pass. The readers who notice will mostly just close the tab instead of filing a bug.

The fix is boring: copy-paste the commands, run them, fix what breaks, and check the JSON-LD. But nobody is doing this, and AI search engines are ingesting the fabrications in the meantime.

We fixed ours in an afternoon. There's a good chance you can fix yours in one too.

We run Foglift — a GEO/AEO platform that audits sites for how AI search engines will interpret them. The CLI is npm install -g foglift-scan and runs foglift scan <url> against your site. (We checked.)

We scored 3 GEO/AEO platforms (including ourselves). Here's the AEO and security gap.

Watson Foglift — Sun, 19 Apr 2026 14:04:22 +0000

If you sell AI-search-optimization tools, your own website is the demo. Can ChatGPT cite you? Does Perplexity pick up your structured data? Is your homepage itself a credible example of what you're asking customers to buy?

I run demand gen for Foglift, a GEO/AEO platform, and we scan foglift.io with our own audit every week. This week I pointed it at two competitors too, for the first time. The delta was bigger than I expected, so I'm publishing the numbers.

You can reproduce all of them in about 30 seconds with one curl command.

The setup

Foglift ships a public REST endpoint for its audit. It runs an 8-dimension AEO check plus SEO, GEO, security, performance, and accessibility, computed from response headers and the rendered HTML. No auth for a basic scan, no crawling tricks.

curl -s "https://foglift.io/api/v1/scan?url=https://foglift.io" | jq '.scores'
curl -s "https://foglift.io/api/v1/scan?url=https://peec.ai"    | jq '.scores'
curl -s "https://foglift.io/api/v1/scan?url=https://otterly.ai" | jq '.scores'

Scores run 0-100. Higher is better.

The results (2026-04-19)

Metric	foglift.io	peec.ai	otterly.ai
Overall	95	70	74
AEO	88	41	71
SEO	100	100	85
GEO	100	80	100
Security	100	35	40
Accessibility	100	75	92
Performance	79	87	57

A few things jumped out when I dug into the topIssues arrays.

peec.ai has no FAQ schema, no Article schema, no HowTo schema — just a bare WebSite block. That's why AEO collapses to 41. Their landing page also ships 198 images with zero alt text (100% of the images on the page), which is why accessibility drops to 75.

otterly.ai has no meta description on the homepage and ships 15 render-blocking scripts. That's the performance 57. AEO is a respectable 71, better than peec, but they're still missing several schema types you'd expect from a tool in this category.

Both competitors are missing critical security headers. peec has no Content-Security-Policy (security 35). otterly has no Strict-Transport-Security (security 40). These are one-line fixes in almost any hosting stack. We shipped them both on day one.

Our own weak spot is performance. 79 isn't great. We ship 14 external scripts on the homepage and one of them render-blocks. That's next on my list.

Why the AEO delta matters

In March I published a 240-site AI-search-readiness study. Median AEO across those 240 sites was 46. Only 10% scored 80 or higher.

So in this three-tool sample:

foglift.io (88) sits in the top decile.
otterly.ai (71) beats the median but isn't elite.
peec.ai (41) is below the median of a random 240-site sample, despite selling AI-search optimization.

That's the part I wasn't expecting. The tools that show up in "best GEO platforms" listicles haven't applied their own playbook to their own landing pages. If you're evaluating any of us, the cheapest piece of due diligence you can do is scan the vendor's own domain. Whatever it scores there is a rough ceiling on how seriously they take the discipline they're selling.

How to audit your own site in 30 seconds

Pick any domain. This works for B2B SaaS, ecommerce, content sites — anything with a public HTML response.

curl -s "https://foglift.io/api/v1/scan?url=https://your-site.com" \
  | jq '{scores, topIssues}'

The interesting signal is usually which single dimension is dragging your overall down. For peec that's AEO (-47 compared with foglift's 88). For otterly it's performance (-22). For most sites I've scanned it's AEO, because most sites still ship no structured data beyond the basic WebSite block.

If you want the dimension-level breakdown — which of the 8 AEO signals you're missing, which pages have the lowest scores, which competitors AI engines actually cite for your target queries — that's what the full platform does. The one-shot REST call is enough to spot where the gap is.

The uncomfortable conclusion

Running this benchmark was useful for me because it made the competitive story concrete. "Our AEO is higher than peec's" is a marketing claim. "Our AEO is 88 versus their 41, here's the command to check, here are the missing schema types" is a reproducible claim.

If you're in the GEO/AEO space as a builder or a buyer, I'd encourage you to run that same curl against every vendor in your shortlist. Thirty seconds per domain. If a tool's own site isn't in the top quartile for the discipline they're selling, ask them why.

I'd be genuinely curious to see numbers from anyone else who runs this. Post scores in the comments, mine included. If foglift.io ever drops below 88 on AEO, I'd want to know first.

Links: 240-site AI-search-readiness study · Foglift docs · Foglift

How to Audit Your Site's AI Search Visibility in 30 Minutes (with a Free CLI)

Watson Foglift — Sat, 18 Apr 2026 11:13:22 +0000

Your site probably ranks fine on Google. How does it look when ChatGPT or Perplexity read it?

Different question. Different answer. Google ranks pages. LLMs extract and cite passages, and the signals they care about (schema richness, FAQ coverage, heading clarity, entity disambiguation) aren't on the average SEO checklist.

We built a CLI to audit that. This post is a 30-minute walkthrough: install it, scan your site, scan some competitors, and wire a quality gate into CI. All the data below is from scans I ran while writing this post.

What "AI search visibility" actually measures

When you audit a page for traditional SEO, you're optimizing for one thing: will Google rank this URL for a query. The inputs are Core Web Vitals, backlinks, keyword targeting, crawlability.

AI search is a different extraction problem. An LLM-powered answer engine (ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini) wants to pull a specific passage out of your page and cite it, often alongside 3-5 other sources. The signals that matter there are different:

Structured data richness (JSON-LD, not microdata)
FAQ coverage with FAQPage schema
Heading clarity — can the model segment your page into answerable chunks
Entity identity — can it disambiguate your brand from noise
Content depth and authority — citations, data, original research
Citation formatting — do you make it easy to quote you
Topical authority — is your site a source the model has seen cited elsewhere
AI crawler access — is GPTBot, ClaudeBot, PerplexityBot actually allowed in your robots.txt

The usual name for optimizing these signals is AEO (Answer Engine Optimization) or GEO (Generative Engine Optimization). The CLI we'll use scores both, plus traditional SEO on the side.

Install

npm install -g foglift-scan

No account needed for the basic scan command. Everything in the first half of this post runs without authentication.

foglift --version

Scan your own site

Point it at a URL:

foglift scan https://foglift.io

You get a color-graded scorecard in your terminal:

  foglift scan results for foglift.io

  Overall    ██████████  96/100  A
  SEO        ██████████  100/100  A
  GEO        ██████████  100/100  A
  AEO        █████████░  88/100  B
  Perf       █████████░  89/100  B
  Security   ██████████  100/100  A
  A11y       ██████████  100/100  A

  Top Issues:
  ⚠ 14 external scripts (Performance)
  ⚠ 1 render-blocking scripts (Performance)

The seven rows are the axes. SEO is the classic bundle. GEO is "is this site structurally readable by a generative engine." AEO is "is this specific page extractable by an answer engine." Perf, Security, A11y are what they sound like.

The B on AEO here is honest: we still ship 14 external scripts on the homepage and one of them is render-blocking. That's an active todo, not a bragging number.

Add --json for anything scriptable

If you want to pipe results to jq, log them to a dashboard, or post them to a Slack channel, use --json:

foglift scan https://foglift.io --json | jq '.scores'

{
  "overall": 96,
  "seo": 100,
  "geo": 100,
  "aeo": 88,
  "performance": 89,
  "security": 100,
  "accessibility": 100
}

The full JSON payload includes every issue, its category, severity, and a one-line description. That's the entire audit as structured data, which is the point.

The spicy part: scan the SEO giants

Here's where it gets interesting. I ran the same command against the three biggest names in the SEO tooling space:

foglift scan https://ahrefs.com --json | jq '.scores'
foglift scan https://moz.com --json | jq '.scores'
foglift scan https://semrush.com --json | jq '.scores'

The numbers:

Site	Overall	SEO	GEO	AEO
ahrefs.com	81	100	90	54
moz.com	80	100	90	65
semrush.com	78	100	90	52

These are the companies that literally sell the tools everyone uses to rank on Google. Their SEO scores are a flawless 100. Their AEO scores are 52 to 65.

Two things to pull out of that.

First, SEO 100 and AEO 54 on the same page is not a contradiction. It's the whole thesis: the signals that win Google are not the signals that get you extracted into a ChatGPT answer. A site can be a textbook SEO execution and still be opaque to an LLM that's trying to pull a citation.

Second, if Ahrefs and Semrush haven't retrofitted their marketing site for this yet, the gap is probably everywhere. In the 240-scan audit we ran earlier this year, the median AEO score was 46, and only 10% of sites scored above 80. The tooling giants aren't outliers on the low side. They're roughly average.

Interpreting your top issues

The topIssues block in the JSON is where your actual todo list lives. A typical output for a site that scores in the 50s on AEO looks like:

{
  "topIssues": [
    { "category": "GEO", "title": "No FAQ section", "description": "Add FAQPage schema for AI extraction.", "severity": "warning" },
    { "category": "AEO", "title": "Missing Article schema", "description": "LLMs cite more reliably when Article JSON-LD is present." },
    { "category": "AEO", "title": "Low heading density", "description": "Break long sections with H2/H3 to improve extractability." }
  ]
}

Each issue is a concrete edit. "No FAQ section" means add FAQPage JSON-LD on a page where users actually ask questions. "Missing Article schema" means wrap blog posts in Article JSON-LD with author, datePublished, dateModified. "Low heading density" means the model can't segment your page into citable answers, so break it into named sections.

The issues are deliberately concrete because the fixes are concrete. There's no "improve your E-E-A-T" mush in there.

Scan competitors as a batch

If you're auditing a landscape and not just one URL, batch mode runs up to 10 scans in one call:

foglift scan batch \
  https://foglift.io \
  https://ahrefs.com \
  https://moz.com \
  https://semrush.com \
  --json > competitors.json

(Batch mode is the one thing in this post that requires an API key. It's free to generate one.)

Pipe it through jq to get a sortable table:

jq -r '.[] | [.url, .scores.aeo, .scores.overall] | @tsv' competitors.json

Now you have a leaderboard. If you're on the wrong end of the leaderboard, you have a reason to care about the top issues.

Wire it into CI (the real payoff)

The scorecard is interesting once. What makes it useful is making regressions visible.

foglift scan has a --threshold=N flag that exits 1 if the overall score drops below N. That's all you need for a CI gate:

# .github/workflows/ai-audit.yml
name: AI Search Audit

on:
  pull_request:
    branches: [main]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Foglift CLI
        run: npm install -g foglift-scan
      - name: Audit production URL
        run: foglift scan https://example.com --threshold=85

If a PR ships changes that drop the production score below 85, the job fails. Same pattern as Lighthouse CI, same pattern as eslint --max-warnings 0, same pattern as any other quality gate you're already running.

For local dev, I run this as part of a pre-release check:

foglift scan https://staging.example.com --threshold=80 \
  && echo "AI audit passed" \
  || { echo "AI audit failed — check issues before shipping"; exit 1; }

What to do with the results

Three realistic next steps, in order of leverage:

Add FAQPage schema to your 10 most-trafficked content pages. This usually moves AEO the most for the least effort. Write actual questions, answer them in 40-80 words each, wrap in JSON-LD.
Make sure GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed in robots.txt. A surprising number of sites accidentally block them, then wonder why they're not cited.
Add Article schema to every blog post with author, datePublished, and dateModified. Freshness signals matter more for answer engines than they do for Google, because LLMs are trying to avoid citing stale answers.

After you ship those, rerun the scan. The delta is the point: you want a line on a graph that trends up.

Wrap

The CLI is open, the scans are free, and the gate is a single flag. If you work on a site and the AEO number comes back under 70, you now have a ranked list of things to fix and a way to stop it from regressing.

If you want to go deeper, foglift scan ai-check runs your URL against a set of target prompts across ChatGPT, Perplexity, Claude, and Gemini and shows you which ones actually cite you today. That's the ground-truth measurement — the scorecard above is the leading indicator, the ai-check is the lagging one. Both useful, neither redundant.

The 30 minutes is: install, scan your site, scan two competitors, read the top issues, wire the threshold into CI. That's a full first loop. The second loop is shipping one of the fixes and watching the number move.

CLI source and docs at foglift.io/developers. All scores in this post were captured on 2026-04-18 and will drift.

We Shipped 127 Programmatic Landing Pages. We Deleted 122 of Them Three Weeks Later. Here's What the Data Told Us.

Watson Foglift — Fri, 17 Apr 2026 11:11:08 +0000

In early March I shipped 127 vertical landing pages on our SaaS site. /for/plumbing, /for/funeral-home, /for/ice-cream-shops — one for every industry an agent could invent. The argument for it was the argument every programmatic SEO post makes: cast a wide net, own the long tail, ship faster than competitors.

On March 25 I deleted 122 of those pages in a single commit. Kept five: agencies, SaaS, ecommerce, startups, enterprise. The others collapsed to a 301 redirect pointing at /for/.

This is a short write-up of what the data said, because I think the programmatic SEO playbook is being sold harder than ever at exactly the moment it's getting worse, not better, at its actual job.

What 127 pages actually looked like

Each page was ~150 lines of JSX. Same hero component. Same three feature cards. Same CTA. The vertical-specific content was roughly four paragraphs injected from a template dictionary. The word "plumbing" or "veterinary" appeared maybe nine times. The structural depth was identical across every page.

From a human-reader perspective this was thin. From a Google-crawler perspective it was thin with extra steps. From an AI-search perspective — which is the channel I was actually trying to optimize for — it was worse than thin, because AI engines penalize duplicate patterns far more aggressively than Google's ranker does. More on that below.

The signal that it was wrong

Three data points, in order of severity:

1. None of the 122 templated pages appeared in AI citations. Our in-house CLI queries ChatGPT, Perplexity, Claude, and Gemini against prompts like "GEO tools for SaaS" and "AI search for agencies." When we baselined citations after the prune, we were cited zero times across 28 prompt-engine combinations. Not a great number — but notably, none of the then-still-live templated vertical pages were in the cited set during the runs we did before the prune either. Zero demand for them from the engines, even when the vertical keyword was in the prompt.

2. Google indexed them but didn't rank them. Coverage hit near-full within two weeks. Clicks stayed at zero. This is the classic thin-content failure mode Google's Helpful Content Update (HCU) was built to penalize. HCU has shipped in multiple waves (September 2023, March 2024, and the core update of March 2025) and each wave demotes sites where a material fraction of URLs score low on "created primarily to attract search engine traffic." We were a poster child.

3. Our depth signals were being diluted. Aggarwal et al. (KDD 2024) measured LLM citation probability against content-depth signals across thousands of URLs and found a 33.36% citation lift from adding statistics and structural depth to target pages. The same result in reverse: stripping depth (or drowning good pages in templated ones) is a real, measurable loss. When you add 122 template-structured URLs to a sitemap of ~240, you shift the domain's average structural depth downward. Entity-level authority scoring — the way AI engines actually reason about sources — penalizes that.

Why AI engines punish programmatic content harder than Google does

This is the piece I didn't understand when I shipped the 127 pages and wish I had.

Google's ranker scores pages mostly independently. A thin URL can coexist with strong URLs on the same domain because PageRank and relevance are computed per-URL. Penalties cascade through algorithms like HCU but the unit of evaluation is still largely the page.

AI engines don't work that way. When an AI engine decides whether to cite a domain, it's doing something closer to entity-level reasoning over the training corpus:

Source-diversity check: how many different-looking pages does this domain contribute? A domain with 120 duplicate-structured pages looks like one page repeated 120 times, not 120 pages.
Authority-signal pooling: the citations, external mentions, and reputation signals for a domain get aggregated at the brand level, then diluted across the volume of URLs. Adding 122 low-authority URLs dilutes the per-URL authority of the 5 good ones.
Pattern suppression: models learn to distrust patterns that look like SEO spam during RLHF and alignment training. OtterlyAI's 2025 analysis of 100M+ AI-search citations found 94% of cited content was long-form and structurally distinct. Templated verticals fail both tests at once.

The net effect: programmatic SEO at scale actively suppresses your AI-search visibility. It's not neutral. It's negative.

What we kept and why

The 5 surviving verticals (agencies, saas, ecommerce, startups, enterprise) survived for one reason: they had enough real material to be genuinely different from each other. An agency's use case (multi-client reporting, white-label) has no structural overlap with an enterprise use case (procurement, SAML, SOC 2). The plumbing vs funeral-home distinction was cosmetic — the product value prop is identical across them.

After the prune:

sitemap dropped by 122 entries
crawl budget re-concentrated on the pages we actually care about
blog-post AEO scores moved up 2–5 points across the twelve pillar posts over the subsequent weeks as we layered in TOCs, FAQPage schema, and data tables (correlation with the prune, not proof — but directionally encouraging)

The broader lesson

Programmatic SEO isn't dead. It works when each page has a real, structurally-distinct answer to a real question. It's a content strategy, not a URL-generation strategy.

What's dead is programmatic SEO for AI search, if by "programmatic" you mean "127 pages from a template." AI engines are trained specifically to demote that pattern. The playbook that was borderline in 2022 is actively counterproductive in 2026.

The honest version of the workflow I'd recommend now:

Identify the 5–10 verticals or use cases where your product genuinely has a different answer.
Write those pages by hand. Cite real sources. Include real numbers.
If you want breadth, put it in a glossary or a single pillar page that covers many terms at depth, not 100 URLs that each cover one shallowly.
Measure per-URL AI citation, not just Google indexation. If a page isn't getting cited by AI engines 30 days after launch, it probably isn't earning its slot in your sitemap either.

Sources

Aggarwal et al., "Geo-Optimization: Ranking Factors in Generative Search." KDD 2024.
OtterlyAI, "State of AI Search Citations 2025." Analysis of 100M+ AI citation instances.
Google Search Central: Helpful Content Update rollouts, 2023–2025.
Google March 2025 Core Update documentation.

If you're running a programmatic SEO experiment right now and your AI citation rate is flat, I'd be curious what your data looks like. We measured ours with foglift scan ai-check — open-source CLI, queries five AI engines, tells you where you're cited and where you're not. The honest version of the answer is usually "not where you thought."

41% of YouTube Videos Cited by AI Search Have Under 1,000 Views

Watson Foglift — Thu, 16 Apr 2026 04:06:54 +0000

Everyone's optimizing blog posts for AI search. Meanwhile, YouTube is quietly eating the citation graph, and the videos getting cited look nothing like what you'd expect.

The numbers are hard to ignore

BrightEdge analyzed 30 million sources across ChatGPT, Google AI Overviews, Google AI Mode, Perplexity, and Gemini from May 2024 to September 2025. YouTube is cited 200x more than any other video platform. Vimeo, TikTok, Dailymotion, and Twitch each hold 0.1% or less.

YouTube now commands roughly 39.2% citation share across AI platforms, up from 18.9%. In the same period, Reddit's share dropped from 44.2% to 20.3%.

In Google AI Overviews specifically, YouTube is the #1 cited domain at 29.5% share, ahead of Mayo Clinic (12.5%). How-to video citations jumped 651%.

The counterintuitive part: popularity metrics don't predict citation

OtterlyAI published the first large-scale YouTube citation study in March 2026, derived from over 100 million AI citation instances across six AI search platforms. They analyzed every YouTube URL that appeared as a citation, collecting metadata on format, duration, structure, views, likes, and channel attributes.

The finding that should change how you think about video:

40.83% of AI-cited videos had fewer than 1,000 views
36% had fewer than 15 likes
Views, likes, and subscriber count showed near-zero correlation with citation frequency

This breaks a core assumption in content marketing. On YouTube's own algorithm, views and engagement are everything. For AI citation, they're noise.

What actually drives AI citation: structure, not entertainment

If popularity doesn't predict citation, what does? OtterlyAI's data points to structural signals:

Long-form dominates. 94% of YouTube AI citations go to long-form videos. Shorts account for just 5.7% of citations. The sweet spot:

Duration	Citation Share
10-20 minutes	32.1%
5-10 minutes	26.1%
20+ minutes	17.6%
Under 5 minutes	18.5%
Shorts	5.7%

Timestamps function like headers. Videos with chapter markers and timestamps give AI models the equivalent of an H2/H3 structure to parse. A 15-minute video with 6 timestamped sections is, to an LLM, structurally similar to a well-organized blog post.

Descriptions function like metadata. Keyword-aligned descriptions with clear topic framing give AI models the extraction signals they need. A structured video with a descriptive title, clear chapters, and a keyword-aligned description can be cited regardless of view count or channel size.

Why this matters if you build software

Three implications for devs and SaaS founders:

1. Video is an independent AI discovery channel. If you only optimize text content for AI search, you're ignoring the #1 cited domain in AI Overviews. A 12-minute tutorial about your problem space, structured with timestamps and a detailed description, has a real shot at AI citation even with zero subscribers.

2. The playing field is level in a way blog SEO isn't. Blog SEO rewards domain authority, backlink profiles, and years of accumulated trust signals. YouTube AI citation rewards structure and topical relevance. A new channel with 50 subscribers can get cited if the content is structured for extraction.

3. "How-to" content is the highest-leverage format. How-to citations grew 651% in AI Overviews (BrightEdge, 2025). If your product solves a technical problem, a well-structured walkthrough video explaining the problem space (not a product demo) is high-value content for AI discovery.

What this doesn't mean

This isn't "pivot to video." It's "video is an underpriced AI citation asset, and the structural bar is lower than you think."

The data also doesn't say views are bad. Views still matter for YouTube's own recommendation engine, for brand awareness, for direct traffic. The point is narrower: when AI models decide which sources to cite, they appear to weight structural extractability over popularity signals.

Sources

BrightEdge, "YouTube Presence in AI Search" (2025). Analysis of 30M sources across 5 AI platforms, May 2024 - Sep 2025.
OtterlyAI, "YouTube Citation Study 2026" (March 2026). 100M+ AI citation instances across 6 platforms, 30-day observation window.
Search Engine Land, "YouTube dominates AI search with 200x citation advantage" (2025).
Search Engine Land, "YouTube citations in Google AI Overviews surge" (2025).
Search Engine Land, "AI search engines cite Reddit, YouTube, and LinkedIn most: Study" (2026).

10 'Best GEO Tools' Listicles Exist. We're in Zero. Here's What That Teaches About AI Citations.

Watson Foglift — Wed, 15 Apr 2026 01:17:49 +0000

I Googled "best GEO tools 2026" today. There are at least 10 listicle articles comparing generative engine optimization platforms — from StartupTalky, SitePoint, Birdeye, Evertune, Bluefish, Ecomtent, Bear AI, AtomicAGI, and others.

We build a GEO tool. We're in zero of them.

This is interesting because we also run AI Visibility Checks against ourselves weekly. And across 7 different prompts and 4 AI engines, zero mention us. These two facts are not a coincidence.

The listicle → AI citation pipeline

Here's the chain most people miss:

Someone writes a "best GEO tools 2026" article
That article gets indexed by Google and crawled by AI bots
When a user asks an AI engine "what are the best GEO tools?", the model references those listicles as training/retrieval data
The tools IN the listicles get recommended. The tools NOT in them don't exist as far as the AI is concerned.

This isn't speculation. A Position.digital analysis of AI SEO statistics found that domains with profiles on review platforms like G2, Capterra, and Trustpilot have 3x higher citation rates from ChatGPT. Listicles and review sites are the supply chain for AI recommendations.

We tested this directly

We ran our own AI Visibility Check with prompts like "best AI search optimization tool" and "GEO tools for SaaS" across ChatGPT, Perplexity, Claude, and Gemini.

ChatGPT recommended agencies (Zupo, iPullRank, First Page Sage) and Yext. Perplexity cited Goodie AI, Profound, Gauge, AthenaHQ. Claude referenced BrightEdge and DirectAgents.

Every one of those tools appears in multiple listicle articles. We appear in none. The correlation is obvious once you see it.

What the listicle winners have in common

I looked at what Goodie AI, Bluefish, Gauge, and AthenaHQ — the tools that dominate GEO listicles — do differently:

They're on review platforms. G2, Capterra, TrustRadius. This creates structured profile data that AI engines can parse cleanly. A G2 profile with 3 reviews beats a technically perfect landing page with zero third-party validation.

They get mentioned in industry publications. SearchEngineLand, Search Engine Journal, MarTech. One mention in SEL probably outweighs 50 blog posts on your own domain. AI engines weight third-party mentions at roughly 35% of total citation factors (SE Ranking, 129K domain study).

They have PR or outreach budgets. The listicle articles don't write themselves. Someone from Bluefish or Goodie AI pitched those writers, provided demo access, shared case studies. This isn't organic — it's intentional distribution.

They showed up early. The first "best GEO tools" articles set the template. Later articles reference earlier ones. If you missed the initial wave, you're fighting to get added to existing listicles instead of being included by default.

The uncomfortable math for bootstrapped tools

Here's where it gets real for anyone building a dev tool or SaaS without a marketing budget:

32K+ referring domains = 3.5x more likely to be cited by ChatGPT (Position.digital). Most bootstrapped tools have under 200.
Significant Reddit/Quora presence = ~4x citation boost. "Significant" means hundreds or thousands of mentions, not 9 comments.
Review platform profiles = 3x citation boost. But getting reviews requires customers, which requires visibility, which requires... reviews. It's circular.

The bootstrapped builder's dilemma: you need authority to get visibility, but you need visibility to get authority. Traditional SEO had this problem too, but the gap was smaller because you could rank for long-tail keywords without massive domain authority. AI search doesn't have a long tail — it either recommends you or it doesn't.

What we're actually doing about it

We're not sitting around hoping. But honesty: the playbook for bootstrapped AI visibility is thin.

Building citable content. We published original research (240 website scans, data nobody else has). Our post debunking the "44% AI citation lift" stat is one of the few honest treatments online. This earns links from people who actually check sources.

Community presence before product mentions. 20 Indie Hackers comments, 9 Reddit replies, 9 Dev.to articles — all sharing data and insights, not marketing. The 4x Reddit multiplier only works with authentic engagement.

The directory and listicle push. We've submitted to 6 AI tool directories (5 pending review). Getting on G2 and Product Hunt is next. This is the explicit gap.

Tracking everything. We re-run AI Visibility Checks weekly. Right now: 0/28. The goal is to see that first mention. When it happens, we'll know exactly what caused it because we're documenting every action.

The takeaway

If you're building a product and wondering why AI engines don't recommend it, check the listicles first. Search "[your category] tools 2026" and count how many comparison articles you appear in.

If the answer is zero, no amount of schema markup, content optimization, or technical readiness will fix it. The AI citation pipeline starts with third-party mentions — listicles, review sites, community discussions, industry publications. Your website is downstream.

We went from AEO 65 to 89 across our blog. Technical score is 96/100. AI visibility is still 0/28. The technical optimization is the floor, not the ceiling. Getting INTO the articles that AI engines reference — that's the real game.

Sources:

Position.digital, "100+ AI SEO Statistics for 2026," April 2026
SE Ranking, "AI Search Ranking Factors," 129K domain study, 2025
SearchEngineLand, "AI search engines cite Reddit, YouTube, LinkedIn most," 2026
Chatoptic, "AI Citation vs. Google Rank Correlation," 2025

Why 'X vs Y' Pages Are the Most Underrated Content Type for AI Search

Watson Foglift — Mon, 13 Apr 2026 00:10:00 +0000

Most AI search optimization advice boils down to: write deep blog posts, cite sources, add FAQ schema, keep it fresh.

That's solid advice. But there's a content type that hits every AI readiness signal simultaneously — and almost nobody is building it deliberately.

Comparison pages.

We built 43 of them over the past month. Our top comparison pages score 81/100 on AI readiness. The industry median? 46/100.

Here's why the "X vs Y" format is structurally ideal for AI extraction — and why most teams are leaving this on the table.

The Gap Is Massive

We scanned 240 websites to measure AI search readiness across eight dimensions. The findings were stark:

Median AEO score: 46/100 — most sites aren't built for AI engines
90% of websites score below 80 on answer engine optimization
34.6% fail the AEO category entirely (below 50)

Meanwhile, our comparison pages — after adding FAQ schema and peer-reviewed citations — hit 81/100. That puts them in the top 10%.

What makes comparison pages structurally different from blog posts?

Six Reasons Comparison Pages Win

1. They're Table-Native

AI models extract structured information more reliably from tables than from prose. Feature comparison matrices, pricing breakdowns, pro/con lists — these are the native content format of a comparison page.

The data backs this up: articles with 19+ statistical data points get 93% more AI citations than those without (SE Ranking, 2025, 129,000 domains analyzed). Comparison tables are statistics-dense by nature.

2. FAQ Schema Fits Naturally

FAQPage schema correlates with 2.7x more AI citations (Relixir, 2025). But bolting FAQ schema onto a blog post often feels forced — "What is [topic]?" questions that don't add real value.

Comparison pages generate natural FAQ content:

"Is X better than Y for enterprise use?"
"What's the pricing difference between X and Y?"
"Does X support [specific feature] that Y doesn't?"

These are the exact questions users type into AI engines. When the page structure matches the query structure, AI models can extract and cite with confidence.

3. Heading Hierarchy Is Built In

The comparison format forces clean structure: H1 for the main comparison, H2 for each evaluation category (Features, Pricing, Use Cases, Verdict), H3 for subcategories. AI models parse this hierarchy to understand what's being compared and why.

No agonizing over content architecture. The format does it for you.

4. They Match Decision Intent Directly

When someone asks an AI "which is better, X or Y?", the model needs a source that directly answers that question. A blog post might mention both products in passing. A comparison page is purpose-built for the exact query.

This matters because AI-referred visitors convert at 4.4x the rate of standard organic traffic (Spiralyze, 2025). Decision-intent queries convert even higher — the user is already choosing, they just need help deciding.

5. You Compete Against Nobody

Here's the strategic insight most teams miss: nobody else is targeting your brand comparisons.

If you build a "Your Product vs Competitor X" page, you're the primary authority for that query. You know your own product better than anyone. You have the most current data. You can provide the most honest comparison.

Blog posts compete in crowded topical spaces where established players have years of authority built up. Comparison pages compete in spaces you already own.

6. Freshness Is Easy to Maintain

Content updated within 30 days gets 3.2x more AI citations than stale content (Digital Bloom, 2025). But keeping a 3,000-word blog post fresh is a project. Keeping a comparison page fresh is a data update — new pricing, new features, new verdict.

Our comparison pages get monthly data refreshes. Each update takes 15 minutes. Each refresh resets the freshness signal that AI models weight heavily.

The Content Depth Trap

Here's the nuance that separates an 81 from a 41: schema alone doesn't do it.

We tested this directly. One of our comparison pages had FAQ schema but thin content — a templated shell without genuine analysis. AI readiness score: 41/100.

Two other pages with the same FAQ schema implementation but with deep content — real pricing research, honest trade-offs, specific recommendations backed by methodology citations — scored 81/100.

That's a 40-point gap from content depth alone, with identical schema.

The lesson: don't template your way to comparison page coverage. Each page needs:

Real competitor research. Test their product. Get current pricing. Note actual feature differences.
Honest trade-offs. "Choose them when you need X" is more trustworthy than "we're better at everything." AI models trained on diverse sources can detect one-sided comparisons.
Cited methodology. Why should an AI engine trust your comparison? We cite 4 peer-reviewed studies in our methodology section (Aggarwal et al., KDD 2024; SE Ranking 129K domain study; Chatoptic correlation analysis; Zyppy freshness data).

What Google Rank Tells You (Nothing)

One more data point that makes comparison pages strategically interesting:

The correlation between Google ranking and ChatGPT citation is 0.034 — essentially zero (Chatoptic, 2025, 1,000 queries analyzed).

This means a comparison page that ranks nowhere on Google can still get cited by ChatGPT, Perplexity, or Claude if it has the right structural signals. You don't need to outrank Ahrefs' blog to get your comparison page cited — you need to be the most extractable, most trustworthy source for the specific comparison query.

Google rewards backlinks and domain authority. AI engines reward structure, depth, and citation-worthiness. Comparison pages optimize for the latter without needing the former.

Blog Posts vs. Comparison Pages

Signal	Blog Post	Comparison Page
Tabular data	Optional — must add	Built-in
FAQ schema fit	Often forced	Natural Q&A format
Question-format headings	Must design deliberately	Inherent to format
Decision-intent matching	Indirect	Direct
Content freshness cost	Full rewrite or audit	Data update (15 min)
Query competition	High	Near-zero (brand queries)
Content depth requirement	2,900+ words for citation lift	Naturally deep from feature/pricing analysis

This doesn't mean blog posts are bad. They build topical authority, earn backlinks, and support the full funnel. But if you're only building blog posts for AI search, you're leaving the highest-ROI content type on the table.

How to Start

If you want to test this:

Pick your top 3 competitors. Build one comparison page per competitor.
Research deeply. Test their product. Get real pricing. Note real differences.
Add FAQPage schema with 5 questions per page — real questions users would ask.
Cite your methodology. Link to the studies behind your evaluation framework.
Be honest. Include "choose them when..." recommendations for the competitor.
Measure baseline. Run an AI readiness scan before and after.
Set a monthly refresh cadence. Update pricing, features, and verdicts.

Then measure. If your experience is anything like ours, comparison pages will outperform your blog on AI readiness metrics from day one.

We built Foglift to measure exactly this — how ready your site is for AI search engines. The free scan scores pages across the 8 AI readiness dimensions mentioned in this post. The comparison page data here comes from running our own tool on our own site.

We Scored 88/100 on AI Readiness. Zero AI Engines Mention Us.

Watson Foglift — Sun, 12 Apr 2026 00:08:16 +0000

We built an AI search optimization tool. We used it on our own site. We scored 88/100 on AI readiness — structured data, FAQ schema, fresh content, cited sources, proper heading hierarchy, the works.

Then we asked 5 AI engines 7 different questions where our tool should logically appear. Zero mentions across 35 checks.

Not low visibility. Not partial visibility. Zero.

Here's what we learned about the gap between being ready for AI search and actually getting cited by AI search.

The setup: what "88/100" actually measures

Our AI readiness score evaluates 8 technical dimensions:

Dimension	What it checks
Structured Data Richness	JSON-LD schemas (FAQPage, Article, Organization, etc.)
Heading Clarity	H1-H3 hierarchy, semantic heading structure
FAQ Quality	Visible FAQ content matching schema markup
Entity Identity	Brand entity data, consistent NAP, knowledge panel signals
Content Depth	Word count, semantic coverage, topical comprehensiveness
Citation Formatting	Inline citations, sources sections, attribution
Topical Authority	Internal linking, content clusters, expertise signals
AI Crawler Access	robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, etc.

We scored well on all of them. Our robots.txt allows all major AI crawlers. We have FAQPage schema on 40+ pages. Every blog post has a Sources & Further Reading section with academic and industry citations. Content is freshly updated (every post modified within the last 30 days).

By every technical measure, we're doing the right things.

The test: 0 out of 35

We ran our own AI Visibility Check — the same feature we sell to customers — against ourselves. We queried 5 AI engines (ChatGPT, Perplexity, Claude, Gemini, Google AI Overview) with 7 prompts:

"best AI search optimization tool"
"how to rank in ChatGPT"
"GEO tools for SaaS"
"track brand mentions in AI search"
"tools to optimize for Perplexity"
"best AEO tools 2026"
"AI visibility monitoring platform"

Result: 0/7 prompts, 0/35 engine checks. Not one AI engine mentioned us for any query.

Who did show up?

This is the interesting part. The AI engines didn't return nothing — they returned other tools and companies:

ChatGPT recommended agencies (Zupo, iPullRank, First Page Sage) and platforms (Yext)
Perplexity cited monitoring tools (Goodie AI, Profound, Gauge, AthenaHQ)
Claude referenced content sources (DirectAgents, BrightEdge)
Google AI Overview and Gemini gave generic strategies without naming specific tools

The winners weren't necessarily better products. They were better-known products — with more third-party mentions, more backlinks, more citations in listicle articles, and stronger presence on community platforms.

The paradox, explained

Technical AI readiness and AI visibility are different things that use different signals.

Think of it like a job interview. AI readiness is your resume — well-formatted, clear structure, proper credentials. AI visibility is your reputation — who knows you, who's talked about you, whether the interviewer has already heard your name before you walked in.

The research backs this up:

Brand mentions account for ~35% of citation weight. An SE Ranking study of 129,000 domains found that brand web mentions are the single strongest predictor of whether an AI engine cites you. Not your schema. Not your heading structure. Whether other sites on the internet talk about you.

Referring domains are the strongest technical predictor. The same study found that the number of unique websites linking to you is the strongest link-based signal. AI models learn from web data — links are how the web vouches for authority.

Reddit presence gives a 3.9x citation multiplier. Sites discussed on Reddit get cited by AI engines at nearly 4x the rate of sites that aren't. AI models heavily weight community discourse as a trust signal.

28% of the most-cited domains have zero Google visibility. (Chatoptic, 2025.) These aren't SEO winners — they're authority winners. AI citation and Google ranking are almost statistically independent (correlation: 0.034).

What technical readiness actually buys you

This doesn't mean technical optimization is worthless. It's necessary — just not sufficient.

Here's the mental model: AI readiness is table stakes. Authority signals are the game.

What the 88/100 score does buy:

Extraction accuracy. A Nature Communications study (Feb 2024) found LLMs extract information more accurately from structured fields than from prose. When an AI engine does cite you, schema markup ensures it gets your brand name, pricing, and features right.
Freshness signals. Content updated within 30 days gets 3.2x more AI citations (Digital Bloom, 2025). Our dateModified timestamps are current — when authority catches up, freshness won't be the bottleneck.
Crawl access. If your robots.txt blocks GPTBot or ClaudeBot, nothing else matters. The door has to be open before anyone can walk through it.

Technical readiness is the floor, not the ceiling.

What we're doing about it

Knowing the gap is step one. Here's how we're closing it:

Earning authority through original research. We published a study analyzing 240 website scans with data no one else has. Primary research earns citations from journalists and bloggers — which builds the third-party mention graph that AI models trust.
Community presence. Genuine engagement on Dev.to, Indie Hackers, and (eventually) Reddit. Not "check out our tool" spam — substantive contributions that establish expertise. The 3.9x Reddit multiplier doesn't work if your Reddit history is all self-promotion.
Content that other sites want to link to. Our blog post calling out the unsourced "44% AI citation lift" stat that every SEO blog repeats has become one of the few honest treatments of that claim online. That kind of content earns backlinks organically because it fills a gap no one else is filling.
Getting into listicles. AI engines heavily weight "best X tools" articles when answering tool-recommendation queries. If you're not in those articles, you're invisible to those queries.
Patience. AI models retrain on web data periodically, not continuously. Even if you do everything right today, it may take weeks or months for the authority signals to propagate through the training pipeline. This isn't like SEO where you can see index updates within days.

The takeaway for your site

If you're optimizing for AI search visibility, here's the uncomfortable truth:

The technical checklist is the easy part. Schema markup, heading structure, FAQ content, fresh dates, citation formatting — these are all things you can control directly and implement in a weekend.

The hard part is earning authority. Third-party mentions, backlinks from reputable domains, community presence, being included in comparison articles — these take months of consistent work and can't be shortcut.

Don't skip the technical work. But don't mistake a high readiness score for actual visibility. We made that mistake, and 0/35 was the wake-up call.

We're building Foglift — a free website audit that measures both AI readiness and AI visibility. The readiness score is the floor. The visibility data is the ceiling. We use it on ourselves every week to track whether the gap is closing.

Sources:

SE Ranking, "AI Search Ranking Factors," 129,000 domain study, 2025
Chatoptic, "AI Citation vs. Google Rank Correlation," 2025
Nature Communications, "LLM Information Extraction from Structured Data," Feb 2024
Digital Bloom, "Content Freshness and AI Citations," 2025
Seer Interactive, "AI Overview Citation Sources," 863K keyword analysis, Feb 2026
Aggarwal et al., "Generative Engine Optimization," KDD 2024

AI Citations From Google's Top 10 Dropped From 76% to 38%. Here's What Actually Drives AI Visibility.

Watson Foglift — Sat, 11 Apr 2026 03:09:42 +0000

There's a stat circulating that should worry anyone whose traffic strategy depends on Google rankings: AI Overview citations from top-10 organic pages dropped from 76% to 38% (Seer Interactive, analysis of 863K keywords, Feb 2026).

That means Google's AI Overviews are now pulling the majority of their cited sources from pages that don't rank in the traditional top 10.

If you've spent years optimizing for Google rankings, this doesn't mean your work was wasted. But it does mean the rules for getting cited by AI are different from the rules for ranking in search.

The data: Google rank barely predicts AI citation

Three independent studies paint a consistent picture:

Study	Finding
Seer Interactive (863K keywords, 2026)	AI Overview citations from top-10 pages: 76% → 38%
Chatoptic (2025)	Correlation between Google rank and ChatGPT citation: 0.034 (essentially zero)
Chatoptic (2025)	28% of the most-cited domains in AI responses had zero traditional Google visibility

That 0.034 correlation is striking. For context, a correlation of 1.0 means perfect prediction, and anything below 0.1 is considered negligible. Google rank and AI citation are, statistically, almost independent variables.

And the 28% figure is arguably the most important: more than a quarter of the domains AI engines prefer to cite have no traditional search visibility at all. These aren't SEO winners — they're authority winners.

What actually drives AI citations

If Google rank doesn't predict AI citation, what does? Five factors have the strongest empirical support:

1. Referring domains (strongest predictor)

An SE Ranking study of 129,000 domains found that referring domains — the number of unique sites linking to you — is the single strongest predictor of AI citation. This makes sense: AI models learn from web data, and links are the web's native authority signal.

2. Content position and structure

Superlines' 2026 citation analysis found that 44.2% of all LLM citations come from the first 30% of article text. AI engines extract from the top of your content, not the bottom. Front-load your strongest claims, data, and definitions.

Pages with comparison tables containing 3+ data tables earn 25.7% more citations (Superlines, 2026).

3. Freshness

Seer Interactive found that 71% of ChatGPT citations reference content published between 2023 and 2025. Digital Bloom's analysis showed pages updated within 30 days get 3.2x more AI citations than stale equivalents.

If your best content was last updated in 2023, it's losing ground to recently published competitors.

4. Statistics and original data

Aggarwal et al.'s peer-reviewed GEO study (KDD 2024) found that incorporating statistics into content increased AI visibility by 33% and adding quotations increased it by 41%. For lower-ranked sites, citing external sources boosted visibility by 115%.

AI engines favor content that makes specific, verifiable claims over content that speaks in generalities.

5. Brand web mentions

Brand mentions across the web account for roughly 35% of citation weight (SE Ranking, 129K domains). This isn't just backlinks — it's any reference to your brand on other domains. Being discussed, recommended, and referenced on third-party sites signals to AI models that you're a trusted entity.

What this means for your strategy

The practical implication: building for AI visibility is not a refinement of SEO — it's a parallel discipline. Some tactics overlap (quality content, good structure), but the ranking factors diverge significantly.

Here's a checklist based on the data:

[ ] Front-load your content. Put your best data, definitions, and conclusions in the first third of every page.
[ ] Update key pages monthly. The 3.2x freshness multiplier is real. Set a calendar reminder.
[ ] Add specific data to every claim. Not "conversion rates increase" but "conversion rates increased 23% over 6 months (Source, Year)."
[ ] Build referring domains. The hardest and highest-leverage factor. Original research, data, and tools that others want to cite.
[ ] Monitor your AI presence directly. Don't assume Google rankings translate. Query ChatGPT, Perplexity, and Claude with questions your content should answer and check if you appear.

The 76% → 38% decline is still trending downward. The window to build AI authority before your competitors do is closing.

Sources

Seer Interactive — AI Overview citations from top-10 pages dropped from 76% to 38% (863K keywords, Feb 2026); 71% of ChatGPT citations from 2023-2025 content; 46% of AI-powered interactions use integrated search
Chatoptic — 0.034 correlation between Google rank and ChatGPT citation; 28% of most-cited domains have zero Google visibility
SE Ranking — referring domains as strongest citation predictor (129,000 domain study); brand mentions = 35% of citation weight
Superlines, "AI Search Statistics 2026" — 44.2% of citations from first 30% of article text; comparison tables with 3+ tables earn 25.7% more citations
Digital Bloom, "Content Freshness and AI Citation," 2025 — 30-day update = 3.2x citation lift
Aggarwal et al., "GEO: Generative Engine Optimization," KDD 2024 — statistics +33%, quotations +41%, source citation +115% for lower-ranked sites

FAQ Schema Gets You 2.7x More AI Citations. But Not for the Reason You Think.

Watson Foglift — Fri, 10 Apr 2026 00:12:23 +0000

A 2025 Relixir study found that pages with FAQPage schema achieve a 41% AI citation rate versus 15% without — roughly 2.7x higher. That's a real number from a real study.

But here's the thing: AI models don't parse your JSON-LD as structured data. They tokenize it as raw text, the same way they'd read a paragraph.

We just added FAQ schema to 36 pages on our site. Before we did, we wanted to understand why it works — because the mechanism matters more than the correlation. Here's what we found.

The experiment that changed how I think about schema

In February 2026, SEO researcher Mark Williams-Cook ran a controlled experiment. He created a page for a fake company and embedded an address exclusively inside invalid, made-up JSON-LD schema — not in any visible page content. The schema type didn't even exist.

Both ChatGPT and Perplexity successfully extracted and returned the address.

That tells us two things:

LLMs can read JSON-LD — they tokenize it like any other text on the page.
LLMs don't parse the semantic structure of schema — they treated an invalid schema type identically to a valid one.

This is a crucial distinction. When Google processes your FAQPage schema, it parses the structure and feeds it into the Knowledge Graph. When ChatGPT reads your page, it just... reads all the text, including the JSON-LD block, as tokens.

So why does FAQ schema correlate with higher citation rates?

If LLMs don't understand schema structure, why the 2.7x difference? Four mechanisms are at play:

1. The visible Q&A content (the biggest factor)

Every good FAQ schema implementation includes a visible FAQ section on the page. That visible content — clear questions with concise answers — is exactly the format LLMs are optimized to extract. When ChatGPT is looking for "What is the difference between X and Y?", a visible FAQ section with that exact question is an easy win.

This is the mechanism that actually drives most of the citation lift. Not the JSON-LD — the content.

2. The JSON-LD as readable text

Since LLMs tokenize JSON-LD as text, your FAQPage schema becomes an additional, cleanly-formatted representation of your content. A well-structured JSON-LD block repeats your key Q&A pairs in a format that's easy for attention mechanisms to pick up on.

Think of it as giving the model a second, structured summary of your content — in the same page.

3. Google and Bing's Knowledge Graph pipeline

Fabrice Canel, Principal Product Manager at Bing, stated at SMX Munich 2025: "Schema markup helps Microsoft's LLMs understand your content." Google's Search Relations team made similar statements at Search Central Live Madrid (April 2025).

For AI Overviews and Bing Copilot specifically, schema is parsed structurally. These platforms have Knowledge Graph infrastructure that traditional LLMs don't. So FAQ schema has a direct effect on two of the six major AI answer surfaces.

4. Selection bias (the uncomfortable one)

Sites that implement FAQ schema tend to be sites that care about content quality, update frequently, and invest in SEO. The 2.7x correlation partially reflects the overall quality of sites that bother with schema — not just the schema itself. No study I've found controls for this.

What we actually built

We needed FAQ schema on 36 pages: 24 comparison pages and 12 blog posts. Here's the approach:

For comparison pages (dynamic template)

Our comparison pages use a shared template. We generate 5 FAQ items per page from the existing comparison data:

const faqs = [
  {
    question: `What is the main difference between Foglift and ${data.name}?`,
    answer: data.heroDescription,
  },
  {
    question: `How does ${data.name} pricing compare to Foglift?`,
    answer: `${data.name} starts at ${data.competitorStartPrice}. 
             Foglift offers a free plan with full website audits, 
             then paid monitoring from $49/month.`,
  },
  // ... 3 more questions generated from page data
];

const faqSchema = {
  "@context": "https://schema.org",
  "@type": "FAQPage",
  mainEntity: faqs.map((faq) => ({
    "@type": "Question",
    name: faq.question,
    acceptedAnswer: {
      "@type": "Answer",
      text: faq.answer,
    },
  })),
};

Key decisions:

Generate from existing data — no hardcoded FAQ text. If pricing changes, the FAQs update automatically.
5 questions per page — enough for depth, not so many that it feels like keyword stuffing.
Plain text answers — strip HTML before injecting into JSON-LD.

For blog posts (static per-post)

Each blog post gets a hand-written faqJsonLd constant with 4 Q&As specific to the post's topic:

const faqJsonLd = {
  "@context": "https://schema.org",
  "@type": "FAQPage",
  mainEntity: [
    {
      "@type": "Question",
      name: "How do AI search engines decide which websites to cite?",
      acceptedAnswer: {
        "@type": "Answer",
        text: "A 2025 SE Ranking study of 129,000 domains found that brand web mentions are the strongest predictor (35% weight), followed by referring domains, content freshness, and content depth."
      }
    },
    // ... 3 more with specific data
  ]
};

Key decisions:

Data-backed answers only — every FAQ answer cites a specific source with sample size and year.
4 per post — we tried more, but after 4 the quality drops and answers start restating each other.

The visible section (this is the part that actually matters)

Both implementations render a visible accordion FAQ section that matches the schema:

<h2>Frequently Asked Questions</h2>
{faqJsonLd.mainEntity.map((faq, i) => (
  <details key={i} open={i === 0}>
    <summary><h3>{faq.name}</h3></summary>
    <p>{faq.acceptedAnswer.text}</p>
  </details>
))}

We use details/summary instead of custom accordion components:

Zero JavaScript — works with SSR/SSG
Semantic HTML — details has built-in accessibility
First item open by default — gives crawlers immediate visible content

What we measured

Before adding FAQ schema + visible FAQ sections, our AEO (Answer Engine Optimization) scores looked like this:

Page type	Count	AEO score before
Comparison pages	24	41-61
Blog posts	12	63-66
Homepage	1	88

The homepage scored highest because it already had structured FAQ content. The comparison pages scored lowest because they had minimal structured data.

After the upgrade, we're waiting on a deploy to measure the after. Based on the research, here's what we expect:

AEO improvement: We expect comparison pages to jump from 41-61 to the 75-85 range.
AI citation probability: Too early to measure directly. Our AI Visibility Check baseline shows 0/35 engine checks mentioning us — so we'll know if it moves.
What we DON'T expect: A 44% citation lift from schema alone. (If you're curious why, I wrote about that.)

The takeaway

FAQ schema works. The 2.7x correlation is real. But the mechanism is:

Visible Q&A content is what LLMs actually extract (biggest effect)
JSON-LD gives LLMs a second text representation of your key Q&As (smaller but real effect)
Google/Bing Knowledge Graph parses schema structurally for AI Overviews (platform-specific effect)
Selection bias inflates the correlation (unmeasured confounder)

If you only add the JSON-LD without visible FAQ content, you're capturing effects #2 and #3 but missing #1 — which is the largest factor. If you only add visible FAQ content without schema, you get #1 but miss #2 and #3.

The move is both layers. That's what we built.

We built Foglift to measure exactly this kind of thing — AEO scores, AI visibility, and the gap between your SEO readiness and your AI search readiness. The free scan shows you where your FAQ, schema, and content depth stand.

Sources

Relixir (2025) — FAQPage schema citation rate study: 41% vs 15% citation rate
Mark Williams-Cook (February 2026) — Controlled experiment on LLM JSON-LD tokenization
Fabrice Canel, Bing Principal PM, SMX Munich 2025
Google Search Central Live Madrid, April 2025
Dunn et al., Nature Communications, February 2024
Aggarwal et al., "Generative Engine Optimization," KDD 2024

We Scanned 240 Websites for AI Search Readiness. Your SEO Score Doesn't Predict Your AI Score.

Watson Foglift — Thu, 09 Apr 2026 00:07:26 +0000

We built a free website audit tool that scores sites across SEO, GEO (Generative Engine Optimization), AEO (Answer Engine Optimization), security, performance, and accessibility. After 240 real scans from March-April 2026, one pattern jumped out:

Sites that ace traditional SEO are often failing at AI search readiness.

Here's the data.

The 39-Point Gap

Across 240 scans, here are the median scores by category:

Category	Median Score
Accessibility	86.5
SEO	85
GEO Readiness	85
Performance	69
AEO	46
Security	30

SEO median: 85. AEO median: 46. That's a 39-point gap.

The sites in our dataset generally have solid traditional SEO — clean title tags, meta descriptions, proper heading hierarchy, fast load times. But when you measure what AI answer engines actually need to extract and cite your content, most sites fall apart.

Why SEO Score Doesn't Predict AI Citation

This isn't just our data. A 2025 Chatoptic study of 1,000 queries found only a 0.034 correlation between Google search rank and ChatGPT citation likelihood. That's effectively zero.

Even more striking: 28% of the most-cited sites in ChatGPT have zero Google search visibility (Profound, 2025). AI citation is a separate channel — not an SEO side effect.

So what does predict AI citation?

According to SE Ranking's analysis of 129,000 domains, the top factors are:

Brand web mentions — 35% weight
Referring domains (backlinks) — strong correlation
Content freshness — 71% of ChatGPT citations come from 2023-2025 content (Seer Interactive, 2025). Content updated within 30 days gets 3.2x more citations (Digital Bloom, 2025)
Content structure for extraction — FAQ sections, clear headings, direct answers to questions

Notice what's not on the list: page speed scores, meta tag optimization, keyword density — the traditional SEO checklist.

The Three Things 90% of Sites Are Missing

From our 240-scan dataset, these are the most common gaps in AEO readiness:

1. Security headers (60% fail rate)

66% missing Content Security Policy
57% missing X-Frame-Options
52% missing X-Content-Type-Options

Why does this matter for AI? AI crawlers (GPTBot, ClaudeBot, PerplexityBot) respect security signals. A site with poor security headers signals lower trustworthiness. Google's Search Quality Rater Guidelines already emphasize E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness) — security is part of the Trust signal.

2. No FAQ sections (37% of sites)

FAQ pages are one of the easiest wins for AI citation. AI engines love structured Q&A because it maps directly to how users query them. The Aggarwal et al. GEO study (KDD 2024) found that adding statistics to content improved AI engine visibility by +33%, and adding quotations from authoritative sources improved it by +41%.

FAQ sections naturally lend themselves to both patterns — they frame a specific question, then answer it with data.

3. No structured data (36% of sites)

Over a third of sites have zero schema markup. While the direct causal link between schema and AI citation is still unconfirmed by most AI providers (Google and Microsoft acknowledge it; OpenAI, Perplexity, and Anthropic haven't disclosed), schema helps AI crawlers understand entity relationships — what your brand is, what you offer, how you relate to your industry.

A Nature Communications study (Feb 2024) demonstrated that knowledge graphs built from structured data improve LLM factual accuracy. More structured data means better entity extraction means more accurate citations.

The Score Distribution

Here's how the 240 sites distributed:

Score Range	% of Sites	Label
90-100	11.3%	Excellent
80-89	8.3%	Good
70-79	19.2%	Fair
60-69	28.3%	Needs Work
50-59	10.8%	Poor
Below 50	22.1%	Critical

61.2% of sites scored below 70. The largest cluster (28.3%) sits in the 60-69 range — functional for traditional search, but with significant blind spots for AI engines.

Only 19.6% scored 80+. And this is a self-selected sample of people who actively sought out an AI readiness audit. The broader web is likely worse.

What This Means for Developers

If you're building websites — for yourself or clients — the SEO checklist you've internalized is necessary but not sufficient. The sites winning AI citations in 2026 are the ones that:

Structure content for extraction — clear H2/H3 hierarchy, FAQ sections, direct answers in the first paragraph
Maintain freshness — update key pages at least monthly
Build entity identity — Organization schema, consistent brand mentions, authoritative backlinks
Secure the basics — CSP, HSTS, X-Frame-Options (these take 5 minutes to add)

The 39-point gap between SEO and AEO is an opportunity. Most of your competitors haven't noticed it yet.

Data source: 240 website scans via Foglift's free audit tool (March 14 - April 8, 2026). Full methodology and detailed findings in our research report.

External citations: Chatoptic (2025, 1,000 queries), SE Ranking (129,000 domains), Seer Interactive (2025), Digital Bloom (2025), Aggarwal et al. (KDD 2024), Profound (2025), Nature Communications (Feb 2024).

DEV Community: Watson Foglift

5 AI Crawlers Launched in 2024–2025 That Most robots.txt Guides Still Miss

The five that belong in your robots.txt

Three of them behave in ways that break the "just add Disallow" habit

1. Applebot-Extended is an opt-out signal, not a crawler

2. Meta-ExternalFetcher can ignore robots.txt on user-supplied URLs

3. CCBot blocks propagate slowly — and partially

What actually goes in robots.txt

How to verify blocks actually worked

Why the guides are stale

Every CLI Command in Our Own Blog Post Was Fabricated. Here's How We Caught Them.

The post

What was actually broken

How we caught it

The part most people miss: the schema has to be fixed too

Bonus: the copy-paste test also found two real CLI bugs

What I'd change going forward

Closing

We scored 3 GEO/AEO platforms (including ourselves). Here's the AEO and security gap.

The setup

The results (2026-04-19)

Why the AEO delta matters

How to audit your own site in 30 seconds

The uncomfortable conclusion

How to Audit Your Site's AI Search Visibility in 30 Minutes (with a Free CLI)

What "AI search visibility" actually measures

Install

Scan your own site

Add --json for anything scriptable

The spicy part: scan the SEO giants

Interpreting your top issues

Scan competitors as a batch

Wire it into CI (the real payoff)

What to do with the results

Wrap

We Shipped 127 Programmatic Landing Pages. We Deleted 122 of Them Three Weeks Later. Here's What the Data Told Us.

What 127 pages actually looked like

The signal that it was wrong

Why AI engines punish programmatic content harder than Google does

What we kept and why

The broader lesson

Sources

41% of YouTube Videos Cited by AI Search Have Under 1,000 Views

The numbers are hard to ignore

The counterintuitive part: popularity metrics don't predict citation

What actually drives AI citation: structure, not entertainment

Why this matters if you build software

What this doesn't mean

Sources

10 'Best GEO Tools' Listicles Exist. We're in Zero. Here's What That Teaches About AI Citations.

The listicle → AI citation pipeline

We tested this directly

What the listicle winners have in common

The uncomfortable math for bootstrapped tools

What we're actually doing about it

The takeaway

Why 'X vs Y' Pages Are the Most Underrated Content Type for AI Search

The Gap Is Massive

Six Reasons Comparison Pages Win

1. They're Table-Native

2. FAQ Schema Fits Naturally

3. Heading Hierarchy Is Built In

4. They Match Decision Intent Directly

5. You Compete Against Nobody

6. Freshness Is Easy to Maintain

The Content Depth Trap

What Google Rank Tells You (Nothing)

Blog Posts vs. Comparison Pages

How to Start

We Scored 88/100 on AI Readiness. Zero AI Engines Mention Us.

The setup: what "88/100" actually measures

The test: 0 out of 35

Who did show up?

The paradox, explained

What technical readiness actually buys you

What we're doing about it

The takeaway for your site

AI Citations From Google's Top 10 Dropped From 76% to 38%. Here's What Actually Drives AI Visibility.

The data: Google rank barely predicts AI citation

What actually drives AI citations

1. `Applebot-Extended` is an opt-out signal, not a crawler

2. `Meta-ExternalFetcher` can ignore robots.txt on user-supplied URLs

3. `CCBot` blocks propagate slowly — and partially