DEV Community: jf lu

I tracked AI crawlers on my new GEO site for 30 days — here's what I learned

jf lu — Fri, 28 Aug 2026 08:27:00 +0000

I launched a site focused on Generative Engine Optimization (GEO) about 2 months ago. Instead of just tracking Google rankings, I set up Cloudflare D1 logging to track every AI crawler visit.

Here's what 30 days of data shows:

Crawler breakdown:

Bingbot: 98 hits/day average (highest volume)
ClaudeBot: 18-22 hits/day (steady, consistent)
GPTBot: 8-12 hits/day (sporadic bursts)
PerplexityBot: 3-5 hits/day (small but high-quality)
Bytespider: 500+ hits/day (massive scanning, likely training data)

Key insights:

Bing is the new Google for AI. Bingbot volume dwarfs everything else. If you're optimizing for GEO, Bing Webmaster Tools is your best free analytics source.
ClaudeBot is remarkably consistent. Same crawl pattern every day, suggesting scheduled indexing rather than on-demand. If you publish content, expect 24-48hr lag before Claude "sees" it.
Bytespider (ByteDance) is aggressive. 500+ hits/day on a small site. They're clearly doing massive data collection. If you want visibility in Doubao/DeepSeek, your content needs to be crawlable.
External referrers matter more than I expected. A single Reddit post drove 8 AI crawler visits from a domain I'd never heard of (sahammurah.com). External mentions = AI discovery signals.

What I'm doing differently now:

Added llms.txt (AI-specific robots.txt equivalent)
Every article now has FAQPage + Article Schema
Answer-first paragraphs (first sentence = direct answer)
Verifiable data points with source citations

Anyone else tracking AI crawlers? What patterns are you seeing?

What GEO Work Actually Moves AI Crawlers? Our Data, and a New-Site Launch Checklist

jf lu — Wed, 19 Aug 2026 06:57:33 +0000

You are about to launch a new site and the advice keeps multiplying: add /ai/summary.json, wire up llms.txt, sprinkle FAQPage markup, build a discovery layer. But which of these does an AI crawler actually care about?
We run a Cloudflare Worker that logs every request to geo010.com and tags it by crawler, so we could answer that question with data instead of opinion. This article attributes 15 days of AI crawler traffic to nine GEO measures, shows which ones changed nothing, and turns the results into a launch checklist for a new site.
How Did We Attribute Traffic to Each GEO Measure?
Every request to geo010.com passes through a Worker that records the date, path, status, page type and a normalized crawler name into a D1 database. For each of nine GEO measures we shipped, we then asked the logs three questions: which crawlers hit the relevant paths, how often, and did visits change after the measure shipped?

Two warnings before the numbers. First, a crawler label is only a user agent — attackers spoof them, so we cross-checked paths and status before counting anything. Second, some "visits" were us: we verify deployments with curl, and our own checks show up in the logs. We filtered those out.

Which GEO Measures Actually Drove Crawler Traffic?
Robots.txt and sitemap.xml were the only paths every major AI crawler used. The homepage was the only page they all visited. Everything else showed up in a minority of crawlers — or none at all.
Here is the full attribution table, ranked by how much real crawler traffic each measure produced:

GEO measure Which crawlers Evidence (Aug 4–19) Impact
robots.txt ClaudeBot, Googlebot, OAI-SearchBot, Bingbot, FacebookBot ClaudeBot 104, Googlebot 32, OAI-SearchBot 26, Bingbot 17 Highest — nearly daily entry point
sitemap.xml ClaudeBot, Bingbot, GPTBot ClaudeBot 104, Bingbot 11, GPTBot 9 Highest — ClaudeBot reads it religiously
Homepage / GPTBot, PerplexityBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Applebot GPTBot 21, PerplexityBot 12, OAI-SearchBot 10, ClaudeBot 8 High — visited by every major crawler daily
New content pages GPTBot (primary), ClaudeBot, PerplexityBot GPTBot 121 content pages; ClaudeBot 2 community articles High — GPTBot traverses all pages
Freshness alignment All (indirect) Crawler bursts on 08-08, 08-12 and 08-17 followed freshness passes Medium — sustains re-visits
llms.txt GPTBot, Amazonbot, Googlebot 4 total requests, GPTBot 2 Low — only GPTBot treats it as an entry
Definition-first layout GPTBot, ClaudeBot, Bingbot, others Entity-density page reached by 7+ crawlers Low-medium — quality, not quantity
FAQPage markup None (AI) 0 visits from major AI crawlers after shipping None (crawl) — a citation-layer lever
AI discovery endpoints (/ai/, ai.txt, plugins) None 0 requests in 15 days None — not used by today's crawlers
The ranking surprised us. The "boring" infrastructure files — robots.txt and sitemap.xml — produced more genuine crawler traffic than every shiny AI feature combined.

Why Do GPTBot and ClaudeBot Crawl Completely Differently?
The two most important AI crawlers could not be more different in behavior, and the difference determines what to build first:

Behavior GPTBot ClaudeBot
Reads robots.txt Rarely (1 request) Religiously (104 requests)
Reads sitemap.xml Occasionally (9) Every day (104)
Visits content pages Yes — 121 pages, including a full-site pass on Aug 17 Rarely (a handful total)
Entry strategy Traverses the whole site per pass Follows robots → sitemap, then mostly home
ClaudeBot is a disciplined path-follower. It checks your robots and your sitemap almost daily, but it barely descends into content unless something in that sitemap or homepage directs it. GPTBot is a deep scanner. It skips robots, reads llms.txt occasionally, and walks the entire site — on Aug 17 it made a single pass touching 60 pages, each one once.

The practical consequence: if your robots.txt or sitemap.xml is broken, you hurt the crawler that reads them daily (ClaudeBot). If your content is shallow, you waste the crawler that reads everything (GPTBot).

What Did We Get Wrong in Earlier Analyses?
Honesty check. Earlier write-ups on this site credited FAQPage and new-article launches with immediate crawler spikes. Our per-request attribution showed those were misattributed:

"A new article got crawled 6-10 times on launch day." Actually most of those hits were browser visitors and our own checks; each AI crawler visited the new page once. New content is crawled once per GPTBot pass, not burst-spiked by the act of publishing.
"FAQPage pages were visited by ClaudeBot and OAI-SearchBot the same day." Not supported by the logs — the six pages got no visits from major AI crawlers after the markup shipped. Their visitors were browsers, Bingbot and our curl checks.
"The 08-11 traffic spike was AI growth." It was a scanner attack — ChatGPT-User and Amazonbot probing /.env.production and /telescope/requests. Same for a spoofed-GPTBot burst on 08-13.
None of those measures are useless. They just do their work at a different layer: FAQPage helps a page be quoted once AI is already considering it; llms.txt helps GPTBot orient; discovery endpoints may help future tooling. The mistake was measuring them against crawl counts at all.

What Three More Days Confirmed
When we first published this article, the data window was 12 days. Three more days (Aug 17–19) gave us a live test of the same predictions:

Date What happened What it proved
Aug 17 GPTBot hit 60 pages in a single pass — the first full-site traversal since launch Content pages are discovered via sitemap, not via AI endpoints. GPTBot never touched /ai/summary.json.
Aug 17 ClaudeBot made 23 requests: all robots.txt + sitemap.xml + homepage ClaudeBot has not changed its entry-only behavior in three weeks. Sitemap quality matters more than content quantity for this crawler.
Aug 17 Total requests 1,066; only 371 returned 200 Scanner noise persists — the same path-spam pattern (old PHP paths, /api/v1/validate/code) returns daily. Noise filtering is ongoing work.
Aug 18 Bingbot made 51 requests (highest single-day for Bing); Meta-ExternalAgent appeared for the first time with 8 requests Not all crawlers behave the same. Bingbot probes old/cached paths; Meta-ExternalAgent is a new entrant worth monitoring.
Aug 18–19 Both community articles (the crawler-data post and this one) each got 1 GPTBot visit + 1 ClaudeBot visit New content is indexed once per crawler type — not burst-on-publish. The "crawled 6-10 times on launch day" claim from earlier articles was definitely noise.
The extra data made the picture sharper, not murkier: robots + sitemap + homepage are the crawl foundation; everything else is a citation-layer optimization you add later.

What Should a New Site Build First?
If we were launching a new site today, we would build three things before anything else: a correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage that explains what the site is — plus a few genuinely useful articles.
Make robots.txt correct. Allow the AI crawlers you want, block only what you must (analytics, staging, private pages). A wrong rule that blocks everything is a silent zero.
Make sitemap.xml 100% valid. Every URL in it must return 200. A sitemap full of 404s teaches the daily visitor (ClaudeBot) that your site is broken.
Serve real HTML on the homepage. AI crawlers do not run JavaScript. The homepage must contain the answer to "what is this site?" in plain text, with links to your best pages.
Publish 3-5 real articles before launch day. GPTBot traverses everything; give it pages worth reading. Five solid pages beat fifty shells.
That is the entire pre-launch crawl work. It took us the least effort of all nine measures, and it produced the most crawler traffic.

What Should a New Site NOT Build First?
Skip these until the foundations above exist. Our logs show they contribute nothing to crawl, however good they look on a roadmap:

AI discovery endpoints (/ai/summary.json, /.well-known/ai.txt, ai-plugin.json, openapi.json) — zero crawler requests in 15 days. Audit tools like them; crawlers ignore them.
FAQPage markup on every page — no crawl effect in our data. Add it later, on pages that are already candidates for citation.
A monitoring dashboard — start with your server logs or platform analytics for the first week; you only need to know whether crawlers are arriving and what they hit.
Freshness plumbing — worth building, but only once you have content and dates to keep honest.
This does not mean those features are bad. It means they are phase-two work, and building them before the crawl foundation is optimizing a pipeline that is not flowing yet.

FAQ
What should a brand-new site build first for AI crawlers? A correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage with a one-line statement of what the site is plus a few real articles. Those were the only entry points every AI crawler used in our logs.

Do AI discovery endpoints like /ai/summary.json help AI crawlers? Not in our 15 days of logs. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot visited robots.txt and sitemap.xml almost every day but never requested any AI-facing endpoint.

Why do GPTBot and ClaudeBot behave differently? ClaudeBot follows the classic crawl path — robots.txt and sitemap.xml daily, but barely any content. GPTBot ignores robots, reads llms.txt occasionally, and traverses the entire site — 121 content pages touched, including a single full-site pass on Aug 17.

Does adding FAQPage markup get a page crawled more? In our data, no measurable crawl effect. It improves how a page is quoted once AI is already considering it — it is a citation-layer lever, not a crawl driver.

Entity Association Density: The GEO Metric That Decides Whether AI Cites You

jf lu — Fri, 14 Aug 2026 11:32:25 +0000

Two websites cover the same topic. The content quality looks roughly equal. Yet AI cites site A again and again — and never mentions site B.

If that sounds familiar, the difference is often not what you wrote, but how densely your entities connect.

**Entity Association Density** measures how interconnected and cross-referenced the entities in your content are — whether an article presents a single isolated concept, or weaves a connected web of related knowledge around its core topic. It is one of the strongest predictors of whether an AI answer engine cites your content, alongside Content Cross-Verification (every claim corroborated by other authoritative sources).

The evidence is concrete. Yext's 2025 analysis of **6.8 million AI citation behaviors** found that 86% of citations come from authoritative, brand-managed sources — websites, listings, and reviews — with brand-owned websites alone accounting for 44% of all citations.

Here is what that means in practice, and how to act on it.

## Low density vs. high density: a before-and-after

**Low entity association density:**

> "CRM systems help businesses manage customer relationships. CRM systems can improve sales efficiency. When choosing a CRM system, you need to consider features, pricing, and implementation difficulty."

One entity — "CRM" — repeated with no connections. AI reads this and concludes: *this author only knows CRM.* There is nothing to extract beyond a single fact.

**High entity association density:**

> "CRM systems (such as Salesforce, HubSpot) serve as the core tool for customer relationship management, working in conjunction with Marketing Automation (MA), Enterprise Resource Planning (ERP), and other systems. According to Yext's 2025 analysis of 6.8 million AI citations, 44% of AI citations come from brand-owned websites and 86% from brand-managed sources overall. Products like Salesforce's Einstein GPT and HubSpot's Breeze AI are redefining industry standards. But CRM implementation challenges are also significant — analysts report that a substantial share of CRM projects fail to meet expectations due to employee resistance."

Entities in one short passage: CRM, Salesforce, HubSpot, Marketing Automation, ERP, Yext, AI citations, Einstein GPT, Breeze AI. Every one supports the core topic from a different angle.

## Why AI rewards density

During RAG generation, AI faces an efficiency problem: it must extract as much useful information as possible from a limited content snippet. A passage that "only discusses CRM" gives AI one fact. A passage that connects CRM to Salesforce, Gartner data, and implementation challenges lets AI extract a multi-dimensional knowledge cluster from a single article.

The underlying logic: **the lower the information entropy, the higher the citation value.** Content that saves AI time is content AI cites.

## Four practical ways to raise your density

**1. Build an entity list before you write.**
Before drafting, list 5-10 entities around your core topic: main products, related technologies, industry authorities, competitors, upstream/downstream concepts, data sources, real-world cases. Then weave them in as logical chains — "because of this entity, that entity follows" — not as a keyword list.

**2. Use comparisons and connections.**
"What's the difference between A and B?" and "How do C and D work together?" are sentence structures that inherently carry high density. They force you to name and connect multiple entities.

**3. Add related-reading links.**
Close with 3-5 internal links to related articles. You are building an entity network within your site — AI crawlers follow these links to map your whole knowledge system.

**4. Balance density with quality.**
Density is not "more is better." Cramming in irrelevant entities creates noise. **Every entity should support or extend the core concept.** A simple test: if removing an entity doesn't weaken the article, it is redundant.

## The partner metric: Content Cross-Verification

Entity Association Density answers "does your content connect enough ideas?" Content Cross-Verification answers "can your claims be confirmed elsewhere?"

AI performs a de facto fact-check before citing. If only you say something, AI hedges — "according to Website A's report." If multiple authoritative sources say the same thing, AI cites it confidently as consensus.

That is why you should not only cite authoritative third-party sources in your content (Yext, Gartner, academic papers) but also aim to *be cited by* them — white papers, media reports, research mentions. A solo voice is an isolated claim; a corroborated claim is consensus.

## A one-sentence summary

AI does not cite the loudest site — it cites the site that gives it the most connected, verifiable knowledge in the fewest snippets. Entity Association Density and Content Cross-Verification are the two levers that make that happen.


![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/vv6vlag9is4pchrlwda6.png)