AI systems are consuming more of the web, but that does not mean publishers and brands are receiving a proportional increase in human visits.
A July 21 Digiday analysis pulled together several recent datasets showing how quickly agent traffic is growing. DataDome recorded 17.7 billion AI agent requests across its network in the second quarter of 2026, up 45% from the first quarter. Digiday also cited Decodo analysis of Cloudflare data showing AI driven traffic grew approximately 187% during 2025.
At the same time, AI referral traffic remains comparatively small. The result is a measurement problem: marketers can see bots fetching content without knowing whether those fetches produce citations, brand mentions, human visits, or business outcomes.
That makes a technical tool such as an AI crawler access checker useful for answering one question, whether major crawlers can access a site, while leaving several much harder questions unanswered.
Crawler traffic and referral traffic are different things
The distinction is easy to lose because both involve AI platforms interacting with a website.
Crawler traffic is generated by automated systems requesting pages or files. Those requests may support model training, search indexing, retrieval for live answers, or other internal purposes. Referral traffic is generated when a human user clicks a link from an AI answer and lands on the website.
One can increase without the other.
Digiday reports that ChatGPT accounts for the large majority of measurable AI driven referral traffic in the DataDome dataset, and human click throughs from ChatGPT increased 17% even while ChatGPT User page fetches fell 6% quarter over quarter. That alone shows why raw bot request counts should not be treated as a proxy for downstream traffic.
A site can be heavily crawled and receive few visits. Another site can receive less crawler activity but earn highly valuable referrals.
More agent traffic does not automatically mean more visibility
Crawling is a prerequisite for some forms of live retrieval, but it is not proof of inclusion.
An AI system can fetch a page and decide not to use it. It can use the information without citing the source. It can cite the source without mentioning the brand. It can mention the brand without sending a click.
Each stage represents a different metric.
That is why AI search visibility reporting should separate technical access, retrieval evidence where available, answer inclusion, citation frequency, referral traffic, and commercial performance.
Blending all of these into a single "AI traffic" number makes it difficult to understand what is actually improving.
The agentic web is becoming a distribution layer
Digiday's larger argument is that publishers are starting to think of AI answer engines less as referral channels and more as distribution layers.
That shift makes sense when users increasingly receive complete answers inside the AI interface. A publisher's reporting, expertise, or product information can influence the answer even when the user never visits the source page.
For brands, the equivalent is zero click discovery. A potential customer can learn that a company exists, compare it with competitors, see strengths and limitations, and move closer to a purchase without generating a session in analytics.
This does not mean traffic stops mattering. Human visits remain much easier to connect to conversion. It means traffic can no longer represent the entire value created by discoverability.
The Mustard Seed guide to AEO is useful in this context because answer visibility focuses on whether a brand's information is available and useful at the point where the user receives the answer, not only on whether the user clicks through.
Bot growth creates a policy question as well as a marketing question
When AI crawler activity rises, website owners also need to decide which agents should have access.
Digiday reports that more publishers are moving away from blanket allow or block policies. Different agents may have different purposes. Some support training. Some retrieve live pages for user questions. Some may send measurable referral traffic. Others may consume large volumes of content with little obvious return.
That means the business value of a crawler depends on what it does, not simply who operates it.
Mustard Seed's AI crawler study 2026 examines how real websites handle access for major AI related crawlers. That kind of technical review should sit beside, not replace, visibility measurement.
A marketer may want a crawler allowed because it supports discovery. A publisher may want a training bot blocked because the commercial tradeoff is unclear. The correct choice can differ by agent and business model.
Measuring AEO requires a funnel of machine and human signals
The new reporting model needs to reflect the sequence between a machine request and a business outcome.
At the top is access. Can the relevant AI system reach the content? Next comes presence. Does the brand or source appear in answers for commercially meaningful prompts? Then comes quality. Is the description accurate and favorable? Next comes citation and referral. Does the answer link to the site, and do users click? Finally comes business value. Do those users convert, or does broader AI visibility correlate with brand demand and qualified pipeline?
Not every company can measure every stage perfectly. The important step is to avoid pretending that an upstream metric proves a downstream result.
A spike in GPTBot requests is not equivalent to a spike in leads.
This is also why SEO reporting needs adaptation in an AI search environment. Search Console clicks, analytics sessions, crawler logs, AI visibility tools, branded search, and CRM data answer different parts of the customer journey.
The metric is changing from traffic returned to influence created
The old web bargain was easy to understand: search engines crawled content and sent people back through links. AI answer engines complicate that exchange because they can consume information and satisfy more of the user's need inside the interface.
The emerging question for marketers is therefore not only how much traffic AI sends. It is how much influence a brand earns before the click, whether that influence is measurable, and whether the remaining referrals are commercially valuable.
AI crawler growth is real. AI referral growth is also real in some datasets. The two should not be confused.
The companies that build separate measurements for access, visibility, citations, referrals, and conversions will have a much clearer view of whether the agentic web is creating value or simply creating server requests.
Originally published on the Mustard Seed blog.
Top comments (0)