<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: leonlesenechal</title>
    <description>The latest articles on DEV Community by leonlesenechal (@leonlesenechal).</description>
    <link>https://dev.to/leonlesenechal</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254025%2F8db87119-769e-4c1b-93c0-7bccb12ecaa6.png</url>
      <title>DEV Community: leonlesenechal</title>
      <link>https://dev.to/leonlesenechal</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/leonlesenechal"/>
    <language>en</language>
    <item>
      <title>Is that really GPTBot? How to see which AI agents visit your site and which ones are fake</title>
      <dc:creator>leonlesenechal</dc:creator>
      <pubDate>Thu, 19 Nov 2026 14:29:36 +0000</pubDate>
      <link>https://dev.to/leonlesenechal/is-that-really-gptbot-how-to-see-which-ai-agents-visit-your-site-and-which-ones-are-fake-4a7j</link>
      <guid>https://dev.to/leonlesenechal/is-that-really-gptbot-how-to-see-which-ai-agents-visit-your-site-and-which-ones-are-fake-4a7j</guid>
      <description>&lt;p&gt;Your analytics says nobody came. Your server logs say ChatGPT, Claude and Perplexity were there all day.&lt;/p&gt;

&lt;p&gt;Both are right. Google Analytics, Plausible, Jetpack and most dashboards count a visit when the browser runs a JavaScript snippet. AI agents don't run your JavaScript. They request the HTML, read the text and leave. So they're invisible to tag-based analytics, even though they're how a growing share of buyers now research products.&lt;/p&gt;

&lt;p&gt;Here's how to see them, and how to tell the real ones from the impostors.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Know the three kinds of AI agents
&lt;/h3&gt;

&lt;p&gt;One company usually runs several bots, and they mean very different things for your site:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Kind&lt;/th&gt;
&lt;th&gt;Examples&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Training crawlers&lt;/td&gt;
&lt;td&gt;GPTBot, ClaudeBot, Google-Extended (a robots.txt token, not a crawler)&lt;/td&gt;
&lt;td&gt;Your content may go into a future model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AI search crawlers&lt;/td&gt;
&lt;td&gt;OAI-SearchBot, Claude-SearchBot, PerplexityBot&lt;/td&gt;
&lt;td&gt;Your pages can show up and be cited in AI answers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fetches for a user&lt;/td&gt;
&lt;td&gt;ChatGPT-User, Claude-User, Perplexity-User&lt;/td&gt;
&lt;td&gt;A person asked the assistant something, and it read your page right now&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The last group is the closest thing to a real visit: a human is on the other end, even if they never load your site.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Find them in your logs
&lt;/h3&gt;

&lt;p&gt;Every request still goes through your server or CDN, and access logs keep the user agent. On a typical Nginx or Apache server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -Ei &lt;span class="s2"&gt;"gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user"&lt;/span&gt; access.log \
  &lt;span class="o"&gt;|&lt;/span&gt; awk -F&lt;span class="s1"&gt;'"'&lt;/span&gt; &lt;span class="s1"&gt;'{print $6}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c &lt;span class="o"&gt;|&lt;/span&gt; sort -rn
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That gives you counts per agent. To see which pages they read, print the request field (&lt;code&gt;$2&lt;/code&gt;) instead.&lt;/p&gt;

&lt;p&gt;On Cloudflare, the free AI Crawl Control dashboard shows this without touching logs. On shared hosting (cPanel), the raw access logs are usually in &lt;code&gt;~/logs&lt;/code&gt; as monthly &lt;code&gt;.gz&lt;/code&gt; files.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Don't trust the user agent
&lt;/h3&gt;

&lt;p&gt;Anyone can send &lt;code&gt;User-Agent: GPTBot&lt;/code&gt;. When we audited a week of logs on a small business site, hundreds of requests claimed to be Claude-User and OAI-SearchBot but came from cloud servers that the operators don't list. Scrapers dress up as AI bots because many sites let AI bots in.&lt;/p&gt;

&lt;p&gt;The operators publish three ways to check a request:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;IP ranges.&lt;/strong&gt; OpenAI, Anthropic, Perplexity, Google and others publish JSON lists of the addresses their bots use. If the request's IP is outside the list, it isn't that bot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reverse DNS.&lt;/strong&gt; For Googlebot, Bingbot, Applebot and others, look up the IP's hostname, check that it ends in the operator's domain, then resolve it forward and confirm you get the same IP back.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signed requests (Web Bot Auth).&lt;/strong&gt; Some agents now sign each request with a key published by their operator. A valid signature proves who sent it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two traps:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A missing method is not proof. If an operator publishes no IP list for a bot, call it &lt;em&gt;unverifiable&lt;/em&gt;, not fake.&lt;/li&gt;
&lt;li&gt;Lists go stale. If an operator's list hasn't changed in months while its bot grows, a request outside the list may still be real. Check the date of the list before you call a request fake.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  4. Decide what to allow
&lt;/h3&gt;

&lt;p&gt;A quick look at robots.txt across the 1,000 most-visited websites (November 2026) showed that most sites don't block anything: 77.5% allow all 14 major AI agents. The ones that block usually do it on purpose: 9.6% block OpenAI's training crawler but keep its search crawler, so they stay in ChatGPT's answers without feeding the next model.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;GPTBot&lt;/span&gt;
&lt;span class="nc"&gt;Disallow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;OAI&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;SearchBot&lt;/span&gt;
&lt;span class="nc"&gt;Allow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Blocking a user-triggered fetcher like ChatGPT-User means telling a person who asked about you "no answer". That's rarely what a business wants.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Connect the read to the result
&lt;/h3&gt;

&lt;p&gt;The agent's read is half the story. The other half is the person who clicks a link in the answer. Those visits do run your JavaScript, and they show up with a referrer like &lt;code&gt;chatgpt.com&lt;/code&gt; or &lt;code&gt;perplexity.ai&lt;/code&gt;, or &lt;code&gt;utm_source=chatgpt.com&lt;/code&gt; (GA4 now groups them under an "AI Assistant" channel). Put the two side by side and you can see which pages the assistants read and which ones turn into visits and leads.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tools
&lt;/h3&gt;

&lt;p&gt;If you'd rather not grep logs every week, there are tools for this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cloudflare AI Crawl Control&lt;/strong&gt; (free on Cloudflare) if you want to block or allow bots at the edge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Known Agents&lt;/strong&gt;, &lt;strong&gt;Salience&lt;/strong&gt; or &lt;strong&gt;Clicktag&lt;/strong&gt; for crawler analytics from your CDN or logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Screaming Frog Log File Analyser&lt;/strong&gt; or &lt;strong&gt;GoAccess&lt;/strong&gt; to analyze log files yourself.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Disclosure: I built &lt;strong&gt;Knocklog&lt;/strong&gt; (knocklog.ai), which does this on any host, including shared cPanel hosting and WordPress: it verifies every agent, shows the pages each one read, and counts the leads each assistant sent. It also has free tools to check whether a bot is real and to test robots.txt against every AI crawler. We keep a sourced comparison of all of these tools here: &lt;a href="https://knocklog.ai/compare/tools-to-see-ai-crawlers-on-your-website" rel="noopener noreferrer"&gt;https://knocklog.ai/compare/tools-to-see-ai-crawlers-on-your-website&lt;/a&gt;&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>seo</category>
      <category>ai</category>
      <category>security</category>
    </item>
  </channel>
</rss>
