<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Marko Savran</title>
    <description>The latest articles on DEV Community by Marko Savran (@markosavran).</description>
    <link>https://dev.to/markosavran</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4228770%2Fbaa487c8-4270-4847-9553-4ebd3dec75ba.jpg</url>
      <title>DEV Community: Marko Savran</title>
      <link>https://dev.to/markosavran</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/markosavran"/>
    <language>en</language>
    <item>
      <title>Five log analyzers, one nginx log, five visitor counts</title>
      <dc:creator>Marko Savran</dc:creator>
      <pubDate>Thu, 19 Nov 2026 14:23:54 +0000</pubDate>
      <link>https://dev.to/markosavran/five-log-analyzers-one-nginx-log-five-visitor-counts-2lp6</link>
      <guid>https://dev.to/markosavran/five-log-analyzers-one-nginx-log-five-visitor-counts-2lp6</guid>
      <description>&lt;p&gt;I generated a month of nginx access log myself, so I knew the right answer before I started. The log has 43,231 people in it. They made 60,706 visits and 109,351 pageviews. Out of 502,103 lines, 118,216 come from bots.&lt;/p&gt;

&lt;p&gt;Then I ran the same file through GoAccess, AWStats, Webalizer (the Stone Steps branch), Matomo's log importer and lnav. I expected small differences. The "unique visitors" numbers came back like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Unique visitors&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AWStats&lt;/td&gt;
&lt;td&gt;49,616&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Webalizer&lt;/td&gt;
&lt;td&gt;50,707&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Matomo&lt;/td&gt;
&lt;td&gt;66,674&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GoAccess with &lt;code&gt;--ignore-crawlers&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;67,863&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GoAccess, default&lt;/td&gt;
&lt;td&gt;105,885&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The true figure is 43,231, and none of the five is close to it. GoAccess in its default mode says 105,885, which is more than twice the real number.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fraelqgc1kjbobpcdv8u9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fraelqgc1kjbobpcdv8u9.png" alt="Five visitor counts for the same 43,231 people" width="800" height="968"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  I blamed the bots first
&lt;/h2&gt;

&lt;p&gt;My first guess was that bots inflated the counts. So I filtered them out and counted again. The numbers still ranged from 49,576 to 60,209. Bots explain part of it at most.&lt;/p&gt;

&lt;h2&gt;
  
  
  Each tool answers a different question
&lt;/h2&gt;

&lt;p&gt;Once I put the outputs side by side, the cause was plain. The tools use three different definitions of a visitor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;AWStats and Webalizer count one visitor per address per month.&lt;/li&gt;
&lt;li&gt;GoAccess counts a visitor as an address plus a date plus a user agent.&lt;/li&gt;
&lt;li&gt;Matomo, run on logs without cookies, treats every visit as a new visitor. It reported 66,675 visits and 66,674 visitors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One reader on a phone at breakfast and a laptop in the evening is one person. Under these rules that reader can be one visitor, two, or a new one on every visit.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcod49v92p84l0nx4xd4y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcod49v92p84l0nx4xd4y.png" alt="One reader, three counting rules" width="800" height="1073"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Visits and pageviews held up
&lt;/h2&gt;

&lt;p&gt;This part surprised me in a good way. Every tool landed within 1% of the true visit count. Pageviews were either exact or 1.4% high. If you compare tools, or compare one month with the next, those two columns are safe to use.&lt;/p&gt;

&lt;p&gt;My conclusion is that "unique visitors" from a log file is a setting you chose, and nobody measured anything. Compare visits and pageviews instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  The scraper nobody caught
&lt;/h2&gt;

&lt;p&gt;I put one scraper in the log that claims to be Chrome 141. It sent 7,285 requests. No bot list in any of the five tools flagged it, and it pushed pageviews up by 6.7%. A list of known user agents only knows what someone already reported, so a fresh disguise passes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Speed on this log
&lt;/h2&gt;

&lt;p&gt;Timings on the same file:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Webalizer: 3.4 s&lt;/li&gt;
&lt;li&gt;lnav: 4.5 s&lt;/li&gt;
&lt;li&gt;GoAccess: 6.1 s&lt;/li&gt;
&lt;li&gt;AWStats: 19.5 s&lt;/li&gt;
&lt;li&gt;Matomo: 265 s&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Matomo took 265 s, far behind the rest. If all you need is a quick answer from a log, the other four finish within 20 seconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do with this
&lt;/h2&gt;

&lt;p&gt;I'd pick the tool by what I want to see and ignore its visitor number. If someone asks me for a headcount from nginx logs, I now say it can't be read from the log alone, and I show visits instead.&lt;/p&gt;

&lt;p&gt;Have you ever compared your own analyzer's visitor count with something you could verify? I'd like to hear how far off yours was.&lt;/p&gt;

&lt;p&gt;The full comparison, with commands and versions for all five tools, is on &lt;a href="https://opensource-analytics.com/access-log-analyzers-compared/" rel="noopener noreferrer"&gt;opensource-analytics.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>nginx</category>
      <category>analytics</category>
      <category>selfhosted</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I checked my nginx log for AI crawlers. A third of the “Googlebots” were fake</title>
      <dc:creator>Marko Savran</dc:creator>
      <pubDate>Mon, 09 Nov 2026 08:50:00 +0000</pubDate>
      <link>https://dev.to/markosavran/i-checked-my-nginx-log-for-ai-crawlers-a-third-of-the-googlebots-were-fake-5b6b</link>
      <guid>https://dev.to/markosavran/i-checked-my-nginx-log-for-ai-crawlers-a-third-of-the-googlebots-were-fake-5b6b</guid>
      <description>&lt;p&gt;I went through two weeks of my site's nginx access log and counted every bot that said it was an AI crawler or a search engine. I only kept requests whose IP address really belongs to the company behind the bot.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdiezdxkv3emtsn12qyr2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdiezdxkv3emtsn12qyr2.png" alt="AI crawlers vs search engines: verified requests over two weeks" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;AI crawlers made 1,590 verified requests. All search engines together made 563. Meta's crawler and ClaudeBot each read the site more often than Googlebot did.&lt;/p&gt;

&lt;p&gt;None of this shows up in Plausible, Umami or Google Analytics. These bots don't run JavaScript, so the tracking script never fires. The server log is the only place you see them.&lt;/p&gt;

&lt;p&gt;The part that surprised me was the fakes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0peh2awdz6yl6e53yxq6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0peh2awdz6yl6e53yxq6.png" alt="1 in 3 Googlebots in my log was fake" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;119 of 331 requests signed "Googlebot" came from rented Google Cloud servers, not from Google. Five servers used 30 bot names between them, including ChatGPT-User and Claude-User. Four of them first read the homepage, robots.txt and about 15 articles like a polite crawler. Then they asked for .env files, cloud keys and Git folders. That's 1,045 requests for secret files, and the server refused them all.&lt;/p&gt;

&lt;p&gt;Two quick checks if you run your own server:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A real Googlebot resolves to a name ending in googlebot.com. Reverse DNS catches these fakes in seconds.&lt;/li&gt;
&lt;li&gt;Make sure your server returns 403 for .env and similar files. robots.txt won't stop a bot that lies about its name.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I also looked for visitors arriving from ChatGPT, Perplexity, Claude or Gemini. In two weeks there were none.&lt;/p&gt;

&lt;p&gt;I put the full write-up on my blog, opensource-analytics.com, with all the charts, the three jobs AI bots do on a site and how I verified each one.&lt;/p&gt;

&lt;p&gt;The full write-up with all the charts and the method is on my blog: &lt;a href="https://opensource-analytics.com/ai-crawlers-in-access-logs/" rel="noopener noreferrer"&gt;AI crawlers in my access logs&lt;/a&gt;&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>security</category>
      <category>devops</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
