The latest articles on DEV Community by Tommy (@tommy2970). https://dev.to/tommy2970 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4002867%2F9e394501-bf03-4d43-983f-1f5e1f9c94a0.png https://dev.to/tommy2970 en Tommy Thu, 25 Jun 2026 18:52:00 +0000 https://dev.to/tommy2970/what-actually-visits-a-self-hosted-website-in-2026humans-ai-crawlers-and-6400-automated-attacks-d6p https://dev.to/tommy2970/what-actually-visits-a-self-hosted-website-in-2026humans-ai-crawlers-and-6400-automated-attacks-d6p <p>I run a small self-hosted website on a Raspberry Pi 4B at home.<br> A few weeks ago I started wondering: who actually visits a website in 2026?<br> Not just humans. Everything.<br> So I built a public observability dashboard on top of GoAccess that separates traffic into four categories: human visitors, search engine crawlers, AI retrieval agents, and automated attacks.<br> The numbers from the last 17 days surprised me:</p> <p><strong>4,523 human visits<br> 6,409 automated attack attempts<br> Thousands of crawler requests from search engines and AI systems</strong></p> <p>The attacks aren't sophisticated. They're mostly automated scanners probing for .env files, WordPress admin panels, and cloud credentials — hitting every public IP on the internet regardless of what's actually running there.<br> What I found more interesting was the AI agent behavior.<br> AI retrieval agents (GPTBot, ClaudeBot, PerplexityBot, Amazonbot) behave differently from traditional search crawlers. They hit semantic files aggressively — llms.txt, sitemap.xml, JSON-LD structured data — and seem to index the knowledge graph structure of a site rather than individual pages. Within hours of publishing new content, multiple AI crawlers had already visited, apparently triggered by the sitemap update rather than any external link.<br> A few observations I didn't expect:</p> <p>Combined machine traffic consistently exceeds human traffic<br> AI agents discovered new content faster than Google did<br> The semantic structure exposed by the site seems almost as important as the content itself<br> Even a Pi on a residential ISP receives constant automated scans (380+ attempts/day average)</p> <p>I made the dashboard public because I think the machine side of the web is underobserved.<br> The modern web feels less like "users visiting pages" and more like a parallel ecosystem of crawlers, AI agents, and automated systems running continuously alongside human visitors.</p> <p><a href="https://dev.tourl">stats.lake8.dev/geo.html</a></p> <p>Two questions:<br> Are others tracking AI agents separately from traditional search crawlers?<br> Has anyone else noticed AI retrieval systems indexing semantic structure (JSON-LD, llms.txt) faster than they index page content?</p> selfhosted webdev raspberrypi showdev