How I Built a Free, Client-Side SEO Log Analyzer with Web Workers (No Data Upload, No Limits)

campaign360 — Sun, 23 Aug 2026 06:29:11 +0000

How I Built a Free, Client-Side SEO Log Analyzer with Web Workers

No data upload. No limits. No signup. Runs entirely in your browser.

The Problem

Technical SEOs know server logs are the ground truth for crawl behavior. Search Console shows sampled data. Crawlers simulate. But logs? Logs show exactly what Googlebot, Bingbot, GPTBot, and 80+ other crawlers actually requested.

The problem: existing tools are either:

Enterprise-priced (Botify, OnCrawl, JetOctopus)
Desktop-only with limits (Screaming Frog Log File Analyser: €99/yr, 1k lines free)
Cloud-based (your logs leave your machine)

The Solution: 100% Client-Side

I built a free alternative that runs entirely in the browser using Web Workers:

‼️ Try it: https://campaign360.io/tools/free-seo-crawl-and-log-file-analyzer/

Key Features

Feature	Description
Screaming Frog + Log Comparison	Upload both, see gap analysis side-by-side
Crawl Budget by Bot	Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot + 80 more
Crawl Budget by Section	Where bots spend time on your site
Spoofer Detection	Reverse DNS + ASN verification for fake crawler UAs
AI vs Search Comparison	See how AI crawlers differ from search crawlers
CSV Exports	Bots, wasted URLs, robots.txt recommendations
Daily Trends	Crawl-over-time charts

Technical Architecture

┌────────────────────────────────────────────────────────────┐
│        Main Thread (UI)             │
│  - File input / drag-drop           │
│  - Results rendering                │
│  - Chart.js visualizations          │
└────────────────────────────────────────────────┘
               │ postMessage()
               ▼
┌────────────────────────────────────────────────────┐
│      Web Worker (Parser)            │
│  - Streams file in chunks           │
│  - Parses Apache/Nginx/IIS/JSON     │
│  - Classifies 196+ bot patterns     │
│  - Aggregates metrics               │
└──────────────────────────────────────────────┘

Why Web Workers?

Large logs (100MB+) don't block UI
Streaming parsing = constant memory
Multi-threaded = faster than single-threaded JS

Bot Detection Logic

// Simplified: 196 patterns across 20+ LLM platforms
const BOT_PATTERNS = {
  googlebot: /Googlebot/i,
  gptbot: /GPTBot/i,
  claude: /ClaudeBot/i,
  perplexity: /PerplexityBot/i,
  // ... 190+ more
};

// Verification: IP → reverse DNS → ASN match
async function verifyBot(ip, userAgent) {
  const rDNS = await reverseDNS(ip);
  const asn = await lookupASN(ip);
  return KNOWN_RANGES[userAgent].some(range => 
    ipInRange(ip, range) || asnMatches(asn, range)
  );
}

Supported Log Formats

Apache Combined / Common
Nginx access logs
IIS / W3C Extended
JSON (Cloudflare Logpush, AWS CloudFront)
Gzipped (.gz) - decompressed in-browser via pako

Privacy-First Design

Aspect	Implementation
Data upload	Zero - all parsing local
Storage	None - close tab, data gone
External calls	Only optional per-IP reverse DNS check
Analytics	None on tool page

Use Cases

Crawl budget optimization - Find 404/redirect waste by section
AI crawler audit - Are GPTBot/ClaudeBot hitting your money pages?
Spoofer blocking - Identify fake Googlebot IPs for firewall rules
Migration monitoring - Compare pre/post logs for crawl changes
Orphan page discovery - URLs in logs but not in SF crawl

Try It Yourself

Export Screaming Frog: Internal → All → Export CSV
Download server logs: Apache /var/log/apache2/access.log, Nginx /var/log/nginx/access.log
Drag both to: https://campaign360.io/tools/free-seo-crawl-and-log-file-analyzer/
Click "Try sample log" if you want to test first

What's Next

[ ] Historical comparison (upload multiple date ranges)
[ ] robots.txt tester integration
[ ] IndexNow submission from findings
[ ] API for automated monitoring

Feedback Welcome

Built this because I couldn't justify €99/yr for occasional log analysis. If you're a technical SEO, developer, or run a large site — try it and tell me what's missing.

Tool: https://campaign360.io/tools/free-seo-crawl-and-log-file-analyzer/

Contact: arun@campaign360.io

Tags: #seo #webdev #javascript #webworkers #technicalseo #loganalysis #crawlbudget

DEV Community: campaign360