DEV Community

campaign360
campaign360

Posted on

How I Built a Free, Client-Side SEO Log Analyzer with Web Workers (No Data Upload, No Limits)

#seowebdevjawebworkersvascript

How I Built a Free, Client-Side SEO Log Analyzer with Web Workers

No data upload. No limits. No signup. Runs entirely in your browser.

The Problem

Technical SEOs know server logs are the ground truth for crawl behavior. Search Console shows sampled data. Crawlers simulate. But logs? Logs show exactly what Googlebot, Bingbot, GPTBot, and 80+ other crawlers actually requested.

The problem: existing tools are either:

  • Enterprise-priced (Botify, OnCrawl, JetOctopus)
  • Desktop-only with limits (Screaming Frog Log File Analyser: €99/yr, 1k lines free)
  • Cloud-based (your logs leave your machine)

The Solution: 100% Client-Side

I built a free alternative that runs entirely in the browser using Web Workers:

‼️ Try it: https://campaign360.io/tools/free-seo-crawl-and-log-file-analyzer/

Key Features

Feature Description
Screaming Frog + Log Comparison Upload both, see gap analysis side-by-side
Crawl Budget by Bot Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot + 80 more
Crawl Budget by Section Where bots spend time on your site
Spoofer Detection Reverse DNS + ASN verification for fake crawler UAs
AI vs Search Comparison See how AI crawlers differ from search crawlers
CSV Exports Bots, wasted URLs, robots.txt recommendations
Daily Trends Crawl-over-time charts

Technical Architecture 

┌────────────────────────────────────────────────────────────┐
│        Main Thread (UI)             │
│  - File input / drag-drop           │
│  - Results rendering                │
│  - Chart.js visualizations          │
└────────────────────────────────────────────────┘
               │ postMessage()
               ▼
┌────────────────────────────────────────────────────┐
│      Web Worker (Parser)            │
│  - Streams file in chunks           │
│  - Parses Apache/Nginx/IIS/JSON     │
│  - Classifies 196+ bot patterns     │
│  - Aggregates metrics               │
└──────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Why Web Workers?

  • Large logs (100MB+) don't block UI
  • Streaming parsing = constant memory
  • Multi-threaded = faster than single-threaded JS

Bot Detection Logic 

// Simplified: 196 patterns across 20+ LLM platforms
const BOT_PATTERNS = {
  googlebot: /Googlebot/i,
  gptbot: /GPTBot/i,
  claude: /ClaudeBot/i,
  perplexity: /PerplexityBot/i,
  // ... 190+ more
};

// Verification: IP → reverse DNS → ASN match
async function verifyBot(ip, userAgent) {
  const rDNS = await reverseDNS(ip);
  const asn = await lookupASN(ip);
  return KNOWN_RANGES[userAgent].some(range => 
    ipInRange(ip, range) || asnMatches(asn, range)
  );
}
Enter fullscreen mode Exit fullscreen mode

Supported Log Formats

  • Apache Combined / Common
  • Nginx access logs
  • IIS / W3C Extended
  • JSON (Cloudflare Logpush, AWS CloudFront)
  • Gzipped (.gz) - decompressed in-browser via pako

Privacy-First Design

Aspect Implementation
Data upload Zero - all parsing local
Storage None - close tab, data gone
External calls Only optional per-IP reverse DNS check
Analytics None on tool page

Use Cases

  1. Crawl budget optimization - Find 404/redirect waste by section
  2. AI crawler audit - Are GPTBot/ClaudeBot hitting your money pages?
  3. Spoofer blocking - Identify fake Googlebot IPs for firewall rules
  4. Migration monitoring - Compare pre/post logs for crawl changes
  5. Orphan page discovery - URLs in logs but not in SF crawl

Try It Yourself

  1. Export Screaming Frog: Internal → All → Export CSV
  2. Download server logs: Apache /var/log/apache2/access.log, Nginx /var/log/nginx/access.log
  3. Drag both to: https://campaign360.io/tools/free-seo-crawl-and-log-file-analyzer/
  4. Click "Try sample log" if you want to test first

What's Next

  • [ ] Historical comparison (upload multiple date ranges)
  • [ ] robots.txt tester integration
  • [ ] IndexNow submission from findings
  • [ ] API for automated monitoring

Feedback Welcome

Built this because I couldn't justify €99/yr for occasional log analysis. If you're a technical SEO, developer, or run a large site — try it and tell me what's missing.

Tool: https://campaign360.io/tools/free-seo-crawl-and-log-file-analyzer/

Contact: arun@campaign360.io

Tags: #seo #webdev #javascript #webworkers #technicalseo #loganalysis #crawlbudget

Top comments (0)