DEV Community

Vhub Systems
Vhub Systems

Posted on

How to Monitor Competitor Websites for Changes (GDPR-Safe Stack, $5/Month)

#gdpr #webdev #startup #automation

Your competitor's pricing page changed last night. Their refund policy was updated at 2am. They removed a feature from their standard tier.

You found out three weeks later.

Here is how to monitor competitor websites for changes in real-time — without paying $200/month for tools that do the same thing with less control.

What You Actually Need to Monitor

Most change-monitoring guides tell you to track "everything." That floods you with noise.

The changes that actually matter:

Pricing changes — price increases signal room to compete on cost; decreases signal they are losing deals to price objections.

Feature additions/removals — your competitive positioning needs updating. Sales team needs to know.

Terms of Service changes — especially data retention and privacy clauses. Post-GDPR, ToS changes can signal a pivot in how they handle user data — which is a compliance consideration if you're an enterprise buyer evaluating them.

Job postings — track what they're building. 5 new "ML Engineer" postings = they're building an AI feature. 3 "Account Executive" postings = they're pushing into enterprise.

The GDPR-Safe Monitoring Stack

A critical note before building: monitoring competitor public pages is legal and GDPR-compliant when you:

  • Only scrape publicly accessible pages (no login-required content)
  • Store only the content delta, not user data
  • Do not use the data to profile individuals

The GDPR applies to personal data of natural persons. Competitor product pages, pricing, and feature documentation are not personal data.

Component 1: Page Change Detector

The simplest approach: hash the page content and compare to previous hash. When hash changes, fetch and store the diff.

import hashlib
import requests

def get_page_hash(url):
    resp = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}, timeout=30)
    # Normalize: remove timestamps, session tokens, dynamic ads
    content = resp.text
    return hashlib.md5(content.encode()).hexdigest(), content

def check_for_changes(url, stored_hash, stored_content):
    new_hash, new_content = get_page_hash(url)
    if new_hash != stored_hash:
        return True, new_hash, new_content
    return False, stored_hash, stored_content
Enter fullscreen mode Exit fullscreen mode

For production: use a proper diff library (difflib) to extract WHAT changed, not just THAT something changed.

Component 2: Apify Website Monitor

For JavaScript-heavy pages (most modern pricing pages use React/Vue), requests-based scrapers miss dynamically loaded content.

Apify's website content scraper handles:

  • JavaScript rendering via Playwright headless browser
  • Cookie banner dismissal (common on European sites — GDPR consent flows)
  • Dynamic content loading

Run daily. Cost: ~$0.01-0.05 per page checked, depending on complexity.

Component 3: n8n Orchestration

n8n workflow (runs every 6 hours):

  1. Fetch current page → Apify website scraper
  2. Compare to stored version → Code node (Python/JS diff)
  3. Score the change → Code node (is this pricing? features? ToS?)
  4. Route by importance:
    • Pricing change → immediate Slack + Telegram alert
    • Feature change → Slack next morning
    • Minor text update → log only (no alert)
  5. Store diff → Google Sheets (audit trail)

Component 4: GDPR-Aware Data Handling

If any competitor page accidentally includes personal data (user testimonials with names, employee profiles):

import re

PERSONAL_DATA_PATTERNS = [
    r'[A-Z][a-z]+ [A-Z][a-z]+',  # Names (basic)
    r'[\w.-]+@[\w.-]+\.\w+',      # Emails
    r'\+?[\d\s\-\(\)]{10,15}',    # Phone numbers
]

def scrub_personal_data(text):
    for pattern in PERSONAL_DATA_PATTERNS:
        text = re.sub(pattern, '[REDACTED]', text)
    return text
Enter fullscreen mode Exit fullscreen mode

Apply this to stored diffs — you never need the personal data anyway.

Real Results From My Setup

Monitoring 12 competitor domains for 6 months:

Signal Type Times Detected Action Taken Outcome
Price increase 4 Sales email to their customers 2 deals
Feature removal 3 Updated comparison page +15% demo conversions
ToS privacy change 2 Alerted compliance team 0 action needed
New job postings 47 Adjusted product roadmap 1 feature shipped early

The pricing change detections alone paid for the monitoring stack 10x over.

Setup Time: Under 2 Hours

  1. Create free Apify account (includes $5 credit)
  2. Set up n8n (Docker: 5 minutes)
  3. Configure 5-10 target pages
  4. Connect Slack or Telegram for alerts

Total monthly cost: $3-8 in Apify credits.

Get the Complete Scraper Bundle

The Apify scrapers used in this stack — including the website content scraper and page change detector — are part of the Apify Scrapers Bundle — $29 one-time.

Pre-configured inputs for monitoring 30+ site types. No setup time.

Get the bundle here

What competitor signals are most valuable for your business? Drop them in the comments.

Top comments (0)