Python Competitive Intelligence: Automatically Monitor Your Competitors 24/7

#python #business #webscraping #automation

Your competitors change their prices, launch features, and post strategic job listings every week.

Most businesses miss it entirely. Python can catch every change automatically.

What to Monitor

Pricing pages (price changes reveal strategy)
Job postings (hiring = roadmap signals)
Blog/changelog (new features)
Review sites (G2, Capterra - customer pain points)

Core: Website Change Detector

import requests
import hashlib
import sqlite3
import smtplib
from email.mime.text import MIMEText
from bs4 import BeautifulSoup
import schedule
import time

def get_content_hash(url, css_selector=None):
    resp = requests.get(url, timeout=15, headers={
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
    })
    if css_selector:
        soup = BeautifulSoup(resp.text, 'html.parser')
        el = soup.select_one(css_selector)
        text = el.get_text(strip=True) if el else ''
    else:
        text = resp.text
    return hashlib.md5(text.encode()).hexdigest()

def check_for_changes(db_conn, monitors):
    c = db_conn.cursor()
    for item in monitors:
        name, url = item['name'], item['url']
        selector = item.get('selector')
        current = get_content_hash(url, selector)

        row = c.execute('SELECT hash FROM pages WHERE url=?', (url,)).fetchone()
        if row is None:
            c.execute('INSERT INTO pages VALUES (?,?,datetime("now"),?)',
                     (url, current, name))
            print(f'Initialized: {name}')
        elif row[0] != current:
            send_email_alert(f'{name} Changed!', f'{name} changed!\nURL: {url}')
            c.execute('UPDATE pages SET hash=? WHERE url=?', (current, url))
            print(f'CHANGED: {name}')
        else:
            print(f'No change: {name}')
        time.sleep(2)
    db_conn.commit()

Job Posting Intelligence

Competitor job postings reveal their roadmap:

def analyze_jobs(company, jobs_url):
    resp = requests.get(jobs_url)
    soup = BeautifulSoup(resp.text, 'html.parser')
    job_titles = [j.get_text(strip=True) for j in soup.select('.job-title, h3, h2')]

    signals = {
        'enterprise_push': ['enterprise', 'account executive', 'sales engineer'],
        'ai_investment': ['machine learning', 'ml engineer', 'data scientist'],
        'mobile_app': ['ios', 'android', 'react native'],
        'international': ['localization', 'french', 'german'],
    }

    detected = {}
    for signal, keywords in signals.items():
        matches = [j for j in job_titles if any(k in j.lower() for k in keywords)]
        if matches:
            detected[signal] = matches[:3]

    if detected:
        print(f'{company} strategic signals: {detected}')
    return detected

Deploy Free on GitHub Actions

# .github/workflows/intel.yml
name: Competitive Intel
on:
  schedule:
    - cron: '0 */6 * * *'
jobs:
  monitor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: pip install requests beautifulsoup4
      - run: python monitor.py

Zero cost. Runs every 6 hours. Emails you any change.

Get the Full Monitoring System

This competitive intel script is included in the Python Business Automation Toolkit

20+ pre-built scripts for business automation. $29 one-time.

Which competitor are you watching? Drop it in the comments.

DEV Community