Build a Competitive Intelligence Tool with Python

tags: python, data, business, tools

---

Build a Competitive Intelligence Tool with Python

Imagine waking up Monday morning to a dashboard that already knows your top competitor just dropped their pricing by 15%, launched a new feature targeting your exact audience, and posted three job openings for AI engineers. You didn’t have to hunt for it. You didn’t have to guess. A tool you built yourself in a weekend did the work. That’s the power of competitive intelligence—and with Python, you can build it today.

Competitive intelligence isn’t just for big corporations with six-figure budgets. It’s for anyone who wants to stay ahead of the curve without drowning in manual research. In this post, you’ll build a real, working Python tool that scrapes competitor websites, detects changes, and delivers actionable insights. No fluff. No theory. Just code you can run immediately.

Why Python for Competitive Intelligence?

Python is the undisputed champion for data automation. Its ecosystem includes powerful libraries for web scraping, data processing, and even AI integration. Whether you’re tracking price changes, monitoring new blog posts, or analyzing ad creatives, Python gives you the flexibility to build exactly what you need.

Unlike generic tools that force you into rigid workflows, a custom Python script lets you:

• Target specific data points (pricing, features, job posts)
• Schedule checks daily, weekly, or hourly
• Customize output formats (CSV, JSON, Markdown reports)
• Integrate with AI models for automated analysis

The Core Architecture

Your tool will follow a simple four-step pipeline:

1. Data Collection: Fetch competitor pages using requests and BeautifulSoup.
2. Change Detection: Compare current content with historical snapshots using hashing.
3. Signal Extraction: Parse structured data like prices, headlines, and metadata.
4. Insight Generation: Summarize findings and flag significant changes.

Let’s build it.

Step 1: Set Up Your Environment

First, create a project folder and set up a virtual environment:

mkdir competitive-intel-tool
cd competitive-intel-tool
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Now, install the required dependencies:

pip install requests beautifulsoup4 python-dotenv

Create a .env file to store any API keys (if you add AI later):

# .env
AI_API_KEY=your_key_here

Step 2: Build the Scraper

Create a file named scraper.py. This script will fetch competitor pages and extract key signals.

# scraper.py
import requests
from bs4 import BeautifulSoup
import hashlib
from dotenv import load_dotenv

load_dotenv()

def fetch_page(url):
    """Fetch webpage content and return cleaned text."""
    try:
        response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
        if response.status_code == 200:
            soup = BeautifulSoup(response.text, 'html.parser')
            # Remove scripts and styles
            for tag in soup(['script', 'style']):
                tag.decompose()
            return soup.get_text(strip=True)
        else:
            return None
    except Exception as e:
        print(f"Error fetching {url}: {e}")
        return None

def detect_changes(old_content, new_content):
    """Detect if content has changed using hash comparison."""
    if old_content is None:
        return True
    old_hash = hashlib.md5(old_content.encode()).hexdigest()
    new_hash = hashlib.md5(new_content.encode()).hexdigest()
    return old_hash != new_hash

def extract_price(text):
    """Simple price extractor (look for patterns like $99 or $99.99)."""
    import re
    prices = re.findall(r'\$[\d,]+\.?\d*', text)
    return prices if prices else []

# Example usage
if __name__ == "__main__":
    competitor_url = "https://example-competitor.com/pricing"
    content = fetch_page(competitor_url)

    if content:
        prices = extract_price(content)
        print(f"Found prices: {prices}")
        # You'd compare this with historical data here
    else:
        print("Failed to fetch page")

Run it:

python scraper.py

You’ll see any prices found on the page. This is your foundation.

Step 3: Add Change Detection & Storage

To track changes over time, you need to store historical snapshots. Create tracker.py:

# tracker.py
import json
import os
from scraper import fetch_page, detect_changes, extract_price

DATA_FILE = "competitor_data.json"

def load_history():
    if os.path.exists(DATA_FILE):
        with open(DATA_FILE, 'r') as f:
            return json.load(f)
    return {}

def save_history(history):
    with open(DATA_FILE, 'w') as f:
        json.dump(history, f)

def track_competitor(url, competitor_name):
    history = load_history()
    current_content = fetch_page(url)

    if not current_content:
        print(f"Could not fetch {url}")
        return

    old_content = history.get(competitor_name, {}).get("content")

    if detect_changes(old_content, current_content):
        prices = extract_price(current_content)
        history[competitor_name] = {
            "content": current_content,
            "prices": prices,
            "last_checked": "just now"
        }
        save_history(history)
        print(f"⚠️ CHANGE DETECTED for {competitor_name}")
        print(f"New prices: {prices}")
    else:
        print(f"No changes for {competitor_name}")

# Run with a competitor
if __name__ == "__main__":
    track_competitor("https://example-competitor.com/pricing", "Competitor A")

Now you’re tracking changes. Run it daily:

python tracker.py

Step 4: Automate with Cron (Optional)

Make your tool run automatically. On Linux/Mac, add this to your crontab:

# Run every day at 8 AM
0 8 * * * /path/to/your/venv/bin/python /path/to/tracker.py >> /path/to/log.txt

On Windows, use Task Scheduler to run tracker.py daily.

What You Can Track

With this foundation, you can expand to track:

• Pricing changes: Extract and compare price points.
• New features: Scan “What’s New” or blog pages.
• Job postings: Monitor hiring for new tech (e.g., “AI engineer”).
• Ad creatives: Use Google Ads Library to grab headlines and CTAs.
• Content updates: Track blog RSS feeds or sitemaps.

Next-Level Enhancements

Once you have the basics, level up:

• AI Analysis: Send deltas to Claude or Ollama for automated summaries.
• Multi-Agent System: Use CrewAI to assign different agents to pricing, content, and jobs.
• Dashboard: Build a Streamlit UI to visualize trends.
• CSV Export: Add pandas to export reports for stakeholders.

Start Today, Win Tomorrow

You just built a competitive intelligence tool that runs on autopilot. No subscriptions. No black boxes. Just code you control.

Your call to action:

1. Copy the code above into your project.
2. Replace example-competitor.com with a real competitor URL.
3. Run tracker.py and watch for changes.
4. Share your results with your team or on Dev.to.

The market moves fast. Now you have a tool that moves faster.

What’s the first competitor you’ll track? Drop your URL in the comments—and let’s build something powerful together.

---

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.