Building a Competitive Intelligence Dashboard with Web Scraping

Understanding competitors' pricing changes, hiring patterns, and marketing strategies gives you a strategic advantage. Here's how to build a Python-powered competitive intelligence system.

Core Engine

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Pricing Intelligence

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Job Posting Intelligence

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Reports

class IntelReport:
    def __init__(self, engine):
        self.e = engine

    def generate(self, cid):
        comp = self.e.db.execute('SELECT name,domain FROM competitors WHERE id=?', (cid,)).fetchone()
        if not comp: return None
        return {'competitor': comp[0], 'domain': comp[1],
                'pricing_changes': self.e.pricing_changes(cid),
                'hiring_trends': self.e.hiring_trends(cid)}

    def display(self, r):
        print(f"\nIntelligence: {r['competitor']}")
        print("="*50)
        if r['pricing_changes']:
            print("\nPricing Changes:")
            for c in r['pricing_changes']:
                d = 'UP' if c['pct'] > 0 else 'DOWN'
                print(f"  {c['plan']}: ${c['old']} -> ${c['new']} ({d} {abs(c['pct'])}%)")
        if r['hiring_trends']:
            print("\nHiring:")
            for dept, n in r['hiring_trends'].items():
                print(f"  {dept}: {n} positions")

intel = CompetitiveIntel(api_key='YOUR_KEY')
intel.add_competitor('c1', 'Competitor A', 'competitor-a.com')
report = IntelReport(intel)
r = report.generate('c1')
report.display(r)

For monitoring dozens of competitors, ScraperAPI handles JS rendering. ThorData provides residential proxies. Track pipelines with ScrapeOps.

---

Follow for more Python business intelligence tutorials.