Build a Multi-Site Competitor Price Monitor in 10 Minutes (No Selectors, No Selenium)

#python #webscraping #dataengineering #ai

How to automate e-commerce price intelligence, Slack webhook alerts, and zero-cost GitHub Actions scheduling.

[!NOTE]
TL;DR / Quick Summary:

The Problem: Writing custom BeautifulSoup / Selenium scrapers for multiple e-commerce sites is fragile and requires continuous selector maintenance whenever layouts update.

The Solution: Use Scraping AI to extract normalized numeric pricing and stock status with a single API call.

Cost: ~$6.60/month to monitor 50 products across 3 retailers daily.

Zero-Cost Cron: Runs automatically on free GitHub Actions compute.

Python SDK: pip install scraping-ai

Zero-Risk Trial: Get 200 free tokens (no credit card required) at https://pig-data.jp/service/scraping-ai/.

The E-Commerce Price Monitoring Dilemma

If you sell products online or run a market intelligence pipeline, tracking competitor prices is essential.

Traditionally, tracking 3 different retailers meant writing 3 separate scraping scripts:

# The Traditional (Brittle) Way
selectors = {
    'amazon': {'title': '.product-title', 'price': '.a-price .a-offscreen'},
    'bestbuy': {'title': '.sku-title', 'price': '.price-current'},
    'niche_store': {'title': 'h1.item-name', 'price': '.sale-price-badge'}
}

This approach breaks constantly:

Prices formatted as $19.99, 19.99 USD, or $19.99 (Save 20%) require complex regex cleanup.
Redesigned product pages cause silent NoneType attribute errors.
Dynamic single-page applications require heavy Selenium/Playwright configurations.

Full Tutorial: Automated Price Drop Alert Pipeline with Webhooks

import sqlite3
import requests
from datetime import datetime
from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="YOUR_API_KEY")
SLACK_WEBHOOK = "https://hooks.slack.com/services/YOUR/WEBHOOK"

# 1. Setup SQLite storage
conn = sqlite3.connect('prices.db')
conn.execute("""
    CREATE TABLE IF NOT EXISTS price_history (
        product_name TEXT,
        price REAL,
        currency TEXT,
        url TEXT,
        checked_at DATETIME
    )
""")

def monitor_url(url: str):
    data = client.extract(
        url=url,
        schema={
            "product_name": "string",
            "price": "number",
            "currency": "string",
            "in_stock": "boolean"
        }
    )

    for item in data.results:
        product = item['data']
        name = product.get('product_name')
        price = float(product.get('price', 0))
        currency = product.get('currency', 'USD')

        # Check previous price
        cursor = conn.execute(
            "SELECT price FROM price_history WHERE product_name = ? ORDER BY checked_at DESC LIMIT 1",
            (name,)
        )
        last_row = cursor.fetchone()

        if last_row:
            last_price = last_row[0]
            if price < last_price:
                drop_pct = ((last_price - price) / last_price) * 100
                alert = f"🚨 PRICE DROP: {name} dropped {drop_pct:.1f}% (${last_price} ➔ ${price}) on {url}"
                print(alert)
                # Send webhook
                requests.post(SLACK_WEBHOOK, json={"text": alert})
            else:
                print(f"✅ Price stable: {name} (${price})")

        # Log to database
        conn.execute(
            "INSERT INTO price_history VALUES (?, ?, ?, ?, ?)",
            (name, price, currency, url, datetime.now())
        )

    conn.commit()

Free Daily Execution via GitHub Actions

You don't need to pay for an EC2 server or cloud cron instance. You can run your monitoring script every morning using free GitHub Actions:

Create .github/workflows/daily_price_monitor.yml:

name: Daily Price Monitor

on:
  schedule:
    - cron: '0 9 * * *' # Runs daily at 9:00 AM UTC
  workflow_dispatch:

jobs:
  check-prices:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install scraping-ai requests
      - run: python monitor.py
        env:
          SCRAPING_AI_KEY: ${{ secrets.SCRAPING_AI_KEY }}
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}

Token Economics: How Much Does It Cost?

Monitoring 50 products across 3 stores daily:

Action	Frequency	Tokens Per Day
Daily URL Extractions	150 pages	~36 tokens
Monthly Total	4,500 pages	~1,080 tokens

On the Growth Tier ($30 / 5,000 tokens), this costs ~$6.60 / month.

Honest Limitations

Social Media: Scraping social media feeds for prices is excluded per platform terms.
Anti-Bot Defense: Automated stealth bypass achieves ~85% success on strict bot defense walls.
Single-Page Learning: If you're building a 1-time script for learning HTML parsing, stick to BeautifulSoup.

Start Monitoring in 60 Seconds

Sign up for a free developer account: https://pig-data.jp/service/scraping-ai/
Claim 200 free tokens (Instantly credited, no credit card required)
Install the Python SDK: pip install scraping-ai
Run your monitoring pipeline!

Pricing Tiers: Free (200 tokens) → Starter ($10 / 1,600 tokens) → Growth ($30 / 5,000 tokens) → Pro ($100 / 20,000 tokens)

API Documentation: https://pig-data.jp/service/scraping-ai/docs/

About the Team & Company

Scraping AI (https://pig-data.jp/service/scraping-ai/) is developed and operated by indigodata Inc., an AI venture subsidiary of SMS DataTech Co., Ltd. (Tokyo, Japan). Built upon PigData's track record of 500+ enterprise data extraction projects, Scraping AI provides a self-serve LLM extraction API for developers worldwide.

DEV Community