DEV Community

agenthustler
agenthustler

Posted on • Edited on

How to Monitor Website Changes and Get Alerts with Python

How to Monitor Website Changes and Get Alerts with Python

Websites change constantly. Automated change detection saves hours of manual checking whether you track pricing, regulatory updates, or stock availability.

Core Monitor

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).
Enter fullscreen mode Exit fullscreen mode

Smart Change Detection

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).
Enter fullscreen mode Exit fullscreen mode

Alert System

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).
Enter fullscreen mode Exit fullscreen mode

Running the Monitor

class Runner:
    def __init__(self, monitor, alerts):
        self.monitor = monitor
        self.alerts = alerts

    def cycle(self):
        cursor = self.monitor.db.execute('''
            SELECT url, selector FROM pages
            WHERE last_checked IS NULL
            OR (julianday('now')-julianday(last_checked))*86400 > interval
        ''')
        changes = 0
        for url, sel in cursor.fetchall():
            change = self.monitor._check(url, sel)
            if change and 'error' not in change and change.get('type') != 'trivial':
                change['url'] = url
                self.alerts.send(change)
                changes += 1
            time.sleep(2)
        return changes

    def run_forever(self, interval=60):
        while True:
            n = self.cycle()
            print(f"[{datetime.now()}] {n} changes found")
            time.sleep(interval)

monitor = WebMonitor(api_key='YOUR_KEY')
alerts = Alerts()
alerts.add_webhook('https://hooks.slack.com/services/YOUR/WEBHOOK')
monitor.add_watch('https://example.com/pricing', '.pricing-table', 3600)
Runner(monitor, alerts).run_forever()
Enter fullscreen mode Exit fullscreen mode

For hundreds of pages, ScraperAPI handles JS rendering. ThorData provides proxy rotation. Monitor health with ScrapeOps.


Follow for more Python monitoring tutorials.

Top comments (0)