Building a Scalable URL Monitoring System with Python and Cloud Services

#productivity #tools #webdev

Introduction

Monitoring website content changes can be a useful tool for various applications, such as price tracking, job alerts, or competitor website changes. In this article, we will explore the technical aspects of building a scalable URL monitoring system using Python and cloud services.

Architecture Overview

Our system will consist of the following components:

URL Storage: a database to store the URLs to be monitored
Content Checker: a service that periodically checks the content of the stored URLs
Notification Service: a service that sends notifications when a content change is detected
Scheduler: a service that schedules the content checks

URL Storage

We will use a relational database management system like MySQL or PostgreSQL to store the URLs. The database schema will consist of a single table with the following columns:

id: a unique identifier for each URL
url: the URL to be monitored
last_checked: the timestamp of the last content check
last_content: the last known content of the URL

Content Checker

The content checker will be a Python script that uses the requests library to fetch the content of the URL and the BeautifulSoup library to parse the HTML content. We will use a hash function like md5 to generate a unique hash for each content. If the hash changes, we will detect a content change.

import requests
from bs4 import BeautifulSoup
import hashlib

def check_content(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    content = soup.get_text()
    hash = hashlib.md5(content.encode()).hexdigest()
    return hash

Notification Service

The notification service will be responsible for sending email notifications when a content change is detected. We will use a cloud-based email service like Amazon SES or Sendgrid to send the emails.

import boto3

ses = boto3.client('ses')

def send_notification(url, new_content):
    ses.send_email(
        Source='your_email@example.com',
        Destination={
            'ToAddresses': ['recipient_email@example.com']
        },
        Message={
            'Body': {
                'Text': {
                    'Data': f'Content change detected on {url}:\n{new_content}'
                }
            }
        }
    )

Scheduler

The scheduler will be responsible for scheduling the content checks. We will use a cloud-based scheduler like Amazon CloudWatch Events or Google Cloud Scheduler to schedule the checks.

import schedule
import time

def schedule_checks():
    schedule.every(1).hours.do(check_all_urls)  # check all URLs every hour

def check_all_urls():
    # fetch all URLs from the database
    # check the content of each URL
    # send notifications if content changes are detected
    pass

Conclusion

Building a scalable URL monitoring system requires a robust architecture and the use of cloud services. By using Python and cloud services, we can build a system that is scalable, reliable, and efficient. If you're interested in trying out a URL monitoring system, check out GetURLMonitor, a SaaS tool that offers a free plan with 3 URLs and checks every 24 hours, as well as a pro plan with unlimited URLs and checks every hour.

DEV Community