DEV Community

bao001 xiao
bao001 xiao

Posted on

How to Monitor Website Changes with the Web to Markdown/JSON API

#tutorial #webdev #api #python

Introduction

Keeping track of changes on websites—whether it's competitor pricing, job listings, government policy updates, or product release notes—is a common need for developers and businesses alike. But writing a full-blown web scraper from scratch for every site you want to monitor? That's brittle, time-consuming, and often breaks the moment the target site tweaks its layout.

Enter the Web to Markdown/JSON API: a simple, powerful API that converts any webpage into clean, structured Markdown or JSON in a single request. It handles all the messy HTML parsing, boilerplate stripping, and formatting so you can focus on what actually matters—detecting changes and acting on them.

In this tutorial, you'll build a website change monitor in Python that:

  • Fetches any web page as clean Markdown via the API
  • Compares snapshots over time to detect meaningful changes
  • Sends email alerts when something important shifts
  • Runs on a cron schedule for fully automated monitoring

Let's dive in.

The API at a Glance

Endpoint: POST https://web2md-api-production-d822.up.railway.app/extract

Request body:

{
  "url": "https://example.com",
  "format": "markdown",
  "max_length": 50000
}
Enter fullscreen mode Exit fullscreen mode

Formats supported: markdown, json, text

Free tier: 50 requests per day

RapidAPI page: web-to-markdown-json-api

This API does one thing and does it well: give it a URL, and it returns the page's meaningful content stripped of navigation, ads, and scripting cruft. Exactly what you need for change detection—no BeautifulSoup or regex wrestling required.

Step 1: Fetching a Page Snapshot

Create a file called monitor.py. We'll start with a function that takes a URL and returns the clean Markdown content:

import requests

API_ENDPOINT = "https://web2md-api-production-d822.up.railway.app/extract"

def fetch_snapshot(url: str, format: str = "markdown", max_length: int = 50000) -> str:
    resp = requests.post(
        API_ENDPOINT,
        json={"url": url, "format": format, "max_length": max_length},
        timeout=30
    )
    resp.raise_for_status()
    data = resp.json()
    return data.get("markdown") or data.get("content", "")

# Quick test
if __name__ == "__main__":
    content = fetch_snapshot("https://example.com")
    print(content[:500])
    print(f"\n--- Total length: {len(content)} characters")
Enter fullscreen mode Exit fullscreen mode

Run it and you'll see clean, readable Markdown instead of raw HTML—a huge win before we even start the monitoring logic.

Step 2: Storing and Comparing Snapshots

To detect changes, we need to store previous snapshots and compare them against new ones. We'll use a simple JSON file as our snapshot database:

import hashlib
import json
import os
from datetime import datetime

SNAPSHOT_FILE = "snapshots.json"

def load_snapshots() -> dict:
    if os.path.exists(SNAPSHOT_FILE):
        with open(SNAPSHOT_FILE) as f:
            return json.load(f)
    return {}

def save_snapshots(snapshots: dict):
    with open(SNAPSHOT_FILE, "w") as f:
        json.dump(snapshots, f, indent=2, default=str)

def compute_hash(content: str) -> str:
    return hashlib.sha256(content.encode("utf-8")).hexdigest()

def compare_snapshot(url: str, new_content: str):
    snapshots = load_snapshots()
    new_hash = compute_hash(new_content)

    if url not in snapshots:
        snapshots[url] = {
            "hash": new_hash,
            "timestamp": datetime.now().isoformat(),
            "length": len(new_content)
        }
        save_snapshots(snapshots)
        return None  # Baseline established

    old = snapshots[url]
    if old["hash"] == new_hash:
        return None  # No change

    length_delta = len(new_content) - old.get("length", 0)

    snapshots[url] = {
        "hash": new_hash,
        "timestamp": datetime.now().isoformat(),
        "length": len(new_content)
    }
    save_snapshots(snapshots)

    return {
        "url": url,
        "previous_checked": old["timestamp"],
        "length_delta": length_delta,
        "length_delta_pct": round(
            (length_delta / old["length"]) * 100, 1
        ) if old["length"] > 0 else 0
    }
Enter fullscreen mode Exit fullscreen mode

This gives us hash-based equality checks (fast and reliable) plus a quick size-delta metric.

Step 3: Detecting Meaningful Changes

Not all changes matter. A page might change its timestamp, or a random ad insertion could shift content slightly. Let's add smart thresholds:

CHANGE_THRESHOLD = 0.02  # 2% content change before we alert

def detect_meaningful_change(url: str, new_content: str):
    diff = compare_snapshot(url, new_content)

    if diff is None:
        return None

    abs_pct = abs(diff["length_delta_pct"])
    if abs_pct < CHANGE_THRESHOLD * 100:
        print(f"Change too small ({abs_pct}%), skipping alert for {url}")
        return None

    return diff
Enter fullscreen mode Exit fullscreen mode

Tune CHANGE_THRESHOLD per URL—higher for noisy pages, lower for pages where every word counts.

Step 4: Sending Email Alerts

When a meaningful change is detected, fire off an email using Python's built-in smtplib:

import smtplib
from email.mime.text import MIMEText

SMTP_HOST = "smtp.gmail.com"
SMTP_PORT = 587
SMTP_USER = "alerts@yourdomain.com"
SMTP_PASS = "your-app-password"
ALERT_EMAIL = "you@yourdomain.com"

def send_alert(change: dict):
    subject = f"Web Change Detected: {change['url']}"
    body = (
        f"A change was detected on a monitored page.\n\n"
        f"URL: {change['url']}\n"
        f"Last checked: {change['previous_checked']}\n"
        f"Size change: {change['length_delta']:+d} chars "
        f"({change['length_delta_pct']:+}%)\n\n"
        f"See: {change['url']}\n"
    )
    msg = MIMEText(body)
    msg["Subject"] = subject
    msg["From"] = SMTP_USER
    msg["To"] = ALERT_EMAIL

    with smtplib.SMTP(SMTP_HOST, SMTP_PORT) as server:
        server.starttls()
        server.login(SMTP_USER, SMTP_PASS)
        server.send_message(msg)

    print(f"Alert sent for {change['url']}")
Enter fullscreen mode Exit fullscreen mode

Step 5: Putting It All Together

The main monitoring loop—feed it a list of URLs and it checks them all:

import time

MONITORED_URLS = [
    "https://example.com/pricing",
    "https://example.com/changelog",
    "https://example.com/jobs",
]

def run_monitor(urls: list[str]):
    print(
        f"Starting monitor for {len(urls)} URLs "
        f"at {datetime.now().isoformat()}"
    )

    for url in urls:
        try:
            print(f"Checking: {url}")
            content = fetch_snapshot(url)
            change = detect_meaningful_change(url, content)

            if change:
                print(
                    f"  -> CHANGE DETECTED! "
                    f"Size delta: {change['length_delta']:+d} chars"
                )
                # send_alert(change)  # Uncomment when SMTP configured
            else:
                print(f"  -> No meaningful change")

            time.sleep(1)  # Respect free tier limits

        except Exception as e:
            print(f"  -> ERROR for {url}: {e}")

    print("Monitor run complete.")

if __name__ == "__main__":
    run_monitor(MONITORED_URLS)
Enter fullscreen mode Exit fullscreen mode

Step 6: Scheduling with Cron

Once working, schedule it to run daily:

# Every day at 8 AM
0 8 * * * cd /path/to/project && python3 monitor.py >> monitor.log 2>&1

# Every 6 hours for fast-moving pages
0 */6 * * * cd /path/to/project && python3 monitor.py >> monitor.log 2>&1
Enter fullscreen mode Exit fullscreen mode

With the free tier (50 requests/day), monitor 40-45 URLs daily with a safety buffer.

Advanced Ideas

1. Diff Highlighting

Store full Markdown content and use difflib for human-readable diffs:

import difflib

def generate_diff(old: str, new: str) -> str:
    return "\n".join(difflib.unified_diff(
        old.splitlines(), new.splitlines(), lineterm=""
    ))
Enter fullscreen mode Exit fullscreen mode

2. Per-URL Thresholds

Some pages change frequently, others rarely:

URL_CONFIG = {
    "https://example.com/news": {"threshold": 0.10},
    "https://example.com/terms": {"threshold": 0.01},
}
Enter fullscreen mode Exit fullscreen mode

3. JSON Format for Structured Data

Switch format to "json" when monitoring structured fields:

resp = requests.post(
    API_ENDPOINT,
    json={
        "url": "https://example.com/product/123",
        "format": "json",
        "max_length": 20000
    }
)
# resp.json() returns structured data with title, content, metadata
Enter fullscreen mode Exit fullscreen mode

4. LLM-Powered Summaries

Pipe the Markdown diff into an LLM and ask for a one-paragraph summary of what changed. Turns raw deltas into actionable intelligence.

Why This API Over DIY Scraping?

Aspect Web to Markdown/JSON API DIY Scraping
Setup time 1 API call Hours of BeautifulSoup + custom parsing
Maintenance Handled by the API You fix it every site redesign
Content quality Clean Markdown, boilerplate stripped Raw HTML to parse yourself
Multiple formats Markdown, JSON, plain text You implement each
Rate limiting Free tier included You write retry logic

Wrapping Up

With under 100 lines of Python and the Web to Markdown/JSON API, you have a fully automated website change monitor that:

  • Fetches clean, readable content from any URL
  • Compares snapshots and detects meaningful changes
  • Alerts by email when something shifts
  • Runs on autopilot via cron

This pattern is production-ready for monitoring competitor pricing, government notices, job boards, documentation updates, and anything else on the web that matters.

Try It Now

Grab a key on RapidAPI or hit the endpoint directly:

curl -X POST https://web2md-api-production-d822.up.railway.app/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown", "max_length": 50000}'
Enter fullscreen mode Exit fullscreen mode

The first 50 requests each day are free. Go build something useful!

Top comments (0)