Monitor Any Website for Changes with Python (No Scraping Required)

We've all been there: waiting for a job posting to open, a product to come back in stock, a price to drop, or documentation to finally ship a fix. Manually refreshing the page every hour is a waste of time — and a sure way to miss the change when it actually lands.

The right tool is a change monitor: a small script that fetches a page on a schedule, compares it to the last version, and pings you when something is different. The annoying part has always been the "fetch and clean" step — most pages are 70% boilerplate (nav bars, ads, cookie banners), and you only care about the actual content.

That's exactly what the Web to Markdown/JSON API solves. It turns any URL into clean Markdown or JSON in one HTTP call, so your monitor compares content instead of noisy HTML.

In this article you'll build a working website change detector in Python. No BeautifulSoup, no selectors, no headless browsers.

The API at a glance

Endpoint:

POST https://web2md-api-production-d822.up.railway.app/extract

Request:

{
  "url": "https://example.com/pricing",
  "format": "markdown",
  "max_length": 50000
}

• format — markdown, json, or text
• max_length — cap the response size (up to 50000 characters)
• Free tier: 50 requests/day — plenty for a personal monitor
• Sign up / grab a key on RapidAPI

Response:

{
  "success": true,
  "title": "Pricing — Example Co",
  "content": "# Pricing\n\n## Plans\n\n...",
  "description": "Plan and pricing details",
  "word_count": 421,
  "response_time_ms": 210
}

The content field is clean Markdown. That's the key insight of this whole approach: we don't hash raw HTML (which changes whenever a tracking pixel or timestamp changes), we hash the readable content, so we only get alerted about changes that actually matter.

The plan

1. POST a URL to /extract and get Markdown back.
2. Hash the content with SHA-256.
3. Compare the hash to the last time we checked.
4. If it changed, save the new snapshot and notify you.

The code

Save this as watch.py:

import hashlib
import json
import sys
from pathlib import Path

import requests

API_URL = "https://web2md-api-production-d822.up.railway.app/extract"
STATE_DIR = Path("watch_state")


def fetch_content(url: str) -> dict:
    """Turn any web page into clean Markdown via the API."""
    r = requests.post(
        API_URL,
        json={"url": url, "format": "markdown", "max_length": 50000},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()


def hash_content(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def watch(url: str) -> bool:
    """Check one URL for changes. Returns True if something changed."""
    data = fetch_content(url)
    content = data["content"]
    digest = hash_content(content)

    STATE_DIR.mkdir(exist_ok=True)
    state_file = STATE_DIR / (hash_content(url)[:16] + ".json")

    if state_file.exists():
        previous = json.loads(state_file.read_text())
        if previous["digest"] == digest:
            print(f"  no change - {url}")
            return False
        print(f"  CHANGED  - {url}")
        print(f"    before: {previous['word_count']} words")
        print(f"    after:  {data['word_count']} words")
    else:
        print(f"  first check - {url} ({data['word_count']} words)")

    # Save the new snapshot (full content, so you can diff it later)
    state_file.write_text(json.dumps({
        "digest": digest,
        "word_count": data["word_count"],
        "title": data["title"],
        "content": content,
    }, indent=2))
    return True


if __name__ == "__main__":
    urls = sys.argv[1:] or [
        "https://example.com",
        "https://news.ycombinator.com",
    ]
    changed = [u for u in urls if watch(u)]
    print(f"\n{len(changed)} page(s) changed.")

How it works

• fetch_content — one POST to the API. The service fetches the page, strips nav/ads/scripts, and returns clean Markdown.
• hash_content — SHA-256 turns the content into a fixed-length fingerprint. Identical content → identical hash.
• watch — stores the last fingerprint (plus the full content so you can diff it) in a local JSON file. On each run it compares hashes and reports no change or CHANGED.

The whole trick is hashing clean Markdown instead of raw HTML. Raw HTML changes constantly — a new <script> tag here, a shuffled inline style there — which triggers false alarms all day long. Clean content only changes when something meaningful actually changed.

Run it

pip install requests
python watch.py https://example.com/pricing

The first run records a baseline; every run after that reports changes:

  first check - https://example.com/pricing (421 words)

  no change - https://example.com/pricing

  CHANGED  - https://example.com/pricing
    before: 421 words
    after:  438 words

Automating it

A change monitor is only useful if it runs on its own. Add a cron job to check every 30 minutes:

# every 30 minutes
*/30 * * * * cd /path/to/project && python watch.py https://example.com/pricing >> watch.log 2>&1

And if you want an actual notification instead of a log file, wire watch() up to any of these:

• Email — smtplib in the stdlib, ~10 lines
• Telegram — send a message via the Bot API in one requests.post
• Slack / Discord — an incoming webhook is a single line
• ntfy — a no-install push service that's designed exactly for this

For a zero-setup option, ntfy is hard to beat — pipe the change straight to your phone:

def notify(title: str, message: str):
    requests.post(
        "https://ntfy.sh/your_topic",
        data=message.encode("utf-8"),
        headers={"Title": title},
    )

Taking it further

• Watch a list — keep a urls.txt and loop over it. 50 free requests/day covers a lot of pages.
• Show the diff — use Python's difflib to print what changed, not just that it changed.
• Use format: "json" when you want structured paragraphs and headings instead of prose — handy for comparing only specific sections of a page.
• Track prices — many sites render prices inside Markdown tables, which turns "did the price change?" into a simple string comparison.

Wrapping up

Monitoring a website for changes is a classic "80% boring plumbing, 20% interesting problem" task. The Web to Markdown/JSON API deletes the boring 80% — fetching and cleaning arbitrary web pages — so you're left with the fun part: deciding what to watch and where to send the alerts.

Grab a free key (50 requests/day) on RapidAPI and stop refreshing pages manually.