DEV Community

Mohammad Ehsan Ansari
Mohammad Ehsan Ansari

Posted on

Monitor Website Changes with Python, Streamlit, Slack & Olostep

#programming #webscraping

Souce code

🛠️ Monitor Website Changes with Python, Streamlit, Slack & Olostep

Keeping tabs on your competitors or your own brand pages is crucial — changes in pricing, content, or job openings can have a direct impact on your strategy. This blog shows how to monitor any public webpage for updates using:

✅ Python

✅ Streamlit

✅ Slack for alerts

✅ Olostep API for web scraping

🎯 What You’ll Learn

  • How to detect pricing updates, new jobs, copy changes, new articles/pages, logos, and more
  • How to use Olostep’s scraping API to retrieve website content
  • How to compare two content versions and find changes
  • How to notify your team via Slack when a change is detected
  • How to visualize and interact with the app using Streamlit

🔧 Step 1: Install Dependencies

Start by installing the necessary packages:

pip install streamlit requests python-dotenv slack_sdk difflib
Enter fullscreen mode Exit fullscreen mode

You’ll also need:

  • A Slack bot token
  • Your Slack channel ID
  • Your Olostep API key

Create a .env file to securely store them:

OLOSTEP_API_KEY=your_olostep_key
SLACK_BOT_TOKEN=xoxb-your-slack-token
SLACK_CHANNEL_ID=your-channel-id
Enter fullscreen mode Exit fullscreen mode

🌐 Step 2: Set Up Web Scraping with Olostep

We use Olostep’s /scrapes endpoint to fetch the raw HTML or markdown content of any URL.

def scrape_url(url):
    headers = {
        "Authorization": f"Bearer {os.getenv('OLOSTEP_API_KEY')}",
        "Content-Type": "application/json"
    }
    payload = {
        "formats": ["markdown"],
        "country": "US",
        "url_to_scrape": url,
    }
    response = requests.post("https://api.olostep.com/v1/scrapes", headers=headers, json=payload)
    return response.json()["result"]["markdown_content"]
Enter fullscreen mode Exit fullscreen mode

🧠 Why markdown? It's easier to analyze than raw HTML and more structured for identifying changes.

🧠 Step 3: Compare New vs Old Content

We compare the newly scraped content with the last saved version using Python’s difflib.

def compare_versions(old_text, new_text):
    diff = list(difflib.unified_diff(
        old_text.splitlines(), 
        new_text.splitlines(), 
        lineterm="", 
        n=2
    ))
    return "
".join(diff)
Enter fullscreen mode Exit fullscreen mode

We store the previous version in a local file (latest.md) and load it when the app runs again.

🔔 Step 4: Notify via Slack

When a change is detected, send a message to your Slack workspace:

from slack_sdk import WebClient

def send_slack_notification(change_text):
    client = WebClient(token=os.getenv("SLACK_BOT_TOKEN"))
    message = f"🔍 *Website Change Detected!*
Enter fullscreen mode Exit fullscreen mode


{change_text[:3000]}

    client.chat_postMessage(channel=os.getenv("SLACK_CHANNEL_ID"), text=message)
Enter fullscreen mode Exit fullscreen mode

📌 Slack messages are capped to ~3000 chars for readability. You can link to a diff file if needed.

🖼️ Step 5: Build the Streamlit App

We use Streamlit for a clean UI to trigger checks and view results.

st.title("🔎 Website Change Monitor")

url = st.text_input("Enter the website URL to monitor")

if st.button("Check for Changes"):
    new_content = scrape_url(url)

    if os.path.exists("latest.md"):
        old_content = Path("latest.md").read_text()
        diff = compare_versions(old_content, new_content)

        if diff:
            send_slack_notification(diff)
            st.success("Changes detected and sent to Slack!")
        else:
            st.info("No changes found.")
    else:
        st.info("First time check. Saving initial version.")

    Path("latest.md").write_text(new_content)
Enter fullscreen mode Exit fullscreen mode

🧪 Example: Monitoring a Competitor's Pricing Page

  • Add your competitor's pricing page: https://example.com/pricing
  • Run the Streamlit app
  • It scrapes content, checks for diff, and posts to Slack if any changes are found
  • Useful for:
    • Pricing wars
    • Detecting seasonal offers
    • New plans or structure

🧠 Use Cases

  • ✅ Monitor your own site for accidental changes
  • ✅ Keep an eye on competitors
  • ✅ Detect new job postings on a careers page
  • ✅ Track new testimonials, logos, or articles

🛠️ Future Enhancements

  • Save diffs to a database
  • Visualize changes over time
  • Monitor multiple pages in parallel
  • Email alerts as a fallback

✅ Final Thoughts

This simple automation stack — built with Python + Streamlit + Slack + Olostep — empowers you to stay ahead of competitors or monitor any site changes. Customize it to track any page that matters.

Happy hacking 🚀

Top comments (0)