Building a Scalable URL Monitoring System with Node.js and MongoDB

#productivity #tools #webdev

Introduction

In today's digital age, monitoring website content changes is crucial for various use cases, such as price tracking, job alerts, and competitor website analysis. A scalable URL monitoring system can help automate this process, providing timely alerts and updates. In this article, we will explore the technical aspects of building such a system using Node.js and MongoDB.

System Requirements

To build a scalable URL monitoring system, we need to consider the following requirements:

URL storage: Store a large number of URLs to be monitored.
Content fetching: Fetch the content of each URL at regular intervals.
Change detection: Detect changes in the content of each URL.
Alert system: Send alerts when changes are detected.

URL Storage with MongoDB

We will use MongoDB as our database to store the URLs. MongoDB is a NoSQL database that allows for flexible and scalable data storage. We can create a collection to store the URLs, with each document containing the URL, last checked date, and last seen content.

const mongoose = require('mongoose');
const urlSchema = new mongoose.Schema({
  url: String,
  lastChecked: Date,
  lastSeenContent: String
});
const Url = mongoose.model('Url', urlSchema);

Content Fetching with Node.js

To fetch the content of each URL, we will use the axios library in Node.js. We can create a function that takes a URL as input, fetches its content, and returns the content as a string.

const axios = require('axios');
async function fetchContent(url) {
  try {
    const response = await axios.get(url);
    return response.data;
  } catch (error) {
    console.error(`Error fetching ${url}: ${error}`);
    return null;
  }
}

Change Detection

To detect changes in the content of each URL, we can use a simple string comparison. We will store the last seen content of each URL in our MongoDB database and compare it with the newly fetched content. If the contents are different, we will send an alert.

async function detectChanges(url) {
  const lastSeenContent = await Url.findOne({ url: url });
  const newContent = await fetchContent(url);
  if (newContent !== lastSeenContent.lastSeenContent) {
    // Send alert
  }
}

Alert System

To send alerts, we can use a library like nodemailer to send emails. We will create a function that takes the URL and the new content as input and sends an email with the updated content.

const nodemailer = require('nodemailer');
async function sendAlert(url, newContent) {
  const transporter = nodemailer.createTransport({
    host: 'smtp.example.com',
    port: 587,
    secure: false, // or 'STARTTLS'
    auth: {
      user: 'username',
      pass: 'password'
    }
  });
  const mailOptions = {
    from: 'sender@example.com',
    to: 'receiver@example.com',
    subject: `Update detected on ${url}`,
    text: newContent
  };
  transporter.sendMail(mailOptions, (error, info) => {
    if (error) {
      console.error(error);
    } else {
      console.log(`Alert sent: ${info.response}`);
    }
  });
}

Conclusion

Building a scalable URL monitoring system requires a robust technical architecture. By using Node.js and MongoDB, we can create a system that can handle a large number of URLs and provide timely alerts. Whether you're building a price tracking tool, a job alert system, or a competitor website analyzer, this system can help you stay on top of website content changes. If you're looking for a pre-built solution, you can try GetURLMonitor, a SaaS tool that offers a free plan with 3 URLs and a pro plan with unlimited URLs, checking every hour.

DEV Community