How to Build a Competitor Monitoring Pipeline in 30 Minutes

Stop checking competitor prices manually. Here's the exact stack I use to monitor 50+ competitor URLs in real-time, without getting blocked.

What We're Building

A system that:

• Checks competitor URLs every 6 hours
• Extracts prices, titles, and availability
• Alerts you only when something significant changes
• Doesn't get blocked by anti-bot protection

The Stack

• XCrawl — residential proxies + scraping API
• Airtable — free, simple data storage with nice UI
• GitHub Actions — free cron scheduling
• Telegram bot — instant alerts on your phone

Total cost: under $20/month for 50 URLs.

Step 1: Define What You're Monitoring

Create a simple Airtable table with columns:

• url — competitor product page
• name — competitor or product name
• last_price — price from last check
• last_checked — timestamp
• change_alert — threshold % for alerts

Start with 10-20 URLs. You can always add more later.

Step 2: The Scraping Function

const { XCrawlScraper } = require('xcrawl-scraper');

const xcrawl = new XCrawlScraper({
  apiKey: process.env.XCRAWL_API_KEY,
});

async function checkCompetitor(url) {
  try {
    const result = await xcrawl.scrapeMarkdown(url, {
      render: true,
    });

    const markdown = result.data.markdown;

    const priceMatch = markdown.match(/\$[\d,]+\.?\d*/);
    const price = priceMatch
      ? parseFloat(priceMatch[0].replace(/[$,]/g, ''))
      : null;

    const titleMatch = markdown.match(/#\s+(.+)/);
    const title = titleMatch ? titleMatch[1].trim() : 'Unknown';

    return {
      url,
      price,
      title,
      checkedAt: new Date().toISOString(),
      success: true,
    };
  } catch (err) {
    return {
      url,
      price: null,
      title: null,
      checkedAt: new Date().toISOString(),
      success: false,
      error: err.message,
    };
  }
}

Step 3: Alert Logic

You don't want to be alerted every time a price changes by $0.01. You want to know when something significant happens:

function shouldAlert(lastPrice, currentPrice, threshold = 5) {
  if (!lastPrice || !currentPrice) return false;

  const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);
  return change >= threshold;
}

function formatAlert(competitor, lastPrice, currentPrice) {
  const direction = currentPrice < lastPrice ? 'DROPPED' : 'INCREASED';
  const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);

  return direction + ' ' + competitor.name + '
' +
    'Price: $' + lastPrice + ' -> ' + currentPrice + ' (' + change.toFixed(1) + '% change)
' +
    'URL: ' + competitor.url;
}

Step 4: GitHub Actions Scheduling

Create .github/workflows/competitor-check.yml:

name: Competitor Price Monitor
on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:        # Manual trigger

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install xcrawl-scraper airtable

      - name: Run competitor checks
        env:
          XCRAWL_API_KEY: ${{ secrets.XCRAWL_API_KEY }}
          AIRTABLE_API_KEY: ${{ secrets.AIRTABLE_API_KEY }}
        run: node competitor-check.js

      - name: Send alert to Telegram
        if: matrix.changed == true
        env:
          TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
          TG_CHAT_ID: ${{ secrets.TG_CHAT_ID }}
        run: |
          curl -s -X POST "https://api.telegram.org/bot$TG_BOT_TOKEN/sendMessage" -d "chat_id=$TG_CHAT_ID" -d "text=${{ matrix.alertMessage }}"

Step 5: The Full Check Loop

async function runChecks() {
  const competitors = await airtable.fetchCompetitors();
  const alerts = [];

  for (const competitor of competitors) {
    const result = await checkCompetitor(competitor.url);

    if (result.success) {
      await airtable.updateRecord(competitor.id, {
        last_price: result.price,
        last_checked: result.checkedAt,
        last_title: result.title,
      });

      if (shouldAlert(competitor.last_price, result.price)) {
        alerts.push(formatAlert(competitor, competitor.last_price, result.price));
      }
    } else {
      console.log('Failed: ' + competitor.url + ' - ' + result.error);
    }

    // Be respectful - don't hammer the servers
    await sleep(2000 + Math.random() * 3000);
  }

  if (alerts.length > 0) {
    await sendTelegramAlerts(alerts);
  }

  console.log('Checked ' + competitors.length + ' competitors. ' + alerts.length + ' alerts.');
}

What This Actually Looks Like in Practice

Here's the alert you'd get on your phone:

DROPPED Amazon Basics Mouse
Price: $29.99 -> $24.99 (16.7% change)
URL: amazon.com/dp/B07ZVPQR5M

That's it. No dashboards to check. No manual research. Just the signal you need to act.

Key Decisions That Made This Work

1. Residential proxies — datacenter IPs get blocked immediately on Amazon, Best Buy, etc.
2. 6-hour check interval — fast enough to catch changes, slow enough to not look like a bot
3. Store everything in Airtable — the UI makes it easy to spot patterns over time
4. Telegram over email — faster, and your phone will buzz to wake you up if needed

The ROI

My client monitors 47 competitor URLs across 8 product categories. In the first month:

• Caught 3 competitor price drops within hours
• Adjusted their own pricing proactively
• Estimated incremental revenue: $2,400 that month

System cost: $49/month (XCrawl) + free (GitHub Actions, Airtable, Telegram).

---

Building something similar? I've open-sourced the monitoring template. Link in bio. Questions below.