DEV Community

Harry
Harry

Posted on

Building a 90MB E-Commerce Price Intelligence Tool for AI Agents using Apify MCP Server

#webscraping #javascript #apify #ai

Running e-commerce competitor price monitoring in production usually forces a painful compromise: pay $200+/month for SaaS platforms like Prisync, or host custom Puppeteer/Playwright scrapers that chew through 1.5GB of RAM per instance just to extract basic price tags.

When I started building price tracking workflows, I wanted something faster and vastly cheaper: a lightweight HTTP scraper that runs on 128MB RAM, extracts Schema.org JSON-LD microdata in under 1 second, and connects directly to AI agents via the Apify Model Context Protocol (MCP) Server (https://mcp.apify.com/).

In this tutorial, I'll walk through how I built the Smart E-Commerce Price & Stock Monitor Actor, how it handles tricky real-world edge cases like SPA detection and 24-hour price bounce-back deduplication, and how you can expose it as an executable tool to LLM agents like Claude, Cursor, and n8n.

The Architecture: Why 90MB HTTP Beats 1.5GB Headless Browsers

Most modern direct-to-consumer (D2C) brand stores built on Shopify, WooCommerce, Magento, or Salesforce Commerce Cloud render product metadata on the server using Schema.org JSON-LD microdata.

Spinning up headless Chrome to render a full DOM when the price data already exists inside a <script type="application/ld+json"> tag is massive resource waste.

Here is the high-level architecture of the Actor:

[ E-Commerce Product URLs ]
          │
          ▼
┌─────────────────────────────────────────┐
│ got-scraping + CheerioCrawler (85MB RAM) │
└────────────────────┬────────────────────┘
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
┌──────────────────┐   ┌──────────────────────┐
│ Tier 1: JSON-LD  │   │ Tier 2: CSS Fallback │
│  Extractor       │   │   Selector Engine    │
└─────────┬────────┘   └──────────┬───────────┘
          └──────────┬────────────┘
                     ▼
┌─────────────────────────────────────────┐
│ Stateful Diffing & Bounce-Back Dedup    │
│  (Key-Value Store: STATE.json)          │
└────────────────────┬────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────┐
│ Output & Webhook (Telegram / n8n / MCP) │
└─────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Key Engineering Decisions

  1. got-scraping + CheerioCrawler: Replaces heavy browser automation. Uses HTTP/2 fingerprint matching to bypass basic anti-bot TLS fingerprinting while staying under 90MB RAM footprint.
  2. 2-Tier Extraction Engine:
    • Tier 1 (JSON-LD): Parses @type: "Product" or "Offer" schemas automatically.
    • Tier 2 (CSS Fallback): Pre-configured DOM selector engine for stores with incomplete microdata.
  3. SPA Auto-Detection: When a site serves an empty client-side React/Angular skeleton (e.g. Zara), the Actor gracefully flags jsRenderingRequired: true rather than crashing or swallowing missing fields silently.

Connecting the Actor to AI Agents via Apify MCP Server

Apify recently launched the Model Context Protocol (MCP) Server (https://mcp.apify.com/). The Model Context Protocol is an open standard that allows LLMs (like Claude Desktop, Cursor, or n8n AI Agent nodes) to discover and run external tools dynamically.

Because any Actor published on Apify Store is automatically exposed through https://mcp.apify.com/, AI agents can invoke our price monitor as a native function call.

How the Input Schema Maps to MCP Tool Call Parameters

For an LLM to call an Actor reliably via MCP without syntax errors, the Actor's .actor/INPUT_SCHEMA.json must be strictly typed. Here is how our input configuration maps:

{
  "productUrls": [
    "https://skims.com/products/fits-everybody-t-shirt-bra-onyx",
    "https://www.allbirds.com/products/mens-tree-runners-fog"
  ],
  "minPriceChangePercentage": 0,
  "notifyOnFirstRun": true,
  "maxConcurrencyPerDomain": 2
}
Enter fullscreen mode Exit fullscreen mode

Keeping Payload Output Compact (<500 Tokens)

A common mistake when building web scrapers for LLM consumption is returning raw HTML or unformatted JSON objects containing thousands of lines. This quickly floods the AI agent's context window and inflates API cost.

Our Actor returns an ultra-compact, token-efficient payload format:

{
  "url": "https://skims.com/products/fits-everybody-t-shirt-bra-onyx",
  "title": "FITS EVERYBODY T-SHIRT BRA | ONYX | 30 AA",
  "currency": "EUR",
  "currentPrice": 68.00,
  "previousPrice": null,
  "priceChanged": false,
  "inStock": true,
  "isNewProduct": true,
  "shouldAlert": true,
  "jsRenderingRequired": false,
  "checkedAt": "2026-08-11T10:36:46.935Z"
}
Enter fullscreen mode Exit fullscreen mode

This record takes ~120 tokens to parse, allowing an AI agent to analyze dozens of product price alerts in a single prompt without blowing token budgets.

Real-World Production Gotchas & Code Solutions

Building a scraper that works in a local test script is easy; making it survive weeks of continuous production monitoring requires handling real-world edge cases.

Gotcha 1: The 24-Hour Price Bounce-Back Bug

During community testing on n8n, a community member (Cloudrocket) pointed out a subtle edge case in our deduplication logic:

Suppose a product price drops from $100 to $80 at 10 AM. An alert is sent and the hash md5(url|80) is saved for 24 hours. At 2 PM, the price reverts to $100 (alert sent). At 6 PM, another flash sale drops the price back to $80. Because md5(url|80) was saved 8 hours prior, the second drop gets swallowed!

To fix this without allowing threshold spam, I refactored stateStore.js to track price transitions rather than static target prices, and automatically invalidate outdated keys for that product URL upon any state change:

// src/services/stateStore.js

if (shouldAlert) {
    const currency = currentData.currency || 'USD';
    const prevPriceVal = prevItem && prevItem.price !== null ? prevItem.price : 'INIT';

    // Hash key combines product URL hash and transition signature (e.g. 100->80)
    const urlHash = crypto.createHash('md5').update(url).digest('hex').substring(0, 12);
    const transitionSignature = `${prevPriceVal}->${currentPrice}|${currency}|${currentInStock}`;
    const transitionHash = crypto.createHash('md5').update(transitionSignature).digest('hex').substring(0, 12);
    const dedupKey = `${urlHash}:${transitionHash}`;

    const DEDUP_WINDOW_MS = 24 * 60 * 60 * 1000; // 24 hours
    const lastSentAt = this.state.sentAlerts[dedupKey];

    if (lastSentAt && (Date.now() - lastSentAt) < DEDUP_WINDOW_MS) {
        shouldAlert = false;
        logger.debug(`Dedup: skipping duplicate alert for ${url} (key ${dedupKey} sent ${Math.round((Date.now() - lastSentAt) / 3600000)}h ago).`);
    } else {
        // Invalidate stale dedup keys for this product URL to allow legitimate bounce-backs
        const urlPrefix = `${urlHash}:`;
        for (const key of Object.keys(this.state.sentAlerts)) {
            if (key.startsWith(urlPrefix)) {
                delete this.state.sentAlerts[key];
            }
        }
        this.state.sentAlerts[dedupKey] = Date.now();
    }
}
Enter fullscreen mode Exit fullscreen mode

Gotcha 2: Client-Side SPA Handling (React / Next.js)

When static scrapers encounter a client-side rendered Single Page Application like Zara, they typically fail silently or return null price fields.

In src/main.js, we detect this condition and attach actionable guidance to the dataset record:

// src/main.js (SPA Detection)

let jsRenderingRequired = false;
let userGuidance = null;

if (productData.price === null || productData.title === 'Unknown Product') {
    jsRenderingRequired = true;
    userGuidance = '⚠️ Product title or price could not be detected statically. This page may require JavaScript rendering (React/SPA) or custom CSS selectors in Actor input settings.';
    logger.warning(`Unable to extract price or title for ${url}. Page may require JavaScript rendering (SPA React/Next.js) or custom CSS selectors.`);
}

productData.jsRenderingRequired = jsRenderingRequired;
if (userGuidance) {
    productData.userGuidance = userGuidance;
}
Enter fullscreen mode Exit fullscreen mode

Real Test Execution Benchmarks

Running a live benchmark test across 5 real D2C e-commerce stores (Skims, Velasca, Allbirds, Gymshark, and Zara):

INFO  System info {"apifyVersion":"3.7.2","crawleeVersion":"3.17.0","osType":"Windows_NT"}
INFO  Starting Smart E-Commerce Price & Stock Monitor for 5 product URL(s).
INFO  Processing product page: https://www.velasca.com/products/spigolatt-marrone-scuro
INFO  Processing product page: https://skims.com/products/fits-everybody-t-shirt-bra-onyx
INFO  Processing product page: https://www.allbirds.com/products/mens-tree-runners-fog
INFO  Processing product page: https://www.zara.com/it/it/t-shirt-strutturata-p00994300.html
WARN  Unable to extract price or title for https://www.zara.com/it/it/t-shirt-strutturata-p00994300.html. Page may require JavaScript rendering (SPA React/Next.js) or custom CSS selectors.
INFO  Processing product page: https://www.gymshark.com/products/gymshark-vital-seamless-2-0-leggings-black-marl-aw21
INFO  CheerioCrawler: Finished! Total 5 requests: 5 succeeded, 0 failed.
Enter fullscreen mode Exit fullscreen mode

Benchmark Results

  • Execution Time: 3.64 seconds for 5 stores.
  • Memory Used: ~85 MB RAM.
  • Success Rate: 100% (5/5 HTTP requests finished cleanly).

Summary & Resources

By pairing lightweight HTTP static scraping with Schema.org JSON-LD microdata extraction, you can build e-commerce price monitoring systems that cost pennies to run and integrate seamlessly into AI agent workflows via Model Context Protocol.

Links & Code

Top comments (0)