Building a Practical Ecommerce Price Intelligence Pipeline

You change a product price, ship the deploy, and a few hours later the numbers are already stale. A competitor dropped the same SKU by 5%, Google Shopping picked it up, and your product detail page now looks expensive. The hard part is not noticing this once. The hard part is building a system that notices it every day without filling your database with bad matches and broken scrape results.

Price intelligence is mostly a data pipeline problem

At a practical level, ecommerce price intelligence means collecting competitor prices, matching them to your catalog, storing snapshots, and deciding what to do with the difference.

A minimal pipeline looks like this:

1. Maintain a list of products you care about.
2. Map each product to competitor URLs or marketplace identifiers.
3. Collect price, availability, seller, shipping, promo, and timestamp.
4. Normalize currencies, units, variants, and taxes.
5. Store every observation, not just the latest one.
6. Alert or recommend actions when the gap crosses a threshold.

The mistake I see most often is treating this like a single scraping script. The scraper matters, but the matching and interpretation matter more. If you compare your 2-pack item against a competitor's single unit, your pricing rule will confidently produce nonsense.

A basic table layout might start like this:

CREATE TABLE competitor_price_observations (
  id BIGSERIAL PRIMARY KEY,
  internal_sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  competitor_url TEXT NOT NULL,
  observed_price_cents INTEGER,
  currency CHAR(3) NOT NULL,
  in_stock BOOLEAN,
  seller_name TEXT,
  raw_title TEXT,
  raw_payload JSONB,
  observed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  scrape_status TEXT NOT NULL
);

CREATE INDEX idx_price_observations_sku_time
  ON competitor_price_observations (internal_sku, observed_at DESC);

Keep raw_payload. You will need it when someone asks why yesterday's price looked wrong.

Scraping the price is the easy part until it isn't

Some sites expose structured product data through JSON-LD. When they do, use it before reaching for fragile CSS selectors.

import json
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 price-monitor/1.0"
}

def extract_json_ld_price(url: str):
    response = requests.get(url, headers=HEADERS, timeout=15)

    if response.status_code == 403:
        raise RuntimeError(f"blocked by target site: HTTP 403 for {url}")

    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "{}")
        except json.JSONDecodeError:
            continue

        items = data if isinstance(data, list) else [data]
        for item in items:
            if item.get("@type") != "Product":
                continue

            offers = item.get("offers")
            if isinstance(offers, list):
                offers = offers[0]

            if not offers:
                continue

            return {
                "title": item.get("name"),
                "price": offers.get("price"),
                "currency": offers.get("priceCurrency"),
                "availability": offers.get("availability"),
            }

    raise RuntimeError(f"no product price found in JSON-LD for {url}")

This works well for simple pages. It fails when the site renders prices client-side, hides prices behind location selection, changes markup during experiments, or blocks repeated requests. The symptoms are usually obvious: HTTP 403, timeouts, None prices, or sudden drops to zero observations for one competitor.

Do not let those failures flow into pricing decisions as if they were real prices. Store scrape_status = 'failed' and keep the last valid observation separate from the latest attempt.

If maintaining site-specific extractors becomes the main work, Wire is one managed extraction layer teams use for ecommerce price, availability, and product data instead of owning every parser themselves.

Product matching needs defensive checks

Exact URL mapping is safest, but it does not always scale. Marketplaces have multiple sellers, duplicate listings, regional URLs, and variant pages. If you use title similarity, add guardrails.

For example, do not match only on text similarity:

def is_reasonable_match(our_product, competitor_product):
    if our_product["brand"].lower() != competitor_product["brand"].lower():
        return False

    if our_product.get("mpn") and competitor_product.get("mpn"):
        return our_product["mpn"] == competitor_product["mpn"]

    if our_product["pack_size"] != competitor_product["pack_size"]:
        return False

    if abs(our_product["size_ml"] - competitor_product["size_ml"]) > 5:
        return False

    return competitor_product["title_score"] >= 0.88

The exact fields depend on your category. Shoes need size and color. Grocery needs unit size and pack count. Electronics need model numbers. If you skip this, your dashboard may look complete while comparing different products.

Frequency should match how prices move

A daily scrape is fine for slow categories. It is not fine for marketplaces, seasonal goods, flash sales, or categories where automated repricing is common.

A simple schedule might be:

high_velocity_skus: every 15 minutes
core_catalog: every 2 hours
long_tail_catalog: daily
new_competitor_discovery: weekly

More frequency means more cost, more blocks, and more noisy data. Less frequency means you miss short-lived promotions. Pick a cadence based on how often the data changes, not how often the cron job is easy to run.

Alerts should explain the decision

A useful alert says more than "competitor cheaper". Include the SKU, competitor, current price, observed price, previous observed price, timestamp, and confidence level.

def should_alert(our_price_cents, competitor_price_cents, min_gap_percent=3):
    if competitor_price_cents is None:
        return False

    gap = (our_price_cents - competitor_price_cents) / our_price_cents * 100
    return gap >= min_gap_percent

Also add constraints before anyone wires this into automatic repricing:

• minimum margin
• MAP or contractual pricing limits
• inventory level
• competitor stock status
• shipping cost differences
• stale observation cutoff

A competitor price from 36 hours ago should not trigger a price cut today.

Start small and measure data quality

Track 20 to 50 important SKUs across two or three competitors first. Measure match accuracy, scrape failure rate, price freshness, and false alerts. Those numbers will tell you whether you need better extraction, better matching, or better pricing rules.

A practical next step: build the observation table, collect one week of snapshots, and review every alert manually before letting the data influence production prices.