Snapshots Are Worthless, Deltas Are the Product: A Change-Detection Engine for Competitor Feeds
If you read the previous article in this series, you already know what to collect from competitor product pages: price, availability, shipping cost, seller identity, offer count. What that article didn't cover — and what quietly decides whether your monitoring system is useful or a firehose of garbage — is what happens after collection. Nobody consumes snapshots. Repricers, alerting pipelines, and dashboards consume events: price changed, stock flipped, a new offer appeared. The snapshot store is a cost center. The delta stream is the product.
The uncomfortable truth is that a naive diff over consecutive snapshots produces mostly phantom events. Over a labeled week of data on one e-commerce vertical, we measured that raw field-level diffs produced roughly 11 candidate "price changes" per SKU, of which fewer than one per SKU was a real competitor move. Everything else was noise. Before you write a single alert rule, you need a change-detection engine that earns the right to say "this changed."
The phantom-change taxonomy
Know your enemies before you tune thresholds against them:
A/B price experiments. Large retailers serve different prices to different sessions, splitting traffic to measure conversion lift. If your scraper's session rotates, you'll observe price A, then price B, then A again — and a naive differ emits three events from zero actual changes. This is the single biggest noise source, and it looks exactly like oscillation.
Floating-point and formatting noise. One crawl renders 1,299.00, the next 1299, the third 1299.00000001 after a bad cast somewhere in your pipeline. These are representation changes, not market changes. Parse to a normalized decimal once, at ingestion, and never diff strings.
Currency and locale. Geo-redirects and locale negotiation can flip the served currency mid-stream. A "price drop" from 100 to 92 might just be EUR rendered instead of USD. Normalize currency upstream or tag observations with the currency and refuse to diff across it.
Seller rotation on marketplaces. On marketplace listings, the buy-box winner rotates among sellers. Price "changes" are often just different sellers winning the box. That's a real signal, but it's a seller event, not a price event — conflating them destroys both signals.
Decoy prices for suspected bots. Sophisticated anti-bot systems serve distorted prices to sessions they distrust. You don't get an error; you get plausible-looking data that's wrong. The only structural defense is upstream: stable residential sessions on consistent IPs so consecutive snapshots come from the same trust level. If you bounce between datacenter IPs and residential exits through a generic gateway, you're comparing decoy pages against real ones and no downstream filter can fully save you.
Out-of-stock price resets. Many stores display list price (or a placeholder) when an item goes out of stock. The sequence 89.99 → out of stock → 129.99 → in stock at 89.99 is one stock event, not two price events.
The common thread: most phantom changes are transient. Real competitor moves persist across observations; noise doesn't. That's the insight the whole engine is built on.
Engine design
The pipeline is three stages:
- Snapshot store: append-only observations per SKU, each with a parsed field payload, timestamp, session/IP fingerprint, and content-quality assertions.
- Candidate diff: compare the current observation against a filtered reference state, not the raw previous observation. This is where an exponential moving average (EMA) or rolling median smooths single-sample outliers.
- Hysteresis filter + event emission: a candidate change only becomes an event if it persists across k consecutive observations, or exceeds a magnitude threshold large enough to be trusted immediately.
Two rules of thumb from running this in production: a k=2 persistence requirement kills most A/B noise (experiments often hit your scraper once), and a 5% immediate-emission threshold catches genuine flash repricing without waiting for the second observation. Stock uses different semantics — it's a categorical state machine with allowed transitions (in_stock → out_of_stock and back), not a numeric filter, and stock events should be emitted immediately because downstream availability logic is time-sensitive.
One caveat on data quality: hysteresis assumes your snapshots are comparable. Run content assertions on every scraped page — expected DOM landmarks, sane price ranges, non-empty offer lists — and drop failed assertions before they reach the engine. Comparing a half-rendered page to a good one is how you manufacture events.
Implementation
This is a trimmed but runnable version of the core engine (stdlib only). It keeps per-SKU history, applies an EMA noise filter, enforces hysteresis, tracks stock as a state machine, and emits events with idempotent keys and confidence scores.
from __future__ import annotations
import statistics
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime, timezone
from decimal import Decimal
from enum import Enum
from typing import Optional
class StockState(Enum):
IN_STOCK = "in_stock"
OUT_OF_STOCK = "out_of_stock"
UNKNOWN = "unknown"
@dataclass(frozen=True)
class Observation:
"""One clean snapshot of a SKU. Assumes upstream parsing and
content assertions already passed: price is a normalized Decimal,
currency is consistent, seller_id is the buy-box winner."""
sku: str
ts: datetime
price: Optional[Decimal] # None when out of stock
currency: str
stock: StockState
seller_id: Optional[str] = None
@dataclass(frozen=True)
class Event:
event_type: str # "price_change" | "stock_change" | "stale_feed"
sku: str
ts: datetime
# Idempotency key: consumers can dedupe on this.
event_key: str
old_value: str
new_value: str
magnitude_pct: Optional[float]
confidence: float # 0.0 - 1.0
class ChangeDetectionEngine:
"""Turns an observation stream into a trustworthy event stream.
Tuning used here (production-tested starting points, not gospel):
- EMA alpha 0.4: responsive enough for daily repricing cycles,
suppresses single-sample A/B and decoy spikes.
- Hysteresis k=2: a candidate change must persist 2 consecutive
observations before emission.
- Immediate emission above 5% magnitude: real flash repricing
should not wait for a confirmation sample.
- Median filter over the last 3 raw prices as a second guard
against one-off outliers when history is short.
"""
EMA_ALPHA = 0.4
PERSIST_K = 2
IMMEDIATE_PCT = 5.0
STALE_DAYS = 14.0
def __init__(self) -> None:
# Per-SKU engine state
self._ema: dict[str, float] = {}
self._stock: dict[str, StockState] = {}
self._recent_prices: dict[str, list[float]] = defaultdict(list)
# Candidate changes waiting for hysteresis confirmation.
# sku -> (new_reference, consecutive_count, first_seen_ts)
self._pending: dict[str, tuple[float, int, datetime]] = {}
self._last_real_change: dict[str, datetime] = {}
# ---- price pipeline -------------------------------------------
def _reference_price(self, sku: str) -> Optional[float]:
ema = self._ema.get(sku)
return ema
def _candidate(self, obs: Observation) -> tuple[float, float] | None:
"""Returns (old_ref, new_ref) if a candidate change exists."""
ref = self._reference_price(obs.sku)
if ref is None or obs.price is None:
return None
pct = abs(float(obs.price) - ref) / ref * 100.0
# Noise floor: ignore anything under 0.5% (formatting residue).
if pct < 0.5:
return None
return ref, float(obs.price)
def _process_price(self, obs: Observation) -> list[Event]:
if obs.stock != StockState.IN_STOCK or obs.price is None:
# Out-of-stock pages often show list price: do not update
# the EMA from data we don't trust as a market price.
return []
old_ref = self._reference_price(obs.sku)
# Update EMA first (the new observation is real data), but keep
# a 3-sample median as a cross-check for single-sample spikes.
window = self._recent_prices[obs.sku][-2:] + [float(obs.price)]
med = statistics.median(window)
prev = self._ema.get(obs.sku, float(obs.price))
self._ema[obs.sku] = prev + self.EMA_ALPHA * (float(obs.price) - prev)
self._recent_prices[obs.sku].append(float(obs.price))
self._recent_prices[obs.sku] = self._recent_prices[obs.sku][-10:]
if old_ref is None:
self._last_real_change[obs.sku] = obs.ts
return [] # first observation: baseline only
pct = abs(med - old_ref) / old_ref * 100.0
if pct < 0.5:
# Median disagrees with the EMA move: treat as outlier,
# reset any pending candidate.
self._pending.pop(obs.sku, None)
return []
pending = self._pending.get(obs.sku)
if pending is None:
self._pending[obs.sku] = (med, 1, obs.ts)
return []
new_ref, count, first_seen = pending
# If the candidate flipped direction, restart the count: this
# is classic A/B oscillation, not a sustained move.
if (med - old_ref) * (new_ref - old_ref) < 0:
self._pending[obs.sku] = (med, 1, obs.ts)
return []
count += 1
large_move = pct >= self.IMMEDIATE_PCT
confirmed = count >= self.PERSIST_K or large_move
if not confirmed:
self._pending[obs.sku] = (new_ref, count, first_seen)
return []
self._pending.pop(obs.sku, None)
self._last_real_change[obs.sku] = obs.ts
# Confidence: persistence and magnitude both add trust.
# 0.55 base for k>=2 confirmation, 0.75 for magnitude-only.
conf = 0.75 if large_move and count < self.PERSIST_K else 0.55
conf += min(pct / 25.0, 0.25) # magnitude bonus, capped
return [Event(
event_type="price_change",
sku=obs.sku,
ts=obs.ts,
event_key=f"{obs.sku}:price:{old_ref:.2f}->{new_ref:.2f}:"
f"{int(first_seen.timestamp())}",
old_value=f"{old_ref:.2f}",
new_value=f"{new_ref:.2f}",
magnitude_pct=round(pct, 2),
confidence=round(min(conf, 0.98), 2),
)]
# ---- stock pipeline -------------------------------------------
def _process_stock(self, obs: Observation) -> list[Event]:
prev = self._stock.get(obs.sku, StockState.UNKNOWN)
self._stock[obs.sku] = obs.stock
if prev == obs.stock or prev == StockState.UNKNOWN:
return []
# Stock transitions are emitted immediately: no hysteresis,
# because availability logic is latency-sensitive. Confidence
# starts moderate — decoy pages sometimes fake stock states.
return [Event(
event_type="stock_change",
sku=obs.sku,
ts=obs.ts,
event_key=f"{obs.sku}:stock:{prev.value}->{obs.stock.value}:"
f"{int(obs.ts.timestamp())}",
old_value=prev.value,
new_value=obs.stock.value,
magnitude_pct=None,
confidence=0.7,
)]
# ---- staleness --------------------------------------------------
def _staleness(self, obs: Observation) -> list[Event]:
last = self._last_real_change.get(obs.sku)
if last is None:
return []
age_days = (obs.ts - last).total_seconds() / 86400.0
if age_days > self.STALE_DAYS:
return [Event(
event_type="stale_feed",
sku=obs.sku,
ts=obs.ts,
event_key=f"{obs.sku}:stale:{int(obs.ts.timestamp())}",
old_value=f"{age_days:.1f}d since last change",
new_value="no events",
magnitude_pct=None,
confidence=0.6,
)]
return []
# ---- public API -------------------------------------------------
def process(self, obs: Observation) -> list[Event]:
events = self._process_price(obs) + self._process_stock(obs)
events += self._staleness(obs)
return events
if __name__ == "__main__":
eng = ChangeDetectionEngine()
base = datetime(2025, 9, 1, 10, 0, tzinfo=timezone.utc)
feed = [
# Baseline at 100.00
Observation("SKU-1", base, Decimal("100.00"), "USD", StockState.IN_STOCK, "S-A"),
# A/B blip: 92.00 once, then back — should emit nothing
Observation("SKU-1", base.replace(hour=12), Decimal("92.00"), "USD", StockState.IN_STOCK, "S-A"),
Observation("SKU-1", base.replace(hour=14), Decimal("100.00"), "USD", StockState.IN_STOCK, "S-A"),
# Real drop: 94.50 twice in a row — hysteresis confirms on 2nd
Observation("SKU-1", base.replace(hour=16), Decimal("94.50"), "USD", StockState.IN_STOCK, "S-A"),
Observation("SKU-1", base.replace(hour=18), Decimal("94.50"), "USD", StockState.IN_STOCK, "S-A"),
# Big jump: 15% move emits immediately
Observation("SKU-1", base.replace(hour=20), Decimal("109.00"), "USD", StockState.IN_STOCK, "S-A"),
# Stock flip — immediate event
Observation("SKU-1", base.replace(hour=22), None, "USD", StockState.OUT_OF_STOCK, "S-A"),
]
for i, obs in enumerate(feed):
for ev in eng.process(obs):
print(f"[obs {i}] {ev.event_type:13} {ev.sku} "
f"{ev.old_value} -> {ev.new_value} "
f"mag={ev.magnitude_pct}% conf={ev.confidence}")
Running this prints two price events (the confirmed drop and the immediate 15% jump) and one stock event — while the A/B blip at 92.00 produces nothing, because it fails the persistence requirement and the median filter disagrees with the EMA spike.
Note what the engine deliberately does not do: it doesn't diff seller identity. Seller rotation deserves its own event type with its own key namespace (sku:seller:S-A->S-B), because a buy-box flip at an unchanged price is actionable intelligence for marketplace sellers — the incumbent lost the box, and the new winner's price history matters. Mixing that into price_change events makes your price stream noisier and your seller stream nonexistent. Same feed, two orthogonal event families.
Also worth being explicit about the tradeoff baked into the immediate-emission rule. At a 5% threshold, a real but small competitor move (say 3%) must wait for a second confirming observation before you see it. If your crawl cadence is hourly, that's up to an hour of extra latency on a fifth of real moves. If you crawl every 15 minutes, the cost is negligible. The thresholds and your crawl frequency are not independent decisions — tune them together. Doubling crawl frequency is often cheaper than lowering the noise floor, because the noise floor is what protects your precision.
Event semantics for downstream consumers
Three contracts matter once events leave the engine:
Idempotent keys. Consumers will see duplicates — retries, at-least-once delivery, replays. Every event carries a key like SKU-1:price:100.00->94.50:1756732800 that deterministically identifies the transition, so a repricer can safely upsert on it. Deriving the key from (sku, old, new, first_seen) rather than a UUID is what makes replays harmless.
Ordering. Emit events with the observation timestamp of first detection, not confirmation. A hysteresis-confirmed event detected at 10:00 and confirmed at 14:00 should be dated 10:00, or your downstream analytics will systematically lag competitor moves.
"No change" is data. The staleness check in the engine encodes this: a competitor who reprices daily and then shows zero movement for two weeks probably isn't stable — your scraper is likely being served cached or decoy pages. A stale_feed event routed to your data-quality channel, not your pricing channel, catches silent scraping degradation that monitoring dashboards miss.
Measuring the engine
Tune thresholds against ground truth, not intuition. Take one week of snapshots, hand-label the real competitor moves (yes, this is an afternoon of tedious work — do it once), and compute precision and recall over emitted events:
- Precision: of emitted price events, what fraction were real? Below 95% and your repricing team will start ignoring alerts, which is worse than having no system.
- Recall: of real moves, what fraction did you catch? Hysteresis inherently trades recall for precision — a move you catch one observation late still counts, so score by detection, not by latency.
What good looks like in practice: precision above 95%, recall above 85%, with the missed 15% concentrated in genuinely ambiguous cases rather than systematic blind spots. If recall loss clusters on small moves (1–3%), your noise floor is too aggressive; if it clusters on fast reversals, your k is too high.
Honest limits
Some events are unresolvable at detection time. A flash sale that lasts exactly one observation is indistinguishable from an A/B blip — same shape in the data, opposite meaning. The wrong move is to suppress it; the right move is to emit it flagged low-confidence and let downstream logic decide. A repricer can treat confidence 0.4 events as "watch, don't act"; an analytics pipeline can aggregate them into a flash-sale detector (many SKUs showing the same low-confidence drop in the same hour is a sale, not noise).
The deeper lesson: a change-detection engine doesn't eliminate ambiguity, it prices it. Snapshots tell you what a page said. Deltas, filtered through hysteresis and stamped with confidence, tell you what a competitor did — and that is the only stream worth building a business on.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)