DEV Community

rhea hollis
rhea hollis

Posted on

An Honest Price Monitor: Session Config, ASN Pinning, and Decoy Detection

#python #webscraping #datascience #tutorial

A price monitor that fails loudly is fine — you fix it. The dangerous one returns 200s and plausible numbers while the dataset rots. This post is the code for the three fixes that matter: session config, ASN pinning, and decoy detection.

1. Size the session to the crawl

The classic mistake is a default timeout that expires mid-catalog. The exit rotates, half the store re-downloads, and the two halves of the catalog are seen from different locations — the prices stop being comparable.

SESSION_MINUTES = max(category_page_count * avg_seconds_per_page / 60 * 1.5, 30)

session = provider.create_session(
    sticky_minutes=SESSION_MINUTES,   # > longest category, plus margin
    country=store.country,
    asn=store.target_asn,             # pin the exit to one network
)
Enter fullscreen mode Exit fullscreen mode

Sticky sessions of 30–90 minutes with country and ASN targeting are standard on residential providers — Thordata exposes all three at session creation (free trial here: https://www.thordata.com/?ls=dev&lk=dev-1) — so this is configuration, not engineering.

2. Log the exit country as a first-class column

Mixed-location data reads as spread but is mostly geo-pricing noise. The fix costs one column:

@dataclass
class PriceRow:
    sku: str
    price: float
    currency: str
    exit_country: str   # from the session, at capture time
    exit_asn: str
    captured_at: datetime
Enter fullscreen mode Exit fullscreen mode

Comparisons then happen GROUP BY exit_country — anything else compares apples to a different country's apples.

3. Variance-check before storage

Decoy pages arrive with a 200 status and plausible digits. Structural parsing won't catch them; statistics will:

def is_plausible(row: PriceRow, history: pd.Series, threshold: float = 0.30) -> bool:
    median = history.tail(7 * 24).median()  # 7-day rolling median
    if median <= 0:
        return True  # not enough history yet
    return abs(row.price - median) / median <= threshold

# plausible -> store; implausible -> flag for review, never store silently
Enter fullscreen mode Exit fullscreen mode

A 30% threshold against a 7-day median catches currency swaps and decoy prices while letting genuine flash sales through to review.

The full loop 

for store in stores:
    with provider.session(country=store.country, asn=store.target_asn,
                          sticky_minutes=SESSION_MINUTES) as s:
        for page in store.catalog:
            row = parse_price(s.get(page.url))
            if is_plausible(row, history[row.sku]):
                store_row(row)
            else:
                flag_for_review(row)
Enter fullscreen mode Exit fullscreen mode

Segment the schedule too: hourly snapshots for flash-deal SKUs, daily for the stable catalog — hourly-everything burns ~20x the bandwidth for marginal signal.

The monitor that assumes its data is lying until proven otherwise is the one that survives.

Top comments (0)