One SKU, One Identity: Why Session Affinity Beats Rotation for Repeated Price Checks
If you've read the earlier posts in this series, you know the usual advice: rotate IPs to avoid detection. That advice is correct for discovery crawls and wrong for price monitoring. This post is about the specific failure mode that kills scheduled price monitors: rotating the exit IP on every request while re-visiting the same URL on a fixed schedule. The fix is the opposite strategy — session affinity. One SKU, one sticky residential session, held for hours or days.
The anomaly argument
Think about what a price monitor looks like from the retailer's side. You have a product page — one URL — that receives a request every 10 minutes, 144 times a day, forever. Now look at the source IPs. If each request comes from a different address, you have a pattern that no human population produces: the same browser fingerprint or request cadence, hitting one listing, from 144 different households per day. No real visitor re-opens the same toaster listing from a different house every 10 minutes.
Per-request rotation is a strategy for breadth. When you crawl a million different URLs, a fresh IP per request is fine because each URL sees you exactly once — there's no per-URL pattern to detect. The anti-bot systems that matter for discovery crawls are rate limiters and IP reputation lists, and rotation defeats both by spreading load.
A price monitor inverts the geometry. You have few URLs, visited many times each. The signal an anti-bot system extracts isn't "this IP makes too many requests" — it's "this URL receives requests from a rotating cast of IPs with identical timing and identical headers." That's a signature. Rotation doesn't hide it; rotation is it. The most common reason a price monitor gets flagged isn't a bad IP pool. It's a good IP pool applied to the wrong problem.
There's a subtlety worth naming: the anti-bot backend doesn't need to see the IPs directly to detect this. Fingerprinting JS collects canvas hashes, font lists, TLS fingerprints, and header order. When 144 distinct IPs present the same fingerprint tuple to one URL on a metronomic schedule, the correlation is trivial — most commercial bot-detection products do exactly this join as their first pass. Rotation doesn't just fail to help here; it actively strengthens the correlation, because the only constant across all 144 requests is the thing you can't easily change: your client's fingerprint and cadence. Affinity works because it changes the shape of the pattern instead of the addresses — one identity, one cadence, gaps at night, jittered intervals. That's what a real price-watcher looks like.
The affinity model
The model is simple: sku_id → sticky_session_id → residential exit IP.
When you start monitoring a SKU, you mint a session ID and encode it in the proxy username (most residential gateways support something like user-sess-abc123:pass@gateway.example.com:8080). The gateway pins that session to one exit node for the session's lifetime — typically 30 to 120 minutes depending on provider and configuration, some longer. Every check of that SKU goes through the same session, so the retailer sees one visitor who keeps coming back to a product they're interested in. That's normal behavior. People watch prices on things they want to buy.
When the session expires (the gateway silently assigns a new exit IP, or starts returning errors), you retire it and mint a fresh one. Two details matter here:
- Persistence. The SKU→session mapping must survive process restarts. Store it in SQLite, not memory. If your monitor restarts every hour and forgets its sessions, you've reinvented rotation.
- Cooldown. When a session dies, don't immediately fire a new one at the same URL. A retirement followed within seconds by a brand-new IP requesting the same page is a smaller but real signature. Wait 60–180 seconds — long enough to look like a human who reopened their browser, short enough that a 10-minute polling cadence barely notices.
Retirement policy: retire after T hours of age or K consecutive failures, whichever comes first. I use T = 6 hours and K = 3 in practice. Retiring on age alone wastes healthy sessions; retiring on failures alone lets a half-dead session pollute your data with errors for hours. Both thresholds together give you a predictable identity churn rate per SKU — roughly 4 identities per day at T = 6h — which is well inside the range of "different devices in one household" plausibility.
Implementation
Here's a working SkuSessionManager. It uses only the standard library plus requests, keeps the mapping in SQLite, tracks session health, and builds per-SKU proxy URLs with the session ID embedded in the username.
import sqlite3
import time
import uuid
import random
import requests
GATEWAY_HOST = "gateway.example.com"
GATEWAY_PORT = 8080
PROXY_USER = "myaccount"
PROXY_PASS = "mypassword"
SESSION_MAX_AGE_S = 6 * 3600 # retire after 6 hours regardless of health
MAX_CONSEC_FAILURES = 3 # retire after 3 consecutive fetch failures
RETIREMENT_COOLDOWN_S = 120 # wait 2 min before minting a replacement
JITTER_FRACTION = 0.20 # +/- 20% on each polling interval
class SkuSessionManager:
"""Binds each SKU to one sticky proxy session, persisted in SQLite."""
def __init__(self, db_path="sku_sessions.db"):
self.db = sqlite3.connect(db_path)
self.db.execute("""
CREATE TABLE IF NOT EXISTS sku_session (
sku_id TEXT PRIMARY KEY,
session_id TEXT NOT NULL,
created_at REAL NOT NULL,
consec_fails INTEGER NOT NULL DEFAULT 0,
last_used_at REAL,
retired_at REAL
)
""")
self.db.commit()
def _mint_session_id(self) -> str:
# Session ID goes into the proxy username; the gateway pins
# requests carrying the same ID to one residential exit IP.
return uuid.uuid4().hex[:12]
def get_proxy(self, sku_id: str) -> dict:
"""Return a requests-style proxy dict for this SKU,
minting or replacing the session if needed."""
row = self.db.execute(
"SELECT session_id, created_at, retired_at FROM sku_session WHERE sku_id=?",
(sku_id,),
).fetchone()
need_new = row is None
if row is not None:
session_id, created_at, retired_at = row
expired = time.time() - created_at > SESSION_MAX_AGE_S
if retired_at is not None:
# Previously retired: honor the cooldown before replacing.
if time.time() - retired_at < RETIREMENT_COOLDOWN_S:
raise RuntimeError(f"SKU {sku_id} in retirement cooldown")
need_new = True
elif expired:
self._retire(sku_id, reason="age")
need_new = True
if need_new:
session_id = self._mint_session_id()
self.db.execute(
"""INSERT INTO sku_session
(sku_id, session_id, created_at, consec_fails, last_used_at, retired_at)
VALUES (?, ?, ?, 0, NULL, NULL)
ON CONFLICT(sku_id) DO UPDATE SET
session_id=excluded.session_id,
created_at=excluded.created_at,
consec_fails=0,
retired_at=NULL""",
(sku_id, session_id, time.time()),
)
self.db.commit()
proxy_url = (
f"http://{PROXY_USER}-sess-{session_id}:{PROXY_PASS}"
f"@{GATEWAY_HOST}:{GATEWAY_PORT}"
)
return {"http": proxy_url, "https": proxy_url}
def report_result(self, sku_id: str, ok: bool):
"""Record fetch outcome; retire the session on repeated failure."""
if ok:
self.db.execute(
"UPDATE sku_session SET consec_fails=0, last_used_at=? WHERE sku_id=?",
(time.time(), sku_id),
)
else:
self.db.execute(
"UPDATE sku_session SET consec_fails=consec_fails+1 WHERE sku_id=?",
(sku_id,),
)
fails = self.db.execute(
"SELECT consec_fails FROM sku_session WHERE sku_id=?", (sku_id,)
).fetchone()[0]
if fails >= MAX_CONSEC_FAILURES:
self._retire(sku_id, reason="failures")
self.db.commit()
def _retire(self, sku_id: str, reason: str):
self.db.execute(
"UPDATE sku_session SET retired_at=? WHERE sku_id=?",
(time.time(), sku_id),
)
self.db.commit()
print(f"[retire] {sku_id}: {reason}")
def fetch_price(sku_url: str, proxy: dict) -> float:
r = requests.get(sku_url, proxies=proxy, timeout=30,
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"})
r.raise_for_status()
# Parse the price out of the page with your usual extractor.
# Placeholder: assumes a data-price attribute in the HTML.
import re
m = re.search(r'data-price="([\d.]+)"', r.text)
if not m:
raise ValueError("price not found")
return float(m.group(1))
def monitor_loop(skus: dict, base_interval_s: int = 600):
"""skus: {sku_id: product_url}. Polls each SKU on its own jittered clock."""
mgr = SkuSessionManager()
next_run = {sku_id: time.time() for sku_id in skus}
while True:
now = time.time()
due = [s for s in skus if next_run[s] <= now]
for sku_id in due:
try:
proxy = mgr.get_proxy(sku_id)
except RuntimeError:
continue # in retirement cooldown; try next cycle
try:
price = fetch_price(skus[sku_id], proxy)
mgr.report_result(sku_id, ok=True)
print(f"{sku_id}: {price}")
except Exception as exc:
mgr.report_result(sku_id, ok=False)
print(f"{sku_id}: failed ({exc})")
# Jitter the next run by +/- 20% so sessions never align on :00.
jitter = base_interval_s * random.uniform(-JITTER_FRACTION, JITTER_FRACTION)
next_run[sku_id] = now + base_interval_s + jitter
time.sleep(5)
if __name__ == "__main__":
monitor_loop({
"SKU-1001": "https://shop.example.com/p/toaster-xyz",
"SKU-1002": "https://shop.example.com/p/coffee-grinder-abc",
})
Note what report_result does not do: it never shares sessions between SKUs, and it never retries inside the fetch. A failed check counts as one failure; the next scheduled check is the retry. Retrying immediately with the same session after a block just burns the session faster.
Sizing the identity pool
The obvious worry: "I monitor 5,000 SKUs. Do I need 5,000 residential IPs?" No. Sessions are minted lazily — a SKU gets a session the first time it's polled, and only currently active sessions hold an exit IP.
Run the arithmetic. With a 10-minute polling interval and 5,000 SKUs, you issue about 8.3 requests per second, but concurrency is what determines simultaneous sessions, not request rate. Each request holds its session for a few seconds (time to fetch the page). If a fetch takes 4 seconds, then at any instant roughly 8.3 req/s × 4 s ≈ 33 sessions are actively mid-request. In practice, gateways keep a session mapped to an exit node for its full TTL whether or not requests are in flight — so the real constraint is distinct sessions alive at once, which equals the number of SKUs with live sessions, i.e. up to 5,000 at steady state.
That's the tradeoff: affinity trades pool size for plausibility. If 5,000 concurrent sticky sessions exceeds your budget or your provider's limits, lengthen the polling interval or shard SKUs by priority — check your top 500 every 10 minutes and the long tail every 2 hours. A 2-hour interval across 4,500 tail SKUs with 6-hour session TTLs still gives each SKU a stable identity across 3 checks before rotation, which preserves the behavioral story at a quarter of the concurrent-session cost.
Failure modes
Shared-session poisoning. The temptation is to assign one session to a group of SKUs from the same retailer to save IPs. Don't, or shard very carefully. If one session covers 200 products on one retailer, then a single flag or CAPTCHA on that session corrupts 200 data streams at once, and the "visitor" browsing 200 unrelated products every 10 minutes starts looking like a crawler anyway. One session per SKU per retailer. If you monitor the same SKU on three retailers, use three sessions — a real person doesn't visit competitors from simultaneously rotating houses either.
Clock alignment. If every SKU's schedule starts at process launch, all sessions fire at :00, :10, :20. Anti-bot systems look for exactly this kind of periodicity at the URL level. The jitter in the loop above (±20%, applied independently per SKU) is not optional polish; it's load-bearing. Reset jitter on every cycle, not just the first.
Cross-retailer session reuse. A session ID that appears on retailer A's logs should never appear on retailer B's. Exit IPs are a shared observable — data brokers and fraud vendors correlate them. Mint sessions per (SKU, retailer) pair and treat the pair as the identity unit.
When affinity is wrong
Affinity is the wrong tool for breadth. Discovery crawls, SERP scraping, one-shot enrichment of a product catalog — these see each target once, so a fresh IP per request is both cheaper and safer. Rotating sessions also matter when a target is already hostile: if a retailer has blocked your monitor's pattern, more affinity just deepens the hole; fall back to rotation, slow down, or reconsider the target.
The rule of thumb from this whole series, distilled: rotation for depth-less breadth, affinity for repeated depth. Price monitoring is repeated depth by definition. Give every product you watch one stable identity, and you stop looking like a bot network and start looking like 5,000 interested shoppers.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)