Scraping Amazon Prices at Scale: The Survival Architecture

Amazon is the final boss of scraping targets: no free price API, one of the most sophisticated anti-bot systems deployed, and product pages that vary by marketplace, region, and even time of day. Which is exactly why price monitoring on Amazon is a paid service everywhere you look.

This is the architecture I've run in production for months — the request patterns, the session strategy, and the failure modes, with runnable Python.

Why Naive Scrapers Die in Minutes

Amazon's bot detection operates on accumulated signals: IP reputation (datacenter IPs start guilty), request volume per IP, session coherence, and mechanical timing. The failure progression is predictable:

• Minute 1: everything works. Your scraper feels invincible.
• Hour 1: soft CAPTCHAs on a percentage of responses ([e-mail] pages)
• Day 1: hard 403s from your IP ranges
• Day 3: your cloud provider's entire subnet is flagged, and even a brand-new server inherits the block

The lesson: by the time you notice blocking, the system flagged you long ago. Architecture has to be right from request one.

The Core Pattern: One Product, One Identity

The pattern that survives is simple to state: each product gets its own short-lived sticky session from a residential IP.

for each product:
    1. create a sticky session (one real ISP IP, holds up to 30 min)
    2. brief "arrival" — fetch the storefront homepage first (1 request)
    3. fetch the product page, extract price/availability
    4. release the session; random delay; next product with a NEW session

Each session is a complete "shopping trip": arrives, browses briefly, looks at one product, leaves. No IP accumulates volume, no session lacks browsing shape, and a flagged session costs you exactly one product before being discarded.

The Implementation

import time, random, uuid
import requests
from bs4 import BeautifulSoup

PROXY_HOST = "gate.thordata.com"
PROXY_PORT = 9000
PROXY_USER, PROXY_PASS = "your-username", "your-password"
GEO = "us"  # marketplace you're monitoring — keep IP country aligned

def make_session() -> requests.Session:
    """Fresh session bound to one sticky residential IP."""
    sid = uuid.uuid4().hex[:12]
    user = f"{PROXY_USER}-session-{sid}-country-{GEO}"
    url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
    s = requests.Session()
    s.proxies = {"http": url, "https": url}
    s.headers.update({
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                      "AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/126.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    })
    return s

def scrape_asin(s: requests.Session, asin: str) -> dict | None:
    try:
        s.get(f"https://www.amazon.com/", timeout=30)          # arrival
        time.sleep(random.uniform(0.8, 2.0))
        r = s.get(f"https://www.amazon.com/dp/{asin}", timeout=30)
        if r.status_code != 200 or "api-services-support" in r.text:
            return None                                          # CAPTCHA/blocked
        soup = BeautifulSoup(r.text, "html.parser")
        price = soup.select_one("span.a-price span.a-offscreen")
        if not price:
            return None
        return {
            "asin": asin,
            "price": price.get_text(strip=True),
            "availability": (soup.select_one("#availability") or {}).get(
                "text", "").strip() if soup.select_one("#availability") else None,
        }
    except requests.RequestException:
        return None

def monitor(asins: list[str]) -> list[dict]:
    results = []
    for asin in asins:
        for attempt in range(3):
            data = scrape_asin(make_session(), asin)  # NEW session each retry
            if data:
                results.append(data)
                print(f"OK   {asin}: {data['price']}")
                break
            time.sleep(random.uniform(3, 8))          # backoff, then fresh IP
        time.sleep(random.uniform(1.5, 4.0))          # breathing room
    return results

The details that matter: retries spin up new sessions (retrying a flagged identity compounds the flag); delays are randomized everywhere; and the marketplace (GEO) matches the proxy country so you get the market's actual prices, not geo-redirected ones.

Cost Arithmetic

Residential proxy traffic is billed per GB; Amazon product pages run 300–600KB including images you don't fetch (we only pull HTML). Rules of thumb:

• 1,000 products ≈ 0.3–0.6 GB
• 500 products hourly ≈ ~1 GB/day
• At Thordata's rotating rate ($0.65/GB), a 500-product hourly monitor costs roughly $0.65/day

Against that: one missed repricing window on a competitive listing can cost more before lunch. New accounts can validate the whole pipeline on the free trial traffic; code thor020 takes 10% off paid plans.

Scheduling Discipline

Continuous monitoring is where good scrapapers go bad:

• Poll every 2–4 hours, not minutes — price strategy rarely needs more
• Add startup jitter (random.uniform(0, 900)) so runs never start at :00
• Rotate your watchlist order between runs — identical sequences are patterns
• Track your block rate; if retries exceed ~5%, slow down before Amazon escalates

The Failure Table

Symptom	Cause	Fix
503 + CAPTCHA page	flagged session or IP	fresh sticky session per retry (built in above)
Wrong currency	proxy geo ≠ marketplace	align GEO with marketplace
Works hours, then dies	per-IP accumulation	fresh session per product, lower frequency
Price parses as None	page layout changed or buybox variant	extend selector set, log raw HTML on failure

Beyond Price

The same page carries stock status, buybox seller, review counts, and ratings — parse them in the same pass; they're free once you've paid for the fetch. Store snapshots with timestamps: price history is where repricing strategies actually come from, and (asin, price, ts) in SQLite is enough to start.

The code above is ~80 lines. The architecture — session-per-product, geo-aligned IPs, randomized pacing — is the part that keeps it running for months instead of minutes.

---

Disclosure: this pipeline runs on Thordata's residential proxies (100M+ IPs, sticky sessions, country/city targeting; rotating from $0.65/GB). The architecture works with any provider offering equivalent session control.