DEV Community

Viktor
Viktor

Posted on

Amazon Scraper in Python: An Honest Developer Guide (2026)

#python #webscraping #amazon #tutorial

Amazon is probably the most-scraped e-commerce site there is, and it's also one of the best defended. People scrape it to track prices, monitor competitors, research product niches, or check what's actually ranking for a keyword.

This guide covers what works in 2026, what breaks, and why. We'll build a small Python scraper that:

  1. Searches Amazon for a keyword
  2. Pulls out ASIN, title, price, rating, and review count
  3. Pages through the results
  4. Stores snapshots in SQLite so you can track prices over time

I'll also be upfront about where the DIY approach stops working.

Legal note: Only scrape publicly visible data, keep your request rate low, and read Amazon's Conditions of Use. If you need product data at scale for a commercial product, look at the official Product Advertising API first (it requires an Associates account with qualifying sales).

How Amazon search pages are structured

A search URL looks like this:

https://www.amazon.com/s?k=wireless+earbuds&page=2
Enter fullscreen mode Exit fullscreen mode
  • k is the keyword
  • page is the 1-based page number (Amazon usually caps search at around 7–20 pages)

Every organic result is a div with data-component-type="s-search-result" and a data-asin attribute. The ASIN (Amazon Standard Identification Number) is the stable ID you want: https://www.amazon.com/dp/<ASIN> always resolves to the product.

Inside each result card:

Field Selector (2026, subject to change)
Title h2 span
Price span.a-price > span.a-offscreen (first one is the current price)
List price span.a-price.a-text-price > span.a-offscreen
Rating span.a-icon-alt (text like "4.4 out of 5 stars")
Review count span.s-underline-text or a[href*="customerReviews"] span

Amazon changes class names regularly and A/B tests layouts, so write your parser defensively: every field may be missing.

Setup 

pip install requests beautifulsoup4 lxml
Enter fullscreen mode Exit fullscreen mode

Step 1: Fetch a search page without getting blocked immediately

The bare requests.get() with the default python-requests/2.x user agent usually gets a 503 or a CAPTCHA page on the first try. At minimum, send realistic browser headers and reuse a session so cookies persist:

import random
import time
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.6 Safari/605.1.15",
]

session = requests.Session()

def headers():
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }

class Blocked(Exception):
    pass

def fetch(url, params=None, retries=3):
    for attempt in range(retries):
        resp = session.get(url, params=params, headers=headers(), timeout=20)
        blocked = (
            resp.status_code in (503, 429)
            or "validateCaptcha" in resp.text
            or "Robot Check" in resp.text
        )
        if not blocked and resp.ok:
            return resp.text
        wait = 2 ** attempt * 5 + random.uniform(0, 3)
        print(f"Blocked or error ({resp.status_code}), sleeping {wait:.0f}s")
        time.sleep(wait)
    raise Blocked(f"Gave up on {resp.url}")
Enter fullscreen mode Exit fullscreen mode

Note that a CAPTCHA page often comes back with HTTP 200, so checking only the status code isn't enough. Look for validateCaptcha in the body.

Step 2: Parse the results 

import re
from bs4 import BeautifulSoup

def to_float(text):
    if not text:
        return None
    m = re.search(r"[\d,]+(?:\.\d+)?", text)
    return float(m.group().replace(",", "")) if m else None

def text_of(el):
    return el.get_text(strip=True) if el else None

def parse_search(html):
    soup = BeautifulSoup(html, "lxml")
    products = []
    for card in soup.select('div[data-component-type="s-search-result"]'):
        asin = card.get("data-asin")
        if not asin:
            continue
        # Skip sponsored placements if you only want organic results
        sponsored = card.select_one('span.puis-label-popover-default, span:-soup-contains("Sponsored")')

        rating_text = text_of(card.select_one("span.a-icon-alt"))  # "4.4 out of 5 stars"
        products.append({
            "asin": asin,
            "title": text_of(card.select_one("h2 span")),
            "price": to_float(text_of(card.select_one("span.a-price > span.a-offscreen"))),
            "list_price": to_float(text_of(card.select_one("span.a-price.a-text-price > span.a-offscreen"))),
            "rating": to_float(rating_text.split(" ")[0]) if rating_text else None,
            "reviews": to_float(text_of(card.select_one("span.s-underline-text"))),
            "sponsored": sponsored is not None,
            "url": f"https://www.amazon.com/dp/{asin}",
        })
    return products
Enter fullscreen mode Exit fullscreen mode

A few gotchas I hit:

  • Prices are locale-formatted. amazon.de gives you 12,99 €. If you scrape other marketplaces, parse per locale instead of stripping commas.
  • Review counts can be abbreviated ("1.2K") in some layouts. Handle that if you see it.
  • Sponsored results are mixed in. Decide up front whether you want them; for rank tracking you usually don't.
  • Duplicate ASINs show up across pages. Dedupe by ASIN.

Step 3: Paginate politely 

SEARCH_URL = "https://www.amazon.com/s"

def search(keyword, max_pages=5):
    seen, results = set(), []
    for page in range(1, max_pages + 1):
        html = fetch(SEARCH_URL, params={"k": keyword, "page": page})
        items = parse_search(html)
        if not items:
            break  # end of results, or a layout we don't recognize
        for item in items:
            if item["asin"] not in seen:
                seen.add(item["asin"])
                results.append(item)
        print(f"page {page}: {len(items)} items")
        time.sleep(random.uniform(3, 7))  # don't hammer it
    return results
Enter fullscreen mode Exit fullscreen mode

Random delays of a few seconds matter more than any header trick. Burst traffic from one IP is the fastest way to get a CAPTCHA wall.

Step 4: Store snapshots for price tracking

The useful thing is usually the history, not a single scrape. SQLite is all you need:

import sqlite3
from datetime import datetime, timezone

def save(products, keyword, db="amazon.db"):
    con = sqlite3.connect(db)
    con.execute("""
        CREATE TABLE IF NOT EXISTS snapshots (
            asin TEXT, keyword TEXT, title TEXT, price REAL, list_price REAL,
            rating REAL, reviews REAL, sponsored INTEGER, position INTEGER,
            scraped_at TEXT
        )""")
    now = datetime.now(timezone.utc).isoformat()
    con.executemany(
        "INSERT INTO snapshots VALUES (?,?,?,?,?,?,?,?,?,?)",
        [(p["asin"], keyword, p["title"], p["price"], p["list_price"], p["rating"],
          p["reviews"], int(p["sponsored"]), i, now)
         for i, p in enumerate(products, 1)],
    )
    con.commit()
    con.close()

if __name__ == "__main__":
    kw = "wireless earbuds"
    data = search(kw, max_pages=3)
    save(data, kw)
    print(f"saved {len(data)} products")
Enter fullscreen mode Exit fullscreen mode

Run it daily with cron and you have price and rank history:

SELECT asin, date(scraped_at) AS day, MIN(price) AS price, MIN(position) AS rank
FROM snapshots
WHERE keyword = 'wireless earbuds'
GROUP BY asin, day
ORDER BY asin, day;
Enter fullscreen mode Exit fullscreen mode

Where DIY falls apart

Here's the honest part. The script above works for a few pages a day from a home connection. Past that, you'll hit:

  1. CAPTCHAs at volume. From datacenter IPs (AWS, GCP, a VPS), you'll often get blocked on the first request. You need residential proxies, which cost real money.
  2. TLS / browser fingerprinting. Amazon looks at more than headers. Libraries like curl_cffi (which impersonates Chrome's TLS fingerprint) help a lot over plain requests.
  3. Layout drift. Selectors break every few weeks. Budget time for maintenance and add an alert when a page returns zero items.
  4. Location-dependent prices. Prices and availability depend on the delivery ZIP code (stored in a cookie). Your numbers may not match what your users see.
  5. Headless browsers aren't a silver bullet. Playwright gets you past some checks but is 10–50x slower and still gets fingerprinted.

Option: use an API for the hard part

If you'd rather not maintain proxies and selectors, a scraping API can return the same data as JSON. For example, ScrapeAtlas* has an Amazon product search endpoint that takes the keyword, marketplace, and a pagination cursor:

import os
import requests

API = "https://api.scrapeatlas.com/v1/amazon/product-search"

def search_api(keyword, country="US", max_pages=3):
    results, cursor = [], None
    for _ in range(max_pages):
        params = {"query": keyword, "country": country}
        if cursor:
            params["cursor"] = cursor
        resp = requests.get(
            API,
            headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
            params=params,
            timeout=60,
        )
        data = resp.json()
        if data["status"] in ("failed", "challenged"):
            raise RuntimeError(f"request failed: {data['requestId']}")
        for item in data["items"]:
            p = item["product"]
            results.append({
                "asin": p["id"],
                "title": p["title"],
                "price": (p.get("price") or {}).get("current"),
                "list_price": (p.get("price") or {}).get("original"),
                "rating": (p.get("rating") or {}).get("average"),
                "reviews": (p.get("rating") or {}).get("count"),
                "sponsored": False,
                "url": p["url"],
            })
        if not data["pagination"]["has_more"]:
            break
        cursor = data["pagination"]["next_cursor"]
    return results
Enter fullscreen mode Exit fullscreen mode

The output fits the same save() function, so you can swap one fetcher for the other. Check status: partial means you got usable data but some fields weren't available. There are also endpoints for single products by ASIN, reviews, offers/sellers, and best-seller lists, and country covers marketplaces like GB, DE, and JP.

Wrapping up

  • For small, occasional scrapes: requests + BeautifulSoup + slow, random delays works fine.
  • For daily tracking of a few keywords: add curl_cffi, retry logic, and a zero-results alert.
  • For anything at scale or from cloud servers: you'll need residential proxies and ongoing maintenance, or an API that handles them for you.

Whichever route you take, store snapshots rather than overwriting rows. The price history is where the value is.

Questions or a selector that broke? Drop a comment and I'll update the guide.

*I build it.

Top comments (0)