DEV Community

Coco
Coco

Posted on

How to scrape Amazon in 2026?

#python #webscraping #tutorial #api

Three Amazon scraping jobs in Python, each driven by a URL: product data, search results, and the star distribution behind a listing. Every request below was executed on 27 July 2026 against Chocodata and the numbers are that run.

HTTP 200 in 3.17s
fields: 60
title: Amazon Kindle Paperwhite 16GB (newest model)
price: 159.99 USD
Enter fullscreen mode Exit fullscreen mode

TL;DR

  • amazon/product takes a product URL and returns 60 top-level fields in about 3 seconds, with no data envelope to unwrap.
  • amazon/search takes a search URL and returned 22 rows, 16 organic and 6 sponsored, for mechanical keyboard.
  • rating_stars_distribution returns 5 bands. Recomputing the mean for the Fire TV Stick gave 4.66 against a listed 4.6.
  • Sponsored rows carry organic_position: None and some rows carry price: None, so sorted() raises TypeError unless you filter first.

Why is it hard to scrape Amazon?

Amazon layers its defences: datacenter IPs are scored before your headers are parsed, product markup is A/B tested so the same URL returns different DOM shapes minutes apart, and sequential access patterns get profiled. The expensive one is the robot check, which arrives as HTTP 200, so any scraper branching on r.status_code == 200 logs a clean run while writing empty rows. That single behaviour causes more wasted debugging than the other three combined.

Prerequisites

  1. A free Chocodata API key, taken from the dashboard after sign-up. Free tier, no card.

Copying the free API key from the dashboard

  1. Python 3.9+ and requests. Tested here on Python 3.13.7 with requests 2.34.2, July 2026. Other languages hit the same endpoints, but every snippet below is Python. 
python -m venv .venv && . .venv/bin/activate
pip install requests
Enter fullscreen mode Exit fullscreen mode
  1. The Amazon URL you want to scrape. Product page or search page, copied from the address bar.

Fetch product data from a product URL

To scrape Amazon product data, send the product URL as url and read the fields straight off the response. There is no wrapper object.

1. Send the URL and assert on a field, not the status 

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
URL = "https://www.amazon.com/dp/B0CFPJYX7P"

r = requests.get(f"{BASE}/amazon/product",
                 params={"url": URL, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
product = r.json()

assert product.get("asin"), "empty payload behind a 200"
print("fields:", len(product))
print("title:", product["title"][:44])
print("price:", product["price"], product["currency"])
Enter fullscreen mode Exit fullscreen mode 
fields: 60
title: Amazon Kindle Paperwhite 16GB (newest model)
price: 159.99 USD
Enter fullscreen mode Exit fullscreen mode

Terminal showing 60 fields and the parsed product title and price

2. Map the fields you actually store 

KEEP = ("asin", "parent_asin", "brand", "price", "price_buybox", "currency",
        "rating", "reviews_count", "max_quantity", "has_videos")

for key in KEEP:
    print(f"{key:>16}: {product[key]}")
Enter fullscreen mode Exit fullscreen mode 
            asin: B0CFPJYX7P
     parent_asin: B0DFYWDGW6
           brand: Amazon
           price: 159.99
    price_buybox: 159.99
        currency: USD
          rating: 4.7
   reviews_count: 21493
    max_quantity: 5
      has_videos: True
Enter fullscreen mode Exit fullscreen mode

The ten stored product fields printed as a key-value table

3. Flatten the variation list to CSV 

import csv

with open("variations.csv", "w", newline="", encoding="utf-8") as fh:
    w = csv.writer(fh)
    w.writerow(["asin", "color", "style", "configuration"])
    for v in product["variations"]:
        d = v["dimensions"]
        w.writerow([v["asin"], d.get("Color"), d.get("Style"), d.get("Configuration")])

print("rows:", len(product["variations"]))
Enter fullscreen mode Exit fullscreen mode 
asin,color,style,configuration
B0CTMS23DZ,Jade,With 3 months of Kindle Unlimited,Without Lockscreen Ads
B0CTMR865D,Raspberry,With 3 months of Kindle Unlimited,Ad-supported
B0CFPTK5JG,Raspberry,Without Kindle Unlimited,Ad-supported
B0CFPJYX7P,Black,Without Kindle Unlimited,Ad-supported
B0DDZJS3SB,Black,Without Kindle Unlimited,Without Lockscreen Ads
rows: 12
Enter fullscreen mode Exit fullscreen mode

dimensions keys vary per product line, which is why .get() beats [] here. The search endpoint returns a flatter shape and is covered next.

The variations CSV written from the product response

Fetch search results from a search URL

To scrape Amazon search results, send the search page URL unchanged. The keyword stays inside the URL, so nothing needs parsing out of it.

1. Request the search URL unchanged 

SEARCH = "https://www.amazon.com/s?k=mechanical+keyboard"

r = requests.get(f"{BASE}/amazon/search",
                 params={"url": SEARCH, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
search = r.json()

print("products:", len(search["products"]))
print("keys:", sorted(search["products"][0])[:6])
Enter fullscreen mode Exit fullscreen mode 
products: 22
keys: ['asin', 'badges', 'best_seller', 'brand', 'climate_pledge_friendly', 'currency']
Enter fullscreen mode Exit fullscreen mode

Product count and row keys from the search response

2. Split sponsored rows from organic rows 

organic = [p for p in search["products"] if not p["is_sponsored"]]
print(len(organic), "organic of", len(search["products"]))

for p in organic[:5]:
    print(f'{p["organic_position"]:>2} {p["asin"]} {str(p["price"]):>7} {p["rating"]} {p["title"][:42]}')
Enter fullscreen mode Exit fullscreen mode 
16 organic of 22
 1 B0CF3VGQFL   29.99 4.3 Redragon Mechanical Gaming Keyboard Wired,
 2 B0CDWP1D58   36.99 4.5 Redragon K668 108-Key Hot-Swap Wired RGB G
 3 B09LK1P1RD   156.9 4.2 Logitech MX Mechanical Wireless Illuminate
 4 B09JG7KRC7   43.31 4.4 Keychron C2 Full Size Wired Mechanical Key
 5 B0C8QYB8W6   89.99 4.5 Razer BlackWidow V4 X Mechanical Gaming Ke
Enter fullscreen mode Exit fullscreen mode

Organic rows separated from sponsored rows with positions and prices

3. Page the URL and dedupe by ASIN 

def fetch_page(keyword, page):
    url = f"https://www.amazon.com/s?k={keyword}&page={page}"
    r = requests.get(f"{BASE}/amazon/search",
                     params={"url": url, "api_key": API_KEY}, timeout=90)
    r.raise_for_status()
    return r.json()["products"]

rows = {}
for page in (1, 2):
    got = fetch_page("mechanical+keyboard", page)
    print(f"page {page}: {len(got)} rows")
    for p in got:
        rows[p["asin"]] = p

print("unique asins:", len(rows))
Enter fullscreen mode Exit fullscreen mode 
page 1: 22 rows
page 2: 22 rows
unique asins: 37
Enter fullscreen mode Exit fullscreen mode

Pages overlapped by 7 ASINs on that run, so keying the dict on asin matters more than it looks. Rating data needs no extra call, which the next section uses.

Two search pages merged and deduplicated by ASIN

Fetch ratings and the star distribution

To scrape Amazon reviews data, read the rating fields off the same product response. One call per listing covers the score, the volume and the five bands.

1. Read rating and reviews_count 

FIRE = "https://www.amazon.com/dp/B0BP9MDCQZ"

r = requests.get(f"{BASE}/amazon/product",
                 params={"url": FIRE, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
item = r.json()

print("rating:", item["rating"])
print("reviews_count:", item["reviews_count"])
Enter fullscreen mode Exit fullscreen mode 
rating: 4.6
reviews_count: 111766
Enter fullscreen mode Exit fullscreen mode

Rating and review count for the Fire TV Stick listing

2. Expand rating_stars_distribution 

for b in item["rating_stars_distribution"]:
    bar = "#" * (b["percentage"] // 2)
    print(f'{b["rating"]} star {b["percentage"]:>3}% {bar}')
Enter fullscreen mode Exit fullscreen mode 
5 star  83% #########################################
4 star   9% ####
3 star   3% #
2 star   1%
1 star   4% ##
Enter fullscreen mode Exit fullscreen mode

The five star bands rendered as a terminal bar chart

3. Convert percentages to counts and recheck the mean 

total = item["reviews_count"]
counts = {b["rating"]: round(total * b["percentage"] / 100)
          for b in item["rating_stars_distribution"]}
mean = sum(k * v for k, v in counts.items()) / sum(counts.values())

for k in sorted(counts, reverse=True):
    print(f"{k} star: {counts[k]:>7,}")
print(f"weighted mean: {mean:.2f} (listed {item['rating']})")
print(f"negative share: {(counts[1] + counts[2]) / sum(counts.values()) * 100:.1f}%")
Enter fullscreen mode Exit fullscreen mode 
5 star:  92,766
4 star:  10,059
3 star:   3,353
2 star:   1,118
1 star:   4,471
weighted mean: 4.66 (listed 4.6)
negative share: 5.0%
Enter fullscreen mode Exit fullscreen mode

The recomputed mean lands 0.06 above the listed rating because the band percentages are rounded to whole numbers, so treat it as a cross-check rather than a replacement. The reviews list on the same response carries the eight most recent reviewers with author, rating, timestamp, is_verified and helpful_count, so reviewer-level tracking works without the written text.

Star bands converted to absolute counts with a recomputed mean

The part that breaks

Nulls in the search rows, not the requests. On the 22-row mechanical keyboard response, 6 rows had organic_position: None because they were sponsored, 1 row had price: None, 1 had rating: None and 4 had sales_volume: None. Sorting straight off those keys raises:

Traceback (most recent call last):
  File "scrape.py", line 32, in <module>
    cheapest = sorted(prods, key=lambda p: p["price"])
TypeError: '<' not supported between instances of 'NoneType' and 'float'

Traceback (most recent call last):
  File "scrape.py", line 38, in <module>
    ranked = sorted(prods, key=lambda p: p["organic_position"])
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
Enter fullscreen mode Exit fullscreen mode

Filter before you sort, and use a sentinel for the rank so sponsored rows sort last instead of blowing up:

prods = search["products"]

priced = [p for p in prods if p["price"] is not None]
cheapest = sorted(priced, key=lambda p: p["price"])
ranked = sorted(prods, key=lambda p: p["organic_position"] or 999)
Enter fullscreen mode Exit fullscreen mode

Two more things worth knowing before you build on this. Search row counts move between runs, 16 on one wireless mouse call and 22 on the keyboard call, so never hard-code a slice length. And the entries in reviews come back with content and title unset, so any pipeline expecting review sentences needs to run on rating_stars_distribution instead.

Full script 

"""Amazon product, search and rating scraper. Python 3.13.7, requests 2.34.2."""
import csv
import sys
import time
import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(path, url, timeout=90, tries=3):
    """One GET, retried, so a batch run survives a dropped response."""
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}",
                         params={"url": url, "api_key": API_KEY}, timeout=timeout)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


def product(url):
    p = call("amazon/product", url)
    if not p.get("asin"):
        raise RuntimeError(f"empty payload for {url}")
    return p


def star_counts(p):
    total = p["reviews_count"]
    return {b["rating"]: round(total * b["percentage"] / 100)
            for b in p.get("rating_stars_distribution") or []}


def search(keyword, pages=2):
    rows = {}
    for page in range(1, pages + 1):
        url = f"https://www.amazon.com/s?k={keyword}&page={page}"
        for row in call("amazon/search", url)["products"]:
            rows[row["asin"]] = row
    return list(rows.values())


def main():
    p = product("https://www.amazon.com/dp/B0CFPJYX7P")
    print(f'{p["title"][:44]} | {p["price"]} {p["currency"]} | {p["rating"]}')

    for star, n in sorted(star_counts(p).items(), reverse=True):
        print(f"  {star} star: {n:>7,}")

    rows = search("mechanical+keyboard", pages=2)
    priced = [r for r in rows if r["price"] is not None]
    priced.sort(key=lambda r: r["price"])
    print(f"{len(rows)} unique asins, {len(priced)} priced")

    with open("amazon.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["asin", "title", "price", "rating", "reviews_count", "sponsored"])
        for r in priced:
            w.writerow([r["asin"], r["title"], r["price"], r["rating"],
                        r["reviews_count"], r["is_sponsored"]])
    print("wrote amazon.csv")


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Summary

Three Amazon jobs run off one request shape: a product URL returns 60 top-level fields including the rating and the five star bands, and a search URL returns product rows with sponsored flags and organic positions. What the payload does not carry is review body text, so this covers price tracking, rank tracking and score monitoring rather than sentiment reading. The habit that saves the most time is asserting on asin rather than on the status code, and filtering None out of every key you intend to sort on.

FAQ

How many fields does the Amazon product response return?

Sixty top-level fields on both products tested, covering price, rating, images, variations and the star distribution.

Why does sorting search rows raise TypeError?

Sponsored rows carry organic_position: None and a few listings carry price: None, so filter the nulls out or supply a sentinel before sorting.

Does the star distribution match the listed rating?

Close but not exact: recomputing the mean from the Fire TV Stick bands gave 4.66 against a listed 4.6, because the band percentages are rounded to whole numbers.

Top comments (0)