Send a plain requests.get() to an Amazon product page from a cloud server and, within a few calls, you get a CAPTCHA page or a 503 instead of a price. Amazon sits behind AWS WAF, which scores every request on its IP, headers and recent activity. Datacenter IPs and default Python headers score badly.
This walkthrough builds three scrapers that work today: product data (name, price, stock, rating, image), every variation of a product (all color, size and carrier combinations with their own prices), and every seller offer for an ASIN. It uses requests, BeautifulSoup and csv, and routes the requests through Scrape.do so the WAF part is handled for you.
1. Setup and one fetch helper
pip install requests beautifulsoup4
Every page goes through Scrape.do's Amazon endpoint with a US ZIP code. The ZIP code matters: Amazon prices and stock depend on the delivery location, so a fixed ZIP gives you the same prices on every run.
import csv
import re
import requests
from bs4 import BeautifulSoup
TOKEN = "<your_scrape.do_token>"
PLUGIN = "https://api.scrape.do/plugin/amazon/"
def fetch(url, zipcode="10001"):
params = {"token": TOKEN, "url": url, "geocode": "us", "zipcode": zipcode, "output": "html"}
response = requests.get(PLUGIN, params=params, timeout=90)
response.raise_for_status()
return BeautifulSoup(response.text, "html.parser")
No render=true and no premium proxies: re-tested in October 2026, the plain call returned 200 in under 4 seconds with the full page.
2. Product data
Five selectors cover the useful fields:
-
#productTitlefor the name -
.a-price-whole+.a-price-fractionfor the price -
#outOfStockBuyBox_feature_div, which only exists when the product is out of stock -
.AverageCustomerReviewsfor the star rating -
img#landingImagefor the main image
def parse_price(soup):
whole = soup.find(class_="a-price-whole")
if whole is None:
return None
fraction = soup.find(class_="a-price-fraction")
digits = re.sub(r"[^\d]", "", whole.get_text())
cents = re.sub(r"[^\d]", "", fraction.get_text()) if fraction else "00"
return float(f"{digits}.{cents or '00'}")
def get_product(asin):
url = f"https://www.amazon.com/dp/{asin}/"
soup = fetch(url)
title = soup.find(id="productTitle")
if title is None:
raise ValueError(f"{asin}: no product title, probably not a product page")
out_of_stock = soup.find("div", id="outOfStockBuyBox_feature_div") is not None
rating_el = soup.find(class_="AverageCustomerReviews")
match = re.search(r"(\d+(?:\.\d+)?) out of", rating_el.get_text()) if rating_el else None
image = soup.find("img", id="landingImage")
return {
"asin": asin,
"name": " ".join(title.get_text().split()),
"price": None if out_of_stock else parse_price(soup),
"in_stock": not out_of_stock,
"rating": float(match.group(1)) if match else None,
"image": image.get("src", "") if image else "",
"url": url,
}
asins = ["B0BLRJ4R8F"] # add your own ASINs
with open("amazon_products.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["asin", "name", "price", "in_stock", "rating", "image", "url"])
writer.writeheader()
for asin in asins:
writer.writerow(get_product(asin))
.a-price-whole holds 1,029. and .a-price-fraction holds 87, so stripping everything but digits and joining them gives 1029.87 as a float. For out-of-stock products the price is set to None on purpose: the page often still shows an old or third-party price.
Three laptop stands from the guide came back at $29.87, $24.99 and $17.99, rated 4.6, 4.6 and 4.8.
3. Every variation of a product
A product like the iPhone 14 Plus (renewed) is one listing with many ASINs: 6 colors, 3 storage sizes, 8 carriers and 2 grades. Each dimension sits in a div#inline-twister-row-<dimension>, and every option is an li with its own data-asin.
TWISTER = re.compile(r"inline-twister-row-.*")
ORDER = ["color", "size", "style", "pattern", "material", "fit"]
def option_name(li):
swatch = li.find("span", class_="swatch-title-text-display")
if swatch:
return swatch.get_text(strip=True)
img = li.find("img")
if img and img.get("alt"):
return img["alt"].strip()
button = li.find("span", class_="a-button-text")
return button.get_text(strip=True) if button else "Unknown"
def dimensions(soup):
dims = {}
for row in soup.find_all("div", id=TWISTER):
name = row["id"].replace("inline-twister-row-", "").replace("_name", "")
options = [
{"name": option_name(li), "asin": li["data-asin"]}
for li in row.find_all("li", attrs={"data-asin": True})
# html.parser lowercases attribute names
if li["data-asin"] and li.get("data-initiallyunavailable") != "true"
]
if options:
dims[name] = options
return dims
The catch: available options change with the selection. Some sizes only exist in some colors, and some carriers only for some sizes. So you can't read all four menus from the first page and multiply them. You have to pick a color, load that page, read the sizes offered there, pick a size, load that page, and so on. That is a depth-first walk over the option tree:
def crawl_variations(url):
first = fetch(url)
dim_names = sorted(dimensions(first), key=lambda d: ORDER.index(d) if d in ORDER else len(ORDER))
pages, rows, done = {}, [], set()
def page(asin):
if asin not in pages: # never fetch the same ASIN twice
pages[asin] = fetch(f"https://www.amazon.com/dp/{asin}/?th=1&psc=1")
return pages[asin]
def walk(asin, level):
if asin in done:
return
soup = page(asin)
dims = dimensions(soup)
if level >= len(dim_names) or not dims:
done.add(asin)
title = soup.find(id="productTitle")
rows.append({
"asin": asin,
"name": " ".join(title.get_text().split()) if title else "",
**selected(soup, dim_names),
"price": parse_price(soup),
})
return
current = dim_names[level]
for option in dims.get(current, [{"asin": asin}]):
walk(option["asin"], level + 1)
walk(re.search(r"/dp/([A-Z0-9]{10})", url).group(1), 0)
return rows
def selected(soup, dim_names):
out = {}
for dim in dim_names:
row = soup.find("div", id=f"inline-twister-row-{dim}_name") or soup.find("div", id=f"inline-twister-row-{dim}")
chosen = row.find("span", class_=re.compile(r"a-button-selected")) if row else None
out[dim] = option_name(chosen) if chosen else "N/A"
return out
rows = crawl_variations("https://www.amazon.com/Apple-iPhone-Plus-128GB-Blue/dp/B0CG84XR6N/")
fields = sorted({k for r in rows for k in r}, key=lambda k: (k not in ("asin", "name"), k == "price", k))
with open("amazon_variations.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fields)
writer.writeheader()
writer.writerows(rows)
On the iPhone 14 Plus listing this took 289 requests and returned 151 variation rows. Budget one request per variation page and put a cap on it for big listings. The page cache matters: many paths through the tree end on the same ASIN, and without it you pay for those pages again.
4. All seller offers for an ASIN
The "Other sellers on Amazon" panel isn't in the product page HTML. It loads from its own endpoint, /gp/product/ajax/aodAjaxMain/?asin=<ASIN>, which you can fetch directly:
def get_offers(asin):
target = f"https://www.amazon.com/gp/product/ajax/aodAjaxMain/?asin={asin}"
r = requests.get("https://api.scrape.do/", params={"token": TOKEN, "url": target, "geoCode": "us"}, timeout=90)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")
offers = []
for sold_by in soup.find_all("div", id="aod-offer-soldBy"):
if sold_by.find_parent(id="aod-sticky-pinned-offer"):
continue # sticky copy of the Buy Box offer, already counted
box = sold_by.parent
while box is not None and box.find("span", class_="a-price-whole") is None:
box = box.parent # climb to the block that holds this offer's price
if box is None:
continue
link = sold_by.find("a", href=lambda h: h and "/gp/aag/main" in h)
rating = sold_by.find("div", id="aod-offer-seller-rating")
stars = re.search(r"(\d+\.?\d*) out of 5", rating.get_text()) if rating else None
count = re.search(r"\((\d[\d,]*)\s*ratings\)", rating.get_text()) if rating else None
positive = re.search(r"(\d+)%\s*positive", rating.get_text()) if rating else None
heading = box.find("div", id="aod-offer-heading")
ships = box.find("div", id="aod-offer-shipsFrom")
offers.append({
"asin": asin,
"price": parse_price(box),
"condition": " ".join(heading.get_text().split()) if heading else "New",
"seller": link.get_text(strip=True) if link else "Amazon.com",
"seller_rating": float(stars.group(1)) if stars else None,
"rating_count": int(count.group(1).replace(",", "")) if count else None,
"positive": f"{positive.group(1)}%" if positive else "",
"ships_from": ships.find("span", class_="a-color-base").get_text(strip=True) if ships else "",
})
return offers
Three details that trip people up:
-
The Buy Box offer appears twice. Once pinned at the top inside
#aod-sticky-pinned-offer, once in the list. Skip the pinned one or every count is off by one. -
No seller link means Amazon sells it. Third-party sellers link to
/gp/aag/main; Amazon itself doesn't. - The offer count isn't fixed. It changes by ASIN and by delivery location. One renewed iPhone 14 Plus listing returned 11 offers from 8 sellers on the first panel. One offer, or none, is also a valid result.
5. Skip the HTML: structured JSON
If you'd rather not maintain selectors, the same API has JSON endpoints that take an ASIN and return parsed data, 1 credit per successful request:
r = requests.get("https://api.scrape.do/plugin/amazon/pdp", params={
"token": TOKEN, "asin": "B0BLRJ4R8F", "geocode": "us", "zipcode": "10001",
})
product = r.json()
print(product["name"], product["price"], product["rating"], product["is_prime"])
offers = requests.get("https://api.scrape.do/plugin/amazon/offer-listing", params={
"token": TOKEN, "asin": "B0BN969PGL", "geocode": "us", "zipcode": "10001",
}).json()
print(offers["totalOfferCount"], [o["merchantName"] for o in offers["offers"]])
The PDP response includes brand, Prime, availability text, all images, best seller rankings and the full technical details table. The offer listing endpoint pages through every offer: on the same iPhone listing it reported 13 offers in total, against the 11 on the first HTML panel.
6. What about reviews?
Since May 2026, Amazon's /product-reviews/ pages return Page Not Found to logged-out visitors, and the review text is gone from the product page HTML. Anything behind a login is off-limits, so this guide doesn't scrape reviews.
7. Packaged version
All of the above, with retries, a variation page cap and a CLI, is on PyPI:
pip install amazon-scraping
export SCRAPEDO_TOKEN=your_token
amazon-scraping product --input asins.txt -f csv -o products.csv
amazon-scraping variations "https://www.amazon.com/dp/B0CG84XR6N/" -f csv -o variations.csv
amazon-scraping offers B0BN969PGL -f csv -o offers.csv
from amazon_scraping import AmazonScraper
scraper = AmazonScraper() # reads SCRAPEDO_TOKEN, ZIP 10001 by default
print(scraper.get_product("B0BLRJ4R8F"))
for offer in scraper.get_offers("B0BN969PGL"):
print(offer["seller"], offer["price"], offer["condition"])
Code and offline tests: github.com/bugrahansss/amazon-scraping. No-code option: the Amazon Scraper actor on Apify runs the JSON version by ASIN list.
Before you run it at scale
- Amazon's Conditions of Use prohibit robots and scrapers. Stick to public product data, keep request rates reasonable and know the laws that apply to you. (Not legal advice.)
-
Markup changes. The parser raises on a page without
#productTitleand leaves missing fields empty instead of guessing, so a layout change shows up as an error, not as wrong data. - Keep the ZIP code fixed when you track prices, so changes come from Amazon and not from a different location.
Scrape.do handles the proxies, headers and CAPTCHAs, with ZIP-level targeting in 11 Amazon marketplaces. The free plan includes 1,000 credits a month, enough to run every script in this post.
Source: Complete 2026 Amazon Scraping Guide: Product Data, Prices, Sellers, and More
Top comments (0)