TL;DR
If you are searching for the Amazon Customer Reviews Dataset (571M reviews) for ML research, UCSD's McAuley Lab has you covered. But if you need live review data for a commercial app or AI Agent, building your own scraper with requests and BeautifulSoup is a bad idea. Amazon's login wall on /product-reviews/, TLS fingerprinting, and dynamic selectors will break your code in weeks. Here is a comparison of your options and Python code to get it running reliably.
The Landscape: Static Datasets vs. Real-Time Scraping
1. The UCSD Academic Dataset (571.54M Reviews)
The gold standard for researchers is the Amazon Reviews 2023 dataset from McAuley Lab at UC San Diego.
- Pros: 571 million reviews, 33 categories, clean user-item graphs, free via Hugging Face.
- Cons: Stale (stops at Sep 2023), non-commercial license (CC BY-NC 4.0), and misses Amazon's recent Customer Says AI summaries.
2. The DIY Python Scraper
Building your own script. Good for learning, bad for production.
3. Managed Review APIs
Services like Pangolinfo, Oxylabs, or Bright Data that manage proxy rotation, TLS fingerprints, and CAPTCHAs, returning clean JSON.
Why DIY Scrapers Fail
In late 2024, Amazon moved the /product-reviews/ path behind a login wall. If you aren't logged in, you get redirected to the sign-in page.
Additionally, Amazon's WAF inspects your TLS Client Hello packet. Python's default TLS handshake (JA3 fingerprint) is easily flagged compared to Chrome's, resulting in instant 503 blocks. To solve this, you need to use HTTP/2 clients like curl_cffi and buy residential proxies ($3–$15/GB).
Here is a basic detail-page scraper that works for the few featured reviews, but fails on deep histories:
import requests
from bs4 import BeautifulSoup
import time
import random
class BasicAmazonScraper:
def __init__(self):
self.headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
"Accept-Language": "en-US,en;q=0.5",
}
def fetch_featured_reviews(self, asin):
url = f"https://www.amazon.com/dp/{asin}"
# ⚠️ This will be blocked by Amazon WAF if run at high volume
time.sleep(random.uniform(3, 8))
res = requests.get(url, headers=self.headers, timeout=15)
if "ap/signin" in res.url:
print("Redirected to login wall!")
return []
soup = BeautifulSoup(res.text, "lxml")
reviews = []
for div in soup.select("[data-hook='review']"):
body = div.select_one("[data-hook='review-body']").get_text(strip=True)
rating = div.select_one("[data-hook='review-star-rating'] span").get_text(strip=True)
reviews.append({"rating": rating, "body": body})
return reviews
The Production Setup: Using a Managed API
Rather than spending hours rebuilding TLS bypasses and paying proxy bills, commercial projects use managed APIs.
import requests
class PangolReviewClient:
API_ENDPOINT = "https://api.pangolinfo.com/v1/amazon/reviews"
def __init__(self, api_key):
self.headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
def get_reviews(self, asin):
payload = {"asin": asin, "marketplace": "US", "page": 1}
# Pangolinfo manages TLS fingerprinting, proxy rotation, and the login wall
response = requests.post(self.API_ENDPOINT, json=payload, headers=self.headers, timeout=20)
return response.json()
if __name__ == "__main__":
client = PangolReviewClient(api_key="YOUR_API_KEY")
data = client.get_reviews("B08N5WRWNW")
print("AI Summary (Customer Says):", data.get("customer_says"))
print("Reviews Count:", len(data.get("reviews", [])))
For AI Agent builders, the Pangolinfo Amazon Data MCP exposes this structured review data via Model Context Protocol, so your agent can query live e-commerce reviews directly.
Summary
- For NLP Research/Offline Training: Use McAuley Lab's static Amazon Reviews 2023 dataset.
- For Production/AI Agent Tooling: Use a managed API like Pangolinfo Amazon Review API to bypass the login wall and get structured JSON.
Top comments (0)