DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Scraping Amazon Reviews: the 200 OK that looks like an empty product

Quick answer

To pull Amazon reviews as structured rows, you need three things the product page won't hand you for free: a request fingerprint that survives Amazon's bot checks, pagination that doesn't silently truncate at page 10, and a schema that keeps verified_purchase and helpful_vote_count as real typed fields instead of scraped strings. The Amazon Reviews Scraper does all three and emits 13 validated fields per review at $0.0025 per review ($2.50 per 1,000) plus a $0.01 run start.

Why Amazon reviews are harder than they look 🔍

Amazon's review surface is one of the most defended pages on the consumer web, and it defends itself in ways that are easy to miss because they don't look like failures.

A naive requests.get() doesn't get blocked with a clean 403. It gets a page. The page just happens to be a stripped-down variant with the review list missing, or a "Sorry, we just need to make sure you're not a robot" interstitial that returns HTTP 200. If your parser is written defensively it returns zero rows and you conclude the product has no reviews. If it isn't, it throws a NoneType error three functions deep and you spend an afternoon debugging your selectors instead of your transport layer.

The second trap is pagination. Review lists are paginated behind a URL pattern that changes shape depending on whether you're hitting the product page's embedded reviews, the dedicated reviews page, or the AJAX endpoint the "Next page" control actually calls. Each surface exposes a different subset of fields, and each caps out at a different depth. A scraper that follows the visible "Next" link will quietly stop well short of the full corpus on high-review products — and it will stop without erroring, which is the worst possible failure mode when you're building a dataset.

The third trap is the fields themselves. "Verified Purchase" is a badge in markup, not a boolean. Helpful votes are a localized string — "1,247 people found this helpful" — that has to be parsed into an integer, with the awkward edge case that a review with zero helpful votes doesn't render the element at all. Dates are localized text, not ISO strings. If you dump raw scraped values into a dataset, every downstream consumer has to re-derive the same parsing logic, and they'll each get it subtly wrong.

What we handle so you don't 🛡️

This is the part we actually built the Actor around. Amazon is not an easy target and we don't pretend otherwise — the value here is that the hard parts are already absorbed.

Browser-grade request fingerprinting. We use curl-cffi to impersonate a real browser's TLS and HTTP/2 fingerprint, not just its User-Agent string. Header-only spoofing is what most scrapers do and it's what most anti-bot stacks catch first, because the TLS handshake gives you away before a single header is read.

Retries and backoff on soft blocks. A 200-with-no-reviews is treated as a block signal, not as an empty result. The run retries with fresh session state rather than recording a false negative.

Proxy-aware transport. Runs route through Apify's proxy infrastructure so requests don't all originate from one address. This matters more on Amazon than on almost any other target.

Post-dedupe billing. You're charged per review row actually written to the dataset, after deduplication. Retries, blocked pages, and duplicate rows across pagination boundaries don't cost you anything. That's a deliberate choice: the pricing shouldn't punish you for the target's flakiness.

What you actually get back 📦

Every row is a Pydantic-validated record with extra="forbid" — meaning the schema is enforced, not aspirational. Thirteen fields, exactly:

Field Type Notes
asin string The ASIN this review belongs to
product_url string \ null
product_title string | null From the product page header, best-effort
review_id string | null Amazon's internal review anchor ID, when exposed
reviewer_name string | null Reviewer display name
rating float Star rating, constrained to 1.0–5.0
review_title string | null Review headline
review_text string Review body
verified_purchase bool True when the badge is present in source markup
review_date string ISO-8601, parsed from localized date text
helpful_vote_count int Defaults to 0 when the element is absent
marketplace_domain string Echoes your input marketplace
scraped_at datetime UTC timestamp at scrape time

Note what's null-able and what isn't. rating, review_text, verified_purchase, review_date, and helpful_vote_count are non-nullable — if we can't produce them, the row doesn't ship. That's the point of validating at the boundary: you never get a half-row that breaks your pipeline at 3am.

Who this is for

AI/ML teams building training corpora. Review text paired with a verified-purchase flag and a numeric rating is one of the cleanest sentiment-labelled datasets available at scale. The verified_purchase boolean is the filter that separates signal from incentivized noise.

E-commerce competitive intelligence. Pull the review corpus for a competitor's ASIN, track rating distribution over time, and find the recurring complaint that shows up in 1- and 2-star reviews. helpful_vote_count is a decent proxy for which complaints other buyers actually care about.

Product teams doing voice-of-customer work. Reviews for your own catalogue plus your three nearest substitutes, in one schema, refreshed on a schedule.

Frequently asked questions

How much does it cost to pull 10,000 reviews?
$25.01 — 10,000 × $0.0025, plus the $0.01 run-start event. Billing is per row written post-dedupe, so blocked pages and retries are free.

Does it work on non-US Amazon marketplaces?
Yes — marketplace_domain is an input, and it's echoed back on every row so multi-marketplace datasets stay unambiguous when you merge them.

Will it get blocked?
Amazon actively works to prevent automated collection, and anyone claiming a 100% success rate on this target is selling you something. What we commit to is that blocks are detected rather than silently recorded as empty results, that runs retry with fresh session state, and that you aren't billed for rows that never landed.

Why per-review pricing instead of per-page?
Because pages have wildly variable review counts and you shouldn't pay the same for a page with 2 reviews as for one with 20. Per-row post-dedupe is the only pricing that tracks the value you actually receive.

Can I get only verified purchases?
Filter on verified_purchase == true downstream. We ship the flag on every row rather than filtering server-side, so you keep the option of analyzing the unverified population too — which is often where the interesting anomalies live.

Try it

The Actor is live on the Apify Store: Amazon Reviews Scraper.

Run it from the Console with an ASIN and a review cap, or drive it from the Apify API and pipe the dataset straight into whatever you're building. Pay-per-event means there's no subscription — you pay for rows you receive.


Built by Devil Scrapes — we build scrapers for the targets that fight back.

Top comments (0)