DEV Community

Cover image for Scraping Amazon Product Reviews Without Hitting Sign-In Walls
Crawler Bros
Crawler Bros

Posted on Fully Autonomous

Scraping Amazon Product Reviews Without Hitting Sign-In Walls

#beginners #devops #webdev #webscraping

Data pipelines monitoring e-commerce customer sentiment face a strict wall on Amazon. Attempting to fetch historical customer feedback via anonymous GET requests to standard /product-reviews/{ASIN} endpoints results in immediate HTTP 302 redirects to Amazon's /ap/signin authentication page. This sign-in requirement applies across country domains regardless of request pacing, custom headers, or TLS fingerprinting techniques.

Because maintaining logged-in session state introduces account maintenance overhead and credential risk, pipeline architectures must shift to public-data modes. The Amazon Reviews Scraper Pro addresses this restriction by targeting the public product detail page (/dp/{ASIN}) directly, extracting structured review data from the inline review block without authenticating.

How Amazon's Public Review Extraction Functions

When a request loads an Amazon product detail page without session cookies, Amazon renders an inline review section containing approximately 8 to 13 "top reviews." Because this content resides directly within the public /dp/ HTML document, fetching it does not trigger the sign-in wall that blocks the dedicated /product-reviews/ routes.

The trade-off is structural: pagination beyond this inline block is impossible without authentication. However, for continuous sentiment tracking, brand monitoring, or catalog-wide sampling, capturing the high-impact top reviews across localized domains provides sufficient data points without account management overhead.

To maintain continuous access without getting blocked, the scraper relies on a cookie-free, residential proxy architecture. Amazon rapidly flags and blocks requests originating from datacenter IP ranges. Utilizing sticky-session residential proxies ensures that traffic mimics standard consumer ISP routing, preventing block screens and captcha challenges on public detail pages.

Key Input Parameters for Targeted Review Runs

To target specific feedback datasets, configure these explicit input parameters in your run payload:

  • productUrls (array of strings): Fully qualified Amazon product URLs (e.g., https://www.amazon.com/dp/B09X7MPX8L).
  • country (enum): Sets the target storefront domain out of 19 supported options (e.g., amazon.com, amazon.co.uk, amazon.de, amazon.co.jp). Reviews are extracted in the native language of the selected domain.
  • sortBy (enum): Set to helpful to prioritize reviews with high community interaction, or recent to capture newly submitted feedback.
  • filterByStar (enum): Pass "1", "2", "3", "4", or "5" to isolate specific star ratings. Because Amazon renders a fixed inline block on public pages, this filter operates on the client side after fetching the inline block, yielding between 0 and 5 reviews per star rating per product execution.
  • includeGdprSensitive (boolean): Controls whether personal identifying data is returned. Defaults to false, omitting reviewer names, profile links, and avatars to help comply with privacy regulations.

An example payload configured for localized, privacy-conscious review collection looks like this:

{
  "productUrls": [
    "https://www.amazon.com/dp/B09X7MPX8L"
  ],
  "country": "amazon.com",
  "maxReviews": 15,
  "sortBy": "recent",
  "filterByStar": "1",
  "includeImages": true,
  "includeGdprSensitive": false,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
Enter fullscreen mode Exit fullscreen mode

Structure of the Extracted Review Records

When the run completes, each review item written to the default dataset includes calculated metadata, clean text, and structured media links. Missing fields on the source page are omitted from the JSON payload rather than set to null.

{
  "productAsin": "B09X7MPX8L",
  "ratingScore": 5,
  "reviewTitle": "Solid build quality",
  "reviewUrl": "https://www.amazon.com/gp/customer-reviews/R1234567890",
  "reviewReaction": "123 people found this helpful",
  "reviewedIn": "Reviewed in the United States on January 15, 2024",
  "reviewDescription": "The material feels durable and matches the product listing dimensions...",
  "isVerified": true,
  "variant": "Size: Large, Color: Blue",
  "reviewImages": [
    "https://images-na.ssl-images-amazon.com/images/I/EXAMPLE.jpg"
  ],
  "position": 1,
  "reviewId": "R1234567890",
  "helpfulCount": 123,
  "reviewDate": "2024-01-15T00:00:00",
  "reviewLocation": "United States",
  "sentimentHint": "positive",
  "wordCount": 150,
  "hasImages": true,
  "imageCount": 1,
  "scrapedAt": "2024-01-20T10:30:00Z",
  "sourceUrl": "https://www.amazon.com/dp/B09X7MPX8L"
}
Enter fullscreen mode Exit fullscreen mode

Notice that isVerified is evaluated individually per item based on page badges. Verified-purchase pre-filtering at fetch time requires an authenticated account, so pipelines requiring exclusively verified reviews should filter on isVerified == true down-stream in post-processing.

Running the Scraper via API

To integrate this extraction process into an automated data pipeline using Python, use the official platform client library to trigger the run and collect dataset items directly into your environment.

1. Initialize the Execution

Install the client SDK (pip install apify-client) and pass your configuration payload:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "productUrls": [
        "https://www.amazon.com/dp/B09X7MPX8L",
        "https://www.amazon.co.uk/dp/B09X7MPX8L"
    ],
    "country": "amazon.com",
    "sortBy": "helpful",
    "includeImages": False,
    "includeGdprSensitive": False,
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

# Run the actor and wait for completion
run = client.actor("crawlerbros/amazon-reviews-scraper-pro").call(run_input=run_input)
Enter fullscreen mode Exit fullscreen mode

2. Fetch Structured Results

Once the process finishes, iterate through the default dataset associated with the execution ID:

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for review in dataset_items:
    print(f"ASIN: {review.get('productAsin')} | Rating: {review.get('ratingScore')}")
    print(f"Title: {review.get('reviewTitle')}")
    print(f"Text: {review.get('reviewDescription')[:100]}...\n")
Enter fullscreen mode Exit fullscreen mode

3. Programmatic Export via cURL

You can also fetch dataset outputs directly via HTTP endpoints for simple pipeline shell scripts:

curl "https://api.apify.com/v2/datasets/{datasetId}/items?token={apiToken}&format=json"
Enter fullscreen mode Exit fullscreen mode

Understanding Event-Based Usage Costs

Billing for this actor follows a strictly event-based model rather than variable compute duration calculations. Charges occur on two specific operational events:

  1. Actor Start (apify-actor-start): Charged at a flat rate of $0.005 per GB of memory allocated to the run upon execution start.
  2. Dataset Result (apify-default-dataset-item): Charged at $0.005 per dataset result returned in the default dataset.

For teams running high-volume extractions, dataset item pricing decreases automatically across standard volume tiers:

  • FREE: $0.005 per result
  • BRONZE: $0.00433 per result
  • SILVER: $0.00367 per result
  • GOLD / PLATINUM / DIAMOND: $0.003 per result

Because costs are tied directly to extracted records and memory allocation, pipeline expenses scale deterministically with the number of products processed.

Architectural Trade-Offs and Limitations

This implementation intentionally trades deep pagination for session stability and operational simplicity. Because anonymous /product-reviews/ requests redirect to sign-in prompts across all geographic regions, this actor cannot extract full historical review catalogs containing hundreds of items for a single product. It extracts only the top 8 to 13 inline reviews exposed on the public /dp/ product page. If your core requirement is extracting thousands of historical reviews per ASIN, an unauthenticated public-data architecture will not meet that objective.

Amazon Reviews Scraper Pro is what these steps drive. The README covers the inputs this article skipped, including the ones that change how much a run costs.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-18. Check the Actor page for the current rates.

Top comments (0)