DEV Community: PsyMall

How to Get Amazon Review & Rating Data at Scale (and What's Actually Public in 2026)

PsyMall — Tue, 14 Jul 2026 09:00:03 +0000

If you sell on Amazon, research a category, or build product-intelligence tooling, you eventually need the same thing: ratings and review signal across many products at once. Not one ASIN. Dozens or hundreds. A competitor's whole catalog. Every product in a subcategory. The question is never "what do people say about this one item." It's "where is the rating strength, where are the complaints clustering, and how is that moving over time."

That's harder than it looks, for two reasons most tutorials skip.

Why naive scraping fails

Amazon treats automated traffic as hostile by default. Send a plain HTTP request or a default headless browser and you'll usually get the "Robot Check" / CAPTCHA page instead of the product. Default headless fingerprints (the giveaway user-agents, the missing browser signals) get flagged fast.

There's a second, quieter failure that costs people more. Amazon serves different pages to different countries. Hit a .com product from an IP in the wrong country and Amazon often returns a stripped page, sometimes with the reviews section missing entirely. Your scraper "works," returns a 200, and hands you a row with no review data. You don't notice until the dataset is already wrong.

What actually works

Four things, together:

A realistic browser fingerprint. A real Chrome user-agent and consistent browser signals, not the default headless profile. You're loading the page the way a person's browser does.
Residential proxies pinned to the marketplace's country. Scraping amazon.co.uk? Use UK residential IPs. .de? German IPs. This is the single biggest fix for the "empty reviews" problem. It's what makes Amazon serve the full page.
Scroll to trigger lazy loading. Reviews and the rating breakdown load as you scroll. Grab the DOM too early and you miss them.
Treat empty as failure. If a run finishes with zero reviews and no rating summary, that's a bug, not a result. Fail loudly. A silent empty return is how you end up trusting a broken dataset.

The honest part: what's actually public in 2026

Here's the reality most tools won't tell you. Since late 2024, Amazon only shows roughly the top ~8 reviews per product to logged-out visitors. The dedicated "all reviews" page now sits behind a login wall. Any tool claiming to pull a product's complete review history without an account is either logging in (against Amazon's terms, and fragile) or quietly returning less than it implies.

So be clear-eyed about scope. Publicly, per product, you can reliably get:

The rating intelligence: overall star score, total number of ratings, and the full star breakdown (% 5★ / 4★ / 3★ / 2★ / 1★).
Amazon's own "Customers say" AI summary.
The top ~8 public reviews, each with reviewer, star rating, title, body, date, verified-purchase flag, and helpful votes.

That's a lot of signal when you're pulling it at scale. What it isn't is deep single-product review mining.

A quick workflow

Collect the ASINs or product URLs you care about (a competitor's catalog, a subcategory).
For each, load the product page with a real browser fingerprint and a country-matched residential proxy.
Scroll to force the reviews and rating breakdown to load.
Parse the rating summary, the star breakdown, the "Customers say" summary, and the visible reviews.
Validate: any product returning nothing is flagged, not silently dropped.

The actor

I built this into an Apify actor: Amazon Reviews & Rating Intelligence Scraper. Give it a list of ASINs or product URLs, and for each it returns the rating summary, the star breakdown, the "Customers say" summary, and the public reviews, across marketplaces (.com, .co.uk, .de, and others). It uses the fingerprint plus country-matched residential proxy approach above, and it fails loudly on empty results.

When to use it: competitor catalog research, category/subcategory analysis, and rating monitoring over time. Basically anywhere the per-product summary plus top reviews across many products is the point.

When not to: if you need one product's entire multi-thousand review history. Publicly that isn't available (~8 reviews/product), and this tool won't pretend otherwise.

Pricing is pay-per-event: platform usage passed through, plus about $1.50 per 1,000 rows.

Takeaway

Amazon rating data at scale is very doable if you match the proxy country, use a real fingerprint, scroll, and fail loudly on empty. Full public review history per product isn't, because Amazon closed that door in late 2024. Knowing exactly where the public line sits is what keeps your dataset honest.

How to Scrape Airbnb Market Data (Prices, Ratings, Availability) at Scale

PsyMall — Sat, 11 Jul 2026 09:00:03 +0000

If you do short-term-rental analysis, competitive pricing, or property research, you eventually need the same thing: prices, ratings, review counts, and locations across a lot of Airbnb listings at once. One listing tells you nothing. A whole neighborhood tells you what the market is actually charging.

The problem is that pulling that data reliably is harder than it looks.

Why naive Airbnb scraping breaks

The obvious approach is to load an Airbnb search page and parse the HTML: find the price element, find the rating element, read the text. This works in a demo and fails in production, for three reasons.

The HTML is not stable. Airbnb's front end is a heavily componentized, class-obfuscated React app. The CSS class names change, the DOM structure shifts, and prices are rendered in ways that are deliberately awkward to select. A selector that works today silently returns nothing next week.

Prices are contextual. The number you see depends on currency, dates, guest count, and whether a discount is active. If you scrape the visible text without controlling those inputs, you get a mix of currencies and can't tell a discounted price from a regular one.

You get blocked, or worse, you get emptiness. Airbnb rate-limits and challenges automated traffic. The dangerous failure isn't an error, it's a run that "succeeds" and returns zero rows, or a page of listings with every price field blank. If your pipeline treats that as valid, your analysis is built on nothing.

What actually works

The reliable path ignores the rendered HTML almost entirely.

Read the embedded page-state JSON. Airbnb ships the data used to render each search page as a structured JSON blob inside the page itself. That's the same source the site uses to draw the listings, so it's far more stable than scraping visual elements. It contains listing name, price, rating, review count, coordinates, room details, and images as clean fields, no selector guessing.

Use a real browser. You still need a real browser session to load the page and get that state, because a plain HTTP request won't pass Airbnb's challenges. Once the page loads, you extract the JSON, not the DOM.

Force the currency explicitly. Pass Airbnb's own currency parameter so every price comes back in one unit. Don't infer it.

Paginate with the site's native cursors. Airbnb's search uses cursor-based pagination. Follow those cursors instead of guessing page-number URLs.

Fail loudly on empty. Treat a zero-listing run as a failure, not a success. A green run that returns nothing is the exact trap that makes bad data look fine.

Step by step

Pick a market: a city name, or paste a full Airbnb search URL (with your dates and filters already applied).
Set your currency.
Load each search page in a real browser and extract the embedded state JSON.
Follow the native cursor to the next page until the results end.
Reject the run if it produced zero listings.

A ready-made option

If you'd rather not maintain this yourself, the Airbnb Market Data Scraper on Apify does exactly the above. Give it a city or a search URL and it returns clean rows: listing name, price per night (including both the discounted and original pre-discount price when a sale is on), rating, review count, lat/lng, bedrooms/beds, images, and the listing URL. It reads the embedded JSON state, forces your chosen currency, follows native cursors, and fails loudly on an empty run. Pricing is pay-per-result, roughly $3.50 per 1,000 listings.

When to use it: STR investors sizing a market, property managers benchmarking nightly rates, pricing and market analysts building comps, and anyone tracking competitors across a whole city.

Honest limits, so you plan correctly: this returns search-result market data, not deep per-listing amenities or host history (that's a separate, future job). Airbnb also caps search pagination at about 15 pages (~270 listings) per query. To cover a full market, run several narrower searches, by neighborhood, by date window, or by price band, and combine them.

Takeaway

For Airbnb market data, don't scrape the HTML. Load the page in a real browser, read the JSON state the page already carries, pin the currency, follow the native cursors, and never trust a run that comes back empty.

How to Take Clean Website Screenshots at Scale (Without Pop-ups Ruining Them)

PsyMall — Wed, 08 Jul 2026 09:00:05 +0000

Screenshotting a webpage sounds trivial until you need to do it a thousand times and have every image come back usable. Anyone who's tried knows how it actually goes. Half the captures show a cookie consent wall, a newsletter modal covering the hero section, or a "we use cookies" bar glued to the bottom. The page you wanted is there, technically, but it's hidden behind an overlay. And if you're feeding those screenshots to an AI agent, a monitoring dashboard, or a client report, the noise makes them close to useless.

Here's why the naive approach fails and what actually produces clean captures at scale.

Why simple screenshotting breaks

A quick script that loads a URL and snaps an image runs into four predictable problems:

Consent banners and pop-ups. Most sites in the EU (and increasingly everywhere) throw a cookie/GDPR banner on first load. Newsletter modals, chat widgets, and app-install interstitials pile on top. None of these are in the page you care about, but they land in front of it.
Lazy-loaded content. Modern pages load images and sections as you scroll. Capture too early and you get gray placeholder boxes instead of real content.
Device sizes. A desktop-width screenshot tells you nothing about how the page looks on a phone, and plenty of use cases need both.
Doing it reliably in bulk. One screenshot is easy. A thousand of them, with retries, timeouts, and consistent output, is an engineering problem.

What actually works

The reliable recipe uses a real headless browser (Chromium via Playwright or Puppeteer), not a lightweight HTTP fetch. Concretely:

Load in a real browser so JavaScript runs and the page renders the way a human would see it.
Wait for the page to settle. Wait for network to go idle or for a load event, and scroll the page to trigger lazy-loaded assets before capturing.
Dismiss overlays before the shot. This is the part most tools skip. Query the DOM for the common consent and pop-up patterns (buttons with text like "Accept", "Accept all", "I agree", "Got it", plus the widespread consent-framework selectors) and click or remove them. Handle close buttons on modals the same way. It's not glamorous, but maintaining a good selector list is what separates clean output from cluttered output.
Set device viewports. Render at a desktop viewport and a mobile viewport (correct width, height, and device scale factor) to get true previews for each.

A short step-by-step

Launch a headless Chromium instance.
Set the viewport to your target device size.
Navigate to the URL and wait for the load/network-idle event.
Scroll to the bottom (and back) to force lazy content to load.
Run your overlay-dismissal pass: click known consent/pop-up selectors, remove leftover sticky bars.
Capture, full-page or viewport, and save the image somewhere with a stable, retrievable URL.
Log failures loudly. A run that "succeeds" but returns a blank or overlay-covered image is worse than one that errors, because you'll end up trusting the bad data.

That last point matters most at scale. Don't accept green-but-empty. If a capture comes back blank or the overlay clearly wasn't dismissed, treat it as a failure you can see, not a silent pass.

A ready-made option

If you'd rather not maintain the browser fleet, the selector lists, and the retry logic yourself, I built an Apify actor that does exactly this. It's called Screenshot Pro: https://apify.com/psymall/screenshot-pro

It captures full-page or device-preset (desktop/mobile) screenshots in bulk, auto-closes cookie banners and pop-ups before capturing, and returns a public signed image URL for each shot. Pricing is a small per-screenshot fee on top of platform usage.

When it's a good fit:

AI agents that need to capture and reason over pages as part of a pipeline.
Visual monitoring / diffing. Snapshot pages on a schedule and compare over time to catch layout or content changes.
Thumbnails for link previews, directories, or dashboards.
QA and design review. Verify how pages render across device sizes without opening each one by hand.

Takeaway

Clean bulk screenshots come down to three things: use a real browser, wait for the page to fully render, and actively dismiss the overlays before you capture. Skip any one of them and you get images you can't trust. Whether you build it yourself or use a ready-made actor, hold the line on failing loudly. A screenshot you can't spot the problem in is the most expensive kind.

How to Scrape Clutch.co Agency Data in 2026 (Without Getting Blocked)

PsyMall — Sun, 05 Jul 2026 11:54:38 +0000

Clutch.co is one of the richest public directories of B2B service providers out there. Agencies for development, marketing, design, and more, each with ratings, verified reviews, hourly rates, team size, and location. If you do lead generation, competitive research, or market sizing, it's a goldmine.

It's also genuinely annoying to scrape. If you've tried, you already know why.

Why most Clutch scrapers fail

Send a plain HTTP request to a Clutch listing page and you get back a 403 with a Cloudflare challenge instead of the data. It happens on almost every request, even for robots.txt. Tools built on simple HTTP clients (requests, axios, cheerio-only actors) hit this wall and either return nothing or break at random.

That's the real reason most Clutch scrapers on the market are unreliable. They never actually get past the front door. What you usually see is a scraper that "succeeds" with an empty result set, which is worse than failing outright, because you think you have data and you don't.

What actually works

The fix isn't a smarter HTTP header. It's using a real browser. A headless Chromium instance (via Playwright) solves Cloudflare's managed challenge on its own within a few seconds, the same way a normal visitor's browser does. After that, the full rendered page is right there.

Two things keep it reliable at scale:

A real browser, not raw HTTP. Playwright or Chromium passes the challenge. A bare request never will.
Residential proxies. Datacenter IPs get flagged fast. Rotating residential IPs keep the success rate high across a lot of pages.

Once you're in, the data is clean and structured. Roughly 60 companies per listing page, each with a name, profile URL, review count, rating, minimum project size, hourly rate, description, locations, and the agency's own website.

The extraction, step by step

Open the listing URL (say, https://clutch.co/agencies/digital-marketing) in a headless browser with a residential proxy.
Wait for the company cards to render. The Cloudflare challenge clears on its own.
Read each card's fields from the page.
Paginate with ?page=N, and stop when a page returns no new companies. Clutch clamps out-of-range pages to the last valid one, so naive pagination just loops on duplicates. De-dupe by the stable provider id.
Fail loudly if a run returns zero results, so you never pay for an empty dataset.

That last point matters more than it looks. The most common complaint about existing Clutch scrapers is "it ran, said success, returned nothing." A scraper you can trust treats an empty run as an error, not a success.

A ready-made option

If you'd rather not build and maintain the browser, proxy, and pagination logic yourself, I published a Clutch.co scraper on the Apify Store that does all of the above:

→ Clutch.co Agency Directory Scraper

Give it a category or location URL and it returns clean, structured company data: name, rating, reviews, min project size, hourly rate, team size, location, and website. It uses a real browser with residential proxies to get past Cloudflare, paginates on its own, de-dupes, and fails loudly on an empty run. Pricing is pay-per-result ($3.00 per 1,000 companies), so you only pay for data you actually get.

When to use it

Lead generation: build a targeted list of agencies by service and location.
Competitive research: track how competitors are rated and priced.
Market analysis: size a category by number of providers, ratings spread, and typical rates.

Build your own or use the actor, the takeaway is the same. On Clutch the wall is Cloudflare, the key is a real browser plus residential proxies, and what separates a scraper you can trust from one you can't is whether it fails loudly when it comes back with nothing.