Trustpilot needs a browser exactly once — the WAF token replays onto curl

Quick answer

Trustpilot sits behind an AWS WAF JavaScript interstitial, so plain HTTP clients get a 991-byte Verifying Connection page no matter which browser TLS fingerprint they present. That sounds like "every request needs a browser", which would make the target expensive enough to be not worth building. It isn't true. We measured what the challenge token is actually scoped to, and the answer is: not the business, not the page, not the filter. One headless browser navigation at startup, roughly eight seconds, and the resulting cookie replays onto ordinary curl-cffi for every business, every page and every star filter in the rest of the run.

The interstitial that doesn't care about your fingerprint 🧱

Four probes, two impersonation profiles, two different endpoints:

URL	Impersonate	Status	Bytes
/review/www.nike.com	chrome131	403	991
/review/www.nike.com	firefox133	403	991
/api/categoriespages/nike.com	chrome131	403	991
/api/categoriespages/nike.com	firefox133	403	991

Identical bodies, all four:

<title>Verifying Connection</title>
<script src="https://a7d575be72e8.edge.sdk.awswaf.com/.../challenge.js" defer></script>

server: CloudFront. This is diagnostic, and the diagnosis matters for cost. An IP-reputation block varies with the exit IP and sometimes with the fingerprint — you answer it with better proxies. A challenge-JS interstitial is served to every unsolved request identically, which is exactly the invariance we measured, and no amount of proxy budget solves it because the missing ingredient is a JavaScript runtime, not a cleaner IP. Rotating tiers here would have burned money to receive the same 991 bytes faster.

Also worth noting for anyone who goes hunting: that /api/categoriespages/ path is not a REST API you can pivot to. It is CloudFront-fronted like everything else, and once the WAF lets you through it answers 404. There's no side door.

The load-bearing test wasn't "can a browser get in" 🔑

Of course a browser gets in. Camoufox navigates to the review page, the interstitial clears itself in about eight seconds with no clicking, no puzzle, no manual step, and the page title becomes the real one. Everybody knows that part. The question that decides whether this Actor is viable is what the resulting aws-waf-token is scoped to — because if it's per-page, you need a browser per page and the unit economics die.

So we solved the challenge once, carried the cookie jar over to plain curl-cffi, and deliberately asked for things the browser had never touched:

Request over the replayed token	Never visited by the browser?	Result
/review/www.nike.com	no	200, 727,579 bytes
/review/www.adidas.com	yes — different business entirely	200, 708,446 bytes
/review/www.nike.com?page=2	yes	200 — 20 review IDs, all different from page 1
/review/www.nike.com?stars=5	yes	200 — star filter honoured

A brand-new business the browser never opened returned a full page. That's the result the architecture hangs on, and it's why the browser code path is fenced into a module that runs at most three times per run, never per request:

CHALLENGE_SOLVE_ATTEMPTS = 3
CARRIED_COOKIE_NAMES = ("aws-waf-token", "TP.uuid", "OptanonConsent")

Three attempts, because a browser launch can fail for boring reasons — a launch timeout is worth retrying, and the module returns None for recoverable failures rather than throwing, so one flaky start doesn't kill a run. If all three produce no aws-waf-token, that's a real failure and it's raised as one.

__NEXT_DATA__, and what we refuse to guess 📄

Every /review/<domain> response is a Next.js SSR page carrying a <script id="__NEXT_DATA__"> blob. We read that and only that — no CSS selectors, which is the difference between a redesign being a nuisance and a redesign being an outage. The same None vs [] discipline runs through the parser: extract_next_data() returns None when the tag is missing or malformed (loud failure, something changed), while parse_reviews() returns [] when reviews is genuinely an empty list (a real, correct answer — that business, that filter, no reviews). A single malformed review entry is skipped, not fatal; the other nineteen on the page still land.

Business-level context rides along on every row — trustScore, numberOfReviews, displayName — so a 50-row pull is self-describing without a second lookup. Nike's page reports 13,038 reviews and a 1.5 trust score, and you get both stamped on each review you pulled from it.

Output

One row per review, with its business context attached:

review_id, reviewer_name, reviewer_country, rating, title, body,
date_experienced, date_published, is_verified, verification_level,
company_reply_text, company_reply_date, language, source_business_domain,
business_display_name, business_trust_score, business_review_count

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/trustpilot-reviews-scraper").call(
    run_input={
        "businesses": ["nike.com", "amazon.com"],
        "maxReviewsPerBusiness": 50,
        "stars": 1,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["rating"], item["title"], "—", item["company_reply_text"] is not None)

company_reply_text is None when the business never replied, which makes "how many of our 1-star reviews went unanswered" a one-line filter. Pricing is $0.20 to start a run plus $0.003 per review — about $3.20 per 1,000 reviews; 100 reviews across a couple of businesses runs about $0.50.

→ Trustpilot Reviews Scraper on Apify

---

Built by Devil Scrapes. We solve the WAF challenge once per run and spend the rest of it on fast plain HTTP, retry the browser step when it flakes, and keep "no reviews matched" and "we got blocked" as two different answers instead of one empty dataset.