DEV Community

Andrii Votiakov
Andrii Votiakov

Posted on

Reading Google Play and App Store reviews straight from their JSON, no browser

#webscraping #javascript #api #node

I spent an afternoon last year fighting a headless Chrome that scraped Play Store reviews. CAPTCHA, then a proxy bill, then a week later the DOM shifted and the whole thing returned empty arrays. Threw it out.

Turns out I never needed the browser. Both Google Play and the Apple App Store serve reviews as plain JSON you can hit with an HTTP request. No login, no proxies, no Playwright. This is the request shapes, the pagination caps that aren't in any docs, and the one place Google's format bit me.

All Node.js, all on got-scraping — a drop-in got replacement that copies a real browser's TLS and header fingerprint. That fingerprint earns its keep. The identical request from stock axios or fetch would sometimes come back 403 while got-scraping walked right through, because Play is fingerprinting the TLS handshake, not reading your User-Agent.

Apple first, because Apple made it easy

Apple publishes reviews as an RSS feed in JSON. One endpoint:

https://itunes.apple.com/{country}/rss/customerreviews/page={1-10}/id={appId}/sortby={mostrecent|mosthelpful}/json
Enter fullscreen mode Exit fullscreen mode
  • country — a storefront code (us, gb, de...). Every storefront keeps its own reviews.
  • appId — the numeric id from the store URL: apps.apple.com/us/app/whatsapp-messenger/id310633997.
  • page — 1 to 10, and 10 is the wall. Fifty reviews a page, so ~500 per storefront. Coming back to that. 
import { gotScraping } from 'got-scraping';

async function fetchAppleReviews(appId, { country = 'us', maxReviews = 200 } = {}) {
    const out = [];
    const pages = Math.min(10, Math.ceil(maxReviews / 50));
    for (let page = 1; page <= pages; page++) {
        const url = `https://itunes.apple.com/${country}/rss/customerreviews/page=${page}/id=${appId}/sortby=mostrecent/json`;
        const res = await gotScraping({ url, responseType: 'json' });
        const entries = res.body?.feed?.entry ?? [];
        // The first entry is sometimes app metadata, not a review — guard on im:rating.
        for (const e of entries) {
            if (!e['im:rating']) continue;
            out.push({
                id: e.id.label,
                author: e.author.name.label,
                rating: Number(e['im:rating'].label),
                title: e.title.label,
                text: e.content.label,
                version: e['im:version'].label,
                date: e.updated.label,
            });
        }
        if (!entries.length) break;
    }
    return out.slice(0, maxReviews);
}
Enter fullscreen mode Exit fullscreen mode

That 500-review cap is hard. No continuation token gets you past it — I looked. What does work: reviews are scoped per storefront, so pull us, then gb, au, ca, de, and the rest of the English-language storefronts, deduping on review id. There are about ten of them; at ~500 each that's roughly 5,000 recent reviews, which is usually plenty.

App metadata — title, average rating, total count, current version — lives at a second, simpler endpoint:

https://itunes.apple.com/lookup?id={appId}&country=us
Enter fullscreen mode Exit fullscreen mode

Google Play, and the batchexecute rabbit hole

Play has no clean REST endpoint. The store front-end talks to an internal RPC called batchexecute, and the payload is ugly and documented nowhere. The payoff for climbing through it: Play paginates with no ceiling. Apple caps you at 500 a storefront; Play just keeps going.

The endpoint:

POST https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Enter fullscreen mode Exit fullscreen mode

The body is a URL-encoded f.req parameter wrapping the RPC id UsvDTd and its arguments:

function buildBody(pkg, { count = 100, token = null, sort = 2 } = {}) {
    const tok = token ? `\\"${token}\\"` : 'null';
    const inner = `[null,null,[2,${sort},[${count},null,${tok}],null,[]],[\\"${pkg}\\",7]]`;
    const freq = `[[["UsvDTd","${inner}",null,"generic"]]]`;
    return 'f.req=' + encodeURIComponent(freq);
}
Enter fullscreen mode Exit fullscreen mode

sort is 2 for newest, 1 for relevance, 3 for rating. pkg is the package name (com.whatsapp). token is the continuation cursor the previous response handed back.

The response is where it gets weird. It opens with an anti-JSON-hijacking guard, the literal )]}', and then a nested envelope where the real data sits as a JSON string inside the outer JSON. You parse twice:

function parse(raw) {
    const envelope = JSON.parse(raw.slice(raw.indexOf('[')));   // strip )]}'
    const inner = envelope?.[0]?.[2];                            // a JSON *string*
    if (!inner) return { reviews: [], nextToken: null };
    const data = JSON.parse(inner);
    const reviews = (data[0] ?? []).map((r) => ({
        id: r[0],
        author: r[1][0],
        rating: r[2],
        text: r[4],
        date: new Date(r[5][0] * 1000).toISOString(),
        thumbsUp: r[6],
        reply: r[7]?.[1] ?? null,     // developer reply text
        appVersion: r[10],
    }));
    const nextToken = data[1]?.[1] ?? null;
    return { reviews, nextToken };
}
Enter fullscreen mode Exit fullscreen mode

Then loop, feeding nextToken back until you've got enough or it comes back null:

async function fetchPlayReviews(pkg, { maxReviews = 200 } = {}) {
    const out = [];
    let token = null;
    while (out.length < maxReviews) {
        const res = await gotScraping({
            url: 'https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us',
            method: 'POST',
            body: buildBody(pkg, { count: 150, token }),
            headers: { 'content-type': 'application/x-www-form-urlencoded;charset=UTF-8' },
        });
        const { reviews, nextToken } = parse(res.body);
        if (!reviews.length) break;
        out.push(...reviews);
        if (!nextToken) break;
        token = nextToken;
    }
    return out.slice(0, maxReviews);
}
Enter fullscreen mode Exit fullscreen mode

A few things I only learned by running this against real apps:

  • Play reviews carry no title. Just a rating and a body. Apple gives you both. If you're merging the two stores into one schema, title has to be nullable or you'll drop half your Play data on a strict validator.
  • Developer replies hide in slot [7] on Play — text at [7][1], timestamp at [7][2][0]. Apple's public feed doesn't surface replies at all.
  • The loop fires requests back to back with no delay. Proxyless, from my laptop and from a datacenter, I haven't been rate-limited doing this — but it's the assumption most likely to break at tens of thousands of reviews, and I'd put a throttle in front of it before trusting it at that scale.

And here's where it got me. I first pulled appVersion from slot [8], eyeballed one response, saw a version string, shipped it. Some apps came back with a country code there instead. The version is [10]. The index parsing is brittle by design — Google can reshuffle slots whenever they like, and nothing tells you. So I pinned a test against a known app with a review I can eyeball and assert the fields on it. When Google moves something, that test screams before my users notice.

Is this actually worth skipping the browser?

For me, yes — and the number that convinced me: 250 Play reviews land in about 0.6 seconds this way. You're reading the exact API the store's own frontend reads, so a store redesign doesn't touch you. My Playwright version was 10 to 20 times slower, wanted a proxy budget the moment I scaled it, and died on the next UI refresh. I don't miss it.

If you'd rather not babysit the slot indices

I bundled both stores into one actor on Apify — one schema, handles the pagination and the batchexecute envelope, runs proxyless: App Reviews Scraper. It's mostly there so I stop re-fixing the [10]-versus-[8] kind of thing every quarter. But the code above is the whole trick, and rolling your own is very doable.

The batchexecute envelope eats afternoons if you go in blind — if you get stuck on it, leave a comment and I'll dig in.

Top comments (2)

Collapse
 
frank_signorini profile image
Frank •

This is brilliant! How reliable have you found the direct JSON endpoints to be for Play Store reviews

Collapse
 
votiakov profile image
Andrii Votiakov •

Thanks Frank. Reliable in practice. It's the same batchexecute RPC the Play web store itself calls, so it stays up as long as that internal endpoint does, and paginating through the continuation tokens has been solid across a lot of apps for me.

Two things to watch. The payload is JSON nested inside JSON, keyed by an internal RPC id, so if Google reshuffles that structure your parser breaks even though the endpoint still responds fine. That's the part that actually needs maintenance, not the request itself. And there's no auth, so at high volume you'll want to throttle or you start getting empty pages back. For normal review pulls I've run it proxyless with no real trouble.

Apple's side is the opposite: dead simple RSS, but it caps at roughly the last 500 reviews per storefront, so you lose the long tail.