DEV Community: Andrii Votiakov

We cut a Lambda bill 96.5% by fixing a retry storm.

Andrii Votiakov — Mon, 20 Jul 2026 16:51:44 +0000

The invocation graph is the first thing I open on any serverless engagement, and this one looked wrong from across the room. 29.3K Lambda invocations per hour. For a workload that justified a few hundred.

Nobody had noticed, because nothing was down. That's the ugly property of a retry storm. The system stays green while it burns money. Every retry eventually succeeds, lands in a dead-letter queue, or quietly expires. Dashboards show traffic, and the only symptom is a bill that grows faster than the product does.

What a retry storm actually is

AWS retries on your behalf in more places than most teams can list. Async Lambda invocations retry twice by default. An SQS event source leaves failed messages on the queue, and after the visibility timeout they become visible again and are redelivered. EventBridge has its own retry policy. Step Functions too. Each layer is individually reasonable. Stacked together, they multiply.

The failure mode that bit this client was surprisingly simple. The handler wrapped everything in try/catch, logged the error, and re-threw. Everything. Including validation errors on malformed messages. A malformed message can never succeed, but the code told SQS, "please try again."

Without a redrive policy, SQS keeps redelivering a failing message until the retention period expires. The default retention period is four days. One poison message became four days of Lambda invocations. There were thousands of poison messages, and the function that consumed their output failed too, creating its own retry storm downstream.

The fix

Four changes, in order of impact.

1. Classify errors before throwing

Retryable and terminal errors are different animals. A network timeout deserves another attempt. A schema violation belongs in a parking lot, not in an infinite retry loop.

export const handler = async (event) => {
  const batchItemFailures = [];

  for (const record of event.Records) {
    try {
      await process(JSON.parse(record.body));
    } catch (err) {
      if (err instanceof ValidationError) {
        await parkMessage(record, err);
        continue;
      }

      batchItemFailures.push({
        itemIdentifier: record.messageId,
      });
    }
  }

  return { batchItemFailures };
};

This uses SQS partial batch responses, so only the failed messages are retried. Without this feature, one failed record can cause the entire batch to be retried, including messages that were already processed successfully.

2. Configure a dead-letter queue with a sensible `maxReceiveCount`

Three attempts, then move the message to a DLQ and alert on queue depth.

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:eu-west-1:123456789:orders-dlq",
    "maxReceiveCount": 3
  }
}

The DLQ is not a trash can. It is a work queue for humans. Someone should periodically inspect it, fix the underlying issue, and decide whether those messages should be replayed.

3. Add a circuit breaker around flaky downstream services

When a dependency is down, ten thousand Lambda invocations hammering it will not make it recover any faster. Fail fast, back off, and let SQS absorb the burst.

One caveat: circuit breakers are awkward in Lambda because execution environments are ephemeral. The breaker state has to live somewhere external. We stored ours in a DynamoDB item with a TTL. It worked well enough, even if it was not the prettiest implementation.

4. Tune timeouts to reality

The function timeout was set close to the Lambda maximum "to be safe." Safe for whom?

Long timeouts increase the maximum cost of every failed invocation and delay retries. We set the timeout to our p99 execution time plus some headroom. That reduced the cost of failures while still giving healthy requests enough time to complete.

The result

Lambda invocations dropped from 29.3K per hour to 669. That's a 97.7% reduction. The Lambda line item fell by 96.5%.

Same product. Same traffic. Same features.

The only difference was that we stopped paying to retry work that could never succeed.

Don't forget idempotency

Retries only work safely if your processing is idempotent.

If processing the same order twice sends two emails, ships two packages, or charges the customer twice, retries become a business problem instead of an infrastructure problem.

Whenever you rely on retries, make sure your handlers can safely process the same message multiple times.

How to catch this in your own account

Spend ten minutes checking these today.

Track the ratio of Errors / Invocations in CloudWatch. Alert when the error rate is consistently above your normal baseline.
For every SQS-triggered Lambda, verify the queue has a redrive policy. Without a DLQ, poison messages keep cycling until the retention period expires.
Compare Lambda invocations with real business events. If you process 10,000 orders per day but see 300,000 invocations, retries are probably inflating the numbers.
Review MaximumRetryAttempts for asynchronous Lambda invocations. The default is two retries, but for some idempotency-sensitive workloads you may want zero.

Retries are a good default for a world where failures are often transient. They are a terrible default for deterministic failures. Your error handling is what tells AWS which kind it is.

If your AWS bill feels wrong and you'd rather not gamble on another generic cloud consultancy, I do this at reducecost.cloud on a pay-for-savings basis. Zero upfront. You only pay a share of what I actually save you.

Stop writing a parser per site. Run five and let confidence decide.

Andrii Votiakov — Mon, 20 Jul 2026 09:40:12 +0000

For a long time I ran product extraction off a database of custom selector configs. Hundreds of retailers, each with its own set of CSS selectors mapped field by field: price here, title there, image over there. It worked. It was also a treadmill. Every retailer that redesigned its frontend silently broke its config, and the maintenance tax grew with every retailer I added. Hundreds of configs is hundreds of things that can rot without telling you, usually right before someone downstream asks why a brand's prices went null.

At some point I stopped feeding it. The scraping platform I ran at my last engagement fed a 20M+ product catalogue at millions of pages a day, and the thing that made that survivable wasn't better per-site config. It was leaning on almost no per-site config at all.

The pattern

Instead of one careful parser per site, run several cheap generic extractors on every page, in parallel:

JSON-LD. A huge share of e-commerce pages ship a schema.org/Product block because Google rewards it. It has name, price, currency, availability, images. It's the closest thing to a public API hiding in the HTML.
OpenGraph tags. og:title, og:image, product:price:amount. Lower quality than JSON-LD, but present on sites too lazy for structured data, because everyone wants pretty link previews.
Microdata / RDFa. Older sites, still surprisingly common outside the US.
Embedded state. __NEXT_DATA__, window.__INITIAL_STATE__ and friends. A JSON blob the site's own frontend hydrates from.
Heuristics. A price-shaped string near a currency symbol inside the main content region. The largest above-the-fold image. The h1. Dumb, and dumb works more often than you'd think.

None of these is reliable alone. OG price tags go stale, JSON-LD sometimes describes the wrong variant, heuristics grab the crossed-out "was" price. The trick is you don't pick an extractor. You pick per field, and you let agreement between sources carry the decision.

Confidence merge

Each extractor emits candidates with a prior based on how trustworthy that source usually is for that field. Then candidates that agree with each other get boosted, and the top score wins per field:

function merge(results, fields) {
  const out = {};
  for (const field of fields) {
    const candidates = results
      .filter((r) => r.data[field] != null)
      .map((r) => ({
        value: r.data[field],
        source: r.name,
        score: PRIORS[r.name][field] ?? 0.3,
      }));

    for (const c of candidates) {
      const agreeing = candidates.filter(
        (o) => o !== c && normalize(o.value) === normalize(c.value)
      ).length;
      c.score *= 1 + 0.4 * agreeing;
    }

    candidates.sort((a, b) => b.score - a.score);
    out[field] = candidates[0] ?? null;
  }
  return out;
}

normalize matters more than the scoring. "1 299,00 zł" and "1299.00 PLN" have to compare equal or your agreement boost never fires. I spent more time on value normalization than on everything else in this system combined.

Add a validation pass after the merge: does the price parse, is the currency a real ISO code, is the title longer than three characters. A merged record that fails validation is a signal, not a result. Log it, sample those pages weekly, and you'll find out which extractor is drifting before your customers do.

Why bother

Because the long tail is where per-site config dies. The top few hundred retailers can justify hand-tuned selectors, and I maintained exactly that for years. Site number 4,000 does not, and with an ensemble it doesn't need any. A new domain comes in, four of five extractors fire, three agree on price, and you have a usable record with zero engineering hours spent. Across that platform we held roughly 95% extraction success at about 3 seconds a page, and most of the domains involved had nobody who had ever looked at their HTML.

The honest limitation: variant-heavy product pages are still rough. When one URL carries twelve colorways at different prices, the ensemble happily returns a confident answer to the wrong question, because every source on the page describes the default variant. I handle that with a separate variant expansion step, and it's the least elegant code I own.

But for "give me name, price, image, availability from an arbitrary shop URL," the ensemble beat every hand-written parser I've replaced with it. Write extractors for formats, not for websites. Formats change maybe once a decade. Websites change on Tuesdays.

This approach is the core of what I'm building at Cartpie, an e-commerce product-data scraping platform, free tier live at cartpie.com if you want to throw some long-tail URLs at it.

Most company job boards are just a public JSON API you can GET

Andrii Votiakov — Sun, 19 Jul 2026 16:24:37 +0000

I spent an afternoon trying to scrape a careers page with a headless browser before I noticed the page itself was calling a JSON endpoint. The company runs Greenhouse. Greenhouse serves the whole board — every open role, full descriptions — over one unauthenticated GET. No Playwright, no proxy, no waiting for React to hydrate. I deleted the browser code and never looked back.

That's the thing nobody tells you when you're building a job aggregator or a "who's hiring" tracker. Stripe, Spotify, Ramp — thousands of companies post through four ATS platforms, and each one exposes a per-company feed that's already public. You just have to know the URL shape.

Here are the four I use, with real slugs so you can curl along.

The endpoints

Every platform keys off a slug — the token buried in the careers URL.

Greenhouse

https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true

?content=true gives you the full HTML description inline. Hit stripe and you get 500+ jobs in a single response — no pagination at all. The slug is whatever sits in boards.greenhouse.io/stripe.

const res = await gotScraping({
    url: 'https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=true',
    responseType: 'json',
});
// res.body.jobs -> [{ id, title, location: {name}, content, absolute_url, departments, offices, first_published, updated_at }]

One trap cost me twenty minutes: Greenhouse double-encodes the content field. You get <p> where you expected <p>. Run it through an HTML-entity decode once before you touch it, or you'll render literal tags in your UI and blame your own template.

Lever

https://api.lever.co/v0/postings/{slug}?mode=json

Try spotify or ro. You get a flat array back, and the useful fields (team, location, commitment) hide one level down under categories:

// [{ id, text, categories: { team, location, commitment }, description, descriptionPlain, hostedUrl, applyUrl, createdAt }]

A 404 here is not an error. It just means the company isn't on Lever, which is exactly what you want to hear when you're guessing which platform a slug belongs to.

Ashby

https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true

Try ramp. Ashby is the one I wish everyone used. Structured location, secondaryLocations, a real isRemote boolean, employmentType, and both descriptionHtml and descriptionPlain so you don't have to strip tags yourself. When the company fills in comp bands, you get those too.

SmartRecruiters

https://api.smartrecruiters.com/v1/companies/{slug}/postings?limit=100&offset=0

Try Visa. This is the odd one out — it paginates. You get a totalFound and you loop limit/offset until you've pulled everything. And here's the catch that bit me: the list endpoint returns metadata only. No description body. If you want the actual job text you have to call a separate per-posting detail endpoint, one request each. So the "free and fast" story falls apart a bit for SmartRecruiters — you're paying N+1 requests for full text.

Turning four shapes into one

The endpoints are the easy part. The value is a single record shape so the rest of your code never has to know or care which ATS a company runs. A thin per-provider normalizer does it:

function normalizeGreenhouse(job) {
    return {
        source: 'greenhouse',
        id: String(job.id),
        title: job.title,
        location: job.location?.name ?? null,
        department: (job.departments ?? []).map((d) => d.name).join(', ') || null,
        remote: /remote/i.test(job.location?.name ?? ''),
        url: job.absolute_url,
        descriptionHtml: decodeEntities(job.content),
        postedAt: job.first_published ?? null,
    };
}

function normalizeLever(job) {
    const c = job.categories ?? {};
    return {
        source: 'lever',
        id: String(job.id),
        title: job.text,
        location: c.location ?? null,
        department: c.team ?? c.department ?? null,
        remote: /remote/i.test(job.workplaceType ?? c.location ?? ''),
        url: job.hostedUrl,
        descriptionHtml: job.description,
        postedAt: job.createdAt ? new Date(job.createdAt).toISOString() : null,
    };
}
// ...ashby, smartRecruiters similarly

Notice remote is a regex guess on Greenhouse and Lever because neither gives you a clean flag. Ashby actually hands you isRemote, so use it there instead of guessing. That inconsistency is the whole reason the normalizer exists.

Guessing the platform when you only have a name

Say a user types "Ramp" and you have no idea which ATS they're on. You could look it up. Or you could throw the slug at all four at once and keep whatever answers:

async function findJobs(slug) {
    const results = await Promise.allSettled([
        fetchGreenhouse(slug),
        fetchLever(slug),
        fetchAshby(slug),
        fetchSmartRecruiters(slug),
    ]);
    return results
        .filter((r) => r.status === 'fulfilled' && r.value.length)
        .flatMap((r) => r.value);
}

The 404s from the three wrong platforms come back in a couple hundred milliseconds and cost nothing. In a real build you'd resolve the actual slug by parsing the careers-page embed — the ATS board is almost always iframed or linked from there — but bare-slug guessing covers more companies than you'd expect: company names and slugs match more often than not.

1,199 roles in one pass

To sanity-check that this actually holds up, I ran it across 8 real companies spanning all four platforms and pulled 1,199 open roles in a single sweep. Proxyless, roughly 200ms per company on the single-request platforms. The stable part is what sells it: these are the same feeds the companies' own job widgets call, so a marketing redesign doesn't touch them. Scrape the rendered page and you're one CSS class rename away from a broken parser.

Where it stops working

Not every ATS plays along, and I'd rather tell you now than have you burn an afternoon.

SmartRecruiters' list feed has no descriptions. Covered above — plan for the N+1 or ship jobs without body text. And log() the pagination totals loudly: a truncated limit/offset loop will quietly return page one and let you think you got the whole board.
Workable is auth-gated now. The old open feed wants a token. Same story with Recruitee. A year ago both were open; that window closed.

So this isn't a universal skeleton key. It's four platforms that happen to still be open, and they happen to cover a large slice of tech hiring. Good enough that I stopped writing careers-page scrapers entirely.

The ready-made version

I packed all four platforms, the unified schema, and the auto-detect into an Apify actor — ATS Job Feed Scraper. But the endpoints above really are most of it. Build your own; it genuinely is a fun afternoon, and you'll understand every field.

If you know another ATS that still serves an open feed, drop it in the comments and I'll add notes. I'm especially curious whether anyone has a live Workable path that doesn't need a token.

Reading Google Play and App Store reviews straight from their JSON, no browser

Andrii Votiakov — Sun, 19 Jul 2026 16:22:13 +0000

I spent an afternoon last year fighting a headless Chrome that scraped Play Store reviews. CAPTCHA, then a proxy bill, then a week later the DOM shifted and the whole thing returned empty arrays. Threw it out.

Turns out I never needed the browser. Both Google Play and the Apple App Store serve reviews as plain JSON you can hit with an HTTP request. No login, no proxies, no Playwright. This is the request shapes, the pagination caps that aren't in any docs, and the one place Google's format bit me.

All Node.js, all on got-scraping — a drop-in got replacement that copies a real browser's TLS and header fingerprint. That fingerprint earns its keep. The identical request from stock axios or fetch would sometimes come back 403 while got-scraping walked right through, because Play is fingerprinting the TLS handshake, not reading your User-Agent.

Apple first, because Apple made it easy

Apple publishes reviews as an RSS feed in JSON. One endpoint:

https://itunes.apple.com/{country}/rss/customerreviews/page={1-10}/id={appId}/sortby={mostrecent|mosthelpful}/json

country — a storefront code (us, gb, de...). Every storefront keeps its own reviews.
appId — the numeric id from the store URL: apps.apple.com/us/app/whatsapp-messenger/id310633997.
page — 1 to 10, and 10 is the wall. Fifty reviews a page, so ~500 per storefront. Coming back to that.

import { gotScraping } from 'got-scraping';

async function fetchAppleReviews(appId, { country = 'us', maxReviews = 200 } = {}) {
    const out = [];
    const pages = Math.min(10, Math.ceil(maxReviews / 50));
    for (let page = 1; page <= pages; page++) {
        const url = `https://itunes.apple.com/${country}/rss/customerreviews/page=${page}/id=${appId}/sortby=mostrecent/json`;
        const res = await gotScraping({ url, responseType: 'json' });
        const entries = res.body?.feed?.entry ?? [];
        // The first entry is sometimes app metadata, not a review — guard on im:rating.
        for (const e of entries) {
            if (!e['im:rating']) continue;
            out.push({
                id: e.id.label,
                author: e.author.name.label,
                rating: Number(e['im:rating'].label),
                title: e.title.label,
                text: e.content.label,
                version: e['im:version'].label,
                date: e.updated.label,
            });
        }
        if (!entries.length) break;
    }
    return out.slice(0, maxReviews);
}

That 500-review cap is hard. No continuation token gets you past it — I looked. What does work: reviews are scoped per storefront, so pull us, then gb, au, ca, de, and the rest of the English-language storefronts, deduping on review id. There are about ten of them; at ~500 each that's roughly 5,000 recent reviews, which is usually plenty.

App metadata — title, average rating, total count, current version — lives at a second, simpler endpoint:

https://itunes.apple.com/lookup?id={appId}&country=us

Google Play, and the batchexecute rabbit hole

Play has no clean REST endpoint. The store front-end talks to an internal RPC called batchexecute, and the payload is ugly and documented nowhere. The payoff for climbing through it: Play paginates with no ceiling. Apple caps you at 500 a storefront; Play just keeps going.

The endpoint:

POST https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us
Content-Type: application/x-www-form-urlencoded;charset=UTF-8

The body is a URL-encoded f.req parameter wrapping the RPC id UsvDTd and its arguments:

function buildBody(pkg, { count = 100, token = null, sort = 2 } = {}) {
    const tok = token ? `\\"${token}\\"` : 'null';
    const inner = `[null,null,[2,${sort},[${count},null,${tok}],null,[]],[\\"${pkg}\\",7]]`;
    const freq = `[[["UsvDTd","${inner}",null,"generic"]]]`;
    return 'f.req=' + encodeURIComponent(freq);
}

sort is 2 for newest, 1 for relevance, 3 for rating. pkg is the package name (com.whatsapp). token is the continuation cursor the previous response handed back.

The response is where it gets weird. It opens with an anti-JSON-hijacking guard, the literal )]}', and then a nested envelope where the real data sits as a JSON string inside the outer JSON. You parse twice:

function parse(raw) {
    const envelope = JSON.parse(raw.slice(raw.indexOf('[')));   // strip )]}'
    const inner = envelope?.[0]?.[2];                            // a JSON *string*
    if (!inner) return { reviews: [], nextToken: null };
    const data = JSON.parse(inner);
    const reviews = (data[0] ?? []).map((r) => ({
        id: r[0],
        author: r[1][0],
        rating: r[2],
        text: r[4],
        date: new Date(r[5][0] * 1000).toISOString(),
        thumbsUp: r[6],
        reply: r[7]?.[1] ?? null,     // developer reply text
        appVersion: r[10],
    }));
    const nextToken = data[1]?.[1] ?? null;
    return { reviews, nextToken };
}

Then loop, feeding nextToken back until you've got enough or it comes back null:

async function fetchPlayReviews(pkg, { maxReviews = 200 } = {}) {
    const out = [];
    let token = null;
    while (out.length < maxReviews) {
        const res = await gotScraping({
            url: 'https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us',
            method: 'POST',
            body: buildBody(pkg, { count: 150, token }),
            headers: { 'content-type': 'application/x-www-form-urlencoded;charset=UTF-8' },
        });
        const { reviews, nextToken } = parse(res.body);
        if (!reviews.length) break;
        out.push(...reviews);
        if (!nextToken) break;
        token = nextToken;
    }
    return out.slice(0, maxReviews);
}

A few things I only learned by running this against real apps:

Play reviews carry no title. Just a rating and a body. Apple gives you both. If you're merging the two stores into one schema, title has to be nullable or you'll drop half your Play data on a strict validator.
Developer replies hide in slot [7] on Play — text at [7][1], timestamp at [7][2][0]. Apple's public feed doesn't surface replies at all.
The loop fires requests back to back with no delay. Proxyless, from my laptop and from a datacenter, I haven't been rate-limited doing this — but it's the assumption most likely to break at tens of thousands of reviews, and I'd put a throttle in front of it before trusting it at that scale.

And here's where it got me. I first pulled appVersion from slot [8], eyeballed one response, saw a version string, shipped it. Some apps came back with a country code there instead. The version is [10]. The index parsing is brittle by design — Google can reshuffle slots whenever they like, and nothing tells you. So I pinned a test against a known app with a review I can eyeball and assert the fields on it. When Google moves something, that test screams before my users notice.

Is this actually worth skipping the browser?

For me, yes — and the number that convinced me: 250 Play reviews land in about 0.6 seconds this way. You're reading the exact API the store's own frontend reads, so a store redesign doesn't touch you. My Playwright version was 10 to 20 times slower, wanted a proxy budget the moment I scaled it, and died on the next UI refresh. I don't miss it.

If you'd rather not babysit the slot indices

I bundled both stores into one actor on Apify — one schema, handles the pagination and the batchexecute envelope, runs proxyless: App Reviews Scraper. It's mostly there so I stop re-fixing the [10]-versus-[8] kind of thing every quarter. But the code above is the whole trick, and rolling your own is very doable.

The batchexecute envelope eats afternoons if you go in blind — if you get stuck on it, leave a comment and I'll dig in.

Stop parsing HTML to get product data — the JSON is already in the page

Andrii Votiakov — Sun, 19 Jul 2026 16:19:49 +0000

CSS selectors against a rendered product page are a trap. They look fine until marketing ships a redesign on a Tuesday and your div.price__amount--v2 returns null for every SKU. Then you're spinning up a headless browser just to read a number that was in the HTTP response the whole time.

Here's the thing I wish someone had told me three storefronts ago: the product data is almost always already JSON, sitting right there in the page. You don't parse it out of the DOM. You pluck it out of a script tag or an XHR. I've built scrapers for a couple dozen retailers now, and I reach for selectors maybe once a year.

Three places to look, best first.

JSON-LD, the one retailers hand you on purpose

Retailers want Google's rich results, so they embed schema.org/Product markup. That's a <script type="application/ld+json"> block with price, currency, availability, sometimes a GTIN — a documented shape you can rely on across completely unrelated sites.

import { gotScraping } from 'got-scraping';

async function fromJsonLd(url) {
    const res = await gotScraping({ url });
    const blocks = [...res.body.matchAll(
        /<script type="application\/ld\+json"[^>]*>([\s\S]*?)<\/script>/g,
    )];
    for (const [, json] of blocks) {
        const data = JSON.parse(json);
        const product = [].concat(data, data['@graph'] ?? [])
            .find((n) => n && n['@type'] === 'Product');
        if (product) {
            return {
                title: product.name,
                price: product.offers?.price ?? product.offers?.[0]?.price,
                currency: product.offers?.priceCurrency,
                availability: product.offers?.availability,   // schema.org/InStock etc.
                gtin: product.gtin13 ?? product.gtin,
            };
        }
    }
    return null;
}

No browser. No selectors. The shape is a public standard, so the same parser works on a shoe store and a hardware store.

Where it falls down: JSON-LD is the marketing-facing summary, not the truth. I've hit pages where the <script> block cheerfully says InStock while the actual buy button is greyed out, because the stock check happens client-side after hydration. So JSON-LD is my first call, never my only one.

Embedded app state — `__NEXT_DATA`, `NUXT`, `window.INITIAL_STATE__`

Anything built on Next, Nuxt, or a similar framework hydrates from a JSON blob baked into the HTML. This is usually the good stuff. Full variant matrices, per-size stock, image sets, seller info — everything the page needs to render, which is a lot more than JSON-LD bothers to expose.

const m = res.body.match(/<script id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/);
const state = JSON.parse(m[1]);
// then dig to state.props.pageProps.product — path varies per site

The path down to the product object is site-specific and annoying to find the first time. Open the blob, JSON.stringify a few candidate nodes, eyeball which one has the price. Once you've mapped it for a retailer it stays put, and it's the richest source you'll get short of the internal API.

The internal mobile JSON API

When there's no JSON-LD and no embedded state — or when the embedded state is deliberately thin — open the network tab and watch what the page fetches. Most SPAs pull the product from an XHR that returns clean JSON, something along the lines of /api/product/{id}. Hit that directly and you skip rendering entirely; you're reading the exact payload the site's own frontend trusts.

This is the approach I lean on hardest for the SHEIN scraper. SHEIN's desktop pages are heavy and defensive, but the site's internal mobile JSON API returns the product already structured — title, price, the full variant list, images — no HTML parsing at any step. I recently added a keyword-search mode on top of it, so you can hand the actor a search term, let it walk the same mobile API for matching product IDs, then run each ID through the detail path. Discovery and extraction, both off the JSON API, no browser in the loop.

One catch that cost me an afternoon the first time: a lot of these internal endpoints return a 403 to a plain fetch. It looks like an IP ban, so you go buy proxies, and it's still 403. It's not your IP — it's your TLS and header fingerprint. Node's default client doesn't handshake like a real browser, and these endpoints notice. got-scraping exists for exactly this; it mimics a browser's TLS fingerprint and header order, and the same request that was getting bounced starts returning 200 with the JSON you wanted.

// plain fetch -> 403, no matter how many proxies you throw at it
// got-scraping -> 200, because the fingerprint reads as a real browser
const res = await gotScraping({ url: internalApiUrl, responseType: 'json' });

Fetching is the easy 20%. Normalization is the other 80%

Getting one store's JSON is a lazy afternoon. The part that actually eats your time is that every retailer names everything differently. Price is price, or current_price, or salePrice.amount. Availability is a boolean here, a schema.org URL there, a raw stock integer somewhere else. Variants come as a flat array on one site and a nested option matrix on the next. A multi-store dataset is worthless unless every product lands in one fixed schema:

{
  title, price, currency, availability,   // normalized to in_stock | out_of_stock
  variants: [{ name, price, available, sku }],
  gtin, images, url
}

That mapping layer — a small adapter per retailer, all feeding one output shape — is where every hour of maintenance goes. It's also the whole difference between "I scraped a store once" and "I have a product feed I can trust on Monday morning."

When you don't want to babysit the adapters

Owning a tuned adapter per retailer, and re-tuning each one when the site shifts a field, is a real recurring cost. If you'd rather POST a product URL and get normalized JSON back across 20-plus retailers, that's the exact chore I'm building Cartpie to eat — one schema across every store. And if SHEIN is your only target, the SHEIN product scraper on Apify does the single-store version.

Scraping one or two stores yourself? The three techniques above cover you with no browser anywhere. JSON-LD first, embedded state when you need the variant matrix, internal API when the site forces your hand.

What's the ugliest product schema you've had to flatten? Mine was a marketplace that base64-encoded the real price and buried it three levels deep, presumably so competitors wouldn't do exactly what I was doing.

Google Trends will 429 you into the ground — here's how I got past it

Andrii Votiakov — Sun, 19 Jul 2026 16:18:34 +0000

The most popular Google Trends actor on Apify sits at three-and-change stars, and if you read the reviews they all say the same thing: it works for a while, then everything comes back 429. That's not a bug someone forgot to fix. It's the actual hard part of scraping Trends, and most tools just don't handle it. I know because I rebuilt mine three times before it stopped falling over.

There's no official API. The internal one hands you single-use tokens and throttles per IP with no mercy. Here's the whole thing — the request flow, the parts Google quietly refuses to serve automated sessions, and the retry machinery that actually holds up.

The token dance

Trends has no "give me the data for this keyword" endpoint. You go through a handshake.

First, explore. You tell it what you want and it hands back a set of widgets, each carrying a one-time token plus the request payload you'll replay:

GET https://trends.google.com/trends/api/explore?hl=en-US&tz=0&req={...}

req is a JSON object describing your comparison items — one keyword, or several to compare, each with a geo and time window:

{"comparisonItem":[{"keyword":"bitcoin","geo":"US","time":"today 12-m"}],"category":0,"property":""}

URL-encode that, drop it into req=, and you're off. The response comes back prefixed with )]}', — that's Google's anti-XSSI guard, sitting in front of every Trends API response so the body can't be executed as a script tag. Strip everything before the first { or your JSON parser chokes on line one.

Then, for each section you want, you call its data endpoint with that widget's token:

GET .../trends/api/widgetdata/multiline?...&token={token}&req={widget.request}

multiline gives you interest over time
comparedgeo gives you interest by region
relatedsearches gives you related queries and topics

Same )]}', prefix on all of them.

Here's the part worth knowing: the token partially signs the request. You can edit some fields of the widget request — bump resolution for finer geo granularity — and it still validates. Touch the wrong field, like userConfig.userType, and you get a flat 401. So no, you can't just promote yourself to a logged-in user by rewriting a string. I tried. It 401s.

The sections Google won't give a robot

Related topics come back empty. Not sometimes — for every anonymous scraper, always. Your explore response gets tagged userType: USER_TYPE_SCRAPER, and for that user type Google serves an empty rankedList on the entity-typed related-topics section while related queries keep working fine. A logged-in browser gets USER_TYPE_LEGIT_USER and the full payload. Consent cookies don't move the needle. I spent a genuinely embarrassing amount of time convinced I'd broken my own parser before I diffed a browser session against my scraper and saw the userType flip.

So I'll say it plainly, because it's the thing I'm most confident about in this whole space: related topics for anonymous sessions is server-side gated and I don't think it's beatable without a real logged-in cookie. If a competing actor advertises full related-topics anonymously, test it on a fresh keyword before you trust the marketing.

City-level geo is walled off the same way. Country, region/state, and US metro (DMA) resolutions all return real data. Ask for CITY and automated sessions get hasData: false.

One thing that is free: Trending Now skips the token dance entirely. There's a plain RSS feed —

https://trends.google.com/trending/rss?geo=US

No token, works anonymously, ~10-20 stories with traffic estimates and the news articles behind each. If trending searches are all you need, you never have to fight the API at all.

The 429 wall

This is the whole game. Google rate-limits per IP, and it does it hard — a naive loop dies inside a few dozen requests and then your IP is cold for hours. Everything below exists because of this one problem.

A session is one IP paired with one cookie jar. Warm it by hitting a Trends page first — any status code sets the cookie you need. Then reuse that pair for a batch.

When a 429 lands, do not retry the same IP. Rotate to a fresh one. This is where residential proxies earn their money and datacenter IPs don't: datacenter ranges are largely burned against Google and you'll see single-digit success rates on them. Between rotations, back off exponentially, and pace requests inside a session instead of firing them all at once.

The piece that saved my completeness numbers is per-section retry. If multiline succeeds but relatedsearches 429s, retry just that one section on a fresh session — don't throw away a keyword you already have three-quarters of.

async function withSession(fn, { maxRotations = 5 } = {}) {
    for (let i = 0; i < maxRotations; i++) {
        const session = newSession();          // fresh proxy IP + cookie jar
        try {
            await warmup(session);             // sets the NID cookie
            return await fn(session);
        } catch (err) {
            if (err.response?.statusCode === 429) {
                await sleep(2 ** i * 1000);     // backoff, then rotate
                continue;
            }
            throw err;
        }
    }
    throw new Error('exhausted sessions (429)');
}

A warning from experience: don't develop this against your home IP. Around 100 calls you'll hit a captcha-302 wall that takes hours to clear, and you'll lose the afternoon locked out of the site you're trying to debug. Point it at a small residential pool from the start.

Once the rotation and per-section retry were both in, I ran the actor across 8 test scenarios covering every section type and geo resolution. All 8 came back clean at 100% section completeness, averaging around 6 seconds per keyword end to end. That number only exists because of the per-section retry — without it, one unlucky 429 on the last section drags a "successful" keyword down to partial.

Values are relative, and no scraper changes that

Trends never gives you absolute search volume. Every series is normalized 0-100 against its own peak in the window you asked for. That's true on trends.google.com itself, so it's not something a scraper can fix or unlock — it's just what the data is. If you want an absolute-ish archive, the only path is to schedule snapshots over time and stitch them together, a daily reading you accumulate rather than a number you fetch.

The box I put it in

Token dance, session rotation, exponential backoff, per-section retry, the Trending RSS shortcut — all of it lives in an Apify actor: Google Trends Scraper. It exists because the 429 handling is genuinely tedious to get right and I got tired of rewriting it from scratch for every project. Nothing in it is proprietary magic, though — the flow above is the real thing, and you could build it yourself from this post.

And if you've actually pulled related topics out of an anonymous session, come tell me how. That's the one piece I've filed under unbeatable, and I would love to be wrong.