Expensive Discovery, Cheap Enrichment: A Marketplace Scraper Architecture Built Around Rate Limits

Most marketplace scrapers are shaped like a human browsing session: open the listing page, read the cards, click into each product, read the detail page, go back, next page. Whether it's Playwright or a stack of HTTP calls, the control flow mirrors what a person would do.

That shape is intuitive and it scales badly, for a reason that has nothing to do with your code.

The measurement that changes the design

Before writing a scraper, I now spend thirty minutes on one question: which hosts does this site's data actually come from, and which of them are rate limited?

Not "is the site rate limited." Which hosts.

Here is what that looked like on Wildberries, one of the largest marketplaces in Russia. Its front end is served by four distinct services. I hit each one with concurrent requests from a single datacenter IP:

Host	What it serves	Test	Result
search.wb.ru	search / discovery	50 parallel	30 × 200, 20 × 429
card.wb.ru	product detail, price, stock	30 parallel	30 × 200
feedbacks1.wb.ru	reviews	30 parallel	30 × 200
basket-NN.wbbasket.ru	attributes, media (CDN)	40 parallel	40 × 200

Sequentially, search was just as unfriendly: 30 requests spread over 69 seconds returned 13 × 200 and 17 × 429. A cold first request can come back 429 before you have sent anything else.

Three of the four services did not throttle me at all.

The limit is per service, and only one service is expensive. That is not a Wildberries quirk; it is the normal consequence of how large marketplaces are built. Search is a ranking system running over a huge index — it is the costly thing to serve and the thing competitors scrape, so it sits behind a gateway. Product cards, reviews and static media are cache-friendly, often on a CDN, and defended lightly or not at all.

Once you know that, the browsing-session shape is obviously wrong. It spends the scarce resource on work the abundant resources would have done for free.

The pattern: two tiers

Split the scraper into two tiers with different economics.

Tier 1 — discovery (scarce). The only job here is to turn a query into a list of identifiers. Nothing else. Paginate, collect ids, stop. This tier is rate limited, so it is serial, backed off, and the only place where proxies are worth paying for.

Tier 2 — enrichment (abundant). Everything else: prices, stock, reviews, attributes, images. Driven entirely by the ids from tier 1, batched, parallel, usually straight from your own IP.

On Wildberries this maps cleanly. Search returns 100 products per page, and each product carries two identifiers: id (the SKU-level nmId) and root (the parent card imtId). Every enrichment endpoint keys off one of those. One throttled request buys 100 items' worth of cheap work.

The ratio is the whole point. If a discovery call yields 100 ids, and enrichment for those 100 ids costs you 2 batched card requests plus 100 unthrottled review requests, then the overwhelming majority of your traffic is on endpoints that don't fight you. Getting more data no longer means hitting the wall more often.

Finding the tiers on a new target

The recipe is short and works on any site with a JSON-backed front end:

1. Read robots.txt first. This is not a formality — it is an architectural input, and it can delete an entire design before you write a line of it. I have a worked example below where it did exactly that.
2. Open the site with devtools on the network tab, filter to XHR/Fetch. Note the distinct hostnames. Most marketplaces use between two and five.
3. Replay each interesting request and strip cookies, tokens and headers one at a time to see what is genuinely required. You often find that nothing was. But do not do this with plain curl or Node's built-in fetch, or you will get false negatives. Some gateways fingerprint the TLS handshake and header ordering, not just the User-Agent. On Wildberries' search host, fetch() with default headers returns an HTML anti-bot page while got-scraping returns JSON — same URL, same IP, seconds apart. Probe with a client that emulates a real browser's TLS profile (got-scraping, curl-impersonate), otherwise you will write off endpoints that work fine.
4. Probe each host separately for its tolerance.

Step 4 is the one people skip. It is about ten lines:

async function probe(url, n = 10) {
  const t0 = Date.now();
  const codes = await Promise.all(
    Array.from({ length: n }, () =>
      fetch(url).then((r) => r.status).catch(() => 0),
    ),
  );
  const tally = {};
  for (const c of codes) tally[c] = (tally[c] ?? 0) + 1;
  return { host: new URL(url).host, n, ms: Date.now() - t0, tally };
}

Run it once per host, with a small n, and stop as soon as you have your answer. You are measuring someone else's infrastructure — a handful of requests tells you what you need, and there is no version of this where hammering a target is the right call. On the targets where I got a clear "no" on the first probe, I stopped there rather than escalating.

Then write the numbers down. They are the input to your design, and they go stale — treat them as dated facts, not permanent truths. The last section of this post is about what happened when I re-measured mine.

What the implementation looks like

Once the tiers are explicit, the code gets simpler, not more complex. Discovery is a generator that yields ids:

const DEST = '-1257786';           // pin region-ish params for the whole run
const PAGE_SIZE = 100;

async function* discover(query, maxPages = 5) {
  let firstIdOfPageOne;

  for (let page = 1; page <= maxPages; page++) {
    const url =
      'https://search.wb.ru/exactmatch/ru/common/v4/search' +
      `?query=${encodeURIComponent(query)}&resultset=catalog` +
      `&dest=${DEST}&curr=rub&spp=30&appType=1&lang=ru&page=${page}`;

    const res = await fetchWithBackoff(url);
    const { products = [] } = await res.json();
    if (products.length === 0) return;

    // This host never serves an empty page at the end of a result set — it starts
    // serving page 1 again. Terminating on "empty" alone yields silent duplicates.
    if (page === 1) firstIdOfPageOne = products[0].id;
    else if (products[0].id === firstIdOfPageOne) return;

    for (const p of products) yield { nmId: p.id, imtId: p.root };
    if (products.length < PAGE_SIZE) return;
  }
}

That duplicate check is not paranoia. Measured on 2026-07-18, page 60 of a ноутбук search returned page 1's exact contents — full 100 products, HTTP 200, no marker. A loop that walks until it sees an empty page never terminates on this host; it just keeps re-emitting the first page and hammering the one service that is actually rate limited. Whatever your target, verify how it signals "no more results" before you trust a termination condition.

The backoff belongs to this tier only, and it has to be heuristic: the 429 response on this host is an HTML body with no Retry-After header, carrying little more than date and server.

async function fetchWithBackoff(url, tries = 5) {
  for (let i = 0; i < tries; i++) {
    const res = await fetch(url);
    if (res.status !== 429) return res;
    await new Promise((r) => setTimeout(r, 1000 * 2 ** i)); // 1s, 2s, 4s, 8s...
  }
  throw new Error(`still throttled after ${tries} tries: ${url}`);
}

Enrichment is the opposite: batched, and indifferent to backoff.

const chunk = (arr, n) =>
  Array.from({ length: Math.ceil(arr.length / n) }, (_, i) => arr.slice(i * n, i * n + n));

async function enrich(nmIds, batchSize = 50) {
  const batches = chunk(nmIds, batchSize);
  const responses = await Promise.all(
    batches.map((ids) =>
      fetch(
        `https://card.wb.ru/cards/v4/detail?appType=1&curr=rub&dest=${DEST}&spp=30&nm=${ids.join(';')}`,
      ).then((r) => r.json()),
    ),
  );

  // Join back by id — see the failure modes below for why this matters.
  const byId = new Map();
  for (const { products = [] } of responses) {
    for (const p of products) byId.set(p.id, p);
  }
  return byId;
}

Two hundred products, four requests, no throttling. Compare that to two hundred detail-page loads through a browser.

What this actually buys you

Bandwidth and compute. A browser fetching a product page downloads the HTML, the JS bundle, the CSS, the fonts and the images, executes the JS, and holds a gigabyte-class process open while doing it. The equivalent JSON call transfers a few kilobytes into JSON.parse. As a planning estimate I use one to two orders of magnitude difference in bytes per item — I have not run a controlled benchmark, and I'd rather say that plainly than quote a number I can't defend. The direction, though, is not in doubt, and on any platform that bills by memory × time it lands directly on your margin.

Proxy spend. This is the concrete one. If only discovery is throttled, only discovery needs premium IPs. Enrichment — usually the large majority of your requests — runs from wherever you like. Proxy bills tend to be the single biggest line item in a scraping operation, and this cuts the traffic that needs premium IPs to a small fraction.

Reliability. Fewer moving parts fail less often. No headless browser to crash, no selectors to break on a redesign, no JS execution to time out.

Four failure modes worth knowing about

Batched endpoints drop items silently. Wildberries' card endpoint accepts many ids separated by ; and returns only the ones it feels like — a batch of 3 came back with 2, and a batch of 21 came back with 14, no error and no indication of which went missing. Unavailable or invalid ids just evaporate. Never assume products.length === ids.length; join by id, and treat absences as data.

Unpinned parameters silently corrupt your time series. Wildberries takes a dest (warehouse/region) parameter that affects delivery estimates and, on the product I measured, the price: on 2026-07-15 the same item came back 1,719 ₽ cheaper under one dest than another, with basic unchanged and the delivery estimate moving from 18 to 27. One product on one day is not a law about their pricing, but it is more than enough to justify the rule — pin it once for the whole run. Rotate it and you will "detect" price changes that are really just geography. Every marketplace has a parameter like this.

Fields vanish rather than going null. Wildberries expresses "unavailable" by omission: reaching into sizes[0].price.product assumes a shape the API never promised, and the batching behaviour above is the same habit in a different place. Find the entry that has what you need rather than indexing blindly, and treat "no value" as a valid state rather than an error.

Response shapes drift. This is the one that turns a working scraper into a silently wrong one. Validate what you parse against a schema — zod, ajv, whatever — and fail loudly. A parse error at 3am is a maintenance ticket; a scraper quietly writing undefined into a client's price feed for a week is a different kind of problem.

And a soft block can arrive dressed as success. Alongside its 429s, Wildberries' search host sometimes answers HTTP 200 with a different shape — {state, version, params, data: {products}} instead of {metadata, products, total}, carrying unrelated content. Status code fine, JSON fine, content wrong. A parser looking for a top-level products key finds nothing and reports zero results, which your pagination reads as "the result set ended" — a truncated dataset that looks like a clean run.

This is why the schema check and the termination condition have to be separate concerns: "the response is not what I expected" must be a retryable failure, never an empty result.

The measurement that humbled me

Getting that distinction right moved one of my scrapers from 91.2% to 98.75% task success over 50 live runs. I was ready to call that the end of the story. It wasn't.

Same workload, same single datacenter IP, no proxies, four runs:

#	Settings	Success rate	Failures	p50 latency
1	250 ms min delay, no retries	91.2%	2 × 429, 4 × substituted response	473 ms
2	500 ms min delay + 2 retries (1.5 s / 3 s)	98.75%	1 × substituted response	471 ms
3	same settings, later the same day	90.14%	7 × substituted, zero 429	1700 ms
4	same settings, next day after ~12 h idle	77.94%	13 × 429, 2 × substituted; 15/50 runs empty	7532 ms

Runs 3 and 4 refuted two conclusions I had already drawn from runs 1 and 2:

• "The failures are pacing artefacts, fixable with delay and retries." Refuted by run 3 — identical settings, eight percentage points lower, latency already tripling.
• "The penalty decays if you let the IP rest." Refuted by run 4 — an overnight idle produced not recovery but the worst result of the four, with mass 429s back and a p50 sixteen times the fresh figure.

Caveats, stated plainly: one IP, four measurements, different times of day. I cannot fully separate accumulated penalty from time-of-day load. But monotonic degradation through a rest period is hard to explain any other way.

What this changes about the architecture is small but important. The two-tier split is still right — three of four hosts never throttled me at any point, so IP rotation applies to a minority of requests. What changed is that rotation on the discovery tier moved from "optimization when you scale" to "precondition for a reliability number you can quote." And a meta-lesson worth more than the architecture: once your IP is penalised, your benchmark measures the depth of your own penalty rather than the target's behaviour, and every additional run makes it worse. At that point the correct move is to stop measuring, not to keep collecting numbers.

When the pattern inverts

The two-tier split assumes discovery is the expensive half. On Avito — Russia's dominant classifieds site, and the country's largest real-estate portal — it is the other way round, and the reason is worth walking through because it starts with robots.txt rather than with a rate limit.

Avito's perimeter is Qrator, and from a datacenter IP it is absolute: the front page returned 429 and the mobile JSON API returned 403, both serving the same captcha page, on the very first request. Realistic TLS and header ordering — the exact trick that turns an anti-bot page into JSON on Wildberries — does not help here. Same firewall page either way. The binding constraint is IP reputation and geography, not request shape.

But the more consequential finding was in robots.txt. Under User-agent: * it disallows /api/ — which covers the mobile JSON API that every tutorial recommends as the cheap path — along with the parameterised search patterns (/*price=, /*pmin=, /*s= and friends) that a filter-driven scraper would depend on. That is not a grey area; it is the target publishing its rules. Both of the obvious designs were out.

What the same file does publish is a sitemap index. And those sitemaps download fine from a datacenter IP, straight past the firewall — because they exist precisely to be crawled. The item maps hold 50,000 URLs each with lastmod timestamps; one property section alone runs to eleven files.

So the tiers invert:

• Discovery is cheap, open and sanctioned — sitemaps, no proxy, no account.
• Enrichment is the expensive half — individual listing pages sit behind the firewall and need residential IPs from the right country.

Built that way, discovery measured 100% success across 50 live runs with no proxy at all (zero failures, zero retries, p50 1.8 s). That number is only impressive with its asterisk attached: it measures the half of the problem that is genuinely easy, and I am reporting it as such.

Two honest consequences that a tutorial would skip. First, sitemaps have no filters or sorting, so "listings under X in district Y" is not something this tier can do — you get inventory and freshness, then filter your own copy. Second, the numeric fields buyers actually want (price, area, floor) are not in a sitemap at all, and guessing them from URL slugs is a trap: a slug fragment like 556_m is ambiguous between 55.6 m² and 556 m². A missing field beats an invented one.

One implementation detail that cost me time: those .xml.gz files arrive as content-type: application/x-gzip without a content-encoding header. No HTTP client will auto-decompress that. It is a file that happens to be gzipped, not a gzipped response, and you have to gunzip it yourself.

When it doesn't apply at all

Sometimes the bottleneck is neither tier. Lazada's review endpoint is a clean H5/BFF JSON API — no token, no signature, no cookies required. The problem is throughput.

From a datacenter IP it sustains roughly one request every five minutes. That is not a figure of speech: three requests spaced ~5 minutes apart came back clean, and a single request sent 8 seconds after the previous one was punished immediately. The punishment arrives as HTTP 200 with an HTML body — a redirect script to a slider-captcha challenge — which is a nastier failure than a 403, because naive code stores it as data.

Three things I expected to help, which didn't:

• Resting the IP. Fifteen hours of idle bought exactly one clean request, not a burst. The budget does not accumulate.
• Rotating regional domains. The flag is shared across Lazada's country domains; a punish on one carries to the next on the first request.
• Asking for more per request. A widely-copied GitHub snippet suggests pageSize=999999 to grab everything at once. It returns success: true with an empty model — the request is rejected, not fulfilled, and code that trusts it records "no reviews" for a product that has plenty.

There is also a ceiling that no architecture can lift. The endpoint's own paging.totalPages is wrong by roughly a factor of three: on the product I measured it advertised 25 pages of 1,243 reviews, and data ran out on page 8. The real ceiling is rateCount − hiddenCount — 1,243 minus 864 hidden, so 379 reviews, which predicted the final short page to the exact item. About 30% of that product's reviews are reachable at all. If you are selling review data, that number belongs in your documentation rather than in a support ticket six weeks later.

The tiering idea still helps here — you still want your scarce resource spent on discovery — but the scarce resource is clean IPs and solved challenges, and no amount of restructuring changes the arithmetic. One product's 379 available reviews is 8 requests, which at datacenter pace is roughly 40 minutes. That is a verdict on the transport, not a design to optimise.

The short version

Measure per host, not per site. Read robots.txt before you design, not after. Put the throttled work in one tier and everything else in another, then spend your proxy budget only on the first. Validate response shape separately from your termination condition, because the ugliest failures arrive with a 200.

And re-measure. My best number was 98.75%, my most recent is 77.94%, and the code did not change between them. The number that matters is the one you can still reproduce today.

I write these up as I go, measurements and refutations included — if you have numbers on any of the open questions above, particularly how long that search-host penalty actually lasts, I'd like to compare notes.