DEV Community: ActorForge

What AI agents actually see when they search for your Apify Actor

ActorForge — Thu, 03 Sep 2026 07:09:02 +0000

Apify has spent this year making the phrase "your Actor as a tool for AI agents" concrete: a hosted Model Context Protocol (MCP) server, per-event billing an agent can actually pay, even crypto payment rails that don't require an Apify account on the caller's side. The pitch writes itself: build a good Actor, describe it well, and a robot with a budget will find you.

What I could not find anywhere was numbers. When an agent goes looking for a tool, what does it actually see? Does it see the same ranking a human sees in Apify Store, or something else? Is there — as I quietly hoped — a side door for new Actors that haven't accumulated users yet?

I had a good reason to want that side door. Between July 29 and August 1 I published three pay-per-event (PPE) Actors under a brand-new developer account: scrapers for Wildberries products and reviews, Avito real estate listings, and Lazada reviews — big marketplaces, quiet corners of the Store. New account, zero reviews, user counts I can list from memory. A textbook cold start.

So on August 6 I measured it. I connected to the Apify MCP server the same way any agent does, ran the searches an agent would run, and compared the results against the human-facing Store search from the same day. This article is the method, the numbers, and the two traps that produced convincingly wrong results before the real ones — one of them had me believing my Actors were invisible to agents entirely.

How an agent reaches your Actor

Before the measurements, a quick map of the pipeline, because it determines what's worth optimizing.

An agent (or its harness — Claude Desktop, an SDK loop, whatever) connects to https://mcp.apify.com, the hosted MCP server. Transport is Streamable HTTP; auth is either OAuth or a plain API token header. The minimal client config is:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

The default toolset exposes, among others, three tools that matter for discovery:

search-actors — keyword search over the Store,
fetch-actor-details — the full card of one Actor: description, pricing, stats, input schema, README,
call-actor — runs an Actor by name with a JSON input, which is validated against your input schema at execution time.

The intended flow is exactly the one you'd guess: search → fetch details → call. Alternatively, a specific Actor can be mounted as a named tool by adding ?tools=username/name to the server URL, in which case the MCP server reads its input schema and generates a dedicated tool from it.

Who is in the searchable pool? Not everyone. The server returns only free and pay-per-event Actors to agentic callers — rental-model Actors are excluded — and it drops Actors that fail platform safety checks. Apify's monetization docs add two more conditions for an Actor to be usable by agentic (and crypto-paying) callers: it must run with limited permissions, and it must not be a Standby-mode Actor. There is no opt-in: a pay-per-event Actor with limited permissions and no Standby is in automatically. As of late July 2026, a bit over 29,000 Actors in the Store passed that bar (counted via the public Store API with the agentic-users filter). Mine are among them, which is what made the next question interesting.

The experiment: searching for my own Actors the way an agent would

I wanted the rawest possible view — no SDK, no client-side magic — so I spoke JSON-RPC to the server directly with curl. Three steps; the only prerequisite is your Apify API token exported as APIFY_TOKEN.

Step 1: initialize and capture the session. The session ID comes back as a response header (-D headers.txt below saves those), and everything after initialize must carry it:

S=$(curl -s -D headers.txt -X POST "https://mcp.apify.com/" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}' \
  > /dev/null; grep -i "^mcp-session-id:" headers.txt | tr -d '\r' | awk '{print $2}')

A small thing that cost me an attempt: grep for ^mcp-session-id: with the anchor. Without it, the first match is the access-control-expose-headers line, which merely mentions Mcp-Session-Id, and you spend a confused minute sending a CORS header list as your session ID.

Step 2: call the search tool. Responses arrive as a Server-Sent Events (SSE) stream; the payload is the last data: line:

curl -s -X POST "https://mcp.apify.com/" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" -H "mcp-session-id: $S" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search-actors","arguments":{"keywords":"Lazada reviews","limit":10,"offset":0}}}'

which comes back looking like this (trimmed):

event: message
data: {"result":{"content":[{"type":"text","text":"# Search results:\n- **Search query:** Lazada reviews\n- **Number of Actors found:** 9\n\n# Actors:\n\n## [Lazada Review Scraper](https://apify.com/hello.datawizards/lazada-review-scraper) ..."}]},"jsonrpc":"2.0","id":3}

Step 3: page through and extract positions. limit is capped at 10, so depth comes from offset. With TOKEN and SESSION set to the values captured in step 1 (paste them in), the probe I actually ran (per query, down to position 50):

import json, re, subprocess

def search(keywords, offset):
    body = json.dumps({"jsonrpc": "2.0", "id": 3, "method": "tools/call",
        "params": {"name": "search-actors",
                   "arguments": {"keywords": keywords, "limit": 10, "offset": offset}}})
    out = subprocess.run(["curl", "-s", "-X", "POST", "https://mcp.apify.com/",
        "-H", f"Authorization: Bearer {TOKEN}", "-H", "Content-Type: application/json",
        "-H", "Accept: application/json, text/event-stream",
        "-H", f"mcp-session-id: {SESSION}", "-d", body],
        capture_output=True, text=True).stdout
    data = [l for l in out.splitlines() if l.startswith("data: ")][-1]
    return json.loads(data[6:])["result"]["content"][0]["text"]

ids = []
for off in (0, 10, 20, 30, 40):
    ids += re.findall(r'https://apify\.com/([a-zA-Z0-9_.-]+/[a-z0-9-]+)', search("Lazada reviews", off))
positions = list(dict.fromkeys(ids))   # dedupe, keep order

That's the whole rig. Except my first two attempts produced garbage, and both failure modes are worth your time, because agents will hit them too.

Trap #1: the argument is `keywords`, and typos are silent

My first pass called the tool with {"search": "avito real estate", "limit": 10} — search felt like the obvious argument name for a search tool. The server didn't complain. It returned a perfectly plausible list of ten popular Actors. So did the next query. And the next.

The tell — which I only spotted the second time it happened — was that the listing was byte-identical for every query. Google Maps scraper, TikTok scraper, Instagram scraper, Google Search scraper, Website Content Crawler… for "avito real estate", for "lazada reviews", for anything. My Actors appeared in none of them, which read exactly like "your Actors are invisible to agents."

Two deliberately different queries, wrong argument name — one identical answer. This is what "your search never executed" looks like.

What actually happens: the tool's input schema declares no required arguments —

{ "properties": {
    "keywords": { "type": "string", "default": "" },
    "limit":    { "type": "integer", "default": 5, "maximum": 10 },
    "offset":   { "type": "integer", "default": 0 } },
  "required": [] }

— so an unknown argument is silently ignored, keywords falls back to "", and empty keywords are documented to return the Store's default sort order, which is popularity. (To be fair, silently dropping unknown properties is stock JSON-Schema behavior, not an Apify quirk — which is exactly why it's worth a warning.) You don't get an error. You get the most-used Actors on the platform, confidently, for any question you ask.

I re-verified it on the day of writing: two deliberately different queries with the wrong argument name returned identical listings, my Actors absent from both; the same queries with keywords returned distinct, correct results.

Same session, correct argument name: a real ranking, with my Actor at #4.

Two lessons. When you're the one measuring: always send a control query — if two obviously different searches return the same list, your search never executed. As a tool author: this is what LLM-facing APIs are like now. A tool that silently substitutes a default for a misspelled argument will feed an agent wrong-but-plausible data, and the agent will act on it. If you build MCP tools of your own, make unknown arguments loud.

Trap #2: the output is Markdown — parse the links, not the prose

search-actors returns Markdown, not JSON: a heading, a bullet with your query, then one section per Actor:

# Search results:
- **Search query:** Lazada reviews
- **Number of Actors found:** 9

# Actors:

## [Lazada Review Scraper](https://apify.com/hello.datawizards/lazada-review-scraper) (`hello.datawizards/lazada-review-scraper`)
- **URL:** https://apify.com/hello.datawizards/lazada-review-scraper
- **Description:** Lazada reviews Scraper Pro extracts detailed product reviews...

My first parser pulled anything shaped like owner/name out of that text. It came back with results like INN/OGRN, pros/cons, JSON/CSV, surface/rooms and 10/10 — README fragments and slashed phrases from Actor descriptions, interleaved with real slugs.

That's not just noise; it shifts positions. With the naive regex, one of my Actors showed at #8 instead of its real #7, another at #3 instead of #2, and a third dropped out of the top 10 entirely. If I'd stopped there, I'd have published wrong numbers that were only slightly wrong — the worst kind.

The reliable anchor is the canonical URL that every listing carries:

re.findall(r'https://apify\.com/([a-zA-Z0-9_.-]+/[a-z0-9-]+)', txt)

with order-preserving dedup afterwards, because each listing block repeats the Actor's URL up to three times (title link, URL field, pricing link).

One more parsing footnote: "Number of Actors found" is the page's own count, not a total. Unlike the public GET /v2/store API, the MCP channel never tells you how many results exist. An agent can't ask "how crowded is this niche" — and neither could I, through this channel.

The result: agent search is a mirror, not a side door

With the rig fixed, here is what an agent sees for seven queries relevant to my three Actors (positions in search-actors output, probed to depth 50, August 6, server 0.14.2):

`keywords`	position	which Actor
`Avito real estate`	#2	Avito real estate
`Avito property`	#2	Avito real estate
`Lazada reviews`	#4	Lazada reviews
`Russian marketplace`	#4	Wildberries
`Wildberries reviews`	#7	Wildberries
`Wildberries products`	#9	Wildberries
`Wildberries`	#20	Wildberries

And here is the same day's human-side Store search — the default, filters-on view a visitor gets, which I measured through the public GET /v2/store API and have spot-checked by eye on the website:

query	web Store search	agent (`search-actors`)
`avito real estate`	#2	#2
`lazada reviews`	#4	#4
`wildberries reviews`	#7	#7
`wildberries`	#20	#20

Four exact matches, including the broadest term. The Actors above me were the same incumbents I compete with in the web Store, seen through a different pipe.

The human side of the mirror, same day: Store search for "avito real estate" — the incumbent at #1 (20 users), my Actor at #2 (2 users).

After measuring this from the outside, I found the receipts on the inside: the MCP server is open source, and search_actors.ts simply calls the same public Store search API, passing your keywords through as the search string. The MCP layer adds formatting, not ranking. Apify's documentation on Store search ranking states it plainly: search ranking evaluates parameters similar to the Actor quality score, and the two correlate strongly in Apify Store search and the MCP search-actors tool.

A short-lived curiosity from six days earlier: on July 30 — when only the first two of my Actors were published and my account's identity verification was still pending — the default web Store search excluded them entirely, while the agent channel already listed them at #2 and #4. For those first couple of days, agents were the only searchers who could find my Actors at all. That divergence has since converged: verification completed, the web results caught up, and both the docs and the server source describe safety filtering on the agentic side too. Ranking was identical all along; treat the episode as trivia.

The strategic conclusion is the honest one, and it's the opposite of what I hoped: there is no separate agent-SEO game and no cold-start bypass. An agent searching for "lazada reviews" sees the same #1 and #2 a human sees. Everything you do for your Store position transfers to the agent channel automatically — and nothing extra is available there. The queue is the queue.

What the agent reads once it finds you

Position gets you into the candidate set. The pick happens on the card. Here's what fetch-actor-details returned for my Wildberries Actor (trimmed — the pricing and stats lines are verbatim):

Wildberries Scraper — Products & Reviews (`actorforgehq/wildberries-scraper`)
- **Pricing:** This Actor is paid per event:
  - **Actor Start**: Charged when the Actor starts running. Number of events
    charged depends on Actor memory (one event per GB, minimum one event).
    ($0.02 per event)
  - **Product scraped**: One Wildberries product record: name, brand, seller,
    region-pinned price before and after discount... ($0.0035 per event)
  - **Review scraped**: One Wildberries review: rating, text, author and date,
    newest first. ($0.002 per event)
- **Stats:** 3 total users, 2 monthly users
- **Developed by:** [actorforgehq](https://apify.com/actorforgehq) (community)
- **Last modified:** 2026-08-06T04:17:14.001Z

The same card as a human sees it. The agent gets the Markdown version — same title, same prices, same user counts.

Three things in that card changed how I think about Actor metadata.

Your user counts are in the agent's context window. "3 total users, 2 monthly users" — the agent sees my cold start as plainly as any human browsing the Store. Social proof reaches agents too: it's right there in the tool output, next to the price. If you assumed agents would judge tools purely on descriptions and schemas: no, the popularity signal ships with the card.

Your PPE event names and descriptions are your pricing page. The event titles and descriptions I wrote in the monetization console are rendered verbatim to the agent as the explanation of what its money buys. Write them as product copy, not as internal event IDs. And mind the grammar of your titles: the Store's price line pluralizes the event title mechanically, so mine renders as "from $3.50 / 1,000 product scrapeds" — the top Actors in my niches sidestep this by naming events as plain nouns: Product, Review, Listing.

Event titles become customer-facing copy — including the mechanically pluralized price line.

Memory is a pricing parameter. Note the fine print on Actor Start: one event per GB, minimum one event. My Actors default to 1,024 MB, so a start bills exactly one event — $0.02 on this Actor. If I raised default memory to 4 GB, every start would silently bill four events without a single price field changing. Price increases on Apify come with a built-in delay before they take effect; a memory bump does not. I now treat defaultRunOptions.memoryMbytes as a line item on the price list, reviewed with the same care as the prices themselves.

Closing the loop: letting an agent run it

Discovery and reading are two-thirds of the story. The third tool is call-actor, so I finished the session the way an agent would — by running my own Lazada Actor through the MCP server, with a budget cap:

{ "name": "call-actor", "arguments": {
    "actor": "actorforgehq/lazada-reviews-scraper",
    "input": { "productUrls": ["https://www.lazada.vn/products/x-i246452966.html"],
               "maxReviews": 3 },
    "waitSecs": 45,
    "callOptions": { "maxTotalChargeUsd": 0.05 } } }

The input is passed through as-is and validated against the Actor's input schema at execution time. waitSecs is capped at 45; a longer run would be collected asynchronously via get-actor-run. Mine came back inside the window:

{ "runId": "v230oDUAehwnWsJSt", "status": "SUCCEEDED",
  "stats": { "runTimeSecs": 9.231 },
  "summary": "SUCCEEDED in 9.231s. 4 items; 39 fields available.",
  "nextStep": "Use get-dataset-items with datasetId=TEXypQcxLYA8sOVBt ..." }

Two details here earn their keep. The response enumerates the dataset's typed fields — the metadata from my dataset schema surfacing exactly where an agent needs it. And nextStep literally coaches the caller through the follow-up, field projection included. get-dataset-items then returned one product record and three Vietnamese reviews with verifiedPurchase flags, projected to just the fields I asked for.

The bill, straight from the run record:

{ "chargedEventCounts": { "apify-actor-start": 1, "product-scraped": 1, "review-scraped": 3 } }

One start ($0.01 on this Actor), one product ($0.002), three reviews (3 × $0.006) — three cents, itemized, against a five-cent cap.

The whole promise of "your Actor as a tool for AI agents" in one round trip: found by keyword, priced per event, run with a budget, billed itemized.

Making your Actor a good tool: what the measurements say actually matters

Given that ranking is shared with the Store and the card is what the agent reads, "my Actor as a tool for AI agents" decomposes into concrete, boring, checkable work. This is my list, each item traceable to something above.

1. Win the text match in the same fields humans search

The search — both channels — runs over title, name, description, username and README. In a web-search sweep I ran in mid-July, everything above roughly position 24 matched the query in its title; README-only matches started behind them. Exact phrases beat popularity: on "avito real estate scraper" that day, an Actor with 15 users outranked one with 608 because the phrase sits in its title. My Actor titles carry the platform and the data type as plain nouns ("Wildberries Scraper — Products & Reviews") for the same reason. It's unglamorous SEO, and the mirror means it's the agent-facing work too.

2. Write input schema descriptions as arguments, not labels

The MCP server converts your input schema into the tool schema the agent reasons over — truncating each property description at 500 characters, enum lists at 2,000. That budget is your entire opportunity to explain a parameter to the caller. In my Lazada Actor, the sort field doesn't say "Sort order (0–3)"; it says why the default is newest first: on a product I measured, Lazada's own "relevance" order put fifty 5-star reviews on page one and pushed every 1–2-star review to the very end — an agent sampling "relevant" reviews would hand its user a positively biased dataset without knowing it. A human skims that; an agent uses it to choose parameters correctly.

3. Fill the dataset schema — field metadata is how agents read your output

Apify's dataset schema docs are direct about this: agents interacting with Actors through MCP "rely on field metadata to understand the data in your dataset," and without it they "must infer field meanings from names alone, which leads to errors." Every field in my dataset schemas has a type, title, description and example — those are the "39 fields available" the call-actor response advertised. And where an Actor mixes row kinds — my Wildberries and Lazada Actors emit both products and reviews — rows carry an explicit discriminator ("type": "product" | "review"), so an agent, or the next Actor in a chain, doesn't have to guess which rows are which.

4. Replace the default example input

A freshly created Actor's public definition ships with exampleRunInput of {"helloWorld": 123}. That junk is part of what agent tooling can read about your Actor. Mine now carry a realistic minimal input (a real search query, a real product URL) — a two-minute API call that removes one entire class of failed first runs.

5. Be runnable and billable by a machine

The pool filter above doubles as a checklist: pay-per-event pricing, limited permissions, no Standby mode — that's what makes an Actor available to agentic callers at all (rental Actors are invisible to this entire channel), and the same three conditions gate the agentic payment rails, like the x402 protocol that landed in June 2026: USDC payments per call, no Apify account on the caller's side. None of it requires opting in; it's simply the default posture of a modern Actor.

6. Respect the caller's budget — it's an API now

An agent can cap a run with maxTotalChargeUsd — my demo run above did. On the Actor side, Actor.charge() does not throw when that cap is reached: it reports eventChargeLimitReached: true in its return value, and the SDK silently stops charging and pushing further items. The platform won't police the cap for you — Apify's pay-per-event guide puts the stop logic in your Actor's hands: check the charge result and end the run yourself. If your code ignores the return value, you'll keep scraping at full speed, for free, into the void. My pipeline checks it and shuts the run down cleanly:

const res = await Actor.charge({ eventName: 'review-scraped', count: batch.length });
if (res.eventChargeLimitReached) {
  log.info('Charge limit reached — stopping gracefully');
  break; // stop fetching; the caller got exactly what they paid for
}

Budget-capped callers are normal callers now. Design for the cap being hit on a good run.

Takeaways

The agent channel is a mirror of Store search — same index, same ranking, confirmed both by measurement (positions matched one-for-one on the same day) and by the server's source code. There is no cold-start side door; your Store position is your agent position.
Silent argument defaults are the measurement hazard. A wrong argument name gave me a confident, popularity-sorted answer to a question I never asked — twice, days apart. Send control queries; distrust identical results for different questions.
Parse anchors, not prose. Position numbers extracted by casual regex from Markdown were off by one to two places — and in one case by enough to knock an Actor out of the visible top 10.
The card is the pitch. Event names, event descriptions, user counts and last-modified date all land in the agent's context. Write every string as if the buyer will read it, because the buyer's agent will.
Memory is a price. One start event per GB means your default memory setting multiplies what every caller pays before the first item arrives.

I went looking for a side door for cold-start Actors and found a mirror instead. I'd rather know. The work that remains is the same work it always was — be findable for the phrase, be legible on the card, be honest per event — except now half the readers parse it with a token budget, and they don't skim.

If you publish Actors, I'd genuinely like to compare notes: have you measured what agents see for your Actors — and did you find a query where the two channels disagree by more than a couple of positions?

Core measurements: August 6, 2026, mcp.apify.com server 0.14.2, probed to position 50 per query and re-verified the same day; supporting web-search sweeps July 20 – August 6, 2026. Positions drift daily — I've watched a term move three places overnight — so expect your numbers to differ. The three Actors involved: Wildberries Scraper, Avito Real Estate Scraper, Lazada Reviews Scraper.

4 ready-to-run recipes for pulling Reddit data as JSON (no API key, no browser)

ActorForge — Thu, 20 Aug 2026 02:29:47 +0000

Getting structured data out of Reddit keeps getting harder: the public .json endpoints were shut down in May 2026, Wayback snapshots stopped in 2025, and official API access now sits behind pre-approval. If you just need a few thousand posts or comments as clean JSON — for market research, social listening, or an AI dataset — the remaining options are mostly heavyweight browser scrapers.

I maintain Reddit Data Scraper, an Apify Actor that takes a different route: plain HTTP requests against Reddit's HTML, no headless browser, no login, no API key. That keeps runs fast and cheap, with a couple of honest trade-offs I'll list at the end.

Below are the four jobs people actually use it for. Each link opens a pre-configured example you can run as-is or tweak.

1. Track every mention of a keyword

Reddit keyword monitoring — search all of Reddit for a phrase (a brand, a competitor, a technology) and export the matching posts: title, subreddit, author, score, comment count, permalink, UTC timestamp. The time filter is relative (week, month), so a scheduled run always returns the fresh slice.

2. Export a full comment thread

Reddit comment thread export — feed it one post URL and get every comment with nesting depth and parent id, enough to rebuild the tree offline or run sentiment analysis on the whole discussion.

3. Top posts of a subreddit

Subreddit top posts — the classic market-research pull: top posts of any subreddit for the past week, with scores, flairs, comment counts and links.

4. A user's public history

Reddit user history — one account's public posts and comments in a single run, NSFW filtered out by default.

The honest trade-offs

Public data only. No logins, no private or quarantined communities.
No CAPTCHA or JS-challenge solving — by design. If Reddit blocks a request, the Actor retries within documented rate limits and then fails loudly instead of pretending the page was empty. You get an error, not silently missing rows.
Documented rate-limit etiquette. The request pacing is in the README, not hidden.

Pricing is pay-per-result ($1.90 per 1,000 posts, $0.75 per 1,000 comments), so an empty or failed run costs nothing.

If one of the recipes almost fits but not quite, open an issue on the Actor page — input schema changes are cheap.

— Oleg Naryzhnykh

Expensive Discovery, Cheap Enrichment: A Marketplace Scraper Architecture Built Around Rate Limits

ActorForge — Fri, 24 Jul 2026 11:28:27 +0000

Most marketplace scrapers are shaped like a human browsing session: open the listing page, read the cards, click into each product, read the detail page, go back, next page. Whether it's Playwright or a stack of HTTP calls, the control flow mirrors what a person would do.

That shape is intuitive and it scales badly, for a reason that has nothing to do with your code.

The measurement that changes the design

Before writing a scraper, I now spend thirty minutes on one question: which hosts does this site's data actually come from, and which of them are rate limited?

Not "is the site rate limited." Which hosts.

Here is what that looked like on Wildberries, one of the largest marketplaces in Russia. Its front end is served by four distinct services. I hit each one with concurrent requests from a single datacenter IP:

Host	What it serves	Test	Result
`search.wb.ru`	search / discovery	50 parallel	30 × 200, 20 × 429
`card.wb.ru`	product detail, price, stock	30 parallel	30 × 200
`feedbacks1.wb.ru`	reviews	30 parallel	30 × 200
`basket-NN.wbbasket.ru`	attributes, media (CDN)	40 parallel	40 × 200

Sequentially, search was just as unfriendly: 30 requests spread over 69 seconds returned 13 × 200 and 17 × 429. A cold first request can come back 429 before you have sent anything else.

Three of the four services did not throttle me at all.

The limit is per service, and only one service is expensive. That is not a Wildberries quirk; it is the normal consequence of how large marketplaces are built. Search is a ranking system running over a huge index — it is the costly thing to serve and the thing competitors scrape, so it sits behind a gateway. Product cards, reviews and static media are cache-friendly, often on a CDN, and defended lightly or not at all.

Once you know that, the browsing-session shape is obviously wrong. It spends the scarce resource on work the abundant resources would have done for free.

The pattern: two tiers

Split the scraper into two tiers with different economics.

Tier 1 — discovery (scarce). The only job here is to turn a query into a list of identifiers. Nothing else. Paginate, collect ids, stop. This tier is rate limited, so it is serial, backed off, and the only place where proxies are worth paying for.

Tier 2 — enrichment (abundant). Everything else: prices, stock, reviews, attributes, images. Driven entirely by the ids from tier 1, batched, parallel, usually straight from your own IP.

On Wildberries this maps cleanly. Search returns 100 products per page, and each product carries two identifiers: id (the SKU-level nmId) and root (the parent card imtId). Every enrichment endpoint keys off one of those. One throttled request buys 100 items' worth of cheap work.

The ratio is the whole point. If a discovery call yields 100 ids, and enrichment for those 100 ids costs you 2 batched card requests plus 100 unthrottled review requests, then the overwhelming majority of your traffic is on endpoints that don't fight you. Getting more data no longer means hitting the wall more often.

Finding the tiers on a new target

The recipe is short and works on any site with a JSON-backed front end:

Read robots.txt first. This is not a formality — it is an architectural input, and it can delete an entire design before you write a line of it. I have a worked example below where it did exactly that.
Open the site with devtools on the network tab, filter to XHR/Fetch. Note the distinct hostnames. Most marketplaces use between two and five.
Replay each interesting request and strip cookies, tokens and headers one at a time to see what is genuinely required. You often find that nothing was. But do not do this with plain curl or Node's built-in fetch, or you will get false negatives. Some gateways fingerprint the TLS handshake and header ordering, not just the User-Agent. On Wildberries' search host, fetch() with default headers returns an HTML anti-bot page while got-scraping returns JSON — same URL, same IP, seconds apart. Probe with a client that emulates a real browser's TLS profile (got-scraping, curl-impersonate), otherwise you will write off endpoints that work fine.
Probe each host separately for its tolerance.

Step 4 is the one people skip. It is about ten lines:

async function probe(url, n = 10) {
  const t0 = Date.now();
  const codes = await Promise.all(
    Array.from({ length: n }, () =>
      fetch(url).then((r) => r.status).catch(() => 0),
    ),
  );
  const tally = {};
  for (const c of codes) tally[c] = (tally[c] ?? 0) + 1;
  return { host: new URL(url).host, n, ms: Date.now() - t0, tally };
}

Run it once per host, with a small n, and stop as soon as you have your answer. You are measuring someone else's infrastructure — a handful of requests tells you what you need, and there is no version of this where hammering a target is the right call. On the targets where I got a clear "no" on the first probe, I stopped there rather than escalating.

Then write the numbers down. They are the input to your design, and they go stale — treat them as dated facts, not permanent truths. The last section of this post is about what happened when I re-measured mine.

What the implementation looks like

Once the tiers are explicit, the code gets simpler, not more complex. Discovery is a generator that yields ids:

const DEST = '-1257786';           // pin region-ish params for the whole run
const PAGE_SIZE = 100;

async function* discover(query, maxPages = 5) {
  let firstIdOfPageOne;

  for (let page = 1; page <= maxPages; page++) {
    const url =
      'https://search.wb.ru/exactmatch/ru/common/v4/search' +
      `?query=${encodeURIComponent(query)}&resultset=catalog` +
      `&dest=${DEST}&curr=rub&spp=30&appType=1&lang=ru&page=${page}`;

    const res = await fetchWithBackoff(url);
    const { products = [] } = await res.json();
    if (products.length === 0) return;

    // This host never serves an empty page at the end of a result set — it starts
    // serving page 1 again. Terminating on "empty" alone yields silent duplicates.
    if (page === 1) firstIdOfPageOne = products[0].id;
    else if (products[0].id === firstIdOfPageOne) return;

    for (const p of products) yield { nmId: p.id, imtId: p.root };
    if (products.length < PAGE_SIZE) return;
  }
}

That duplicate check is not paranoia. Measured on 2026-07-18, page 60 of a ноутбук search returned page 1's exact contents — full 100 products, HTTP 200, no marker. A loop that walks until it sees an empty page never terminates on this host; it just keeps re-emitting the first page and hammering the one service that is actually rate limited. Whatever your target, verify how it signals "no more results" before you trust a termination condition.

The backoff belongs to this tier only, and it has to be heuristic: the 429 response on this host is an HTML body with no Retry-After header, carrying little more than date and server.

async function fetchWithBackoff(url, tries = 5) {
  for (let i = 0; i < tries; i++) {
    const res = await fetch(url);
    if (res.status !== 429) return res;
    await new Promise((r) => setTimeout(r, 1000 * 2 ** i)); // 1s, 2s, 4s, 8s...
  }
  throw new Error(`still throttled after ${tries} tries: ${url}`);
}

Enrichment is the opposite: batched, and indifferent to backoff.

const chunk = (arr, n) =>
  Array.from({ length: Math.ceil(arr.length / n) }, (_, i) => arr.slice(i * n, i * n + n));

async function enrich(nmIds, batchSize = 50) {
  const batches = chunk(nmIds, batchSize);
  const responses = await Promise.all(
    batches.map((ids) =>
      fetch(
        `https://card.wb.ru/cards/v4/detail?appType=1&curr=rub&dest=${DEST}&spp=30&nm=${ids.join(';')}`,
      ).then((r) => r.json()),
    ),
  );

  // Join back by id — see the failure modes below for why this matters.
  const byId = new Map();
  for (const { products = [] } of responses) {
    for (const p of products) byId.set(p.id, p);
  }
  return byId;
}

Two hundred products, four requests, no throttling. Compare that to two hundred detail-page loads through a browser.

What this actually buys you

Bandwidth and compute. A browser fetching a product page downloads the HTML, the JS bundle, the CSS, the fonts and the images, executes the JS, and holds a gigabyte-class process open while doing it. The equivalent JSON call transfers a few kilobytes into JSON.parse. As a planning estimate I use one to two orders of magnitude difference in bytes per item — I have not run a controlled benchmark, and I'd rather say that plainly than quote a number I can't defend. The direction, though, is not in doubt, and on any platform that bills by memory × time it lands directly on your margin.

Proxy spend. This is the concrete one. If only discovery is throttled, only discovery needs premium IPs. Enrichment — usually the large majority of your requests — runs from wherever you like. Proxy bills tend to be the single biggest line item in a scraping operation, and this cuts the traffic that needs premium IPs to a small fraction.

Reliability. Fewer moving parts fail less often. No headless browser to crash, no selectors to break on a redesign, no JS execution to time out.

Four failure modes worth knowing about

Batched endpoints drop items silently. Wildberries' card endpoint accepts many ids separated by ; and returns only the ones it feels like — a batch of 3 came back with 2, and a batch of 21 came back with 14, no error and no indication of which went missing. Unavailable or invalid ids just evaporate. Never assume products.length === ids.length; join by id, and treat absences as data.

Unpinned parameters silently corrupt your time series. Wildberries takes a dest (warehouse/region) parameter that affects delivery estimates and, on the product I measured, the price: on 2026-07-15 the same item came back 1,719 ₽ cheaper under one dest than another, with basic unchanged and the delivery estimate moving from 18 to 27. One product on one day is not a law about their pricing, but it is more than enough to justify the rule — pin it once for the whole run. Rotate it and you will "detect" price changes that are really just geography. Every marketplace has a parameter like this.

Fields vanish rather than going null. Wildberries expresses "unavailable" by omission: reaching into sizes[0].price.product assumes a shape the API never promised, and the batching behaviour above is the same habit in a different place. Find the entry that has what you need rather than indexing blindly, and treat "no value" as a valid state rather than an error.

Response shapes drift. This is the one that turns a working scraper into a silently wrong one. Validate what you parse against a schema — zod, ajv, whatever — and fail loudly. A parse error at 3am is a maintenance ticket; a scraper quietly writing undefined into a client's price feed for a week is a different kind of problem.

And a soft block can arrive dressed as success. Alongside its 429s, Wildberries' search host sometimes answers HTTP 200 with a different shape — {state, version, params, data: {products}} instead of {metadata, products, total}, carrying unrelated content. Status code fine, JSON fine, content wrong. A parser looking for a top-level products key finds nothing and reports zero results, which your pagination reads as "the result set ended" — a truncated dataset that looks like a clean run.

This is why the schema check and the termination condition have to be separate concerns: "the response is not what I expected" must be a retryable failure, never an empty result.

The measurement that humbled me

Getting that distinction right moved one of my scrapers from 91.2% to 98.75% task success over 50 live runs. I was ready to call that the end of the story. It wasn't.

Same workload, same single datacenter IP, no proxies, four runs:

#	Settings	Success rate	Failures	p50 latency
1	250 ms min delay, no retries	91.2%	2 × 429, 4 × substituted response	473 ms
2	500 ms min delay + 2 retries (1.5 s / 3 s)	98.75%	1 × substituted response	471 ms
3	same settings, later the same day	90.14%	7 × substituted, zero 429	1700 ms
4	same settings, next day after ~12 h idle	77.94%	13 × 429, 2 × substituted; 15/50 runs empty	7532 ms

Runs 3 and 4 refuted two conclusions I had already drawn from runs 1 and 2:

"The failures are pacing artefacts, fixable with delay and retries." Refuted by run 3 — identical settings, eight percentage points lower, latency already tripling.
"The penalty decays if you let the IP rest." Refuted by run 4 — an overnight idle produced not recovery but the worst result of the four, with mass 429s back and a p50 sixteen times the fresh figure.

Caveats, stated plainly: one IP, four measurements, different times of day. I cannot fully separate accumulated penalty from time-of-day load. But monotonic degradation through a rest period is hard to explain any other way.

What this changes about the architecture is small but important. The two-tier split is still right — three of four hosts never throttled me at any point, so IP rotation applies to a minority of requests. What changed is that rotation on the discovery tier moved from "optimization when you scale" to "precondition for a reliability number you can quote." And a meta-lesson worth more than the architecture: once your IP is penalised, your benchmark measures the depth of your own penalty rather than the target's behaviour, and every additional run makes it worse. At that point the correct move is to stop measuring, not to keep collecting numbers.

When the pattern inverts

The two-tier split assumes discovery is the expensive half. On Avito — Russia's dominant classifieds site, and the country's largest real-estate portal — it is the other way round, and the reason is worth walking through because it starts with robots.txt rather than with a rate limit.

Avito's perimeter is Qrator, and from a datacenter IP it is absolute: the front page returned 429 and the mobile JSON API returned 403, both serving the same captcha page, on the very first request. Realistic TLS and header ordering — the exact trick that turns an anti-bot page into JSON on Wildberries — does not help here. Same firewall page either way. The binding constraint is IP reputation and geography, not request shape.

But the more consequential finding was in robots.txt. Under User-agent: * it disallows /api/ — which covers the mobile JSON API that every tutorial recommends as the cheap path — along with the parameterised search patterns (/*price=, /*pmin=, /*s= and friends) that a filter-driven scraper would depend on. That is not a grey area; it is the target publishing its rules. Both of the obvious designs were out.

What the same file does publish is a sitemap index. And those sitemaps download fine from a datacenter IP, straight past the firewall — because they exist precisely to be crawled. The item maps hold 50,000 URLs each with lastmod timestamps; one property section alone runs to eleven files.

So the tiers invert:

Discovery is cheap, open and sanctioned — sitemaps, no proxy, no account.
Enrichment is the expensive half — individual listing pages sit behind the firewall and need residential IPs from the right country.

Built that way, discovery measured 100% success across 50 live runs with no proxy at all (zero failures, zero retries, p50 1.8 s). That number is only impressive with its asterisk attached: it measures the half of the problem that is genuinely easy, and I am reporting it as such.

Two honest consequences that a tutorial would skip. First, sitemaps have no filters or sorting, so "listings under X in district Y" is not something this tier can do — you get inventory and freshness, then filter your own copy. Second, the numeric fields buyers actually want (price, area, floor) are not in a sitemap at all, and guessing them from URL slugs is a trap: a slug fragment like 556_m is ambiguous between 55.6 m² and 556 m². A missing field beats an invented one.

One implementation detail that cost me time: those .xml.gz files arrive as content-type: application/x-gzip without a content-encoding header. No HTTP client will auto-decompress that. It is a file that happens to be gzipped, not a gzipped response, and you have to gunzip it yourself.

When it doesn't apply at all

Sometimes the bottleneck is neither tier. Lazada's review endpoint is a clean H5/BFF JSON API — no token, no signature, no cookies required. The problem is throughput.

From a datacenter IP it sustains roughly one request every five minutes. That is not a figure of speech: three requests spaced ~5 minutes apart came back clean, and a single request sent 8 seconds after the previous one was punished immediately. The punishment arrives as HTTP 200 with an HTML body — a redirect script to a slider-captcha challenge — which is a nastier failure than a 403, because naive code stores it as data.

Three things I expected to help, which didn't:

Resting the IP. Fifteen hours of idle bought exactly one clean request, not a burst. The budget does not accumulate.
Rotating regional domains. The flag is shared across Lazada's country domains; a punish on one carries to the next on the first request.
Asking for more per request. A widely-copied GitHub snippet suggests pageSize=999999 to grab everything at once. It returns success: true with an empty model — the request is rejected, not fulfilled, and code that trusts it records "no reviews" for a product that has plenty.

There is also a ceiling that no architecture can lift. The endpoint's own paging.totalPages is wrong by roughly a factor of three: on the product I measured it advertised 25 pages of 1,243 reviews, and data ran out on page 8. The real ceiling is rateCount − hiddenCount — 1,243 minus 864 hidden, so 379 reviews, which predicted the final short page to the exact item. About 30% of that product's reviews are reachable at all. If you are selling review data, that number belongs in your documentation rather than in a support ticket six weeks later.

The tiering idea still helps here — you still want your scarce resource spent on discovery — but the scarce resource is clean IPs and solved challenges, and no amount of restructuring changes the arithmetic. One product's 379 available reviews is 8 requests, which at datacenter pace is roughly 40 minutes. That is a verdict on the transport, not a design to optimise.

The short version

Measure per host, not per site. Read robots.txt before you design, not after. Put the throttled work in one tier and everything else in another, then spend your proxy budget only on the first. Validate response shape separately from your termination condition, because the ugliest failures arrive with a 200.

And re-measure. My best number was 98.75%, my most recent is 77.94%, and the code did not change between them. The number that matters is the one you can still reproduce today.

I write these up as I go, measurements and refutations included — if you have numbers on any of the open questions above, particularly how long that search-host penalty actually lasts, I'd like to compare notes.

The three Actors this architecture came out of, if you want to see the shape in production: Avito Real Estate (discovery tier, no proxies), Lazada Reviews (enrichment tier behind residential IPs) and Wildberries (both tiers in one).

The Wildberries API in English: How to Get Products, Prices and Reviews as JSON

ActorForge — Fri, 24 Jul 2026 11:27:49 +0000

Wildberries is one of the largest online marketplaces in Russia. If you build price monitoring, assortment analytics or review mining for e-commerce, sooner or later a client asks for its data — and you discover there is no official English-language API documentation to point at.

The good news is that the Wildberries storefront is a JavaScript app talking to a small set of public JSON endpoints. No API key, no request signature, no session token — anywhere. Products, prices, stock, delivery estimates, full attributes, media and reviews are all reachable with plain HTTP requests.

I mapped those endpoints from a datacenter IP. The endpoint behaviour below was measured on 2026-07-15 and re-verified on 2026-07-18; the reliability numbers at the end run through 2026-07-19. Where something is a guess, I say so. Where a measurement of mine turned out to be wrong, I say that too — the last section of this post exists because my own first conclusion did not survive contact with a fourth run.

Rule zero: every price is in kopecks

Every price field in every Wildberries response is an integer in kopecks. Divide by 100.

{ "basic": 202400, "product": 23800, "logistics": 0, "return": 0, "cashback": 0 }

That is basic = 2,024 ₽ and product = 238 ₽ for the same headphones (live response, July 2026). basic is the price before discount, product is what the customer actually pays. Get this wrong and you ship a dashboard that overstates every price by 100× — and misses the discount on top.

1. Discovery: the search endpoint

https://search.wb.ru/exactmatch/ru/common/v4/search
  ?query=iphone
  &resultset=catalog
  &dest=-1257786
  &curr=rub
  &spp=30
  &appType=1
  &lang=ru
  &page=1

Returns HTTP 200 with three top-level keys: metadata, products, total. metadata.rs is 100 — you get 100 products per page — and total is the size of the result set.

The interesting parameter is dest, which I cover below. resultset=catalog gets you products; page paginates.

Pagination does not end — it loops

This one is a trap, and it is the reason I am writing this section at all. Wildberries does not signal the end of a result set with an empty page. Past the last real page, it starts serving page 1 again — a full 100 products, HTTP 200, no marker of any kind.

Walking page until you get an empty response therefore never terminates. You loop forever, writing duplicates, hammering the one endpoint that is actually rate-limited.

Measured on the query ноутбук (2026-07-18): pages 2 and 3 returned distinct results, page 60 returned page 1's exact contents — same leading id.

The fix is three lines: remember the first id from page 1 and stop when you see it again.

let firstIdOfPageOne;
for (let page = 1; ; page++) {
  const { products } = await search(query, page);
  if (!products?.length) break;
  if (page === 1) firstIdOfPageOne = products[0].id;
  else if (products[0].id === firstIdOfPageOne) break; // wrapped — we are done
  yield* products;
}

Do not substitute a hardcoded page cap for this. Where the wrap happens depends on the query.

Each entry in products[] carries the fields you would expect — brand, brandId, name, supplier, supplierId, supplierRating, rating / reviewRating / nmReviewRating, feedbacks / nmFeedbacks, pics, colors[], subjectId, totalQuantity — plus a sizes[] array where price and stock actually live.

The two identifiers are the thing to understand:

Field in `products[]`	What it is	What it's for
`id`	`nmId` — the SKU-level id	product cards, the static CDN card, the public URL
`root`	`imtId` — the parent card id	reviews, the static card's `imt_id`

I verified the second one against the static card: for nmId 1097975275 the search result gives root: 2227481309, and the CDN card for that product reports imt_id: 2227481309. Same number. Miss this and you will spend an hour wondering why the reviews endpoint returns nothing.

Extracting the price

const toRub = (k) => (typeof k === 'number' ? k / 100 : undefined);

function mapProduct(p) {
  // An item can carry several size entries; take the first one that actually has a price.
  const size = p.sizes?.find((s) => s.price?.product != null);
  return {
    nmId: p.id,
    imtId: p.root,
    name: p.name,
    brand: p.brand,
    price: toRub(size?.price.product),      // final price
    basicPrice: toRub(size?.price.basic),   // before discount
    rating: p.reviewRating ?? p.rating,
    feedbacks: p.feedbacks ?? p.nmFeedbacks,
    inStock: (p.totalQuantity ?? 0) > 0,
    url: `https://www.wildberries.ru/catalog/${p.id}/detail.aspx`,
  };
}

That .find() is not defensive programming for its own sake. Wildberries expresses "unavailable" by omission rather than by a null or an error — you will see the same habit in the batching behaviour below, where sold-out ids simply vanish from a response. Reaching straight into sizes[0].price.product assumes a shape the API never promised you. Treat a missing price as "not purchasable right now", not as an error.

2. Product detail: the card endpoint

https://card.wb.ru/cards/v4/detail?appType=1&curr=rub&dest=-1257786&spp=30&nm=1097975275

The response is products[] with the same product schema as search, plus a promotions array. If you have written a mapper for search results, it works here unchanged.

Two practical notes:

Only v4 exists. I checked the variants that older blog posts and GitHub projects still reference:

GET /cards/v1/detail   -> 404
GET /cards/v2/detail   -> 404
GET /cards/detail      -> 404
GET u-card.wb.ru/...   -> 404
GET /cards/v4/detail   -> 200

If you are copying a snippet from a 2024 article, this is probably why it stopped working.

It batches. Pass several nmId values separated by ;:

&nm=863160612;1097975275;14671602

One request, many products. This is the single most useful thing in this whole post for anyone scraping at volume. One caveat: invalid or unavailable ids are silently dropped. A batch of 3 returned 2 products; a batch of 21 returned 14. Never assume response.products.length === ids.length — join the results back by id yourself, and treat an absence as data rather than as a bug.

3. Full attributes and media: the basket CDN

Search and card give you the commercial view. The full merchandising card — description, attribute list, certificate, composition, media — lives on a static CDN with a computed hostname:

https://basket-{NN}.wbbasket.ru/vol{vol}/part{part}/{nmId}/info/ru/card.json

vol  = floor(nmId / 100000)
part = floor(nmId / 1000)

For nmId 1097975275 that is vol=10979, part=1097975, served by basket-42. For nmId 14671602 it is vol=146, part=14671, served by basket-02.

{NN} is assigned by a stepped lookup on vol that Wildberries extends over time (known reference points: vol=146 → basket-02, vol=8826 → basket-39, vol=10979 → basket-42). Don't hardcode that table. Probe and cache:

const hostCache = new Map(); // vol -> host number, e.g. 10979 -> "42"

async function fetchStaticCard(nmId) {
  const vol = Math.floor(nmId / 100000);
  const part = Math.floor(nmId / 1000);
  const path = `/vol${vol}/part${part}/${nmId}/info/ru/card.json`;

  const cached = hostCache.get(vol);
  if (cached) {
    const res = await fetch(`https://basket-${cached}.wbbasket.ru${path}`);
    if (res.ok) return res.json();
    hostCache.delete(vol); // shard moved — fall through and re-probe
  }

  for (let n = 1; n <= 60; n++) {
    const nn = String(n).padStart(2, '0');
    const res = await fetch(`https://basket-${nn}.wbbasket.ru${path}`);
    if (res.ok) {
      hostCache.set(vol, nn);
      return res.json();
    }
  }
  throw new Error(`no basket host found for nmId ${nmId}`);
}

One probe per vol bucket, then straight hits. The payload keys are imt_id, nm_id, imt_name, subj_name, vendor_code, description, options, compositions, certificate, colors, media, data and a few more; options is the attribute list as {name, value} pairs.

And again: imt_id here equals root from search.

4. Reviews

https://feedbacks1.wb.ru/feedbacks/v1/{imtId}

Note: imtId, not nmId. Take it from root in search, or imt_id in the static card.

The response gives you aggregates — valuation, valuationSum, valuationDistribution, feedbackCount, feedbackCountWithPhoto / WithText / WithVideo — and the reviews themselves in feedbacks[].

async function fetchReviews(imtId) {
  const res = await fetch(`https://feedbacks1.wb.ru/feedbacks/v1/${imtId}`);
  if (!res.ok) throw new Error(`feedbacks HTTP ${res.status}`);
  const data = await res.json();

  return (data.feedbacks ?? []).map((f) => ({
    stars: f.productValuation,        // 1-5
    text: f.text,
    pros: f.pros,
    cons: f.cons,
    size: f.size,
    color: f.color,
    date: f.createdDate,
    author: f.wbUserDetails?.name,
    sellerReply: f.answer?.text ?? null,
  }));
}

pros and cons are separate fields from text and are frequently empty — for sentiment work you want all three. answer.text is the seller's public reply, which is a genuinely underrated signal if you are profiling sellers rather than products.

feedbacks2.wb.ru serves identical content — it's a mirror, not a shard, so don't bother splitting requests across the two for coverage.

The hard limit nobody mentions: one response returns at most 1000 reviews, and there is no pagination on this endpoint. I checked a product reporting feedbacks: 132597 in search: the endpoint returned exactly 1000 entries, the most recent ones, sorted by createdDate descending. The aggregates at the top of the response (feedbackCount, valuationDistribution) still describe all reviews — so you can compute an accurate rating distribution for a product whose individual reviews you can only sample.

If you are building review mining, design for that now: you are working with a recent-1000 window, not a full corpus. Anyone promising you "all reviews for any product" from this endpoint has not tested it on a product with six-figure review counts. If you ship this to someone else, say which of the two numbers you are handing them.

Two more things that will save you a debugging session:

A product with no reviews returns HTTP 200 with feedbacks: null — not a 404, not an empty array. So does a nonexistent imtId. Guard for null explicitly, or every review-less product will look like a failed request.
photos and video behave differently. The photos key is present only on reviews that actually have photos (43 of 1000 on one product I sampled) and holds {id, key, isBlurred, isReady} entries; video is present on every review but is null for almost all of them (2 of 1000 on the same product). Check for presence and non-null, not for type.

One endpoint to avoid: /feedbacks/v1/summary/full?imtId=. It is referenced in various places and it returned zeroes for products that demonstrably have reviews. It looks like it works, which is worse than failing. Use /feedbacks/v1/{imtId}.

Worth noting what else rides along in that same response for free: valuationDistribution (the 1–5 star breakdown), the three feedbackCountWith* counters, and matchingSizePercentages (runs small / runs large). Those are product-level aggregates that cost you no extra request.

5. The `dest` parameter, and why it will bite you

dest is a warehouse/region code. It is not cosmetic: it changes what the API tells you about delivery and, on the product I measured, about price.

On 2026-07-15, nmId 1097975275 returned product: 11589400 with dest=-1257786 and product: 11417500 with dest=123585487 — the same item, 1,719 ₽ cheaper, purely from the region code. basic did not change; time2 (the delivery estimate) went from 18 to 27.

That is one product on one day, so I would not turn it into a law about how Wildberries prices everything. It is more than enough to justify the rule:

Pin one dest for an entire run. If you rotate it, your price time series is comparing different things and you will "discover" price movements that are really just geography.

-1257786 is the Moscow default and a reasonable choice. Some values are not valid: dest=0 returns products: null, and -1 and -446488 come back empty.

Resolving a dest code from coordinates (there is a user-geo-data.wb.ru/get_geo_info endpoint) is something I could not test — the host did not resolve from my environment. Treat it as unverified.

6. Anti-bot: two independent layers

This is where most posts about Wildberries are either vague or wrong. There are two separate mechanisms, and the one people worry about is not the one that will stop you.

Layer 1 — proof-of-work (`x-pow`), currently not enforced

Requests to search.wb.ru and card.wb.ru come back from a gateway (server: wbaas) with a header like x-pow: status=invalid;challenge=8,8,1,.... Wildberries has published how this works: adaptive difficulty, more hashes demanded from clients that look suspicious, described there in the context of their mobile app.

Here is the part that matters: on the web JSON endpoints it was not enforced. The response arrives with status=invalid and the data is in it anyway. You do not need to compute anything or send a token back.

I would not build on the assumption that this lasts forever. If enforcement is switched on, an API-level scraper needs either a hash solver or a headless-browser fallback. It costs nothing to leave a seam in your code for that today.

Layer 1.5 — your HTTP client is part of the fingerprint

Before you blame the endpoint, check what you are calling it with. On 2026-07-18 I ran the exact same search URL two ways from the same machine, seconds apart:

// Node's built-in fetch, default headers → HTML anti-bot page, not JSON
await fetch(searchUrl);

// got-scraping → HTTP 200, JSON
await gotScraping({ url: searchUrl });

Same URL, same IP, completely different outcome. search.wb.ru fingerprints the client — TLS handshake and header ordering, not just the User-Agent string, which is why setting a browser-ish UA on plain fetch does not rescue it.

This matters more than it sounds: it is a very common way to conclude "the endpoint is dead" when it is fine. Use a client that emulates a real browser's TLS profile (got-scraping, curl-impersonate, or a real browser). Plain curl and plain fetch will mislead you.

The other hosts (card, feedbacks, basket) were far less picky in my tests — they sit behind a different stack. feedbacks1 answers with server: Angie, no x-pow, and CORS *.

Layer 2 — rate limiting (the one that actually stops you)

Only search.wb.ru is throttled, and it is aggressive. From a single datacenter IP:

A cold first request can return 429 immediately, before you have sent anything else.
30 sequential requests over 69 seconds: 13 × 200, 17 × 429.
50 parallel requests at roughly 3.8 RPS: 30 × 200, 20 × 429.

The 429 body is HTML, and there is no Retry-After header — the response carries little more than date and server: wbaas. Your backoff has to be heuristic. The upside, at least initially: it behaves as a throttle rather than an outright ban, and a retry a few seconds after a 429 returned 200.

And there is a quieter failure mode than 429. Sometimes search answers HTTP 200 with a different response shape — instead of {metadata, products, total} you get {state, version, params, data: {products: [...]}} carrying unrelated content. No error code, no header to tell you apart.

That one is more dangerous than a 429, because a parser looking for a top-level products key finds nothing, returns zero results, and your pagination loop reads that as "the result set ended". You get a silently truncated dataset that looks like a clean run. Validate the shape of the response, not just the status code, and treat a shape mismatch as a retryable failure — after a backoff, the same URL returned normal data.

The other hosts were not throttled in my tests:

Host	Test	Result
`search.wb.ru`	50 parallel	30 × 200, 20 × 429
`card.wb.ru/v4`	30 parallel	30 × 200
`feedbacks1.wb.ru`	30 parallel	30 × 200
`basket-42.wbbasket.ru`	40 parallel	40 × 200

The limit is per service, and only one service is expensive. That single fact should shape your whole design.

7. What four reliability runs taught me, including where I was wrong

I want to be careful here, because this is the section where I initially drew a conclusion that later measurements destroyed.

I ran the same workload — 50 runs against live endpoints, one datacenter IP, no proxies — four times, changing the pacing settings once and then leaving them alone.

#	Settings	Success rate	Failures	p50 latency
1	250 ms min delay, no retries	91.2%	2 × 429, 4 × substituted response	473 ms
2	500 ms min delay + 2 retries (1.5 s / 3 s)	98.75%	1 × substituted response	471 ms
3	same settings, later the same day	90.14%	7 × substituted, zero 429	1700 ms
4	same settings, next day after ~12 h idle	77.94%	13 × 429, 2 × substituted; 15/50 runs empty; 48 retries	7532 ms

After run 2, the obvious story was: the failures are pacing artefacts; slow down, retry transient errors, and you get 98.75%. I believed that. It is also the kind of number that ends up in a README.

Run 3 was the same code with the same settings a few hours later: 90.14%, and the p50 had already more than tripled. Run 4, after leaving the IP completely idle overnight, was worse again — 77.94%, with mass 429s back and a p50 sixteen times the "fresh" figure.

So two of my own conclusions were wrong:

"It's fixed by pacing and retries" — refuted by run 3. Pacing and retries are necessary. They are not sufficient.
"The penalty decays if you let the IP rest" — refuted by run 4. Roughly twelve hours of idling produced not recovery but further degradation. Whatever search.wb.ru is tracking against that IP outlives a night, or the IP has landed on a reputation list.

The honest caveats: this is one IP, four measurements, at different times of day. I cannot fully separate "accumulated penalty" from "time-of-day load". But monotonic degradation through a rest period is hard to explain any other way.

Three practical consequences:

98.75% is a best case on a fresh IP, not a steady state. Quoting it as your service level would be a lie with a spreadsheet attached.
Rotating IPs on the discovery host is a precondition, not an optimization — if you want a reliability figure above 95% that holds. The other three hosts aren't throttled, so this applies to a small fraction of your traffic.
Stop benchmarking from a burned IP. Once your address is penalised, your success rate measures the depth of your own penalty, not the target's behaviour — and each run digs deeper. The next honest number has to come from a fresh address.

That third point is the one I would most like to have known earlier.

Putting it together

The architecture that falls out of the measurements is: expensive discovery, cheap enrichment.

Use search v4 only to discover ids — collect id (nmId) and root (imtId), paginate with page, 100 per page. Rate-limit yourself here, and if you need volume, this is the only place proxies are worth paying for.
Enrich with card v4 in batches of many nmId per request. Not throttled.
Fetch reviews from feedbacks1/v1/{imtId} using the root you already have. Not throttled.
Pull attributes and media from the basket CDN. Not throttled, no gateway in front of it.

Nothing in that pipeline needs a browser. No JS rendering, no proof-of-work solving, no login. It is HTTP and JSON.parse, which means it is cheap to run and boring to maintain — the two properties you actually want in a scraper. What it does not make cheap is the discovery step, and that is where your budget goes.

What I still don't know

In the spirit of not pretending:

The RPS threshold for 429 on a genuinely fresh, unpenalised IP. My numbers are a lower bound measured from an IP that had already been throttled.
How long the penalty on search.wb.ru actually lasts. I know it survives twelve hours. I don't know whether it is days, or permanent for that address.
The complete vol → basket-NN mapping. It drifts; probing is the correct answer.
How to resolve a dest code from coordinates, and the full list of valid values.
What the opaque qv and kcl tokens in metadata are for. Results come back fine without them; my guess is deep pagination, unverified.
What exactly triggers the substituted-200 response, and whether the boundary where pagination wraps is a fixed result-count cap or query-dependent. I only know how to detect both, not what drives them.
Whether and when Wildberries starts enforcing proof-of-work on web endpoints.

If you have measured any of these — particularly the penalty duration, which I can no longer test cleanly from this address — I would genuinely like to compare notes.

The scraper I built on top of these endpoints is now live: Wildberries Scraper on Apify — search, product cards and reviews as JSON, billed per record. The limits above are documented on its page too, night window included.