Quick answer
Alibaba's search page used to embed its product data behind a fixed marker: window.__page__data=. It doesn't anymore. Alibaba now ships window.__page__data_sse10._offer_list = {...} — a build-tagged variable name, assigned per sub-key instead of as one blob. A scraper still hunting the old literal string finds nothing, silently falls back to a regex scrape of the raw HTML, and keeps running. Every row still gets a product_id. Almost nothing else does: measured field fill on that fallback path was title ~17%, and price / moq / supplier_name / image_url all 0%. The run reports SUCCEEDED with real rows in it. Most of those rows are structurally useless. That's the bug the Alibaba Products Scraper was built to close — along with three smaller ones that showed up chasing it.
Why did the price field silently drop to zero? 🔍
Nothing threw an exception. extract_page_payload() searches the HTML for the marker, finds no match, and returns None — which is the correct behavior for a page that genuinely has no inline data. Alibaba's page just wasn't that page. The code fell through to regex_fallback_items(), a best-effort scrape built as a last resort for exactly this scenario ("Alibaba changes its inline JSON shape" is in the spec's own risk table), and the fallback did what fallbacks do: it produced something, which is worse than producing an error, because something looks like success.
The fix is a pattern, not a literal string: PAGE_DATA_MARKER_RE now matches window\.__page__data\w*\.\w*[Oo]ffer\w*\s*=\s*, tolerant of the build-tag suffix (_sse10, confirmed stable across five fresh proxy sessions on 2026-09-17 — not guaranteed to stay that way) and of the exact sub-key spelling. A hardcoded string is a bet that a vendor's minifier never changes; a pattern that describes the shape of the assignment survives the next rename too.
How do you tell a real zero-match search from an anti-bot challenge? 🛡️
Here's the trap underneath the trap: a served challenge page and a genuine "nothing matched your search" page look identical to a naive parser. Both return 200 OK. Both have no window.__page__data. Both parse to zero product items. If your zero-row logic treats every empty parse as "no matches," you'll report a clean success on a run the target actually blocked.
We fixed that by looking for a positive signal instead of trusting the absence of one: Alibaba's own challenge assets — punish-component, awsc.js, baxia-punish — show up in the HTML of a real challenge and never in a genuine results page. is_challenge_page() checks for those markers explicitly. Only a page with none of them and no product data gets to succeed with zero rows; a challenge page now retries instead.
What happens when a recommendations module outranks the real product list? 🎯
The extractor's fallback path, when the known field is missing, BFS-walks the whole decoded JSON payload and keeps the largest list of dicts that carries a productId. That's a reasonable general strategy — until Alibaba ships a second productId-bearing list on the same page. A 226-row deep cloud verification run caught one page where every row filled title and product_id but every other field was None. The cause: a sibling pc_less_recommend block — Alibaba's own "less relevant recommendations" module — happened to outsize the real offerResultData.offers list on a low-match page, and the pure "biggest list wins" rule picked the recommendations instead of the results.
The fix keeps the BFS as a fallback but no longer trusts it first: search_result_items() now checks the confirmed-real path (offerResultData.offers) before falling back to the largest-list search. Known shape beats guessed shape whenever the known shape is actually present.
Does this target need a residential proxy? 📡
Recon said no. Production said otherwise. The 0.2 build's cloud QA run — both search terms, default (non-residential) Apify Proxy — failed outright: every request hit a served 200 OK anti-bot challenge. Not intermittently. Every time, that day, on that pool. Switching the default to RESIDENTIAL pinned to US cleared the same target on every attempt. We now default every run to residential proxy sessions, rotated on every detected challenge, with a fresh browser-impersonation profile picked on retry too — Chrome, Firefox, and Safari TLS fingerprints, not one identity hammered repeatedly.
That retry loop is bounded by the run's actual platform kill deadline (Actor.configuration.timeout_at), not a guessed fixed attempt count — a caller can shorten a run's timeout via the API, and a constant baked in at build time won't know that. If a term still can't clear a challenge before the deadline, the run finishes and its status message names exactly which terms delivered and which didn't, instead of a single aggregate row count that hides a shut-out term inside a technically-true "SUCCEEDED."
What you get per row
| Field | Notes |
|---|---|
product_id |
Alibaba's internal ID |
title |
HTML badges/highlight tags stripped |
price / moq
|
raw strings as served |
supplier_name |
when present |
image_url / product_url
|
absolute URLs |
page_number / scraped_at
|
provenance |
Duplicate product_ids are removed run-wide, not just per page — Alibaba's sponsored listings repeat heavily across search pages, so per-page dedup alone leaves obvious repeats in the dataset.
What does it actually cost?
Pay-Per-Event: $0.20 per run start + $0.006 per unique row — $6.20 per 1,000 results at that rate. Measured, not estimated: two full-depth cloud runs today (two search terms, 100 results each, 200 rows total per run) settled at roughly $0.015 of platform cost per 200 rows — about $0.075 per 1,000 rows of underlying infrastructure spend, comfortably inside the $6.20 price band even with residential proxy sessions now the default.
FAQ
Why did earlier runs return rows with a product_id but no price, MOQ, or image?
The extraction marker Alibaba's page relies on changed to a build-tagged variable name. The old literal-string marker matched nothing, so every run silently fell through to a low-fidelity regex fallback that only ever recovers product_id, and inconsistently title. That's fixed — the marker is now pattern-based, tolerant of the build tag.
Do I need to configure a residential proxy myself?
No. It's the default now. Non-residential sessions hit a served anti-bot challenge reliably in testing; residential sessions, rotated on every block with a fresh browser-impersonation profile, cleared the same target every time tried.
Will a search that matches nothing fail my run?
No — but a blocked run won't masquerade as a real zero-match either. We check for Alibaba's own challenge-page markers before accepting an empty parse as "genuinely no results."
What happens if one search term gets blocked and another doesn't in the same run?
The run finishes and its status message reports delivered-versus-requested term coverage, so a partial result is visible, not buried inside a single total row count.
Built by Devil Scrapes. We do the dirty work so your dataset stays clean. 😈
Top comments (0)