DEV Community

Devil Scrapes
Devil Scrapes

Posted on

A SUCCEEDED run with zero log lines: the probe that never executed

Quick answer

A Python script with no if __name__ == "__main__": guard, run as a container entrypoint, exits 0 having done nothing. The platform records SUCCEEDED. If that script was your reconnaissance probe, you now hold a green run as evidence about a target you never actually contacted — and a 36-second SUCCEEDED run with no log lines reads, at a glance, exactly like a clean pass. That phantom result kept the AliExpress Products Scraper shelved as "blocked by a challenge wall" for nine days. When a probe with a __main__ guard finally ran, the target returned HTTP 200 on 9 of 9 pages with zero challenge markers. The wall did not exist. The parser was broken.

Why did a green run prove nothing? 🕳️

The original recon note was confident: AliExpress serves a challenge, we cannot reach product data, shelve it. The evidence was a cloud run that finished SUCCEEDED in 36 seconds.

It had no log lines at all. Not an empty result — no output whatsoever. That is not what a successful probe looks like; that is what an unexecuted module looks like. The recon file defined its functions, defined a main(), and never called it. Container starts, Python imports the module, module defines some names, interpreter exits 0.

The trap is that every signal you would normally trust pointed the right way. Exit code 0. Status SUCCEEDED. No traceback. No error to investigate. The absence of evidence got filed as evidence of absence, and the shelve note hardened into a verdict that nobody re-litigated because "we already probed that."

Rule we now apply: a run that produced no log lines is not a result, whatever its status says. Zero output is a failed probe, not a clean one.

So what was actually wrong? 🔧

A re-probe — same target, same proxy tier, this time with a __main__ guard — fetched a search page of 995 KB, harvested 8 product ids, and pulled 8 detail pages. Nine requests, nine HTTP 200, no challenge markers anywhere.

The real defect was one layer up: 0 of those 9 pages carried the run_params JSON payload the parser was written against. AliExpress had moved the data; our extractor was hunting a shape that no longer shipped. That is ordinary scraper maintenance, not an infrastructure wall — and the two need completely different responses. One is a rewrite you can do in an afternoon. The other is a purchase decision.

Diagnosing it as the wrong one cost nine days.

How do you know the rows are real? 🔬

Because "the run SUCCEEDED and rows exist" is not proof of delivery, and we have been burned by exactly that in this fleet: another Actor returned a SUCCEEDED run with three rows where every row was fabricated from page furniture that happened to satisfy generic fallback selectors.

So delivery got checked from outside the Actor. Three of the five dataset rows were re-fetched directly from live aliexpress.com product pages by a separate process, and their og:title and og:image were compared byte-for-byte against what the Actor had written. All three matched.

That check is now structural rather than a one-off. ResultRow asserts things only a genuine product page can satisfy:

  • product_id is numeric
  • price is greater than zero
  • product_url embeds the same product_id as the row

A blocked-but-200 page cannot satisfy all three by accident, so it can never again pass silently as data.

The bug behind the bug: fixed code that was never on main 🌿

There is a second failure worth naming, because it is the one that makes the first one durable.

The parser rewrite, its 97 passing tests and its verified cloud run all lived on a branch that was never fast-forwarded onto main. Running git log in the repo showed nothing but an unrelated Dockerfile restore. So the daily status report kept reading this Actor as shelved and broken while working, verified code sat one merge away — and any push from main would have reverted production to the broken parser.

"Committed" and "deployed" and "on the default branch" are three different states, and they diverge quietly. A green run on a branch nobody merged buys you nothing.

What you get per row

Field Example
product_id 1005006284718063
title full product title as listed
price / currency numeric, asserted > 0
original_price / discount_pct where the listing shows one
rating / orders_count demand signal
store_name / store_url seller attribution
image_url primary product image
product_url must embed product_id — the delivery assertion

Feed it search_keywords and it pages through results. Export as JSON, CSV or Excel.

FAQ

What does it cost?
$0.05 per run start plus $0.002 per product row — $2.05 per 1,000 products. That is set against a measurement, not a guess: a clean search-only run cost $0.0101 of platform spend for 5 rows, about $2.03 per 1,000, read after billing settled. Reading a run's cost immediately after it finishes understates it, because residential-proxy transfer lands late.

Do I need an account, an API key or a proxy?
No. Proxy rotation, the browser engine, retries and backoff are handled inside the Actor.

Is the product-detail enrichment reliable?
Not yet, and it ships labelled best-effort for that reason. The direct productIds / productUrls path — which renders a detail page in a browser — failed to resolve one specific QA product on 3 of 3 observed attempts across two builds. It is fault-isolated, so a failed enrichment skips that item and never fabricates a row. search_keywords is the reliable, cheap core path and is what to use in production. We would rather say this out loud than sell you a feature that quietly returns nothing.

Will an empty search fail the run?
No. A search that legitimately matches zero products finishes SUCCEEDED with a status message naming what was searched. Only real errors fail — and after this Actor's history, we are careful about the difference.


Built by Devil Scrapes. We do the dirty work so your dataset stays clean. 😈

Top comments (0)