DEV Community

Devil Scrapes
Devil Scrapes

Posted on

We doubled Mercado Libre's scraper memory and clearance went from 50% to 90% - with zero code changes.

Quick answer

Mercado Libre retired anonymous access to its public search API — api.mercadolibre.com/sites/{MLA,MLM}/search comes back a flat 403 forbidden on every proxy tier we tested, datacenter and residential alike — so a JSON-based Mercado Libre scraper simply doesn't exist anymore. The HTML search page survives, but only past a proof-of-work challenge that a datacenter or bare-runner IP never even reaches: it gets served Mercado Libre's own suspicious-traffic-frontend decoy bundle instead, a 200 that reads exactly like a soft block. A residential exit gets a real, solvable challenge — and clearing it turned out to depend on memory more than evasion: the identical Camoufox code cleared 50% of attempts at 4 GB of Actor memory and 90% at 8 GB, because the low-memory runs were getting OOM-killed mid-challenge, not blocked. This Actor ships at 8192 MB, clears the challenge once per run, then walks up to 9 pages (offset 385) inside that one cleared session, at $5.20 per 1,000 results.

Why doesn't Mercado Libre's public search API work anymore?

Because it's not a WAF flinching at your fingerprint — it's a permanent auth wall. GET api.mercadolibre.com/sites/MLA/search?q=laptop&limit=50&offset=0 returns 403, application/json, a 67-byte body: {"message":"forbidden","error":"forbidden","status":403,"cause":[]}. That response was identical across four separate instruments: a laptop on a home connection, a bare Apify runner with no proxy, Apify datacenter, and Apify RESIDENTIAL pinned to MX/AR. No proxy tier moves a 403 that's the same on every tier — that's a company retiring an API, not a bot filter working. We don't retry that endpoint and we don't build on it; every row this Actor returns comes from the rendered search page instead.

Why does a proof-of-work challenge from a datacenter IP look exactly like a normal block?

Because the failure Mercado Libre serves a weak IP isn't a 403 at all — it's a 200. GET listado.mercadolibre.com.mx/laptop came back 200 text/html on every single instrument we tried, including this laptop, the bare Apify runner, and Apify datacenter — same ~25.6 KB body every time, and that body is Mercado Libre's own suspicious-traffic-frontend asset bundle with zero real result markers (ui-search-layout, poly-card) anywhere in it. It reads like success until you read the body.

Only the residential-proxy request got something different: a 10.9 KB page carrying the real title (Laptop | MercadoLibre 📦), a snoopy-script.js payload, trackPowResult({...with_pow:true...}), a _bmc cookie written after the computation runs, and the visible text "This page requires JavaScript to work." That's not a refusal — it's an invitation aimed at a browser, and an HTTP-only instrument can't answer it. So the recon stopped there rather than filing a false NO-GO on a channel that genuinely just needed a different kind of client.

A 200 status code can mean three completely different things that look identical in a logs table — a decoy bundle, a solvable challenge, and the real page — and they only tell apart when you actually read the body.

Why did doubling the Actor's memory turn a 50% clearance rate into 90%?

Because the failures weren't anti-bot at all — they were the container running out of RAM before the browser finished rendering. Mercado Libre's search page renders a 3.3 MB DOM, and at 4096 MB, Camoufox got killed mid-page and reported TargetClosedError: Target page, context or browser has been closed — an error message that reads exactly like the target hanging up on us. Same code, same proxy tier, same hour, run at 8192 MB instead: 9 of 10 attempts cleared, with the one miss an account-verification page, a different and recoverable refusal.

Twelve attempts at 4096 MB cleared six (50%). Ten attempts at 8192 MB cleared nine (90%). The two failure modes — real OOM kills and real anti-bot refusals — were interleaved across the same batch of runs, which is exactly how a recon done only at 4 GB ends up averaging them into "this target blocks us about half the time" and filing a verdict with no physical meaning behind the number.

A killed browser process and a hostile target can produce an identical-looking crash. Before writing off a target as too defended, check whether your own container had enough memory to finish the page.

How far can you paginate once you're past the challenge?

Further than one page, and the clearance carries you the whole way. Inside a context that already cleared the challenge once, walking offsets 49 through 385 — eight deeper pages — all came back 200, 2.65–2.77 MB, 54–59 grid ids each, in every one of four separate attempts. No challenge, captcha wall, or soft-block reappeared after the first page in any of them. You pay the residential-proxy cost of clearing the challenge once per run, then the rest of the pagination is close to free.

How did you catch a pagination check that was quietly comparing page 1 to itself?

By checking where the request actually landed, not the URL we typed. An early version of this recon guessed page 2 as ..._Desde_51, the arithmetically obvious offset, and got a 302 back to page 1 every time — so the "page 1 vs page 2" overlap check was silently diffing page 1 against page 1, and reported a suspiciously high overlap that looked like a working pager. The real page 2 is ..._Desde_49_NoIndex_True (48 results per page, 1-based offset), and the fix only surfaced once we logged the response's finalUrl instead of trusting the request URL.

If you want to prove two pages are actually different, check where your request landed after redirects, not the URL you sent. A redirect that quietly serves page 1 again will pass every overlap check that only ever looks at the outgoing request.

Is this target defended, and what does the Actor do about it?

Yes, and we build around it rather than pretending otherwise. Every request runs through a dedicated Camoufox context on a residential proxy pinned to Mexico, sized to survive the render (8192 MB) rather than get killed mid-challenge. We rotate sessions and exit IPs on every recoverable refusal and retry with exponential backoff. When a refusal exhausts retries — a captcha wall, an account-verification page, or the decoy bundle — the run surfaces a clear status message instead of quietly returning an empty dataset. None of the three refusal shapes we've seen is a permanent Access denied; all three are worth a fresh context and another attempt.

What do you get back?

One row per organic search result: itemId, title, seller, rating, listPrice + salePrice, discountPercent, installmentsText, shippingText, itemUrl, currency, the query that produced the row, the source page, and an ISO-8601 scrapedAt timestamp. Sponsored placements and recommendation carousels are excluded — every row comes from the organic result grid.

Pricing

Pay-Per-Event: $0.20 per run (a flat warm-up charge, fired once) + $0.005 per organic listing row written to your dataset. 1,000 results run about $5.20. No subscription, no minimum — Apify hands every new account $5 of free trial credit.

FAQ

Why does this need a browser instead of a fast API call?
Because the public JSON API is gone — a flat 403 on every tier — and the HTML search page is fronted by a proof-of-work challenge that only clears from a real, adequately-resourced browser on a residential exit. We run that browser and absorb the challenge so your dataset arrives clean.

Which Mercado Libre country domains are supported?
.mx (Mexico) only in v1. Other country domains need their own recon before they're added.

Are sponsored listings included?
No — only the organic result grid. Sponsored placements and recommendation carousels are dropped.

What happens if my search matches nothing?
The run succeeds with zero rows and a status message naming the query — a genuinely empty search is not a failure and isn't billed per row.

How deep can I paginate?
Up to page 9 (offset 385), the proven-clean ceiling from live recon. Depth past that is unmeasured, so we don't market it.


😈 Mercado Libre Search Results Scraper clears the marketplace's proof-of-work wall once per run and hands back normalized MX listings — price, discount, installments, shipping — for $5.20 per 1,000 results, no card required to try.

No trick fingerprint here — just a browser that needed enough RAM to finish the page before we could call the target defended at all. 😈

Top comments (0)