DEV Community

Devil Scrapes
Devil Scrapes

Posted on

OLX Brazil embeds a decoy listings array with the exact shape of the real one. We had to prove which was which.

Quick answer

OLX Brazil's search runs entirely on the nationwide host www.olx.com.br/brasil?q= — not on the regional subdomains its own listing URLs resolve to (ba.olx.com.br, sp.olx.com.br, and dozens more) — and paginates cleanly with &o=<page>: page 2 returned a fully disjoint 50-id set against page 1, zero overlap, at 777 KB. The page ships its data as Next.js App Router streaming chunks (self.__next_f.push(...)), not the classic __NEXT_DATA__ blob you'd expect, and one of those chunks is a decoy with the exact same shape as the real listings array — topoVipSelection.ads, always empty, no listId field — sitting right next to the real one (ads paired with totalOfAds). Get that disambiguation wrong and you ship an OLX Brazil scraper that runs green and returns nothing. This one turns a keyword into a deduplicated, region-tagged dataset — title, price in BRL, location, category, image — for $3.00 per 1,000 results.

Why do OLX Brazil listing URLs point at dozens of subdomains if you search from one host?

Because search and hosting are two different concerns on OLX Brazil, and only one of them is regional. Every search — regardless of which Brazilian state or city you're actually interested in — goes to the same nationwide endpoint, www.olx.com.br/brasil?q=<query>. The listings that endpoint returns each carry their own canonical detail URL on a regional subdomain: a notebook listing might resolve to ba.olx.com.br/grande-salvador/informatica/notebooks/.... We parse the region straight out of that URL (^https?://([a-z]{2,20})\.olx\.com\.br/) rather than out of anything you searched with, which is what lets you filter a nationwide search down to one region ("ba", "sp", "rj") after the fact — no separate regional endpoint to hit, no extra request.

What's a decoy payload, and how did it almost break this parser?

It's a chunk of embedded JSON that looks, structurally, exactly like the data you actually want — sitting a few keys away from it on the same page. OLX Brazil's search page streams its data in Next.js App Router chunks rather than the single classic __NEXT_DATA__ blob an older design might assume. Inside those chunks sits topoVipSelection.ads: same list-of-objects shape as the real listings, same ads naming pattern, and it is always empty with a sibling seller key instead of a listId. A parser that grabs "the first thing shaped like an ads array" grabs that one, and ships an Actor that builds, runs, and reports success — with zero rows, forever.

The real container is the chunk-payload dict that carries both an ads list and a totalOfAds key. That pairing is what disambiguates it, confirmed against real captured pages rather than hand-typed fixtures.

A payload shaped exactly like the data you want, sitting right next to the data you want, is the quietest way for a scraper to ship completely wrong. Disambiguate by the field that has to be there for real data — not by position or a plausible-sounding key name.

A handful of entries inside the real ads array are also sponsored ad-slot placeholders ({"advertisingId": ..., "deviceType": ...} with no listId) — genuine, live-observed malformed cards, not fixture noise, and the parser skips them rather than emitting a half-populated row.

Why does the Actor prefer the embedded JSON over the rendered HTML when both are on the page?

Because a page's own internal data contract changes far less often than its CSS classes and DOM structure do. This Actor checks for that Next.js chunk payload first and only falls back to parsing rendered <section class="olx-adcard"> cards and their child selectors when the embedded data is missing. A visual redesign — new class names, restructured cards — doesn't necessarily touch the JSON the page ships itself; a change to the JSON contract is a much rarer, more deliberate event on OLX's side.

When a page embeds its own data as JSON, parse that first and treat the rendered DOM as the fallback, not the primary source — a redesign changes markup far more often than it changes an internal data contract.

How do you know pagination isn't just re-serving page 1?

By diffing the actual id sets, not by trusting the parameter. o omitted or o=1 is page 1; o=2 returned a full 50-id page with zero overlap against page 1's 50 ids, confirmed live against a 777 KB response. That's the same discipline this fleet applies everywhere a page/offset parameter exists: request it, then check whether the ids actually changed — a redirect or a silently-repeated feed reads identically to real pagination until you check.

Why is price_brl sometimes null when price_raw is always filled in?

Because not every card carries a number. Cards can show a real BRL amount ("R$ 2.200"), or non-numeric price text like "Grátis" or "A combinar" (negotiable) — both real, common answers, not parse failures. price_raw keeps the verbatim card text always; price_brl is the parsed numeric value, null whenever there's genuinely no number to parse. Filtering on price_brl is not None is the deliberate way to drop the negotiable/free listings if your use case needs strict pricing.

What happens when a search runs out of new listings before it hits your page cap?

Pagination stops the moment a page contributes zero new listing ids — not when it hits maxPagesPerQuery. That means you can legitimately get fewer rows than maxResultsPerQuery asked for: it's the search running out of real inventory, not a bug, and nothing is billed for pages you didn't actually get data from.

What do you get back?

One row per listing card: listing_id, title, price_raw + price_brl, url, region (parsed from the URL's subdomain), location, category_path, image_url, posted_at_raw when the card shows one, the query and page that produced the row, and an ISO-8601 scraped_at timestamp. Point it at up to 20 keywords in one run; each runs its own paginated search, deduplicated by listing_id.

Pricing

Pay-Per-Event: $0.20 per run (a flat warm-up charge, fired once) + $0.0028 per listing row written to your dataset. 1,000 results run about $3.00. No subscription, no minimum — Apify hands every new account $5 of free trial credit.

FAQ

Can I limit results to one Brazilian state or city?
Yes — set region to the regional subdomain code (e.g. "ba", "sp", "rj"), parsed straight from each listing's own URL. Leave it empty to get every region.

Why did I get fewer rows than maxResultsPerQuery?
Pagination stops early once a page contributes zero new listing ids — that's the real end of that search's results, not a bug.

Do sponsored slots ever leak into the dataset as broken rows?
No. Real sponsored placeholders inside the listings payload (no listId) are detected and skipped, and the always-empty decoy list (topoVipSelection.ads) is never read as the source at all.

Is this legal to run?
The Actor only fetches what OLX Brazil's own search pages already serve publicly, with no login. You're responsible for how you use the output against OLX's terms of service.

How do I export to Google Sheets?
After the run, go to Storage → Dataset → Export and pick CSV — Google Sheets imports it directly.


😈 OLX Brazil Listings Scraper turns one keyword into a deduplicated, region-tagged dataset off Brazil's biggest classifieds site — title, price in BRL, location, category, image. $3.00 per 1,000 results, pay only for results that land, no card required to try.

No captcha to fight on this one — just a decoy list wearing the real one's clothes. We checked for the field that couldn't lie. 😈

Top comments (0)