Scraping OLX and the marketplaces that block datacenter IPs

#tutorial

Most "just use a scraping library" tutorials quietly assume the site lets you in. Classified-ad marketplaces like OLX don't. Here's what actually worked for me in 2026, including the two-layer block that ate a full day of my life.

Datacenter IPs get nothing

OLX Brazil embeds its listings in a Next.js __NEXT_DATA__ JSON blob. Easy to parse - if you can fetch the page. From any cloud IP you get an empty response or a challenge. It turned out to be two separate blocks stacked:

Datacenter IP filtering - most cloud ranges are blocked outright.
TLS fingerprinting - even from a residential IP, a vanilla Python requests handshake gets flagged as a bot.

You have to beat both, and I wasted hours assuming it was just one.

Fix 1: impersonate a real browser's TLS

curl_cffi mimics Chrome's actual TLS fingerprint:

from curl_cffi import requests as cffi_requests

def fetch(url, timeout=30):
    r = cffi_requests.get(url, impersonate="chrome", timeout=timeout)
    return r.status_code, r.text

That alone gets you past the fingerprint check - but not the IP block.

Fix 2: egress through a residential IP

The trick still fails from a datacenter. My setup: a small FastAPI service running on a residential connection that does the actual fetch, exposed over HTTPS with an API key. The cloud-side code is a thin proxy that calls it. Datacenter-only competitors simply can't reach the data.

The part nobody documents: OLX EU is a different stack

OLX Brazil uses the Next.js __NEXT_DATA__ approach. But OLX's European sites - Portugal, Poland, Romania, Bulgaria, Ukraine, Kazakhstan - run a completely different platform with a clean public JSON API:

https://www.olx.pt/api/v1/offers/?query=iphone&offset=0&limit=40

One parser covers all six countries. The lesson that cost me time: never assume a brand's regional sites share a backend. Probe the real response before writing the parser.

Making it actually useful

A one-off scrape is noise. The value is on a schedule: cron a category every morning, diff against yesterday's listings, and push price drops to Slack or email through Make or n8n. That turns it into an automated deal-finder instead of a thing you run by hand.

I packaged both into ready-to-run actors if you'd rather not host the residential side yourself (free tier covers testing):