DEV Community

Laurent HALBRUN
Laurent HALBRUN

Posted on

Scraping French sites that block datacenter IPs — what actually worked (Pages Jaunes, Google Maps)

#webscraping #api #javascript #apify

Some sites just refuse to be scraped from the cloud. Point a headless browser at them from AWS/GCP and you get a bot wall, a captcha, or an empty results feed. The fix that finally worked for me wasn't a smarter parser — it was where the request came from.

The problem: datacenter IPs are a dead giveaway

Two French sources I needed:

  • Pages Jaunes (the French Yellow Pages) — anti-bot (DataDome), and the phone number is lazy-loaded so a naive scrape only sees name + address.
  • Google Maps — aggressively rate-limits and blocks cloud IP ranges; the results feed often loads empty from a datacenter.

From a cloud IP, both mostly failed. From a residential IP, both worked.

Fix 1 — route the browser through a residential IP

Running real Chromium through a residential IP (not a cloud range) passes the DataDome challenge on Pages Jaunes and lets the Google Maps results feed actually populate. Same code, different exit IP — that's the whole trick for the network layer.

Note: not every anti-bot falls to this. Pages Jaunes' DataDome is passable this way; some others (Leboncoin, SeLoger) run a much harder challenge that a plain render won't beat. Test before you assume.

Fix 2 — read JSON-LD, don't fight the lazy-load

On Pages Jaunes the phone is deliberately hidden until a JS interaction, so scraping the search page gives you phone: null. But each business detail page ships a <script type="application/ld+json"> block with the real number:

const m = html.match(/"telephone"\s*:\s*"([^"]+)"/);
const phone = m ? m[1].replace(/^\+33/, "0") : null;
Enter fullscreen mode Exit fullscreen mode

Read the structured data instead of the rendered DOM. On a Bordeaux "avocat" search this recovered 100% of phone numbers (5/5) vs ~0% from the search page alone.

Fix 3 — Google Maps is a SPA, so wait + scroll

Google Maps loads its results feed via XHR after the initial HTML. If you grab page.content() too early you get an empty shell. You have to:

  1. Set the Google consent cookies (SOCS / CONSENT) so you skip the consent interstitial.
  2. Wait for div[role="feed"].
  3. Scroll the feed until you have enough cards, then read them. 
for (let i = 0; i < 12; i++) {
  const n = await page.evaluate(() =>
    document.querySelectorAll('div[role="feed"] a[href*="/maps/place/"]').length);
  if (n >= max) break;
  await page.evaluate(() => {
    const f = document.querySelector('div[role="feed"]');
    if (f) f.scrollTop = f.scrollHeight;
  });
  await page.waitForTimeout(1300);
}
Enter fullscreen mode Exit fullscreen mode

Each card gives you name (the link's aria-label), rating (.MW4etd), reviews (.UY7F9), category (.W4Efsd), phone and website. Clean B2B leads — name + phone + website + rating — the kind cloud scrapers miss because the feed never loads for them.

Turned it into pay-per-result actors

Rather than keep this to myself, I packaged them as Apify actors — pay per result, no subscription:

Happy to answer questions on the residential-IP approach or the JSON-LD phone trick in the comments.

Top comments (0)