DEV Community

Cover image for Why Your Headless Browser Can't Scrape Everything
Hoelee
Hoelee

Posted on Originally published at blog.hoelee.com

Why Your Headless Browser Can't Scrape Everything

Originally published on my blog — read it there with the full context, code and updates.

I hit the same wall a few weeks apart, on two different second-hand marketplaces,
and the contrast taught me more about scraping than a year of tutorials did.

On Carousell, a headless browser pulled freshly-listed products into a database
with prices, conditions, and photo URLs — no problem, running for months. On
Goofish (闲鱼), the exact same setup returned the exact same block message every time:

为了保障您的体验,请使用正常浏览器访问闲鱼~
(translation: "to protect your experience, please use a normal browser")
Enter fullscreen mode Exit fullscreen mode

Two marketplaces, one browser, opposite outcomes. This is the story of why, and
the one decision that would have saved me most of a day if I'd made it first.

The goal, badly framed

I wanted to search Goofish for products and pull the results out — titles, prices,
links. The obvious first answer is "use a headless browser." It's the answer
everyone gives: render the page with Playwright or Puppeteer, wait for the
JavaScript, read the DOM.

So I built exactly that. I deployed a self-hosted browserless container — a
ready-made headless-Chrome-as-an-API — and pointed it at Goofish search. Then I
spent hours trying every lever the tool exposes:

  • --disable-blink-features=AutomationControlled to hide the automation flag
  • patching navigator.webdriver to undefined via evaluateOnNewDocument
  • a realistic Chrome user-agent
  • a stealth launch option

Each time, the product list stayed stuck at "loading…" — the page frame rendered,
the header rendered, the footer rendered, but the actual listings never populated.
The HTML came back with zero prices and zero product links.

Meanwhile I had a memory nagging me: my Carousell monitor grabs the same kind of
data with no stealth, no anti-detection tricks, and it works flawlessly. Why?

The frame change: read how the data gets to the page

The answer wasn't in the browser flags. It was in how each site delivers its data.

Carousell embeds its search results in the page's initial payload. The server
sends the listings as JSON baked into the HTML (a __PRELOADED_STATE__-style blob).
The JavaScript enhances them afterward, but a plain HTTP request already contains
everything — product ID, title, price, photos. A scraper can parse that JSON out of
the raw HTML without ever rendering anything.

Goofish delivers nothing server-side. Its search results are fetched after
the page loads, from a signed API endpoint. That endpoint requires:

  • a signature the app computes from request parameters and a secret
  • anti-bot checks that flag a headless browser's fingerprint (WebDriver detection, CDP artifacts, canvas/audio fingerprints)

Strip the signature away, and the API returns nothing. Strip the browser off, and
the page returns "illegal access."

That single distinction — server-rendered JSON vs. a signed async API — is the
whole game. It tells you in one minute whether "render the page in a browser" will
take you an afternoon or a month.

Why the headless browser hit a wall on Goofish

The honest reality: an open-source, self-hosted browser is not a stealth tool.

browserless (the container I deployed) is a generic renderer. Its free tier
renders pages, executes JavaScript, takes screenshots, converts PDFs. The
anti-detection features — real stealth mode, captcha solving, unblocked proxies —
are explicitly the paid Cloud/Enterprise tier. The message in their docs is
unambiguous.

So I was asking a general-purpose renderer to beat a platform whose entire
business depends on defeating automated clients. That's not a configuration bug;
it's a mismatch between the tool and the adversary.

To actually scrape Goofish I'd have to choose one of a few much heavier paths:

  1. Reverse the app's signing algorithm and call the search API directly. This is how the serious scrapers do it — but the signature changes, so it's an ongoing maintenance cost, not a one-time build.
  2. Buy a third-party data API (Alibaba Cloud's marketplace and similar sell item_search endpoints). Clean data, but you pay per call and you're at the mercy of the vendor.
  3. Drive a real, logged-in browser at low frequency and hope the rate stays under the radar. Works for occasional manual checks, useless for reliable monitoring at scale.

None of these are "spin up a container and go." That's the point.

The reusable checklist

Before I touch Playwright again on a new target, I now answer three questions in
order:

  1. Where does the data actually live? Open the page with JavaScript disabled (or read the raw HTML). If the data is in the HTML, I don't need a browser at all — I need an HTTP client and a parser. The browser is waste.
  2. If it's not in the HTML, what does the async request look like? Open DevTools → Network, find the XHR/fetch that returns the data, and read its headers and query params. Is there a signature? Is it stable, or does it change per request?
  3. Is there a signed API? If yes, stop. Decide up front whether the target is worth either (a) reverse-engineering the sign, or (b) paying a data vendor. If neither, the honest answer is "this costs more than it's worth" — and knowing that in the first hour is itself a win.

Most scraping guides skip all three and jump straight to "install Puppeteer."
That's the expensive path, the one that burns a day discovering the hard way what
a 60-second investigation would have told you.

What I'd do differently

  • Disable JavaScript before reaching for a browser. It's the single fastest signal for "server-rendered vs. client-fetched." Ten seconds, no setup.
  • Don't buy stealth from a generic renderer. Stealth is either a product you pay for (browserless Cloud, ScrapingBee, etc.) or something you build and maintain. It is not a checkbox in an open-source container.
  • Distinguish "rendering" from "extraction." A headless browser returns HTML; it does not hand you structured product fields. Even when rendering works, someone still has to write the selectors or the evaluate script. That's a separate task — plan for it, don't assume it's free.

The result

The contrast sharpened into a rule I now apply to every project: decide whether
the target is server-rendered or API-gated before choosing a tool.
On Carousell
(server-rendered), a plain HTTP parser has archived thousands of listings with
prices and photos for months. On Goofish (signed API), I stopped after an honest
look at the effort and the maintenance burden — which is itself the right
outcome for a project where low-frequency spot-checks fulfill all business requirements.

A headless browser is the right tool sometimes — for screenshots, PDFs, taking
over a human session, or targets that render server-side. It is not a universal
scraping key, and treating it as one is the expensive mistake.


Want me to look at a site before you commit to scraping it?

Most scraping projects fail in the first hour because nobody checked whether the
target even lets data be scraped cheaply. I do that check before writing any
code — and I build the monitors, notification pipelines, and self-hosted
infrastructure that actually run. If you're eyeing a data source and want to know
if it's an afternoon or a month, let's talk:


I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Top comments (1)

Collapse
 
userxnom profile image
Abdullah Naseer •

The Carousell vs Goofish contrast is the part that will stick with me — Carousell embedding its listings in a PRELOADED_STATE-style blob while Goofish returns nothing until a signed API call clears. Your first checklist question, disabling JavaScript to see if the data is already in the HTML, is such a cheap test for such an expensive mistake. When a site does turn out to be API-gated, how do you usually decide whether it is worth reversing the signing or just walking away?