DEV Community

Cover image for I pointed a stealth browser at 5 sites that hate scrapers. Two shut it out completely
George Kioko
George Kioko

Posted on

I pointed a stealth browser at 5 sites that hate scrapers. Two shut it out completely

Everyone selling scraping right now sells the browser. Stealth Chromium, fresh fingerprints, "undetectable". So i tested one properly before trusting it with a paid product.

Five sites known for fighting bots. Straight from a cloud server, no proxy, real pages.

  • Wellfound (startup jobs): 20 of 20 pages, about 7 seconds each
  • StepStone (European jobs): 16 of 16, but 45 seconds a page
  • Trustpilot (reviews): 17 of 20
  • Glassdoor job details: 0 of 15
  • Seek job pages: 0 of 10

Two total wipeouts out of five, and one that works but is too slow to sell. Thats the honest state of browser scraping. Its a coin you flip, not a key.

So i mostly stopped needing one

This week i shipped five new scrapers. Three of them never open a browser at all.

Most sites build their pages out of a JSON feed the site itself calls. The page is packaging. The feed is the product. Bot protection sits on the packaging, because thats where people and ads are.

  • Seek (Australia and NZ jobs, plus Jobstreet and JobsDB in Asia). The search runs on a JSON feed. 20 data engineer jobs in 4.2 seconds, no browser, no proxy. The job pages are the ones that scored 0 of 10 above, so the actor ships what the feed gives and the README says so.
  • Substack has public archive, post and comment endpoints on every publication. 10 full posts from Lenny's Newsletter in 11.6 seconds. No login, no cookies, no proxy.
  • Telegram has a public web preview for every open channel, plain HTML with its own search box. Thats a keyword search across public channels with no account and no API key.

None of these are tricks. Theyre the same requests a normal visit makes.

When the browser earns its cost

Wellfound rejects plain HTTP, and it was the clean winner above. So that one runs on the stealth browser. 20 jobs in 57 seconds. It costs more compute and it can break the day the site changes its bot rules, which is why it costs $3 per 1,000 jobs and the Seek one costs $1.50.

The rule i build by now. Look for the feed first. Reach for the browser only when the site leaves you nothing else, and charge for it honestly.

Four of the actors from this week

All pay per result, no subscription:

The best scraper is the one that never has to pretend to be a person.

Top comments (0)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.