I scrape data for a living, and most of that job is the same fight over and over: a site has data I need, and it really doesn't want me to have it. This is the story of one wall that beat every trick in the usual playbook, and how I finally got through.
The target
I needed a set of listings from a large European car marketplace. Nothing exotic: title, price, a few fields per listing. The kind of thing that should take twenty minutes.
It took a lot more than twenty minutes.
Attempt 1: plain requests
I started where everyone starts: a plain HTTP request with a normal User-Agent header.
Blocked. Not a 404, not a timeout, a challenge page. And here's the first trap that eats a lot of beginners: the response came back 200 OK. Status says success, but the body is a bot wall, not the data. If your scraper only checks the status code, it thinks it won and happily saves a page of garbage.
Lesson 1: a 200 is not a win. Read the body. A block page returned as 200 OK is a failure, not data.
This wasn't a missing header or a bad IP. The site was running Akamai Bot Manager, and it had already decided I wasn't a browser.
Attempt 2: look like a real browser (TLS spoofing)
If the problem is "you don't look like a browser," the next move is to look more like one. Not just the User-Agent string, anyone can fake that, but the TLS fingerprint: the exact way a real browser negotiates the HTTPS connection.
Still blocked.
Matching the handshake gets you past the front door, but Akamai doesn't stop there. It runs JavaScript in the page that measures things a bare HTTP client simply doesn't have.
Lesson 2: TLS fingerprinting is necessary but not sufficient. Modern bot managers check what happens after the connection, inside a real page.
Attempt 3: headless browser
Fine, if it wants a browser, I'll give it a browser. Headless automation.
Closer. But still flagged.
Headless browsers leak. They announce themselves in dozens of small ways, and Akamai's script is built precisely to notice those. You can spend days patching each tell one by one, and the wall moves the following week.
Lesson 3: a headless browser is still not a normal browser. The gap is exactly what these systems are trained to detect.
What finally worked
The breakthrough came from changing the question. Instead of asking "how do I fake a legitimate browser better?", I asked "what is this wall actually measuring, and what would it take to give it nothing to catch?"
That shift led to an approach the wall had no reason to flag. The listings loaded, and I extracted 64 complete listings, every field intact. The method is documented and I reuse it on similar sites.
The lesson under the lesson
Here's the part that matters more than any single technique:
Getting past the wall once is an afternoon. Keeping it working for six months is a different job.
The wall changes. IPs get flagged. What passes today gets caught next month. The real work isn't the clever trick, it's the diagnosis (which wall am I even facing?), the maintenance, and knowing when to stop hammering and change approach entirely. Anyone can copy a snippet. Reading the wall in front of you is the actual skill.
If you're stuck on a wall
If you've got a site that keeps blocking you and you can't tell why, that's the interesting part, and I'm happy to look. Tell me the symptom (403? a 200 that's secretly a challenge? works in the browser but not in code?) and I'll tell you what you're most likely up against.
That diagnosis is free. What you do with it is up to you.
I build and maintain scrapers for protected and fast-changing sites (Akamai, DataDome, Cloudflare). If yours is getting blocked or returning empty results, reach out on Upwork.
Top comments (0)