DEV Community: Hugo.H

Six of my collectors returned zero. Three of them were never blocked. published: false

Hugo.H — Thu, 08 Oct 2026 08:44:54 +0000

I spent a morning pointing one collector at fourteen different sources. By lunchtime six of them were returning zero rows.

My first instinct was the same one everybody has: they are blocking me. I was wrong on half of them, and that half cost me far more time than the real blocks did.

Here is what the fourteen sources actually did, and why a zero tells you almost nothing on its own.

SIX DIFFERENT WAYS TO GET NOTHING

A block is not one thing. Across those fourteen sources I hit six genuinely different behaviours, and only two of them look like failure:

a flat refusal, immediate and honest
a rate limit that appears after a few calls and then never lifts
an interstitial challenge page served instead of the content
a success code with an empty body
a validation error that rejects the request before it is even considered
a not found that is not actually a not found

Two of those are easy. The other four all answer in ways that your error handling will happily wave through.

THE ONE THAT ACTUALLY FOOLED ME

One source answered with a success status code. Not a 403, not a redirect, not a challenge. A code in the 200 family, which every HTTP client on earth treats as "fine".

The body was empty.

My collector did what any collector does: it took the success code at face value, parsed nothing, found nothing, and moved on. It did that 270 times in a row. The run finished cleanly. No exception, no retry, no warning.

Then it handed me a qualification rate of 0.4 percent and I sat there for twenty minutes rewriting my filters, convinced my targeting criteria were wrong.

My criteria were fine. I had simply read 270 blank pages and measured the blankness very precisely.

That is the whole lesson of the morning. A collector that is being quietly starved produces output that is indistinguishable from a collector that is working on a bad dataset. You will debug the wrong layer, and you will do it confidently.

THREE OF THE SIX WERE NEVER BLOCKED AT ALL

This is the part I did not expect.

Of the six sources returning zero, three were serving me everything I asked for, perfectly, the whole time.

One had a page titled after the exact thing I was searching for. The content was right there. My collector discarded all of it because of an assumption I had made about the shape of its links, and that assumption was wrong.

Another one told me, in plain text on the page, that my results were in a different section of the site. It printed the counts. I was querying the one section that happened to be empty, and it had been telling me so from the first attempt.

A third had simply changed where it keeps its records since the last time anyone looked.

Three sources out of fourteen. Zero rows each. Not a single one of them was protected. I had spent the morning building defences against an enemy that was not in the room, while the actual problem was my own reading of pages I was already being handed.

WHAT I CHECK BEFORE I BELIEVE A ZERO

A zero is a question, not an answer. Before I touch anything defensive, I now establish three things in order:

Did I receive a real page, or a shape of a page? A success code proves a conversation happened. It proves nothing about whether anything was said. The size and the substance of what came back are separate facts from the status, and they have to be measured separately.

Does the page itself disagree with me? Sites are surprisingly talkative about their own state. Counts, section labels, empty state messages. More than once the page has told me exactly why it had nothing for me, in a sentence I had not bothered to read because I was looking at my parser.

Is my own assumption the newest thing in the system? Every collector carries beliefs about the shape of its target. Those beliefs were true when they were written. The target does not know that and does not care.

WHY THIS COSTS REAL MONEY

On fourteen sources in one morning, this is an annoying detour.

On a collector that has been feeding a product for eight months, it is a different story. The pipeline keeps running. The dashboards keep refreshing. The numbers stop moving, and nobody notices for weeks, because every system that watches whether the job ran will tell you it ran.

Almost nothing watches whether it delivered. That gap is where the expensive failures live, and it is the same gap whether you are blocked, half blocked, or not blocked at all and simply wrong.

If your collector is returning less than it used to and nothing in the logs explains why, that is usually not a mystery, it is an unmeasured layer. I do that diagnosis as a fixed scope piece of work, and most of the time the cause is not where the team expected it to be.

I take that work on Upwork. Bring me the symptom and what you have already ruled out, and I will tell you which layer is actually lying to you.

The site said 141 listings. I delivered 109. It was 100%.

Hugo.H — Tue, 06 Oct 2026 09:06:11 +0000

I set myself a real-world test: collect every apartment listed for sale in one city, from a large property portal sitting behind a Cloudflare challenge, and deliver it the way a paying client would expect.

The site's own counter said 141 listings.

I delivered 109.

And I could prove that 109 was 100% of what the site actually publishes.

This post is about that gap, because it's the question every buyer of scraped data should ask, and almost nobody does: how do you know you got everything?

THE WALL WAS NOT THE HARD PART

The first request came back 403. So did the homepage. The site was running a JavaScript challenge that refuses anything that doesn't behave like a real visitor.

Getting through took a session that respected the site's own rules: its crawling rules, a human pace, no forbidden paths, and no personal data collected at any point. Once that was in place, the listings loaded.

That's where most scraping jobs stop. Rows in a file, job done.

It's also exactly where the expensive mistakes start.

141 VS 109: THE QUESTION THAT MATTERS

A gap of 32 listings. Two possible stories:

My scraper missed 32 listings. The dataset is incomplete, and every average, every price per m² built on it is quietly wrong.
The counter overstates what the site actually serves. The dataset is complete, and the counter is the thing that's off.

From the outside, both look identical. A file with 109 rows. The only way to know is to check.

So I didn't trust a single path. I collected the same city through three independent entry points, then compared them line by line.

All three converged on the same 109 listings. Not one listing appeared in one path and not the others.

The counter was counting things the site doesn't actually publish in its lists. 109 was the full set. Nobody has to take my word for it: the proof comes with the file.

THE SILENT TRAPS ALONG THE WAY

None of these threw an error. All of them would have shipped a wrong dataset:

Number formats. The listings write 72,30 m², with a decimal comma. Read the wrong way, that becomes 7,230 m². One apartment the size of a shopping mall, and the average surface of the whole city is garbage.
Infinite scroll that gives up early. The list loads more results as you scroll. Stop a little too soon and you silently collect 11 listings, or 29, instead of 109.
Partner ads mixed into the results. Some cards look exactly like listings but come from other sites. Count them and your totals are inflated; drop them blindly and you lose real data.
A city page that isn't only the city. Nearby towns get mixed in with distances attached. Fine if you know it, misleading if you don't.

Every one of these is invisible if you only check that the job finished.

WHAT A CLIENT ACTUALLY BUYS

Not 109 rows. Anyone can hand over rows.

A client buys certainty: a clean file, one row per real listing, every field normalised, and a short quality report showing coverage, missing fields per column, and how the total was verified.

That report is the difference between data you can build decisions on and data you hope is right.

TAKEAWAYS

The wall is rarely the hardest part. Proving completeness is.
A site's own counter is not ground truth. Verify against what it actually serves.
Silent traps don't crash. They just make your numbers wrong.
Ask your data supplier one question: how do you know nothing is missing?

I build and maintain scrapers for protected and fast-changing sites, and every delivery comes with proof of what was collected and what wasn't. If you're not sure your dataset is complete, or your scraper keeps getting blocked, reach out on Upwork and I'll take a look.

My scraper returned HTTP 200 and zero rows. Nothing crashed.

Hugo.H — Tue, 29 Sep 2026 08:45:07 +0000

The worst scraping failure I've dealt with didn't throw a single error.

No 403. No captcha. No timeout. The request came back HTTP 200, 884 KB of HTML, in 557 ms. The parser ran fine. And it returned 0 listings.

If nobody is watching, that run gets logged as a success. So does the next one, and the one after that. You find out weeks later, when someone asks why the price chart went flat on the 12th.

Here's what happened, what it cost, and what I now put on every scraper I run.

WHAT CHANGED

The target was a large European classifieds site. One night they shipped a front end redesign. Same pages, same URLs, same data on screen. Different markup underneath.

The scraper had been written against the old layout. Overnight, the parts it relied on simply stopped existing. Not one of them threw an error: they just matched nothing.

The page itself was perfectly healthy. The data was still there, in plain sight. The scraper just couldn't see it anymore.

WHY NOBODY NOTICED

Every check that usually catches a broken scraper was green:

the server answered 200
the page size was normal
the response time was normal
the job finished without a single exception

From the outside, the run looked identical to a good day. The only thing missing was the only thing that mattered: the data.

That's the trap. Most monitoring watches whether the scraper ran. Almost none watches whether it delivered.

THE FIX

I rebuilt the extraction so it no longer depends on the parts of a page that designers change every few months. Redesigns happen; the collector now rides through them instead of breaking on them.

Result: 0 → 505 listings across 4 requests, 100% field completeness on price, mileage, year and location.

And there was a second silent bug hiding inside the first fix. Some fields came back empty, with no error, no warning, nothing in the logs. It only showed up because I was counting empty fields column by column. A silent failure inside the fix for a silent failure. Without that count, it would have shipped.

WHAT I CHECK ON EVERY RUN NOW

Status codes tell you whether the server answered. They say nothing about whether you got the data. So every run gets judged on its output:

Zero rows on a 200 is a failure, never an empty day.
Volume against the recent average: a sudden drop means something changed.
Field completeness: a scraper can find every row and still lose a field.

Any of these trips an alert within minutes, instead of someone noticing weeks later.

TAKEAWAYS

A 200 is not a success. Judge a run by what it extracted.
Sites will change. The question is whether your scraper survives it or quietly dies.
The question isn't whether your scraper will break. It's how fast you'll know.

I build and maintain scrapers for protected and fast-changing sites, with this kind of monitoring built in. If your scraper is quietly returning less than it should, or getting blocked, reach out on Upwork and I'll take a look.

[Boost]

Hugo.H — Sun, 20 Sep 2026 08:02:10 +0000

Hugo.H

Sep 20

Why websites block your data (explained without the tech jargon)

#webscraping #beginners #data #datascience

3 min read

Why websites block your data (explained without the tech jargon)

Hugo.H — Sun, 20 Sep 2026 07:51:26 +0000

If you've ever tried to pull data off a website and hit a wall — an error, a blank page, a "prove you're human" screen — this is for you. No code, no jargon. Just what's actually happening and why it's harder than it looks.

Think of the website as a nightclub

A protected website has a bouncer at the door. His whole job is to let real people in and keep bots out. When your script tries to grab data, it walks up to that door — and the bouncer is watching.

Here's the thing most people don't realise: the bouncer doesn't just check one thing. He checks a whole list, and he gets smarter every month.

Trick 1 — "just ask nicely"

The simplest way to get data is to send a plain request, like knocking politely and asking for the list.

On an unprotected site, this works. On a protected one, the bouncer takes one look and hands you a fake page instead of the real data.

And here's the sneaky part: the site often says "OK, success!" while actually giving you a block page. So your tool thinks it worked, saves a page full of nothing, and you don't find out until much later. A "success" message doesn't mean you got the real thing.

Trick 2 — "dress like a regular"

Next idea: make your script look like a real web browser (Chrome, Safari…). Put on the right outfit so the bouncer thinks you belong.

Problem: the bouncer doesn't just look at your clothes. He watches how you move, how you talk, tiny details a real regular has and a stranger in a costume doesn't. A good disguise gets you a second look — not through the door.

Trick 3 — "send a robot dressed as a person"

So you send a more convincing stand-in: an automated browser that really does load the page like a person would.

Closer. But these automated browsers have tells — little tics that give them away. And the bouncer has been trained specifically to spot exactly those tics. Some nights you slip through; most nights you get pulled aside.

What actually works — send a real regular

The trick that works isn't a better disguise. It's sending someone who genuinely belongs.

Instead of faking a browser, you use a real one — the exact same program a normal person uses every day — and quietly guide it to collect what you need. From the bouncer's point of view there's nothing to catch, because there's nothing fake. It's a real regular walking in.

That's the whole secret: stop pretending to be legit, and just be legit.

Why this matters if you're paying for data

Two things worth knowing if you're hiring someone (or thinking of DIY-ing it):

1. "It loaded" doesn't mean "it worked." The most expensive mistake is a file that looks full but is quietly full of block pages. Whoever does the job should check the data is real, not just that the page returned something.

2. Getting in once is easy. Staying in is the job. The bouncer changes his checklist constantly. A scraper that works today can be dead next month. The real value isn't the clever trick to get past once — it's keeping the data flowing reliably over time.

That second point is where most cheap solutions fall apart, and it's the part worth paying for.

If you've got a site that keeps blocking you and you don't know why, tell me the symptom — a blank page, an error, a "prove you're human" screen — and I'll tell you what you're likely dealing with. Costs nothing to ask.

I build and maintain scrapers for protected and fast-changing sites (Akamai, DataDome, Cloudflare). If yours is getting blocked or returning empty results, feel free to reach out on Upwork.

When curl, TLS spoofing and headless all fail: getting past Akamai Bot Manager

Hugo.H — Sat, 19 Sep 2026 07:39:21 +0000

I scrape data for a living, and most of that job is the same fight over and over: a site has data I need, and it really doesn't want me to have it. This is the story of one wall that beat every trick in the usual playbook, and how I finally got through.

The target

I needed a set of listings from a large European car marketplace. Nothing exotic: title, price, a few fields per listing. The kind of thing that should take twenty minutes.

It took a lot more than twenty minutes.

Attempt 1: plain requests

I started where everyone starts: a plain HTTP request with a normal User-Agent header.

Blocked. Not a 404, not a timeout, a challenge page. And here's the first trap that eats a lot of beginners: the response came back 200 OK. Status says success, but the body is a bot wall, not the data. If your scraper only checks the status code, it thinks it won and happily saves a page of garbage.

Lesson 1: a 200 is not a win. Read the body. A block page returned as 200 OK is a failure, not data.

This wasn't a missing header or a bad IP. The site was running Akamai Bot Manager, and it had already decided I wasn't a browser.

Attempt 2: look like a real browser (TLS spoofing)

If the problem is "you don't look like a browser," the next move is to look more like one. Not just the User-Agent string, anyone can fake that, but the TLS fingerprint: the exact way a real browser negotiates the HTTPS connection.

Still blocked.

Matching the handshake gets you past the front door, but Akamai doesn't stop there. It runs JavaScript in the page that measures things a bare HTTP client simply doesn't have.

Lesson 2: TLS fingerprinting is necessary but not sufficient. Modern bot managers check what happens after the connection, inside a real page.

Attempt 3: headless browser

Fine, if it wants a browser, I'll give it a browser. Headless automation.

Closer. But still flagged.

Headless browsers leak. They announce themselves in dozens of small ways, and Akamai's script is built precisely to notice those. You can spend days patching each tell one by one, and the wall moves the following week.

Lesson 3: a headless browser is still not a normal browser. The gap is exactly what these systems are trained to detect.

What finally worked

The breakthrough came from changing the question. Instead of asking "how do I fake a legitimate browser better?", I asked "what is this wall actually measuring, and what would it take to give it nothing to catch?"

That shift led to an approach the wall had no reason to flag. The listings loaded, and I extracted 64 complete listings, every field intact. The method is documented and I reuse it on similar sites.

The lesson under the lesson

Here's the part that matters more than any single technique:

Getting past the wall once is an afternoon. Keeping it working for six months is a different job.

The wall changes. IPs get flagged. What passes today gets caught next month. The real work isn't the clever trick, it's the diagnosis (which wall am I even facing?), the maintenance, and knowing when to stop hammering and change approach entirely. Anyone can copy a snippet. Reading the wall in front of you is the actual skill.

If you're stuck on a wall

If you've got a site that keeps blocking you and you can't tell why, that's the interesting part, and I'm happy to look. Tell me the symptom (403? a 200 that's secretly a challenge? works in the browser but not in code?) and I'll tell you what you're most likely up against.

That diagnosis is free. What you do with it is up to you.

I build and maintain scrapers for protected and fast-changing sites (Akamai, DataDome, Cloudflare). If yours is getting blocked or returning empty results, reach out on Upwork.