I spent a morning pointing one collector at fourteen different sources. By lunchtime six of them were returning zero rows.
My first instinct was the same one everybody has: they are blocking me. I was wrong on half of them, and that half cost me far more time than the real blocks did.
Here is what the fourteen sources actually did, and why a zero tells you almost nothing on its own.
SIX DIFFERENT WAYS TO GET NOTHING
A block is not one thing. Across those fourteen sources I hit six genuinely different behaviours, and only two of them look like failure:
- a flat refusal, immediate and honest
- a rate limit that appears after a few calls and then never lifts
- an interstitial challenge page served instead of the content
- a success code with an empty body
- a validation error that rejects the request before it is even considered
- a not found that is not actually a not found
Two of those are easy. The other four all answer in ways that your error handling will happily wave through.
THE ONE THAT ACTUALLY FOOLED ME
One source answered with a success status code. Not a 403, not a redirect, not a challenge. A code in the 200 family, which every HTTP client on earth treats as "fine".
The body was empty.
My collector did what any collector does: it took the success code at face value, parsed nothing, found nothing, and moved on. It did that 270 times in a row. The run finished cleanly. No exception, no retry, no warning.
Then it handed me a qualification rate of 0.4 percent and I sat there for twenty minutes rewriting my filters, convinced my targeting criteria were wrong.
My criteria were fine. I had simply read 270 blank pages and measured the blankness very precisely.
That is the whole lesson of the morning. A collector that is being quietly starved produces output that is indistinguishable from a collector that is working on a bad dataset. You will debug the wrong layer, and you will do it confidently.
THREE OF THE SIX WERE NEVER BLOCKED AT ALL
This is the part I did not expect.
Of the six sources returning zero, three were serving me everything I asked for, perfectly, the whole time.
One had a page titled after the exact thing I was searching for. The content was right there. My collector discarded all of it because of an assumption I had made about the shape of its links, and that assumption was wrong.
Another one told me, in plain text on the page, that my results were in a different section of the site. It printed the counts. I was querying the one section that happened to be empty, and it had been telling me so from the first attempt.
A third had simply changed where it keeps its records since the last time anyone looked.
Three sources out of fourteen. Zero rows each. Not a single one of them was protected. I had spent the morning building defences against an enemy that was not in the room, while the actual problem was my own reading of pages I was already being handed.
WHAT I CHECK BEFORE I BELIEVE A ZERO
A zero is a question, not an answer. Before I touch anything defensive, I now establish three things in order:
Did I receive a real page, or a shape of a page? A success code proves a conversation happened. It proves nothing about whether anything was said. The size and the substance of what came back are separate facts from the status, and they have to be measured separately.
Does the page itself disagree with me? Sites are surprisingly talkative about their own state. Counts, section labels, empty state messages. More than once the page has told me exactly why it had nothing for me, in a sentence I had not bothered to read because I was looking at my parser.
Is my own assumption the newest thing in the system? Every collector carries beliefs about the shape of its target. Those beliefs were true when they were written. The target does not know that and does not care.
WHY THIS COSTS REAL MONEY
On fourteen sources in one morning, this is an annoying detour.
On a collector that has been feeding a product for eight months, it is a different story. The pipeline keeps running. The dashboards keep refreshing. The numbers stop moving, and nobody notices for weeks, because every system that watches whether the job ran will tell you it ran.
Almost nothing watches whether it delivered. That gap is where the expensive failures live, and it is the same gap whether you are blocked, half blocked, or not blocked at all and simply wrong.
If your collector is returning less than it used to and nothing in the logs explains why, that is usually not a mystery, it is an unmeasured layer. I do that diagnosis as a fixed scope piece of work, and most of the time the cause is not where the team expected it to be.
I take that work on Upwork. Bring me the symptom and what you have already ruled out, and I will tell you which layer is actually lying to you.
Top comments (0)