The worst scraping failure I've dealt with didn't throw a single error.
No 403. No captcha. No timeout. The request came back HTTP 200, 884 KB of HTML, in 557 ms. The parser ran fine. And it returned 0 listings.
If nobody is watching, that run gets logged as a success. So does the next one, and the one after that. You find out weeks later, when someone asks why the price chart went flat on the 12th.
Here's what happened, what it cost, and what I now put on every scraper I run.
WHAT CHANGED
The target was a large European classifieds site. One night they shipped a front end redesign. Same pages, same URLs, same data on screen. Different markup underneath.
The scraper had been written against the old layout. Overnight, the parts it relied on simply stopped existing. Not one of them threw an error: they just matched nothing.
The page itself was perfectly healthy. The data was still there, in plain sight. The scraper just couldn't see it anymore.
WHY NOBODY NOTICED
Every check that usually catches a broken scraper was green:
- the server answered 200
- the page size was normal
- the response time was normal
- the job finished without a single exception
From the outside, the run looked identical to a good day. The only thing missing was the only thing that mattered: the data.
That's the trap. Most monitoring watches whether the scraper ran. Almost none watches whether it delivered.
THE FIX
I rebuilt the extraction so it no longer depends on the parts of a page that designers change every few months. Redesigns happen; the collector now rides through them instead of breaking on them.
Result: 0 → 505 listings across 4 requests, 100% field completeness on price, mileage, year and location.
And there was a second silent bug hiding inside the first fix. Some fields came back empty, with no error, no warning, nothing in the logs. It only showed up because I was counting empty fields column by column. A silent failure inside the fix for a silent failure. Without that count, it would have shipped.
WHAT I CHECK ON EVERY RUN NOW
Status codes tell you whether the server answered. They say nothing about whether you got the data. So every run gets judged on its output:
- Zero rows on a 200 is a failure, never an empty day.
- Volume against the recent average: a sudden drop means something changed.
- Field completeness: a scraper can find every row and still lose a field.
Any of these trips an alert within minutes, instead of someone noticing weeks later.
TAKEAWAYS
- A 200 is not a success. Judge a run by what it extracted.
- Sites will change. The question is whether your scraper survives it or quietly dies.
- The question isn't whether your scraper will break. It's how fast you'll know.
I build and maintain scrapers for protected and fast-changing sites, with this kind of monitoring built in. If your scraper is quietly returning less than it should, or getting blocked, reach out on Upwork and I'll take a look.
Top comments (0)