Here is a list of places that your web crawler will probably fail at. You will need to build in checks for each and also expect them to happen. Send yourself alerts by having portions of the scripts check for unexpected behavior.
If your web crawler is stuck, you need to know
If your web crawler is slowing down, you need to know
if you are having internet issues, you need to know
if the data you are getting is weird, you need to know
Use can also use external tools like these to help keep them up.
Log all the steps your web crawler is taking and the time it took for each. Build in a check where your code sends you an alert when the time has taken is too long and if it 'knows' the data that should be fetched, but it is not fetched this time.
Here is a list of places that you need to pay special attention to in your code to prevent breakages
We the web pages dont load
Internet is down
When the content at the URL has moved
You are shown a CAPTCHA challenge.
The web page changes its HTML, so your scraping doesn't work.
Some fields that you scrape are empty some of the time, and there is no handler for that.
The web pages take a long time to load
The web site has blocked you completely
Top comments (0)