Sudhir Bahadure

Posted on Jul 8

Build a Web Scraper in Python That Runs on a Schedule

#automation #python #beginners #tutorial

Introduction

According to a recent report, over 70% of businesses rely on web scraping to gather valuable data, yet many developers struggle to build and schedule their own web scrapers. In this tutorial, you will learn how to build a web scraper in Python that runs on a schedule, allowing you to automate data collection and focus on more strategic tasks. To get started, you will need Python 3.8 or higher, requests and beautifulsoup4 libraries, and a basic understanding of Python programming.

Introduction
Setup and Background
Inspecting the Website and Creating a Scraper
Scheduling the Scraper
Real-World Application
Conclusion

Setup and Background

Before we dive into building the web scraper, let's understand why scheduling is essential. Many websites update their content regularly, and to stay up-to-date, you need to scrape them at regular intervals. Python's schedule library makes it easy to schedule tasks to run at specific times or intervals. Here's an example of how to use the schedule library:

import schedule
import time

def job():
    print("Hello World")

schedule.every(10).seconds.do(job)  # run job every 10 seconds

while True:
    schedule.run_pending()
    time.sleep(1)

This code will print "Hello World" every 10 seconds.

Inspecting the Website and Creating a Scraper

To build a web scraper, you need to inspect the website you want to scrape and identify the data you want to extract. For this example, let's scrape the title and all the links from the Python website. You can use the requests library to send an HTTP request to the website and get the HTML response:

import requests
from bs4 import BeautifulSoup

url = "https://www.python.org/"
response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')

title = soup.title.text
links = [a.get('href') for a in soup.find_all('a') if a.get('href')]

print(title)
print(links)

This code will print the title and all the links from the Python website.

Scheduling the Scraper

Now that you have a working web scraper, you can schedule it to run at regular intervals using the schedule library. Let's schedule the scraper to run every hour:

import schedule
import time
import requests
from bs4 import BeautifulSoup

def scrape_python_website():
    url = "https://www.python.org/"
    response = requests.get(url)

    soup = BeautifulSoup(response.text, 'html.parser')

    title = soup.title.text
    links = [a.get('href') for a in soup.find_all('a') if a.get('href')]

    print(title)
    print(links)

schedule.every(1).hours.do(scrape_python_website)  # run scraper every hour

while True:
    schedule.run_pending()
    time.sleep(1)

This code will scrape the Python website every hour and print the title and links.

Scheduling the Scraper using YAML configuration

You can also use a YAML configuration file to schedule the scraper. This approach is more flexible and allows you to easily modify the schedule without changing the code. Here's an example YAML configuration file:

schedule:
  - scraper: python_website
    interval: 1h

And here's the updated code:

import schedule
import time
import requests
from bs4 import BeautifulSoup
import yaml

with open('config.yaml', 'r') as f:
    config = yaml.safe_load(f)

def scrape_python_website():
    url = "https://www.python.org/"
    response = requests.get(url)

    soup = BeautifulSoup(response.text, 'html.parser')

    title = soup.title.text
    links = [a.get('href') for a in soup.find_all('a') if a.get('href')]

    print(title)
    print(links)

for task in config['schedule']:
    if task['scraper'] == 'python_website':
        schedule.every(int(task['interval'].split('h')[0])).hours.do(scrape_python_website)

while True:
    schedule.run_pending()
    time.sleep(1)

This code will read the YAML configuration file and schedule the scraper accordingly.

Real-World Application

Web scraping has many real-world applications, from monitoring website changes to gathering data for machine learning models. For example, you can use web scraping to track prices on e-commerce websites like Amazon and receive alerts when the price drops. You can also use web scraping to gather data for cybersecurity research, such as monitoring NordVPN (68% off + 3 months free) for vulnerabilities. Additionally, you can use web scraping to monitor website uptime and performance, and receive alerts when the website is down, using tools like Hostinger (up to 80% off hosting) and Namecheap (cheapest domains online).

Conclusion

In this tutorial, you learned how to build a web scraper in Python that runs on a schedule. You also learned how to use the schedule library to schedule tasks and how to use YAML configuration files to make the schedule more flexible. Here are three specific takeaways:

Use the requests library to send HTTP requests and get HTML responses.
Use the beautifulsoup4 library to parse HTML and extract data.
Use the schedule library to schedule tasks and run them at regular intervals. To further improve your Python automation skills, check out the next article in the Python Automation Mastery series and learn how to build a price tracker bot in Python. ---

💡 Found this helpful?

If this tutorial saved you time or solved a problem, consider:

Every coffee keeps me writing free tutorials like this one!

This article was written with AI assistance and reviewed for technical accuracy.
Part of the **Python Automation Mastery* series — Follow for more free tutorials*

#aBotWroteThis

DEV Community