Build a Web Scraper in 10 Minutes - Complete Tutorial

If you can read a webpage in your browser, you can scrape it with Python. In about 10 minutes, you can build a real scraper that pulls data from a site, parses the HTML, and saves the results to CSV.

What you’re building

A web scraper has a simple job: fetch a page, extract the data you care about, and store it somewhere useful. That basic flow is consistent across beginner tutorials and professional guides: request the page, parse the HTML, select the fields, and export the results.[3][7][8][10]

For this tutorial, we’ll build a scraper for a simple practice site, which is ideal because it is designed for scraping and avoids the complexity of JavaScript-heavy pages. Tutorials from Real Python and others commonly start with a site like this because it lets you focus on the core mechanics instead of fighting the page.[11][10]

What you need

You only need:

• Python installed
• A terminal
• A text editor or IDE
• Two libraries: requests and beautifulsoup4[1][11]

Install them with:

pip install requests beautifulsoup4

How web scraping works

Before writing code, it helps to understand the workflow. Web scraping usually means:

• Sending an HTTP request to a page
• Receiving the HTML response
• Parsing that HTML
• Locating the elements you want
• Saving the extracted data[3][7][8][10]

That’s it. The “magic” is mostly just HTML parsing and a little pattern recognition.

Inspect the page first

Open the target site in your browser and use developer tools or “View page source” to inspect the HTML. That step is one of the most important parts of scraping because it tells you what tags, classes, and structure your code should look for.[1][11]

You are looking for:

• Repeating blocks of data
• Tags that wrap each item
• Class names or IDs that uniquely identify the content
• Links, prices, titles, or other fields you want to extract

Once you understand the structure, the code becomes straightforward.

Build the scraper

Here’s a complete working example that scrapes quotes and authors from a practice site and saves them to a CSV file.

import csv
import requests
from bs4 import BeautifulSoup

URL = "https://quotes.toscrape.com"

def scrape_quotes():
    response = requests.get(URL, timeout=10)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    quotes = []

    for quote_block in soup.select("div.quote"):
        text = quote_block.select_one("span.text").get_text(strip=True)
        author = quote_block.select_one("small.author").get_text(strip=True)
        tags = [tag.get_text(strip=True) for tag in quote_block.select("div.tags a.tag")]

        quotes.append({
            "quote": text,
            "author": author,
            "tags": ", ".join(tags)
        })

    return quotes

def save_to_csv(data, filename="quotes.csv"):
    with open(filename, "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=["quote", "author", "tags"])
        writer.writeheader()
        writer.writerows(data)

if __name__ == "__main__":
    quotes = scrape_quotes()
    save_to_csv(quotes)
    print(f"Saved {len(quotes)} quotes to quotes.csv")

How the code works

1) Fetch the HTML

requests.get() downloads the page content.[3][7][11] The raise_for_status() call is a small but valuable habit because it stops the script if the page returns an error.

2) Parse the document

BeautifulSoup(response.text, "html.parser") turns the raw HTML into something you can search with CSS selectors.[8][9][11]

3) Select repeating items

div.quote identifies each quote card. Then the scraper pulls the quote text, author, and tags from inside each card. This pattern—select a repeated block, then extract fields from inside it—is one of the most common ways to scrape HTML.[7][10]

4) Save the output

Using the built-in csv module keeps the script lightweight and portable. CSV is one of the most practical formats for scraped data because you can open it in Excel, Google Sheets, pandas, or a database import tool.[2][10][12]

Make it useful for real projects

The example above is a template you can adapt today. Here are a few practical upgrades:

Add pagination

Many sites split results across multiple pages. A common next step is looping through page URLs and collecting results from each one. Pagination is part of many real scraping workflows, especially for product listings and job boards.[1][12]

Clean the data as you go

Scraped data often includes extra whitespace, currency symbols, or mixed formatting. It is usually best to normalize values before saving them. Tutorials on practical scraping workflows consistently include a cleanup step after extraction.[7][10]

Handle errors gracefully

Network requests fail, pages change, and servers sometimes block automated traffic. Good scrapers check for request failures and keep going when one page breaks instead of crashing the whole run.[4][10]

Respect the website

Before scraping a site, check its terms of service and robots rules, and avoid aggressive request rates. Many guides emphasize that responsible scraping means limiting load and being intentional about what you collect.[1][10][11]

Common beginner mistakes

Scraping the wrong HTML

The text you see in the browser is not always the same as the HTML you get from the server. Some sites render data with JavaScript, which means a simple requests-based scraper may not see the content you expect.[4][10]

Hardcoding fragile selectors

If you rely on overly specific CSS paths, the scraper can break when the site layout changes. Prefer selectors that match the logical structure of the page rather than the exact visual layout.

Ignoring response errors

Always check whether the request succeeded. A scraper that silently processes error pages often produces bad data without warning.

When this approach is enough

A requests + BeautifulSoup scraper is perfect when the data is in static HTML. That covers a lot of useful cases: blogs, simple product catalogs, directories, and documentation pages.[3][8][11]

If the content is loaded dynamically with JavaScript, you may need a browser automation tool like Selenium or a more advanced scraping setup. Some 2026 tutorials also note that browser headers and other anti-blocking techniques matter more on modern sites.[4][10]

Your next move

You now have a real scraper you can run, edit, and extend. Replace the target URL, inspect the HTML, adjust the selectors, and export the data you actually need. That same workflow scales from a tiny practice site to a serious data collection project.[7][10][11]

If you want a strong next step, try one of these today:

• Scrape headlines from a news site
• Pull product prices from a category page
• Collect job titles and company names from a listing page
• Save the results to CSV and open them in a spreadsheet

Build the script, run it, break it, fix it, and then point it at something useful. That is how web scraping becomes a practical skill instead of just another tutorial you read once and forgot.

---

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.