DEV Community

lenevo02487
lenevo02487

Posted on

I Scraped a 1,000-Book Catalogue with Python — Full Walkthrough

#python #webscraping #tutorial #beginners

Every web scraping tutorial stops at "here's how to grab one page." Real catalogues don't fit on one page — so for this practice project I scraped the entire books.toscrape.com catalogue: 1,000 books across 50 pages, into one clean CSV with title, price, star rating, and stock status.

books.toscrape.com exists specifically for scraping practice, which makes it the perfect place to learn the full workflow — pagination included. Here's exactly how it works, with the real code.

What we're building

A script that:

  1. Opens the catalogue's first page
  2. Extracts every book card on the page
  3. Follows the "next" link until there isn't one
  4. Writes everything to books.csv

Final output (real rows from the run):

title,price_gbp,rating,availability
A Light in the Attic,51.77,3,In stock
Tipping the Velvet,53.74,1,In stock
Soumission,50.10,1,In stock
Enter fullscreen mode Exit fullscreen mode

Setup 

pip install requests beautifulsoup4
Enter fullscreen mode Exit fullscreen mode

That's it. No browser automation, no Selenium — the site serves plain HTML, so requests + BeautifulSoup is enough. Reaching for a headless browser when the HTML is right there is the most common beginner mistake I see.

Step 1: Understand one book card

Every book on the site lives in an <article class="product_pod"> element. Inside it:

  • the title is in the title attribute of the <h3><a> link (the visible text is truncated with "…", the attribute has the full title — grab the attribute!)
  • the price is in .price_color
  • the rating is a word — One to Five — hidden in the class list of .star-rating
  • availability is the text of .availability 
RATING_WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

def parse_book(article):
    """Pull title, price, rating and availability out of one book card."""
    title = article.h3.a["title"]
    price = article.select_one(".price_color").get_text(strip=True)
    rating_word = article.select_one(".star-rating")["class"][1]
    availability = article.select_one(".availability").get_text(strip=True)
    return {
        "title": title,
        "price_gbp": price.replace("£", ""),
        "rating": RATING_WORDS.get(rating_word, ""),
        "availability": availability,
    }
Enter fullscreen mode Exit fullscreen mode

Step 2: Follow the pagination

Instead of guessing page URLs (page-2.html, page-3.html…), let the site tell you where to go next. Each page has a li.next a link — follow it until it disappears:

import time
import requests
from bs4 import BeautifulSoup

BASE = "https://books.toscrape.com/"

def scrape(max_pages=None):
    books = []
    page_url = BASE
    page_no = 0
    session = requests.Session()
    session.headers["User-Agent"] = "Mozilla/5.0 (demo scraper)"

    while page_url:
        page_no += 1
        resp = session.get(page_url, timeout=25)
        resp.raise_for_status()
        resp.encoding = "utf-8"  # see lesson 1 below

        soup = BeautifulSoup(resp.text, "html.parser")
        for article in soup.select("article.product_pod"):
            books.append(parse_book(article))
        print(f"page {page_no}: {len(books)} books so far")

        if max_pages and page_no >= max_pages:
            break
        next_link = soup.select_one("li.next a")
        page_url = requests.compat.urljoin(page_url, next_link["href"]) if next_link else None
        if page_url:
            time.sleep(0.3)  # be polite to the server

    return books
Enter fullscreen mode Exit fullscreen mode

Two details matter here:

  • urljoin — the next-page href is relative (catalogue/page-2.html), and urljoin resolves it correctly against whatever page you're on. Concatenating strings by hand breaks the moment the site structure changes.
  • The small sleep — 0.3s between pages costs you 15 seconds over 50 pages and keeps you a good citizen. On real projects this matters even more.

Step 3: Write the CSV 

import csv

books = scrape()
with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price_gbp", "rating", "availability"])
    writer.writeheader()
    writer.writerows(books)

print(f"Done: {len(books)} books written to books.csv")
Enter fullscreen mode Exit fullscreen mode

Output: Done: 1000 books written to books.csv — exactly 1,000, which matches the catalogue's own count. Always check your row count against a number the site itself reports; "it ran without errors" is not the same as "it got everything."

3 lessons this project teaches

1. Encoding will bite you. The site is UTF-8, but requests sometimes guesses Latin-1 from the headers — and the £ sign turns into £ in your CSV. One line (resp.encoding = "utf-8") fixes it. I found this the fun way: by opening the CSV.

2. Attributes beat visible text. The visible title is cut off ("A Light in the …"); the full title sits in the title attribute. When scraped data looks truncated, inspect the element before blaming your code.

3. Ratings hide in class names. There's no "3 stars" text anywhere — the rating is a CSS class. Real sites hide data in classes, data attributes, and JSON blobs all the time; reading the raw HTML is a core scraping skill.

Full code

The complete script (with a --pages flag for quick tests) is on GitHub: python-web-scraper — the repo includes the actual books.csv output with all 1,000 rows.

Demo first — on YOUR sample — before you pay anything. That's how I work: if you need a catalogue, price list, or directory scraped, send me the site and a few rows of the output you want, and I'll build a small working demo on your sample before you spend anything.

👉 My Python web scraping & automation gig on Fiverr

— Abhishek, Python Automation Developer

Top comments (0)