DEV Community

Alock75
Alock75

Posted on

How to Collect Product Data from a Website with Python

#python #webscraping #tutorial #beginners

In this guide, you will learn how to write a Python script that visits a website, grabs product names and prices, and saves everything into a spreadsheet file. No experience needed — we explain every step.

You can find the complete source code for this project in this GitHub repository.

🧰 What You Need Before Starting

  • Python 3 installed on your computer (download here)
  • A code editor — VS Code works great (download here)
  • Your terminal (the black window where you type commands)
  • A basic understanding of Python (variables and for loops)

Step 1 — Install the Tools

We need to install 3 extra tools for Python. Open your terminal and copy-paste this line:

pip install requests beautifulsoup4 pandas
Enter fullscreen mode Exit fullscreen mode

Then press Enter and wait for it to finish.

📦

What are these tools?

  • requests — Goes to a website and downloads its content (like a browser, but in code)
  • beautifulsoup4 — Reads the downloaded content and helps you find specific information in it
  • pandas — Puts your data into a table and saves it as a spreadsheet (.csv file)

Step 2 — Download the Website Content

First, we tell Python which website to visit and download its content.

import requests
from bs4 import BeautifulSoup
import pandas as pd

# The website you want to scrape
URL = "https://toscrape.com"

# This tells the website our script is a normal browser (not a robot)
headers = {
    "User-Agent": "Mozilla/5.0"
}

# Visit the website and download the page
response = requests.get(URL, headers=headers)

# If something goes wrong, Python will tell us
response.raise_for_status()

print("Page downloaded successfully!")
Enter fullscreen mode Exit fullscreen mode

💡

The User-Agent line is important. Without it, some websites will refuse our script because they think it's a robot.

Step 3 — Find the Products on the Page

Every website is built with HTML — a language that describes what goes where on a page. We use BeautifulSoup to read that HTML and find the products.

# Read the HTML of the page
soup = BeautifulSoup(response.text, "html.parser")

# Find all product cards on the page
# On toscrape.com, each product is inside an <article> tag with class "product_pod"
products = soup.find_all("article", class_="product_pod")

print(f"We found {len(products)} products!")
Enter fullscreen mode Exit fullscreen mode

🔍

How do you know which class name to use?

  1. Open the website in Chrome or Firefox
  2. Right-click on a product → click Inspect
  3. Look for the HTML tag and the class="..." name around that product
  4. Use that class name in your code

Step 4 — Get the Name and Price of Each Product

Now we go through each product one by one and grab its name and price.

# This list will store all our products
data = []

for product in products:
    # Find the name of the product
    name = product.h3.a if product.h3 else None

    # Find the price of the product
    price = product.find("p", class_="price_color")

    # Save the name and price (or "N/A" if not found)
    # We use name["title"] because the full product name is stored in the title attribute
    data.append({
        "name": name["title"] if name and name.has_attr("title") else "N/A",
        "price": price.get_text(strip=True) if price else "N/A"
    })

# Show the first 3 results to check everything looks good
print(data[:3])
Enter fullscreen mode Exit fullscreen mode

Step 5 — Save Everything to a Spreadsheet

Finally, we take all the data we collected and save it as a .csv file (which you can open in Excel or Google Sheets).

# Turn the list into a table
df = pd.DataFrame(data)

# Save the table as a CSV file
df.to_csv("products.csv", index=False, encoding="utf-8")

print(f"✅ Done! {len(df)} products saved to products.csv")
Enter fullscreen mode Exit fullscreen mode

⚠️ Common Problems & How to Fix Them

Problem What it means How to fix it
403 Forbidden The website blocked your script Make sure you added the User-Agent header
AttributeError: NoneType The name or price was not found on the page Check that the class names in your code match the website's HTML
Empty results (0 products found) The website loads products with JavaScript, not HTML This guide won't work — you'll need a more advanced tool like Selenium
ConnectionError No internet or the website is down Check your connection and try again

Top comments (0)