How to Collect Product Data from a Website with Python

Alock75 — Sun, 26 Jul 2026 22:50:15 +0000

In this guide, you will learn how to write a Python script that visits a website, grabs product names and prices, and saves everything into a spreadsheet file. No experience needed — we explain every step.

You can find the complete source code for this project in this GitHub repository.

🧰 What You Need Before Starting

Python 3 installed on your computer (download here)
A code editor — VS Code works great (download here)
Your terminal (the black window where you type commands)
A basic understanding of Python (variables and for loops)

Step 1 — Install the Tools

We need to install 3 extra tools for Python. Open your terminal and copy-paste this line:

pip install requests beautifulsoup4 pandas

Then press Enter and wait for it to finish.

📦

What are these tools?

requests — Goes to a website and downloads its content (like a browser, but in code)
beautifulsoup4 — Reads the downloaded content and helps you find specific information in it
pandas — Puts your data into a table and saves it as a spreadsheet (.csv file)

Step 2 — Download the Website Content

First, we tell Python which website to visit and download its content.

import requests
from bs4 import BeautifulSoup
import pandas as pd

# The website you want to scrape
URL = "https://toscrape.com"

# This tells the website our script is a normal browser (not a robot)
headers = {
    "User-Agent": "Mozilla/5.0"
}

# Visit the website and download the page
response = requests.get(URL, headers=headers)

# If something goes wrong, Python will tell us
response.raise_for_status()

print("Page downloaded successfully!")

💡

The User-Agent line is important. Without it, some websites will refuse our script because they think it's a robot.

Step 3 — Find the Products on the Page

Every website is built with HTML — a language that describes what goes where on a page. We use BeautifulSoup to read that HTML and find the products.

# Read the HTML of the page
soup = BeautifulSoup(response.text, "html.parser")

# Find all product cards on the page
# On toscrape.com, each product is inside an <article> tag with class "product_pod"
products = soup.find_all("article", class_="product_pod")

print(f"We found {len(products)} products!")

🔍

How do you know which class name to use?

Open the website in Chrome or Firefox
Right-click on a product → click Inspect
Look for the HTML tag and the class="..." name around that product
Use that class name in your code

Step 4 — Get the Name and Price of Each Product

Now we go through each product one by one and grab its name and price.

# This list will store all our products
data = []

for product in products:
    # Find the name of the product
    name = product.h3.a if product.h3 else None

    # Find the price of the product
    price = product.find("p", class_="price_color")

    # Save the name and price (or "N/A" if not found)
    # We use name["title"] because the full product name is stored in the title attribute
    data.append({
        "name": name["title"] if name and name.has_attr("title") else "N/A",
        "price": price.get_text(strip=True) if price else "N/A"
    })

# Show the first 3 results to check everything looks good
print(data[:3])

Step 5 — Save Everything to a Spreadsheet

Finally, we take all the data we collected and save it as a .csv file (which you can open in Excel or Google Sheets).

# Turn the list into a table
df = pd.DataFrame(data)

# Save the table as a CSV file
df.to_csv("products.csv", index=False, encoding="utf-8")

print(f"✅ Done! {len(df)} products saved to products.csv")

⚠️ Common Problems & How to Fix Them

Problem	What it means	How to fix it
`403 Forbidden`	The website blocked your script	Make sure you added the `User-Agent` header
`AttributeError: NoneType`	The name or price was not found on the page	Check that the class names in your code match the website's HTML
Empty results (0 products found)	The website loads products with JavaScript, not HTML	This guide won't work — you'll need a more advanced tool like Selenium
`ConnectionError`	No internet or the website is down	Check your connection and try again

DEV Community: Alock75