DEV Community

Alex Vance
Alex Vance

Posted on

Web Scraping for Consumer Research: A Python & BeautifulSoup Tutorial

#python #datascience #webscraping #tutorial

Introduction: The "Why" Behind the Code

As a data analyst, I'm obsessed with turning chaos into clarity. One of the most chaotic environments for consumers is the UK's online entertainment market. It's a wall of noise: flashy promises, complex terms, and dozens of near-identical platforms. How can a regular person make an informed decision?

The answer is data. But where does that data come from? You have to gather it.

This tutorial is a deep dive into the 'how'. I'm going to walk you through a complete, beginner-friendly web scraping project using Python, requests, and BeautifulSoup. We'll build a conceptual scraper to gather data from a sample webpage, clean it, and structure it for analysis. This is the foundational skill for any data-driven consumer research project.

Part 1: The Ethics and The Setup

Before we write a single line of code, let's talk ethics. Web scraping can be a powerful tool, but it comes with responsibilities:

  1. Respect robots.txt: This is a file on every website that tells bots which pages they are and are not allowed to access. Always check it.
  2. Don't Overload Servers: Send requests at a reasonable rate. A simple time.sleep(1) between requests is a good start. Be a polite guest.
  3. Identify Yourself: Set a user-agent in your request headers that identifies your script or project.
  4. Scrape Public Data Only: Never attempt to scrape data that is behind a login or is not intended for public consumption.

Our goal is ethical data collection for consumer empowerment, not spam.

The Toolbox:

You'll need Python 3 installed. Then, let's get our libraries.

pip install requests beautifulsoup4 pandas
Enter fullscreen mode Exit fullscreen mode
  • requests: To handle the HTTP requests and fetch the HTML content.
  • beautifulsoup4: The magic wand for parsing HTML and XML documents. It creates a parse tree from page source code that can be used to extract data.
  • pandas: The ultimate tool for data manipulation and analysis in Python. We'll use it to structure and save our data.

Part 2: Making the First Request

For this tutorial, we can't scrape a live, complex website. It's bad practice and the structure might change. Instead, let's work with a sample, static HTML structure that mimics a typical review listing page.

Imagine we have a page with the following HTML:

<html>
<body>
  <div class="review-card" id="site-1">
    <h2 class="site-name">PlaySafe UK</h2>
    <div class="rating-badge">9.5/10</div>
    <div class="bonus-offer">100% up to £50</div>
    <div class="payout-speed"><span>Payout Speed:</span> 24 Hours (e-wallets)</div>
    <a href="/reviews/playsafe-uk" class="review-link">Read More</a>
  </div>
  <div class="review-card" id="site-2">
    <h2 class="site-name">Gambit Palace</h2>
    <div class="rating-badge">8.8/10</div>
    <div class="bonus-offer">Get 200 Free Spins</div>
    <div class="payout-speed"><span>Payout Speed:</span> 2-3 Days</div>
    <a href="/reviews/gambit-palace" class="review-link">Read More</a>
  </div>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode

Our goal is to extract the Name, Rating, Bonus, and Payout Speed from each review-card.

First, let's write the Python code to fetch this content. In a real script, you'd use a URL. Here, we'll just use a multiline string.

import requests
from bs4 import BeautifulSoup

# In a real project, this would be:
# URL = "http://example-review-site.com/uk-reviews"
# headers = {'User-Agent': 'TDUX-Research-Bot/1.0'}
# response = requests.get(URL, headers=headers)
# html_content = response.text

# For our tutorial, we'll use a local string
html_content = """
<html>
<body>
  <div class="review-card" id="site-1">
    <h2 class="site-name">PlaySafe UK</h2>
    <div class="rating-badge">9.5/10</div>
    <div class="bonus-offer">100% up to £50</div>
    <div class="payout-speed"><span>Payout Speed:</span> 24 Hours (e-wallets)</div>
    <a href="/reviews/playsafe-uk" class="review-link">Read More</a>
  </div>
  <div class="review-card" id="site-2">
    <h2 class="site-name">Gambit Palace</h2>
    <div class="rating-badge">8.8/10</div>
    <div class="bonus-offer">Get 200 Free Spins</div>
    <div class="payout-speed"><span>Payout Speed:</span> 2-3 Days</div>
    <a href="/reviews/gambit-palace" class="review-link">Read More</a>
  </div>
</body>
</html>
"""

# Create a BeautifulSoup object to parse the HTML
soup = BeautifulSoup(html_content, 'html.parser')

print("Successfully parsed the HTML content.")
Enter fullscreen mode Exit fullscreen mode

Part 3: Extracting Data with BeautifulSoup

Now the fun begins. BeautifulSoup gives us powerful methods to find elements based on their tags, classes, or IDs. The most useful are find() (for one element) and find_all() (for multiple elements).

Let's start by isolating all the review cards.

review_cards = soup.find_all('div', class_='review-card')
print(f"Found {len(review_cards)} review cards.")
Enter fullscreen mode Exit fullscreen mode

Now, let's process the first card to figure out our extraction logic.

first_card = review_cards[0]

# --- Extract the Name ---
# The name is inside an <h2> tag with class 'site-name'
name_element = first_card.find('h2', class_='site-name')
# .text gets the text content of the element. .strip() removes whitespace.
name = name_element.text.strip()
print(f"Name: {name}")

# --- Extract the Rating ---
rating_element = first_card.find('div', class_='rating-badge')
rating = rating_element.text.strip()
print(f"Rating: {rating}")

# --- Extract the Bonus ---
bonus_element = first_card.find('div', class_='bonus-offer')
bonus = bonus_element.text.strip()
print(f"Bonus: {bonus}")

# --- Extract the Payout Speed ---
# This one is trickier. The text is "Payout Speed: 24 Hours (e-wallets)"
# We want to remove the "Payout Speed:" part.
payout_element = first_card.find('div', class_='payout-speed')
# We can find the <span> inside and remove it
payout_element.find('span').decompose() # This removes the tag and its content
payout_speed = payout_element.text.strip()
print(f"Payout Speed: {payout_speed}")
Enter fullscreen mode Exit fullscreen mode

This logic works perfectly for one card. Now, we just need to loop through all the cards we found.

Part 4: Scaling Up and Storing the Data

We'll create a loop and store our results in a list of dictionaries—a very standard and useful format. We'll also add some error handling with try-except blocks, because real-world HTML is messy and elements can be missing.

import pandas as pd

scraped_data = []

for card in review_cards:
    try:
        name = card.find('h2', class_='site-name').text.strip()
        rating = card.find('div', class_='rating-badge').text.strip()
        bonus = card.find('div', class_='bonus-offer').text.strip()

        payout_element = card.find('div', class_='payout-speed')
        payout_element.find('span').decompose()
        payout_speed = payout_element.text.strip()

        # Store the extracted data in a dictionary
        site_data = {
            'Name': name,
            'Rating': rating,
            'Bonus': bonus,
            'Payout Speed': payout_speed
        }
        scraped_data.append(site_data)

    except AttributeError:
        # This will catch errors if a tag is not found (e.g., a card is missing a rating)
        print("Skipping a card due to missing data.")
        continue

# Now, let's use Pandas to see our beautiful, structured data
df = pd.DataFrame(scraped_data)
print(df)
Enter fullscreen mode Exit fullscreen mode

Output:

          Name  Rating              Bonus           Payout Speed
0  PlaySafe UK  9.5/10    100% up to £50  24 Hours (e-wallets)
1  Gambit Palace  8.8/10  Get 200 Free Spins              2-3 Days
Enter fullscreen mode Exit fullscreen mode

Look at that! We've turned messy HTML into a clean, structured table. The final step is to save it.

# Save the DataFrame to a CSV file
df.to_csv('uk_review_data.csv', index=False)
print("\nData successfully saved to uk_review_data.csv")
Enter fullscreen mode Exit fullscreen mode

Conclusion: From a Simple Script to a Full-Scale Project

What we've built here is a simple, conceptual scraper. But this exact process is the foundation of any large-scale data analysis project in the consumer research space.

This tutorial mirrors the foundational work we do at the Casimo.org project. We take this methodology and apply it across the entire UK market, running automated scripts to gather, structure, and analyze tens of thousands of data points on everything from bonus terms to payout speeds. The goal is always the same: to turn a confusing market into a transparent, data-driven resource for players.

This script is the first step. The end result is a platform where players can make decisions based on data, not just marketing hype.

To see the results of this methodology applied at scale, you can explore the full data and reviews on our public research portal: Casimo.org.

In my next post, I'll show you how to take the uk_review_data.csv file we just created and build some powerful visualizations with Matplotlib and Seaborn.

Thanks for reading, and happy scraping!

Top comments (0)