Building a Hacker News Scraper with Python and BeautifulSoup

#webscraping #python #tutorial #beginners

Introduction

I built a Python web scraper that collects Hacker News stories and saves them to a CSV file.

The scraper collects:

Story titles
Story links
Story scores
Comment counts

Tools I Used

Python
Requests
BeautifulSoup
lxml
CSV

How It Works

First, I sent a request to the Hacker News website using the Requests library. Then I used BeautifulSoup to find the story information in the HTML page. Finally, I saved the extracted data into a CSV file.

Code

from bs4 import BeautifulSoup as bs
import requests
import csv

url = "https://news.ycombinator.com/news"
headers = {
    "User-Agent": "Mozilla/5.0"
}
source = requests.get(url, headers=headers)
source.raise_for_status()

soup = bs(source.content, "lxml")
stories = soup.find_all("tr", class_="athing")
data =[]

for story in stories:
    title_tag = story.find("span", class_="titleline").find("a")
    title = title_tag.text
    link = title_tag["href"]

    subtext = story.find_next_sibling("tr")
    score_tag = subtext.find("span", class_="score")
    score = score_tag.text if score_tag else "0 points"

    comments ="0 comments"
    comment_links = subtext.find_all("a")
    if comment_links:
        last = comment_links[-1].text
        if "comment" in last or "discuss" in last:
            comments = last

    data.append([title, score, comments, link])

with open("hacker_news.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow([
        "Title",
        "Score",
        "Comments",
        "Link"
    ])
    writer.writerows(data)

print(f"Successfully scraped {len(data)} stories")

What I Learned

This project helped me practice sending HTTP requests, parsing HTML, extracting data, and saving information to CSV.

What I Want to Improve

Next, I want to add pagination, improve error handling, and store the scraped data in a database.

You can find the complete project on my GitHub:
(https://github.com/logicshadowdev/web-scraping-project)

DEV Community