Avoid Manual HN Job Scraping with Python Automation

When you're hunting for remote developer jobs on Hacker News, the repetitive task of manually scraping job listings can eat into your time and reduce the quality of your search. This article walks you through a Python-based solution that automates the process of fetching and exporting Hacker News job postings—without any manual scraping. The technique is practical, and the script is ready to run, so you can focus on what matters: applying for jobs, not gathering them.

Hacker News is a popular platform for developers to find job opportunities, but the volume of posts can be overwhelming. Manually copying and pasting job details into spreadsheets is not only time-consuming but also error-prone. By automating this process, you can save hours each week and ensure your job tracking is consistent and reliable.

The script we'll walk through uses Python's requests and BeautifulSoup libraries to fetch job listings from Hacker News. It handles pagination, network errors, and exports the data into a clean CSV file. This script is a solid example of how to structure a web scraping automation task in Python, even if you're not using the tool specifically.

First, let's look at how to fetch the job listings from Hacker News. The script makes HTTP requests to the HN job page and parses the HTML to extract job details. Here's a basic version of the script that fetches the first page of job listings:

import requests
from bs4 import BeautifulSoup
import csv

def fetch_jobs():
    url = "https://news.ycombinator.com/jobs"
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    jobs = []

    for item in soup.select('.athing'):
        title = item.select_one('.titlelink').text.strip()
        company = item.select_one('.subtext a').text.strip()
        link = item.select_one('.titlelink')['href']
        jobs.append({
            'company': company,
            'role': title,
            'link': link
        })

    return jobs

jobs = fetch_jobs()
print(jobs)

This script fetches the first page of job listings and extracts the company name, job title, and link. It uses CSS selectors to find the relevant elements in the HTML. However, this is just the beginning of the automation process.

To make this script more robust, we can add support for pagination and error handling. Here's an enhanced version that fetches multiple pages and handles network errors gracefully:

import requests
from bs4 import BeautifulSoup
import csv
import time

def fetch_jobs(max_pages=5):
    url = "https://news.ycombinator.com/jobs"
    jobs = []
    for page in range(1, max_pages + 1):
        params = {'p': page}
        try:
            response = requests.get(url, params=params)
            response.raise_for_status()
            soup = BeautifulSoup(response.text, 'html.parser')
            for item in soup.select('.athing'):
                title = item.select_one('.titlelink').text.strip()
                company = item.select_one('.subtext a').text.strip()
                link = item.select_one('.titlelink')['href']
                jobs.append({
                    'company': company,
                    'role': title,
                    'link': link
                })
            time.sleep(1)  # Be polite to the server
        except requests.RequestException as e:
            print(f"Error fetching page {page}: {e}")
            continue
    return jobs

jobs = fetch_jobs()
print(jobs)

This version of the script fetches up to five pages of job listings and includes error handling for network issues. It also adds a delay between requests to avoid overwhelming the server. The script can be extended to export the data into a CSV file, which is a common format for job tracking.

If you're looking for a complete solution that includes exporting to CSV and handling all the edge cases, you can find the full script and setup instructions at HN Job Exporter. The tool is designed to save you time and reduce the manual effort involved in tracking jobs on Hacker News.