DEV Community

IntelliTools
IntelliTools

Posted on

Automate HN Job Tracking with Python and CSV Export

#python #automation #tutorial #productivity 
import requests
from bs4 import BeautifulSoup
import csv

url = 'https://news.ycombinator.com/jobs'
response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')
job_listings = soup.find_all('tr', class_='athing')

jobs = []
for job in job_listings:
    title = job.find('span', class_='titleline').text.strip()
    link = job.find('span', class_='titleline').find('a')['href']
    jobs.append({'title': title, 'link': link})

with open('hn_jobs.csv', 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ['title', 'link']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for job in jobs:
        writer.writerow(job)
Enter fullscreen mode Exit fullscreen mode

Tracking job listings on Hacker News can be tedious, especially when manually sifting through threads for relevant opportunities. The HN Job Scraper is a Python tool designed to automate this process, saving developers hours of manual work each week. It extracts job titles, company names, locations, and links from Hacker News job listings and exports them to a clean CSV file. This makes it easy to stay updated on remote development opportunities without having to scroll endlessly through threads.

The tool is built using Python and leverages two popular libraries: requests for fetching the HTML content of Hacker News job pages and beautifulsoup4 for parsing the HTML and extracting the relevant data. Once the data is scraped, it's exported to a CSV file with columns for title, company, location, and link. This structured format allows for easy integration with job tracking tools or manual review.

# Example of scraping a single job listing
job = job_listings[0]
title = job.find('span', class_='titleline').text.strip()
link = job.find('span', class_='titleline').find('a')['href']
print(f"{title} - {link}")
Enter fullscreen mode Exit fullscreen mode

The script includes robust error handling and retry logic to ensure reliable execution, even when faced with network issues or changes in the structure of the Hacker News website. This reliability is crucial for maintaining an up-to-date job list, especially for remote developers who rely on these opportunities to find work.

To use the HN Job Scraper, you'll need Python 3.7 or later. The script is designed to be run with minimal setup—just install the required dependencies using pip install requests beautifulsoup4 and run the script. The tool also includes a README.txt file with step-by-step instructions and a sample_output.txt to show exactly what the script produces. This makes it easy for new users to get started and understand the output format.

For developers who want to track job listings efficiently and avoid the manual sifting process, the HN Job Scraper is an invaluable tool. It's ideal for remote developers and job seekers who want to stay on top of the latest opportunities without spending excessive time on manual tasks. The script's clean output and ease of use make it a great addition to any developer's automation toolkit.

If you're looking for a simple yet powerful way to automate HN job tracking, you can find the tool at https://intellitools.gumroad.com/l/hn-job-scraper. The tool is lightweight, easy to use, and designed to save you time while keeping you informed about the latest remote job opportunities.

Top comments (0)