Export Hacker News stories to CSV with the official API

#python #api #beginners #opensource

Hacker News has a free, official API. You don't need to scrape the website. This post shows how to pull the top stories into a CSV file, filter them by keyword, and keep a feed running.

How the API works

Hacker News publishes its data through a Firebase-hosted API at https://hacker-news.firebaseio.com/v0/. Two calls cover most needs:

topstories.json returns up to 500 story ids. newstories.json, beststories.json, askstories.json, showstories.json and jobstories.json work the same way.
item/<id>.json returns one story: title, link, author, score, comment count and time.

The API documentation states no rate limit, but keep your requests polite.

The script

import csv
import requests

BASE = "https://hacker-news.firebaseio.com/v0"

ids = requests.get(f"{BASE}/topstories.json", timeout=20).json()[:50]

rows = []
for story_id in ids:
    item = requests.get(f"{BASE}/item/{story_id}.json", timeout=20).json()
    if not item or item.get("dead") or item.get("deleted"):
        continue
    rows.append({
        "id": item["id"],
        "title": item.get("title", ""),
        "url": item.get("url", f"https://news.ycombinator.com/item?id={item['id']}"),
        "author": item.get("by", ""),
        "score": item.get("score", 0),
        "comments": item.get("descendants", 0),
    })

with open("hn_top.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

print(f"Saved {len(rows)} stories")

Run it and you get hn_top.csv, ready for a spreadsheet.

Filter by keyword or score

Add one line inside the loop to keep only what you care about:

if "rust" not in item.get("title", "").lower() or item.get("score", 0) < 20:
    continue

Following a feed over time

A script like this returns the same stories until the list changes. To follow a feed, store the ids you have already seen, for example in a small JSON file, and skip them on the next run. Run the script from cron or Task Scheduler every hour.

Two details are easy to miss. Ask HN and job posts have no url, so fall back to the discussion link as the script does. And each feed lists at most 500 ids, so an hourly run is enough to keep up.

If you don't want to maintain it

I built the same thing as a small hosted tool, the Hacker News Stories Scraper on Apify. It handles the feed choice, keyword and score filters and a "new stories only" mode, and returns JSON, CSV or Excel. It costs $0.0001 per story plus $0.005 per run. The script above does the same job for free if you are happy to run it yourself.

If you want a field added or hit a problem, leave a comment or open an issue on the actor's page.

DEV Community