How I built a Hacker News scraper that pulled 1,000 posts in 10 seconds

Most HN scrapers I've seen over-engineer the problem. Proxy rotation, headless browsers, captcha solving — all to pull data that's already exposed through a free JSON API. Here's the version that takes 10 seconds and costs nothing

The API nobody charges for

Hacker News uses Algolia to power search, and Algolia exposes a public endpoint with no key required

https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=200

That's it. No auth. No rate limit. Returns clean JSON with title, URL, points, comments, author, timestamp — everything you need to analyze HN

The fetch (no Apify, no SDK, just urllib

`import json
import urllib.request
import time

stories = []
for page in range(10):
url = f"https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=100&page={page}"
req = urllib.request.Request(url, headers={"User-Agent": "hn-research/1.0"})
with urllib.request.urlopen(req) as r:
data = json.loads(r.read())
stories.extend(data.get("hits", []))
time.sleep(0.3) # be polite

print(f"Pulled {len(stories)} stories")`

Total runtime on my VPS: about 12 seconds for 1,000 posts. The 0.3s sleep is optional but polite — Algolia serves everyone from the same backend

What the data actually says

I ran the script for the last 7 days and bucketed by points

• Median points: 1. Half of all HN stories get exactly one upvote

• Only 1.5% reach 50 points — that's "visible" by HN's standards

• Zero posts crossed 500 points in the entire 7-day window

The top story of the week landed at 104 points (a Jane Street reverse engineering writeup). The HN virality curve is steep. If you're posting there hoping for traction, you're competing in a 1.5% funnel

When the data was posted (UTC

06:00 UTC 5 posts
07:00 UTC 35 posts
08:00 UTC 28 posts
09:00 UTC 27 posts
10:00 UTC 40 posts
11:00 UTC 46 posts
12:00 UTC 19 posts
(other hours: ~0)

Almost everything posts between 07:00 and 12:00 UTC (US morning). Counterintuitively, the best window to post is 06:00–07:00 UTC — you catch the morning traffic spike with less competition

Domain analysis

Top linked domains across 1,000 posts

twitter.com 45
github.com 29
openai.com 18
nytimes.com 13
anthropic.com 13

AI-related domains are over-represented. HN readers actively look for AI news, so if you're posting in that space, you get structural tailwind

Packaged as an Apify actor

I wrapped the script as an Apify actor so I can re-run it weekly without rewriting the boilerplate. The actor charges per result (PAY_PER_EVENT), so cost scales with what you actually pull

`// src/main.js — Apify actor (free Algolia endpoint)
import { Actor } from 'apify';

await Actor.main(async () => {
const input = await Actor.getInput() || {};
const queries = input.queries || [];
const tags = input.tags || 'story';
const maxResults = Math.min(500, input.maxResults || 100);

for (const q of queries) {
const url = https://hn.algolia.com/api/v1/search?query=${encodeURIComponent(q)}&tags=${tags}&hitsPerPage=${Math.min(100, maxResults)};
const res = await fetch(url, { headers: { accept: 'application/json' } });
const data = await res.json();
for (const hit of data.hits || []) {
await Actor.pushData({
title: hit.title || hit.story_title,
url: hit.url || hit.story_url,
author: hit.author,
points: hit.points,
comments: hit.num_comments,
createdAt: hit.created_at,
});
try { await Actor.charge({ eventName: 'result', count: 1 }); } catch {}
}
}
});`

The full pipeline

• Pull — fetch 1,000 fresh stories (10 seconds

• Score — bucket by points, comments, posting hour (Python pandas, 30 seconds

• Analyze — extract the patterns above (5 minutes of thinking

• Write up — turn the data into a Medium article (the long-form version

Total time: about an hour from API call to published article. Total cost: free (Algolia is free, the actor charges per-result on Apify

Why this matters

You don't need a scraping service to do competitive research on public forums. HN, Reddit, GitHub, Wikipedia, Stack Overflow — all of them have free public APIs or fetch-clean endpoints. The bottleneck isn't data access, it's analysis

The same approach works for any forum where posts are timestamped. The trick is: pick a 7-day window, pull everything, score by engagement, and look for the patterns. You don't need ML — a 50-line script does it

This is part of a broader weekly data routine I run — HN virality, GitHub trends, and (separately) the remote job market. If you like this style of low-cost data analysis, I publish a weekly report on remote hiring signals every Wednesday