This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.
TL;DR
Scrape Hacker News using AlterLab's API with Python or Node.js. Start at T1 tier, let the API auto-escalate if needed, and extract structured data via CSS selectors or Cortex. Respect rate limits and robots.txt.
Why collect tech data from Hacker News?
Hacker News aggregates real-time tech discussions, product launches, and industry sentiment. Practical use cases include:
- Tracking startup funding announcements and job postings for market research
- Monitoring technology trends by analyzing upvote patterns on specific topics
- Building competitor intelligence feeds by scraping links to rival products
Technical challenges
Hacker News implements standard anti-bot protections: rate limiting by IP, User-Agent header validation, and occasional JavaScript challenges for suspicious traffic. Raw HTTP requests (curl/urllib) frequently receive 429 or 403 responses. AlterLab's Smart Rendering API automates proxy rotation, header optimization, and tier escalation to maintain access while respecting site policies.
Quick start with AlterLab API
Begin with our Getting started guide. Here's how to fetch the Hacker News front page:
```python title="scrape_hackernews.py" {3-5}
client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://news.ycombinator.com")
print(response.text[:500]) # First 500 chars of HTML
```javascript title="scrape_hackernews.js" {3-5}
const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const response = await client.scrape("https://news.ycombinator.com");
console.log(response.text.slice(0, 500));
```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://news.ycombinator.com"}'
## Extracting structured data
Target these common elements using CSS selectors:
- Story titles: `.titleline > a`
- Scores: `.score`
- Author names: `.hnuser`
- Comment counts: `.age > a:nth-child(3)`
Example Python extraction:
```python title="extract_hn_titles.py"
from parsel import Selector
client = alterlab.Client("YOUR_API_KEY")
html = client.scrape("https://news.ycombinator.com").text
selector = Selector(text=html)
titles = selector.css(".titleline > a::text").getall()
print(f"Found {len(titles)} stories")
Structured JSON extraction with Cortex
For typed output without manual parsing, use Cortex AI extraction. Define a schema for story objects:
```python title="extract_hn_structured.py"
client = alterlab.Client("YOUR_API_KEY")
result = client.extract(
url="https://news.ycombinator.com",
schema={
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {"type": "string"},
"url": {"type": "string", "format": "uri"},
"score": {"type": "integer"},
"author": {"type": "string"}
},
"required": ["title", "url"]
}
}
)
print(result.data) # List of validated story objects
## Cost breakdown
Hacker News typically requires T2 (standard headers) or T3 (stealth) tiers due to anti-bot measures. AlterLab auto-escalates: start at T1, pay only for the tier that succeeds.
| Tier | Use Case | Cost per Request | Cost per 1,000 | Requests per $1 |
|------|----------|-----------------|----------------|------------------|
| T1 — Curl | Static HTML, no JS needed | $0.0002 | $0.20 | 5,000 |
| T2 — HTTP | Standard pages with headers | $0.0003 | $0.30 | 3,333 |
| T3 — Stealth | Protected pages, anti-bot active | $0.002 | $2.00 | 500 |
| T4 — Browser | Full JS rendering required | $0.004 | $4.00 | 250 |
| T5 — CAPTCHA | CAPTCHA solving + JS rendering | $0.02 | $20.00 | 50 |
See [AlterLab pricing](/pricing) for volume discounts. For most Hacker News scraping, expect $0.30-$2.00 per 1,000 requests.
## Best practices
- **Rate limiting**: AlterLab respects `Crawl-delay` in robots.txt. Add `wait_time=1` parameter for 1-second intervals between requests.
- **Robots.txt**: Hacker News allows scraping with `User-agent: *` and `Crawl-delay: 30`. Adjust frequency accordingly.
- **Dynamic content**: Use `render_js=true` for AJAX-loaded comments (triggers T4 tier only when necessary).
- **Error handling**: Implement exponential backoff for 429 responses. AlterLab auto-retries failed tiers.
## Scaling up
For large datasets:
1. **Batch requests**: Send 100 URLs per API call using `urls` array parameter
2. **Scheduling**: Use AlterLab's cron endpoint for daily/weekly scrapes
3. **Storage**: Stream results directly to S3 or your database via webhooks
4. **Responsibility**: Monitor response codes; pause if 4xx errors exceed 1%
<div data-infographic="stats">
<div data-stat data-value="99.2%" data-label="Success Rate"></div>
<div data-stat data-value="1.2s" data-label="Avg Response"></div>
<div data-stat data-value="$0.002" data-label="Per Request (T3)"></div>
</div>
<div data-infographic="steps">
<div data-step data-number="1" data-title="Configure headers" data-description="Set User-Agent and Accept-Language to mimic browser"></div>
<div data-step data-number="2" data-title="Handle pagination" data-description="Follow 'More' links while respecting crawl-delay"></div>
<div data-step data-number="3" data-title="Extract and validate" data-description="Use Cortex for schema-enforced JSON output"></div>
</div>
## Key takeaways
- AlterLab manages anti-bot challenges so you focus on data extraction
- Always verify public data accessibility and comply with robots.txt
- Use Cortex for type-safe JSON output instead of brittle CSS selectors
- Start scraping at T1 tier—pay only for what succeeds
- Scale responsibly with rate limiting and error handling
Related resource: [Hacker News scraping guide](/scrape/hacker-news)
Top comments (0)