You're building a React analytics tool. A major incident hits HN with the title "React Performance Issues at Scale: Here's What We Found" and it hits #3. By the time you see it, 8 hours later, the conversation is 200 comments deep and you've missed the window to contribute meaningfully.
Or worse, you don't see it at all, and miss a critical discussion about your product space.
Hacker News moves fast. Trends emerge, discussions peak, and fade within hours. Manual monitoring means either constantly refreshing the site or missing valuable signals about your market, competitors, and community sentiment.
What if instead you could set up a system that automatically monitors HN, filters for topics relevant to your niche, identifies viral posts early, and sends you real-time alerts? That's the power of an automated HN tracker.
In this post, I'll walk you through building one that catches the signals that matter for your business.
Why Automated HN Monitoring Matters
Hacker News is where engineers congregate. If your product is developer-focused, your market is here. But it's not just about finding mentions of your product. It's about understanding:
What problems are engineers actively discussing? If 500 engineers are discussing "database migration horror stories" in a single thread, that's market research. Maybe your tool should position around frictionless migrations.
How fast do certain topics trend? Posts about AI security shoot to #1 in 3 hours. Traditional cloud infrastructure discussions slowly climb over days. Understanding velocity tells you where excitement is building.
What are the sentiment patterns? A post about a competitor hits #5. Read the comments. Are people praising them or criticizing them? That's competitive intel.
Where are discussion gaps? No one's talking about a feature you know matters. Either the market doesn't care yet, or there's an education opportunity.
How early can you detect trends? A post about Rust for systems programming sits at #50 with 20 comments. Six hours later, it's #2 with 400 comments. Catching it at #50 means you can be an early voice, shaping the narrative.
Manual monitoring can't scale to capture this. Automated monitoring can.
Setting Up Your Hacker News Scraper
Start by understanding what you want to track. For a developer tools company, I'd monitor:
- Posts about the languages/frameworks you support
- Posts about your competitors
- Posts about the problems you solve
- General trends in your category (databases, APIs, testing, etc.)
Use the Apify Hacker News Scraper to extract the front page and top stories on a regular schedule.
Your basic input configuration:
{
"startUrls": [
{
"url": "https://news.ycombinator.com/newest"
},
{
"url": "https://news.ycombinator.com/best"
}
],
"maxPostsPerPage": 30,
"scrapeComments": true,
"maxComments": 20,
"pageTimeout": 30000,
"useChrome": true,
"proxyConfiguration": {
"useApifyProxy": true
}
}
This pulls the newest and best posts with top comments, giving you context.
Understanding Your HN Scraper Output
Here's what a typical run returns (abbreviated):
{
"posts": [
{
"postId": "41234567",
"title": "We Rewrote Our Backend in Rust and Gained 10x Performance",
"url": "https://example.com/rust-backend",
"author": "john_eng",
"score": 1247,
"commentCount": 342,
"timestamp": "2026-04-04T14:23:00Z",
"category": "full-stack",
"rank": 1,
"comments": [
{
"commentId": "41234568",
"author": "sarah_dev",
"text": "Did you consider Go instead? Rust has a steep learning curve for teams.",
"score": 245,
"timestamp": "2026-04-04T14:35:00Z",
"depth": 1
},
{
"commentId": "41234569",
"author": "john_eng",
"text": "Go wasn't fast enough for our use case. Rust gave us the performance we needed without GC pauses.",
"score": 189,
"timestamp": "2026-04-04T14:42:00Z",
"depth": 2
}
]
},
{
"postId": "41234570",
"title": "PostgreSQL Sharding Lessons from Running 1000 Databases",
"url": "https://example.com/postgres-sharding",
"author": "db_expert",
"score": 892,
"commentCount": 156,
"timestamp": "2026-04-04T12:11:00Z",
"category": "databases",
"rank": 2,
"comments": [
{
"commentId": "41234571",
"author": "ops_person",
"text": "Good breakdown. We went with CockroachDB instead for similar guarantees.",
"score": 134,
"timestamp": "2026-04-04T12:45:00Z",
"depth": 1
}
]
},
{
"postId": "41234571",
"title": "Why We're Migrating Away from Kubernetes",
"url": "https://example.com/k8s-migration",
"author": "infra_lead",
"score": 654,
"commentCount": 201,
"timestamp": "2026-04-04T10:05:00Z",
"category": "infrastructure",
"rank": 3,
"comments": [
{
"commentId": "41234572",
"author": "k8s_maintainer",
"text": "What specific pain points drove the decision? K8s can be overkill for many orgs.",
"score": 178,
"timestamp": "2026-04-04T10:22:00Z",
"depth": 1
}
]
}
],
"scrapingRunId": "run-20260404-1",
"scrapedAt": "2026-04-04T15:00:00Z",
"totalPostsScraped": 30
}
Notice the structure: posts have scores and comment counts, making it easy to identify which discussions are heating up.
Building Topic Filtering and Alerts
Raw data isn't actionable. You need filtering. Let's say you're a DevOps tool company focused on Kubernetes. Create a filter that catches relevant posts:
const filterRelevantPosts = (posts) => {
const relevantKeywords = [
'kubernetes', 'k8s', 'docker', 'container', 'devops',
'infrastructure', 'deployment', 'orchestration', 'microservices'
];
const competitorKeywords = [
'hashicorp', 'terraform', 'aws', 'gke', 'eks'
];
return posts.filter(post => {
const titleLower = post.title.toLowerCase();
const isRelevant = relevantKeywords.some(kw => titleLower.includes(kw));
const isCompetitorMention = competitorKeywords.some(kw => titleLower.includes(kw));
return isRelevant || isCompetitorMention;
}).map(post => ({
...post,
relevanceScore: calculateRelevance(post),
sentimentTrend: analyzeSentiment(post.comments)
}));
};
const calculateRelevance = (post) => {
// Score based on engagement and recency
const ageMinutes = (Date.now() - new Date(post.timestamp).getTime()) / (1000 * 60);
const engagementScore = post.score + (post.commentCount * 5);
const velocityBoost = ageMinutes < 60 ? 1.5 : 1.0; // Recent posts weighted higher
return Math.round(engagementScore * velocityBoost);
};
const analyzeSentiment = (comments) => {
const positive = comments.filter(c => c.text.includes('great') || c.text.includes('excellent')).length;
const negative = comments.filter(c => c.text.includes('bad') || c.text.includes('issue')).length;
return {
positiveCount: positive,
negativeCount: negative,
trend: positive > negative ? 'positive' : negative > positive ? 'negative' : 'neutral'
};
};
Running this filter on your scraped data returns only the posts that matter to your niche, scored by relevance.
Setting Up Apify Schedules and Webhooks
Don't run this manually. Schedule your scraper to run every 2 hours during business hours:
- Go to the Apify Hacker News Scraper
- Create a scheduled task (Schedules tab)
- Set frequency: "0 /2 __ " (every 2 hours)
- Configure a webhook to POST results to your backend
Your webhook endpoint receives the fresh data and immediately:
- Filters for relevant posts
- Compares against the previous run (has post ranking changed?)
- Identifies newly trending posts (went from #50 to #5)
- Routes high-relevance posts to Slack
Slack Integration: Real-Time Alerts
Here's how to send filtered posts to Slack:
const sendSlackAlert = async (post, relevanceScore) => {
const webhookUrl = process.env.SLACK_WEBHOOK_URL;
const color = relevanceScore > 500 ? '#ff6b35' : '#4CAF50';
const emoji = post.sentimentTrend === 'positive' ? ':thumbsup:' : ':warning:';
const slackPayload = {
attachments: [
{
color: color,
title: `${emoji} ${post.title}`,
title_link: `https://news.ycombinator.com/item?id=${post.postId}`,
fields: [
{
title: 'Score',
value: `${post.score}`,
short: true
},
{
title: 'Comments',
value: `${post.commentCount}`,
short: true
},
{
title: 'Relevance',
value: `${relevanceScore}/1000`,
short: true
},
{
title: 'Sentiment',
value: `Positive: ${post.sentimentTrend.positiveCount}, Negative: ${post.sentimentTrend.negativeCount}`,
short: true
}
],
footer: `Posted by ${post.author} at ${new Date(post.timestamp).toLocaleTimeString()}`
}
]
};
await fetch(webhookUrl, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(slackPayload)
});
};
Now your team gets a Slack notification within 5 minutes of a relevant post going viral. You can comment early, get visibility, build credibility.
Tracking Trends Over Time
To identify emerging trends, compare multiple runs:
const identifyTrendingTopics = (currentRun, previousRun) => {
const currentPosts = new Map(currentRun.posts.map(p => [p.postId, p]));
const previousRanks = new Map(previousRun.posts.map((p, idx) => [p.postId, idx]));
const trending = [];
currentRun.posts.forEach((post, currentRank) => {
const previousRank = previousRanks.get(post.postId);
if (previousRank !== undefined) {
const rankChange = previousRank - currentRank;
const scoreChange = post.score - (currentPosts.get(post.postId)?.score || 0);
if (rankChange > 5 || scoreChange > 100) {
trending.push({
postId: post.postId,
title: post.title,
rankImprovement: rankChange,
scoreGain: scoreChange,
currentRank: currentRank
});
}
}
});
return trending.sort((a, b) => b.rankImprovement - a.rankImprovement);
};
Output shows which posts are accelerating, helping you spot trends early.
Real-World Application: Building a Competitive Moat
Imagine you work on a database company. Your HN tracker catches a post: "We Switched from PostgreSQL to TimescaleDB for Time-Series Data" at rank #40 with 15 comments.
You're notified immediately. You read the discussion. The author mentions pain points you've solved in your product. You comment thoughtfully, sharing relevant experience. Your comment gets upvoted. You gain visibility among your target market.
By 2 PM, the post is #5 with 200 comments. You're one of the visible voices in the conversation. That's brand building.
Meanwhile, competitors who monitor HN manually missed it entirely. They're not in the conversation. You captured the moment.
Scale this across dozens of relevant posts per week. Over months, your team becomes the go-to voice in critical discussions. That's how you build a technical moat.
Getting Started Today
Head to Apify Hacker News Scraper and run one manual scrape. See what posts you find. Then set up a 2-hour schedule.
Connect it to a Slack webhook. Within a day, you'll be seeing HN trends as they emerge. Your competitive advantage depends on moving faster than the market. Automation gives you that speed.
Top comments (0)