DEV Community

WaterShow
WaterShow

Posted on Originally published at watershowdigital.com

How to search Hacker News using the free Algolia API

Hacker News is useful to track trends, find mentions of your product, or read discussions. Scraping the HTML directly can get your IP address blocked.

You can use the free Algolia Hacker News API. It provides a JSON interface to search through stories and comments without needing an API key.

The base URL and endpoints

Algolia hosts the official search for Hacker News. The API has two main endpoints.

For the most relevant results for a keyword, use the standard search endpoint at https://hn.algolia.com/api/v1/search. For the newest mentions, use https://hn.algolia.com/api/v1/search_by_date. Both endpoints accept a query string parameter called query.

Filtering by stories or comments

Hacker News has different types of items like stories, comments, polls, and job postings. You can restrict your search using the tags parameter.

To search only the main submissions, add tags=story to your URL. To find mentions in threads, use tags=comment. You can also filter by a specific author using tags=author_username.

Setting a minimum point threshold

The API allows you to use numeric filters to return items that meet certain criteria.

Pass a numericFilters parameter to set conditions on fields like points or number of comments. Passing numericFilters=points>50 returns submissions with more than 50 points. You can chain these together by passing multiple filters separated by a comma.

A simple Node.js fetch script

Here is a TypeScript script that pulls recent stories mentioning a specific topic with more than 20 points. You can run this with Node.js.

async function searchHackerNews(keyword: string) {
  const baseUrl = 'https://hn.algolia.com/api/v1/search_by_date';
  const params = new URLSearchParams({
    query: keyword,
    tags: 'story',
    numericFilters: 'points>20'
  });

  const response = await fetch(`${baseUrl}?${params.toString()}`);

  if (!response.ok) {
    throw new Error(`API error: ${response.statusText}`);
  }

  const data = await response.json();

  for (const hit of data.hits) {
    console.log(`[${hit.points} pts] ${hit.title}`);
    console.log(`URL: ${hit.url || 'No external link'}`);
    console.log(`HN Link: https://news.ycombinator.com/item?id=${hit.objectID}\n`);
  }
}

searchHackerNews('supabase');
Enter fullscreen mode Exit fullscreen mode

Things that bit me

The API does not require authentication, but it does rate limit you based on your IP address. If you run a script that polls too aggressively, you will start getting 429 Too Many Requests errors.

Algolia caps the pagination at 1000 hits per query. If your search matches 50,000 comments, you can only page through the first 1000 results unless you narrow the time window using numeric filters on the created_at_i field.

The url field in the response is sometimes null. Text-only posts, like Ask HN or Show HN submissions, do not have an external link, so you have to handle that missing property in your code.

If you need help building scrapers or connecting APIs, check out other things I build.

FAQ

How do I search for newest results instead of relevant ones?

You use the search_by_date endpoint instead of the standard search endpoint. The base URL for this is https://hn.algolia.com/api/v1/search_by_date.

How do I find comments instead of stories?

You pass the tags parameter in your API request. Setting it to tags=comment will restrict the results to only return comments.

Why can I only see the first 1000 results?

Algolia restricts the pagination on this API to 1000 hits maximum per query. To see more, you must narrow down your search using date filters.


Originally published at watershowdigital.com/blog/search-hacker-news-algolia-api. Need a hand with your app or automation? WaterShow Digital fixes and builds it, fixed quote first.

Top comments (0)