Andy

Posted on Jul 31, 2023

Building an HackerNews to Discord notification system using Node.js, node-cron and discord-notification.

#javascript #node #beginners #webdev

TL;DR

In this article I am going to show how to build a simple service which monitors our HackerNews account and sends a notification to our Discord channel when there are new comments to our submissions.

We are going to use th following packages/technolgies:

Node.js / express
discord-notification
node-cron

Before we start… I have a favour to ask. 🤗

I am building an open source feature flags platform and if you could star the project on github it would really help me to grow the project and would allow me keep writing articles like this!

https://github.com/switchfeat-com/switchfeat

Setup the Express server

Let’s start creating a server folder and install express and some other depedencies we are going use:

cd server & npm init -y
npm install express cors typescript 
npm install @types/node @types/express @types/cors dotenv
npx tsc --init

Let’s create the index.ts file, and push the esieast Express configuration you can have in tyescript:

import express, { Express } from 'express';
import cors from "cors"; 

const app: Express = express();
const port = 4000;

app.use(express.urlencoded({ extended: false }));
app.use(express.json());
app.use(cors());

app.get("/api", (req, res) => {
    res.json({
        message: "(HackerNews -> Discord) notifier",
    });
});

app.listen(port, () => {
    console.log(`⚡️[server]: Server is running on port 4000`);
});

In package.json of the sever, we need to add a script to run it:

"scripts": {
    "start": "node index.js"
  }

To finish our setup, let’s compile typescript and run the server:

tsc & npm run start

Cool! Let’s go!

Installing server dependancies

In the server folder let’s install the dependancies we need to interact with both HackerNews API and Discord webhook.

npm i discord-notification node-cron @types/node-cron

Building the HackerNews API connection

The HackerNews API exposes a bunch of public endpoints, there is no authentication, so I would assume there is some sort of rate limits in place, so be careful not to overload it with requests. More information here.

Let’s create a simple abstraction layer which would hold the connection to HackerNews and run simple GET requests to their API. Having these calls in separate file enforces separation of concerns and allows to use other libraries in the future with minimal effort.

The hackerNews.ts file would look like this:

export const getUser = async (userName: string) => {
    const resp = await fetch(getUserURL(userName), {
        method: "GET",
        headers: {
            Accept: "application/json"
        }
    });
    return await resp.json();
};

export const getSubmission = async (submissionId: string) => {
    const resp = await fetch(getItemURL(submissionId), {
        method: "GET",
        headers: {
            Accept: "application/json"
        }
    });
    return await resp.json();
};

Let’s start building the scheduling logic, in a way that our app will automatically fetch the latest data from HN on a schedule. For this, we are going to use the node-cron library which is a very solid and mature library which allows to run different sorts of scheduling patterns.

We are going to setup the scheduler to check every 15min if there are new comments to our latest submission.

node-cron uses cron-tab based cron expressions, you can see how to write more complex once here.

Let’s create a scheduler.ts file as follows:

import * as cronJob from "node-cron";

export const initScheduledJobs = () => {
  const scheduledJobFunction = cronJob.schedule("*/15 * * * *", () => {
    // - intaract with HN API
    // - send nofitication to Discord
  });

  scheduledJobFunction.start();
}

Finally let’s instruct our express server to start the scheduler at startup in the index.ts:

import * as scheduler from "./scheduler";
scheduler.initScheduledJobs();

Great! Now that we have a scheduler in place, let’s add the HackerNews fetching logic to the initScheduledJobs function:

const userData = await hackerNews.getUser(userName);
console.log(userData);

Running this code, we are going to see the response from HN for my user data dev-bre on the console.

{
  created: 1587898826,
  id: 'dev-bre',
  karma: 23,
  submitted: [
    36846077, 36712635, 36712606, 36548356,
    36547441, 36147979, 36131968, 36131217,
    36131215, 36128924, 36022807, 34781854,
    34109805, 34109230, 34109200, 34105053,
    33909549, 33909537, 33887733, 33388763
  ]
}

The submitted field in the response contains an array of itemIds each one referring to a single submission I made, sorted by date. For the purpose of this article we are going only to monitor the latest submission, which means we are only interested in the first item in that array.

In order to get the list of comments for this post, we need to run an additional API call as follows:

const submissionData = await hackerNews.getSubmission(selectedPost);

The HN API returns the data in the form of a Tree datastructure, which means this API response would contain a field: kids in the form of an array of numbers which are the direct children of the current item, i.e. direct comments to the requested comment. In order to traverse the entire set of comments, on each level we need to use recursion.

A recursive approach to get HN data

In order to get all comments to a post on any level, we need to go with a recursive approach!

Without going too much into the details of how recursion works, in a nutshell a recursive function is a function which calls itself progressively on a smaller data sets, until it reaches the base case which allows the function to return.

In our scenario, the base case is when we find a comment with no children (no one replied to that comment). Here it is how our function would look like:

const callRecursively = 
    async (node: ResultType, results: ResultType[]): Promise<void> => {

  // call HN for the current node
    let curr = await hackerNews.getSubmission(node.id);
    const kids = curr.kids as number[];
    results.push({ id: node.id, text: node.text, time: node.time });

  // check kids and go recursive
    if (kids && kids.length > 0) {
        for (let x = 0; x < kids.length; x++) {
            const childNode = {
                id: kids[x],
                text: curr.text,
                time: curr.time
            };
            await callRecursively(childNode, results);
        }
    }
};

At the end of the processing, the results array is going to contain all the comments for the requested post.

Next step is to keep track of the previous execution and compare the current execution with the previous one. If the two arrays are different, then we have updates and we need to send a discord notification.

Here it is how all this is going to be integrated with the rest of the app we are builduing:

    console.log("Starting check HackerNews -> Discord");
    const currentResults = await grabContentFromHN(
       userName, 
       selectedPostId);
    console.log(JSON.stringify(currentResults));

    // Compare the latest results with the previous run, 
    // if there is a change, then there are updates to this post.
    const updateAvailable = compareWithLastRun(currentResults);

    if (updateAvailable) {
        console.log("New updates available!");
    // notify Discord! 
    }

    // update the global state
    globalResults = [...currentResults];

The super simple comparing logic simply converts the 2 arrays in JSON strings and use the string comparator to do the work. This not the best solution, simply because it's not the fastest, but does the job:

const compareWithLastRun = 
    (currentResults: ResultType[]) : boolean => {
    if (globalResults.length < currentResults.length) {
        return true;
    }

    if (JSON.stringify(globalResults) !== JSON.stringify(currentResults)) {
        return true;
    }

    return false;
}

For the purpose of this article we are going to save the state of the previous run locally, which means everytime we restart our app, we are going to receive an intial notification, the first execution sets the baseline. A more elegant approach would save the previous run in a storage instead.

Discord integration

In order to send Discord notifications, we are going to use the discord-notification library, this is not a library which covers every single feature that the Discord API offers, but it is good enough for our usecase.

Before using the library we need to create a new Webhook on Discord. I am not going into the details on how to create a webhook, the Discord documentation is really well done. Here it is the link.

Now that we have our webhook, let’s save it in the .env file of our project. The .env should have this structure:

HN_USERNAME=
HN_POST_ID=
DISCORD_WEBHOOK=

Let’s create a new file discordNotifier.ts and let’s push this code in it.

const discordWebHook = process.env.DISCORD_WEBHOOK as string;
export const discordNotification = 
     new DiscordNotification('HN-Notifier', discordWebHook);

export const notifyDiscord = (message: string, 
     postId: string, 
     postText: string, 
     comments: number) : Promise<void> => {
    return discordNotification
    .sucessfulMessage()
    .addTitle(message)
    .addDescription(`[Link to the post](https://news.ycombinator.com/item?id=${postId})`)
    .addContent(postText)
    .addField({name: 'postId', value: postId, inline: true })
    .addField({name: 'comments', value: comments.toString() }) 
    .sendMessage();
};

The code above receives in input the notification data and uses the Webhook to send the notification. Couldn’t be easier than this!

Finally let’s integrate this logic in our scheduler, simply using this function like this:

if (updateAvailable) {
    console.log("New updates available!");
    // notify Discord!
    await discordNotifier.notifyDiscord(
            "New comments(s) available!", 
            selectedPostId, 
            currentResults[0].text, 
            currentResults.length - 1);
}

Here it is how our new shiny Discord notification will look like!

Well done!

We now have fully working Hacker News monitoring system and we won’t lose comments anymore!

We could expand on this putting additional data into the notification, like, what was the text of the latest comment, the time when the latest comment has been published, etc. This system we just built is flexible enough to allow these chanegs with minimal effort.

Github link here!.