DEV Community: Hazim J

I built a data platform to make finding datasets more accessible for everyone

Hazim J — Thu, 05 Nov 2020 09:19:05 +0000

Tl;dr: I started a project to try and make large, clean, and reliable datasets easier for anyone to share for free or through a monthly subscription.

Why build this?

There's no doubt that there is a lot of value in structured data, especially in large quantities. But finding the correct datasets you need can usually be pretty tough and sometimes a bit messy. We also hear a lot about how tech giants gain so much value from data science because of the sheer amount of data they have control over.

So I decided to build stackup.sh to try and solve these problems:

Make the user experience of finding a good dataset as simple as possible.
By making data more accessible this can hopefully make the value of data science more available to anyone.

What's next?

Right now the platform only has a single dataset available for tech news articles. I've used this to build a separate news feed for myself and also to learn some NLP algorithms. It's available for free and is updated every 30 minutes with new articles from a bunch of popular tech sources.

Over the next few weeks, I'll be sharing some more datasets I'm planning on using in other projects. If anyone is interested in helping me build out this platform by sharing their own data or requesting something they need, let me know!

How to schedule a task in Node.JS

Hazim J — Fri, 26 Jun 2020 11:49:49 +0000

Scheduling a task in node.js can be achieved in several different ways depending on your requirements. For this example, let's say we've created a library function to scrape data off a website and we wanted to run this on an hourly basis. We'll see how the solution to scheduling this task changes as the function becomes more robust. As with most things in software, we'll start trading off simplicity for a more scalable but complex solution

Running on a single node.js process

To start with let's assume the WebScraper library only fetches data from a single website and dumps it in a data store somewhere. This is quite a simple use case and we can probably get away with just using an open-source library like node-cron and running a single process.

Start by installing the dependencies:

npm install cron

Next, the entry point would look something like this:

const CronJob = require("cron").CronJob;
const WebScraper = require("./lib/webscraper");

const job = new CronJob("0 * * * *", WebScraper);

job.start();

Here we've defined the schedule using a crontab that calls the WebScraper function every hour.

Scaling to multiple jobs

Let's say you've now iterated over the WebScraper function a few times and added a feature where it accepts an arbitrary URL to crawl and fetch data from.

The same code above can be expanded for multiple jobs like this:

const CronJob = require("cron").CronJob;
const WebScraper = require("./lib/webscraper");

const URLS = [
  // list of urls to scrape...
];

const jobs = URLS.map((url) => {
  const job = new CronJob("0 0 * * *", () => WebScrapper(url));
  job.start();
  return job;
});

However, this isn't scalable as the number of jobs grows for a couple of reasons.

It's inefficient and hard to scale horizontally. As the number of jobs grows, the process they're running on will become a bottleneck. Eventually, you'll have to figure out how to run the jobs in parallel over multiple processes in order to complete them in a reasonable time.
Tracking and retying failures get tricky. As the number of jobs increases, it becomes more likely that some will fail intermittently. With our current approach, we have no way to easily track which jobs have failed and why. And if they failed we also have no way to retry them.

Using a job queue

To deal with the scaling problems we can look into using a task queue instead. A queue will have many jobs that can be distributed to a worker. Since they're stateless, the workers can also be scaled horizontally to multiple processes as required.

There are a few different libraries for implementing a task queue in Node.js, but for this example, we'll take a look at Bull. The message queue implemented by this library is also backed by Redis which handles the distribution of jobs to workers.

Start by installing the dependencies:

npm install bull

Following on from our last example we can set up and add jobs to the queue with the following code (this also assumes you have access to a Redis cluster):

const Queue = require("bull");

const webScraperQueue = new Queue("Fetch Data", process.env.REDIS_URL);

const URLS = [
  // list of urls to scrape...
];

URLS.forEach((url) =>
  webScraperQueue.add({ url }, { repeat: { cron: "0 0 * * *" } })
);

The worker code would then look something like this:

const Queue = require("bull");
const WebScraper = require("./lib/webscraper");

const webScraperQueue = new Queue("Fetch Data", process.env.REDIS_URL);

webScraperQueue.process("*", async (job) => {
  const { url } = job.data;
  const res = await WebScraper(url);

  return res;
});

Although slightly more complicated, this system is more scalable as the queue grows with more jobs. The two bottlenecks when scaling are:

The size of the queue which can be fixed by scaling up the Redis cluster.
The processing time which can be fixed by scaling up the number of worker processes.

When adding jobs to the queue we can also set additional options for retires in the case of an inevitable failed attempt.

Adding observability and making queues easier to manage

While the system can now be easily scaled, we still need a way to add some observability into what is happening in production. Information on the state of the jobs, stack traces, and logs are all things that can help us to determine the overall health of the queue while also being useful tools for debugging when needed.

Bull has a few different third-party UI options to enable observability:

All three options are open source and can be straightforward to integrate. Arena and bull-board can both be mounted to an existing express.js app while ZeroQueue is built with the aim of having as little code as possible. The latter comes with authentication and the ability to also create and manage queues from the UI rather than through code.

Summary

The simplest way to schedule a job in node.js would be to use an open-source library like node-cron.
As the number of jobs grows, you will likely want to transition to a task queue such as Bull to overcome processing bottlenecks and continue scaling.
Because of the added complexity, you'll also likely want to leverage a UI to easily manage your queues and get better observability into how your system is performing. For this, you can leverage dashboards such as arena, bull-board and zeroqueue.

Creating scheduled tasks in Node.js with ZeroQueue

Hazim J — Sun, 07 Jun 2020 01:26:37 +0000

Often times in your project you'll find a need to run certain tasks on a schedule. Whether it's every few minutes, hours, days, weeks, or even months. This is a perfect use case for a job queue which will often let you do just that by specifying the schedule through a crontab.

Here we'll see how you can easily do this with minimal code using ZeroQueue. As a simple example lets say you wanted to aggregate article links from multiple different news sites you follow on a daily basis and save them to an Airtable spreadsheet for easy access.

ZeroQueue Setup

Since a built docker image is currently hosted on Docker Hub, the app should easily work on many different platforms such as Kubernetes or Heroku. For this example we'll keep things simple and just run it locally with docker.

Backing services

To run ZeroQueue we'll first need a SQL database and Redis instance. A SQL database is used to track the different queues and users on the system while Redis is used to manage the jobs on a queue and distribute them to workers accordingly.

First, spin up a postgreSQL database with the following command:

docker run --name zeroqueue-db -e POSTGRES_USER=admin -e POSTGRES_PASSWORD=password -e POSTGRES_DB=zeroqueue_db -p 5432:5432 -d postgres

Next, spin up a Redis instance:

docker run --name zeroqueue-redis -p 6379:6379 -d redis

Starting a ZeroQueue web server

To start the web server we first need to run a migration on the database with the following command:

docker run --rm -e DATABASE_URL=postgres://admin:password@host.docker.internal:5432/zeroqueue_db zeroqueue/zeroqueue npm run db:sync

Once that is completed we can spin up the web server with:

docker run --rm -d -e DATABASE_URL=postgres://admin:password@host.docker.internal:5432/zeroqueue_db -e REDIS_URL=redis://host.docker.internal:6379 -e SESSION_SECRET=this-is-a-secret-value-with-at-least-32-characters -p 9376:9376 --name zeroqueue zeroqueue/zeroqueue

Once that's all done, you should be able to access ZeroQueue via http://localhost:9376 with the default credentials:

username: admin
password: password

Creating a queue

After logging into the dashboard click the "New Queue" button. We'll call our queue "Fetch Articles" and set the cron as 0 0 * * * to run daily.

Adding jobs

For this example lets aggregate articles from the RSS feeds of two common sources, Dev.to and HackerNews.

First create the following jobs.json file:

[
  {
    "name": "Dev.to",
    "data": {
      "source": "Dev.to",
      "url": "https://dev.to/feed/"
    }
  },
  {
    "name": "HackerNews",
    "data": {
      "source": "HackerNews",
      "url": "https://news.ycombinator.com/rss"
    }
  }
]

Next we'll add it to the queue by clicking into "Fetch Articles" on the dashboard and uploading the JSON file via the "New Jobs" button.

Adding workers

The final step is to now write some code for the workers. For each job, we want to fetch the RSS feed and push each item to an Airtable spreadsheet.

Start by installing the necessary dependancies:

npm install airtable bull rss-parser

Bull is the main library we need for the worker to be able to start processing jobs.

We can then create a file called worker.js with the following code:

const Queue = require("bull");
const Parser = require("rss-parser");
const Airtable = require("airtable");

const queue = new Queue("Fetch Articles", process.env.REDIS_URL);
const parser = new Parser();
const base = new Airtable({ apiKey: "API_KEY" }).base("BASE_ID");

queue.process("*", async (job, done) => {
  const { source, url } = job.data;

  try {
    const feed = await parser.parseURL(url);
    await Promise.all(
      feed.items.map((item) =>
        base("Articles").create(
          [
            {
              fields: {
                Name: item.title,
                Source: source,
                Link: item.link,
                Timestamp: new Date(item.pubDate).toISOString(),
              },
            },
          ],
          { typecast: true }
        )
      )
    );
  } catch (error) {
    job.log(error);
  }

  job.progress(100);
  done(null, null);
});

By running node worker.js, the above script will execute according to the schedule we previously set. Everyday the worker will process the jobs in the "Fetch Article" queue. For each job it will pull the RSS feed from the given url and dump the article links into an Airtable spreadsheet.

Summary

In summary we've managed to:

Setup a queue management system
Scheduled jobs
Added a worker

We've also managed to do this with less than 50 lines of code 🎉. You can also check out the example repo at https://github.com/thezeroqueue/zeroqueue-rss-example and the end result at https://airtable.com/shrcr2ihRSNWRBsPC.

If you're interested in using ZeroQueue for your own projects, consider starring it on GitHub at https://github.com/thezeroqueue/zeroqueue.

Note: although the worker is quite basic and won't cover a lot of edge cases such as de-duplication, the main point is that it can still be very easy to schedule tasks for anything when needed.

ZeroQueue: A low-code queue management system for Node.js

Hazim J — Fri, 05 Jun 2020 05:34:58 +0000

Link: https://github.com/thezeroqueue/zeroqueue

I recently created this open source project to simplify a lot of my other side projects that required the capability to schedule jobs. For context, I've been using Bull as my default job queue for awhile now. It's a really nice library, but I also felt like I was just copy pasting the same boiler plate code over and over again. So I wanted to see if there was an easier way to spin up and maintain a queue with the least amount of code possible.

This led me to building ZeroQueue so that I could focus my time on writing business logic code for the workers while just using a simple dashboard to handle everything else.

I wanted to share this here for other node.js devs incase it can help save some time in any of your projects too. It's completely free and open source if you want a self hosted solution. I'm also interested in building this out a bit more and would appreciate any feedback if you got it.