DEV Community: Sarbik Betal

Simple Node.js task queue with bee-queue and redis

Sarbik Betal — Fri, 26 Jun 2020 16:27:53 +0000

As you saw in the previous article that task queues are pretty awesome 🌟 and in this tutorial we would be using a task queue in our own application let's get our hands dirty and write some code.

We will be building our restaurant as explained in the previous article.

This tutorial would be much of a demonstration rather than a working application, so stick with me if you wanna check out how to plug in a task queue into your app.
In the next article we will be building a real application. (I know it's exciting and you can't wait for that 😉).

👨‍💻 Link to the entire github repo for this project is at the end of the article ⬇

Let's get started.

Pre-requisites

Install Node.js on your machine, and then run the following command to verify proper installation. ```bash

$ node --version

v12.16.1

+ Redis running on your pc or the cloud. Install [Redis](https://redis.io/) or create an instance on [RedisLabs](https://redislabs.com/) for free.

And we're good to go :grin:

## Initialization
Run:
```bash


$ npm init

After that install the necessary packages by running



$ npm install express bee-queue dotenv

In case you are wondering what each package does, here's some info:

express helps us to create a server and handle incoming requests with ease.
bee-queue is our task queue manager and will help to create and run jobs
dotenv helps us to load environment variables from a local .env file

After that create a file restaurant.js and edit your package.json so it looks something like this



{
  ...
  "main": "restaurant.js",
  "scripts": {
    "start": "node restaurant.js"
  }
  ...
}

Time for some real code

Open restaurant.js in the editor of your choice and add the following lines of code



require('dotenv').config();
const express = require('express');
const http = require('http');

// Inits
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

// Routes
app.get('/', (req, res) => {
    res.send("😋 We are serving freshly cooked food 🍲");
});


// Create and start the server
const server = http.createServer(app);
const PORT = process.env.PORT || 5000;
server.listen(PORT, () => {
    console.log(`Restaurant open at:${PORT}`);
});

What it does is basically start a local webserver at the specified port (here, 5000) and listens for incoming GET requests on the base url / and replies with a text.

Run the following command to start the server and head over to localhost:5000 in your browser.



$ npm start
> restaurant@1.0.0 start /mnt/code/dev/queue
> node restaurant.js

Restaurant open at port:5000

You will get a blank page with a neat little 😋 We are serving freshly cooked food 🍲 message

Now it's time to create our task queue

First create a file named .env and paste in it your database credentials like so, (You can use your local redis instance here as well) and remember, never to commit .env to your source control.



DB_HOST=redis-random-cloud.redislabs.com
DB_PORT=14827
DB_PASS=pTAl.not-my-password.rUlJq

And you are done with the basic configuration.

Let's go ahead and create our waiter. Start by creating a file waiter.js and add the following chunk of code:




const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);


const placeOrder = (order) => {
    return cookQueue.createJob(order).save();
};

serveQueue.process((job, done) => {
    console.log(`🧾 ${job.data.qty}x ${job.data.dish} ready to be served 😋`);
    // Notify the client via push notification, web socket or email etc.
    done();
})
    // Notify the client via push notification, web socket or email etc.
    done();
})


module.exports.placeOrder = placeOrder;

🤯 Whoa! what was that? Well, let me explain.

We first import the bee-queue package as Queue,
and then pass in the database configuration to our two new Queue objects. One of the queue will have the list of orders to be prepared by the cook and the other will have the list of orders that are ready to be served by the waiter.

We then create a new function placeOrder that takes in an order as the parameter. We will define this order object later, but keep in mind that it has a structure like this



order = {
    dish: "Pizza 🍕", 
    qty: 2,
    orderNo: "kbv9euic"
}

The placeOrder function takes this order and adds it to the queue by calling .createJob(order).save() method on the cookQueue Queue object. This acts as the task publisher.

and lastly the process method on serveQueue Queue object executes the handler function (job, done) => {...} everytime an order is prepared and ready to be served. This acts as the task consumer.

We call done() to acknowledge out task queue that the job in done so that it can send the next task to be processed from the queue. We simply call done() to indicate the task was successful and call done(err) i.e with the first parameter (where err is an error message) to indicate job failure. You can also call done(null, msg) to indicate job success with the second parameter msg being the success message.

And our waiter 👨‍💼 is ready

Now its time for the kitchen with the cooks 👨‍🍳
create another file kitchen.js and paste in it the following lines of code:



const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);

cookQueue.process(3, (job, done) => {
    setTimeout(() => console.log("Getting the ingredients ready 🥬 🧄 🧅 🍄"), 1000);
    setTimeout(() => console.log(`🍳 Preparing ${job.data.dish}`), 1500);
    setTimeout(() => {
        console.log(`🧾 Order ${job.data.orderNo}: ${job.data.dish} ready`);
        done();
    }, job.data.qty * 5000);
});

cookQueue.on('succeeded', (job, result) => {
    serveQueue.createJob(job.data).save();
});

😌 Well that looks familiar.

Yeah exactly, but the only change is that here our cooks are consuming from the cookQueue and publishing to the serveQueue for the waiters to take and serve the the orders.

One thing to note here is that, anything published via createJob(order) is available to the cosumer as job.data in the Queue.process() method's handler function (job, done) => {...}, and if you look closely, there's something different in cookQueue.process(3, (job, done) => {...}) too. Yeah, we pass in a number, before the actual handler function. It is known as concurrency (the number of tasks in the queue that can be processed simultaneously). Here we have set it to 3 because our kitchen has 3 cooks, who can work together.

And we use the cookQueue.on('succeeded', (job, result) => {...}) method to call the handler function whenever a task is successful (i.e. whenever you have called done() in the process() method).

Believe me we are almost done 🤞

Final step: Hook everything together

Open restaurant.js and add these final lines of code



// ...
// Add these lines before the Inits.
require('./kitchen');
const { placeOrder } = require('./waiter');

// Inits
// ...
// Routes

// ...

app.post('/order', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }

    if (order.dish && order.qty) {
        placeOrder(order)
            .then(() => res.json({ done: true, message: "Your order will be ready in a while" }))
            .catch(() => res.json({ done: false, message: "Your order could not be placed" }));
    } else {
        res.status(422);
    }
})

// Create and start the server
// ...

What we've done here is imported our kitchen and waiter and added a POST route /order to receive orders from our customers. Remember the order object?



order = {
    dish: "Pizza 🍕", 
    qty: 2,
    orderNo: "kbv9euic"
}

We are creating a order object from the JSON body of the POST request and passing it to our waiter and sending a JSON response to acknowledge our customer. In case the request is not properly made, we will send some error message also. And we're done ✌ .

Yeah really, we are done. Now it's time to test it out 😁

Start the server by running $ npm start on your terminal.
Send a get request to localhost:5000 and see if you get a response like this:
Next send a POST request to localhost:5000/order and check the response and look at your console.

You can send multiple requests one after another to check that it does not hang to any request.

Let's add another POST route, to compare it with a normal restaurant without a task queue.

Add these lines to restaurant.js:



//  ...
app.post('/order-legacy', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }
    if (order.dish && order.qty) {
        setTimeout(() => console.log("Getting the ingredients ready... 🥬 🧄 🧅 🍄"), 1000);
        setTimeout(() => console.log(`🍳 Preparing ${order.dish}`), 1500);
        setTimeout(() => {
            console.log(`🧾 Order ${order.orderNo}: ${order.dish} ready`);
            res.json({ done: true, message: `Your ${order.qty}x ${order.dish} is ready` })
        }, order.qty * 5000);
    } else {
        console.log("Incomplete order rejected");
        res.status(422).json({ done: false, message: "Your order could not be placed" });
    }
});


// Create and start the server
// ...

Next send a POST request to localhost:5000/order-legacy and check the response and look at your console.

Note the difference in the response time 🤯

With the task queue

Without task queue

Here is the Github repo, containing the complete project

sarbikbetal / nodejs-task-queue

This repo contains the sample code for the article "Simple Node.js task queue with bee-queue and redis"

Please comment down below if you have any questions or suggestions and feel free to reach me out 😄 and also check out the section below for Q&A.

📸Instagram	📨Email	👨‍💼LinkedIn	👨‍💻Github

🤔 Hmmm.. I have some questions though.

I know, so here are some common ones, feel free to ask more in the comments section below.

How do we send the food to our customer once it is prepared?

For that we need to implement some additional logic to our server side and client side application. Example of how we can achieve that, is through Websockets, push-notifications, emails, etc. Don't worry though I'll be covering that in details in the next article.
Aren't there better things out there like RabbitMQ?

Yeah sure there is, but for small scale projects that doesn't need a lot of advanced features but still want to maintain a decent back-end infrastructure RabbitMQ would be an overkill and bee-queue might just turn out simple and easy to use.

Task queues and why do we need them.

Sarbik Betal — Sun, 21 Jun 2020 20:50:29 +0000

^{Cover photo: ©Unsplash/Camille Chen}

Some Background:

What is a task queue and why do you need it?

Analogy

Well, to answer that question let's consider a scenario.
There's a restaurant, and the restaurant has several employees (let's say 10) like waiters, chefs, cashier, receptionist, manager, etc. Now just recall what happens in a restaurant when you place your order.

You inform what you require 🗣️. (Request)
The waiter notes it down 📄, and assures you that your food will be ready in a while 🛎️. (Acknowledge)
The waiter passes you order to a chef 🧑‍🍳, and the chef adds it to the list of orders. (Enqueue)
Then the waiter goes to take orders from another customer 👥. (Next request).
Multiple chefs may be preparing the food 🥪 from the list of orders, one by one or may be many at a time ⌛. (Process)
After a while when your food is ready, the chef calls the waiter and passes the food 🥪. (Dequeue)
The waiter comes and serves you the food 😋. (Response)
Then the waiter goes to some another customer. (Next request).

The waiter and the chef are decoupled from one another, and the waiter takes orders and the chef prepares food independently.

Now imagine the same scenario where all the employees were capable of doing all kinds of jobs (take order, cook, etc.).
If that would have been the case, then the work flow would have changed to something like this.

A waiter arrives, takes your order 📄 and tells you that your food will be ready.
The same waiter goes to the kitchen 🏃 with your order and starts preparing them 🧑‍🍳.
When he/she is done preparing your food, comes back 🏃 and serves you the food 🥪.

You might not see much of a problem here, do you? Well think again, the restaurant has 10 employees only, what would happen if there are 20 or 25 customers waiting to order food?
The former way of handling the orders will easily deal with the pressure. But the latter would just break 🚧, because if all the employees are busy preparing food for the first 10 customers, who 👻 is gonna take orders from the remaining customers? And if the new customers are not addressed within a few minutes, they will surely leave 😠.

Where do we need them?

When we are building web applications/services 🖥️ that does some heavy-lifting in the server that takes time (anything over a few milliseconds) or is a long running job ⏱️ unlike simple CRUD operations like complex calculation, file handling or data analysis, we should always use a task queue. You can think of this as asynchrony (like Promises or Async-await in js) taken to the next level. This would help us to enqueue the task for processing and send the client some kind of acknowledgement immediately before we do the actual processing ⚙️ and move on to the next request (like the waiter). Another server (or maybe the same server which spins off another worker instance/process) would just check for the list 📃 if there is any pending task and process them (like the chef). Once it's done with a job, it will acknowledge the API server which would communicate to the client that the job is done ✔️ (through web-sockets, push notifications, emails or whatever implementation you could think of).

Now if it happens to process the job in one go with your API server (like the restaurant in the second case), things will get really sluggish ⏱️ because the server will take your request, process it, do the heavy lifting 🏋️(which takes time) and respond you back, all in one go. This means that the client would have to wait while the entire operation is complete and your browser will load on and on 🐌 till the server sends the response and if anyone sends a request in between would have to wait for the server to finish the first request before it can even address the second one and then send back the response. Now imagine the same case for thousands of requests per second, that would be really slow and painful and you can imagine that it would result in a very bad UX 🙅.

How do we make it work?

Before getting into the the details of using a task queue, let me introduce some of the terms used extensively over the context of this series.

Queue - Queues are like actual queues in which similar jobs/tasks are grouped together waiting to be processed by a worker in a FIFO (first in first out) manner.
Jobs/Tasks - They are the objects which contain the actual details about the job that is waiting to be processed.
Publisher - It is the one who adds the task in a queue.
Consumer - It watches the job queue for any pending job and sends it for processing.
Worker - The actual powerhouse which processes the job and notifies if it was successful or not. The worker logic can be housed inside of the consumer if you wish to do so.

Now that you have a basic overview, let's get into the details.

First we set up an API server with some endpoints which would respond to the client's HTTP requests.
The API server publishes the job to its respective queue and sends some kind of acknowledgement to the client like ```json

{
"job": "conversion",
"id": "dcj32q3",
"status": "ok"
}

or in case it fails
```json


{
  "job": "conversion",
  "id": "dcj32q5",
  "status": "failed",
  "reason": "auth_failed"
}

and closes the connection.

A consumer watches and consumes the queue and sends the task for processing to a worker.
The worker processes the job (one or many at a time), reports the progress in between (if it wishes to) and dispatches an event once it is done with the job. You may note that the task can fail at this stage also, so it dispatches a success or a failure event which can be handled accordingly.
The API server queries the progress and reports it to the client (through web-sockets or polling XHR/Fetch requests) so that the application can show a nice progress bar in the UI.
It also listens for the success or failure events and sends a notification to the client.
The client can now request the resource through another API call and the server responds with the requested resource to the client and closes the connection.

This way the clients are assured immediately that

Hey, I'm working on your job. I'll notify you once it is done, in the meanwhile you can do some other stuff.

and no one has to keep waiting for long and the server can efficiently handle more incoming requests.
The task queue essentially glues all these pieces (the API server and the workers) and makes them work together shifting the load from the API server to the worker and thus ensuring a much lower response time and lower down-time.

Conclusion

Hurray! 🎉, now you hopefully understand the basics of a task queue, why do we need them and what are its advantages ✨. If you think about it, this architecture is highly scalable (horizontally) and increased demand can be addressed by adding more worker processes.
I hope this post was helpful for beginners and if you liked this article please show some love, give it a 💗 and stay tuned 📻 for more.
Please comment down below if you have any questions or suggestions and feel free to reach me out 😄.

📸Instagram	📨Email	👨‍💼LinkedIn	👨‍💻Github

In the next article, we will see a step by step guide on how to setup a simple task queue in node js

Simple Node.js task queue with bee-queue and redis

Sarbik Betal ・ Jun 26 '20

#node #javascript #webdev #tutorial

DEV Community: Sarbik Betal

Simple Node.js task queue with bee-queue and redis

Let's get started.

Pre-requisites

Time for some real code

Now it's time to create our task queue

Final step: Hook everything together

Here is the Github repo, containing the complete project

sarbikbetal / nodejs-task-queue

This repo contains the sample code for the article "Simple Node.js task queue with bee-queue and redis"

🤔 Hmmm.. I have some questions though.

How do we send the food to our customer once it is prepared?

Aren't there better things out there like RabbitMQ?

Task queues and why do we need them.

Some Background:

What is a task queue and why do you need it?

Analogy

Where do we need them?

How do we make it work?

Conclusion

Simple Node.js task queue with bee-queue and redis

Sarbik Betal ・ Jun 26 '20