Simple Node.js task queue with bee-queue and redis

^{Cover Photo by Bimo Luki on Unsplash}

As you saw in the previous article that task queues are pretty awesome 🌟 and in this tutorial we would be using a task queue in our own application let's get our hands dirty and write some code.

We will be building our restaurant as explained in the previous article.

This tutorial would be much of a demonstration rather than a working application, so stick with me if you wanna check out how to plug in a task queue into your app.
In the next article we will be building a real application. (I know it's exciting and you can't wait for that 😉).

👨‍💻 Link to the entire github repo for this project is at the end of the article ⬇

Let's get started.

Pre-requisites

• Install Node.js on your machine, and then run the following command to verify proper installation.

$ node --version
> v12.16.1

• Redis running on your pc or the cloud. Install Redis or create an instance on RedisLabs for free.

And we're good to go 😁

Initialization

Run:

$ npm init

After that install the necessary packages by running

$ npm install express bee-queue dotenv

In case you are wondering what each package does, here's some info:

• express helps us to create a server and handle incoming requests with ease.
• bee-queue is our task queue manager and will help to create and run jobs
• dotenv helps us to load environment variables from a local .env file

After that create a file restaurant.js and edit your package.json so it looks something like this

{
  ...
  "main": "restaurant.js",
  "scripts": {
    "start": "node restaurant.js"
  }
  ...
}

Time for some real code

Open restaurant.js in the editor of your choice and add the following lines of code

require('dotenv').config();
const express = require('express');
const http = require('http');

// Inits
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

// Routes
app.get('/', (req, res) => {
    res.send("😋 We are serving freshly cooked food 🍲");
});


// Create and start the server
const server = http.createServer(app);
const PORT = process.env.PORT || 5000;
server.listen(PORT, () => {
    console.log(`Restaurant open at:${PORT}`);
});

What it does is basically start a local webserver at the specified port (here, 5000) and listens for incoming GET requests on the base url / and replies with a text.

Run the following command to start the server and head over to localhost:5000 in your browser.

$ npm start
> restaurant@1.0.0 start /mnt/code/dev/queue
> node restaurant.js

Restaurant open at port:5000

You will get a blank page with a neat little 😋 We are serving freshly cooked food 🍲 message

---

Now it's time to create our task queue

First create a file named .env and paste in it your database credentials like so, (You can use your local redis instance here as well) and remember, never to commit .env to your source control.

DB_HOST=redis-random-cloud.redislabs.com
DB_PORT=14827
DB_PASS=pTAl.not-my-password.rUlJq

And you are done with the basic configuration.

Let's go ahead and create our waiter. Start by creating a file waiter.js and add the following chunk of code:

const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);


const placeOrder = (order) => {
    return cookQueue.createJob(order).save();
};

serveQueue.process((job, done) => {
    console.log(`🧾 ${job.data.qty}x ${job.data.dish} ready to be served 😋`);
    // Notify the client via push notification, web socket or email etc.
    done();
})
    // Notify the client via push notification, web socket or email etc.
    done();
})


module.exports.placeOrder = placeOrder;

🤯 Whoa! what was that? Well, let me explain.

We first import the bee-queue package as Queue,
and then pass in the database configuration to our two new Queue objects. One of the queue will have the list of orders to be prepared by the cook and the other will have the list of orders that are ready to be served by the waiter.

We then create a new function placeOrder that takes in an order as the parameter. We will define this order object later, but keep in mind that it has a structure like this

order = {
    dish: "Pizza 🍕", 
    qty: 2,
    orderNo: "kbv9euic"
}

The placeOrder function takes this order and adds it to the queue by calling .createJob(order).save() method on the cookQueue Queue object. This acts as the task publisher.

and lastly the process method on serveQueue Queue object executes the handler function (job, done) => {...} everytime an order is prepared and ready to be served. This acts as the task consumer.

We call done() to acknowledge out task queue that the job in done so that it can send the next task to be processed from the queue. We simply call done() to indicate the task was successful and call done(err) i.e with the first parameter (where err is an error message) to indicate job failure. You can also call done(null, msg) to indicate job success with the second parameter msg being the success message.

And our waiter 👨‍💼 is ready

---

Now its time for the kitchen with the cooks 👨‍🍳
create another file kitchen.js and paste in it the following lines of code:

const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);

cookQueue.process(3, (job, done) => {
    setTimeout(() => console.log("Getting the ingredients ready 🥬 🧄 🧅 🍄"), 1000);
    setTimeout(() => console.log(`🍳 Preparing ${job.data.dish}`), 1500);
    setTimeout(() => {
        console.log(`🧾 Order ${job.data.orderNo}: ${job.data.dish} ready`);
        done();
    }, job.data.qty * 5000);
});

cookQueue.on('succeeded', (job, result) => {
    serveQueue.createJob(job.data).save();
});

😌 Well that looks familiar.

Yeah exactly, but the only change is that here our cooks are consuming from the cookQueue and publishing to the serveQueue for the waiters to take and serve the the orders.

One thing to note here is that, anything published via createJob(order) is available to the cosumer as job.data in the Queue.process() method's handler function (job, done) => {...}, and if you look closely, there's something different in cookQueue.process(3, (job, done) => {...}) too. Yeah, we pass in a number, before the actual handler function. It is known as concurrency (the number of tasks in the queue that can be processed simultaneously). Here we have set it to 3 because our kitchen has 3 cooks, who can work together.

And we use the cookQueue.on('succeeded', (job, result) => {...}) method to call the handler function whenever a task is successful (i.e. whenever you have called done() in the process() method).

Believe me we are almost done 🤞

---

Final step: Hook everything together

Open restaurant.js and add these final lines of code

// ...
// Add these lines before the Inits.
require('./kitchen');
const { placeOrder } = require('./waiter');

// Inits
// ...
// Routes

// ...

app.post('/order', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }

    if (order.dish && order.qty) {
        placeOrder(order)
            .then(() => res.json({ done: true, message: "Your order will be ready in a while" }))
            .catch(() => res.json({ done: false, message: "Your order could not be placed" }));
    } else {
        res.status(422);
    }
})

// Create and start the server
// ...

What we've done here is imported our kitchen and waiter and added a POST route /order to receive orders from our customers. Remember the order object?

order = {
    dish: "Pizza 🍕", 
    qty: 2,
    orderNo: "kbv9euic"
}

We are creating a order object from the JSON body of the POST request and passing it to our waiter and sending a JSON response to acknowledge our customer. In case the request is not properly made, we will send some error message also. And we're done ✌ .

---

Yeah really, we are done. Now it's time to test it out 😁

• Start the server by running $ npm start on your terminal.
• Send a get request to localhost:5000 and see if you get a response like this:
• Next send a POST request to localhost:5000/order and check the response and look at your console.

You can send multiple requests one after another to check that it does not hang to any request.

Let's add another POST route, to compare it with a normal restaurant without a task queue.

Add these lines to restaurant.js:

//  ...
app.post('/order-legacy', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }
    if (order.dish && order.qty) {
        setTimeout(() => console.log("Getting the ingredients ready... 🥬 🧄 🧅 🍄"), 1000);
        setTimeout(() => console.log(`🍳 Preparing ${order.dish}`), 1500);
        setTimeout(() => {
            console.log(`🧾 Order ${order.orderNo}: ${order.dish} ready`);
            res.json({ done: true, message: `Your ${order.qty}x ${order.dish} is ready` })
        }, order.qty * 5000);
    } else {
        console.log("Incomplete order rejected");
        res.status(422).json({ done: false, message: "Your order could not be placed" });
    }
});


// Create and start the server
// ...

• Next send a POST request to localhost:5000/order-legacy and check the response and look at your console.

Note the difference in the response time 🤯

With the task queue

Without task queue

---

Here is the Github repo, containing the complete project

sarbikbetal / nodejs-task-queue

This repo contains the sample code for the article "Simple Node.js task queue with bee-queue and redis"

Please comment down below if you have any questions or suggestions and feel free to reach me out 😄 and also check out the section below for Q&A.

📸Instagram	📨Email	👨‍💼LinkedIn	👨‍💻Github

---

🤔 Hmmm.. I have some questions though.

I know, so here are some common ones, feel free to ask more in the comments section below.

• How do we send the food to our customer once it is prepared?

  For that we need to implement some additional logic to our server side and client side application. Example of how we can achieve that, is through Websockets, push-notifications, emails, etc. Don't worry though I'll be covering that in details in the next article.

• Aren't there better things out there like RabbitMQ?

  Yeah sure there is, but for small scale projects that doesn't need a lot of advanced features but still want to maintain a decent back-end infrastructure RabbitMQ would be an overkill and bee-queue might just turn out simple and easy to use.

Comments

[Swarup Kumar Mahapatra]

Great start. But shouldn't your POST api return an orderId, so that customer can ask status of his order?

[Sarbik Betal]

Yeah you are right, this was a very basic example, but I will be adding it to this article. Thanks for your feedback 😊

[Sarbik Betal]

Swarup, I have updated the code in the Github repo, will be updating the article soon. Thanks for your nice idea.

[Swarup Kumar Mahapatra]

I have put my review comments in github github.com/sarbikbetal/nodejs-task....
Let me know what you think of it.
Cheers

[Sarbik Betal]

Thank you so much, I checked it out. I would always keep them in mind. 😊

[Kos-M]

Nice post!

• How do we send the food to our customer once it is prepared?
  • Instead of websockets , you can pick Server Sent Events , if clients needs only to read from server in one way channel fasion.

by the way : Came here searching for time consuning common tasks usually devs wants to run in nodejs.. Im developing a distributed queue/task runner , and searching for good use cases to apply as examples..
queueXec

[SUMAN DAS]

👍👍👍

[Joe Previte (he/him)]

Hey Sarbik! Great article. Do you plan to write a third answering "How do we send the food to our customer once it is prepared?"

[Simone Double B]

Very interesting article.
Looking forward to read the next one... Nice way to explain it with the analogy.
Would you give as well some additional use cases with real world examples?

[Sarbik Betal]

Yes Simone, that's what I would do in the next article 😁. We would build something interesting, Stay tuned 📻.

[Galih Anggara]

Wow,, great post, thanks for great explanation of task queue and the example project, love it.