loading...
Cover image for Simple Node.js task queue with bee-queue and redis

Simple Node.js task queue with bee-queue and redis

sarbikbetal profile image Sarbik Betal ใƒป7 min read

Cover Photo by Bimo Luki on Unsplash

As you saw in the previous article that task queues are pretty awesome ๐ŸŒŸ and in this tutorial we would be using a task queue in our own application let's get our hands dirty and write some code.

We will be building our restaurant as explained in the previous article.

This tutorial would be much of a demonstration rather than a working application, so stick with me if you wanna check out how to plug in a task queue into your app.
In the next article we will be building a real application. (I know it's exciting and you can't wait for that ๐Ÿ˜‰).

๐Ÿ‘จโ€๐Ÿ’ป Link to the entire github repo for this project is at the end of the article โฌ‡

Let's get started.

Pre-requisites

  • Install Node.js on your machine, and then run the following command to verify proper installation.
$ node --version
> v12.16.1
  • Redis running on your pc or the cloud. Install Redis or create an instance on RedisLabs for free.

And we're good to go ๐Ÿ˜

Initialization

Run:

$ npm init

After that install the necessary packages by running

$ npm install express bee-queue dotenv

In case you are wondering what each package does, here's some info:

  • express helps us to create a server and handle incoming requests with ease.
  • bee-queue is our task queue manager and will help to create and run jobs
  • dotenv helps us to load environment variables from a local .env file

After that create a file restaurant.js and edit your package.json so it looks something like this

{
  ...
  "main": "restaurant.js",
  "scripts": {
    "start": "node restaurant.js"
  }
  ...
}

Time for some real code

Open restaurant.js in the editor of your choice and add the following lines of code

require('dotenv').config();
const express = require('express');
const http = require('http');

// Inits
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

// Routes
app.get('/', (req, res) => {
    res.send("๐Ÿ˜‹ We are serving freshly cooked food ๐Ÿฒ");
});


// Create and start the server
const server = http.createServer(app);
const PORT = process.env.PORT || 5000;
server.listen(PORT, () => {
    console.log(`Restaurant open at:${PORT}`);
});

What it does is basically start a local webserver at the specified port (here, 5000) and listens for incoming GET requests on the base url / and replies with a text.

Run the following command to start the server and head over to localhost:5000 in your browser.

$ npm start
> restaurant@1.0.0 start /mnt/code/dev/queue
> node restaurant.js

Restaurant open at port:5000

You will get a blank page with a neat little ๐Ÿ˜‹ We are serving freshly cooked food ๐Ÿฒ message


Now it's time to create our task queue

First create a file named .env and paste in it your database credentials like so, (You can use your local redis instance here as well) and remember, never to commit .env to your source control.

DB_HOST=redis-random-cloud.redislabs.com
DB_PORT=14827
DB_PASS=pTAl.not-my-password.rUlJq

And you are done with the basic configuration.

Let's go ahead and create our waiter. Start by creating a file waiter.js and add the following chunk of code:


const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);


const placeOrder = (order) => {
    return cookQueue.createJob(order).save();
};

serveQueue.process((job, done) => {
    console.log(`๐Ÿงพ ${job.data.qty}x ${job.data.dish} ready to be served ๐Ÿ˜‹`);
    // Notify the client via push notification, web socket or email etc.
    done();
})
    // Notify the client via push notification, web socket or email etc.
    done();
})


module.exports.placeOrder = placeOrder;

๐Ÿคฏ Whoa! what was that? Well, let me explain.

We first import the bee-queue package as Queue,
and then pass in the database configuration to our two new Queue objects. One of the queue will have the list of orders to be prepared by the cook and the other will have the list of orders that are ready to be served by the waiter.

We then create a new function placeOrder that takes in an order as the parameter. We will define this order object later, but keep in mind that it has a structure like this

order = {
    dish: "Pizza ๐Ÿ•", 
    qty: 2,
    orderNo: "kbv9euic"
}

The placeOrder function takes this order and adds it to the queue by calling .createJob(order).save() method on the cookQueue Queue object. This acts as the task publisher.

and lastly the process method on serveQueue Queue object executes the handler function (job, done) => {...} everytime an order is prepared and ready to be served. This acts as the task consumer.

We call done() to acknowledge out task queue that the job in done so that it can send the next task to be processed from the queue. We simply call done() to indicate the task was successful and call done(err) i.e with the first parameter (where err is an error message) to indicate job failure. You can also call done(null, msg) to indicate job success with the second parameter msg being the success message.

And our waiter ๐Ÿ‘จโ€๐Ÿ’ผ is ready


Now its time for the kitchen with the cooks ๐Ÿ‘จโ€๐Ÿณ
create another file kitchen.js and paste in it the following lines of code:

const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);

cookQueue.process(3, (job, done) => {
    setTimeout(() => console.log("Getting the ingredients ready ๐Ÿฅฌ ๐Ÿง„ ๐Ÿง… ๐Ÿ„"), 1000);
    setTimeout(() => console.log(`๐Ÿณ Preparing ${job.data.dish}`), 1500);
    setTimeout(() => {
        console.log(`๐Ÿงพ Order ${job.data.orderNo}: ${job.data.dish} ready`);
        done();
    }, job.data.qty * 5000);
});

cookQueue.on('succeeded', (job, result) => {
    serveQueue.createJob(job.data).save();
});

๐Ÿ˜Œ Well that looks familiar.

Yeah exactly, but the only change is that here our cooks are consuming from the cookQueue and publishing to the serveQueue for the waiters to take and serve the the orders.

One thing to note here is that, anything published via createJob(order) is available to the cosumer as job.data in the Queue.process() method's handler function (job, done) => {...}, and if you look closely, there's something different in cookQueue.process(3, (job, done) => {...}) too. Yeah, we pass in a number, before the actual handler function. It is known as concurrency (the number of tasks in the queue that can be processed simultaneously). Here we have set it to 3 because our kitchen has 3 cooks, who can work together.

And we use the cookQueue.on('succeeded', (job, result) => {...}) method to call the handler function whenever a task is successful (i.e. whenever you have called done() in the process() method).

Believe me we are almost done ๐Ÿคž


Final step: Hook everything together

Open restaurant.js and add these final lines of code

// ...
// Add these lines before the Inits.
require('./kitchen');
const { placeOrder } = require('./waiter');

// Inits
// ...
// Routes

// ...

app.post('/order', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }

    if (order.dish && order.qty) {
        placeOrder(order)
            .then(() => res.json({ done: true, message: "Your order will be ready in a while" }))
            .catch(() => res.json({ done: false, message: "Your order could not be placed" }));
    } else {
        res.status(422);
    }
})

// Create and start the server
// ...

What we've done here is imported our kitchen and waiter and added a POST route /order to receive orders from our customers. Remember the order object?

order = {
    dish: "Pizza ๐Ÿ•", 
    qty: 2,
    orderNo: "kbv9euic"
}

We are creating a order object from the JSON body of the POST request and passing it to our waiter and sending a JSON response to acknowledge our customer. In case the request is not properly made, we will send some error message also. And we're done โœŒ .


Yeah really, we are done. Now it's time to test it out ๐Ÿ˜

  • Start the server by running $ npm start on your terminal.
  • Send a get request to localhost:5000 and see if you get a response like this: Restaurant open
  • Next send a POST request to localhost:5000/order and check the response and look at your console. Testing out the API

You can send multiple requests one after another to check that it does not hang to any request.

Let's add another POST route, to compare it with a normal restaurant without a task queue.

Add these lines to restaurant.js:

//  ...
app.post('/order-legacy', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }
    if (order.dish && order.qty) {
        setTimeout(() => console.log("Getting the ingredients ready... ๐Ÿฅฌ ๐Ÿง„ ๐Ÿง… ๐Ÿ„"), 1000);
        setTimeout(() => console.log(`๐Ÿณ Preparing ${order.dish}`), 1500);
        setTimeout(() => {
            console.log(`๐Ÿงพ Order ${order.orderNo}: ${order.dish} ready`);
            res.json({ done: true, message: `Your ${order.qty}x ${order.dish} is ready` })
        }, order.qty * 5000);
    } else {
        console.log("Incomplete order rejected");
        res.status(422).json({ done: false, message: "Your order could not be placed" });
    }
});


// Create and start the server
// ...
  • Next send a POST request to localhost:5000/order-legacy and check the response and look at your console. Alt Text

Note the difference in the response time ๐Ÿคฏ

With the task queue
With task queue

Without task queue
without task queue


Here is the Github repo, containing the complete project

GitHub logo sarbikbetal / nodejs-task-queue

This repo contains the sample code for the article "Simple Node.js task queue with bee-queue and redis"

Please comment down below if you have any questions or suggestions and feel free to reach me out ๐Ÿ˜„ and also check out the section below for Q&A.

๐Ÿ“ธInstagram ๐Ÿ“จEmail ๐Ÿ‘จโ€๐Ÿ’ผLinkedIn ๐Ÿ‘จโ€๐Ÿ’ปGithub

๐Ÿค” Hmmm.. I have some questions though.

I know, so here are some common ones, feel free to ask more in the comments section below.

  • How do we send the food to our customer once it is prepared?

    For that we need to implement some additional logic to our server side and client side application. Example of how we can achieve that, is through Websockets, push-notifications, emails, etc. Don't worry though I'll be covering that in details in the next article.

  • Aren't there better things out there like RabbitMQ?

    Yeah sure there is, but for small scale projects that doesn't need a lot of advanced features but still want to maintain a decent back-end infrastructure RabbitMQ would be an overkill and bee-queue might just turn out simple and easy to use.

Posted on by:

sarbikbetal profile

Sarbik Betal

@sarbikbetal

All things javascript

Discussion

markdown guide
 

Great start. But shouldn't your POST api return an orderId, so that customer can ask status of his order?

 

Yeah you are right, this was a very basic example, but I will be adding it to this article. Thanks for your feedback ๐Ÿ˜Š

 

Swarup, I have updated the code in the Github repo, will be updating the article soon. Thanks for your nice idea.

 

I have put my review comments in github github.com/sarbikbetal/nodejs-task....
Let me know what you think of it.
Cheers

Thank you so much, I checked it out. I would always keep them in mind. ๐Ÿ˜Š

 

๐Ÿ‘๐Ÿ‘๐Ÿ‘

 

i read the two article, what an awesome ones, you really made all of this so simple !!
im waiting for next article !! keept it up

 

Very interesting article.
Looking forward to read the next one... Nice way to explain it with the analogy.
Would you give as well some additional use cases with real world examples?

 

Yes Simone, that's what I would do in the next article ๐Ÿ˜. We would build something interesting, Stay tuned ๐Ÿ“ป.