Cover Photo by Bimo Luki on Unsplash
As you saw in the previous article that task queues are pretty awesome ๐ and in this tutorial we would be using a task queue in our own application let's get our hands dirty and write some code.
We will be building our restaurant as explained in the previous article.
This tutorial would be much of a demonstration rather than a working application, so stick with me if you wanna check out how to plug in a task queue into your app.
In the next article we will be building a real application. (I know it's exciting and you can't wait for that ๐).
๐จโ๐ป Link to the entire github repo for this project is at the end of the article โฌ
Let's get started.
Pre-requisites
- Install Node.js on your machine, and then run the following command to verify proper installation. ```bash
$ node --version
v12.16.1
+ Redis running on your pc or the cloud. Install [Redis](https://redis.io/) or create an instance on [RedisLabs](https://redislabs.com/) for free.
And we're good to go :grin:
## Initialization
Run:
```bash
$ npm init
After that install the necessary packages by running
$ npm install express bee-queue dotenv
In case you are wondering what each package does, here's some info:
-
express
helps us to create a server and handle incoming requests with ease. -
bee-queue
is our task queue manager and will help to create and run jobs -
dotenv
helps us to load environment variables from a local.env
file
After that create a file restaurant.js
and edit your package.json
so it looks something like this
{
...
"main": "restaurant.js",
"scripts": {
"start": "node restaurant.js"
}
...
}
Time for some real code
Open restaurant.js
in the editor of your choice and add the following lines of code
require('dotenv').config();
const express = require('express');
const http = require('http');
// Inits
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
// Routes
app.get('/', (req, res) => {
res.send("๐ We are serving freshly cooked food ๐ฒ");
});
// Create and start the server
const server = http.createServer(app);
const PORT = process.env.PORT || 5000;
server.listen(PORT, () => {
console.log(`Restaurant open at:${PORT}`);
});
What it does is basically start a local webserver at the specified port (here, 5000) and listens for incoming GET
requests on the base url /
and replies with a text.
Run the following command to start the server and head over to localhost:5000
in your browser.
$ npm start
> restaurant@1.0.0 start /mnt/code/dev/queue
> node restaurant.js
Restaurant open at port:5000
You will get a blank page with a neat little ๐ We are serving freshly cooked food ๐ฒ
message
Now it's time to create our task queue
First create a file named .env
and paste in it your database credentials like so, (You can use your local redis instance here as well) and remember, never to commit .env
to your source control.
DB_HOST=redis-random-cloud.redislabs.com
DB_PORT=14827
DB_PASS=pTAl.not-my-password.rUlJq
And you are done with the basic configuration.
Let's go ahead and create our waiter
. Start by creating a file waiter.js
and add the following chunk of code:
const Queue = require('bee-queue');
const options = {
removeOnSuccess: true,
redis: {
host: process.env.DB_HOST,
port: process.env.DB_PORT,
password: process.env.DB_PASS,
},
}
const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);
const placeOrder = (order) => {
return cookQueue.createJob(order).save();
};
serveQueue.process((job, done) => {
console.log(`๐งพ ${job.data.qty}x ${job.data.dish} ready to be served ๐`);
// Notify the client via push notification, web socket or email etc.
done();
})
// Notify the client via push notification, web socket or email etc.
done();
})
module.exports.placeOrder = placeOrder;
๐คฏ Whoa! what was that? Well, let me explain.
We first import the bee-queue
package as Queue
,
and then pass in the database configuration to our two new Queue
objects. One of the queue will have the list of orders to be prepared by the cook and the other will have the list of orders that are ready to be served by the waiter.
We then create a new function placeOrder
that takes in an order
as the parameter. We will define this order object later, but keep in mind that it has a structure like this
order = {
dish: "Pizza ๐",
qty: 2,
orderNo: "kbv9euic"
}
The placeOrder
function takes this order and adds it to the queue by calling .createJob(order).save()
method on the cookQueue
Queue object. This acts as the task publisher.
and lastly the process
method on serveQueue
Queue object executes the handler function (job, done) => {...}
everytime an order is prepared and ready to be served. This acts as the task consumer.
We call done()
to acknowledge out task queue that the job in done so that it can send the next task to be processed from the queue. We simply call done()
to indicate the task was successful and call done(err)
i.e with the first parameter (where err
is an error message) to indicate job failure. You can also call done(null, msg)
to indicate job success with the second parameter msg
being the success message.
And our waiter ๐จโ๐ผ is ready
Now its time for the kitchen with the cooks ๐จโ๐ณ
create another file kitchen.js
and paste in it the following lines of code:
const Queue = require('bee-queue');
const options = {
removeOnSuccess: true,
redis: {
host: process.env.DB_HOST,
port: process.env.DB_PORT,
password: process.env.DB_PASS,
},
}
const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);
cookQueue.process(3, (job, done) => {
setTimeout(() => console.log("Getting the ingredients ready ๐ฅฌ ๐ง ๐ง
๐"), 1000);
setTimeout(() => console.log(`๐ณ Preparing ${job.data.dish}`), 1500);
setTimeout(() => {
console.log(`๐งพ Order ${job.data.orderNo}: ${job.data.dish} ready`);
done();
}, job.data.qty * 5000);
});
cookQueue.on('succeeded', (job, result) => {
serveQueue.createJob(job.data).save();
});
๐ Well that looks familiar.
Yeah exactly, but the only change is that here our cooks are consuming from the cookQueue
and publishing to the serveQueue
for the waiters to take and serve the the orders.
One thing to note here is that, anything published via createJob(order)
is available to the cosumer as job.data
in the Queue.process()
method's handler function (job, done) => {...}
, and if you look closely, there's something different in cookQueue.process(3, (job, done) => {...})
too. Yeah, we pass in a number, before the actual handler function. It is known as concurrency (the number of tasks in the queue that can be processed simultaneously). Here we have set it to 3 because our kitchen has 3 cooks, who can work together.
And we use the cookQueue.on('succeeded', (job, result) => {...})
method to call the handler function whenever a task is successful (i.e. whenever you have called done()
in the process()
method).
Believe me we are almost done ๐ค
Final step: Hook everything together
Open restaurant.js
and add these final lines of code
// ...
// Add these lines before the Inits.
require('./kitchen');
const { placeOrder } = require('./waiter');
// Inits
// ...
// Routes
// ...
app.post('/order', (req, res) => {
let order = {
dish: req.body.dish,
qty: req.body.qty,
orderNo: Date.now().toString(36)
}
if (order.dish && order.qty) {
placeOrder(order)
.then(() => res.json({ done: true, message: "Your order will be ready in a while" }))
.catch(() => res.json({ done: false, message: "Your order could not be placed" }));
} else {
res.status(422);
}
})
// Create and start the server
// ...
What we've done here is imported our kitchen
and waiter
and added a POST route /order
to receive orders from our customers. Remember the order object?
order = {
dish: "Pizza ๐",
qty: 2,
orderNo: "kbv9euic"
}
We are creating a order object from the JSON body of the POST request and passing it to our waiter and sending a JSON response to acknowledge our customer. In case the request is not properly made, we will send some error message also. And we're done โ .
Yeah really, we are done. Now it's time to test it out ๐
- Start the server by running
$ npm start
on your terminal. - Send a get request to
localhost:5000
and see if you get a response like this: - Next send a POST request to
localhost:5000/order
and check the response and look at your console.
You can send multiple requests one after another to check that it does not hang to any request.
Let's add another POST
route, to compare it with a normal restaurant without a task queue.
Add these lines to restaurant.js
:
// ...
app.post('/order-legacy', (req, res) => {
let order = {
dish: req.body.dish,
qty: req.body.qty,
orderNo: Date.now().toString(36)
}
if (order.dish && order.qty) {
setTimeout(() => console.log("Getting the ingredients ready... ๐ฅฌ ๐ง ๐ง
๐"), 1000);
setTimeout(() => console.log(`๐ณ Preparing ${order.dish}`), 1500);
setTimeout(() => {
console.log(`๐งพ Order ${order.orderNo}: ${order.dish} ready`);
res.json({ done: true, message: `Your ${order.qty}x ${order.dish} is ready` })
}, order.qty * 5000);
} else {
console.log("Incomplete order rejected");
res.status(422).json({ done: false, message: "Your order could not be placed" });
}
});
// Create and start the server
// ...
- Next send a POST request to
localhost:5000/order-legacy
and check the response and look at your console.
Note the difference in the response time ๐คฏ
Here is the Github repo, containing the complete project
sarbikbetal / nodejs-task-queue
This repo contains the sample code for the article "Simple Node.js task queue with bee-queue and redis"
Please comment down below if you have any questions or suggestions and feel free to reach me out ๐ and also check out the section below for Q&A.
๐ค Hmmm.. I have some questions though.
I know, so here are some common ones, feel free to ask more in the comments section below.
-
How do we send the food to our customer once it is prepared?
For that we need to implement some additional logic to our server side and client side application. Example of how we can achieve that, is through Websockets, push-notifications, emails, etc. Don't worry though I'll be covering that in details in the next article.
-
Aren't there better things out there like RabbitMQ?
Yeah sure there is, but for small scale projects that doesn't need a lot of advanced features but still want to maintain a decent back-end infrastructure RabbitMQ would be an overkill and bee-queue might just turn out simple and easy to use.
Top comments (11)
Great start. But shouldn't your POST api return an orderId, so that customer can ask status of his order?
Yeah you are right, this was a very basic example, but I will be adding it to this article. Thanks for your feedback ๐
Swarup, I have updated the code in the Github repo, will be updating the article soon. Thanks for your nice idea.
I have put my review comments in github github.com/sarbikbetal/nodejs-task....
Let me know what you think of it.
Cheers
Thank you so much, I checked it out. I would always keep them in mind. ๐
Nice post!
by the way : Came here searching for time consuning common tasks usually devs wants to run in nodejs.. Im developing a distributed queue/task runner , and searching for good use cases to apply as examples..
queueXec
๐๐๐
Hey Sarbik! Great article. Do you plan to write a third answering "How do we send the food to our customer once it is prepared?"
Very interesting article.
Looking forward to read the next one... Nice way to explain it with the analogy.
Would you give as well some additional use cases with real world examples?
Yes Simone, that's what I would do in the next article ๐. We would build something interesting, Stay tuned ๐ป.
Wow,, great post, thanks for great explanation of task queue and the example project, love it.