So I recently got a project that requires real-time location + chat data streaming, and I wanted to know the right steps to take.
Most articles I've read online state to spawn up multiple servers and set up a load balancer like nginx+pm2.
However, I wanted to make this thing DevOps ready, with CI/CD and best practices, which I am not able to judge from the articles. Containerisation, database optimisation, and everything.
Would love to learn the right way!
Top comments (20)
Articles about scaling Node are going to recommend pm2 and load balancing because that's how you scale Node. There are plenty of articles here & elsewhere about containerization if you look for those instead; the important thing to remember is that once you have a container that's listening on whatever port and serving content, the fact that it's doing that with Node stops mattering. Same goes for CI/CD and database tuning: the content is out there, and easily found with a little exploration.
Anyway, from 30,000 feet the approach I'd take is to build Docker images as part of CI, deploy them to DockerHub or a private registry, and manage them with Kubernetes, since that's more or less become the default orchestration tool. You'd want to look into high-availability Kubernetes specifically since that involves some extra configuration to avoid single points of failure.
So container management with kubernetes will avoid single points of failure, and this I'll have to integrate with my load balancer and process manager right?
On a side note, are there any good methods to identify the ideal amount of RAM and processing power i should deploy with my servers, to optimise costs? Are there any metrics?
XMPP server is best solution for chat. Do not use node.js as chat data streaming.
Hey! A little late to the discussion, but I have a question. What if my application has custom chat requirements, such as only some users being able to message other users. For this we do need some sort of proxy/server in between the client and the XMPP server. How exactly should this case be handled where requirement is based on data stored outside of XMPP? A little overview/nudge in the right direction would be highly appreciated.
Thanks!
Hi Sandeep, are you available for some on going development work? If you are please contact me on glen.alexis@mradigital.com to discuss.
Thanks
Could you provide some details / data on your statement. Why would you choose XMPP over a NodeJS app? Do you have any personal experience / empirical evidence the poster could use for references?
Yes. 8 Android app built by me use XMPP (Openfire).
I previously use Node.js but there are many limitations in node.js
1) You need to built all things from scratch but in openfire XMPP server each and every thing that i want to use is already prebuild.
2) Node.js is good for broadcasting messages. But for one-to-one messaging. It is disastrous. On the other hand Openfire provides both of these.
3) Main problem with node.js is offline messaging.
Example. When User_A send message to User_B and User_B is offline at the time of messaging. Then message lost. To solve this problem you need to code all thing for this from scratch in node.js. But in openfire you don't need to worry.
4) You need to track all users sockets. But in openfire there is no such problem.
5) You don't detect which user is online or not in node.js. You need to program it that node.js ping for every 'X' seconds to users to track their availability. But In openfire you don't need to worry.
There are many other points.
In other words You just need to install XMPP server like openfire. And Go for your project. you don't create all thing from scratch like in node.js
I have one question. Why google,facebook,whatsapp use XMPP server for instant messaging and push notification. Why they don't even try to use Node.js
Node.JS real time API is just for fun.
I use Node.js previously But I face many problem . My client complaint that they don't receive messages. Buy when i heard of XMPP server. i used that in all my apps for push notifications and for chat.
Thank you for the details. That is some very helpful information. I sure @presto412 has a better idea of the route to take now.
cubettech.com/blog/xmpp-chat-using...
How would you rate this? I'd like to keep my codebase consistent
Hey I would like to discuss this a lot further, could you follow me back or send me a mail at priyansh.jain0246@gmail.com ?
I have followed you here on dev.to :).
Mr. Priyansh Jain. That is node.js xmpp client not XMPP server.
I think you don't read comment section in first block of code.
Oh darn, didn't realise it.
Love all the knowledge sharing in this thread. Just a friendly reminder everyone’s here to learn from each other’s experiences. 👍🏻
What @dmfay said; if you containerize the application the process running inside the container is of minimal impact; outside of resource requirements per container (IE cost to run each instance). However, scaling becomes much easier.
The current industry direction is options like Amazon ECS or Google GCE. When configured correctly the running process can finally not be the bottle neck anymore.
AS for CI/CD: you will want a task runner (TravisCI/Jenkins/CodeShip) and setup security scanning, performance checking, linting, test runners, and any other step you come across later. The name of the game here is automation with the goal of quality enforcement.
As for Database optimization, that really depends on the needs of the application, SQL, NoSQL, Key/Value store, Document store? How about media/binary storage? Does it have to be ACID compliant? Is latency or accuracy more important? The database question would need more information to really sort it out.
That all being said, I am available, and the dev.to community is here, to help out along the way. No matter what, it sounds like you have an interesting project ahead.
Thanks for this, looks like I'll start with containerization
Do follow me back so we can stay in touch!
I read that SocketCluster can be used for scalable real-time services, but I never tried it out.
It looks promising, thanks for the suggestion!
To scale an application that serves so many users i would consider Go language.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.