I work at a startup and have been working on a cool project NodeJS based project. Even though, it is a WordPress position, when I'm free, I take my time to work on interesting projects. Given what we're trying to achieve in this soon-to-be proprietary system, I noticed optimization issues with it. I tried bombarding the endpoints with 200 requests at a time and
pm2 monit showed a few issues.
Before I talk about the pm2 stats, I would like to talk about what the system actually does. We sent a domain name to our endpoint, for example, say, dev.to. The endpoint makes an entry to database and it emits an event to analyze that website. The endpoint quickly does what it is supposed and sends back a 200 response but the process afterwards takes a lot of time as it involves HTTP requests a huge sets of things it needs to analysis the website for, a long with Puppeteer, potentially a lot of loops with hundreds, if not thousands of iterations.
What I talked about above, resulted in at almost 100% of CPU usage in PM2 stats, the heap was almost 100% too, EventEmitter gave us memory leaks warnings. Given that there is no queue or in-memory DS like Redis, I think the event loop was overwhelmed with 200 requests at the same time which involves a lot of processing! I'm not happy with the results and will present issues further down the road.
NOTE: Keep in mind that the slow process I am talking about takes place on an app level. The ExpressJS route only emits and event to start the process for that specific domain that it just saved into the database. I did this because there was no need to keep a user waiting for 10 seconds!
I discussed this with my boss and he encouraged me to take time and ask for help around the community. This is why I am here! Have you worked on such a project? If so, how did you handle it? What would you recommend me to do in this case? Should I go for cronjobs instead of event based system? Would really appreciate an answer.
Thank you! Also, if you have any questions regarding the project, let me know and I shall answer them.