I see that you are already using an asynchronous way of handling requests by saving the domain in the database and triggering an event, but the other request will also add the job in the database and will run the event.
I might try the following approaches
Simple Scheduled Job: trigger an event in an interval to fetch a job from the database and execute it and keep track of it using some sort of flag. This can also happen in the following way
SetInterval (in same code) or cron job --> trigger an event --> fetch batches from database --> spawn a few threads and process the batches either parallelly or synchronously.
Even a synchronous approach won't harm this as you can notify the user once the process is done but it can be slow, depends on the use case.
Utilize a queue (any queue will be okay for POC): Push the domain to the queue and configure consumers to fetch from the queue and do the task. You will have to control the number of consumers that can run parallelly to reduce the load.
Make sure that the producer does not produce more than what the consumer can handle which might also lead to a memory issue.
You can even isolate the main server to just get the request and push it to the database and you can have another instance to take care of the heavy lifting if you think that the load can be a lot on the main server.
It can't be the best solution, I would also appreciate it if others can correct my approach and let me learn more.
I looked into general optimizations for NodeJS apps and your answer is on point, I doubt that there will be a better one but if someone gives it, I'll notify you in the comment perhaps. One article, called, "CPU Intensive Node.js" suggested that one should use multiple treads along with Redis and a priority queue to handle CPU intensive tasks. It is exactly what you suggested.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I see that you are already using an asynchronous way of handling requests by saving the domain in the database and triggering an event, but the other request will also add the job in the database and will run the event.
I might try the following approaches
Simple Scheduled Job: trigger an event in an interval to fetch a job from the database and execute it and keep track of it using some sort of flag. This can also happen in the following way
SetInterval (in same code) or cron job --> trigger an event --> fetch batches from database --> spawn a few threads and process the batches either parallelly or synchronously.
Even a synchronous approach won't harm this as you can notify the user once the process is done but it can be slow, depends on the use case.
Utilize a queue (any queue will be okay for POC): Push the domain to the queue and configure consumers to fetch from the queue and do the task. You will have to control the number of consumers that can run parallelly to reduce the load.
Make sure that the producer does not produce more than what the consumer can handle which might also lead to a memory issue.
You can even isolate the main server to just get the request and push it to the database and you can have another instance to take care of the heavy lifting if you think that the load can be a lot on the main server.
It can't be the best solution, I would also appreciate it if others can correct my approach and let me learn more.
Thanks!
I looked into general optimizations for NodeJS apps and your answer is on point, I doubt that there will be a better one but if someone gives it, I'll notify you in the comment perhaps. One article, called, "CPU Intensive Node.js" suggested that one should use multiple treads along with Redis and a priority queue to handle CPU intensive tasks. It is exactly what you suggested.