I'd suggest using something like the async library to have a maximum n processes running at once - this should reduce thrashing. You'd use forEachLimit or something like that and create a promise awaiting the result of the external process. Try to balance the number of concurrent processes against the number of CPUs etc.
How’s it going, I'm a Adam, a Full-Stack Engineer, actively searching for work. I'm all about JavaScript. And Frontend but don't let that fool you - I've also got some serious Backend skills.
Location
City of Bath, UK 🇬🇧
Education
10 plus years* active enterprise development experience and a Fine art degree 🎨
Ah yes that library came up alot digging around Stack Overflow, it used to be really popular, anyway you mentioned balancing. Where I have 8 CPUs and 3xx process I should find a limit divisible by those numbers?
How’s it going, I'm a Adam, a Full-Stack Engineer, actively searching for work. I'm all about JavaScript. And Frontend but don't let that fool you - I've also got some serious Backend skills.
Location
City of Bath, UK 🇬🇧
Education
10 plus years* active enterprise development experience and a Fine art degree 🎨
I'd suggest using something like the async library to have a maximum
nprocesses running at once - this should reduce thrashing. You'd use forEachLimit or something like that and create a promise awaiting the result of the external process. Try to balance the number of concurrent processes against the number of CPUs etc.Ah yes that library came up alot digging around Stack Overflow, it used to be really popular, anyway you mentioned balancing. Where I have 8 CPUs and 3xx process I should find a limit divisible by those numbers?
Yeah I still use it occasionally when I hit those kind of challenges. I'd go with a limit of 8 or 16 and see which works out best.
Yeah seems to run slower now but also more stable so it's a trade-off I'm happy with, thanks for the tips!