DEV Community

Discussion on: Node.js CPU intensive 🔥

Collapse
mse99 profile image
Mohamed Edrah

You could use multiple processes, but I suggest a simpler solution, boost your thread pool size and use parallel scheduling and scheduling queues to schedule more than one task just remember to keep the number of spawned tasks under the number of threads in the thread pool.

Collapse
adam_cyclones profile image
Adam Crockett Author

It's a shame I didn't see this comment, I went the fork process route (1 per jar), it's now 300 processes for allocated 'jobs'

  • Parse jars
  • bytecode forEach classpath
  • Parse bytecode to typescript

Although it's not sequential, some jobs finish sooner than others and then when the child proc does we pass on to the next one

The multi process is a hell of a lot faster but now running kind of stalls and lags my system... One of the jobs is the classpaths processing, that could have quite a lot of org.paths.goo paths so I think instead of multiprocess here, maybe a queue and limit the amount of processing within that job?

Your suggestions I will need to research 🦉