loading...
Play Button Pause Button

CLUSTER MULTI THREADED NODEJS -- Run threads in parallel to speed up long processes

joelnet profile image JavaScript Joel ・1 min read

My JavaScript that creates thumbnails for hundreds of files is taking too long. So I have decided to speed things up with cluster.fork() that is built into Node.js. This will allow the workload to be spread out across my multi-core processor system.

8:53 should read: index % forks === cluster.worker.id - 1

To follow along, you can fork this repository. I have added a tag you can check out that will bring you right to the point this video was created.

git checkout https://github.com/joelnet/mojo-gallery.git
cd mojo-gallery
git checkout youtube-video-cluster

Be sure to subscribe for more videos like this!

Alt Text

Posted on by:

joelnet profile

JavaScript Joel

@joelnet

Cofounded Host Collective (DiscountASP.net). Cofounded Player Axis (Social Gaming). Computer Scientist and Technology Evangelist with 20+ years of experience with JavaScript!

Discussion

markdown guide
 

I had a similar requirement, for image manipulation, I used workerpool library. It made task pretty easy. Initially I did achieve it with worker_threads , but the issue I faced was that I couldn't actually keep a track of threads, and would quickly run out of cpus, but workerpool managed it automatically.

 

Ahh! this is a cool library. I haven't seen this one before. Definitely helps with the managing of threads. I'll check this out next time I need something multi-threaded. Thanks for sharing!

 

What about using Working Threads? Wouldn't be lighter and faster?

 

I think you are right. I am familiar with cluster because I use with with Express.js. And I think cluster might be the better use-case when using Express. But in this instance because there isn't much communication back and forth between processes, worker_threads would probably run faster. I think the difference would be small though.

Want to convert it and run some benchmarks? 😁