DEV Community

Cover image for Run multiple tasks Concurrently (in parallel) using JavaScript
King Elisha
King Elisha

Posted on

10 3

Run multiple tasks Concurrently (in parallel) using JavaScript

When you have multiple time-consuming tasks/functions to execute, there are two main solutions to optimize the execution time and speed up your app:

Run everything at once with Promise.all()

If your functions are promise-based, they can easily be executed concurrently using Promise.all()

const axios = require('axios').default
const fetchPosts = () => axios.get('/api/to/posts')
const fetchUsers = () => axios.get('/api/to/users')
const fetchDocs = () => axios.get('/api/to/docs')
Promise.all([
fetchPosts(),
fetchUsers(),
fetchDocs(),
]).then(([postsRes, usersRes, docsRes]) => {
// do something
}).catch((err) => console.log(err))
view raw concurrent-1.js hosted with ❤ by GitHub

Functions that work with properly formatted call-backs — where the first argument of the call-back is reserved for errors and the second argument is the value to be returned — can easily be promisified using the promisify utility function and executed concurrently.

const fs = require('fs')
const { promisify } = require('util')
const readFilePromise = promisify(fs.readFile)
Promise.all([
readFilePromise('file-1.txt'),
readFilePromise('file-2.txt'),
readFilePromise('file-3.txt'),
]).then(([file1, file2, file3]) => {
//do something
}).catch((err) => console.log(err))
view raw concurrent-2.js hosted with ❤ by GitHub

Run a fixed batch concurrently

If your functions require significant resources to execute, running them all at once with Promise.all() may cause your application to crash. A solution to this is to create a TaskQueue that can execute a fixed number of tasks concurrently

class ConcurrentTaskQueue {
constructor(taskPromisesFunc = [], batchSize = 1) {
this.batchSize = batchSize > taskPromisesFunc.length ? taskPromisesFunc.length : batchSize
this.todoTasks = taskPromisesFunc
this.resolvedValues = []
}
run(resolve, reject) {
if (this.todoTasks.length > 0) {
const taskPromises = this.todoTasks.splice(0, this.batchSize);
Promise.all(taskPromises.map((p) => p()))
.then((resolvedValues) => {
this.resolvedValues = [...this.resolvedValues, ...resolvedValues]
this.run(resolve, reject)
})
.catch((err) => reject(err))
} else {
resolve(this.resolvedValues)
}
}
runTasks() {
return new Promise((resolve, reject) => {
this.run(resolve, reject)
})
}
}
// some arbitrary function that consumes resources
const costlyFunction = (arg) => new Promise((resolve) => {
// do something costly here
resolve(arg);
})
const batchSize = 2;
const taskQueue = new ConcurrentTaskQueue([
// wrap all functions to prevent direct execution
() => costlyFunction(10),
() => costlyFunction(20),
() => costlyFunction(100),
() => costlyFunction(50),
], batchSize);
taskQueue.runTasks()
.then(([res1, res2, res3, res4]) => {
console.log(res1, res2, res3, res4);
});
view raw concurrent-3.js hosted with ❤ by GitHub

The runTask method executes each batch concurrently and resolves with the results of all the functions after executing all batches. This way, the speed of execution is improved without going overboard on computing resources.

Thanks 👍 for making it to the end 👨‍💻 and I really hope you found the content useful.

Top comments (0)