Avoid the Promise.all pitfall! Rate limit async function calls

Have you fallen into the Promise.all pitfall? You know the one; you grab a list of things from somewhere and run a parallel function against all of them:

     const list = await getMeSomeList()
     const results = await Promise.all(list.map(someAsyncFunction))

Works a treat when the list has a few things in it, but lets say there are suddenly 10,000 records returned - this could really get messy.

You are really trying to spin too many plates, and memory or resources are going to become tight...

The Solution

Well you could just install the async package which has lots of useful functions like mapLimit which will reduce the burden and only run a number in parallel.

If that's overkill - then you can achieve a similar result using a simple rate limiter:

class Semaphore {
    constructor(maxConcurrency) {
        this.maxConcurrency = maxConcurrency
        this.currentConcurrency = 0
        this.queue = []
    }

    async acquire() {
        return new Promise((resolve) => {
            if (this.currentConcurrency < this.maxConcurrency) {
                this.currentConcurrency++
                resolve()
            } else {
                this.queue.push(resolve)
            }
        })
    }

    release() {
        if (this.queue.length > 0) {
            const resolve = this.queue.shift()
            resolve()
        } else {
            this.currentConcurrency--
        }
    }
}

export function rateLimit(asyncFunction, rate) {
    const semaphore = new Semaphore(rate)

    return async function process(...args) {
        await semaphore.acquire()
        try {
            return await asyncFunction(...args)
        } finally {
            semaphore.release()
        }
    }
}

With that in hand your code would change to be just this:

     const list = await getMeSomeList()
     const results = await Promise.all(list.map(rateLimit(someAsyncFunction, 20))

This would mean that it would keep 20 running at a time until the list way finished. Every time one of the someAsyncFunctions returns another one is started until the list is exhausted. Easy right :)

Comments

[wakywayne]

Is 20 the recommended number? Surely you should have a much higher limit, correct?

[Mike Talbot ⭐]

It depends on what the async function is doing :) Normally I'd set a 100 or so. But it depends, if you are querying a database then perhaps no more than 20 or 30 in parallel to avoid too much contention or too many database connections required. Given the speed of Redis, my tests have shown 20 - 30 concurrent is optimal under my configuration.

[Voltra]

If you run HTTP requests in parallel, you'll hit that limit

[Mike Talbot ⭐]

Dead right, especially on HTTP rather than HTTP2

[Yeom suyun]

My library has a function that was written for a similar purpose.
How is it?

/**
 * Run multiple functions in parallel with a specified limit on the number of parallel executions.
 * @param {number} size
 * @param {Function[]} callbacks
 * @returns {Promise<({ value: * }|{ reason: * })[]>}
 */
const parallel = (size, ...callbacks) => {
    /** @type {({ value: * }|{ reason: * })[]} */
    const result = []
    return new Promise(resolve => {
        const length = callbacks.length
        if (length < size) size = length
        let index = 0
        const finally_callback = () => {
            if (index < length) run(index++)
            else if (++index == length + size) resolve(result)
        }
        /** @param {number} i */
        const run = (i) =>
            Promise.resolve(callbacks[i]())
                .then(value => result[i] = { value }, reason => result[i] = { reason })
                .finally(finally_callback)
        while (index < size) run(index++)
    })
}

[Mike Talbot ⭐]

I see that you are returning an array of results of exceptions, that's an interesting concept and would fit certain circumstances well. If you require all of the results then the Promise.all works well as it throws an exception on the first one that fails which stops the execution of the other remaining ones.

[Midas/XIV]

Just added this to my code base! it's really useful, thanks!