Async Iteration in Nodejs

Node.js is an amazing JavaScript run-time environment, no doubt. Its non-blocking IO model makes applications built with it amazingly fast and highly efficient. But sometimes its non-blocking nature provides some interesting challenges when performing asynchronous iterations.

Lets say we have an array of user ids, and we want to loop through the array and query our users collection(or table) for more information such as their first name, last name and email address. In the end we want to return another array of objects containing these data.

One approach we may use will be as follows:

const getUserDetails = (userIds, callback) => {

    let userInfoArr = [];

    for(let i = 0; i < userIds.length; i++){
        User.findById(userIds[i], (err, user) => {
            if(err){
                return next(err);
            }

            userInfoArr.push({
                firstName: user.firstName,
                lastName: user.lastName,
                email: user.email
            });
        })
    }

    callback(userInfoArr);
}

getUserDetails(ids, (users) => {
    console.log(users);
})

You might think this will work and that we successfully get back the array of users that we wanted, but when we log the resulting users array to the console, we get an empty array. Why is that?

Well, as we already know, node is asynchronous: it doesn't sit around and wait for the result of an execution and instead goes back only when the result is available.
So the reason we are getting an empty array is that at the point where we logged the resulting array, the execution of our code was not complete.

This means we have to wait for the execution to be complete before we return our results. One way that has worked for me is by introducing a new counter variable.
In every iteration, the value of the counter is increased and then checked to see if its equal to the length of the array we are iterating over. If the counter is equal to the array's length, then we assume the execution has completed and we return the resulting array, like so:

const getUserDetails = (userIds, callback) => {

    let userInfoArr = [];
    let counter = 0;

    for(let i = 0; i < userIds.length; i++){
        User.findById(userIds[i], (err, user) => {
            if(err){
                return next(err);
            }

            userInfoArr.push({
                firstName: user.firstName,
                lastName: user.lastName,
                email: user.email
            });

            counter++;

            if(counter == userIds.length){
                return callback(userInfoArr);
            }
        })
    }
}

getUserDetails(ids, (users) => {
    console.log(users);
});

Happy coding!

Comments

[Ryan]

Hey Kwabena, it’s funny that you posted this today because I posted my solution to the async problem as well today 😂 I like the creative counter approach that you used. You might be able to benefit from Promises and Async/Await! I talk about them in my last article: dev.to/ryhenness/the-path-to-conqu...

[Robert Lippens]

Nice! This one has snuck up on me (and I'd wager most javascript developers) more often than I'd like to admit. Javascript's non-blocking I/O is nice but can be a bit counter-intuitive for async operations.

I've become a big fan of doing operations like this which can be done in parallel using something like Promise.all over callbacks. (I actually use generators most of the time, as the project I work on has issues upgrading past node v6 - but async/await would work great as Ryan mentioned)

I would wrap the findById in a Promise (I do this manually but something like BluebirdJS can handle entire libraries if you are dealing with old node packages):

const findById = (userId) => {
  return new Promise((resolve, reject) => {
    User.findById(userId, (err, user) => {
       if (err)  {
         reject(err);
       } else {
         resolve({
          firstName: user.firstName,
          lastName: user.lastName,
          email: user.email
         });
    });
  });
}

With this helper, we can just do something like:

return Promise.all(ids.map((id) => findById(id)));

(where our caller would use the result of the promise.)

Note that with Promise.all, a single rejection will terminate the entire call, so if you can tolerate partial results, I recommend just console.error(err) with a resolve() instead of reject(err). You can pick out the empty objects before returning.

[Kwabena Bio Berko]

Cool!

[Manuel Romero]

if(error) return callback(error)

If there is no error:

return callback(null, userInfoArr)

When working with callbacks is important error first management