Async/Await and the forEach Pit of Despair

The JavaScript VM is like a 4 year-old on a sugar high; screaming through your code with no pants on and just executing everything at all once while it draws on your wall with a Sharpie.

Async/Await is like that same 4 year-old taking a nap. Your code is just there, like a perfect angel, doing exactly what you want and you wonder how you could ever love anything so much. Your heart might burst with pride. And you probably need to get out more because we're talking about programming here.

But async/await is not all finger-painting and "first day of Kindergarten". It's got some weirdo edge cases that can make you wonder if you have made a huge mistake by ever deciding to even have JavaScripts. Stop looking at me like that. If you're a parent, you know exactly what I'm talking about.

I ran into one of these strange edge cases on a large project and the complexity of the code made it really hard to debug. That, and I'm not terribly smart to begin with so it was kind of the perfect storm.

Async/Await Basics

Just to ensure we're all playing our Recorders to the same version of "Hot Crossed Buns", let's look at a simple async/await example.

Axios returns a Promise, but we can await that Promise so we don't have to deal with any more socially acceptable callbacks.

This is all well and good, but what if we wanted to do an async call inside that forEach loop? Easy, just mark the callback as async and go about your bizness.

This works. Or at least it looks like it does. What's the problem here? The problem is that this loop is not executing items in order. Don't believe me? Watch what happens when we throw the index into the loop…

Chaos. That's what happens. Like a heard of 4-year old's right about nap time.

Why is this happening? Isn't forEach supposed to be a synchronous operation?

Yes. But we are passing in an async function, which tells the VM that it can execute these things however it wants and "however it wants" is "watching the world burn".

This is no good. This is not the solution to anything. Even worse is that this is incredibly hard to debug if you put a lot of misplaced faith in forEach. I should know. Async/await is basically negating an extremely important aspect of the built-in array loop.

So how do we fix this? The fix is to go to a for of loop.

And that's what we wanted all along. 

Now I'm not the first person to write about this. The internet is littered with posts about how async/await changes forEach. There are also plenty of people saying not to use forEach because it's "slower", or something like that. I don't know. I didn't actually read the articles.

I'm not going to recommend that you do or don't use forEach, it definitely has it's place. Just remember that when you are doing async/await inside of a forEach loop, you are doing it wrong. And don't worry - you'll know you are doing it wrong because it won't work properly and that psychotic toddler of a JavaScript VM will turn your life upside down.

Further Reading: Moving from Callbacks to Promises and Async/Await

Comments

[simo]

The difference between forEach and for of in your example is that with forEach you are executing all HTTP requests in parallel, and with for of sequentially.

So it's not about 'fixing' things but instead understanding them.

Hope that helps.

[Thomas van Latum]

By doing the for loop instead of the ForEach you just killed the whole purpose of doing the async code.
supposed axios(${baseURL}/users/${post.userId}) takes a couple of seconds to run...

Listen to your Toddler, he just wants to move fast and execute stuff as fast as possible!

You should just put the result in an array and sort the order out in the end.

[Douglas Gubert]

Hey Burke, nice one!

I have ran into a similar problem when I was making some scripts to populate a database with some records. I had this "async loop" and wanted to wait for it to finish to show some messages and do some cleanup, but obviously the clean up was being executed before all the asynchronous tasks I've started in the .forEach had finished :(

My solution at the time was to take out the .forEach and make it something like this

await Promise.all(data.map(async () => { .... } ))

But your solution is ways more elegant!

[Vincent Brouwers]

I've solved this problem before by using Array#reduce to basically build a big Promise chain.

It's something like

await posts.data.reduce((prev, curr) => prev.then(async post => {
    let user = await axios(`${baseURL}/users/${post.userId}`);
    console.log(user.data.name);
}), Promise.resolve()) 

If I only need the result to ordered and don't care about the actual execution order, I use Array#map with Promise#all

const result = Promise.all(posts.data.map(async post => await axios(`${baseURL}/users/${post.userId}`))) 

(in this example both the async and await can be left out since only the plain promises get used, but it should get the thought across)

[Dipun Mistry]

Everything you describe from a code point of view is correct, but as a developer, when you write code, you should know what you are writing. Let's talk about await async first:

async function main() {
    await somethingAsync();
    return 20;
}

function main2() {
    return somethingAsync().then(_ => 20);
}

The two methods above are identical. This is what is happening behind the scenes.

now a quick primer on forEach:

myArray.forEach(function(val) {
  return val + 20;
});

most people know that returning something here is pointless, forEach will run your function for every item in the array, but it won't do anything with what you return.

Let's go back to await async:

async function main() {
  await doSomethingAsync();
}

Although this doesn't look like it returns anything, it actually translates to this:

function main() {
  return doSomethingAsync();
}

This is why you need to await on async functions and just like in any language, the async/await paradigm bleeds out and suddenly everything is asynchronous. If you don't await on that function when you call it, you risk running your next operation before the first operation finished.

In some cases this is fine, sometimes you want your application to go off and do a bunch of things at once, eg: when a page loads in the browser, you want the browser to go fetch your images, js and css as quickly as possible, making use of all it's resources; you don't want it to fetch them one at a time.

Now remember, forEach does nothing with whatever you return in your function, but async functions need to be awaited if you want things to go in sequencial order.

Looking at your example, you might actually be missing a trick: just because the console logs need to run sequentially, doesn't mean you should force your http calls to happen sequentially:

const p_users = data.map((post, index) => axios(post, index));
const users = await Promise.all(p_users); // Promise.all maintains the order of values, and waits for all promises to resolve before continuing.
users.forEach(user => console.log(user));

The above call will allow you to make all your http calls at the same time instead of one at a time but allows you to log your messages in the correct order. Overall, same result, just faster.

Everything you saw happen, happened by design; this may be an insightful article for the next developer, but let's educate and empower without scaring and blaming.

[Jon 🇪🇺]

await Promise.all(data.map(async () => { .... } )) proves to be more performant in my case - because it runs them all asynchronously and then resolves the list in the correct order as opposed to the for of loop which awaits on each iteration of the loop - making it slower because things don't run in parallel ...I think

[Frederik 👨‍💻➡️🌐 Creemers]

You can use backticks "``" around in-line source code to improve the formatting. Just a heads-up. Also since this seems to be your first comment on DEV.to, welcome!

[Mihail Malo]

You need to be aware that you not only changed the order of the output, but also limited concurrent requests from "all at once" to "only ONE at a time" which is not what you'd often want.
If you wanted to log them asynchronously, but in the returned order, you only had to make a small change:

ConsoleLogHTML.connect(document.getElementById('log'));

const baseURL = 'https://jsonplaceholder.typicode.com';

async function main() {
  // get all posts
  const posts = await axios(`${baseURL}/posts`);
  const names = posts.data.map(async (post, index) => {
    // get the user who posted this post
    const user = await axios(`${baseURL}/users/${post.userId}`);
    return `${index}: ${user.data.name}`;
  });
  for (const name of names) {
    console.log(await name)
  }
}

main();

This way, all requests are started at the beginning, and then we go through the list of promises in order, effectively sleeping on every unfinished request.
Usually, in a situation like that I won't splurge for an inner async function, so it would actually end up looking like this:

ConsoleLogHTML.connect(document.getElementById('log'))

const baseURL = 'https://jsonplaceholder.typicode.com'

async function main() {
  // get all posts
  const posts = await axios(`${baseURL}/posts`)
  // map user ids to user name promises
  const names = posts.data.map((post, index) =>
    axios(`${baseURL}/users/${post.userId}`).then(
      user => `${index}: ${user.data.name}`
    )
  )
  for (const name of names) {
    console.log(await name)
  }
}

main()

No brakes on the ajax train

If you truly wanted to display them asynchronously, and as soon as possible, you'd reserve space for each request's output, like this:

All at once (feat. Promise.all)

Finally, if you only care about all the data together, and don't need it to get in front of the user's eyes as soon as humanly possible, just use Promise.all:

ConsoleLogHTML.connect(document.getElementById('log'))

const baseURL = 'https://jsonplaceholder.typicode.com'

async function main() {
  // get all posts
  let posts = await axios(`${baseURL}/posts`)
  const users = await Promise.all(
    posts.data.map(post => axios(`${baseURL}/users/${post.userId}`))
  )
  for (const user of users) {
    console.log(`${users.indexOf(user)}: ${user.data.name}`)
  }
}

main()

[iurquiza]

'forEach' and 'map' are declarative functions used to traverse/manipulate all items in an array. You are not supposed to assume how they work internally, including how they access the items. In many cases/languages they are use to parallelize execution, which basically throw order of completion out of the window. If your task depends on the order of completion use a procedural loop. forEach's internal implementation can use a typical loop but it could also be implemented using a different traversal method.

[Adebiyi Adedotun]

The truth I learnt from experiment is in two parts:

According to the MDN docs,

1. forEach() method executes a provided function once for each array element.
   Which doesn't mean sequentially (but in here, in parallel), but we are already used to executing things synchronously most of the times we never take async into consideration. So, practically, async has the right to change the order of execution, whatever promise gets the baton goes first...

2. I also figured something with functions that require callbacks... again the forEach() MDN says: forEach() method executes a provided function once for each array element. "Once for each array element" And based on my experience with this, what we should really aim to iterate and perform an action upon is the array(posts.data) item, in this case "post" not "user" because that breaks the purpose of the forEach() method.

[Frederik 👨‍💻➡️🌐 Creemers]

I love your toddler metaphor ❤.