loading...
Cover image for πŸ’‘πŸŽ JavaScript Visualized: Generators and Iterators

πŸ’‘πŸŽ JavaScript Visualized: Generators and Iterators

lydiahallie profile image Lydia Hallie ・10 min read

ES6 introduced something cool called generator functions πŸŽ‰ Whenever I ask people about generator functions, the responses are basically: "I've seem them once, got confused, never looked at it again", "oh gosh no I've read so many blog posts about generator functions and I still don't get them", "I get them but why would anyone ever use that" πŸ€” Or maybe that's just the conversations I've been having with myself because that's how I used to think for a long time! But they're actually quite cool.

So, what are generator functions? Let's first just look at a regular, old-fashioned function πŸ‘΅πŸΌ

Yep absolutely nothing special about this! It's just a normal function that logs a value 4 times. Let's invoke it!

Alt Text

"But Lydia why did you just waste 5 seconds of my life by making me look at this normal boring function", a very good question. Normal functions follow something called a run-to-completion model: when we invoke a function, it will always run until it completes (well, unless there's an error somewhere). We can't just randomly pause a function somewhere in the middle whenever we want to.

Now here comes the cool part: generator functions don't follow the run-to-completion model! 🀯 Does this mean that we can randomly pause a generator function in the middle of executing it? Well, sort of! Let's take a look at what generator functions are and how we can use them.

We create a generator function by writing an asterisk * after the function keyword.

But that's not all we have to do to use generator functions! Generator functions actually work in a completely different way compared to regular functions:

  • Invoking a generator function returns a generator object, which is an iterator.
  • We can use the yield keyword in a generator function to "pause" the execution.

But what does that even mean!?

Let's first go over the first one: Invoking a generator function returns a generator object. When we invoke a regular function, the function body gets executed and eventually returns a value. However when we invoke a generator function, a generator object gets returned! Let's see what that looks like when we log the returned value.


Now, I can hear you screaming internally (or externally πŸ™ƒ) because this can look a little overwhelming. But don't worry, we don't really have to use any of the properties you see logged here. So what's the generator object good for then?

First we need to take a small step back, and answer the second difference between regular functions and generator functions: We can use the yield keyword in a generator function to "pause" the execution.

With generator functions, we can write something like this (genFunc is short for generatorFunction):

What's that yield keyword doing there? The execution of the generator gets "paused" when it encounters a yield keyword. And the best thing is that the next time we run the function, it remembered where it previously paused, and runs from there on! πŸ˜ƒ Basically what's happening here (don't worry this will be animated later on):

  1. The first time it runs, it "pauses" on the first line and yields the string value '✨'
  2. The second time it runs, it starts on the line of the previous yield keyword. It then runs all the way down till the second yield keyword and yields the value 'πŸ’•'.
  3. The third time it runs, it start on the line of the previous yield keyword. It runs all the way down until it encounters the return keyword, and returns the value 'Done!'.

But... how can we invoke the function if we previously saw that invoking the generator function returned a generator object? πŸ€” This is where the generator object comes into play!

The generator object contains a next method (on the prototype chain). This method is what we'll use to iterate the generator object. However, in order to remember the state of where it previously left off after yielding a value, we need to assign the generator object to a variable. I'll call it genObj short for generatorObject.

Yep, the same scary looking object as we saw before. Let's see what happens when we invoke the next method on the genObj generator object!

The generator ran until it encountered the first yield keyword, which happened to be on the first line! It yielded an object containing a value property, and a done property.

{ value: ... , done: ... }

The value property is equal to the value that we yielded.
The done property is a boolean value, which is only set to true once the generator function returned a value (not yielded! 😊).

We stopped iterating over the generator, which makes it look like the function just paused! How cool is that. Let's invoke the next method again! πŸ˜ƒ

First, we logged the string First log! to the console. This is neither a yield nor return keyword, so it continues! Then, it encountered a yield keyword with the value 'πŸ’•'. An object gets yielded with the value property of 'πŸ’•' and a done property. The value of the done property is false, since we haven't returned from the generator yet.

We're almost there! Let's invoke next for the last time.

Alt Text

We logged the string Second log! to the console. Then, it encountered a return keyword with the value 'Done!'. An object gets returned with the value property of 'Done!'. We actually returned this time, so the value of done is set to true!

The done property is actually very important. We can only iterate a generator object once. What?! So what happens when we call the next method again?

Alt Text

It simply returns undefined forever. In case you want to iterate it again, you just have to create a new generator object!


As we just saw, a generator function returns an iterator (the generator object). But.. wait an iterator? Does that mean we can use for of loops, and the spread operator on the returned object? Yas! 🀩

Let's try to spread the yielded values in an array, using the [... ] syntax.

Alt Text

Or maybe by using a for of loop?!

Alt Text

Heck so many possibilities!

But what makes an iterator an iterator? Because we can also use for-of loops and the spread syntax with arrays, strings, maps, and sets. It's actually because they implement the iterator protocol: the [Symbol.iterator]. Say that we have the following values (with very descriptive names lol πŸ’πŸΌβ€β™€οΈ):

The array, string, and generatorObject are all iterators! Let's take a look at the value of their [Symbol.iterator] property.

Alt Text

But then what's the value of the [Symbol.iterator] on the values that aren't iterable?

Alt Text

Yeah, it's just not there. So.. Can we simply just add the [Symbol.iterator] property manually, and make non-iterables iterable? Yes, we can! πŸ˜ƒ

[Symbol.iterator] has to return an iterator, containing a next method which returns an object just like we saw before: { value: '...', done: false/true }.

To keep it simple (as lazy me likes to do) we can simply set the value of [Symbol.iterator] equal to a generator function, as this returns an iterator by default. Let's make the object an iterable, and the yielded value the entire object:

See what happens when we use the spread syntax or a for-of loop on our object object now!

Alt Text

Or maybe we only wanted to get the object keys. "Oh well that's easy, we just yield Object.keys(this) instead of this"!

Hmm let's try that.

Alt Text

Oh shoot. Object.keys(this) is an array, so the value that got yielded is an array. Then we spread this yielded array into another array, resulting in a nested array. We didn't want this, we just wanted to yield each individual key!

Good news! πŸ₯³ We can yield individual values from iterators within a generator using the yield* keyword, so the yield with an asterisk! Say that we have a generator function that first yield an avocado, then we want to yield the values of another iterator (an array in this case) individually. We can do so with the yield* keyword. We then delegate to another generator!

Alt Text

Each value of the delegated generator gets yielded, before it continued iterating the genObj iterator.

This is exactly what we need to do in order to get all object keys individually!

Alt Text


Another use of generator functions, is that we can (sort of) use them as observer functions. A generator can wait for incoming data, and only if that data is passed, it will process it. An example:

A big difference here is that we don't just have yield [value] like we saw in the previous examples. Instead, we assign a value called second, and yield value the string First!. This is the value that will get yielded the first time we call the next method.

Let's see what happens when we call the next method for the first time on the iterable.

Alt Text

It encountered the yield on the first line, and yielded the value First!. So, what's the value of the variable second?

That's actually the value that we pass to the next method the next time we call it! This time, let's pass the string 'I like JavaScript'.

Alt Text

It's important to see here that the first invocation of the next method doesn't keep track of any input yet. We simply start the observer by invoking it the first time. The generator waits for our input, before it continues, and possibly processes the value that we pass to the next method.


So why would you ever want to use generator functions?

One of the biggest advantages of generators is the fact that they are lazily evaluated. This means that the value that gets returned after invoking the next method, is only computed after we specifically asked for it! Normal functions don't have this: all the values are generated for you in case you need to use it some time in the future.

Alt Text

There are several other use cases, but I usually like to do it to have way more control when I'm iterating large datasets!

Imagine we have a list of book clubs! πŸ“š To keep this example short and not one huge block of code, each book club just has one member. A member is currently reading several books, which is represented in the books array!

Now, we're looking for a book with the id ey812. In order to find that, we could potentially just use a nested for-loop or a forEach helper, but that means that we'd still be iterating through the data even after finding the team member we were looking for!

The awesome thing about generators, is that it doesn't keep on running unless we tell it to. This means that we can evaluate each returned item, and if it's the item we're looking for, we simply don't call next! Let's see what that would look like.

First, let's create a generator that iterates through the books array of each team member. We'll pass the team member's book array to the function, iterate through the array, and yield each book!

Perfect! Now we have to make a generator that iterates through the clubMembers array. We don't really care about the club member itself, we just need to iterate through their books. In the iterateMembers generator, let's delegate the iterateBooks iterator in order to just yield their books!

Almost there! The last step is to iterate through the bookclubs. Just like in the previous example, we don't really care about the bookclubs themselves, we just care about the club members (and especially their books). Let's delegate the iterateClubMembers iterator and pass the clubMembers array to it.

In order to iterate through all this, we need to get the generator object iterable by passing the bookClub array to the iterateBookClubs generator. I'll just call the generator object it for now, for iterator.

Let's invoke the next method, until we get a book with the id ey812.

Alt Text

Nice! We didn't have to iterate through all the data in order to get the book we were looking for. Instead, we just looked for the data on demand! of course, calling the next method manually each time isn't very efficient... So let's make a function instead!

Let's pass an id to the function, which is the id of the book we're looking for. If the value.id is the id we're looking for, then simply return the entire value (the book object). Else, if it's not the correct id, invoke next again!

Alt Text

Of course this was a tiny tiny data set. But just imagine that we have tons and tons of data, or maybe an incoming stream that we need to parse in order to just find one value. Normally, we'd have to wait for the entire dataset to be ready, in order to begin parsing. With generator functions, we can simply require small chunks of data, check that data, and the values are only generated when we invoke the next method!


Don't worry if you're still all "what the heck is happening" mindset, generator functions are quite confusing until you've used them yourself and had some solid use cases for it! I hoped some terms are a bit clearer now, and as always: if you have any questions, feel free to reach out! πŸ˜ƒ

✨ Twitter πŸ‘©πŸ½β€πŸ’» Instagram πŸ’» GitHub πŸ’‘ LinkedIn πŸ“· YouTube πŸ’Œ Email

Discussion

pic
Editor guide
Collapse
vaibhavkhulbe profile image
Vaibhav Khulbe

This visualization series is so good. I can totally relate to the context! Thank you for doing all the hard work 😁

PS: Like I said earlier, "better than my college professors!" πŸ˜›

Collapse
daviddaxi95 profile image
David Daxbacher

Very nice explanation and visualizations of the code. Well done. But I am still missing valid use cases where to use the generator functions in "real" applications, because your last example can also simple and efficient realized by using the es5+ features (flatMap and find):

const id = 'ey812';
const book = bookclubs
  .flatMap(club => club.clubMembers)
  .flatMap(member => member.books)
  .find(book => book.id === id);

Maybe I have to explicitly search for some use cases and dive a little deeper into this topic, until I finally have the "ahhhhh" effect :)

Collapse
lydiahallie profile image
Lydia Hallie Author

Your example works when we already have the all data available (to give bookclubs a value) and for datasets that aren't too big. However, we would be storing the entire bookclubs array in memory, which is something we sometimes want to avoid when working with a lot of data which might be useless.

A good example for which I often use generators is decoding a stream. In your example, we'd have to wait before we've received the entire stream before we can start decoding it (in order to give bookclubs a value). By iterating over smaller pieces of the stream, and decoding these smaller pieces, we can already start decoding it right from the beginning instead of having to wait. If you're looking for a specific piece of data that may be right at the beginning of the stream, it means that we don't have to call next again and don't have to use more memory in order to store the rest of our data, which would be useless.

(Although this is a micro-optimization which doesn't matter in most cases, I'm also not sure about performance of flatMap when working with larger, deeply nested datasets.)

Collapse
tstsankov profile image
TSTsankov

const findbook = (bookID) => {
for(var i=0; i<members.length; i++)
for(var j=0; j<member[i].Books.length; j++)
if(members[i].Books[j].id == bookID)
return members[i].Books[j];
}

findbook("ey812");

Shouldn't the return statement just finish execution if/when book is found?

Collapse
daviddaxi95 profile image
David Daxbacher

Ok now i got the point, makes sense. Thanks! πŸ‘

Collapse
thepeoplesbourgeois profile image
Josh

Teach them reduce, teach them reduce!

Collapse
glencodes profile image
Glen Codes

This is the most insane kung fu magic ever. Thank you for explaining this. Very useful. The visual animations are very good too, definitely think you would be struggling to explain it without the animations to assist :)

Collapse
conermurphy profile image
Coner Murphy

Love this series and more importantly your blog posts. From someone, who is relatively new to the dev world these posts are a god send. I'd love to know what you make the animations in, would love to use it for similar stuff in my own blog posts. πŸ”₯

Collapse
pinutz23 profile image
Jannik Wempe

Wow, awesome work with all these animations (and emojis) πŸ†’! It really helps me understanding the concept. I think there would be no way of understanding it without visuals. Keep on that great work!

Collapse
codenutt profile image
Jared

This is going to help to solve Project Euler problems, thank you!

Collapse
jsn1nj4 profile image
JSn1nj4β€β€πŸ‘¨β€πŸ’»

I was doing a little exercise recently that this would've helped me very much on. Now I'll be able to simplify what I wrote very much. Thanks!

And yeah, I realize I'm reading this article late, but it's still just as helpful. This is pretty awesome. I bet there are a number of things I could refactor in a simpler way using these concepts...

Collapse
imdanwu profile image
Daniel Wu

Hey love this series! This was one topic I had trouble diving into. Thanks!

Also, I think the last example has a small mistake. If the book you're looking for is the very last book, then the while loop will end before checking it.

Collapse
artoodeeto profile image
aRtoo

Ma'am, you are the GOAT! I'm telling you! You helped my brain cells.

Collapse
hugoliconv profile image
Hugo

What do you use to create those amazing animations?

Collapse
corentinbettiol profile image
Corentin Bettiol

I had already found the previous posts but I didn't have an account at the time.
Thank you for writing such wonderful posts!

Collapse
hem profile image
Hem

This is great and it inspires me ❀️

Collapse
rhymes profile image
rhymes

Love it! Great explanation and gifs :)

Collapse
fabricioffv profile image
Fabricio Filipe Viapiana

Hi Lydia
Does the iterators have more performance than array.filter/reduce on searching for data in a huge array? I mean, which one is the best, an interatior funcion or filter/reduce?

Collapse
holtkam2 profile image
Jason Holtkamp

Animations on point!

Collapse
gigamegs profile image
Gigamegs

Why not using break?

Collapse
hellovietduc profile image
Duc Nguyen

I have the same question. Still in the stage of "oh gosh no I've read so many blog posts about generator functions and I still don't get them"!

Collapse
suzettemccanny profile image
Suzette McCanny

This is so fun! Thank you!

Collapse
diegomgar profile image
Dieg Oto

Very nice series! Continue like this. I can say now that i understand generators :) TY!