JavaScript Generators

Below is an excerpt from my free e-book Async JavaScript. You can read the full book on https://asyncjsbook.com

Generators are special functions that generate values when you need them to. When you call a generator it will not execute like a normal function. It will execute to the point where it sees a yield statement and it will exit until you need a new value. When you want a new value, you ask the generator for the next value and it will execute the function again from where it left off until there are no more values to generate. In the following sections we will learn how to create generators, how to iterate over them, how to stop them and more.

Creating Generators

You can create a generator by placing a * after the function keyword:

function* myGenerator() {
 //...
}

Next, in the body of the generator function, we can generate values using the yield statement:

// [file]: code/generators/simple.js
function* simpleGenerator() {
  yield 1;
  yield 5;
}
const g = simpleGenerator();
const v1 = g.next().value; // --> 1
const v2 = g.next().value; // --> 5
const v3 = g.next().value; // --> undefined

You can even define an infinite loop and generate values:

// [file]: code/generators/inf-loop.js
function* myGenerator() {
  let i = 0;
  while(true) {
    i += 1;
    yield i;
  }
}

Now if it were a normal function, it would get stuck in an infinite loop. But because this is a generator we can read values generated by calling next on the generator object returned:

const g = myGenerator();
const v1 = g.next(); // --> { value: 1, done: false }
const v2 = g.next(); // --> { value: 2, done: false }
const v3 = g.next(); // --> { value: 3, done: false }
// and so on...

Essentially, we enter and exit the function every time we call next and we pick up from where we last left off. Notice how the value of i is "remembered" every time we call next. Now let's update the code above and make the generator finish generating values. Let's make it so that it won't generate any values if i is bigger than 2:

function* myGenerator() {
  let i = 0;
  while(true) {
    i += 1;
    if(i > 2) {
      return;
    }
    yield i;
  }
}

or we can simplify the code above and move the condition to the while loop:

// [file]: code/generators/inf-loop-terminate.js
function* myGenerator() {
  let i = 0;
  while(i < 2) {
    i += 1;
    yield i;
  }
}

Now if we read the generated values, we will only get two values out:

const g = myGenerator();
const v1 = g.next(); // --> { value: 1, done: false }
const v2 = g.next(); // --> { value: 2, done: false }
const v3 = g.next(); // --> { value: undefined, done: true }

Notice that after the second value, if we keep calling next, we will get the same result back. That is, a generator object with a value of undefined and the done property set to true indicating that there will be no more values generated.

Return Statements

A return statement in a generator marks the last value and no values will be generated after that:

// [file]: code/generators/return-statement.js
function* withReturn() {
  yield 1;
  yield 55;
  return 250;
  yield 500;
}
const g = withReturn();
const v1 = g.next().value; // --> 1
const v2 = g.next().value; // --> 55
const v3 = g.next().value; // --> 250
const v4 = g.next().value; // --> undefined

The code above will generate 1, 55 and 250. It will not reach the final yield statement, because the return statement marks the end of the generator.

Passing Values to Next

Using generators, you can pass a value to the next callback to use in place of the previously calculated yield statement. Let's look at a simple example to demonstrate what that means.

// [file]: code/generators/pass-next.js
function* myGenerator(n) {
  const a = (yield 10) + n;
  yield a;
}

const g = myGenerator(1);
g.next().value; // --> 10
g.next(100).value; // --> 101

Let's go through the snippet above and explore what happens step by step:

• First we call the generator and we pass 1 for n, and store the iterator object in g. Nothing new here.
• Then, we call g.next to start the generator. The function is executed until it reaches the first yield statement: const a = (yield 10). At this point the value next to yeild is generated which is 10.
• Then we call g.next and we pass 100. The function resumes from where it left off: + n but it will replace 100 for (yield 10) resulting in const a = 100 + n where n is 1. It will continue until it hits the next yield. In this case yield a which will generate 100 + 1 = 101.

We will use this special behavior of generators in later sections to implement a helper to handle async flows.

Calling Another Generator Within a Generator

You can use yield* inside a generator if you want to call another generator. In the example below, we have two generators, g1 and g2. We want to call g2 inside g1 and read the generated values:

// [file]: code/generators/call-another.js
function* g2() {
  yield 2;
  yield 3;
}
function* g1() {
  yield 1;
  yield* g2();
  yield 4;
}

const vals = [...g1()];

console.log(vals); // -> [1,2,3,4]

In the snippet above we call the g1 generator and below is a summary of what happens:

• The 1 value is generated from the first yield statement
• Next, we hit yield* g2() which will generate all the values that g2 would generate, that is 2 and 3
• Next, we come back to g1 and generated the final value, which is 4

Iterating Through Values

Using for-of

Since a generator function returns an iterable, we can use the for-of loop to read each generated value. Using the simple generator from above, we can write a loop to log each generated value:

// [file]: code/generators/use-for-of.js
function* myGenerator() {
  let i = 0;
  while(i < 2) {
    i += 1;
    yield i;
  }
}

const g = myGenerator();
for(const v of g) {
  console.log(v);
}

The code above will output 1 and then 2.

Using while Loop

You can also use a while loop to iterate through a generator object:

// [file]: code/generators/use-while-loop.js
const g = myGenerator();
let next = g.next().value;
while(next) {
  console.log(next);
  next = g.next().value;
}

In the while loop above, first we get the first generated value and we assign it to next. Then in the while loop, we set next to the next generated value. The while loop will keep going until next becomes undefined when the generator yields the last value.

Spread Operator and Array.from

Because a generator object is an iterable you can also use the spread operator to read the values:

// [file]: code/generators/use-spread.js
function* myGenerator() {
  let i = 0;
  while(i < 2) {
    i += 1;
    yield i;
  }
}
const vals = [...myGenerator()]; // -> [1, 2]

In the example above first we call the generator myGenerator() and we place it in an array. And finally we use the spread operator right before it to essentially read each value out. The result is stored in the vals variable as an array with two values [1, 2].

In addition to the spread operator, you can also use the Array.from method to read the values and put them in an array:

// [file]: code/generators/use-array-from.js
function* myGenerator() {
  let i = 0;
  while(i < 2) {
    i += 1;
    yield i;
  }
}
const vals = Array.from(myGenerator()); // --> [1, 2]

In the snippet above we call the generator and we pass it to Array.from which will read each value and store them in an array, resulting in [1, 2].

It's worth mentioning that if you are iterating through a generator object that includes a return statement terminating the sequence, you won't be able to read the last value if you use any of the internal iteration methods like for-of loop or the spread operator:

function* withReturn() {
  yield 1;
  yield 55;
  return 250;
  yield 500;
}
for(const v of withReturn()) {
  console.log(v);
}

The code above will output 1 and then 55 but it won't output 250. This is also true if you use the spread operator:

function* withReturn() {
  yield 1;
  yield 55;
  return 250;
  yield 500;
}
const vals = [...withReturn()];
console.log(vals);

The code above will output [1, 55] and will not include 250. But notice that if we use a while loop, we can read all the values up until the value at the return statement:

function* withReturn() {
  yield 1;
  yield 55;
  return 250;
  yield 500;
}

const g = withReturn();
let next = g.next().value;

while(next) {
  console.log(next);
  next = g.next().value;
}

The while loop above will read all the values, including the value at the return statement, logging 1, 55, and 250 to the console.

Generating Infinite Sequences

In this section we are going to look at creating a Fibonacci sequence using a generator function. Note that the code used in this section is only for demonstration purposes. For practical purposes, you probably would want to use a pre-generated list to retrieve values for better performance.

The Fibonacci sequence is a sequence of numbers that starts with 0, and 1. And the rest of the numbers in the sequence is calculated by adding the current value with the previous one:

0, 1, 1, 2, 3, 5, 8, 13, 21, ...

or recursively, the sequence can be define as:

fib(n) = fib(n - 1) + fib(n - 2)

We can use the definition above and define a generator to produce n number of values:

// [file]: code/generators/fibo.js
function* fibo(n, prev = 0, current = 1) {
  if (n === 0) {
    return prev;
  }
  yield prev;
  yield* fibo(n - 1, current, prev + current);
}

let vals = [...fibo(5)];
console.log(vals); //-> [ 0, 1, 1, 2, 3 ]

In the snippet above we define the first two numbers as default argument values using prev = 0 and current = 1. Below is a summary of what happens for n = 5:

1. The first yield will generate the prev value, that is 0. Note that n is 4 now.
2. Next, fibo(4 - 1, 1, 0 + 1) = fib(3, 1, 1) will generate 1.
3. Next, fibo(3 - 1, 1, 1 + 1) = fibo(2, 1, 2) will generate 1.
4. Next, fibo(2 - 1, 2, 1 + 2) = fibo(1, 2, 3) will generate 2.
5. Next, fibo(1 - 1, 3, 2 + 3) = fibo(0, 3, 5) will generate 3, marking the end since n is 0 and we hit the return statement.

Generators and Async Operations

We can take advantage of the unique features of generators to essentially wait for async operations to finish before moving onto other parts of a function. In this section, we are going to write a helper function to allow us to do just that. But, first let's review what happens when you pass g.next an argument. If you remember from the previous sections, if you pass g.next an argument, it is going to replace the given value with the previously yielded result:

function* myGenerator(n) {
  const a = (yield 10) + n;
  yield a;
}

const g = myGenerator(1);
g.next().value; // --> 10
g.next(100).value; // --> 101

We are going to use that as the foundation for our helper function. Now, first let' start by making an asynchronous function that returns a promise:

const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));

This function returns a promise that resolves to the value 1 after 1 second. Now, let's create a generator function and call our async function inside it:

const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));

function* main() {
  const result = yield asynTask1();
}

const g = main();
console.log(g.next());

What do you think the code above will output? Let's go through it and figure out what's going to happen:

• First, we call the generator and store the generator object in g.
• Then, we call next to get the first yield result. In this case it's going to be a promise since asynTask1 returns the promise.
• Finally we log the value to the console: { value: Promise { <pending> }, done: false }.
• After 1 second the program ends.

After the program ends we won't get access to the resolved value. But imagine, if we could call next again and pass the resolved value to it at the "right" time. In that case, yield asynTask1() will be replaced with the resolved value and it would be assigned to result! Let's update the code above and make that happen with one promise:

const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));

function* main() {
  const result = yield asynTask1();
  return result; //<-- return the resolved value and mark the end.
}

const g = main();
const next = g.next();
console.log(next); // --> { value: Promise { <pending> }, done: false }
next.value.then(v => { // Resolve promise.
  const r = g.next(v); // passing the resolved value to next.
  console.log(r); // -> { value: 1, done: true }
});

In the snippet above we added a return statement in the generator to simply return the resolved value. But the important part is when we resolve the promise. When we resolve the promise, we call g.next(v) which replaces the yield asynTask1() with the resolved value and will assign it to result. Now, we are ready to write our helper function. This helper function is going to accept a generator and do what we discussed above. It is going to return the resolved value if there are no more values to be generated. We'll start by defining the helper function:

const helper = (gen) => {
  const g = gen();
};

So far nothing special, we pass our helper a generator function and inside the helper we call the generator and assign the generator object to g. Next, we need to define a function that is going to handle calling next for us:

const helper = (gen) => {
  const g = gen();
  function callNext(resolved) {
    const next = g.next(resolved); // replace the last yield with the resolved value
    if(next.done) return next.value; // return the resolved value if not more items
    return next.value.then(callNext); // pass `callNext` back again.
  }
};

This function is going to take a single argument, the resolved value of a promise. Then, we call g.next with the resolved value, and will assign the result to the next variable. After that we will check if the generator is done. If so, we will simply return the value. And finally, we call next.value.then() and we will pass callNext back to it to recursively call the next for us until there no more values to generate. Now, to use this helper function, we will simply call it and we will pass our generator to it:

helper(function* main() {
  const a = yield asynTask1();
  console.log(a);
});

Now if you run the code above, you won't see the logged result, and that's because we have one missing piece. The callNext function in our helper needs to be immediately self-invoked, otherwise no one will call it:

const helper = (gen) => {
  const g = gen();
  (function callNext(resolved) {
    const next = g.next(resolved);
    if(next.done) return next.value;
    return next.value.then(callNext);
  }()); // <-- self invoking
};

Now that we have our helper function, let's throw an error in the helper so that we can catch it later:

const helper = (gen) => {
  const g = gen();
  (function callNext(resolved) {
    const next = g.next(resolved);
    if(next.done) return next.value;
    return next.value.then(callNext)
    .catch(err => g.throw(err)); // <-- throw error
  }());
};

The catch block will throw an error from the generator if any of the promises throws an error. And we can simply use a try-catch in the passed in generator function to handle errors. Putting it all together we will have:

// [file]: code/generators/async-flow.js
const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));
const asynTask2 = () => new Promise((r, j) => setTimeout(() => j(new Error('e')), 500));

const helper = (gen) => {
  const g = gen();
  (function callNext(resolved) {
    const next = g.next(resolved);
    if(next.done) return next.value;
    return next.value.then(callNext)
    .catch(err => g.throw(err));
  }());
};

helper(function* main() {
  try {
    const a = yield asynTask1();
    const b = yield asynTask2();
    console.log(a, b);
  } catch(e) {
    console.log('error happened', e);
  }
});

If you are curious, you can take a look the co library for a more comprehensive implementation. We will however look at the async-await abstraction in the next chapter which is a native abstraction over generators for handling async flows.

Comments

[Kushan Joshi]

Great article AJ, you can also use for-await-of but it works for async iterables, you can also use them with an array of promises.

Here's how you would create an async iterable.

const fetchNextPage = page => Promise.resolve(page);

async function* asyncGen() {
  let page = 0;
  while (page < 10) yield fetchNextPage(page++);
}

for await (const g of asyncGen()) {
  console.log(g);
}

The best part about this approach is concurrency, the async generator asyncGen only creates a new promise when asked for, and the for-await-of automatically awaits at the start of for loop, resolves it and puts it as the const g.

[AJ Meyghani]

Thanks for your comment Kushan!