Understanding Go 1.23 iterators

wmdanor — Sat, 17 Aug 2024 15:12:42 +0000

Many people seem to be confused by newly added iterators in Go, that is why I decided to write another one article attempting to explain them in a simple as possible way.

How are they called by Go?

First, I think it is important to understand how iterators are even being called and used by Go, and it is actually pretty simple, let's use slices.All iterator as an example. Here is how you would normally use this iterator:

package main

import (
    "fmt"
    "slices"
)

func main() {
    slice := []string{
        "Element 1",
        "Element 2",
        "Element 3",
        "Element 4",
    }

    for index, element := range slices.All(slice) {
        if index >= 2 {
            break
        }
        fmt.Println(index, element)
    }

    // Output:
    // 0 Element 1
    // 1 Element 2
}

And here is how it actually looks like:

package main

import (
    "fmt"
    "slices"
)

func main() {
    slice := []string{
        "Element 1",
        "Element 2",
        "Element 3",
        "Element 4",
    }

    slices.All(slice)(func (index int, element string) bool {
        if index >= 2 {
            return false // break
        }
        fmt.Println(index, element)

        return true // continue loop as normal
    })

    // Output:
    // 0 Element 1
    // 1 Element 2
}

What happens is the loop body being "moved" to yield function that is passed to the iterator, while continue and break are being transformed to return true and return false respectively. return true is also added to end of the loop to signalize that we would like to get next element, if nothing else made another decision before.

This is not exact unfold of what is compiler is doing and I have not checked Go implementation to check this, but they do produce equivalent results from my observations.

How to create your own iterator and its execution

Now, that you understand how they are being called and realized how simple it actually is, it will be much easier to understand how to create your own iterator and its execution.

Let's create a debug iterator that will print debug messages for each step of iterator implementation that will walk over all elements in slice (slices.All functionality).

First, I will create small helper function to log out message with current execution time.

import (
    "fmt"
    "time"
)

var START time.Time = time.Now()

func logt(message string) {
    fmt.Println(time.Since(START), message)
}

Back to iterator:

import (
    "iter"
)

func DebugIter[E any](slice []E) iter.Seq2[int, E] {
    logt("DebugIter called")

    // the same way iter.All returned function
    // we called in order to iterate over slice
    // here we are returning a function to
    // iterate over all slice elements too
    return func(yield func(int, E) bool) {
        logt("Seq2 return function called, starting loop")
        for index, element := range slice {
            logt("in loop, calling yield")
            shouldContinue := yield(index, element)
            if !shouldContinue {
                logt("in loop, yield returned false")
                return
            }
            logt("in loop, yield returned true")
        }
    }
}

I have added few debug print statements so we could better see the order of execution of the iterator and how it will react to different keywords like break and continue.

Finally, let's use implemented iterator:

func main() {
    slice := []string{
        "Element 1",
        "Element 2",
        "Element 3",
        "Element 4",
    }

    for index, element := range DebugIter(slice) {
        message := "got element in range of iter: " + element
        logt(message)
        if index >= 2 {
            break
        }
        if index > 0 {
            continue
        }
        time.Sleep(2 * time.Second)
        logt("ended sleep in range of iter")
    }
}

Will give us the output:

11.125µs DebugIter called
39.292µs Seq2 return function called, starting loop
42.459µs in loop, calling yield
44.292µs got element in range of iter: Element 1
2.001194292s ended sleep in range of iter
2.001280459s in loop, yield returned true
2.001283917s in loop, calling yield
2.001287042s got element in range of iter: Element 2
2.001291084s in loop, yield returned true
2.001293125s in loop, calling yield
2.0012955s got element in range of iter: Element 3
2.001297542s in loop, yield returned false

This example shows pretty well how iterators works and executed. When using iterator in range loop all instructions in loop block are kind of "moved" to a function that is called yield. When we call yield we essentially ask go runtime to execute whatever located in loop block with following value for this iteration, that is also why yield will be blocked, if the loop body will get blocked. In case runtime determines that this loop iteration supposed to stop, yield will return false, it can happen when break keyword is met during loop block execution, we should not call yield anymore if that happens. Otherwise, we should continue calling yield.

Full code:

package main

import (
    "fmt"
    "time"
    "iter"
)

var START time.Time = time.Now()

func logt(message string) {
    fmt.Println(time.Since(START), message)
}

func DebugIter[E any](slice []E) iter.Seq2[int, E] {
    logt("DebugIter called")

    // the same way iter.All returned function
    // we called in order to iterate over slice
    // here we are returning a function to
    // iterate over all slice elements too
    return func(yield func(int, E) bool) {
        logt("Seq2 return function called, starting loop")
        for index, element := range slice {
            logt("in loop, calling yield for")
            shouldContinue := yield(index, element)
            if !shouldContinue {
                logt("in loop, yield returned false")
                return
            }
            logt("in loop, yield returned true")
        }
    }
}

func main() {
    slice := []string{
        "Element 1",
        "Element 2",
        "Element 3",
        "Element 4",
    }

    for index, element := range DebugIter(slice) {
        message := "got element in range of iter: " + element
        logt(message)
        if index >= 2 {
            break
        }
        if index > 0 {
            continue
        }
        time.Sleep(2 * time.Second)
        logt("ended sleep in range of iter")
    }

    // unfold compiler magic
    //  DebugIter(slice)(func (index int, element string) bool {
    //    message := "got element in range of iter: " + element
    //    logt(message)
    //    if index >= 2 {
    //      return false
    //    }
    //    if index > 0 {
    //      return true
    //    }
    //    time.Sleep(2 * time.Second)
    //    logt("ended sleep in range of iter")
    //
    //    return true
    //  })
}

DEV Community: wmdanor

Understanding Go 1.23 iterators

How are they called by Go?

How to create your own iterator and its execution