Generator Concurrency Pattern in Go: A Comprehensive Guide

⚠️ How to go about this series?

1. Run Every Example: Don't just read the code. Type it out, run it, and observe the behavior.
2. Experiment and Break Things: Remove sleeps and see what happens, change channel buffer sizes, modify goroutine counts.
Breaking things teaches you how they work
3. Reason About Behavior: Before running modified code, try predicting the outcome. When you see unexpected behavior, pause and think why. Challenge the explanations.
4. Build Mental Models: Each visualization represents a concept. Try drawing your own diagrams for modified code.

In our previous post, we explored the basics of goroutines and channels, the building blocks of Go's concurrency. Read here:

Understanding and visualizing Goroutines and Channels in Golang

Souvik Kar Mahapatra ・ Dec 20

#go #programming #learning #tutorial

Now, let's look at how these primitives combine to form powerful patterns that solve real-world problems.

In this post we'll cover Generator Pattern and will try to visualize them. So let's gear up as we'll be hands on through out the process.

Generator Pattern

A generator is like a fountain that continuously produces values that we can consume whenever needed.

In Go, it's a function that produces a stream of values and sends them through a channel, allowing other parts of our program to receive these values on demand.

Let's look at an example:

// generateNumbers creates a generator that produces numbers from 1 to max
func generateNumbers(max int) chan int {
    // Create a channel to send numbers
    out := make(chan int)

    // Launch a goroutine to generate numbers
    go func() {
        // Important: Always close the channel when done
        defer close(out)

        for i := 1; i <= max; i++ {
            out <- i  // Send number to channel
        }
    }()

    // Return channel immediately
    return out
}

// Using the generator
func main() {
    // Create a generator that produces numbers 1-5
    numbers := generateNumbers(5)

    // Receive values from the generator
    for num := range numbers {
        fmt.Println("Received:", num)
    }
}

In this example, our generator function does three key things:

1. Creates a channel to send values
2. Launches a goroutine to generate values
3. Returns the channel immediately for consumers to use

Why Use Generators?

1. Separate value production from consumption
2. Generate values on-demand (lazy evaluation)
3. Can represent infinite sequences without consuming infinite memory
4. Allow concurrent production and consumption of values

Real-world Use Case

Reading large files line by line:

func generateLines(filename string) chan string {
    out := make(chan string)
    go func() {
        defer close(out)
        file, err := os.Open(filename)
        if err != nil {
            return
        }
        defer file.Close()

        scanner := bufio.NewScanner(file)
        for scanner.Scan() {
            out <- scanner.Text()
        }
    }()
    return out
}

Now you might be thinking, what's so special about it? we can do the same like generating sequence of data or read line by line without goroutines. Isn't it an overkill? Let's try to visualize both cases:

Without the goroutines

// Traditional approach
func getNumbers(max int) []int {
    numbers := make([]int, max)
    for i := 1; i <= max; i++ {
        numbers[i-1] = i
        // Imagine some heavy computation here
        time.Sleep(100 * time.Millisecond)
    }
    return numbers
}

Here you have to wait for everything to be ready before you can start processing.

With goroutines

// Generator approach
func generateNumbers(max int) chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := 1; i <= max; i++ {
            out <- i
            // Same heavy computation
            time.Sleep(100 * time.Millisecond)
        }
    }()
    return out
}

You can start processing the data while the data is still being generated.

Key Benefits of Generator Pattern:

1. Non-Blocking Execution: Generation and processing happen simultaneously

2. Memory Efficiency: Can generate and process one value at a time, no need to store in the memory right away

3. Infinite Sequences: Can generate infinite sequences without memory issues

4. Backpressure Handling: If your consumer is slow, the generator naturally slows down (due to channel blocking), preventing memory overload.
   

// Generator naturally handles slow consumers (backpressure handling)
for line := range generateLines(bigFile) {
    // Take our time processing each line
    // Generator won't overwhelm us with data
    processSlowly(line)
}

Common Pitfalls and Solutions

1. Forgetting to Close Channels

// Wrong ❌
func badGenerator() chan int {
    out := make(chan int)
    go func() {
        for i := 1; i <= 5; i++ {
            out <- i
        }
        // Channel never closed!
    }()
    return out
}

// Right ✅
func goodGenerator() chan int {
    out := make(chan int)
    go func() {
        defer close(out)  // Always close when done
        for i := 1; i <= 5; i++ {
            out <- i
        }
    }()
    return out
}

1. Not Handling Errors

// Better approach with error handling
func generateWithErrors() (chan int, chan error) {
    out := make(chan int)
    errc := make(chan error, 1)  // Buffered channel for error

    go func() {
        defer close(out)
        defer close(errc)

        for i := 1; i <= 5; i++ {
            if i == 3 {
                errc <- fmt.Errorf("error at number 3")
                return
            }
            out <- i
        }
    }()

    return out, errc
}

1. Resource Leaks: When using generators with resources (like files), ensure proper cleanup:

func generateFromFile(filename string) chan string {
    out := make(chan string)
    go func() {
        defer close(out)

        file, err := os.Open(filename)
        if err != nil {
            return
        }
        defer file.Close()  // Important: Close file when done

        scanner := bufio.NewScanner(file)
        for scanner.Scan() {
            out <- scanner.Text()
        }
    }()
    return out
}

That wraps up our deep dive into the Generator pattern! Coming up next, we'll explore the Pipeline concurrency pattern, where we'll see how to chain our generators together to build powerful data processing flows.

If you found this post helpful, have any questions, or want to share your own experiences with generators - I'd love to hear from you in the comments below. Your insights and questions help make these explanations even better for everyone.

Stay tuned for more Go concurrency patterns! 🚀