DEV Community: Tiago Temporin

Mastering memory management in Go: Avoiding slice-related leaks

Tiago Temporin — Thu, 05 Dec 2024 17:28:30 +0000

Go is a programming language recognized for its efficiency and automatic memory management through the Garbage Collector (GC). However, even with these advantages, applications written in Go can experience memory leaks, especially when slices are improperly handled.

In this post, we’ll explore what memory leaks are, how they can occur in slices, and best practices to avoid them.

What is a Memory Leak

A memory leak happens when a program allocates memory for temporary use and fails to release it afterward. This results in an increasing memory footprint, which can degrade performance or even exhaust available memory, causing application failures.

In languages with automatic memory management, such as Go, the Garbage Collector is responsible for freeing unused memory. However, if there are active references to memory regions that are no longer needed, the GC cannot reclaim them, leading to a memory leak.

To better understand how the GC works, I recommend reading the post “Unveiling the Garbage Collector in Go”.

Memory Leak in Slices

When you create a slice from an array or another slice, it references the same underlying array. In other words, if the original slice is large, and you create a small sub-slice, the entire array remains in memory as long as the sub-slice exists.

Example:

func main() {
    largeSlice := make([]byte, 1<<20) // 1MB slice
    smallSlice := largeSlice[:10]    // 10-byte sub-slice

    // largeSlice is no longer used but still occupies 1MB of memory
    process(smallSlice)
}

func process(data []byte) {
    // Process the data
}

In this example, even though only 10 bytes are used, the entire 1MB remains in memory due to the reference held by smallSlice.

Essential Rule!

Whenever a slice element is a pointer or a struct field is a pointer, the elements will not be removed by the Garbage Collector (GC).

How to Avoid It

1. Copy Only the Needed Data

If you only need a small part of a large slice, copy the data to a new slice to eliminate the reference to the original array.

Corrected Example:

func main() {
    largeSlice := make([]byte, 1<<20) // 1MB slice
    smallSlice := make([]byte, 10)
    copy(smallSlice, largeSlice[:10]) // Copy only the necessary 10 bytes

    largeSlice = nil // Remove the reference to the large slice
    process(smallSlice)
}

func process(data []byte) {
    // Process the data
}

Now, the 1MB array can be collected by the GC since there are no active references to it.

2. Set Unused Slices to `nil`

After finishing with a large slice, set it to nil to remove references to the underlying array.

Example:

func main() {
    data := loadData()
    // Use the data
    processData(data)
    data = nil // Allow GC to release memory
}

func loadData() []byte {
    // Load data into a large slice
}

func processData(data []byte) {
    // Process the data
}

3. Manage Slice Growth in Loops

Avoid slices growing indefinitely in loops. If possible, preallocate the required capacity or reset the slice after use.

Example:

func main() {
    data := make([]int, 0, 1e6) // Preallocate capacity

    for i := 0; i < 1e6; i++ {
        data = append(data, i)
        if len(data) == cap(data) {
            processData(data)
            data = data[:0] // Reset the slice for reuse
        }
    }
}

func processData(data []int) {
    // Process the data
}

Conclusion

Even with Go’s automatic memory management, it’s crucial for developers to understand how slices work to avoid memory leaks.

By being aware of how references in slices can keep large arrays in memory and applying practices like copying necessary data and clearing references, you can write more efficient and reliable code.

Always monitor your application’s memory usage and leverage available tools to identify and fix potential memory leak issues.

See you next time!

Unveiling the Garbage Collector in Go

Tiago Temporin — Wed, 27 Nov 2024 09:33:58 +0000

The Garbage Collector (GC) is one of the key features of the Go programming language, designed to simplify memory management for developers. Unlike languages like C and C++, where programmers must manually allocate and release memory, the GC in Go automates this process.

In this post, we'll explore how the Garbage Collector works in Go, understand its behavior in different scenarios, and identify pitfalls that can lead to memory leaks even with GC in place.

What is a Garbage Collector?

A Garbage Collector is an automated mechanism responsible for reclaiming memory allocated to objects that are no longer used in a program. In Go, it identifies variables and data structures that are no longer accessible or referenced in the code, then releases their memory for reuse. This improves application efficiency and prevents issues like memory leaks.

Go employs a mark-and-sweep garbage collection model. This algorithm operates in two main phases:

Mark Phase: The GC traverses all references of active objects in memory, starting from entry points (such as global variables and execution stacks). Every reachable object is marked as “in use” or “active.”
Sweep Phase: After marking, the GC scans the memory to identify objects not marked as active. These objects are deemed "unreachable" and their memory is freed, making it available for reuse.

This method effectively ensures that memory used by unreferenced objects is reclaimed. While the algorithm is straightforward and helps prevent memory leaks, it can have drawbacks, such as long pauses (stop-the-world) during garbage collection, especially in larger or more complex programs.

To address performance issues, starting with Go version 1.5, the GC became concurrent (executing in parallel with the application code). This minimizes stop-the-world pauses during garbage collection, offering better performance.

Garbage Collector in Action

The Garbage Collector in Go is triggered primarily in two scenarios:

Memory Allocation: Whenever a new variable or object is created, Go allocates memory for it. When the GC detects that these objects are no longer referenced, it collects them and releases the memory.
Memory Fragmentation: The GC ensures that fragmented memory is reclaimed and reused. Without this, an application might run out of free memory even when much of the allocated memory is unused.

Although the Garbage Collector handles most of the heavy lifting, certain coding patterns can cause objects to remain in memory longer than necessary.

This topic is vast and requires a deeper understanding of Go internals. In the next two posts, I’ll cover scenarios involving maps and slices to explain these patterns in more detail without making this post overly long.

Conclusion

Go’s Garbage Collector is a powerful ally for automatic memory management, allowing developers to focus on other aspects of their applications. However, it’s essential to understand its limitations and the common pitfalls that can lead to memory leaks. By learning these nuances, you can write more efficient code and prevent memory-related issues from impacting the performance of your Go application.

To dive deeper into this topic, subscribe to my newsletter and don’t miss the upcoming posts.

See you next time!

Mastering ENUMs in Go

Tiago Temporin — Thu, 21 Nov 2024 13:36:59 +0000

Often, within the systems we develop, we encounter constant values. One example of these values could be the representation of a registration status. In this case, consider a status that includes more variations beyond active and inactive.

If these statuses are defined as strings, their validation within the system could become a major headache. Additionally, this approach might “inflate” the binary, as each validation would involve two strings (the expected value and the value being validated).

To avoid these problems, we can use the well-known enum type. If you’re unfamiliar with this type, it is essentially a fixed or limited-size type.

To make it clearer, let’s dive into some code. Following the idea presented earlier, we’ll create an enum type to validate registration statuses.

Defining a New Type

Creating an enum based on Go’s standard types can be problematic. Let me explain. Imagine we define our status as an uint8 type. Now, suppose our system also has another enum of type uint8 for genre.

Now imagine that the value 1 represents both the Pending status and the Country music genre. What will happen if the validation if Pending == Country is performed? Exactly, it will return true.

To prevent this, we’ll create a new type specifically for handling status. This type will be based on uint8, but since it’s a distinct type, the validation mentioned earlier will not return true.

type Status uint8

Creating the ENUM

With a new type defined, let’s create the constants and their corresponding values for the registration statuses.

const (
  Created Status = 0
  Pending = 1
  Approved = 2
  Rejected = 3
)

Although there’s nothing inherently wrong with assigning values as we did above, there’s a simpler way. Instead of assigning a value to each constant, we can use the iota keyword. This keyword makes Go assign 0 to the first constant and then increment the value by 1 sequentially for each subsequent constant.

const (
  Created Status = iota
  Pending
  Approved
  Rejected
)

Printing the ENUM

As the enum is currently implemented, printing the constant Created would display the value 0. However, for better readability, it’s more helpful to display the word Created instead of the value 0.

The solution is very simple. Just implement the magic String() method.

func (s Status) String() string {
  switch s {
    case Created:
      return "created"
    case Pending:
      return "pending"
    case Approved:
      return "approved"
    case Rejected:
      return "rejected"
   }

   return "unknown"
}

Conclusion

To test this, let’s do a simple print of the Pending status.

package main

import "fmt"

type Status uint8

const (
  Created Status = iota
  Pending
  Approved
  Rejected
)

func (s Status) String() string {
  switch s {
    case Created:
      return "created"
    case Pending:
      return "pending"
    case Approved:
      return "approved"
    case Rejected:
      return "rejected"
  }

  return "unknown"
}

func main() {
  fmt.Println(Pending)
}

Executing the command go run should output pending on terminal.

That’s it! I hope this was helpful.

See you next time!

The difference between pointers and values on methods

Tiago Temporin — Tue, 19 Nov 2024 12:38:08 +0000

When writing methods in Go, one of the key decisions is whether to pass a struct by value or by pointer. This choice can impact performance, code behavior, and memory allocation. In this post, we’ll explore this difference with a practical example and understand when each approach is more appropriate.

Let’s start with a small struct and two methods: one where the struct is passed by value and another by pointer.

package main

import (
    "fmt"
)

type Person struct {
    Name string
    Age  int
}

// Method with struct passed by value
func (p Person) CelebrateBirthdayValue() {
    p.Age++
}

// Method with struct passed by pointer
func (p *Person) CelebrateBirthdayPointer() {
    p.Age++
}

func main() {
    person := Person{Name: "Alice", Age: 30}

    // Passing by value
    person.CelebrateBirthdayValue()
    fmt.Println("After CelebrateBirthdayValue:", person.Age) // Output: 30

    // Passing by pointer
    person.CelebrateBirthdayPointer()
    fmt.Println("After CelebrateBirthdayPointer:", person.Age) // Output: 31
}

Difference between value and pointer

When we pass a struct by value to a method, Go creates a copy of the struct. Any changes made to the struct inside the method won’t affect the original struct because we’re working with an independent copy.

On the other hand, when we pass a struct by pointer, we pass the memory address of the original struct. This means any changes made to the struct inside the method will directly modify the original struct since we’re manipulating the same instance.

In summary:

By value: The method receives a copy of the struct, creating a new memory space.
By pointer: The method receives the memory address of the original struct, pointing to the same memory space.

Heap

When a struct is passed by value, the copy is allocated on the stack, which is generally fast and efficient. However, if the struct is large, the copy can consume significant stack memory.

When a struct is passed by pointer, the pointer itself is allocated on the stack, but the original struct might be allocated on the heap, especially if it was created using new, make, or captured by an anonymous function.

Heap allocation is more expensive in terms of allocation time and garbage collection but allows efficient manipulation of large amounts of data without copying the entire struct.

When to use each approach

By value

Passing structs by value is useful when:

You want to ensure the original struct is not modified.
The struct is small, and copying it does not significantly impact performance.
The method simply reads data without needing to alter the internal state.

Example:

func (p Person) GetName() string {
    return p.Name
}

Here, GetName only reads the Name field and returns a string without modifying the struct’s state.

By pointer

Passing structs by pointer is beneficial when:

You need to modify the original struct.
The struct is large, and copying its data would be costly in terms of memory and performance.
You want to avoid unnecessary copies to improve code efficiency.

Example:

func (p *Person) UpdateName(newName string) {
    p.Name = newName
}

In this case, UpdateName directly modifies the Name field of the original struct, which is more efficient than creating a copy.

Conclusion

Deciding whether to pass a struct by value or by pointer when writing methods in Go is an important choice that can affect performance, code behavior, and memory allocation.

Passing by value is useful for ensuring immutability of the struct within the method, while passing by pointer is essential for modifying the original struct and optimizing performance when working with larger structs.

Functional Options Pattern in Go

Tiago Temporin — Mon, 18 Nov 2024 17:45:35 +0000

In software development, flexibility in configuring objects is a common need, especially when dealing with functions and structures with multiple optional parameters.

In Go, as it doesn’t support function overloading, finding an elegant solution to this problem can be challenging. This is where the Functional Options Pattern comes in as an effective and idiomatic approach.

What is the Functional Options Pattern?

The Functional Options Pattern is a design pattern widely used in Go to facilitate the configuration of complex structures or functions, especially when there are multiple optional parameters. It allows you to build objects or configure functions flexibly by using functions that act as “options.”

Instead of directly passing all parameters to a constructor or function, you create separate functions that apply incremental and optional configurations. This approach helps to avoid functions with long lists of parameters or the need for multiple constructors for different use cases.

Applying it in Practice

Let’s consider a common problem in API development: creating a custom HTTP client, where some configurations (like timeout, headers, and retries) are optional. Without the Functional Options Pattern, you would need to pass all possible parameters to the creation function, resulting in less readable and harder-to-maintain code.

Creating an HTTP Client with Optional Settings

Suppose we want to create an HTTP client that allows us to set an optional timeout, custom header*, and the number of retries (reconnection attempts). The traditional solution without the Functional Options Pattern would look something like this:

type HttpClient struct {
    Timeout time.Duration
    Headers map[string]string
    Retries int
}

func NewHttpClient(timeout time.Duration, headers map[string]string, retries int) *HttpClient {
    return &HttpClient{
        Timeout: timeout,
        Headers: headers,
        Retries: retries,
    }
}

If you don’t want to set all parameters, you would still need to pass default or zero values, which could cause confusion and hinder readability.

Now, let’s look at how to apply the Functional Options Pattern to improve this approach.

Implementing the Functional Options Pattern

Here, we’ll use functions that encapsulate options, allowing you to configure only the parameters you really need.

type HttpClient struct {
    Timeout time.Duration
    Headers map[string]string
    Retries int
}

// Option defines the function type that will apply settings to HttpClient
type Option func(*HttpClient)

// NewHttpClient creates an HttpClient instance with functional options
func NewHttpClient(opts ...Option) *HttpClient {
    client := &HttpClient{
        Timeout: 30 * time.Second, // default value
        Headers: make(map[string]string),
        Retries: 3, // default value
    }

    // Apply the provided options
    for _, opt := range opts {
        opt(client)
    }

    return client
}

// WithTimeout allows you to set the client timeout
func WithTimeout(timeout time.Duration) Option {
    return func(c *HttpClient) {
        c.Timeout = timeout
    }
}

// WithHeaders allows you to add custom headers
func WithHeaders(headers map[string]string) Option {
    return func(c *HttpClient) {
        for k, v := range headers {
            c.Headers[k] = v
        }
    }
}

// WithRetries allows you to set the number of reconnection attempts
func WithRetries(retries int) Option {
    return func(c *HttpClient) {
        c.Retries = retries
    }
}

Now, when creating a new HTTP client, you can specify only the options you want:

func main() {
    client := NewHttpClient(
        WithTimeout(10*time.Second),
        WithHeaders(map[string]string{"Authorization": "Bearer token"}),
    )

    fmt.Printf("Timeout: %v, Headers: %v, Retries: %d\n", client.Timeout, client.Headers, client.Retries)
}

This allows flexibility and clarity in configuring the client without needing to pass all parameters every time.

Packages that Use this Pattern

Many popular packages in the Go ecosystem use the Functional Options Pattern. Notable examples include:

grpc-go: Go’s package for gRPC uses functional options to configure gRPC servers and clients.
zap: The popular high-performance logger uses this pattern to configure multiple logging options, such as log level, output format, and more.

Conclusion

The Functional Options Pattern is a powerful and idiomatic solution in Go for handling functions or structures that require flexible configuration. By adopting this pattern, you enhance code readability, provide flexibility for end-users, and avoid the proliferation of functions with extensive parameter lists.

Whether creating a custom HTTP client or configuring a complex library, the Functional Options Pattern is a valuable tool for designing APIs in Go.

DEV Community: Tiago Temporin

Mastering memory management in Go: Avoiding slice-related leaks

What is a Memory Leak

Memory Leak in Slices

Essential Rule!

How to Avoid It

1. Copy Only the Needed Data

2. Set Unused Slices to nil

3. Manage Slice Growth in Loops

Conclusion

Unveiling the Garbage Collector in Go

What is a Garbage Collector?

Garbage Collector in Action

Conclusion

Mastering ENUMs in Go

Defining a New Type

Creating the ENUM

Printing the ENUM

Conclusion

The difference between pointers and values on methods

Difference between value and pointer

Heap

When to use each approach

By value

By pointer

Conclusion

Functional Options Pattern in Go

What is the Functional Options Pattern?

Applying it in Practice

Creating an HTTP Client with Optional Settings

Implementing the Functional Options Pattern

Packages that Use this Pattern

Conclusion

2. Set Unused Slices to `nil`