Gotcha while appending to slices in Golang

#go #gotchas

When you pass a slice to a function and try to change the element of slice, it works. But when you try to add a new element to slice inside a function, added elements are not visible to caller function.

package main

import (
    "fmt"
)

func main() {
    var slice = make([]int, 3, 4)
    fmt.Println(slice) //[0,0,0]

    slice[0] = 1
    fmt.Println(slice) //[1 0 0]

    slice = append(slice, 4) // works, new element gets added to the slice
    fmt.Println(slice)       // [1 0 0 4]

    modifySlice(slice)
    fmt.Println(slice) //[11 0 0 4]
}

func modifySlice(s []int) {
    s[0] = 11

    s = append(s, 2) // new element gets added to the slice inside function, but not visible to caller function.
    s[2] = 3         // changes after append will not be visible to caller

    fmt.Println(s) //[11 0 3 4 2]
}

Output
[0 0 0] [1 0 0] [1 0 0 4] [11 0 3 4 2] [11 0 3 4]

Go Playground Link.

To understand above issue, lets see internal representation of slice in detail.

Internally a slice is represented by three things.

- Pointer to the underlying array
- Length of underlying array
- Total Capacity which is the maximum capacity to which the underlying array can expand.

type SliceHeader struct {
        Pointer uintptr
        Len  int
        Cap  int
}

It’s important to understand that even if slice contains a pointer to underlying array, it is itself a value. So When we are passing slice to modifySlice, we are passing a struct value which is holding a pointer to underlying array, length and capacity of array. A point to note here is slice is not a pointer to a struct.

Even though the slice header is passed by value, the header includes a pointer to elements of an array, so when we modified 0th index of slice s[0] = 11 in our modifySlice function, the underlying array element got changed. Therefore, when the function returns, the modified index was visible to the caller function.

Why newly added element inside modifySlice and changes made after that are not visible to caller function?

Whenever any append operations is made on the slice and capacity is not available, a new array is created with double capacity and the pointer to underlying array is overwritten. Now, the elements present in the original array are copied into a new array and returns a new slice pointing to the new array.
When a slice grows via append, it takes time for the Go runtime to allocate new memory and copy the existing data from the old memory to the new. The old memory also needs to be garbage collected. For this reason, the Go runtime usually increases a slice by more than one each time it runs out of capacity. The rules as of Go 1.14 are to double the size of the slice when the capacity is less than 1,024 and then grow by at least 25% afterward.
So when the our slice got reallocated, a new location of the memory is used. Even if the values in slice are the same, the slice points to a new memory location and therefore changes post slice re-allocation are not visible to the caller function.

Solution:
Return the value after appending to the slice and then assign it to the original slice.

package main

import (
    "fmt"
)

func main() {
    var slice = make([]int, 3, 4)
    fmt.Println(slice) //[0,0,0]

    slice[0] = 1
    fmt.Println(slice) //[1 0 0]

    slice = append(slice, 4)
    fmt.Println(slice) // [1 0 0 4]

    slice = modifySlice(slice)
    fmt.Println(slice) //[11 0 3 4 2]
}

func modifySlice(s []int) []int {
    s[0] = 11

    s = append(s, 2)
    s[2] = 3

    fmt.Println(s) //[11 0 3 4 2]

    return s
}

Go Playground link.

One thing to be considered is even if you are returning the updated slice and assigning to the same value, its original length and capacity will change, which will lead to a new underlying array of different length. Check the length and capacity before and after changing the slice to see the difference.

fmt.Println(len(slice), cap(slice))