Golang Garbage Collector 101

Quick Look at Garbage Collector

A simple program that creates a byte array of 100000000 elements and lets print the memory allocations.
package main

package main 

import (
    "fmt"
    "runtime"
    "time"
)

func printStats(mem runtime.MemStats) {
    runtime.ReadMemStats(&mem)
    fmt.Println("mem.Alloc:", mem.Alloc)
    fmt.Println("mem.TotalAlloc:", mem.TotalAlloc)
    fmt.Println("mem.HeapAlloc:", mem.HeapAlloc)
    fmt.Println("mem.NumGC:", mem.NumGC)
    fmt.Println("-----")
}

func main() {
    var mem runtime.MemStats
    for i := 0; i < 2; i++ {
        s := make([]byte, 100000000)
        if s == nil {
            fmt.Println("Operation failed!")
        }
        printStats(mem)
    }
    time.Sleep(time.Second)
    // adding time.Sleep so that GC finishes it works and print out the output to terminal
}

Understanding the code

There is a package in Golang standard library called runtimewhich contains alot of useful function like :

• runtime.ReadMemStats(&mem)
• runtime.GC()

Take a loot at : https://golang.org/pkg/runtime/

1. func ReadMemStats

func ReadMemStats(m *MemStats)

ReadMemStats populates m with memory allocator statistics.
The returned memory allocator statistics are up to date as of the call to ReadMemStats. This is in contrast with a heap profile, which is a snapshot as of the most recently completed garbage collection cycle.

1. func GC

func GC()

GC runs a garbage collection and blocks the caller until the garbage collection is complete. It may also block the entire program.

*Run this program using the following flag : *

$ GODEBUG=gctrace=1 go run gc.go 

Output i receive [ don't get scared, will clean up ] :

// some other output //

gc 3 @0.029s 5%: 0.020+2.8+0.009 ms clock, 0.081+0.12/2.7/0.19+0.036 ms cpu, 13->15->10 MB, 15 MB goal, 4 P
mem.Alloc: 100092312
mem.TotalAlloc: 100092312
mem.HeapAlloc: 100092312
mem.NumGC: 0
-----
gc 1 @0.001s 3%: 0.035+0.20+0.002 ms clock, 0.14+0.041/0.037/0.10+0.010 ms cpu, 95->95->0 MB, 96 MB goal, 4 P
mem.Alloc: 100081456
mem.TotalAlloc: 100101064
mem.HeapAlloc: 100081456
mem.NumGC: 1
-----
gc 2 @0.021s 0%: 0.014+0.11+0.003 ms clock, 0.058+0.068/0.050/0.049+0.012 ms cpu, 95->95->0 MB, 96 MB goal, 4 P

Let look at the second run :

-----
gc 1 @0.001s 3%: 0.035+0.20+0.002 ms clock, 0.14+0.041/0.037/0.10+0.010 ms cpu, 95->95->0 MB, 96 MB goal, 4 P
mem.Alloc: 100081456
mem.TotalAlloc: 100101064
mem.HeapAlloc: 100081456
mem.NumGC: 1
-----
gc 2 @0.021s 0%: 0.014+0.11+0.003 ms clock, 0.058+0.068/0.050/0.049+0.012 ms cpu, 95->95->0 MB, 96 MB goal, 4 P

• In first line : ignoring the value from CPU profiler, take a look at "95->95>0 MB". The first number is the heap size when the garbage collector is about to run.
• The second value is the heap size when the garbage collector ends its operation. The last value is the size of the live heap that is 0
• It allocated 100081456 bytes of memory, more than the 100000000 because some extra memory required for some internal stuff, you can verify this by reducing 100000000 to 3000000o or even 30000, on my machine it always took somewhere around 70k to 90k bytes, your number may vary.
• Thus before running through the second iteration of the loop, it clears out the memory allocated during the first.

Also you can read the comments below for some interesting insights.

Comments

[David Kröll]

Hi, nice insights you gave about the Go runtime and GC internals - never seen GODEBUG=gctrace=1, tho!

But doesn't Go always allocate the same len and cap when using the make statement? I think it would be 100000000 for the capacity too, until you append to the slice, then it would double the capacity.

Some additional further improvement would be code highlighting. This can be enabled by specifying the programming language after the three backticks at the code block. Just put"go" after them and it will work.

[Uday Yadav]

Thank you for the feedback

i looked into the Go's length and capacity concept. Yes when you use make to allocate, and print the len and cap of slice, they are same but still compiler allocates some more memory on heap. I dont exactly remember where i learned about this, but there is something called backing array in go which is basically some extra memory allocated for slice to expand. Running GODEBUG=gctrace=1 always shows more memory being allocated than specified

In the article saying the length and capacity is not same is definitely wrong (i will edit this soon). I will to look into this more and try to provide a better explaination

Also in my markdown editor, i did specify and language for code block but somehow didn't get copied from there, will edit that too

Thanks again

[David Kröll]

Hi again,

I've done some additional research about the concepts and paradigm of the slice data type. Basically there are the three fields len, cap and data (the backing array you mentioned). As you already pointed out, there is more memory allocated than the cap and len (which are the same when using make) state.

These two values len and cap have to be stored also, in order to not access memory outside the bounds of the backing array. There is probably some memory usage hidden in these fields (as far as my understanding goes, this will only take two words of memory).

So where does the additional memory come from?
I believe this is to some extend based on the Go runtime. There is some space needed for the stack, Goroutine handling, the GC bookkeeping, and possibly many more.

[Uday Yadav]

I think its really good explaination of what's happening underneath. When i ran the program with s := make([]byte, 5000) it still show mem.TotalAlloc in between 80000 - 120000

finding resoure & explaination about this is bit hard, thanks you giving your time and sharing knowledge

i will update the article soon ( i have my uni exams coming up )