Enabling Memory Profiling on KEDA v2

I have a minor memory leak issue to solve. So that I'm wondering how can we enable profiler on KEDA. KEDA uses Operator SDK.

Go lang has a good profiler.

• Getting started with Go CPU and memory profiling

• KEDA

It is very simple to use it. Just add this line for memory profiling.

Configuration and Issue

According to the documentation, the configuration will be like this.

go get github.com/pkg/profile

then add following line to the main.go

import (
    //...
    "github.com/pkg/profile"
)
   :
defer profile.Start(profile.MemProfile).Stop()

Run the KEDA on my local machine. However, it doesn't work. The root cause is, we need to stop the operator sdk with ctr+c. In this case, defer method doesn't execute Stop() method.

ushio@DESKTOP-KIUTRHV:~/Code/keda$ make run ARGS="--zap-log-level=debug"
/home/ushio/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go run \
-ldflags "-X=github.com/kedacore/keda/version.GitCommit=d0788b943ecf94f7f7c5b1a09f777cc3af6b746e -X=github.com/kedacore/keda/version.Version=v2" \
./main.go --zap-log-level=debug
2020/10/20 09:19:49 profile: memory profiling enabled (rate 4096), /tmp/profile174831365/mem.pprof
                   :
^C2020/10/20 09:20:02 profile: caught interrupt, stopping profiles
Exit 0 for profiling
make: *** [Makefile:96: run] Interrupt

$ ls -l /tmp/profile287657069/mem.pprof
-rw-r--r-- 1 ushio ushio 0 Oct 20 09:18 /tmp/profile287657069/mem.pprof

Solution

If you want to handle the SIG interrupt, you can use signal.Notify method for handling the signal.

func SetupCloseHandler() {
    c := make(chan os.Signal)
    signal.Notify(c, os.Interrupt, syscall.SIGTERM)
    go func() {
        <-c
        fmt.Println("\r- Ctrl+C pressed in Terminal")
        DeleteFiles()
        os.Exit(0)
    }()
}

I read the main.go they are also having the signal handling part. Let's change it to call profile.Stop() method.

func main() {
    // Memory profiling
    f := profile.Start(profile.MemProfile, profile.NoShutdownHook)
    :
    //  if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
    //      setupLog.Error(err, "problem running manager")
    //      os.Exit(1) // Profiling test
    //  }
    fmt.Println("-----SetupCloseHandler")
    setupSignalHandler := func() (stopCh <-chan struct{}) {
        stop := make(chan struct{})
        c := make(chan os.Signal, 2)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        go func() {
            <-c
            fmt.Println("Exit 0 for profiling")
            f.Stop()
            os.Exit(1) // second signal. Exit directly.
        }()

        return stop
    }

    if err := mgr.Start(setupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

Run it. Works!

$ ls -l /tmp/profile388737932/mem.pprof
-rw-r--r-- 1 ushio ushio 47888 Oct 20 09:31 /tmp/profile388737932/mem.pprof

Viewer

The next step will be the view the analysis report.
It looks requires graphviz. Install it.

$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf
failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH
$ sudo apt install graphviz

Works!

$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf

I don't know if there is better way to enable profiling using Operator SDK. Once I know it, I'll update this blog.

Reference

• Handle Ctrl+C (Signal Interrupt) Close in the Terminal