Benchmark Testing Kubernetes Controller Runtime With Envtest

The envtest docs push you into Ginko, but I'm a huge proponent of idiomatic (add another t to this word and I still fit) Go and Ginko doesn't fit me.

Thankfully you can use standard Go test benchmark functions without any extra testing packages and profile code to improve controllers.

Here is how I did it...

package something
import (
    "context"
    "path/filepath"
    "testing"

    "k8s.io/apimachinery/pkg/types"
    "k8s.io/client-go/kubernetes/scheme"
    controllerruntime "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"

    "something/mypkg"
)
func BenchmarkMy_Reconcile(b *testing.B) {
    testEnv := &envtest.Environment{
        CRDDirectoryPaths: []string{"path/to/crds"},
    }
    cfg, err := testEnv.Start()
    if err != nil {
        b.Fatal(err)
    }
    defer func() {
        err := testEnv.Stop()
        if err != nil {
            b.Error(err)
        }
    }()
    err = mypkg.AddToScheme(scheme.Scheme)
    if err != nil {
        b.Fatal(err)
    }
    k8sClient, err := client.New(cfg, client.Options{Scheme: scheme.Scheme})
    if err != nil {
        b.Fatal(err)
    }
    r := &MyController{
        Client:      k8sClient,
        Log:         controllerruntime.Log.WithName("controllers").WithName("mypkg"),
        Scheme:      scheme.Scheme,
        InitMetrics: map[string]string{},
    }
    rreq := reconcile.Request{NamespacedName: types.NamespacedName{
        Namespace: "test", Name: "test"}}
    for i := 0; i < b.N; i++ {
        got, err := r.Reconcile(context.Background(), rreq)
        if err != nil {
            b.Fatalf("MyController.Reconcile() error = %v", err)
            return
        }
        b.Logf("%#v", got)
    }
}

On my mac, I use brew to install etcd and kubectl. I clone kubernetes and build kube-apiserver myself and copy it to /usr/local/bin

brew install kubectl etcd
git clone [e-mail]:kubernetes/kubernetes.git && \
pushd kubernetes && \
git checkout v1.26.1 && \
go install ./cmd/kube-apiserver && \
cp ~/go/bin/kube-apiserver /usr/local/bin && popd

Then I run this test with KUBEBUILDER_ASSETS=/usr/local/bin go test -benchmem -run=^$ -bench ^BenchmarkMy_Reconcile$ -memprofile mem.prof ./something

The output is the standard Go test benchmark output and the mem.prof which I examine using go tool pprof mem.prof.

I haven't been programming much lately and so I didn't have dot installed on my Mac and I had to lookup where to get it. I had to brew install graphviz.

I use top10 to see the highest memory users and I use web to generate an SVG of the memory creation call graph.

In my case, the code base was over using fmt.Sprint(f) and I was able to optimize these string allocations. I consider this a win.