kustomize: alternative to Helm for lightweight deployments with secrets

My journey started with the simple wish to move some of my applications from a docker environment to a kubernetes cluster I configured for some of my microservices.

Table Of Contents

• Why I looked for an alternative to Helm
• kustomize as an alternative
• The proposed deployment
• Using system environment variables

Target applications for my thoughts were small applications, especially Discord bots. Those applications also need a couple of secrets like tokens and other credentials which might change depending on the deployment.

Why I looked for an alternative to Helm

Since I was already used to utilising Helm for quick and easy deployments, it seemed reasonable to also use it for a microservice like a Discord bot. For those who don't know Helm, it's a fully grown package manager for kubernetes with dozens of articles about it.

Being a - as mentioned "fully grown package manager" - also comes with the downside of shipping a lot of unnecessary bloat for very simple applications. To bring this into perspective, the helm create command allows us to generate a basic template - or - Helm chart. This very basic bootstrapped chart already comes with 10 files, spread over 3 directories (as shown below).

|-- Chart.yaml
|-- charts
|-- templates
|   |-- NOTES.txt
|   |-- _helpers.tpl
|   |-- deployment.yaml
|   |-- hpa.yaml
|   |-- ingress.yaml
|   |-- service.yaml
|   |-- serviceaccount.yaml
|   `-- tests
|       `-- test-connection.yaml
`-- values.yaml

I might be a bit of a purist, but 10 files for the deployment of an application that doesn't even need a service sounds like a lot of bloat. Sure you can start cleaning the chart of unnecessary templates and keep the Helm benefits like versioning. This tho takes a considerable amount of time and doesn't really remove the clutter.

kustomize as an alternative

kustomize is a built-in configuration management system that allows to patch deployments if needed. It also ships a handy tool called secretGenerator that allows to dynamically generate secrets from a given source, something I'll heavily rely on in this example.

One sad thing about kustomize that I realised quite early: the documentation is still very poor and it lacks a lot of functionality that is already implemented.

The proposed deployment

It's recommended to structure a kustomize deployment into two folders base for the default deployment and overlays for the patching deployments.

The simplest deployment that infuses our secrets can look like the following:

base
├── deployment.yaml
├── kustomization.yaml
└── secrets.env

The deployment.yaml contains the kubernetes manifest that we'll use for the deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: ghcr.io/staubrein/myapp
        imagePullPolicy: Always
        env:
        - name: TOKEN
          valueFrom:
            secretKeyRef:
              name: my-app-secrets
              key: TOKEN

This manifest expects a secret my-app-secret with a key TOKEN and adds it to the container environment.

The content of this secret can be read from a casual .env file or in our case secrets.env.

TOKEN=XXXXXXXXXYXYYXYXYXYXYXYXYXYXXYXXXXXXXXYYY

To generate the secret from the file (and also system environment variables!) we first need to write the kustomize.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

secretGenerator:
- name: my-app-secrets
  env: secrets.env

This kustomize file takes the deployment.yaml as resource and combines it with generated secret from the secretGenerator.
It takes the values from the given secrets.env file into context and creates a secret with the given name my-app-secrets. It's important to import the file via the env key and not the file key, since file will turn the whole file into a secret while env makes all containing keys of the secrets.env accessible.

We can take a look at the finished manifests with kubectl kustomize base.

apiVersion: v1
data:
  TOKEN: |
    T888as8d8dja8sdjjajd8ahjjdknkasndanddnasd8a8sda8sd
kind: Secret
metadata:
  name: my-app-secrets-fmk84tgkh5
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - env:
        - name: TOKEN
          valueFrom:
            secretKeyRef:
              key: TOKEN
              name: my-app-secrets-fmk84tgkh5
        image: ghcr.io/staubrein/myapp
        imagePullPolicy: Always
        name: my-app

The secretGenerator of kustomize dynamically generated a secret from the secrets.env file with base64 encoded values and those are accessible within the container.

Using system environment variables

The best part on this solution, what makes it also very interesting for usage in pipelines or environments that don't use variables from files, is that it also works with system environment variables.

By not assigning a value to the key in a environment file, it will automatically try to read out the value for this key from the system environment. Let's demonstrate this in a simple example and add another key to the secrets.env.

TOKEN=XXXXXXXXXYXYYXYXYXYXYXYXYXYXXYXXXXXXXXYYY
FOO

It is important, that there is no equal sign after the key, otherwise the value will be considered empty and not overwritten with a system environment variable.
Let's fill this variable locally with export FOO=bar and generate the secret with kustomize by running kubectl kustomize base.

apiVersion: v1
data:
  FOO: YmFy
  TOKEN: WFhYWFhYWFhYWVhZWVhZWFlYWVhZWFlYWVhZWFhZWFhYWFhYWFhZWVk=
kind: Secret
metadata:
  name: dailyottr-secrets-h7b26g7c74
  namespace: my-app
type: Opaque

As seen in the output above, it set the variable FOO with the value we set in the system environment variable :)

The deployment can be triggered via kubectl kustomize base | kubectl apply -f -.

To fully show the potential of kustomize, especially the functionality of merging and overwriting different environments, I'll write another article soon that patches the base deployment.