Level up your secrets management in Kubernetes using AWS Secret Manager and Helm

Engineers' biggest struggle when writing Kubernetes resources is to keep all secrets secure. To be honest, the secrets k8s resource is not secured at all because base64 is not encryption!

Required skills:

• Understanding basic concepts about Kubernetes
• Cloud providers resources knowledge
• Helm v3 installed on a Kubernetes cluster.

Understanding Helm

For this article and for Formance infrastructure we chose to use Helm as a templating system for our k8s resources. This software is commonly known as the package manager for Kubernetes but they also provide a good system for deploying many versions of the same Charts, like for production and staging environments.

Initialize the application

First, we have to create a new application using the Helm CLI. You could find how to install it here: https://helm.sh/docs/intro/quickstart/. Once all is installed correctly, you could run the command:

helm create "<your_application_name>"

The directory Helm creates should look like this ⬇️

In this article, we’re going to stay basic and remove some templates to keep it simple. We’re only keeping:

• The deployment template
• The service template
• The service account template
• The helpers, Notes, and Tests

Finally, we will create a file named secret.yaml for the template of our SecretProvider object.

If you’re not familiar with the Helm templating language, I recommend you take a look at the documentation: https://helm.sh/docs/chart_template_guide/

The secret provider object

Concept

What are CSI Drivers?

To perform what we want, we’re going to use a Kubernetes resource named SecretProviderClass from the API secrets-store.csi.x-k8s.io/v1alpha1. The secret store is provided by the Kubernetes Container Storage Interface which here helps us to connect basic resources like volumes or secrets to cloud providers' services.

The main problem of Secrets resources on Kubernetes is the lack of security and if I want to share some private values as database passwords to my application deployment, I need more than just a base64 encryption.

Here at Formance we use Amazon Web Services as cloud provider so I will use the service AWS Secret Manager (docs: https://docs.aws.amazon.com/secretsmanager/?id=docs_gateway). When we have created our Relational Database Service with MariaDB or PostgreSQL or whatever, using InfraAsCode tools Terraform, we have automatically created a Secret with the database login information. The problem is to retrieve those values directly on my application container automatically.

Simplified overview of the workflow

Installing the Secret Store CSI driver

That’s here where Kubernetes CSI enters in action ! It will create an interface between our Secret object on AWS and our application deployment. First of all, we have to install the Secret Store CSI driver using helm.

helm repo add secrets-store-CSI-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

You could find an alternative installation here: https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html

It will install things like the operator, and service account with the right RBAC permissions and most important, it will declare the Custom Resources Definition of the SecretProviderClass. Without this, the Kubernetes Scheduler will not understand the API version and the object kind.

Secret store CSI driver also supports GCP, Azure, and Vault secrets providers.

Deep dive into the secret template

The requirement is to make a template where we could inject some specific values like:

1. The name obviously
2. The secret provider (note that this example is only tested on AWS)
3. And the secrets to retrieve using the secret name or in AWS its Amazon Resource Name (ARN)

Keep in mind that this example is a simple template with multiple secrets but you personalize it as you wish. You can find various

Let’s take a look at the template and its values:

Once you create all templates, all you have to do is to apply the command:

helm install <application name> .

And you're all done!

Image credits: Midjourney