Managing app secrets in Kubernetes at Firmhouse

At Firmhouse we're gradually migrating from a Dokku-based infrastructure onto a new Kubernetes-based infrastructure setup running on DigitalOcean Kubernetes. Out with manual work. In with automation!

In the new setup we heavily rely on Terraform for Infrastructure as Code collaboration and automation. One thing that we're adding to our Terraform repository is automated management of secrets and ENV vars for our app deployments.

In this article I'll show you how we're managing secrets and ENV vars via Terraform, Azure Key Vault, and Kubernetes Secrets.

App Deployment in Kubernetes

First of all, let's show you the thing that this is all about in the end: the application deployment. Our Ruby on Rails applications are deployed using a Kubernetes Deployment. Here is a snippet of one of our applications from our Terraform repository:

As you can see, we load the application's environment from a single Kubernetes Secret named dispatch-env.

We also put all Kubernetes objects for a given application in a specific Kubernetes Namespace. Following this practice makes sure that by default we don't let secret stuff from one application deployment leak over to a different application.

ENV vars via a single Kubernetes Secret

Simply loading the application's environment from a single Kubernetes secret makes it easy to manage the whole runtime environment via a single Terraform resource. Here is a redacted and shortened example of one of our apps:

Secret storage: Azure Key Vault

In our new infrastructure setup we store our secrets in an Azure Key Vault. Azure Key Vault is comparable to AWS Key Management Service or HashiCorp Vault.

We used to not store secrets at all or have only certain manually generated secrets stored in our 1Password business account.

Because Terraform can talk to Azure Key Vault via it's Azure provider, we can now start managing secrets without ever touching them or making then visible to a human eye. This allows us to read a secret key from Azure Key Vault in our Terraform repository:

Setting up the application ENV variables with secrets from Azure Key Vault

So uptil this point we have three things prepared.

1. Applications are deployed using Kubernetes Deployments in their own namespace.
2. An application uses one single Kubernetes Secret to load the environment.
3. Azure Key Vault is set up and we can load secrets via Terraform resources.

Now let's bring it all together.

In the following redacted sample you'll see how we're pulling in SMTP login credentials and a database password to load those into the application deployment's ENV:

Conclusion

Why this is great

Such a setup is great for the following reasons:

• Anyone in the team can take a look at our Infrastructure as Code git repository to see what ENV vars are configured for a given application deployment.
• Secrets are safely stored in Azure Key Vault, including versioning, timestamps, access logs, and access policies for team members and applications.
• We don't have to manually log in to Dokku servers or into Heroku anymore to define the ENV variables for our apps.

What can be improved

One thing that still bugs me is that the actual secret values are stored in a Kubernetes Secret. In practice, this is not worse than storing them in Heroku or Dokku. But it would even be greater to have a setup where the secret value is not visible to humans with access to the Kubernetes cluster.

Jon Arild Tørresdal has a solution in place at Sparebanken Vest. He has written great blog post about this setup. Check it out here: Introducing Azure Key Vault to Kubernetes