DEV Community: Kevin Davin

Taming FluxCD HelmReleases: The Kustomize Way approach

Kevin Davin — Tue, 04 Jun 2024 16:00:00 +0000

FluxCD is a powerful tool for managing deployments in Kubernetes using GitOps principles. While it offers a wide range of features, this post will explore scenarios where a simpler approach might be preferable, aligning with the #SimplerIsBetter philosophy.

NOTE: This article shares insights and perspectives gained through experience in various contexts and companies. Feel free to disagree, but with respect! 😉

`HelmRelease`, what is this?

HelmRelease is a custom resource provided by FluxCD, which provides users
a way to automatically install helm charts using FluxCD and its declarative system.

For example, if you want to install the PodInfo app, you have to declare
the following manifest:



apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
spec:
  chart:
    spec:
      chart: podinfo
      version: '6.5.*'
      sourceRef:
        kind: HelmRepository
        name: podinfo
      interval: 5m
  releaseName: podinfo
  values: # part dedicated to the all `values` the chart accept
    replicaCount: 2

NOTE: For the sake of simplicity, I kept only relevant attributes, but I can say FluxCD offers a rich API to cover most of use cases.

The simplified workflow can be described like this:

The HelmOperator verifies and retrieves the Helm chart from a remote source like GitHub, GitLab, Artifactory or an OCI registry (recommended)
It then renders the chart using configuration from the HelmRelease object, which can include values provided in different ways (inlined, using ConfigMap or Secret)
Finally, the HelmOperator applies the generated YAML manifests to the Kubernetes API to install or upgrade the application.

This approach can work for simple deployments, but as an operator, I've encountered several design drawbacks that I'd like to discuss. Let's see these issues!

NOTE Like every GitOps solution, FluxCD requires a connection to a GitRepository too. To keep the diagram simple, I've not materialized this item.

GitOps, aka "only source of truth"

The GitOps philosophy is built on top of 3 key principles:

Declarative System: You define the desired state of your system (what you want) using declarative language (4GL). This approach focuses on the "what" instead of the "how," making your configuration easier to understand and maintain.
System State Captured in a Git Repository: The desired state of your system is stored in a Git repository. This provides a central location for managing your infrastructure configurations, enabling version control, collaboration, and easy rollbacks if needed.
Automatic Deployment system: Any changes pushed to the Git repository trigger an automated deployment process. This automates the process of translating your desired state into actual changes within your system, reducing manual intervention and the risk of errors.

The HelmRelease is a Declarative approach, where automation is managed by the controller. However, "desired state of your system is stored in a Git repository" and all it implies are not respected.

External Chart Dependencies: A Potential Weak Point

A core principle of building resilient systems is ensuring the availability of all their components. When using HelmRelease, we introduce an external dependency, the location where the Helm chart resides.

If communication with this external location fails (due to server downtime, network issues, etc.), you might not be able to install or update your application. This introduces a potential single point of failure (SPOF) in your deployment process. Additionally, the desired state of your cluster is not solely defined by your git repository; it also depends on all the charts your system downloads.

Limited Visibility into Helm Chart Content

Another purpose of this System State Captured in a Git Repository is its auditability. If you capture the complete state of a system in git, you can review it before an installation or upgrade. However, using HelmRelease introduces a layer of opacity.

While you declare the specific chart you want to use, you might not have a complete picture of what resources the chart will actually install in your cluster. It could potentially create various resources like ClusterRoles, NetworkPolicies, or DaemonSets. You can't say without… deploying it, running it locally or worth, reading the chart's source. 😞

// Detect dark theme var iframe = document.getElementById('tweet-1648619974934642689-925'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1648619974934642689&theme=dark" }

Lack of Immutability in Helm Charts!

The problem we already have with container images applied the same way to helm charts. A chart produced and published at a specific date might be un-published or re-published with a different content. In those case, you expose your system to the two previous points again…

NOTE We can now store charts in OCI registries, with immutability feature. To leverage it, you have to complexify your system with digest pinning, because in security, we can't blindly trust a 3rd party 😇.

Chart customisation, another nightmare 👻

While Helm charts offer a convenient way to package deployments, maintaining them can be challenging due to their complexity. Kubernetes itself provides a wide range of configuration options, which can further complicate matters.

This complexity can lead to situations where the desired configuration isn't readily available within a chart. For example, you might want to add an annotation to a workload or modify taints and tolerations, but the chart may not offer built-in ways to do so.

// Detect dark theme var iframe = document.getElementById('tweet-1622925402845970433-191'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1622925402845970433&theme=dark" }

To address this challenge, FluxCD introduced the concept of postRenderers (see documentation within HelmRelease resources. This feature leverages the Kustomize API to customize deployments after the initial Helm chart rendering.



apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
spec:
  releaseName: podinfo
  chart: { … }
  values: { … }
  postRenderers:
    - kustomize:
        patches:
          - target:
              version: v1
              kind: Deployment
              name: metrics-server
            patch: |
              - op: add
                path: /metadata/labels/environment
                value: production              
        images:
          - name: docker.io/bitnami/metrics-server
            newName: docker.io/bitnami/metrics-server
            newTag: 0.4.1-debian-10-r54

The configuration for postRenderers is separate from the Helm chart result. This can make it difficult to understand the complete picture of how the final deployment will be configured, potentially leading to hidden errors. 🤯

FluxCD doesn't provide robust mechanisms to analyze the resources created after the combined rendering and post-rendering steps. This can make troubleshooting issues arising from these modifications cumbersome. 😞

Solution is simplicity 🚀!

We've discussed the challenges associated with relying solely on Helm charts within FluxCD deployments. These challenges can compromise the visibility, maintainability, and overall health of your GitOps workflow.

So, how can we achieve the ideal balance: a declarative system, automatic deployments, and a clear picture of your system state captured entirely within your Git repository?

Render the chart and store it into `git` for better auditability

The answer lies in a simple yet powerful sub-command – helm template (or helmfile template if you use Helmfile). This command allows you to locally render helm charts along with your desired values, generating the final deployment manifest files.



$ helm repo add podinfo https://stefanprodan.github.io/podinfo
"podinfo" has been added to your repositories
$ helm repo update podinfo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "podinfo" chart repository
Update Complete. ⎈Happy Helming!⎈
$ echo "replicaCount: 2" > values.yaml
$ helm template podinfo/podinfo -f values.yaml --version 6.6.3 > podinfo.yaml

And that's it! It was not so complicated 😇. We have a file called podinfo.yaml, located in /k8s/podinfo of our git repository.

This file is now yours, it can be read, analyzed and pushed to kubernetes. After this generation, your system is free from the external chart registry where the chart is located.

NOTE I recommend to never modify a file "generated" by another tool, because you will loose this modification during the next rendering. tldr; treat them as "read-only".

How to deploy generated files with FluxCD?

Instead of using HelmRelease, we're going to use Kustomization resources provided by FluxCD. It is way simpler than HelmRelease, because it just deploys manifests located in a specific location.



apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
spec:
  sourceRef:
    kind: GitRepository
    name: our-gitops-repository
  path: "/k8s/podinfo" # our location in our GitOps repo
  prune: true
  timeout: 1m

Because we use the GitRepository, called our-gitops-repository and eventually used by FluxCD itself, there is no extra dependency in our system.

Direct Customization with Kustomize

While HelmRelease offers postRenderers for some customizations, the Kustomization resource provides full access to the powerful Kustomize capabilities.
This allows for more granular and flexible control over your manifests. Here's how to achieve a similar customization as the previous postRenderer example using a kustomization.yaml file:



# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - podinfo.yaml # your generated file from the previous step

patches:
- target:
    version: v1
    kind: Deployment
    name: metrics-server
  patch: |
    - op: add
      path: /metadata/labels/environment
      value: production              

images:
- name: docker.io/bitnami/metrics-server
  newName: docker.io/bitnami/metrics-server
  newTag: 0.4.1-debian-10-r54

kustomize offers a wider range of functionalities compared to postRenderers. You can manipulate resources using features like namePrefix, labels, replacements, components… The list is too long to be detailed here 😇.

As a bonus point, you can run kustomize build /k8s/podinfo/ and see the complete result of the generation before any interaction with FluxCD.

Enjoy reviews and audit with rich diff!

One of the significant advantages of managing your full Kubernetes state with GitOps is the ability to leverage Git's powerful version control capabilities for reviewing and auditing deployments.

From an operator perspective, there is nothing better than a clear and detailed diff views during tool upgrade:

Obviously, your IDE will be your best friend to understand what happened, with clear context and details of changes:

NOTE Upgrade can be automated using tools like renovate or dependabot

Conclusion

While Helm charts offer a convenient way to package deployments, maintaining them in a FluxCD workflow can introduce challenges related to transparency, maintainability, and control.

This article explored the limitations of HelmReleases and presented helm template … as a more powerful and flexible alternative, leveraging FluxCD Kustomize resource. Using Kustomize directly within your git repository, you gain greater control, visibility, and the benefits of Git version control for reviewing and auditing changes.

Ultimately, by adopting a Kustomize-based approach within your FluxCD workflows, you can achieve a more declarative, transparent, and auditable approach to managing your Kubernetes deployments.

Deploy with Kustomize, FluxCD and Remote Resources

Kevin Davin — Fri, 12 Aug 2022 14:41:32 +0000

Introduction

As a Kubernetes user, I chose to use the default tooling available, named Kustomize, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that, Kustomize is a powerful solution based on concepts of inheritance (resources) and composition (components).

But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where helm is ahead of Kustomize now.

However, Kustomize provides a mechanism to use remote elements, for both resources and components using git under the hood. This solution is perfect, because we can use any git repository as a yaml registry for our reousrces and components 🔥.

But, the downside is FluxCD does not supports so well, for a lot of good reasons (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote resources and components, the FluxCD way!

Cluster & FluxCD configuration

In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!

For this article, I have set up a cluster using Google Kubernetes Engine (aka GKE) using this documentation and installed FluxCD connected to a GitLab repository with the following command (extracted from the official documentation).

$ flux bootstrap gitlab \
  --owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \
  --repository=clusters \
  --branch=gke \
  --path=fluxcd

NOTE: I don't like to use default branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of GitLab Flow for environment.

With this, I have a complete Kubernetes Cluster ready for GitOps, using clusters repository as source-of-truth. Every modification made in this repository will be automatically deployed by FluxCD 🤩.

`bases` repository

I have set up another git repository dedicated to resources and components. It hosts what we usually call base, because it contains the common core of an application and also many components to enable some extra features (at infrastructure or application level).

This repository is available in GitLab and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an helm registry. Because it is managed in git, I can use branch & tags for management 🚀.

NOTE: Usually, this kind of yaml repository is only populated by CI from other projects automatically.

PodInfo base publication

I have chosen the well known podinfo application as an example, and I have developed the base and some components associated to it.

.
├── README.md
└── podinfo
    └── base
        ├── components
        │   ├── hpa
        │   │   ├── hpa.yaml
        │   │   └── kustomization.yaml
        │   ├── ingress
        │   │   ├── ingress.yaml
        │   │   └── kustomization.yaml
        │   └── redis
        │       ├── kustomization.yaml
        │       ├── redis.conf
        │       └── redis.yaml
        ├── kustomization.yaml
        └── podinfo.yaml

Core PodInfo

This part is the common part of all potential deployment. It contains:

The podinfo Kubernetes Service
The podinfo Kubernetes Deployment

And nothing else… if you need to deploy any other part of the app, you have to use components 👇.

Components

In this components, we can find all optional part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.

Here, we have:

hpa with a default preconfigured Horizontal-Pod-Autoscaler if you want the application to scale up and down automatically
ingress definition to expose the application to the outside world (see official documentation)
redis component to enable cache on the PodInfo application with default configuration.

It's interesting to note components can be of any sort, hpa and ingress are focused on infrastructure when redis is a feature of the application. Without this redis component, the application is able to work, just without any caching system. This redis component is here to deploy a redis server (of course 😅) but also to enable caching at podinfo level too.

NOTE: If you want a better description of this component feature, I advise you to discover this article from the official documentation

So, we have a complete, environment agnostic resource and components committed to our base repository. Now, It is time to use it!

Publish PodInfo to `dev`

Back to our cluster, I want to deploy this podinfo application, but of course I want to apply some extra customizations on it, like:

Define components I want to enable
Define a domain name for the ingress definition
Define the required tls configuration to the ingress
Define an alternative registry for the image

So, to do that, I need to have one (or multiple) kustomization.yaml to define this:

# <clusters-repo>/podinfo-dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: podinfo-dev

images:
  - name: podinfo
    newName: ghcr.io/stefanprodan/podinfo

resources:
  - ../base
  - ingress

components:
  - ../base/components/redis

NOTE: The ingress modification is in its own file to keep this one simple enough.

But, what is this ../base folder? There is no base at all in cluster repository. And you are right… we will use FluxCD feature available in flux to include the base repository at the right location during reconciliation. To do that, we need to define multiple flux resources.

`podinfo-base` flux repository

First, we need a representation of our flux.GitRepository for bases, which is materialized in flux with:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-base-dev
  namespace: flux-system
spec:
  interval: 5m
  ref:
    branch: podinfo
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
  secretRef:
    name: flux-system

NOTE: Here, I have chosen to define spec.ref.branch: podinfo to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.

`podinfo` flux repository

Now, we need to create a flux.GitRepository for manifests in clusters repository and manifests from the bases repository created in the step before. To do that, we are going to use the include feature from FluxCD:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-dev
  namespace: flux-system
spec:
  interval: 5m
  include:
    - repository:
        name: podinfo-base-dev
      fromPath: podinfo/base
      toPath: base
  ref:
    branch: gke
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters
  secretRef:
    name: flux-system

The important part is the following:

spec:
  include:
    - repository:
        name: podinfo-base-dev # <1>
      fromPath: podinfo/base # <2>
      toPath: base # <3>

This means the folder podinfo/base (2) previously defined in the podinfo-base-dev flux.GitRepository (1) will be injected in the path base (3) of the current podinfo-dev flux.GitRepository during FluxCD reconciliation… making all resources and components available.

`podinfo-dev` flux Kustomization

This one is the standard flux.Kustomization you would use in any case. This one is just using the composite flux.GitRepository from the previous step as source (spec.sourceRef):

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: podinfo-dev
  namespace: flux-system
spec:
  suspend: false
  interval: 10m0s
  path: podinfo-dev # file path in the <clusters> repository  
  prune: true
  sourceRef:
    kind: GitRepository
    namespace: flux-system
    name: podinfo-dev
  validation: client

If we deploy all those files (you can put them in the same one), you will see your application deployed by FluxCD with your provided custom configuration.

❯ kubectl get all -n podinfo-dev
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-869ff7c78b-jjbbk     1/1     Running   0          103m
pod/podinfo-6dbf56d6d7-ctxxr   1/1     Running   0          103m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.43.115.207   <none>        9898/TCP,9999/TCP   103m
service/redis     ClusterIP   10.43.192.84    <none>        6379/TCP            103m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis     1/1     1            1           103m
deployment.apps/podinfo   1/1     1            1           103m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-869ff7c78b     1         1         1       103m
replicaset.apps/podinfo-6dbf56d6d7   1         1         1       103m

Publish PodInfo to `prod`

Now, we want to use again the same base this time for a production environment.

Because we want something relatively stable, we chose to define spec.ref.tag to a specific tag in the bases repository (instead of the podinfo branch).

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-base-prod
  namespace: flux-system
spec:
  interval: 5m
  ref:
    tag: podinfo-1.0
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
  secretRef:
    name: flux-system

All others flux.Kustomization and flux.GitRepository are the same.

Because we are in production, we also want to enable different modules, here the HorizontalPodAutoscaler. So we have a different kustomization.yaml. Thanks to the full control we have over the kustomization.yaml per environment, we can modify this only in production 😍.

Once everything is published and FluxCD reconciliation is OK, we have:

❯ kubectl get all -n podinfo-prod
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-869ff7c78b-lnll5     1/1     Running   0          113m
pod/podinfo-6dbf56d6d7-k8rjl   1/1     Running   0          113m
pod/podinfo-6dbf56d6d7-qldr8   1/1     Running   0          113m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.43.109.213   <none>        9898/TCP,9999/TCP   113m
service/redis     ClusterIP   10.43.252.54    <none>        6379/TCP            113m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis     1/1     1            1           113m
deployment.apps/podinfo   2/2     2            2           113m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-869ff7c78b     1         1         1       113m
replicaset.apps/podinfo-6dbf56d6d7   2         2         2       113m

NAME                                          REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/podinfo   Deployment/podinfo   4%/99%    2         4         2          113m

Conclusion

It was tough, with many yaml but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (yaml) registry, like helm already have 😇.

There is some follow-up improvements to come, like OCI registries (announced in FluxCD 0.32), to replace flux.GitRepository. We still need this issue to be solved to be able to use it with all features provided by Kustomize.

In a more distant future, some other solution like porch could be a good candidate to simplify again our deployment strategy… but it is currently in alpha stage.

I hope you liked this article, you can find all resources used in GitLab, and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter @davinkevin

Scaling Keycloak on Distroless into Kubernetes

Kevin Davin — Fri, 04 Jun 2021 11:40:28 +0000

In the two previous articles, we discovered how to build and run Keycloak with a Distroless base image in a Kubernetes cluster. The previously seen configuration was Ok for one instance, but the clustering capabilities of Keycloak was not used, which can cause some problems.

Keycloak has a built-in clustering mode, based on Wildfly & Infinispan. To activate it, some start-up scripts are using environment values to set up everything for you… and of course, those scripts are bash based, not compatible with our version of Keycloak. Here, we will see how to configure this and deploy it to Kubernetes

`standalone-ha.xml` extraction

We will use the same strategy seen before to generate the standalone-ha.xml, by running the official image with parameters we want to use and extract the file with docker cp command line. Let's see:

# In the first shell
# Creation of a docker network
first-shell$ docker network create keycloak-network
4da77163731b584bef2c6d0b00386b9d62e31fa216204c6c6795f66e109ba1a6
# Launching PostgreSQL linked to the network previously created
first-shell$ docker run --rm -d --name postgres --net keycloak-network \
-e POSTGRES_DB=keycloak \
-e POSTGRES_USER=keycloak \
-e POSTGRES_PASSWORD=password postgres
229816da42707e772542f1b089c616a2333a6fbe1aea2be7efe658d6f2c934a1
first-shell$ docker run -it --rm --name keycloak \
-e DB_ADDR=postgres \
-e DB_USER=keycloak \
-e DB_PASSWORD=password \
-e KEYCLOAK_USER=foo \
-e KEYCLOAK_PASSWORD=bar \
-e JGROUPS_DISCOVERY_PROTOCOL="dns.DNS_PING" \
-e JGROUPS_TRANSPORT_STACK=tcp \
-e JGROUPS_DISCOVERY_PROPERTIES="dns_query=keycloak-headless" \
--net keycloak-network jboss/keycloak:13.0.1

=========================================================================

  Using PostgreSQL database

=========================================================================

19:15:45,322 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.11.0.Final
19:15:45,389 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.12.Final
19:15:45,399 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
19:15:45,542 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) starting
...
19:16:23,596 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
19:16:23,671 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
19:16:23,679 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) started in 25820ms - Started 692 of 978 services (686 services are lazy, passive or on-demand)
19:16:23,685 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
19:16:23,686 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

You can see we add some extra parameters for the clustering mode, based on JGROUPS. Some details are in the docker official documentation but you will find more in the keycloak server installation documentation.

The simplest solution to set up cluster mode in a Kubernetes environment is to use DNS_PING over TCP. This is why we defined the following environment values in the previous shell example:

JGROUPS_DISCOVERY_PROTOCOL="dns.DNS_PING" to activate DNS_PING.
JGROUPS_TRANSPORT_STACK=tcp to activate clustering over TCP.
JGROUPS_DISCOVERY_PROPERTIES="dns_query=keycloak-headless" to provide a way to find other instance (we will describe it in the next paragraph).

Then, in another shell, we will steal again the standalone-ha.xml.

NOTE: In the previous article, we were targeting the standalone.xml, the HA version contains a more robust configuration for our use case in a cluster mode.

second-shell$ docker cp keycloak:/opt/jboss/keycloak/standalone/configuration/standalone-ha.xml .
second-shell$ ls
standalone-ha.xml
# We can now stop the keycloak container
second-shell$ docker stop keycloak
keycloak
second-shell$

NOTE: If you want to set up other parameters, you can use this method for almost everything 🤩.

If we look into the standalone-ha.xml file, we can see an important configuration for our clustering mode:

<!-- standalone-ha.xml -->
<subsystem xmlns="urn:jboss:domain:jgroups:8.0">
    <channels default="ee">
        <channel name="ee" stack="tcp" cluster="ejb"/>
    </channels>
    <stacks>
        <stack name="tcp">
            <transport type="TCP" socket-binding="jgroups-tcp"/>
            <protocol type="dns.DNS_PING">
                <property name="dns_query">keycloak-headless</property>
            </protocol>
            <protocol type="MERGE3"/>
            <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
            <protocol type="FD_ALL"/>
            <protocol type="VERIFY_SUSPECT"/>
            <protocol type="pbcast.NAKACK2"/>
            <protocol type="UNICAST3"/>
            <protocol type="pbcast.STABLE"/>
            <protocol type="pbcast.GMS"/>
            <protocol type="MFC"/>
            <protocol type="FRAG3"/>
        </stack>
    </stacks>
</subsystem>

This file will configure Keycloak to find other instances through the DNS_PING protocol. In fact, Keycloak will forge a DNS request to find IPs behind the domain name keycloak-headless… easy as pie!

Kubernetes deployment

Keycloak is ready for clustering mode, but we have to adapt our deployment to allow this specific configuration where each instance can communicate to each other.

The first modification is at deployment level, to expose some extra ports dedicated to instance-to-instance communication:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          ports:
            # Standard HTTP port used by keycloak
            - containerPort: 8080
              protocol: TCP
            # Port used by Jgroups to communicate
            - containerPort: 7600
              protocol: TCP

To work well, Jgroups has to be bound to the Pod IP. In Kubernetes world, we usually don't know the Pod IP in advance, so we will have to inject the Pod IP in the deployment and use it in the args part, like below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          args:
            - "-D[Standalone]"
            - "-server"
            - "-Xms64m"
            - "-Xmx512m"
            - "-XX:MetaspaceSize=96M"
            - "-XX:MaxMetaspaceSize=256m"
            - "-Djava.net.preferIPv4Stack=true"
            - "-Djboss.modules.system.pkgs=org.jboss.byteman"
            - "-Djava.awt.headless=true"
            - "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED"
            - "-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log"
            - "-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties"
            - "-jar"
            - "/opt/jboss/keycloak/jboss-modules.jar"
            - "-mp"
            - "/opt/jboss/keycloak/modules"
            - "org.jboss.as.standalone"
            - "-Djboss.home.dir=/opt/jboss/keycloak"
            - "-Djboss.server.base.dir=/opt/jboss/keycloak/standalone"
            # Note we have changed the command here to use the standalone-ha.xml file
            - "-c=standalone-ha.xml"
            - "-b=0.0.0.0"
            - "-bprivate=0.0.0.0"
            - "-bmanagement=0.0.0.0"
           # Thanks to the Kubernetes interpolation, we are able to launch the app
           # with a custom parameter for each pods. 
            - '-Djgroups.bind_addr=$(HOST_IP)'
          env:
            # the HOST_IP environment value is populated by Kubernetes with 
            # the current Pod IP coming from `status.podIP`.
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP

With those modifications, Keycloak will be able to work in cluster mode… but it won't be able to find any other instances 😔. We have to add a way to discover other instances 💇‍♀️!

Headless Service to the rescue!

In Kubernetes, usually we are using Service to expose one domain name with multiple instances of an application behind it. In our case, we want to be able to fetch every IPs behind a domain name, and this is what Headless Service is for!

apiVersion: v1
kind: Service
metadata:
  name: keycloak-headless
spec:
  # Important parameter to discover every instance even before its complete startup
  publishNotReadyAddresses: true
  clusterIP: None
  ports:
    - name: ping
      port: 7600
      targetPort: 7600
  selector:
    app: keycloak

Thanks to this, every DNS query made by Jgroups on the domaine keycloak-headless will result to the complete list of Keycloak pod IPs in namespace!

Demo time!

We will deploy and scale our keycloak application and see clustering mode in action. The kustomization.yaml is similar to version in the second part of this series:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: keycloak

resources:
  - keycloak.yaml
  - database.yaml

configMapGenerator:
  - name: keycloak
    files:
      - standalone-ha.xml
  - name: database
    literals:
      - user=keycloak
      - name=keycloak

secretGenerator:
  - name: database
    literals:
      - password=sPCwZjuq8CMvrBn7

When we deploy it, we will have the following result:

$ kubectl apply -k .
configmap/database-56h9f7gfdh created
configmap/keycloak-k97c6gkct6 created
secret/database-8g8gk22d26 created
service/database created
service/keycloak-headless created
service/keycloak created
deployment.apps/database created
deployment.apps/keycloak created
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
database-5dcc69b7b6-m48h9   1/1     Running   0          7s
keycloak-7f5f7bd8c6-7s2br   0/1     Running   0          7s
# After few seconds…
$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
database-5dcc69b7b6-m48h9   1/1     Running   0          67s
keycloak-7f5f7bd8c6-7s2br   1/1     Running   0          67s

If we look at the Keycloak logs, everything looks good. We can scale it up and see if clustering mode do its job:

$ kubectl scale deploy/keycloak --replicas=2
deployment.apps/keycloak scaled

Now, in the log of the previously running instance, we can see the following messages:

$ kubectl logs keycloak-7f5f7bd8c6-7s2br
20:05:51,480 INFO  [org.infinispan.CLUSTER] (thread-19,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 10
20:05:51,480 INFO  [org.infinispan.CLUSTER] (thread-27,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=offlineSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 10
20:05:51,480 INFO  [org.infinispan.CLUSTER] (thread-28,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11
20:05:51,482 INFO  [org.infinispan.CLUSTER] (thread-12,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=offlineClientSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 10
20:05:51,471 INFO  [org.infinispan.CLUSTER] (thread-25,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=clientSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 9
20:05:51,486 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p6-t2) [Context=sessions] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11
20:05:51,493 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p6-t2) [Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11
20:05:51,493 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p6-t1) [Context=loginFailures] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 10
20:05:51,499 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p6-t1) [Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11
20:05:51,503 INFO  [org.infinispan.CLUSTER] (thread-28,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11
20:05:51,506 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p6-t2) [Context=clientSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 10
20:05:51,512 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p6-t2) [Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11
20:05:51,522 INFO  [org.infinispan.CLUSTER] (thread-28,ejb,keycloak-7f5f7bd8c6-7s2br) [Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-7f5f7bd8c6-7s2br, keycloak-7f5f7bd8c6-dbfxh], topology id 11

We can see the successful operation made by Infinispan to communicate between instances. In the log we found the name of our current pod keycloak-7f5f7bd8c6-7s2br and the name of the new one created through the scale command keycloak-7f5f7bd8c6-dbfxh. If we scale it back to 1 instance, new logs will be available:

$ kubectl logs keycloak-7f5f7bd8c6-7s2br
20:10:28,787 INFO  [org.infinispan.CLUSTER] (thread-34,ejb,keycloak-7f5f7bd8c6-7s2br) ISPN100001: Node keycloak-7f5f7bd8c6-dbfxh left the cluster
20:10:28,790 INFO  [org.infinispan.CLUSTER] (thread-34,ejb,keycloak-7f5f7bd8c6-7s2br) ISPN000094: Received new cluster view for channel ejb: [keycloak-7f5f7bd8c6-7s2br|4] (1) [keycloak-7f5f7bd8c6-7s2br]
20:10:28,791 INFO  [org.infinispan.CLUSTER] (thread-34,ejb,keycloak-7f5f7bd8c6-7s2br) ISPN100001: Node keycloak-7f5f7bd8c6-dbfxh left the cluster

And Voila!

Conclusion

This ends this 3-part article on Keycloak, Distroless and Kubernetes. You are now able to deploy a rock-solid, less vulnerable and scalable instance of Keycloak in your own cluster 🚀.

I hope you enjoyed it as mush as I enjoyed writing this article and share this experience about Keycloak configuration. You can find all the sample files from this article in this GitLab repository: davinkevin/keycloak-distroless.

Keycloak on Distroless into Kubernetes

Kevin Davin — Tue, 01 Jun 2021 09:14:27 +0000

In the previous article, we've seen how to build and run a Keycloak application based on Distroless base image. However, this article only used Docker to launch containers, and you mostly use an orchestrator to do that. In this article, we will see how to run our previously created image in a Kubernetes cluster.

Challenges

Using a Keycloak image based on Distroless requires some adaptation:

Launching the application require a specific command
Stopping the application require a specific command
Probes should be defined
Database connection should be provided via env variables
Configuration file should be mounted by the orchestrator

Database

Before doing some configuration, we will "install" a PostgreSQL instance inside our cluster. This will be done with this manifest:

# database.yaml
apiVersion: v1
kind: Service
metadata:
  name: database
  labels:
    app: database
spec:
  ports:
    - name: pg-port
      port: 5432
      targetPort: 5432
      protocol: TCP
  selector:
    app: database
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
spec:
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
        - image: postgres:13.3-alpine
          imagePullPolicy: IfNotPresent
          name: database
          env:
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database
                  key: password
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: name
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - su - postgres -c "pg_ctl stop -m fast"
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - -e
                - exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
          ports:
            - name: pg-port
              containerPort: 5432
              protocol: TCP

NOTE: This manifest is some kind of hello-world at PostgreSQL level and SHOULD NOT be used for production… because it doesn't manage any state. Every data will be lost if the pod restart. This is just for this example.

Launching Keycloak

In the previous step, we fetched the java command used to launched Keycloak. We will reuse it, in the Kubernetes manifest. For information, the command was:

java '-D[Standalone]' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=172.17.0.2 -Djboss.bind.address.private=172.17.0.2 -c=standalone-ha.xml

We will set $.spec.template.spec.containers[0].command and $.spec.template.spec.containers[0].args to the previously used ENTRYPOINT in the Dockerfile.

# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          # We define Java as the main command of the image.
          command: [ "java" ]
          # We put back all the parameters from the previous step.
          args:
            - "-D[Standalone]"
            - "-server"
            - "-Xms64m"
            - "-Xmx512m"
            - "-XX:MetaspaceSize=96M"
            - "-XX:MaxMetaspaceSize=256m"
            - "-Djava.net.preferIPv4Stack=true"
            - "-Djboss.modules.system.pkgs=org.jboss.byteman"
            - "-Djava.awt.headless=true"
            - "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED"
            - "-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log"
            - "-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties"
            - "-jar"
            - "/opt/jboss/keycloak/jboss-modules.jar"
            - "-mp"
            - "/opt/jboss/keycloak/modules"
            - "org.jboss.as.standalone"
            - "-Djboss.home.dir=/opt/jboss/keycloak"
            - "-Djboss.server.base.dir=/opt/jboss/keycloak/standalone"
            - "-c=standalone.xml"
            - "-b=0.0.0.0"
            - "-bprivate=0.0.0.0"
            - "-bmanagement=0.0.0.0"

NOTE: Like previously, the parameter can and should be adapted to your needs, for performance and security reasons.

Providing configuration to Keycloak

Another part of the Keycloak configuration is the standalone.xml. Our goal here will be to mount the file inside the running container, at the right place, to be used as source of configuration at boot time.

# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      # We declare a volume containing the configuration of keycloak
      volumes:
        - name: keycloak-config
          # The configMap targeted here will be declared in the next step of the article
          configMap:
            name: keycloak
      containers:
        - name: keycloak
          # And then, we mount the file from the previously seen volume at the 
          # location where keycloak want to find it…  
          volumeMounts:
            - name: keycloak-config
              mountPath: /opt/jboss/keycloak/standalone/configuration/standalone.xml
              subPath: standalone.xml

Probes

In Kubernetes world, the orchestrator needs to know when the system is live and ready. For this, we have to define some liveness and readyness probes allowing Kubernetes to act and react to problem with Keycloak if required. Again, this is just some values in our manifest:

# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          # Liveness, if in error, keycloak will be restarted
          livenessProbe:
            httpGet:
                path: /auth/
                port: http
            initialDelaySeconds: 30
            timeoutSeconds: 5
          # Readiness, if in success, traffic will be routed to this pod
          readinessProbe:
            httpGet:
                path: /auth/realms/master
                port: http
            initialDelaySeconds: 30
            timeoutSeconds: 1

NOTE: RedHat team will introduce more advanced and precise liveness and readyness probes in future version of Keycloak 😇.

Lifecycle Pre-Stop Command

To gracefully shutdown Keycloak, we usually have to execute a specific shell script. In our case, without any shell, this will be problematic… like for ENTRYPOINT, we will extract the java command from the shell script and use it in our manifest.

# We launch the keycloak original image and launch a bash inside it
$ docker run -it --rm --entrypoint=bash jboss/keycloak:13.0.1
# We use our magic trick to see commands executed by the jboss-cli.sh we will use
bash-4.4$ awk -i inplace 'NR==2 {print "set -x"} 1' /opt/jboss/keycloak/bin/jboss-cli.sh
# Finally, we launch the script with parameters we want to execute in our pre-hook
bash-4.4$ /opt/jboss/keycloak/bin/jboss-cli.sh --connect command=:shutdown --timeout=20
++ dirname /opt/jboss/keycloak/bin/jboss-cli.sh
+ DIRNAME=/opt/jboss/keycloak/bin
+ GREP=grep
+ . /opt/jboss/keycloak/bin/common.sh
++ '[' x = x ']'
++ COMMON_CONF=/opt/jboss/keycloak/bin/common.conf
++ '[' -r /opt/jboss/keycloak/bin/common.conf ']'
+ cygwin=false
+ darwin=false
+ case "`uname`" in
++ uname
+ false
++ cd /opt/jboss/keycloak/bin/..
++ pwd
+ RESOLVED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' x/opt/jboss/keycloak = x ']'
++ cd /opt/jboss/keycloak
++ pwd
+ SANITIZED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' /opt/jboss/keycloak '!=' /opt/jboss/keycloak ']'
+ export JBOSS_HOME
+ '[' x = x ']'
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
+ '[' x = x ']'
+ '[' x '!=' x ']'
+ JAVA=java
+ setDefaultModularJvmOptions
+ setModularJdk
+ java --add-modules=java.se -version
+ MODULAR_JDK=true
+ '[' true = true ']'
++ echo
++ grep '\-\-add\-modules'
+ DEFAULT_MODULAR_JVM_OPTIONS=
+ '[' x = x ']'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ false
+ false
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing'
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true'
++ eval echo '"/opt/jboss/keycloak/modules"'
+++ echo /opt/jboss/keycloak/modules
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
++ echo --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true
++ grep logging.configuration
+ LOG_CONF=
+ '[' x = x ']'
+ exec java --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true -Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.cli --connect command=:shutdown --timeout=20
Failed to connect to the controller: The controller is not available at localhost:9990: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: Connection refused

The execution ends in error, which is normal because, in this case, no Keycloak instance are running. The java command is displayed, starting with + exec java. All of this can be moved into our yaml manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          lifecycle:
            preStop:
              exec:
                command:
                  - "java"
                  - '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
                  - '--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
                  - '--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
                  - '-Djboss.modules.system.pkgs=com.sun.java.swing'
                  - '-Dcom.ibm.jsse2.overrideDefaultTLS=true'
                  - '-Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties'
                  - "-jar"
                  - "/opt/jboss/keycloak/jboss-modules.jar"
                  - "-mp"
                  - "/opt/jboss/keycloak/modules"
                  - "org.jboss.as.cli"
                  - "--connect"
                  - '--commands=shutdown --timeout=20'

Environment Configuration

Finally, we have to provide all the configuration to Keycloak. Until now, we only define how the value will be linked but not which value at all. To do that, we will first include some environment values into the Keycloak manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          env:
            - name: DB_ADDR
              value: database
            - name: DB_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: name
            - name: DB_USER
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: user
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database
                  key: password

Here, we are mainly targeting values required for database connection. To provide it to the deployment, we will use a kustomization.yaml file. It will gather every manifests presented until now and add concret values:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Install everything in the kubernetes namespace named Keycloak
namespace: keycloak

# Load both keycloak.yaml and database.yaml
resources:
  - keycloak.yaml
  - database.yaml

# Generate two ConfigMap
configMapGenerator:
  # One for Keycloak, containing the standalone.xml
  - name: keycloak
    files:
      - standalone.xml
  # One for database, containing user and name
  - name: database
    literals:
      - user=keycloak
      - name=keycloak

# Generate a Secret
secretGenerator:
  # For the database, containing just the password of the database
  - name: database
    literals:
      - password=sPCwZjuq8CMvrBn7

NOTE: The password is here stored in clear, you can choose to use some tooling to encrypt the secret at git level (with SOPS) or cluster level (with SealedSecret).

Deploying Keycloak

In this step, we will deploy our version of Keycloak in a Kubernetes Cluster. For this example, I choose to use docker-for-mac and its Kubernetes integration. You can, of course, use any Kubernetes distribution (Google Kubernetes Engine, Azure Kubernetes Service, Amazon Elastic Kubernetes Service…).

$ ls
database.yaml  keycloak.yaml  kustomization.yaml  standalone.xml
# We deploy all the manifests
$ kubectl apply -k .
configmap/database-56h9f7gfdh created
configmap/keycloak-96k2tfg747 created
secret/database-8g8gk22d26 created
service/database created
service/keycloak created
deployment.apps/database created
deployment.apps/keycloak created
# We create a port-forward to the keycloak pod
$ kubectl -n keycloak  port-forward pod/keycloak-567797b6bb-6vqvz 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

To access it, we create a port-forward between my computer and the cluster. Thanks to that, I'm able to access the UI:

And Voila!

Conclusion

In this article, we've seen how to deploy our custom Keycloak based on Distroless Java. This is a challenging setup, but at the end we can use a more secure version of Keycloak without any shell, which prevent any attack using shell as a vector of code execution for example!

I hope you liked it, you can find all the sample files from this article in this GitLab repository: davinkevin/keycloak-distroless.

Keycloak on Distroless

Kevin Davin — Fri, 28 May 2021 06:04:52 +0000

⚠️ This article has been written and tested on Keycloak v13 and is working with version until 16. For later version, using Quarkus based distribution (v17+), another article will be redacted.

Keycloak is a wonderful piece of software, managed with success by RedHat, to be used as an Identity and Access Management software. RedHat distribute it as a zip package to be run on a machine with a JVM installed or as a container. Nowadays, container is a simpler solution, especially if you are using an orchestrator like Kubernetes.

The Keycloak image is available on the DockerHub or Quay. It provides an important level of configuration through environment variables, which is useful if you are not familiar with WildFly configuration. But, this solution has an important downside, especially for a tool dedicated to security… tags are not maintained at OS level over time and has many vulnerabilities.

You can see below, a lot of vulnerabilities in the latest Keycloak image, especially at the OS level. In some case, you can't choose to rely on so many vulnerabilities and need to fix that, or at least reduce them.

$ trivy image jboss/keycloak:13.0.1
2021-05-26T19:23:14.416+0200    INFO    Detected OS: redhat
2021-05-26T19:23:14.416+0200    INFO    Detecting RHEL/CentOS vulnerabilities...
2021-05-26T19:23:14.432+0200    INFO    Number of PL dependency files: 621

jboss/keycloak:13.0.1 (redhat 8.4)
==================================
Total: 118 (UNKNOWN: 0, LOW: 49, MEDIUM: 67, HIGH: 2, CRITICAL: 0)
...

NOTE: Number of CVEs in an image evolves over time, so reports in this article can be way different if you run it by yourself.

On one side, you can choose to upgrade every packages in the image manually, hoping a fix is available in the official CentOS registry. Another solution is to change the base image to something with less vulnerability like Google Distroless. Those images only contain the runtime for your application and nothing less… no shell, no package manager, nothing… just your runtime. For Keycloak, we will use the Distroless Java image to sanitize our workload.

Crafting the best Dockerfile possible

The original Keycloak image use a lot of bash scripts to configure the whole system. This is a good idea, but here, we don't have any shell in our Distroless base image, so we will have to extract the application, and the way to launch it from scratch.

Moving Keycloak into Distroless

If we analyse the jboss/keycloak:13.0.1 image with Dive, we can see all Keycloak related files are stored into /opt/jboss/.

We will copy them into our distroless then, with the following Dockerfile:

FROM jboss/keycloak:13.0.1 as base

FROM gcr.io/distroless/java:11-nonroot
COPY --chown=nonroot:nonroot --from=base /opt/jboss /opt/jboss

The execution is pretty simple:

$ docker build -t keycloak-distroless .
[+] Building 0.6s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 37B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for gcr.io/distroless/java:11-nonroot         0.5s
 => [internal] load metadata for docker.io/jboss/keycloak:13.0.1           0.0s
 => [base 1/1] FROM docker.io/jboss/keycloak:13.0.1                        0.0s
 => [stage-1 1/2] FROM gcr.io/distroless/java:11-nonroot@sha256:07d017944  0.0s
 => CACHED [stage-1 2/2] COPY --chown=nonroot:nonroot --from=base /opt/jb  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:06e849f0ab369043be9c071a446484e2a699a114dd988  0.0s
 => => naming to docker.io/library/keycloak-distroless                     0.0s

Sadly, if we are launching it like this, we will see the following error:

$ docker run --rm -it -p 8080:8080 keycloak-distroless
Error: -jar requires jar file specification
Usage: java [options] <mainclass> [args...]
           (to execute a class)
   or  java [options] -jar <jarfile> [args...]
           (to execute a jar file)
   or  java [options] -m <module>[/<mainclass>] [args...]
       java [options] --module <module>[/<mainclass>] [args...]
           (to execute the main class in a module)
   or  java [options] <sourcefile> [args]
           (to execute a single source-file program)

 Arguments following the main class, source file, -jar <jarfile>,
 -m or --module <module>/<mainclass> are passed as the arguments to
 main class.
...

This is because the default ENTRYPOINT of this distroless image want to launch a (fat) JAR, but keycloak is more complex than this, so we will have to find the right ENTRYPOINT for our use case.

Generating the ENTRYPOINT

For this one, we will use the original image to see how Keycloak is launched in its natural state. To do that, we will edit the standalone.sh file to make it more verbose and copy the java command generated from it. We will follow the official documentation to launch keycloak, but we will log into the container to do our magic trick:

# Starting the container with the minimal configuration and log into it thanks to the custom entrypoint
$ docker run -it --rm -e DB_VENDOR=h2 --entrypoint=bash jboss/keycloak:13.0.1
# From here, we are IN the Keycloak image!
# The following command update the standalone.sh file to be a lot verbose
bash-4.4$ awk -i inplace 'NR==2 {print "set -x"} 1' /opt/jboss/keycloak/bin/standalone.sh
# Finally, we will launch keycloak from here and stop it when we found the line starting with "++ java"
bash-4.4$ /opt/jboss/tools/docker-entrypoint.sh

=========================================================================

  Using Embedded H2 database

=========================================================================

+ DEBUG_MODE=false
+ DEBUG_PORT=8787
+ GC_LOG=
+ SERVER_OPTS=
+ '[' 3 -gt 0 ']'
+ case "$1" in
+ SERVER_OPTS=' '\''-Djboss.bind.address=172.17.0.2'\'''
+ shift
+ '[' 2 -gt 0 ']'
+ case "$1" in
+ SERVER_OPTS=' '\''-Djboss.bind.address=172.17.0.2'\'' '\''-Djboss.bind.address.private=172.17.0.2'\'''
+ shift
+ '[' 1 -gt 0 ']'
+ case "$1" in
+ SERVER_OPTS=' '\''-Djboss.bind.address=172.17.0.2'\'' '\''-Djboss.bind.address.private=172.17.0.2'\'' '\''-c=standalone-ha.xml'\'''
+ shift
+ '[' 0 -gt 0 ']'
++ dirname /opt/jboss/keycloak/bin/standalone.sh
+ DIRNAME=/opt/jboss/keycloak/bin
++ basename /opt/jboss/keycloak/bin/standalone.sh
+ PROGNAME=standalone.sh
+ GREP=grep
+ . /opt/jboss/keycloak/bin/common.sh
++ '[' x = x ']'
++ COMMON_CONF=/opt/jboss/keycloak/bin/common.conf
++ '[' -r /opt/jboss/keycloak/bin/common.conf ']'
+ MAX_FD=maximum
+ MALLOC_ARENA_MAX=1
+ export MALLOC_ARENA_MAX
+ cygwin=false
+ darwin=false
+ linux=false
+ solaris=false
+ freebsd=false
+ other=false
+ case "`uname`" in
++ uname
+ linux=true
+ false
++ cd /opt/jboss/keycloak/bin/..
++ pwd
+ RESOLVED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' x/opt/jboss/keycloak = x ']'
++ cd /opt/jboss/keycloak
++ pwd
+ SANITIZED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' /opt/jboss/keycloak '!=' /opt/jboss/keycloak ']'
+ export JBOSS_HOME
+ '[' x = x ']'
+ RUN_CONF=/opt/jboss/keycloak/bin/standalone.conf
+ '[' -r /opt/jboss/keycloak/bin/standalone.conf ']'
+ . /opt/jboss/keycloak/bin/standalone.conf
++ '[' x = x ']'
++ JBOSS_MODULES_SYSTEM_PKGS=org.jboss.byteman
++ '[' x = x ']'
++ JAVA_OPTS='-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true'
++ JAVA_OPTS='-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true'
++ JAVA_OPTS='-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true '
+ '[' false = true ']'
+ '[' x = x ']'
+ '[' x '!=' x ']'
+ JAVA=java
+ true
+ CONSOLIDATED_OPTS='-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true   '\''-Djboss.bind.address=172.17.0.2'\'' '\''-Djboss.bind.address.private=172.17.0.2'\'' '\''-c=standalone-ha.xml'\'''
+ for var in $CONSOLIDATED_OPTS
++ echo -Xms64m
++ tr -d \'
+ p=-Xms64m
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo -Xmx512m
++ tr -d \'
+ p=-Xmx512m
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo -XX:MetaspaceSize=96M
++ tr -d \'
+ p=-XX:MetaspaceSize=96M
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo -XX:MaxMetaspaceSize=256m
++ tr -d \'
+ p=-XX:MaxMetaspaceSize=256m
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo -Djava.net.preferIPv4Stack=true
++ tr -d \'
+ p=-Djava.net.preferIPv4Stack=true
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo -Djboss.modules.system.pkgs=org.jboss.byteman
++ tr -d \'
+ p=-Djboss.modules.system.pkgs=org.jboss.byteman
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo -Djava.awt.headless=true
++ tr -d \'
+ p=-Djava.awt.headless=true
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo ''\''-Djboss.bind.address=172.17.0.2'\'''
++ tr -d \'
+ p=-Djboss.bind.address=172.17.0.2
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo ''\''-Djboss.bind.address.private=172.17.0.2'\'''
++ tr -d \'
+ p=-Djboss.bind.address.private=172.17.0.2
+ case $p in
+ for var in $CONSOLIDATED_OPTS
++ echo ''\''-c=standalone-ha.xml'\'''
++ tr -d \'
+ p=-c=standalone-ha.xml
+ case $p in
+ false
+ false
+ false
+ false
+ '[' x = x ']'
+ JBOSS_BASE_DIR=/opt/jboss/keycloak/standalone
+ '[' x = x ']'
+ JBOSS_LOG_DIR=/opt/jboss/keycloak/standalone/log
+ '[' x = x ']'
+ JBOSS_CONFIG_DIR=/opt/jboss/keycloak/standalone/configuration
+ '[' x = x ']'
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
+ false
+ '[' '' '!=' true ']'
++ echo -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
++ grep '\-d64'
+ JVM_D64_OPTION=
++ echo -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
++ grep '\-d32'
+ JVM_D32_OPTION=
++ echo -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
++ grep '\-server'
+ SERVER_SET=
++ echo -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
++ grep '\-client'
+ CLIENT_SET=
+ '[' x '!=' x ']'
+ '[' x '!=' x ']'
+ false
+ '[' x = x -a x = x ']'
+ false
+ PREPEND_JAVA_OPTS=' -server'
+ setModularJdk
+ java --add-modules=java.se -version
+ MODULAR_JDK=true
+ '[' '' = true ']'
+ setDefaultModularJvmOptions -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+ setModularJdk
+ java --add-modules=java.se -version
+ MODULAR_JDK=true
+ '[' true = true ']'
++ echo -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
++ grep '\-\-add\-modules'
+ DEFAULT_MODULAR_JVM_OPTIONS=
+ '[' x = x ']'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ JAVA_OPTS='-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true   --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ JAVA_OPTS=' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true   --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
++ echo -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED
++ grep 'java\.security\.manager'
+ SECURITY_MANAGER_SET=
+ '[' x '!=' x ']'
+ MODULE_OPTS=
+ '[' '' = true ']'
++ echo ''
++ grep '\-javaagent:'
+ AGENT_SET=
+ '[' x '!=' x ']'
+ echo =========================================================================
=========================================================================
+ echo ''

+ echo '  JBoss Bootstrap Environment'
  JBoss Bootstrap Environment
+ echo ''

+ echo '  JBOSS_HOME: /opt/jboss/keycloak'
  JBOSS_HOME: /opt/jboss/keycloak
+ echo ''

+ echo '  JAVA: java'
  JAVA: java
+ echo ''

+ echo '  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true   --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true   --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED
+ echo ''

+ echo =========================================================================
=========================================================================
+ echo ''

+ true
+ '[' x1 = x ']'
+ eval '"java"' '-D"[Standalone]"' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED '"-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log"' '"-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties"' -jar '"/opt/jboss/keycloak/jboss-modules.jar"' -mp '"/opt/jboss/keycloak/modules"' org.jboss.as.standalone '-Djboss.home.dir="/opt/jboss/keycloak"' '-Djboss.server.base.dir="/opt/jboss/keycloak/standalone"' ' '\''-Djboss.bind.address=172.17.0.2'\'' '\''-Djboss.bind.address.private=172.17.0.2'\'' '\''-c=standalone-ha.xml'\''' '&'
+ JBOSS_PID=122
+ trap 'kill -HUP  122' HUP
++ java '-D[Standalone]' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=172.17.0.2 -Djboss.bind.address.private=172.17.0.2 -c=standalone-ha.xml
+ trap 'kill -TERM 122' INT
+ trap 'kill -QUIT 122' QUIT
+ trap 'kill -PIPE 122' PIPE
+ trap 'kill -TERM 122' TERM
+ '[' x '!=' x ']'
+ WAIT_STATUS=128
+ '[' 128 -ge 128 ']'
+ wait 122
18:08:24,393 INFO  [org.jboss.modules] (main) JBoss Modules version 1.11.0.Final
18:08:25,034 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.12.Final
18:08:25,050 INFO  [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
18:08:25,219 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) starting
18:08:25,412 INFO  [org.jboss.vfs] (MSC service thread 1-4) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
18:08:26,228 INFO  [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.15.3.Final
^C
bash-4.4$ exit
$

In the huge starting log, we can see the following command, starting with ++ java:

java '-D[Standalone]' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=172.17.0.2 -Djboss.bind.address.private=172.17.0.2 -c=standalone-ha.xml

This is the java command we will put inside our Dockerfile, as an ENTRYPOINT to make Keycloak start.

FROM jboss/keycloak:13.0.1 as base

FROM gcr.io/distroless/java:11-nonroot
COPY --chown=nonroot:nonroot --from=base /opt/jboss /opt/jboss

ENTRYPOINT [ "java", "-D[Standalone]", "-server", "-Xms64m", "-Xmx512m", "-XX:MetaspaceSize=96M", "-XX:MaxMetaspaceSize=256m", "-Djava.net.preferIPv4Stack=true", "-Djboss.modules.system.pkgs=org.jboss.byteman", "-Djava.awt.headless=true", "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED", "--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED", "--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED", "-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log", "-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties", "-jar", "/opt/jboss/keycloak/jboss-modules.jar", "-mp", "/opt/jboss/keycloak/modules", "org.jboss.as.standalone", "-Djboss.home.dir=/opt/jboss/keycloak", "-Djboss.server.base.dir=/opt/jboss/keycloak/standalone", "-Djboss.bind.address=0.0.0.0", "-Djboss.bind.address.private=1720.0.0.0", "-c=standalone.xml" ]

NOTE: You can tune this command to increase or decrease the memory setup, the private/public bind address of your keycloak instance and many other parameters. Here, we changed the configuration file used (-c=standalone.xml instead of -c=standalone-ha.xml for simplicity reasons) and the bound ip adresses (to 0.0.0.0)

If we build and run this, we will be able to access the Keycloak UI:

$ docker build -t keycloak-distroless .
[+] Building 0.6s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 37B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for gcr.io/distroless/java:11-nonroot         0.5s
 => [internal] load metadata for docker.io/jboss/keycloak:13.0.1           0.0s
 => [base 1/1] FROM docker.io/jboss/keycloak:13.0.1                        0.0s
 => [stage-1 1/2] FROM gcr.io/distroless/java:11-nonroot@sha256:07d017944  0.0s
 => CACHED [stage-1 2/2] COPY --chown=nonroot:nonroot --from=base /opt/jb  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:100908720c19018f2408bb53a5d78ef3d9eb51391b165  0.0s
 => => naming to docker.io/library/keycloak-distroless                     0.0s

$ docker run --rm -it -p 8080:8080 keycloak-distroless
18:15:22,645 INFO  [org.jboss.modules] (main) JBoss Modules version 1.11.0.Final
18:15:23,283 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.12.Final
18:15:23,292 INFO  [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
18:15:23,452 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) starting
18:15:23,694 INFO  [org.jboss.vfs] (MSC service thread 1-5) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
18:15:24,457 INFO  [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.15.3.Final
...
...
18:15:44,642 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) WFLYUT0021: Registered web context: '/auth' for server 'default-server'
18:15:44,778 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
18:15:44,886 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
18:15:44,892 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) started in 22800ms - Started 692 of 977 services (686 services are lazy, passive or on-demand)
18:15:44,896 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
18:15:44,896 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

If we try to access http://localhost:8080/, we can see the following page 🎉.

This is a good start, but this is just the minimal setup with H2 database, we often want something more robust for production!

Generating the perfect configuration

The jboss/keycloak image use a lot of environment variables to configure keycloak (and the underlying standalone.xml) for you… but in our case, we can't use that because:

We don't have a shell to run those scripts.
We don't want to run those scripts at every startup / scale-up.

So, we will have to steal the generated standalone.xml file from the original container, post start-up, and include it in our container. For this example, I will use PostgreSQL as our main database.

To do this, I will use two shells side-by-side, one to launch Keycloak, and the other one to fetch the configuration.

# In the first shell
# Creation of a docker network
first-shell$ docker network create keycloak-network
4da77163731b584bef2c6d0b00386b9d62e31fa216204c6c6795f66e109ba1a6
# Launching PostgreSQL linked to the network previously created
first-shell$ docker run --rm -d --name postgres --net keycloak-network \
-e POSTGRES_DB=keycloak \
-e POSTGRES_USER=keycloak \
-e POSTGRES_PASSWORD=password postgres
229816da42707e772542f1b089c616a2333a6fbe1aea2be7efe658d6f2c934a1
first-shell$ docker run -it --rm --name keycloak \
-e DB_ADDR=postgres \
-e DB_USER=keycloak \
-e DB_PASSWORD=password \
-e KEYCLOAK_USER=foo \
-e KEYCLOAK_PASSWORD=bar \
--net keycloak-network jboss/keycloak:13.0.1

=========================================================================

  Using PostgreSQL database

=========================================================================

18:32:25,172 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.11.0.Final
18:32:25,279 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.12.Final
18:32:25,302 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
18:32:25,453 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) starting
...
18:32:59,128 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
18:32:59,129 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

In another shell, while the previous is still running, we will execute the following command to get the standalone.xml file used to configure Keycloak:

second-shell$ docker cp keycloak:/opt/jboss/keycloak/standalone/configuration/standalone.xml .
second-shell$ ls
standalone.xml
# We can now stop the keycloak container
second-shell$ docker stop keycloak
keycloak
second-shell$

Now, we will start the Distroless Keycloak and mount the standalone.xml inside the container.

$ docker run --rm -it -e DB_USER=keycloak -e DB_PASSWORD=password --net keycloak-network -v $(pwd)/standalone.xml:/opt/jboss/keycloak/standalone/configuration/standalone.xml -p 8080:8080 keycloak-distroless
19:42:20,707 INFO  [org.jboss.modules] (main) JBoss Modules version 1.11.0.Final
19:42:21,317 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.12.Final
19:42:21,329 INFO  [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
19:42:21,470 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) starting
19:42:21,651 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
19:42:22,577 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.15.3.Final
...
19:43:58,356 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 13.0.1 (WildFly Core 15.0.1.Final) started in 17828ms - Started 595 of 873 services (584 services are lazy, passive or on-demand)
19:43:58,362 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
19:43:58,363 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

And Voila!

What about security?

The original and main purpose of this manipulation is to reduce the number of CVEs present in our image. We will be able to compare it using trivy again on our newly image.

$ trivy image keycloak-distroless
2021-05-26T21:11:15.959+0200    INFO    Detected OS: debian
2021-05-26T21:11:15.959+0200    INFO    Detecting Debian vulnerabilities...
2021-05-26T21:11:15.963+0200    INFO    Number of PL dependency files: 621
2021-05-26T21:11:15.963+0200    INFO    Detecting jar vulnerabilities...

keycloak-distroless (debian 10.9)
=================================
Total: 27 (UNKNOWN: 0, LOW: 23, MEDIUM: 3, HIGH: 1, CRITICAL: 0)

We can see, our image contain fewer vulnerabilities, at LOW, MEDIUM or HIGH level. Again, this depends on when you are doing this analysis. With the solution provided in this article, you'll be able to rebuild your keycloak on a new, up-to-date, Distroless base image without updating keycloak. With the original keycloak image, the keycloak version is tied to the OS version (and security flaws).

NOTE: The jboss/keycloak:13.0.1 was released few hours before the creation of this article while the distroless/java-debian10:non-root was released 1 month ago. This is the worst comparison scenario possible for the Distroless base image.

Another benefit of this alternative is to create a smaller image for keycloak. The previous dive reports stated 698 MB for the official image when our custom image weight only 519 MB, so around 179 MB reduction 🏋️‍♂️, and I'm sure we can remove almost 100MB by removing all useless binaries in the image (useless drivers, command line tools, documentation…).

Conclusion

With this article, you should be able to build, from the official jboss/keycloak image a custom one based on the Distroless/java and even fix CVEs by doing it again when a new version of Distroless/java image is released.

I hope you liked it, you can find all the sample files from this article in this GitLab repository: davinkevin/keycloak-distroless.

Anthos Service Mesh, Istio on Google Cloud ⛵️

Kevin Davin — Tue, 18 May 2021 05:55:46 +0000

Anthos Service Mesh, a managed Istio ⛵️

Istio is one of the most advanced pieces of software in the Kubernetes ecosystem. It allows to redefine the way our services are communicating with each other, without being invasive. Istio works by taking control over the network of your Kubernetes cluster and allows applying configurations (through YAML). If you want to discover Istio, I invite you to read the excellent documentation provided in istio.io

The main problem with Istio, is the complexity to manage and configure it. Like kubernetes, this system is complex, and errors could lead to downtime in your cluster… 😓. Lots of software-company starts to provide a pre-configured version of Istio, and here we will talk about Anthos Service Mesh.

Anthos Service Mesh is available on Anthos clusters running in Google Cloud, AWS or on-premise (different features are available depending on the cluster locality). Here, we will describe the Google Cloud version based on Anthos Service Mesh version 1.9.3.asm-2 (the last version may be different when you read this article).

NOTE: A fully managed version of Anthos Service Mesh exists but is actually in preview/beta. I prefer, for now, using the standard version of Anthos Service Mesh (aka Customer-managed control plane).

Installation

The installation is pretty straight-forward, Google provides a script install_asm to automate the installation on an already existing GKE cluster:

./install_asm \
  --project_id kevin-anthos-asm \
  --cluster_name anthos-asm-demo  \
  --cluster_location europe-north1-a  \
  --mode install \
  --output_dir ./asm-downloads \
  --enable_all

NOTE: I am installing the latest version of ASM here, but you can choose a different one with the --revision_name parameter if required.

To be executed, the script has some requirements. I invite you to check that here. The best solution is to use the Google Cloud Shell to do the installation, it fulfills requirements by default.

install_asm: Setting up necessary files...
install_asm: Fetching/writing GCP credentials to kubeconfig file...
install_asm: [WARNING]: nc not found, skipping k8s connection verification
install_asm: [WARNING]: (Installation will continue normally.)
install_asm: Checking installation tool dependencies...
install_asm: Getting account information...
install_asm: Confirming cluster information for kevin-anthos-asm/europe-north1-a/anthos-asm-demo...
install_asm: Downloading ASM..
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 41.7M  100 41.7M    0     0  31.5M      0  0:00:01  0:00:01 --:--:-- 31.5M
install_asm: Downloading ASM kpt package...
fetching package "/asm" from "https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages" to "asm"
install_asm: Confirming node pool requirements for kevin-anthos-asm/europe-north1-a/anthos-asm-demo...
install_asm: Checking Istio installations...
install_asm: Enabling required APIs...
install_asm: Binding user:kevin.davin@stack-labs.com to required IAM roles...
install_asm: Checking for project kevin-anthos-asm...
install_asm: Reading labels for europe-north1-a/anthos-asm-demo...
install_asm: Adding labels to europe-north1-a/anthos-asm-demo...
install_asm: Enabling Workload Identity on europe-north1-a/anthos-asm-demo...
install_asm: (This could take awhile, up to 10 minutes)
install_asm: Initializing meshconfig API...
install_asm: Enabling Stackdriver on europe-north1-a/anthos-asm-demo...
install_asm: Querying for core/account...
install_asm: Binding kevin.davin@stack-labs.com to cluster admin role...
clusterrolebinding.rbac.authorization.k8s.io/kevin.davin-cluster-admin-binding created
install_asm: Creating istio-system namespace...
namespace/istio-system created
install_asm: Configuring kpt package...
asm/
set 22 field(s) of setter "gcloud.container.cluster" to value "anthos-asm-demo"
asm/
set 40 field(s) of setter "gcloud.core.project" to value "kevin-anthos-asm"
asm/
set 2 field(s) of setter "gcloud.project.projectNumber" to value "62405001080"
asm/
set 6 field(s) of setter "gcloud.project.environProjectNumber" to value "62405001080"
asm/
set 21 field(s) of setter "gcloud.compute.location" to value "europe-north1-a"
asm/
set 2 field(s) of setter "gcloud.compute.network" to value "kevin-anthos-asm-default"
asm/
set 6 field(s) of setter "anthos.servicemesh.rev" to value "asm-193-2"
asm/
set 2 field(s) of setter "anthos.servicemesh.tag" to value "1.9.3-asm.2"
install_asm: Installing validation webhook fix...
service/istiod created
install_asm: Installing ASM control plane...
install_asm: ...done!
install_asm: Installing ASM CanonicalService controller in asm-system namespace...
namespace/asm-system created
customresourcedefinition.apiextensions.k8s.io/canonicalservices.anthos.cloud.google.com created
role.rbac.authorization.k8s.io/canonical-service-leader-election-role created
clusterrole.rbac.authorization.k8s.io/canonical-service-manager-role created
clusterrole.rbac.authorization.k8s.io/canonical-service-metrics-reader created
serviceaccount/canonical-service-account created
rolebinding.rbac.authorization.k8s.io/canonical-service-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/canonical-service-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/canonical-service-proxy-rolebinding created
service/canonical-service-controller-manager-metrics-service created
deployment.apps/canonical-service-controller-manager created
install_asm: Waiting for deployment...
deployment.apps/canonical-service-controller-manager condition met
install_asm: ...done!
install_asm:
install_asm: *****************************
client version: 1.9.3-asm.2
control plane version: 1.9.3-asm.2
data plane version: 1.9.3-asm.2 (2 proxies)
install_asm: *****************************
install_asm: The ASM control plane installation is now complete.
install_asm: To enable automatic sidecar injection on a namespace, you can use the following command:
install_asm: kubectl label namespace <NAMESPACE> istio-injection- istio.io/rev=asm-193-2 --overwrite
install_asm: If you use 'istioctl install' afterwards to modify this installation, you will need
install_asm: to specify the option '--set revision=asm-193-2' to target this control plane
install_asm: instead of installing a new one.
install_asm: To finish the installation, enable Istio sidecar injection and restart your workloads.
install_asm: For more information, see:
install_asm: https://cloud.google.com/service-mesh/docs/proxy-injection
install_asm: The ASM package used for installation can be found at:
install_asm: /home/kevin_davin/anthos/asm/2021-05-11-apres-midi/asm-downloads/asm
install_asm: The version of istioctl that matches the installation can be found at:
install_asm: /home/kevin_davin/anthos/asm/2021-05-11-apres-midi/asm-downloads/istio-1.9.3-asm.2/bin/istioctl
install_asm: A symlink to the istioctl binary can be found at:
install_asm: /home/kevin_davin/anthos/asm/2021-05-11-apres-midi/asm-downloads/istioctl
install_asm: The combined configuration generated for installation can be found at:
install_asm: /home/kevin_davin/anthos/asm/2021-05-11-apres-midi/asm-downloads/asm-193-2-manifest-raw.yaml
install_asm: The full, expanded set of kubernetes resources can be found at:
install_asm: /home/kevin_davin/anthos/asm/2021-05-11-apres-midi/asm-downloads/asm-193-2-manifest-expanded.yaml
install_asm: *****************************
install_asm: Successfully installed ASM.

This script will install a custom version of Istio, named Antos Service Mesh. At the end, your cluster will have two new namespaces, asm-system and istio-system:

$ kubectl get ns
NAME              STATUS   AGE
asm-system        Active   127m
default           Active   141m
istio-system      Active   128m
kube-node-lease   Active   141m
kube-public       Active   141m
kube-system       Active   141m

You have successfully installed Anthos Service Mesh on your GKE Cluster… we have to use it now!

Namespace Activation

Istio is a cluster-wide tool, which can be activated at a namespace level or at a component level (but less common). We have to add a label to our namespace to trigger Istio functionalities on it:

$ kubectl create namespace workshop
$ kubectl label namespace workshop istio.io/rev=asm-193-2 --overwrite

Here, asm-193-2 is the version provided by the install_asm command logs. With this information, ASM knows it will have to inject side-car container for every component of this namespace.

NOTE: If you want more details on the installation process, the official documentation is available here and provide a lot of information for various use cases.

Functionalities

Anthos Service Mesh is a branded version of Istio. Modifications provided by Google are pretty soft and here mainly to make the system compatible with the cloud console. In Customer-managed control plane, you have access to every functionality the vast majority of feature provided by Istio 1.9.

You can consult the complete list of features available here

Dashboards

The main advantage of ASM over the OpenSource version of Istio is its integration in the Google Cloud console.

For this example, I've deployed 3 applications in the workshop namespace. Those applications came from our Stack Labs Workshop on Istio (accessible here, and fully open source). With this, we can use theirs dashboards provided.

Global overview

We can have a global point-of-view on our micro-services deployed in our cluster. We have a tabular view, which can be filtered on namespace, providing a clear view of our services status and performance.

A global topology view (still in beta 🧪) allowing us to drill down on our services, components, deployments, pods… very useful if we want to understand the communication schema in our cluster.

If we want to have deeper understanding on each component, we have access to a service specific view, accessible by clicking on a service on the tabular view.

Service Dashboard

This dashboard will provide a specific point of view on the behaviour of your service. Here, we will focus on the middleware service. For this example, we configure this application to send back 500 errors 50% of the time.

This main view is here to summarize every following dashboards into one. It is an entry point for our service
analysis.

The Health view is here to present us the Service Level Objectives (SLO) based on Service Level Indicators (SLI) we defined on our service.

NOTE: If you want to learn more about SLOs and SLIs, this blog post summarizes it (a lot) the idea behind it. You can also read the documentation and books provide freely by Google SRE team here.

We can define SLOs and SLIs on our service with multiple information gathered by the cloud console and Anthos Service Mesh for us.

You can define an alerting strategy for each SLO. Multiple systems are available from Slack, Pager-Duty, Email for the most standard to web-hook, cloud function or any other programmating system. I choose email for this example and I receive this after few minutes, because the SLO defined was not reached anymore.

We have access to a global metrics view of the service. CPU, RAM, requests by seconds… all the required information to be able to follow the health of the application.

We can analyse all the connectivity of our service. Here, we have a complete list of every services connecting to our service (inbound) or services reached by our service (outbound).

The infrastructure pan allows us to see every instance of our service over time. Each of them has its own metrics (CPU, RAM, error rate…) and so, we can analyse at a fine grain level the performance and behaviour of our system.

Security pane allows analysing communication security level between services. Istio and ASM provides a built-in way to communicate with mTLS between components. Here, you will see if exchanges are made "in clear" or with a secure protocol. I didn't configure anything and communications between frontend, middleware and database are secured by default.

Finally, a useful view is available to consult YAML resources deployed in the cluster corresponding to this service. Deployment, VirtualService, DestinationRoute… All resources are available from the web-ui, simplifying analysis again.

Timeline everywhere!

When you analyze production, especially when a problem occurs, you can't only use current data, you have to compare data with past data of the same system to elaborate a conclusion. Here, in every dashboard I introduce to you, you have the capacity to activate a timeline and narrow down your observations to a specific period of time.

Every table, graph, metrics will be adapted to the given time span to present you the information at a specific moment. This will be convenient when you will want to compare the behavior of a service before and after an upgrade, for example.

This feature is really awesome because you won't have to configure a dashboard for each needs. The system provided to you is made for operator, no need to customize a PromQL or Grafana query to analyse what's currently happening in production…

Conclusion

I focused this article mainly on Observability, because Anthos Service Mesh provides it out of the box. Even if Istio has wonderful features for traffic splitting, mirroring, authorization… the first reason you will want to use it for is Observability 🕵️‍♂️.

Google is now doing with Istio what it did with Kubernetes many years ago. It integrated it and simplify its usage to make it available for everyone with ease. The future version, with a Google Managed Control Plane should simplify it even more.

If you want to increase your observability with a managed and preconfigured system, I advise you to test Anthos Service Mesh!

Manage your secrets in Git with SOPS for Kubectl & Kustomize 🔧

Kevin Davin — Sat, 20 Jun 2020 10:55:53 +0000

In previous parts, we see how to manage our secrets in Kubernetes format directly from Git with SOPS. For that we use the standard Kubernetes format, and some SOPS parameters to encrypt only some keys. We will go further here and see how to do that with if we want to use kubectl -k or kustomize.

DISCLAIMER If you want to discover kustomize, I've written an article about it available here.

Simpler solutions

We will use here the solution provided by kubectl / kustomize to generate a secret from a secret generator (see the official documentation about it).
This solution induces two steps, one to decrypt the secret and another to produce the YAML.

📄 With files

SecretGenerator allows us to include secret from files. So, we can use SOPS to decrypt the file, and then we can use the files directive to include them into manifests.

Devon has the following files:

# kustomization.yaml
secretGenerator:
- name: app-secret
  files:
  - secret # sops encrypted file
  - another-secret # sops encrypted file

secret file before encryption:

TlZSNk1sRk9lR3RwTnpnNVdVWkVZUT09

And another-secret file before encryption:

CWNRUt3MPSTX3TizkhX2GVh5pN

After encryption with command sops -i -e secret, the file look like this:

# secret file
{
    "data": "ENC[AES256_GCM,data:FUiwHLcn9X1Js+4w5CxQ4Qh+SF3VC4AHGcxT9W3Taqmy,iv:Jh/QcOpcWelFr8cwpv2VtzJQ8/67aam9peVGImHZQdg=,tag:rzGayLJ1Blcymkk7R/Iq9A==,type:str]",
    "sops": {
        "kms": null,
        "gcp_kms": null,
        "azure_kv": null,
        "lastmodified": "2020-06-20T10:12:05Z",
        "mac": "ENC[AES256_GCM,data:3aHK0NLSTT+XdFi44y4xCAfoXy+1PSc+FVTkBe8EpGCl3HlROoUbdjj/nDJTaAHQBctGg1E2U0pSgY2cIx/tWHR61XCtAZf59CL2pzMNeSRSuwJr1Atqq0ltonk08VWDVQ4DpVWBRvUPl5oxAvZpPmSk0LeuHVGeqXO2cTNa5gs=,iv:zUpUHeXghIs+/9hsHQuZtZ5UcgBzC/sF1ccJ6JHzN/U=,tag:rcwzfioUgN6H4wLcZtcrQw==,type:str]",
        "pgp": [
            {
                "created_at": "2020-06-20T10:12:04Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA4Hzarga0atVAQv9EQ4B8KqC41L1h4N+1jpApv/wHeDUMzsp1q5VisRndjHU\ncqmE0YtaJsPbIAMt1cdGZr+koZG5PH25Q6bby4lph3zMFQHZhyrKnGzMftsbCE92\nDPK64vmGQ1QLpJ/3897acu1NtKJicigi14Dr18ujv9kDG4HV1EeqI8o0ylycpUDr\nVVctbUP7WNKd23ShXTymVOJjjNgH4fZxoCHXnf5ndEaVKGcM6wLkPO5VtZBI0D1N\ncyy+UxWV3fZRXqjWx/qFwVGfs9wayRSy16WfSlFCNFzM4bztAxb+bLzNPWVQR4Q3\nSf7eDPVbPHF2VWvmNsODVkg8kr+flf0yKD2T1BOZ9uc+fQiZW6FQYY7qP9fSgPjk\n8Bto7rtqbcCqyQE7i7E7Xuw6hVjw8dB3nIUnr5WcLD7uuHOFTfk1YNPFV+DLdT4L\nch4bFkYLtlAz1EGavZuuGwe6vod/anS3BQyp1dNc34Z4Xoc2fZ7G3oeBCXzVlFK4\nbYLba8pAfYfVQOoMKepp0l4BtV7JTcdNWB+hCCRfPy1nAmWS8SJYvZqFhPzL9VrY\nL1kPWZ74Ooz6H1srlnNHR4YmxUHqtkHnkDv2GADZxvq0JPpKSv0fpHUi2cCnPO6d\n7VQQaUr53Gl2blMJJPBg\n=MdW8\n-----END PGP MESSAGE-----\n",
                "fp": "5844C613B763F4374BAB2D2FC735658AB38BF93A"
            },
            {
                "created_at": "2020-06-20T10:12:04Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA+itJvd1gkukAQv7BzY7dpzpafy1CPpyEaUyZLHkr32KPWypU+x5Mgs1BXFo\n7+Zo92bmnidK6SAgMTJFJZlZP/81AM9s9VYXkbTmpS6OaKgloPsCe4/TQYA7ktAa\nc9mRAESQSt1gDX0xupfWqdmS8yUqnAM6bULNvRkwFgV8Fjz9014sE9yM5DZckLfY\n9FxinBvEfnB2kg/8f82hu7/JIqbvc5uweUCfRglqORHx9L6wrx0suGnBeyWacNB8\nF3Dw4ICMxOBLrvohomqjGQqX7uzmYF8akZVl1vqtHz6vLRu818NIpap9xuyPMzpJ\niXiC+NthfVCAAlKOtUxWMSC+ptZu2JtPaU+WALEwLhAGj26UVsghn1w3u2cEtNwi\n09odd63jt6vXyT7KXPPyJN5stSuHZnRveN5pmXXY/+ZE4VNQXxBW8XzEW1jXJNWH\n8153wy1ObJUG5vOftt8L/NjpYEOBh9TRK+W+DIjGXCcunAHJ2MYH4JRQCvZvinZK\nsw27mdhGe1MTyfPUsLvp0l4BxIvJoO/ac/JcsWwUZBhNDynNzOObL++E0TMoJUmE\n69Gn3UmPwNcFBVHjZRBZh2IpYL5J0Hp+SBxQtKf/XI5AuLQ3jL504teiXt1YuWf4\n9JiEUVhwnGheaWIIWHCc\n=cDXu\n-----END PGP MESSAGE-----\n",
                "fp": "AE0D6FD0242FF896BE1E376B62E1E77388753B8E"
            },
            {
                "created_at": "2020-06-20T10:12:04Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA1ZWtaFCHvLMAQv+NkbxKcDPgNdtTs/n6OaXiFpCgCPRsjuXOmCFaZou33Od\ntNCPTHeAiMseq+Hxl3c0cdLh8wj1C5Sgqc6OBGaTOki3ZKfWA7H6DyPlbFR2d+af\nTZRQc6j7uGISJkPRVGQ5X5rJ1BcoT19rhUQm4vr3x3LHHbLvRXccKugiI7fxpDo4\nrVl7Dc9jhwz8XlV1WdalBeXvupvgxV+6aMqsnt6TFKc0vS/ECSRNoGD/cp2ZnEb8\nNRgIQXH7jv6cYSl1is7VIHvRIxyCukRznlQc6Bhcu/nDI2+vPJEIdiIX8qgG/AF7\nlfFPg8S6FALWcjNmCZksInIGltAIaQir0D2RxPfH7JdYu34c0TwgZCjMow1fBMiS\n82sa9cTWQStGh6lJALTlGWTmZeDtbL22+NX0tG0gtVGlnAmJqZbDCWunL61qrt1n\n1vvAPiYM/KZJOUibn5/gVyyb8PuVyY5Y0GahiTPPXL5h2gFo11xYpHsA9Ir0r9iB\nB2/9gyR8dqaI6xoLWJnc0l4BDdKoO+4+hhkkSneEC5FG6p0u0a2E+bjkfNKGuPas\nEnlIaWEYPh4Ndz2Km/GbII0kOuXvxdt9fxBTmB2k7/weDusUhNYY00sNxLd6rGW4\niQeX7+n5ajUO1efzqmXM\n=B2bM\n-----END PGP MESSAGE-----\n",
                "fp": "3FF9E6938905023D25AB56EEFFEFFE9451381735"
            },
            {
                "created_at": "2020-06-20T10:12:04Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA5/a4uV8wP1EAQv/YmHFgZNe/FgrL9etsny96gztis5qnIDCus+GdZam6j+K\nmVJ21tt5lqqPi3eXm/1OyEQKBgYjQzjDvLNFJRpQMyxogAgD062jKOOHooZWsGaA\nuhQ8Pz05pzMAdmCS2YBQmKKBHbzcue8g0iiJCKd/LAr5Ude8VTAa+Z/Re5fbZAK7\nY7UX/uz7dg0msAF8PyYxovZG907Dsin0FC/ObAcyL5GW24YOWeEHp5VCwpnyWa6v\nVr3uopiWn0n9J1sqhlb9ESq/M5qz8tqBLh1Z0yw8en6Bite+kEiDNW1o16XH4zKA\nB+SN0R/jS5uutijrneAUOQulYN148W9md1rvbL5U9VT5pJXEKPj55rSogvQkv6tK\nPH1m4WiEu7GeujO4oTD2gL5k17c5VzSLYDg3QmkYsJqLh0Z6/DvYELTKo9icMhrb\nZUj2DhTB9RtctJAbfGhZHeRp+V2Lt6at3c/bt7lINAqkEAEUqymrRu9CZwhdDcM9\nzDoanJrDqAInhUyYo7sq0l4Bbd0eyO2FNqsHTlwASkSNgP3bFdSGL5u2t4WlK+gx\nyQd/AZO8C1dcd5VWLi90j6gYe/8zrxrfnnzY1E0CvYRL21YxI2lUeS7+7EAnGmXz\nvw3ZXpnrNX44jC4RZ8FX\n=NTXR\n-----END PGP MESSAGE-----\n",
                "fp": "57E6DA39E907744429FB07871141FE9F63986243"
            }
        ],
        "unencrypted_suffix": "_unencrypted",
        "version": "3.5.0"
    }
}

# another-secret file
{
    "data": "ENC[AES256_GCM,data:5MqAdBMFuvXolRkAiiRZzEEaOt6Tm7HU1aOd,iv:7Z7DGCzH9pQJfuMKs9/BJWsHNfK2RX+xg4/GZgQXWxg=,tag:uXwTc+2fzXmixc27RuhhCQ==,type:str]",
    "sops": {
        "kms": null,
        "gcp_kms": null,
        "azure_kv": null,
        "lastmodified": "2020-06-20T10:12:08Z",
        "mac": "ENC[AES256_GCM,data:HMjZeBiuiN5oW6Z56Sd4O1+1stR7EyO1cNfAmlEUL8XSR97vSJ8mD+It0yZf7Y2/qcTBi1TQS+0Ic6YRL5RycBbRXzCwdVdAUsrQ0netosv1Zcz+jF+sRwzbtJY72hVA16XnAB/vGOBqdAOzzYt60yMI2Hw56AYvK2tnYihGJdk=,iv:bPdVIcUaNifmavFmcHroVU24dj6gqz3jLZHUIKupt5o=,tag:XucX0NIdbXxN3mNXE08Q8Q==,type:str]",
        "pgp": [
            {
                "created_at": "2020-06-20T10:12:07Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA4Hzarga0atVAQwAmfkQySL6NV4gJL57RKTgxsJA9VYCZeww4vhZcCCNy1pN\nzJF97UZSjrO9Pn5LZKgAHi4Ao0RANCH5S2+Se8t3t43SktJdN+TEBIG9gWa3WJ4W\nA0yyR1Xx/bD81ogYmU1CNBxbxF4QO7wZcPtBwhN7whDDLsSvd1ETOR0ZifhhGXDw\nWMieRpf3VannmCu8BugQBqYILyHCwkL/PFzZhVjBT08Vvy9EJwd0oSOKQ4LajDli\nhJkiWwyzoyLmwqZDxSOhOhjF3FHc3cowAwrNDnXs0W1k+NZE+97M96Bkd4JcRvdX\n6K7zD9oKQMT8jUToD6qyzyFGdGOnqh9odTInRDhcWGahbp8vkBsGPWrLFVxboRUy\n1yT92AbWjfLsmhkPQOfpHZcy5zqg8/lrnuYtuw7i5v3OEGjEOoJA5ddjB16roX3I\nJbyHCgrO+60dbDLFKDyYyt9kNBwo7tsFwVdoCtLdvs1/y4jwvRGR9udLaapZV/3H\nzOwFzDpLDblM/eMjIB4G0l4B0dFU0EpjTwX6XDOjf2NVDJYhIu9hwBbqQiTQ2Fhe\nlfVdSLMBwsgTmYmZsTz6Vm4QdwOY9gfNhZm6wrutJ7ZqKb0mZCYi5qtZIa96C2cG\n/pRUsrtTnp8gZao/XjAR\n=Lj0c\n-----END PGP MESSAGE-----\n",
                "fp": "5844C613B763F4374BAB2D2FC735658AB38BF93A"
            },
            {
                "created_at": "2020-06-20T10:12:07Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA+itJvd1gkukAQwAhdEWK1TK+FsnqQAsD5KVCYNxySWfDUqoRgD5QWI6Axnm\nTXo9d5g5qUsz6NrSjMIMQEZ4injIGj0aezxrrAQ+A5h2KnNNc76/Mhy1HdvrlnRz\nqhP0pfwyB0MFyNEoD2nJoGONzg69Ob2mMBULsVxja4p+nl7V9xcNe8o7bgfbdobJ\nYtkgNEJxfahMamQecED8/r7rUhbxxuc3yYx4ijU96dcjpnPl2HpXiD9kUVImMRuQ\nwGdV1GrUYBYFb5U/PgD24Vz2V1D5hIGifbG5MLa0ddOkiKAlz6IXj8HKbR6nNklK\nTGhs9FunpGuwgowef3BMwrKmQocvsHRgrv6Tu/sndwh7ThnV2tH0yoYOz3vaM57+\n1VDGMZSe9QHJjsgaU36lgJBgq4SfzYeWs8zE08QzjZe07iAg6YKms2M+giZ0FDNx\nsHwtCf6p432NTYokYd1NHS09c8s+fyxr271muNC5Q10+NltVJNRYuqfNPdZ9A9Aw\nUBvdyLs9X6fSs+0yPCyq0l4BbF3pL5KfR+lRXkwZmqFr5yUfkaykCpF1sIGwHbfj\n5VAnzcwTgcpSFdc9rqVq4E+HH4V5DZYbXTvqQ9vJGOZeqwUdhYF5NpRTDntzbqoe\n2PolvaOWzvbCGu+DP4Ga\n=6Tfo\n-----END PGP MESSAGE-----\n",
                "fp": "AE0D6FD0242FF896BE1E376B62E1E77388753B8E"
            },
            {
                "created_at": "2020-06-20T10:12:07Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA1ZWtaFCHvLMAQv+Ndwgiv2ch5F/3FcIEtzqlxoq4yuy4L/xzD9Brhoqfgqx\nZPQsrSIez93SlR7JBBGmSaPZiCeUtNuQapi70f25LugGb3HHNVs6Brss05AabTUJ\nWMNjlMBl7uk03bGtx83xNoJAnI8BGzkmrE5JlY3bYE+fTfzxY3CP8L0Tf/7lzbKy\n9uTF3Jf3sWWX0bvSJY7KKuELImTJynfBj8z29DKl6Tmt15LbwVGpqSknhsuZBMBx\nEI9JmwfPnA233jAz+f6QCDodx4GDv9+Ak5CJL9NleMBGJdG6VONrKRe8a8yNCI1k\nVyqgx9oWzpEZK9Go4eBIrbv9GpnyRDo7ULX6luqAeys0co0Cp8N/f8T20K24vEm+\nGhi1dIwcIj3gyFZZ+o7OkI/355CAdmYAldyEkBycZMV+oC7r3zyLniLRrTEzrtwt\nQZCdu/lejBoUK/fWY++L8slZjTL8jVX9lVMHKQZqmC4LMsXtoPv9uS6xlA9I8usp\nOzhrdGCiapQVfPAvZ3Fa0l4BHpAWInZcfWjYXCWfOfis+ygb/YV4I9BLRFTzRiRR\nEpug1dNDEDX3uYO5BsQBt8qMPMsH7T+XI21081oLIH95UD5iOSiYsu600whQV7XO\ng6p/T5LaVdoreMBDcyy/\n=DSoD\n-----END PGP MESSAGE-----\n",
                "fp": "3FF9E6938905023D25AB56EEFFEFFE9451381735"
            },
            {
                "created_at": "2020-06-20T10:12:07Z",
                "enc": "-----BEGIN PGP MESSAGE-----\n\nhQGMA5/a4uV8wP1EAQv/Z3Z2n4inarsPkhSQllWWG7+O7nDwdOK7c6S01f/eGBQ9\nbzwhsZmyuYQ9ReKHRpxhbLjYnQBgc8JqDHCDgznZ8aIFp4XQ3zZeZwBzWYX2D+Ok\ndXt2Ht1yd1zkhMGgrj55yjojnFZRRuNclXFZtypkVFzlU7trrBknlCjQDGIDqOS4\nZXUQ3IHeDOW7F0+wPxz96i4ZfxcZcOtRhsGGqxrMPAXRZGwgRwdOSJNC1a5iwEsc\nk9DPcluf9+1Px/eOW2lmExntqkscNCgkjgT4ACsUCmI4p1V/LUrwvkPDPCH0EE4h\n9MuBKQNuqp+gjjPwR9NOLOtoa/Njsp2V0vZsQ39Y+nfkqtE9w1bAX/R1VDABU+Bm\nml0WjT0eqGQwnZsBY4pfEvRwIS4JVQCa8sGhZXS4iax0uXgEFh4jma2uWBWKBnGd\nD9mJ53RC6Xb4eTSkvNtF2VBFtqdu0N587xhykrprxVf7q/s2RXu3haP+JmdJHfMW\nx2utciyuNIWAGm13u3Cn0l4BK5z9hlBxZfOeGAIyQIWQjYn/ZDG8smHmo8jGUPIR\nki/bs+uzo5Fnyk7OJhr0+bkQjG6weVoZJIUxZr61Tb0JhV4+xXaPVMZoZpg96Bii\n+VWaRWaE+nTfH4wrXwDT\n=svPU\n-----END PGP MESSAGE-----\n",
                "fp": "57E6DA39E907744429FB07871141FE9F63986243"
            }
        ],
        "unencrypted_suffix": "_unencrypted",
        "version": "3.5.0"
    }
}

After decryption of the secret and execution of the command kubectl apply -k . --dry-run -o yaml or kustomize build ., we have the following YAML

# Generated manifest
apiVersion: v1
data:
  another-secret: Q1dOUlV0M01QU1RYM1RpemtoWDJHVmg1cE4K
  secret: VGxaU05rMXNSazlsUjNSd1RucG5OVmRWV2tWWlVUMDkK
kind: Secret
metadata:
  name: app-secret-8m97554t4c
  namespace: default
type: Opaque

You see here, the secret and another-secret keys used in the file kustomization.yaml ($.secretGenerator[*].files[*]) are both filenames on the file-system, and keys in the generated manifests ($.data[*]). This could lead to problems if key name can't be a file (unauthorized characters for example).

🗂 With `.env` files

A feature available into kustomize but not yet in kubectl (see issue) is the possibility to use .env file as secret source.

secretGenerator:
- name: app-secret
  envs:
  - secrets.env

The secrets.env before encryption:

secret=TlZSNk1sRk9lR3RwTnpnNVdVWkVZUT09
another-secret=CWNRUt3MPSTX3TizkhX2GVh5pN

After encryption with command sops -i -e secrets.env:

secret=ENC[AES256_GCM,data:LZxAINIj7/HOIqSJ/M1YTjGmjGIFk24KKEKyXxcFHvQ=,iv:OJNKetoqlJdmYKFpFsfr1ihQjpZ6nJfm8INW474EfE8=,tag:Ea8G+IGH+aORM0sePL9Njw==,type:str]
another-secret=ENC[AES256_GCM,data:OFgAiBkTzEjPwxpo0qn17wvSXm3T164f1Wg=,iv:+kvczR+vGDl6W/UG8KOl++cCKDmx3zA4cGC2vIsge7w=,tag:n7Qbs/I/qo4uy4FfCOMQkg==,type:str]
sops_pgp__list_3__map_fp=57E6DA39E907744429FB07871141FE9F63986243
sops_pgp__list_2__map_enc=-----BEGIN PGP MESSAGE-----\n\nhQGMA1ZWtaFCHvLMAQv/S/gV3liTwmA8Rm3v08kjV39mfBF66mSbjgljoVMvlgoQ\nZ6l2djvzUfzTpgdfACwVVbzQ+KWYfYWnHtOlEHKGQ1abi4Ik/WRS9J6xfzf0aPj0\n9GAy43wCqblvKLKBPHzfYXsQQLP/IA3gHgj1T0hRMx0Nbv+bHi16DEilzxyhCQQf\nzV+w0IMfF1CJrTvTgn4ohaN1KvuQcqOJNlWyr0/fIL420QYL09rQvjA20ds8Jwaq\nZ61O6XOtjyCVAB1nAYfCIOtd0t/+ApnI6Qwe5FKKcN1GrCiqJ4rt4mCK9/nLRcqg\nF1h825h5yuAWYjIyIb8zbulfLxUUN8ztbn4myIUKTcXu+Onq6dq2jaCKOceVc90m\njpXwvcDKYRdeG1HUcmOeQu9ioyt+4xGxOkWY8/hg1YJp9lA+/n4uU5OE9Su0o8LP\n71EzXfISx2PeROkY2W83e8qhzgQQNsCOI9Y5k8Tmd8ctgI4i9Pw0j1S2+sdiD5Y9\njOW7hPVLNiQ/avAkfKVr0lwBZSwkvC3d0Rbw4Wskj8/meRNMYf8DsbFOYi05ysjO\nwTvWQFYPy4cvMtaIvMKn05f+dQIm/6jwOwu6tyytKxqYuq/vEPOrUMQU72wA25eS\nTyXEd7AjE4boT/hQ9g==\n=gUpS\n-----END PGP MESSAGE-----\n
sops_pgp__list_3__map_enc=-----BEGIN PGP MESSAGE-----\n\nhQGMA5/a4uV8wP1EAQv8DQgTw5K4u3E+J9rRxiNK3AvdM+ajifrhZ/NEXu1hgj6g\nMhp8eqQoG1P7/1OJAYL1B20fTatol6oHXgpE7/+f8W6pY+wLfo4d6JSL+KSAAxJk\nWizkR87JxWZFyQB109vgZRDagPylA02yq/2XSgI47JFGVPB14/MAxJkr3O47Si3D\nc3NkqUew8jwPNAToLxM/oCnnDuEoSKmB6smSiL5UHk2/04h3PqbcwUnEPheGR9LK\neA6LVhQ/AiEtbZqAeSU46KGLVHFvQdQVKO+sAHkia21Y041tfS1HDiijTk3UxYTV\nSIB5OKln0qVdDZPYDg4y6Pc/qvj+DHUE+gDoSFOOaGPl1BMw5m9AoY/UQ+Tip66k\nBpG54b20GNmlCuI+7N2QG/lpoML4CGDP2AJ40o6KbZrlo4iwGieg9gdSAstJKlvg\nPpi/p/Fo2zyWin1Gjf/T3QYk5PjbeDkqDqyehToI90qDv8KF7ZDf/t3bz9Geitef\nt9Vg74WCXK/yNcyGAGbC0lwBaqxUf2erAGoTDBkmjjlyEGiEsIgpZzhNj85/G1Mp\n1NxM7zM+5nJGEquzXe/FEoOVX4+LRm1W8+y0T2FRxiOUhlE3anWs+576BYZ+1zZl\nourCUh/MJPpDz3R6Cg==\n=D5A/\n-----END PGP MESSAGE-----\n
sops_pgp__list_1__map_enc=-----BEGIN PGP MESSAGE-----\n\nhQGMA+itJvd1gkukAQv+NZtoRyHQfmOEA2zLM3HaK2HDofMNqcA8Gy66sOfMRwM0\nDvnyu7gcurDE75hYiY4Dd/k5wy+HE7c6fITIWZ62nGAU33RQKgWzZ/vuZX2JnzcA\n8d2XsyoBLe3KGOB75tfepYVkPuRdzDqCY+IH6bHvr9MtNW/CGL58OBXfGTKs1NnI\nSpBWPhF/8Zj0l9S5QkA5ENYGM2u/yutav1z281PNLmZmTrmZ9VYnytwwPX2dY9LS\nupyJnywKqwuw1iaFh7f8BhbuqTjNzAkmjycL8ZFLQB1uGI5CwxCGZ9a3KByNJ6Eh\nH/KVrK5rBBP09ByoIAYoiSBraejoovNFns9O2oUin8HEjv4tziuZ3KeqR51uJCS0\n31nZ/JBlx7GyJ7WGKoVsthlrOebpUuDRbmcKhzhNZT7umTCOYCHdTvGH5p6nMvsE\ngcuuqiTJjrp1WECkr1mMhQqD1Ef708Cw9TAK1LbzdHz5ePBWZ9b+FP0PpBUBHZOc\npWlzd8nYATEK7kQsRbUw0lwB5XQZZKltYrzy1n0dmI98HpqWUpx/f0BX9WRQMkgz\nmbZ8iEQIOBSexNvcM3yA9RMVaTW6M1WoHPGoQdiz4kw4gWtaKqxkK/RlBa6YPnwG\ncrToyqNOO5+HxrSIJg==\n=3esV\n-----END PGP MESSAGE-----\n
sops_pgp__list_1__map_fp=AE0D6FD0242FF896BE1E376B62E1E77388753B8E
sops_pgp__list_2__map_created_at=2020-06-20T10:30:18Z
sops_pgp__list_0__map_created_at=2020-06-20T10:30:18Z
sops_pgp__list_0__map_fp=5844C613B763F4374BAB2D2FC735658AB38BF93A
sops_unencrypted_suffix=_unencrypted
sops_lastmodified=2020-06-20T10:30:19Z
sops_pgp__list_3__map_created_at=2020-06-20T10:30:18Z
sops_pgp__list_2__map_fp=3FF9E6938905023D25AB56EEFFEFFE9451381735
sops_version=3.5.0
sops_pgp__list_0__map_enc=-----BEGIN PGP MESSAGE-----\n\nhQGMA4Hzarga0atVAQv/cIT+IM2097+hQ5NWuod9D7WamqfLTkYeCA+eQqMWRy7l\nBeUGX+JNrh8Wzmq3jAT4NP61/WZiPdX/sqmw/VKd7f+RoLoT5vvT/omHqXgt07Qv\nG0Sp/lz5KZF9d5ujiEmvOTbHZgFhA/8CzLl6rfWLzyHocdV5585vcnRK6RUa8Ezh\nUG9+nLCecq5xIAiqUfWpwdt+bznHAgb9VMW8m8T3UFDnUDa0Y/ankLJ9wIaReoVB\nTCRpJnr5fKxUw4r+kJDFxhnSg21Iw5Rx4rGJHAlDmazXGHazXD5z7f9g3ZiHxrOQ\n6WpoxZw/X/d3697viqe/yemp+CDQJUWQBolhAqfrMN95KllNo7vb7dOpNYG4PFIK\nRLNSTgg+2Ph4mE7AQHTqCdV1jPhtONApcf7CQmkQG/KqyJ5YCfwDLzfo+7+pv1Xx\nhSYEGwMVX+jBIW7ZlLsqcYUaJNQeF3DYNrfgdsTiMB7aGdhohOJvpL/771t37P2s\nyJ/rgu/K8J1InTcEfRPs0lwBeXjohJIpTOL09QgXNSQ0Dsio19VhPjWTr0QI6cAP\nYedVlwrAebeQAOkBlC8RPSUG35CYUqnSRpGHKrl03xCP0Z4TXZ+v60XphehDpdOn\nROm168cPbVF+K2BTTQ==\n=Pdik\n-----END PGP MESSAGE-----\n
sops_mac=ENC[AES256_GCM,data:kXxkiJmgK5nyDiEwStytEHwKLw5JIN1Y1xiNUAXKJLNGEk4VyZ+VkqMNBcUED8HQycHIRle550/Mtov4aVtviM9jM7MnTMIdtg/bp6f0AKiXcl5nCWt26b3JP2nvLN/COMUnvxHxW3uwhdnrsJupBMNDSVdDOv2wXbyazfuor+U=,iv:LR2dRc054UDQoWCVJ+bynri5j5zFKvE69ggLh19sXLY=,tag:nvYpc7ez5Hb4jYLNUnd9Fw==,type:str]
sops_pgp__list_1__map_created_at=2020-06-20T10:30:18Z

After decryption of the secrets.env and execution of the command kustomize build ., we have the following YAML

apiVersion: v1
data:
  another-secret: Q1dOUlV0M01QU1RYM1RpemtoWDJHVmg1cE4=
  secret: VGxaU05rMXNSazlsUjNSd1RucG5OVmRWV2tWWlVUMDk=
kind: Secret
metadata:
  name: app-secret-4g57fkmb8b
type: Opaque

Both way can be used without any problem (files or envs). This depends of the restriction on tooling you have to generate/deploy your manifests.

🔧 Alternative with Kustomize Plugins

The main goal about using the kustomize plugin system is to be able to remove the step of files decryption before manifests creation. Thanks to that, you will be able to use SOPS secrets with some tools like skaffold and or others usingkustomize under the hood.

kustomize has two plugins system (official plugin documentation)

native Go extension
exec shell extension

This native solution is very useful because all can be done directly from kustomize code, and we don't need to have SOPS installed (for local or CI execution).

But this solution is complex and requires compiling kustomize & plugins from sources 😓... with all the Go build system. There is some SOPS plugins available:

The shell solution is simpler to manage, but it will require installing SOPS or other binary to perform decrypt operation. I found only one implementation of shell plugin for kustomize available: kustomize-sopssecretgenerator

Both solutions have another downside, it requires extracting secrets configuration outside the kustomization.yaml (in a generator.yaml file), so it multiplies files and make it harder to read / manage 😓. Depending on your needs, I advise you to use KSOPS or kustomize-sopssecretgenerator.

🔐 Conclusion

We have seen here a “simpler” solution to manage our secret and to use them with Kubernetes relying on the builtin Kustomize system. The plugin ecosystem is still young, so I hope integrations will be simpler in futur.

You can find the source code of this article, files and scripts in this GitLab repository.

Manage your secrets in Git with SOPS for Kubernetes ☸️

Kevin Davin — Sat, 13 Jun 2020 13:25:56 +0000

In previous parts, we see how to manage our secrets in Git and usable by humans and CI. Right now, we will discover how
to integrate it with Kubernetes, and especially with Kubernetes Secrets.

😊 Naive usage

Since the beginning, Alice, Bobby and Devon are using only *.env file to store secrets. They have to use them
in a Kubernetes context, to create a secret named app-secret. The simple (and naive way) to do this is to use
the kubectl application to create the secret from an env files:

This solution is pretty basic and rely on the capacity of the kubectl cli to create secret from an *.env file. The main problem here is the fact we have to generate our secret from the env file. In Kubernetes context, when we have multiple manifests, we like to have the same semantic and organisation, which is simpler to read and debug.

🔒 Sops & YAML

To match the team expectation, Devon will move to a Kubernetes YAML format for secrets instead of relying on *.env files. To do so, he will use the output generated from the previous example and use sops to encrypt it.

The SOPS output is like this:

apiVersion: ENC[AES256_GCM,data:gEA=,iv:2jeZ0y0SEPXZTgrsmYXc7LgPydoPwnKzF4GVqykSw5M=,tag:YzEMrctU19sUIqat+LUFIw==,type:str]
data:
    secret: ENC[AES256_GCM,data:uJIDP3yMdFyW/7bnBAU0MJp07xmF0nlh,iv:ardoVW+p9HLNpaWfXOWrevZFdNJqGJvt00jPbkrr7p4=,tag:mB3wqa7kNE7gZ9J87IboNg==,type:str]
kind: ENC[AES256_GCM,data:tu+OHbXI,iv:QfqZeYOkTPTzJ618gh+/zGWPMDBJfjp+GwM8yBZCD+Y=,tag:XrJfNhmyDhOQNncd/uNvxA==,type:str]
metadata:
    name: ENC[AES256_GCM,data:X8WMTByIcUrjag==,iv:pQ6xt5DPchrxwXtt98l4mgCixpcpaA2ewST0lAqYY4E=,tag:fqj/1PbzbhjIhRx9Ogitxg==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2020-06-13T12:21:16Z'
    mac: ENC[AES256_GCM,data:dU1bm2GBwdkqMQqEa+b0EfuMG5BRiiHDxNjB3tvB4JTVxqamQ17A47tzMld93/lZnJWRb4BtDFUI2xHh/5BCpxdI4J67AvWdv91usRgpgh7z7SF4pK4UUVah3WDYd3tvo+VzugvN6b4V+oGZlfJM789dJrTDOjSCAyyaaa/Qfb0=,iv:Cdaet+2Sowq50TDIkDO1RXi4vgaVqe7rYz08PHybAv0=,tag:1aeR/f4NnydiP2Vuuvljvw==,type:str]
    pgp:
    -   created_at: '2020-06-13T12:21:16Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQGMA5/a4uV8wP1EAQwAshoGsWCKiYnsi1Rr/HoMtvp8Tx5cmUWLPz0OsGsQUFc/
            HQ48h7KUAjemTBnyQ6SpZUK2uI5YOLnOIqNcQJvNb4o30o0M3kuD2KNzSbsRGt3j
            vODVKcs699sT1mX/rc6/EzpLy0NUgrAlXn7IUG+rhnk97OvfMdjP6+i77OpSU9Uc
            5l6FWokBzFBvL6EENay6EO54C17MBXijoM4h6x7YajOjk5VNFtUl7wh039VGWZpQ
            7vMuTbla06DWAHIFKIOs54yEh+vMTMgnsg8NrPzz4xqKDMcmZoQIm1jwc8EvxYK4
            nGI4LpXO8VMgCyHBEcMjCGhZ5xXAiMRzF9M8XAWRfDFgYjDosL8QOQkl4vVnsf4e
            5bhITFz1ZGaJwEeVzlt82cxWy4+BNq+lfAghqEnHGAjLGFIgTiRtVB47NLk9WPIT
            T7Qv5gvATdW0rNHecfoGCKWx15ZJezXSo8Vtt5eVPgdYd3wO76Uw6XWhkxwSgQNT
            vE7W5qvLoqxZrJs4Kyd20l4BQzSEiHJRg7wmWx9Y0BhyzF1Zh6F0oQBxTzGjP0f1
            J9THfwIOhyOBwm8bWkEz7ac84o2NEItdrvlhrB0Y61ZfRl6xZKWqFxJ3N1yVHxxM
            MWRYzaQcuoh5670lRhY5
            =KjAd
            -----END PGP MESSAGE-----
        fp: 57E6DA39E907744429FB07871141FE9F63986243
    unencrypted_suffix: _unencrypted
    version: 3.5.0

SOPS is encrypting every value of the previously generated YAML, including kind or apiVersion and metadata.name. This is problematic because the whole team need to be able to read keys, which are not sensible in this Kubernetes secret.

🔏 Encrypt specific keys only

SOPS provides an handy parameter to choose which keys should be encrypted. This could be used directly from the command line sops --encrypt --encrypted-regex '^(data|stringData)$' app-secret.yaml.

DISCLAIMER: For concision, I've removed a part of the generated files. SOPS parameters are included into the YAML and are verbose.

The file looks like this now:

apiVersion: v1
data:
    secret: ENC[AES256_GCM,data:RLfYML8bJ0d5kN0nudgGgJJQIJMHK9MB,iv:6rurG5rErwn2O7PrHVoPjRnkMT5MjenhRmF3Vxh/cVw=,tag:4B5NeHbTmm0yl5VFlA9fww==,type:str]
kind: Secret
metadata:
    name: app-secret
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2020-06-13T12:32:40Z'
    mac: ENC[AES256_GCM,data:jl00u8YtegEi3lvuHv70PtI6Zywo8tSDH2DKk2jw+b4yiUGU2y9T82t3bE6ahY1MWy1U6CRK9NYKBS8p5HJqDjH/N21rhQy7k3g4BuzG/CJt2S/xio0KaAwwGXURUtXbHpXLrko22q0cgqCrsr/IFvz65r6WNhji0utSC1FKCPQ=,iv:XoilcDIZR8ir/ySbu+xVGNO0NkEkQr/IlyKXvAyyeyM=,tag:Hraabep8cq85M3OjvjoQBA==,type:str]
    pgp: ... # Removed for concision
    encrypted_regex: ^(data|stringData)$
    version: 3.5.0

We can do better, because SOPS provides us a solution to store configuration required for every secret directly into the .sops.yaml. We can add the key encrypted-regex to simplify the command line. This is helpful when you need to add another secret key in an existing secret.

Every team member can now read and edit Kubernetes secret simply 🎉.

🦊 CI Integration

The CI needs to be adjusted to be able to deploy those manifests. We are using the base described in the previous article.

deploy int:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
  before_script:
  - *auth
  script: 
  - sops -d app-secret.yaml | kubectl apply -f -

Conclusion

If you are using Kubernetes, and you want to manage your secrets in Git, SOPS will be a good match ❤️. You will be able to encrypt only desired value in a YAML file to keep it readable, by a human and other tools working on Kubernetes manifests.

You can find the source code of this article, files and scripts in this GitLab repository.

Manage your secrets in Git with SOPS & GitLab CI 🦊

Kevin Davin — Sat, 06 Jun 2020 12:50:18 +0000

In this series, we saw how to create and manage our secrets in git and restrict their access to team members.

What if I want do use my secret from my CI for Continuous Deployment? This is exactly what we will see in this article, based on GitLab CI 🦊.

DISCLAIMER: Every CI system should be compatible, you just need to have access to CI secrets and be able to fetch them during execution.

🤖 Creation of the CI identity

To be able to decrypt secrets within Continuous Integration, we should create an identity for our CI. Alice, Bobby and Devon will have a new friend named CI 🤖 !

Devon, in charge of the CI, will create a new key-pair for this new special user. He will use the gpg --full-gen-key command and answer questions like if it was for a real human.

Then, he will extract a backup of the private/public key with the command gpg -o ci.key --armor --export-secret-keys ci@domain.fr. This key is in a textual format, which is easy to use as an environment variable.

NOTE: This key and the passphrase should be stored securely, in a keypass, vault or something else, because they represent the identity of your CI. By design, it will be able to decrypt all secrets from the repository.

We also need to extract the public key separately to distribute it to every team members. Without it, they won't be able to encrypt a secret and include the CI key in the process.

To do so, Devon will use the command gpg -o ci.public.key --armor --export.

🔑 Import public key of CI

Each team member should import the public key extracted by Devon in the previous step. Here, Booby will use the gpg --import ci.public.key command to perform this.

At the end, he, and every other team members should have in their key list (gpg --list-keys) the public key of CI.

🔏 Update keys of every secret

CI is a new team member, as any other human. The team has to let this new "person" access and decrypt every secret. So we will use the sops updatekeys <secret> on every secret.

This requires a correctly configured .sops.yaml file. For more information about this, you can read the second part of this series.

🦊 GitLab CI Configuration

Now, the CI is ready to be configured. Devon, in charge of this, will import into GitLab secret pane (in Settings > CI/CD > Variables) the content of the ci.key created previously and the passphrase defined for it.

NOTE: The variable Type is defined to File for the KEY. For more information about this, look at the official GitLab documentation about file variable

Then, Devon will define a GitLab CI job able to read the value from the encrypted secret. For our example, this will just do a cat on the standard output. Of course, you can do what you want with those values in your CI pipeline.

We need to take care of the image used in our Continuous Integration. We should be able to access the gpg command. In this example, we will install it.

deploy int:
  image: google/cloud-sdk # <1>
  before_script:
  - apt-get update && apt-get install -y curl gnupg # <2>
  - curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops # <3>
  - chmod +x /usr/local/bin/sops
  - cat $KEY | gpg --batch --import # <4> 
  - echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp) # <5>
  script: 
  - sops -d int.encrypted.env > int.env # <6>
  - cat int.env

Define a job image based required for our deploy phase... it should fit team needs, this is just for example here.
We install gpg and curl thanks to apt on debian distribution. Should be adapted if you are using another distribution.
We use curl to download sops binary from GitHub. Take care to get the latest version 😉.
Import the ci.key into our gpg toolchain.
Provide the PASSPHRASE to the gpg toolchain to be totally "not interactive".
Then, the CI can decrypt the int.encrypted.env file and use it where we want in our deploy phase.

Running with gitlab-runner 13.0.0 (c127439c)
   on docker-auto-scale 0277ea0f
Preparing the "docker+machine" executor
 Using Docker executor with image google/cloud-sdk ...
 Pulling docker image google/cloud-sdk ...
 Using docker image sha256:5d096c48b3b4bab72df240dcf94940be3e0397606bb4cc94143f2a12189dba1f for google/cloud-sdk ...
Preparing environment
 Running on runner-0277ea0f-project-19225532-concurrent-0 via runner-0277ea0f-srm-1591443365-4e9ad7c3...
Getting source from Git repository
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes with git depth set to 50...
 Initialized empty Git repository in /builds/davinkevin/sops-blog-post-repository/.git/
 Created fresh repository.
 From https://gitlab.com/davinkevin/sops-blog-post-repository
  * [new ref]         refs/pipelines/153546622 -> refs/pipelines/153546622
  * [new branch]      master                   -> origin/master
 Checking out 84edc518 as master...
 Skipping Git submodules setup
Restoring cache
00:02
Downloading artifacts
00:01
Running before_script and script
 $ apt-get update && apt-get install -y curl gnupg
 Hit:1 http://deb.debian.org/debian buster InRelease
 Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
 Get:3 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
 Hit:4 https://packages.cloud.google.com/apt cloud-sdk-buster InRelease
 Get:5 http://deb.debian.org/debian sid InRelease [146 kB]
 Get:6 http://security.debian.org/debian-security buster/updates/main amd64 Packages [201 kB]
 Get:7 http://deb.debian.org/debian sid/main amd64 Packages.diff/Index [27.9 kB]
 Get:8 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-05-2003.49.pdiff [23.8 kB]
 Get:9 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0208.52.pdiff [16.2 kB]
 Get:10 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0803.34.pdiff [12.5 kB]
 Get:10 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0803.34.pdiff [12.5 kB]
 Fetched 543 kB in 2s (243 kB/s)
 Reading package lists...
 Reading package lists...
 Building dependency tree...
 Reading state information...
 The following packages were automatically installed and are no longer required:
   dh-python libperl5.28 libpython3.7 libpython3.7-minimal libpython3.7-stdlib
   perl-modules-5.28 python3.7-minimal
 Use 'apt autoremove' to remove them.
 The following additional packages will be installed:
   dirmngr gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server
   gpgconf gpgsm gpgv libbrotli1 libcurl4
 Suggested packages:
   dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage
   scdaemon
 The following NEW packages will be installed:
   libbrotli1
 The following packages will be upgraded:
   curl dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client
   gpg-wks-server gpgconf gpgsm gpgv libcurl4
 13 upgraded, 1 newly installed, 0 to remove and 132 not upgraded.
 Need to get 8559 kB of archives.
 After this operation, 1241 kB of additional disk space will be used.
 Get:1 http://deb.debian.org/debian sid/main amd64 gpg-wks-client amd64 2.2.20-1 [507 kB]
 Get:2 http://deb.debian.org/debian sid/main amd64 dirmngr amd64 2.2.20-1 [740 kB]
 Get:3 http://deb.debian.org/debian sid/main amd64 gnupg-utils amd64 2.2.20-1 [889 kB]
 Get:4 http://deb.debian.org/debian sid/main amd64 gpg-wks-server amd64 2.2.20-1 [500 kB]
 Get:5 http://deb.debian.org/debian sid/main amd64 gpg-agent amd64 2.2.20-1 [641 kB]
 Get:6 http://deb.debian.org/debian sid/main amd64 gpg amd64 2.2.20-1 [894 kB]
 Get:7 http://deb.debian.org/debian sid/main amd64 gpgconf amd64 2.2.20-1 [532 kB]
 Get:8 http://deb.debian.org/debian sid/main amd64 gnupg-l10n all 2.2.20-1 [1035 kB]
 Get:9 http://deb.debian.org/debian sid/main amd64 gnupg all 2.2.20-1 [749 kB]
 Get:10 http://deb.debian.org/debian sid/main amd64 gpgsm amd64 2.2.20-1 [627 kB]
 Get:11 http://deb.debian.org/debian sid/main amd64 gpgv amd64 2.2.20-1 [608 kB]
 Get:12 http://deb.debian.org/debian sid/main amd64 libbrotli1 amd64 1.0.7-6.1 [267 kB]
 Get:13 http://deb.debian.org/debian sid/main amd64 curl amd64 7.68.0-1 [249 kB]
 Get:14 http://deb.debian.org/debian sid/main amd64 libcurl4 amd64 7.68.0-1 [321 kB]
 debconf: delaying package configuration, since apt-utils is not installed
 Fetched 8559 kB in 0s (22.3 MB/s)
 (Reading database ... 85863 files and directories currently installed.)
 Preparing to unpack .../00-gpg-wks-client_2.2.20-1_amd64.deb ...
 Unpacking gpg-wks-client (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../01-dirmngr_2.2.20-1_amd64.deb ...
 Unpacking dirmngr (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../02-gnupg-utils_2.2.20-1_amd64.deb ...
 Unpacking gnupg-utils (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../03-gpg-wks-server_2.2.20-1_amd64.deb ...
 Unpacking gpg-wks-server (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../04-gpg-agent_2.2.20-1_amd64.deb ...
 Unpacking gpg-agent (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../05-gpg_2.2.20-1_amd64.deb ...
 Unpacking gpg (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../06-gpgconf_2.2.20-1_amd64.deb ...
 Unpacking gpgconf (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../07-gnupg-l10n_2.2.20-1_all.deb ...
 Unpacking gnupg-l10n (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../08-gnupg_2.2.20-1_all.deb ...
 Unpacking gnupg (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../09-gpgsm_2.2.20-1_amd64.deb ...
 Unpacking gpgsm (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../10-gpgv_2.2.20-1_amd64.deb ...
 Unpacking gpgv (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Setting up gpgv (2.2.20-1) ...
 Selecting previously unselected package libbrotli1:amd64.
 (Reading database ... 85881 files and directories currently installed.)
 Preparing to unpack .../libbrotli1_1.0.7-6.1_amd64.deb ...
 Unpacking libbrotli1:amd64 (1.0.7-6.1) ...
 Preparing to unpack .../curl_7.68.0-1_amd64.deb ...
 Unpacking curl (7.68.0-1) over (7.64.0-4+deb10u1) ...
 Preparing to unpack .../libcurl4_7.68.0-1_amd64.deb ...
 Unpacking libcurl4:amd64 (7.68.0-1) over (7.64.0-4+deb10u1) ...
 Setting up libbrotli1:amd64 (1.0.7-6.1) ...
 Setting up gnupg-l10n (2.2.20-1) ...
 Setting up gpgconf (2.2.20-1) ...
 Setting up libcurl4:amd64 (7.68.0-1) ...
 Setting up curl (7.68.0-1) ...
 Setting up gpg (2.2.20-1) ...
 Setting up gnupg-utils (2.2.20-1) ...
 Setting up gpg-agent (2.2.20-1) ...
 Installing new version of config file /etc/logcheck/ignore.d.server/gpg-agent ...
 Setting up gpgsm (2.2.20-1) ...
 Setting up dirmngr (2.2.20-1) ...
 Setting up gpg-wks-server (2.2.20-1) ...
 Setting up gpg-wks-client (2.2.20-1) ...
 Setting up gnupg (2.2.20-1) ...
 Processing triggers for libc-bin (2.30-8) ...
 $ curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops
 $ chmod +x /usr/local/bin/sops
 $ cat $KEY | gpg --batch --import
 gpg: directory '/root/.gnupg' created
 gpg: keybox '/root/.gnupg/pubring.kbx' created
 gpg: /root/.gnupg/trustdb.gpg: trustdb created
 gpg: key FFEFFE9451381735: public key "continuous-integration <ci@domain.fr>" imported
 gpg: key FFEFFE9451381735: secret key imported
 gpg: Total number processed: 1
 gpg:               imported: 1
 gpg:       secret keys read: 1
 gpg:   secret keys imported: 1
 $ echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
 $ sops -d int.encrypted.env > int.env
 $ cat int.env
 secret=5Tz2QNxki789YFDa
Running after_script
00:01
Saving cache
00:02
Uploading artifacts for successful job
00:01
 Job succeeded

Like we see here (and in the GitLab job view), every time the pipeline is triggered, it will re-install every software, from gpg to sops, which can be time-consuming and source of error.

🦊 GitLab CI Optimization

Devon will create another project just for CI tooling image, and he will prepare the image of the CI to be able to have gpg and sops installed. Here is for example, the Dockerfile:

FROM tutum/curl AS downloader

RUN curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /opt/sops && \
    chmod +x /opt/sops

FROM google/cloud-sdk as final

COPY --from=downloader /opt/sops /usr/local/bin/sops

RUN apt-get update && apt-get install -y gnupg --no-install-recommends

Then, he will activate a schedule to build this image every day and publish it in their own docker registry. Thanks to this, he can improve the previous .gitlab-ci.yml:

deploy int:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops # Image created by the previous Dockerfile 🐳
  before_script:
  - cat $KEY | gpg --batch --import 
  - echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
  script: 
  - sops -d int.encrypted.env > int.env
  - cat int.env

You can see, the job execution is much straightforward now:

Running with gitlab-runner 13.0.0 (c127439c)
   on docker-auto-scale ed2dce3a
Preparing the "docker+machine" executor
 Using Docker executor with image registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
 Authenticating with credentials from job payload (GitLab Registry)
 Pulling docker image registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
 Using docker image sha256:ec7e941ae9f66656c9c0b7b4ce61b229eeb215492c65ebfe21c69026aa166013 for registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
Preparing environment
00:05
 Running on runner-ed2dce3a-project-19225532-concurrent-0 via runner-ed2dce3a-srm-1591446538-6f33846d...
Getting source from Git repository
00:03
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes with git depth set to 50...
 Initialized empty Git repository in /builds/davinkevin/sops-blog-post-repository/.git/
 Created fresh repository.
 From https://gitlab.com/davinkevin/sops-blog-post-repository
  * [new ref]         refs/pipelines/153552640 -> refs/pipelines/153552640
  * [new branch]      master                   -> origin/master
 Checking out 625b7479 as master...
 Skipping Git submodules setup
Restoring cache
00:01
Downloading artifacts
00:02
Running before_script and script
00:04
 Authenticating with credentials from job payload (GitLab Registry)
 $ cat $KEY | gpg --batch --import
 gpg: directory '/root/.gnupg' created
 gpg: keybox '/root/.gnupg/pubring.kbx' created
 gpg: /root/.gnupg/trustdb.gpg: trustdb created
 gpg: key FFEFFE9451381735: public key "continuous-integration <ci@domain.fr>" imported
 gpg: key FFEFFE9451381735: secret key imported
 gpg: Total number processed: 1
 gpg:               imported: 1
 gpg:       secret keys read: 1
 gpg:   secret keys imported: 1
 $ echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
 $ sops -d int.encrypted.env > int.env
 $ cat int.env
 secret=5Tz2QNxki789YFDa
Running after_script
00:02
Saving cache
00:01
Uploading artifacts for successful job
00:02
 Job succeeded

NOTE: You can use this method with any docker registry, not only the one from GitLab 🦊.

Of course, the before_script should be copy/paste for every job requiring access to secrets. Devon will refactor this to optimize the Don't Repeat Yourself aspect:

.auth: &auth | # <1>
  function gpg_auth() {
    cat $KEY | gpg --batch --import > /dev/null
    echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp) > /dev/null
  }

  gpg_auth

deploy alice:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
  before_script:
  - *auth # <2>
  script: 
  - sops -d dev_a.encrypted.env > dev_a.env
  - cat dev_a.env

deploy int:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
  before_script:
  - *auth # <3>
  script: 
  - sops -d int.encrypted.env > int.env
  - cat int.env

A YAML anchor used to set a custom shell function and calling it right away.
Usage of the anchor as a before_script step in the job.
Re-use of the auth anchor in another job.

You can see deploy alice and deploy int logs in the GitLab UI.

Of course, this is just an extract of the real .gitlab-ci.yaml, which includes linting, test, build & cie 👍.

Conclusion

Here, we learn how to configure our CI identity and configure the GitLab CI 🦊 to be able to decrypt secrets. We also see how to optimize it in a GitLab environment leveraging usage of home made docker image.

You can find the source code of this article, files and scripts in this GitLab repository. The CI is triggered in this GitLab repository

Kustomize - The right way to do templating in Kubernetes

Kevin Davin — Thu, 04 Jun 2020 05:56:02 +0000

Note: this article was originally published on Stack Labs blog.

We always need to customize our deployment with Kubernetes and, I don't know why but the main tool around for now is HELM which throws away all the logic we learn on docker and Kubernetes.
Here I will introduce to you an alternative called Kustomize ❤️

Kustomize isn't a new tool, it is under construction since 2017 and has been introduced as a native kubectl sub-command in the version 1.14. Yeah, you've heard correctly, this is now embedded directly inside the tool you use everyday... so you will be able to throw that helm command away 😉.

Philosophy

Kustomize tries to follow the philosophy you are using in your everyday job when using Git as VCS, creating Docker images or declaring your resources inside Kubernetes.

So, first of all, Kustomize is like Kubernetes, it is totally declarative !
You say what you want and the system provides it to you. You don't have to follow the imperative way and describe how you want it to build the thing.

Secondly, it works like Docker. You have many layers and each of those is modifying the previous ones.
Thanks to that, you can constantly write things above others without adding complexity inside your configuration.
The result of the build will be the addition of the base and the different layers you applied over it.

Lastly, like Git, you can use a remote base as the start of your work and add some customization on it.

Installation

Of course, for 🍎 Mac users, you can use brew to install it :

$ brew install kustomize

If you are on another operating system, you can directly download the binary from the release page and add it to your path.

For the others, you also can build it from source, why not 😅.

Your base

To start with Kustomize, you need to have your original yaml files describing any resources you want to deploy into your cluster.
Those files will be stored for this example in the folder ./k8s/base/.

Those files will NEVER (EVER) be touched, we will just apply customization above them to create new resources definitions

Note: You can build base templates (e.g. for dev environment) at any point in time using the command kubectl apply -f ./k8s/base/.

In this example, we will work with a service and a deployment resources:

apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
    - name: http
      port: 8080
  selector:
    app: sl-demo-app

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - name: app
        image: foo/bar:latest
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

We wil add a new file inside this folder, named kustomization.yaml :

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - service.yaml
  - deployment.yaml

This file will be the central point of your base and it describes the resources you use.
Those resources are the path to the files relatively to the current file.

Note: This kustomization.yaml file could lead to errors when running kubectl apply -f ./k8s/base/, you can either run it with the parameter --validate=false or simply not running the command against the whole folder

To apply your base template to your cluster, you just have to execute the following command:

$ kubectl apply -k k8s/base

To see what will be applied in your cluster, we will mainly use in this article the command kustomize build instead of kubectl apply -k.

The result of kustomize build k8s/base command will be the following, which is for now only the two files previously seen, concatenated:

apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - image: foo/bar:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Kustomization

Now, we want to kustomize our app for a specific case, for example, for our prod environement.
In each step, we will see how to enhance our base with some modification.

The main goal of this article is not to cover the whole set of functionnalities of Kustomize but to be a standard example to show you the phiplosophy behind this tool.

First of all, we will create the folder k8s/overlays/prod with a kustomization.yaml inside it.

The k8s/overlays/prod/kustomization.yaml has the following content:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

If we build it, we will see the same result as before when building the base.

$ kustomize build k8s/overlays/prod

This will output the following yaml

apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - image: foo/bar:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

We are now ready to apply kustomization for our prod env

Define Env variables for our deployment

In our base, we didn't define any env variable. We will now add those env variables above our base.
To do so, it's very simple, we just have to create the chunk of yaml we would like to apply above our base and referece it inside the kustomization.yaml.

This file custom-env.yaml containing env variables will look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  template:
    spec:
      containers:
        - name: app # (1)
          env:
            - name: CUSTOM_ENV_VARIABLE
              value: Value defined by Kustomize ❤️

Note: The name (1) key here is very important and allow Kustomize to find the right container which need to be modified.

You can see this yaml file isn't valid by itself but it describes only the addition we would like to do on our previous base.

We just have to add this file to a specific entry in the k8s/overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml

If we build this one, we will have the following result:

$ kustomize build k8s/overlays/prod

apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - env:
        - name: CUSTOM_ENV_VARIABLE # (1)
          value: Value defined by Kustomize ❤️
        image: foo/bar:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

You can see our env block has been applied above our base and now the CUSTOM_ENV_VARIABLE (1) will be defined inside our deployment.yaml.

Change the number of replica

Like in our previous example, we will extend our base to define variables not already defined

Note: You can also override some variables already present in your base files.

Here, we would like to add information about the number of replica. Like before, a chunk or yaml with just the extra info needed for defining replica will be enought:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  replicas: 10
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

And like before, we add it to the list of patchesStrategicMerge in the kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml

The result of the command kustomize build k8s/overlays/prod give us the following result

apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  replicas: 10
  selector:
    matchLabels:
      app: sl-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - env:
        - name: CUSTOM_ENV_VARIABLE
          value: Value defined by Kustomize ❤️
        image: foo/bar:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

And you can see the replica number and rollingUpdate strategy have been applied above our base.

Use a secret define through command line

One of the things we often do is to set some variables as secret from command-line.
In our case, we are doing this directly from our Gitlab-CI on Gitlab.com.

But you can do this from anywhere else, the main purpose here is to define Kubernetes Secret without putting them inside Git 😱.

To do so, kustomize has a sub-command to edit a kustomization.yaml and create a secret for you. You just have to use it in your deployment like if it already exists.

$ cd k8s/overlays/prod
$ kustomize edit add secret sl-demo-app --from-literal=db-password=12345

These commands will modify your kustomization.yaml and add a SecretGenerator inside it.

Note: You can also use secret comming from properties file (with --from-file=file/path) or from env file (with --from-env-file=env/path.env)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml

secretGenerator:
- literals:
  - db-password=12345
  name: sl-demo-app
  type: Opaque

If you run the kustomize build k8s/overlays/prod from the root folder of the example project, you will have the following output

apiVersion: v1
data:
  db-password: MTIzNDU=
kind: Secret
metadata:
  name: sl-demo-app-6ft88t2625
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  replicas: 10
  selector:
    matchLabels:
      app: sl-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - env:
        - name: CUSTOM_ENV_VARIABLE
          value: Value defined by Kustomize ❤️
        image: foo/bar:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Note: The secret name is sl-demo-app-6ft88t2625 instead of sl-demo-app, it's normal and this is made to trigger a rolling update of the deployment if secrets content is changed.

If we want to use this secret from our deployment, we just have, like before, to add a new layer definition which uses the secret.

For example, this file will mount the db-password value as environement variables

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  template:
    spec:
      containers:
      - name: app
        env:
        - name: "DB_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: sl-demo-app
              key: db.password

And, like before, we add this to the k8s/overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml
- database-secret.yaml

secretGenerator:
- literals:
  - db-password=12345
  name: sl-demo-app
  type: Opaque

If we build the whole prod files, we now have

apiVersion: v1
data:
  db-password: MTIzNDU=
kind: Secret
metadata:
  name: sl-demo-app-6ft88t2625
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  replicas: 10
  selector:
    matchLabels:
      app: sl-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: db.password
              name: sl-demo-app-6ft88t2625 # (1)
        -  name: CUSTOM_ENV_VARIABLE
          value: Value defined by Kustomize ❤️
        image: foo/bar:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

You can see the secretKeyRef.name used is automatically modified to follow the name defined by Kustomize (1)

Note: Don't forget, the command to put the secret inside the kustomization.yaml file should be made only from safe env and should not be commited.

The same logic exists with ConfigMap with hash at the end to allow redeployement of your app if ConfigMap changes.

Change the image of a deployment

Like for secret, there is a custom directive to allow changing of image or tag directly from the command line.
This is very useful if you need to deploy the image previously tagged by your continuous build system.

To do that, you can use the following command:

$ cd k8s/overlays/prod
$ TAG_VERSION=3.4.5 # (1)
$ kustomize edit add secret sl-demo-app --from-literal=db-password=12345 # To create my required secret
$ kustomize edit set image foo/bar=foo/bar:$TAG_VERSION

Note: the TAG_VERSION here is usualy defined by your CI/CD system

The k8s/overlays/prod/kustomization.yaml will be modified with those values:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml
- database-secret.yaml

secretGenerator:
- literals:
  - db-password=12345
  name: sl-demo-app
  type: Opaque

images:
- name: foo/bar
  newName: foo/bar
  newTag: 3.4.5

And if we build it, with the kustomize build k8s/overlays/prod/ we have the following result:

apiVersion: v1
data:
  db-password: MTIzNDU=
kind: Secret
metadata:
  name: sl-demo-app-6ft88t2625
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: sl-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  replicas: 10
  selector:
    matchLabels:
      app: sl-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: db.password
              name: sl-demo-app-6ft88t2625
        - name: CUSTOM_ENV_VARIABLE
          value: Value defined by Kustomize ❤️
        image: foo/bar:3.4.5 # (1)
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

You see the first container.image of the deployment have been modified to be run with the version 3.4.5 (1).

Conclusion

We see in these examples how we can leverage the power of Kustomize to define your Kubernetes files without even using a templating system.
All the modification files you made will be applied above the original files without altering it with curly braces and imperative modification.

There is a lot of advanced topic in Kustomize, like the mixins and inheritance logic or other directive allowing to define a name, label or namespace to every created object...
You can follow the official Kustomize github repository to see advanced examples and documentation.

Note: You can find all code from this article in this Gitlab project.

Manage your secrets in Git with SOPS - Common operations

Kevin Davin — Sat, 30 May 2020 14:27:23 +0000

We saw in the previous article how to use SOPS to store our secrets in Git. Here, we will see some common operations required when you are using SOPS.

Edit a secret

Alice would like to change the value in dev_a secret. To do so, she can use the sops dev_a.encrypted.env command to open the $EDITOR and allow in-place changes.

After the edition, the secret is encrypted back, and she can commit the file in Git.

Add secret access to someone else

Alice would like to let Bobby read the dev_a secret. To do that, she will use sops --rotate --in-place --add-pgp <bobby-key-id> dev_a.encrypted.env command.

After this modification, Bobby can fetch modifications. He is now able to read (and modify) the secret.

Remove secret access to someone else

Alice now wants to remove Bobby access to dev_a secret. She is able to do this by using the sops --rotate --in-place --rm-pgp <bobby-key-id> dev_a.encrypted.env.

After this, Bobby is unable to decrypt the secret anymore.

Configure automatic key selection

Like we saw before, sops commands often requires references to key-id of people concerned by the modification... and this is error prone and hard to manage if you share access with a lot of people.

To simplify this, the team can create a file, named .sops.yaml and placed it in the root of our Git repository.

creation_rules:

  # Specific to `dev_a` env
  - path_regex: dev_a\.encrypted\.env$
    # Here, only the `Alice` key-id
    pgp: >-
      5844C613B763F4374BAB2D2FC735658AB38BF93A

  # Specific to `int` env
  - path_regex: int\.encrypted\.env$
    # Here, we have :
    # * `Alice` key-id: 5844C613B763F4374BAB2D2FC735658AB38BF93A
    # * `Bobby` key-id: AE0D6FD0242FF896BE1E376B62E1E77388753B8E
    # * `Devon` key-id: 57E6DA39E907744429FB07871141FE9F63986243
    pgp: >-
      5844C613B763F4374BAB2D2FC735658AB38BF93A,
      AE0D6FD0242FF896BE1E376B62E1E77388753B8E,
      57E6DA39E907744429FB07871141FE9F63986243

  # Specific for new env `dev_a_and_b`
  - path_regex: dev_a_and_b\.encrypted.env$
    # Here, we have only `Alice` and `Bobby` :
    # * `Alice` key-id: 5844C613B763F4374BAB2D2FC735658AB38BF93A
    # * `Bobby` key-id: AE0D6FD0242FF896BE1E376B62E1E77388753B8E
    pgp: >-
      5844C613B763F4374BAB2D2FC735658AB38BF93A,
      AE0D6FD0242FF896BE1E376B62E1E77388753B8E

Here, Bobby will create a new secret for dev_a_and_b env just with the command sops dev_a_and_b.encrypted.env. No more --pgp <key-id>, sops automatically selects the closest .sops.yaml file from the CWD (see this).

Keys are selected by matching a regex againt the path of the file, so possibilities are wide and this is simpler than using parameters in command line !

Add or Remove access with `.sops.yaml`

If a .sops.yaml file is used, Alice can simplify the add-pgp or rm-pgp command previously seen. She just need to change the .sops.yaml and use the command sops updatekeys dev_a.encrypted.env to update who can decrypt the file.

Conclusion

Those are the most common operations required to use sops in your project. This is simple and still helps you to keep your secrets in sync with your code !

You can find the source code of this article, files, and scripts in this GitLab repository.

Manage your secrets in Git with SOPS

Kevin Davin — Sat, 23 May 2020 10:35:16 +0000

We use Git for everything now, from code source to organization, history, and even for Kubernetes Cluster Management (aka GitOps).

But, there is still something not widely adopted... managing our secrets in Git. Some tools like HashiCorp Vault, Google Secret Management, or AWS Secret Manager provide us a solution to manage our secrets in a dedicated system, but they are still not in sync with our source code.

We will see here, thanks to Mozilla SOPS how to integrate our secrets management directly in Git.

getsops / sops

Simple and flexible tool for managing secrets

SOPS: Secrets OPerationS

SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP (demo)

1 Download

1.1 Stable release

Binaries and packages of the latest stable release are available at https://github.com/getsops/sops/releases.

1.2 Development branch

For the adventurous, unstable features are available in the main branch, which you can install from source:

$ mkdir -p $GOPATH/src/github.com/getsops/sops/
$ git clone https://github.com/getsops/sops.git $GOPATH/src/github.com/getsops/sops/
$ cd $GOPATH/src/github.com/getsops/sops/
$ make install

(requires Go >= 1.19)

If you don't have Go installed, set it up with:

$ {apt,yum,brew} install golang
$ echo 'export GOPATH=~/go' >> ~/.bashrc
$ source ~/.bashrc
$ mkdir $GOPATH

Or whatever variation of the above fits your system and shell.

To use SOPS as a library, take a look at the decrypt package.

…

View on GitHub

DISCLAIMER: I've previously written an article on the same subject about a project named kubesec specialized in Kubernetes Secret. The project seems to be stopped and Mozilla SOPS is a better alternative right now, because it can manage every kind of secrets, not only Kubernetes ones.

Sops Installation

Sops is very simple to install, like every golang application, you just have to download the binary for your specific Operating System (Linux, Mac, Windows) directly from the release page on GitHub.

By the way, you can install it thanks to brew on Mac & Linux (sops formuale).

Use case

What we will try to achieve is to store secrets in Git but with restrictions on "who can access what".

For example, we have 4 environments, dev_a, dev_b, int, and prod and 3 team members, Alice, Bobby, and Devon.

We want to restrict secrets access with the following requirements:

dev_a secrets should be only accessible by Alice.
dev_b secrets should be only accessible by Bobby.
int secrets should be accessible by Alice, Bobby, and Devon.
prod secrets should be accessible only by Devon.

Each of them already has configured their GPG key pairs. If you need to set them up, you can follow the official GitLab documentation 🦊 about this.

In this example, secrets are just plain old env files.

`dev_a` configuration

Alice will generate a file containing a secret:

Thanks to the command:

sops -pgp 5844C613B763F4374BAB2D2FC735658AB38BF93A -e dev_a.env > dev_a.encrypted.env

Alice has encrypted the file dev_a.env and stored the result in dev_a.encrypted.env. She is the only one able to decrypt it.

Let's see, if we run this as Bobby:

dev_b and prod configurations are similar to the one created by Alice. Then, Bobby and Devon will respectively create dev_b and prod.

`int` configuration

In this configuration, we would like every developers to be able to read this file. But, only developers from the project and not everyone with access to the git repository... so we still have to encrypt this file.

To do so, Devon will execute the following commands:

Devon has to create the secret with the command

sops --pgp 57E6DA39E907744429FB07871141FE9F63986243,\
           5844C613B763F4374BAB2D2FC735658AB38BF93A,\
           AE0D6FD0242FF896BE1E376B62E1E77388753B8E \
           -e int.env > int.encrypted.env`

This command contains every public key ids, comma sparated.

We can check that both Alice and Bobby can decrypt the int.encrypted.env file:

alice $ sops -d int.encrypted.env
secret=5Tz2QNxki789YFDa

bobby $ sops -d int.encrypted.env
secret=5Tz2QNxki789YFDa

All the *.encrypted.env files are now stored in Git and can be managed like any other resources, with history and diff in commits. Only those defined during encryption can read them edit them.

You can find the source code of this article, files, and scripts in this GitLab repository.

I hope this will help you to use Git & SOPS to manage your secrets.

DEV Community: Kevin Davin

Taming FluxCD HelmReleases: The Kustomize Way approach

HelmRelease, what is this?

GitOps, aka "only source of truth"

External Chart Dependencies: A Potential Weak Point

Limited Visibility into Helm Chart Content

Lack of Immutability in Helm Charts!

Chart customisation, another nightmare 👻

Solution is simplicity 🚀!

Render the chart and store it into git for better auditability

How to deploy generated files with FluxCD?

Direct Customization with Kustomize

Enjoy reviews and audit with rich diff!

Conclusion

Deploy with Kustomize, FluxCD and Remote Resources

Introduction

Cluster & FluxCD configuration

bases repository

PodInfo base publication

Core PodInfo

Components

Publish PodInfo to dev

podinfo-base flux repository

podinfo flux repository

podinfo-dev flux Kustomization

Publish PodInfo to prod

Conclusion

Scaling Keycloak on Distroless into Kubernetes

standalone-ha.xml extraction

Kubernetes deployment

Headless Service to the rescue!

Demo time!

Conclusion

Keycloak on Distroless into Kubernetes

Challenges

Database

Launching Keycloak

Providing configuration to Keycloak

Probes

Lifecycle Pre-Stop Command

Environment Configuration

Deploying Keycloak

Conclusion

Keycloak on Distroless

Crafting the best Dockerfile possible

Moving Keycloak into Distroless

Generating the ENTRYPOINT

Generating the perfect configuration

What about security?

Conclusion

Anthos Service Mesh, Istio on Google Cloud ⛵️

Anthos Service Mesh, a managed Istio ⛵️

Installation

Namespace Activation

Functionalities

Dashboards

Global overview

Service Dashboard

Timeline everywhere!

Conclusion

Manage your secrets in Git with SOPS for Kubectl & Kustomize 🔧

Simpler solutions

📄 With files

🗂 With .env files

🔧 Alternative with Kustomize Plugins

🔐 Conclusion

Manage your secrets in Git with SOPS for Kubernetes ☸️

😊 Naive usage

🔒 Sops & YAML

🔏 Encrypt specific keys only

🦊 CI Integration

Conclusion

Manage your secrets in Git with SOPS & GitLab CI 🦊

🤖 Creation of the CI identity

🔑 Import public key of CI

🔏 Update keys of every secret

🦊 GitLab CI Configuration

🦊 GitLab CI Optimization

Conclusion

Kustomize - The right way to do templating in Kubernetes

`HelmRelease`, what is this?

Render the chart and store it into `git` for better auditability

`bases` repository

Publish PodInfo to `dev`

`podinfo-base` flux repository

`podinfo` flux repository

`podinfo-dev` flux Kustomization

Publish PodInfo to `prod`

`standalone-ha.xml` extraction

🗂 With `.env` files

Add or Remove access with `.sops.yaml`

`dev_a` configuration

`int` configuration