In the previous article, we've seen how to build and run a Keycloak application based on Distroless base image. However, this article only used Docker to launch containers, and you mostly use an orchestrator to do that. In this article, we will see how to run our previously created image in a Kubernetes cluster.
Challenges
Using a Keycloak image based on Distroless requires some adaptation:
- Launching the application require a specific command
- Stopping the application require a specific command
- Probes should be defined
- Database connection should be provided via env variables
- Configuration file should be mounted by the orchestrator
Database
Before doing some configuration, we will "install" a PostgreSQL instance inside our cluster. This will be done with this manifest:
# database.yaml
apiVersion: v1
kind: Service
metadata:
name: database
labels:
app: database
spec:
ports:
- name: pg-port
port: 5432
targetPort: 5432
protocol: TCP
selector:
app: database
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: database
spec:
selector:
matchLabels:
app: database
template:
metadata:
labels:
app: database
spec:
containers:
- image: postgres:13.3-alpine
imagePullPolicy: IfNotPresent
name: database
env:
- name: POSTGRES_USER
valueFrom:
configMapKeyRef:
name: database
key: user
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: database
key: password
- name: POSTGRES_DB
valueFrom:
configMapKeyRef:
name: database
key: name
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- su - postgres -c "pg_ctl stop -m fast"
livenessProbe:
exec:
command:
- /bin/sh
- -c
- exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
readinessProbe:
exec:
command:
- /bin/sh
- -c
- -e
- exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
ports:
- name: pg-port
containerPort: 5432
protocol: TCP
NOTE: This manifest is some kind of hello-world at PostgreSQL level and SHOULD NOT be used for production… because it doesn't manage any state. Every data will be lost if the pod restart. This is just for this example.
Launching Keycloak
In the previous step, we fetched the java command used to launched Keycloak. We will reuse it, in the Kubernetes manifest. For information, the command was:
java '-D[Standalone]' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=172.17.0.2 -Djboss.bind.address.private=172.17.0.2 -c=standalone-ha.xml
We will set $.spec.template.spec.containers[0].command and $.spec.template.spec.containers[0].args to the previously used ENTRYPOINT in the Dockerfile.
# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
labels:
app: keycloak
spec:
template:
spec:
containers:
- name: keycloak
# We define Java as the main command of the image.
command: [ "java" ]
# We put back all the parameters from the previous step.
args:
- "-D[Standalone]"
- "-server"
- "-Xms64m"
- "-Xmx512m"
- "-XX:MetaspaceSize=96M"
- "-XX:MaxMetaspaceSize=256m"
- "-Djava.net.preferIPv4Stack=true"
- "-Djboss.modules.system.pkgs=org.jboss.byteman"
- "-Djava.awt.headless=true"
- "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED"
- "--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED"
- "--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED"
- "-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log"
- "-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties"
- "-jar"
- "/opt/jboss/keycloak/jboss-modules.jar"
- "-mp"
- "/opt/jboss/keycloak/modules"
- "org.jboss.as.standalone"
- "-Djboss.home.dir=/opt/jboss/keycloak"
- "-Djboss.server.base.dir=/opt/jboss/keycloak/standalone"
- "-c=standalone.xml"
- "-b=0.0.0.0"
- "-bprivate=0.0.0.0"
- "-bmanagement=0.0.0.0"
NOTE: Like previously, the parameter can and should be adapted to your needs, for performance and security reasons.
Providing configuration to Keycloak
Another part of the Keycloak configuration is the standalone.xml. Our goal here will be to mount the file inside the running container, at the right place, to be used as source of configuration at boot time.
# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
labels:
app: keycloak
spec:
template:
spec:
# We declare a volume containing the configuration of keycloak
volumes:
- name: keycloak-config
# The configMap targeted here will be declared in the next step of the article
configMap:
name: keycloak
containers:
- name: keycloak
# And then, we mount the file from the previously seen volume at the
# location where keycloak want to find it…
volumeMounts:
- name: keycloak-config
mountPath: /opt/jboss/keycloak/standalone/configuration/standalone.xml
subPath: standalone.xml
Probes
In Kubernetes world, the orchestrator needs to know when the system is live and ready. For this, we have to define some liveness and readyness probes allowing Kubernetes to act and react to problem with Keycloak if required. Again, this is just some values in our manifest:
# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
labels:
app: keycloak
spec:
template:
spec:
containers:
- name: keycloak
# Liveness, if in error, keycloak will be restarted
livenessProbe:
httpGet:
path: /auth/
port: http
initialDelaySeconds: 30
timeoutSeconds: 5
# Readiness, if in success, traffic will be routed to this pod
readinessProbe:
httpGet:
path: /auth/realms/master
port: http
initialDelaySeconds: 30
timeoutSeconds: 1
NOTE: RedHat team will introduce more advanced and precise liveness and readyness probes in future version of Keycloak 😇.
Lifecycle Pre-Stop Command
To gracefully shutdown Keycloak, we usually have to execute a specific shell script. In our case, without any shell, this will be problematic… like for ENTRYPOINT, we will extract the java command from the shell script and use it in our manifest.
# We launch the keycloak original image and launch a bash inside it
$ docker run -it --rm --entrypoint=bash jboss/keycloak:13.0.1
# We use our magic trick to see commands executed by the jboss-cli.sh we will use
bash-4.4$ awk -i inplace 'NR==2 {print "set -x"} 1' /opt/jboss/keycloak/bin/jboss-cli.sh
# Finally, we launch the script with parameters we want to execute in our pre-hook
bash-4.4$ /opt/jboss/keycloak/bin/jboss-cli.sh --connect command=:shutdown --timeout=20
++ dirname /opt/jboss/keycloak/bin/jboss-cli.sh
+ DIRNAME=/opt/jboss/keycloak/bin
+ GREP=grep
+ . /opt/jboss/keycloak/bin/common.sh
++ '[' x = x ']'
++ COMMON_CONF=/opt/jboss/keycloak/bin/common.conf
++ '[' -r /opt/jboss/keycloak/bin/common.conf ']'
+ cygwin=false
+ darwin=false
+ case "`uname`" in
++ uname
+ false
++ cd /opt/jboss/keycloak/bin/..
++ pwd
+ RESOLVED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' x/opt/jboss/keycloak = x ']'
++ cd /opt/jboss/keycloak
++ pwd
+ SANITIZED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' /opt/jboss/keycloak '!=' /opt/jboss/keycloak ']'
+ export JBOSS_HOME
+ '[' x = x ']'
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
+ '[' x = x ']'
+ '[' x '!=' x ']'
+ JAVA=java
+ setDefaultModularJvmOptions
+ setModularJdk
+ java --add-modules=java.se -version
+ MODULAR_JDK=true
+ '[' true = true ']'
++ echo
++ grep '\-\-add\-modules'
+ DEFAULT_MODULAR_JVM_OPTIONS=
+ '[' x = x ']'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ JAVA_OPTS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ false
+ false
+ JAVA_OPTS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing'
+ JAVA_OPTS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true'
++ eval echo '"/opt/jboss/keycloak/modules"'
+++ echo /opt/jboss/keycloak/modules
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
++ echo --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true
++ grep logging.configuration
+ LOG_CONF=
+ '[' x = x ']'
+ exec java --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true -Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.cli --connect command=:shutdown --timeout=20
Failed to connect to the controller: The controller is not available at localhost:9990: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: Connection refused
The execution ends in error, which is normal because, in this case, no Keycloak instance are running. The java command is displayed, starting with + exec java. All of this can be moved into our yaml manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
labels:
app: keycloak
spec:
template:
spec:
containers:
- name: keycloak
lifecycle:
preStop:
exec:
command:
- "java"
- '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
- '--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
- '--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
- '-Djboss.modules.system.pkgs=com.sun.java.swing'
- '-Dcom.ibm.jsse2.overrideDefaultTLS=true'
- '-Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties'
- "-jar"
- "/opt/jboss/keycloak/jboss-modules.jar"
- "-mp"
- "/opt/jboss/keycloak/modules"
- "org.jboss.as.cli"
- "--connect"
- '--commands=shutdown --timeout=20'
Environment Configuration
Finally, we have to provide all the configuration to Keycloak. Until now, we only define how the value will be linked but not which value at all. To do that, we will first include some environment values into the Keycloak manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
labels:
app: keycloak
spec:
template:
spec:
containers:
- name: keycloak
env:
- name: DB_ADDR
value: database
- name: DB_DATABASE
valueFrom:
configMapKeyRef:
name: database
key: name
- name: DB_USER
valueFrom:
configMapKeyRef:
name: database
key: user
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: database
key: password
Here, we are mainly targeting values required for database connection. To provide it to the deployment, we will use a kustomization.yaml file. It will gather every manifests presented until now and add concret values:
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Install everything in the kubernetes namespace named Keycloak
namespace: keycloak
# Load both keycloak.yaml and database.yaml
resources:
- keycloak.yaml
- database.yaml
# Generate two ConfigMap
configMapGenerator:
# One for Keycloak, containing the standalone.xml
- name: keycloak
files:
- standalone.xml
# One for database, containing user and name
- name: database
literals:
- user=keycloak
- name=keycloak
# Generate a Secret
secretGenerator:
# For the database, containing just the password of the database
- name: database
literals:
- password=sPCwZjuq8CMvrBn7
NOTE: The password is here stored in clear, you can choose to use some tooling to encrypt the secret at git level (with SOPS) or cluster level (with SealedSecret).
Deploying Keycloak
In this step, we will deploy our version of Keycloak in a Kubernetes Cluster. For this example, I choose to use docker-for-mac and its Kubernetes integration. You can, of course, use any Kubernetes distribution (Google Kubernetes Engine, Azure Kubernetes Service, Amazon Elastic Kubernetes Service…).
$ ls
database.yaml keycloak.yaml kustomization.yaml standalone.xml
# We deploy all the manifests
$ kubectl apply -k .
configmap/database-56h9f7gfdh created
configmap/keycloak-96k2tfg747 created
secret/database-8g8gk22d26 created
service/database created
service/keycloak created
deployment.apps/database created
deployment.apps/keycloak created
# We create a port-forward to the keycloak pod
$ kubectl -n keycloak port-forward pod/keycloak-567797b6bb-6vqvz 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
To access it, we create a port-forward between my computer and the cluster. Thanks to that, I'm able to access the UI:
And Voila!
Conclusion
In this article, we've seen how to deploy our custom Keycloak based on Distroless Java. This is a challenging setup, but at the end we can use a more secure version of Keycloak without any shell, which prevent any attack using shell as a vector of code execution for example!
I hope you liked it, you can find all the sample files from this article in this GitLab repository: davinkevin/keycloak-distroless.
Top comments (0)