Optimizing Spring Boot dynamic configuration using Kubernetes ConfigMap (part 1)

Disclaimer

This article represents my perspective on this solution, thus any suggestions, fixes, or discussions will be highly appreciated.

The short story

In the ever-evolving landscape of container orchestration, Kubernetes stands as a beacon of efficiency and scalability. However, for those deeply entrenched in the realm of managing Kubernetes Pods, the recurrent need to update properties across multiple instances can become a daunting task, consuming valuable time and resources. Imagine a scenario where a critical change needs to be applied uniformly across a myriad of Pods, demanding a repetitive and error-prone process. It is in this challenge that the quest for a seamless solution arises — a single update that ripples through the entire Kubernetes environment, ensuring consistency without the hassle of individually addressing each Pod.
ConfigMaps offer a method that allows for a one-time update, streamlining operations and bringing efficiency to the forefront of containerized environments.

The Spring framework currently offers two distinct approaches, both integrated within the Spring Cloud project:

1. Direct access from pods to Kubernetes resources.
   This method involves accessing Kubernetes resources directly from each pod through a scheduled process. However, this approach was deprecated in 2020.

2. Spring Cloud Kubernetes Configuration Watcher Service.
   Alternatively, the Spring Cloud team has introduced a specialized service, the Spring Cloud Kubernetes Configuration Watcher. This service is designed to recognize changes in ConfigMaps and trigger a reload of the IoC Spring container by reading a locally mounted ConfigMap file on each pod.

While both approaches have their pros and cons, it's important to highlight concerns with the first approach. This method is considered suboptimal as it necessitates resource allocation for a scheduled process. Additionally, the service's pod needs to trigger the process, which is architecturally problematic when a service attempts to trigger an external resource that has changed when each pods actually is not aware of such changes.

On the other hand, the second approach initiates a series of flows when an external resource changes, monitoring and responding to these changes. Both methods involve restarting beans or the entire context, potentially disrupting service. To address this, a flow can be created, triggered by the Spring Cloud Kubernetes Configuration Watcher. This flow would utilize a service's HTTP client to connect to the Kubernetes API, pulling ConfigMap key-value pairs to update properties for each serving class holding these properties.

The Spring Cloud Kubernetes Configuration Watcher Helm chart deploys an instance of this service. This service detects changes in a specific ConfigMap and sends empty HTTP POST requests to a custom actuator endpoint (http:<hostname>:<port>/actuator/configmap/refresh). This invocation prompts the service's HTTP client to send a GET request to the Kubernetes API, obtaining the updated ConfigMap file with key-value pairs that override the current values within the service.

Below the first part that will help to implement my approach with changes relates mostly to Kubernetes. The second part will describe service's changes that will support the chosen approach.

What is Kubernetes ConfigMap?

In Kubernetes, ConfigMap is a resource designed to handle the configuration needs of applications separately from their codebase. It serves as a key-value store or a provider of configuration data in the form of plain text or configuration files. ConfigMap offers a versatile and efficient solution for handling configuration data in containerized applications.

Key characteristics of ConfigMap

• Decoupling configuration from an application code.

ConfigMap promotes the separation of configuration concerns from the application code. By externalizing configuration data, changes can be made to application settings without modifying or redeploying the application itself. This decoupling aligns with best practices, fostering modular and adaptable application architectures.

• Key-Value pairs and configuration files.

ConfigMaps support two primary modes of storing configuration data. The first is through key-value pairs, allowing for simple configurations. The second involves mounting entire configuration files, which is particularly useful for complex configurations or scenarios where maintaining the structure of a configuration file is essential.

• Environment specific configurations.

Microservices and containerized applications often traverse various environments, from development and testing to production. ConfigMap enables the creation of environment-specific configurations, ensuring that applications seamlessly adapt to different runtime conditions.

• Live updates and dynamic configuration.

ConfigMaps support live updates, allowing for dynamic configuration changes at runtime. This feature is especially valuable in scenarios where applications need to adapt quickly to changing requirements without the need for restarts or redeployment.

Creation and usage examples of ConfigMaps

1. Declarative configuration in YAML or JSON: ConfigMaps are typically defined declaratively using YAML or JSON files. These files specify the configuration data, either as key-value pairs or by referencing external configuration files.

apiVersion: v1
kind: ConfigMap
metadata:
    name: example-config
data:
    key1: value1
    key2: value2 

1. Imperative creation: ConfigMaps can also be created imperatively using the kubectl command line:

kubectl create configmap \
    example-config \
    --from-literal=key1=value1 \
    --from-literal=key2=value2

1. Mounting ConfigMap in pods: Once a ConfigMap is created, it can be mounted into the file system of pods as a volume or used as environment variables.

apiVersion: v1
kind: Pod
metadata:
    name: example-pod
spec:
    containers:
    - name: example-container
      image: example-image
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
volumes:
    - name: config-volume
configMap:
    name: example-config

Remember, mounting to each pod will increase amount of time for updating ConfigMap on the mounted volume. The maximum time Kubernetes will perform it can take more than 120,000 milliseconds.

Use cases and considerations

• Configuration of application.properties.

ConfigMap is particularly useful for managing application properties, environment variables, and other configuration details that might vary across different deployment environments. The foundation of Spring Boot's configuration lies in properties and YAML files. These files provide a simple and effective means of configuring applications. However, as microservices environments grow in complexity, the need for externalized configuration management becomes more pronounced.

• Configuration for multiple microservices.

In a microservices architecture, ConfigMaps can be employed to manage configurations for multiple services, providing a centralized and consistent approach to configuration management.

• Sensitive information and security.

While ConfigMap is suitable for non-sensitive configuration data, Kubernetes Secrets are recommended for managing sensitive information such as passwords and API keys.

• Integration with Spring Cloud Kubernetes Watcher.

ConfigMaps seamlessly integrate with tools like Spring Cloud Kubernetes Watcher enabling dynamic updates to configurations without requiring application restarts.

• Comparison with Consul, etcd, or Spring Cloud Config Server.

Alternatively, some organizations adopt tools like Consul, etcd, or Spring Cloud Config Server for centralized configuration management. While these tools offer robust solutions, the simplicity and native integration of ConfigMap make it an attractive choice, especially in Kubernetes-centric environments.

Implementation of ConfigMaps

Let's assume our services have 2 ports: one for a main flow (8888) and another one for service's management (9999).

ConfigMap

Each service will have its own ConfigMap within its Helm chart. ConfigMap file will include key-value pairs of application properties to be updated dynamically. These pairs can be added into data key only:

{{- if .Values.configmap.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: "{{ .Release.Name }}"
  namespace: production
  labels:
    spring.cloud.kubernetes.config: "true"
    spring.cloud.kubernetes.secret: "false"
  annotations:
    spring.cloud.kubernetes.configmap.apps: "{{ .Release.Name }}"
data:
  kubernetes.config.map.external-api-enabled: "true"
  kubernetes.config.map.internal-api-enabled: "true"
{{- end }}

{{ .Release.Name }} will be you actual spring.application.name property you've already defined within your application. Otherwise it can be a default application, but I suppose you will not do it ad define for each service the dedicated name for future convenience.

The key has the following naming convention: <source name>.<resource name>.<key/property name> where:

• <source name> will be kubernetes
• <resource name> will be config.map (splitter by a dot for a better readability)
• <key name> will be external-api-enabled property name (splitter by a hyphen to allow for Spring Boot parse a key-value into a camel case class field, e.g.: externalApiEnabled)

Implementation

The following steps need to be performed in order to add ConfigMap support for services:

1. Add the following configuration into values.yaml file within the Helm charts of a service:

configmap:
  enabled: true

It will allow you to trigger the first clause within a ConfigMap, that actually enables it: {{- if .Values.configmap.enabled -}}.

1. Add ConfigMap.yaml file into Helm charts templates directory.

2. Rename service.yaml file into service-one-main.yaml file and update the metadata.name value by adding the suffix -main:
   

apiVersion: v1
kind: Service
metadata:
  name: "{{ .Release.Name }}-main"
spec:
  ports:
    - port: {{ .Values.port }}
      targetPort: {{ .Values.port }}
      protocol: TCP
      name: "http-{{ .Values.port }}"
  selector:
    app: {{ .Release.Name }}

For each application's port there is a dedicated service.yaml file defines specifications related to this port: main or maintenance port. The value {{ .Values.port }} allows you to add a port's variable into the main values.yaml file for a convenient management of charts.

1. Update service-act.yaml file content by adding annotations key-value pair:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
  labels:
    app: {{ .Release.Name }}
  annotations:
    boot.spring.io/actuator: "http://:{{ .Values.act_port }}/actuator/configmap"
spec:
  ports:
    - port: {{ .Values.act_port }}
      targetPort: {{ .Values.act_port }}
      protocol: TCP
      name: "http-{{ .Values.act_port }}"
  selector:
    app: {{ .Release.Name }}

This file defines a Kubernetes service for a management port I've added (9999).
boot.spring.io/actuator key represents the custom actuator endpoint /configmap I've added to our service. It can any path you want because Spring Boot allows to you programmatically define any new actuator path. The reason I've added it is to control the behavior of this endpoint by adding my own logic and metrics if needed.

1. Update a service.name value by adding the suffix -main within ingress.yaml file to allow a traffic split (if you have such implementation) to the reconfigured main Service resource:

  - host: service-one.com
    http:
      paths:
      - backend:
          service:
            name: "{{ .Release.Name }}-main"
            port:
              number: {{ .Values.port }}
        path: /readiness
        pathType: Prefix
  - host: service-one.com
    http:
      paths:
      - backend:
          service:
            name: "{{ .Release.Name }}-main"
            port:
              number: {{ .Values.port }}

1. Update region/deployment related *-values.yaml files to include a new reconfigured Service name within the traffic split configuration if you have some, e.g. us-east-1-service-values.yaml:

ts:
  backends:
  - service: service-one
    weight: 99
  - service: service-one-main
    weight: 1
  service: service-one

1. Add new Service name and namespace into role-binding.yaml file within Helm charts of Spring Cloud Kubernetes Configuration Watcher:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-configmap-binding
subjects:
- kind: ServiceAccount
  name: username
  namespace: production
- kind: ServiceAccount
  name: service-one
  namespace: production
roleRef:
  kind: Role
  name: read-configmap
  apiGroup: rbac.authorization.k8s.io

username represents a role binding for each service that will connect to Kubernetes API.

1. Add Helm chart project name into read-configmap-role.yaml file within Helm charts of Spring Cloud Kubernetes Configuration Watcher:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-configmap
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["service-one", "service-two"]
  verbs: ["get", "watch", "list"]

Each name will be added into the resourceName array.

1. Update spring.application.name property within application.properties file of the service in order to allow to Kubernetes recognize the pod of this service.

This application name must be the same as a Helm chart Service name (e.g: service-one, service-two). The same name will be provided into the property kubernetes.config.map.client.uri in order to get the ConfigMap file by Kubernetes API from the dedicated resource defines the service deployment.

info.app.name=@project.name@
spring.application.name=${info.app.name}
kubernetes.config.map.client.uri=https://<Kubernetes API address>:<Kubernetes API port>/api/v1/namespaces/production/configmaps/${spring.application.name}

1. Enable /configmap actuator endpoint by updating the following application properties:

management.endpoint.configmap.enabled=true
management.endpoints.web.exposure.include=configmap,...

Implementation validation

For a convenient monitoring the Spring Cloud Watcher runs with the DEBUG logging level.
The following log clearly describes steps during updating ConfigMap key-value pairs:

2023-11-22 11:44:24.150  INFO 1 --- [           main] .c.w.HttpBasedSecretsWatchChangeDetector : Kubernetes event-based secrets change detector activated
2023-11-22 11:44:24.153 DEBUG 1 --- [//10.32.0.1/...] .w.HttpBasedConfigMapWatchChangeDetector : Scheduling remote refresh event to be published for ConfigMap service-one to be published in 120000 milliseconds
2023-11-22 11:44:24.165 DEBUG 1 --- [//10.32.0.1/...] .w.HttpBasedConfigMapWatchChangeDetector : Scheduling remote refresh event to be published for ConfigMap service-two to be published in 120000 milliseconds
2023-11-22 11:46:24.188 DEBUG 1 --- [oundedElastic-4] .w.HttpBasedConfigMapWatchChangeDetector : Metadata actuator uri is: http://:<pod's port>/actuator/configmap
2023-11-22 11:46:24.188 DEBUG 1 --- [oundedElastic-1] .w.HttpBasedConfigMapWatchChangeDetector : Metadata actuator uri is: http://:<pod's port>/actuator/configmap
2023-11-22 11:46:24.254 DEBUG 1 --- [oundedElastic-2] .w.HttpBasedConfigMapWatchChangeDetector : Found actuator URI in service instance metadata
2023-11-22 11:46:24.254 DEBUG 1 --- [oundedElastic-1] .w.HttpBasedConfigMapWatchChangeDetector : Found actuator URI in service instance metadata
2023-11-22 11:46:24.254 DEBUG 1 --- [oundedElastic-2] .w.HttpBasedConfigMapWatchChangeDetector : Sending refresh request for service-one to URI http://<pod's IP address>:<pod's port>/actuator/configmap/refresh
2023-11-22 11:46:24.254 DEBUG 1 --- [oundedElastic-1] .w.HttpBasedConfigMapWatchChangeDetector : Sending refresh request for service-two to URI http://<pod's IP address>:<pod's port>/actuator/configmap/refresh
2023-11-22 11:46:24.337 DEBUG 1 --- [or-http-epoll-6] .w.HttpBasedConfigMapWatchChangeDetector : Refresh sent to service-one at URI address http://<pod's IP address>:<pod's port>/actuator/configmap/refresh returned a 200 OK
2023-11-22 11:46:24.345 DEBUG 1 --- [r-http-epoll-10] .w.HttpBasedConfigMapWatchChangeDetector : Refresh sent to service-two at URI address http://<pod's IP address>:<pod's port>/actuator/configmap/refresh returned a 200 OK

If by any reason the changes in ConfigMap weren’t recognized by a watcher, the deleting of its pod or restarting it will trigger sending HTTP POST requests to all defined services within a watcher. Therefore you will update all services with their ConfigMaps.

Possible issues

This complex implementation mainly depends on Kubernetes resources configurations. Spring Cloud Kubernetes Configuration Watcher uses regular reactive Spring Boot. You can check its code here.

To prevent Kubernetes API failures there is a delay configuration (a service WebClient flow) in sending HTTP GET requests from each pod defined by a random value of time but less than 5 seconds.
It’s not recommended to change the watcher refresh delay configuration (2 minutes) because this value represents a possible maximum time for Kubernetes to update resources (it relates mainly for mounted ConfigMap).

References

1. Kubernetes ConfigMaps documentation
2. Spring Cloud Kubernetes Configuration Watcher documentation
3. Spring Cloud Kubernetes Configuration Watcher source code
4. Spring Boot Actuator: Production-ready Features

---

Finding my articles helpful? You could give me a caffeine boost to keep them coming! Your coffee donation will keep my keyboard clacking and my ideas brewing. But remember, it's completely optional. Stay tuned, stay informed, and perhaps, keep the coffee flowing!