Use cert-manager with for Karavi Observability

TL; DR

Check out how works Prometheus metrics & Grafana dashboard for CSI PowerFlex from that karavi observability video.

Learn more about karavi and karavi-observability from their respective repositories. And use these configurations to use cert-manager as a certificate source for the karavi components.

The premise

Dell Technologies launched the project Karavi in December 2020 with the objective to complement functionalities not covered by the CSI specification.

Karavi focuses in three domains :

• Observability
• Data mobility
• Security

The first project brought to life as a tech-preview is the metrics & topology collection for PowerFlex Kubernetes volumes.

The video below will give you an introduction on the architecture, use-cases and what is coming in the future.

At the time of the publication of this post, the communication between the OpenTelemetry component and Prometheus server, and, the communication between the Karavi topology component and Grafana are secured with TLS by default :

As indicated in the documentation, you have to supply certificates to the helm installer :

helm install karavi-observability dell/karavi-observability -n karavi --create-namespace \
    --set-file karaviTopology.certificateFile=<location-of-karavi-topology-certificate-file> \
    --set-file karaviTopology.privateKeyFile=<location-of-karavi-topology-private-key-file> \
    --set-file otelCollector.certificateFile=<location-of-otel-collector-certificate-file> \
    --set-file otelCollector.privateKeyFile=<location-of-otel-collector-private-key-file> \
    --set karaviMetricsPowerflex.powerflexPassword=<base64-encoded-password> \
    --set karaviMetricsPowerflex.powerflexUser=<base64-encoded-username> \
    --set karaviMetricsPowerflex.powerflexEndpoint=https://<powerflex-endpoint>

This command assumes you generated the different certificates with correct DNS names. For example with openssl:

openssl req -new -newkey rsa:2048 -nodes -keyout karavi-topology.key -out karavi-topology.csr -subj '/CN=karavi-topology'
openssl x509 -req -in karavi-topology.csr -signkey karavi-topology.key -out karavi-topology.crt
openssl req -new -newkey rsa:2048 -nodes -keyout otel-collector.key -out otel-collector.csr -subj '/CN=otel-collector'
openssl x509 -req -in otel-collector.csr -signkey otel-collector.key -out otel-collector.crt

Obviously, anytime the certificate expires or you move the components to different namespace you have to manually generate new certificates.

So let us see how to take advantage of cert-manager to automate that process.

The implementation

cert-manager is the goto Kubernetes certificate management tool. It bundled with some distribution like GKE. If you need to install it few kubectl apply will do the job.

The first step to use certificate delivered by cert-manager is to configure an Issuer. The easiest is to use a SelfSigned without a dedicated certificate authority :

400: Invalid request

The next step is to define the parameters for your certificate (size, rotation, DNS names, etc.) ; here is, for example, what we have for the otel-collector :

400: Invalid request

The same for karavi-topology can be downloaded here.

The last step is to patch the deployment (otel-collector and karavi-topology) to use the new certificate stored as a secret.

You can choose to do it directly in the karavi-metrics-powerflex & karavi-topology Helm charts ; or patch it in Kubernetes if already deployed.

For example, with the otel-collector we have:

400: Invalid request

For karavi-topology, the patch looks like this:

400: Invalid request

The last step is to apply the patch to the deployments:

kubectl patch deployments.apps -n karavi otel-collector --patch "$(cat otel-cert_manager.patch)"
kubectl patch deployments.apps -n karavi karavi-topology --patch "$(cat karavi-topology-cert_manager.patch)"

Go further

If you want to learn more about Karavi observability, you can check that other blog post or ask for help on the Dell container community forum.