Kubernetes Add-ons: Visibility and Control

When managing a fleet of Kubernetes clusters, visibility of where and which Kubernetes add-ons are deployed is important for a number of reasons. It can help you ensure that your clusters are consistent and compliant with your organization’s standards, troubleshoot problems more quickly, and plan for future changes.

Sveltos is a Kubernetes add-on controller that provides insights into the add-ons installed on each cluster and how their configurations have evolved over time.

Setup

The setup used for this tutorial consists of the following:

• A management cluster with Sveltos and ClusterAPI.
• A GKE cluster.
• A ClusterAPI-powered cluster with Docker as the infrastructure provider.
• A ClusterAPI-powered cluster with Hetzner as the infrastructure provider.

All YAMLs used can be found here.

Sveltosctl can be used to display add-ons and configuration changes over time.

The Sveltos configuration used in this tutorial consists of the following three ClusterProfiles:

• ClusterProfile kyverno: This ClusterProfile instructs Sveltos to deploy the Kyverno Helm chart version v3.0.1 and a Kyverno ClusterPolicy that disallows using the latest tag in each managed cluster with the label env: production.

• ClusterProfile prometheus-grafana: This ClusterProfile instructs Sveltos to deploy the Prometheus and Grafana Helm charts in each managed cluster with the label team: devops.

• ClusterProfile clusterprofile-nginx: This ClusterProfile instructs Sveltos to deploy the Nginx Helm chart in each managed cluster with the label tier: frontend.

Display deployed add-ons

kubectl exec -it -n projectsveltos sveltosctl-0 -- ./sveltosctl show addons                    
+----------------------------+--------------------------+------------+-----------------------------+---------+-------------------------------+----------------------+
|          CLUSTER           |      RESOURCE TYPE       | NAMESPACE  |            NAME             | VERSION |             TIME              |   CLUSTER PROFILES   |
+----------------------------+--------------------------+------------+-----------------------------+---------+-------------------------------+----------------------+
| docker/clusterapi-workload | helm chart               | kyverno    | kyverno-latest              | 3.0.1   | 2023-09-03 06:57:02 -0700 PDT | kyverno              |
| docker/clusterapi-workload | helm chart               | prometheus | prometheus                  | 23.4.0  | 2023-09-03 07:06:44 -0700 PDT | prometheus-grafana   |
| docker/clusterapi-workload | helm chart               | grafana    | grafana                     | 6.58.9  | 2023-09-03 07:06:53 -0700 PDT | prometheus-grafana   |
| docker/clusterapi-workload | kyverno.io:ClusterPolicy |            | disallow-latest-tag         | N/A     | 2023-09-03 07:21:36 -0700 PDT | kyverno              |
| gke/cluster1               | helm chart               | kyverno    | kyverno-latest              | 3.0.1   | 2023-09-03 06:57:03 -0700 PDT | kyverno              |
| gke/cluster1               | kyverno.io:ClusterPolicy |            | disallow-latest-tag         | N/A     | 2023-09-03 07:21:38 -0700 PDT | kyverno              |
| hetzner/hetzner            | helm chart               | kyverno    | kyverno-latest              | 3.0.1   | 2023-09-03 06:57:02 -0700 PDT | kyverno              |
| hetzner/hetzner            | helm chart               | nginx      | nginx-latest                | 0.17.1  | 2023-09-03 07:11:19 -0700 PDT | clusterprofile-nginx |
| hetzner/hetzner            | kyverno.io:ClusterPolicy |            | disallow-latest-tag         | N/A     | 2023-09-03 07:21:36 -0700 PDT | kyverno              |
+----------------------------+--------------------------+------------+-----------------------------+---------+-------------------------------+----------------------+

At any point in time, Sveltos can show:

• The list of deployed add-ons and applications in each cluster.
• The reason each of the add-ons and applications are deployed, which is the ClusterProfile instance that deployed them.
• The time of the last deployment.
• For Helm charts, the deployed version.

Sveltos can also be used as a centralised location for viewing a summary of resources. More info can be found here.

Preview the effects of a change

When you’re about to make changes to your configuration, it’s important to be sure of the results. You don’t want to risk causing any unwanted side effects. That’s where the Sveltos DryRun syncMode configuration comes in.

The DryRun syncMode configuration allows you to launch a simulation of all the operations that would normally be executed in a live run. This means that you can see exactly what changes will be made to your Kubernetes clusters without actually making any changes.

To use the DryRun syncMode configuration, you need to add the following to your ClusterProfile definition:

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: clusterprofile-nginx
spec:
  syncMode: DryRun
  ...

Once you’ve updated your ClusterProfile with the DryRun syncMode configuration, you can check the status of the dry run workflow to see a detailed list of all the potential changes that would have been made to each matching cluster. This allows you to carefully inspect and validate these changes before deploying the new ClusterProfile configuration.

kubectl exec -it -n projectsveltos sveltosctl-0 -- ./sveltosctl show dryrun
+-----------------+---------------+-----------+--------------+---------+--------------------------------+----------------------+
|     CLUSTER     | RESOURCE TYPE | NAMESPACE |     NAME     | ACTION  |            MESSAGE             |   CLUSTER PROFILE    |
+-----------------+---------------+-----------+--------------+---------+--------------------------------+----------------------+
| hetzner/hetzner | helm release  | nginx     | nginx-latest | Upgrade | Current version: "0.17.1".     | clusterprofile-nginx |
|                 |               |           |              |         | Would move to version:         |                      |
|                 |               |           |              |         | "0.18.1"                       |                      |
+-----------------+---------------+-----------+--------------+---------+--------------------------------+----------------------+

Here are some of the benefits of using the DryRun syncMode configuration:

• It allows you to safely test changes to your ClusterProfile without actually making any changes to your clusters.
• It helps you to avoid causing unwanted side effects.
• It allows you to carefully inspect and validate changes before deploying them.

Display Configuration Changes Over Time

Sveltos Snapshot allows administrators to:

• Take live snapshots of the running Sveltos configuration.
• Schedule recurring snapshots.
• Store snapshots in a versioned format.
• View any snapshot configuration, including the differences between snapshots.
• Rollback to any previous configuration snapshot, either fully or partially.

For example, an administrator could take a snapshot of the Sveltos configuration with the system in its current state. Then, they could make some changes to the Kyverno ClusterPolicy, such as modifying the content of an existing ClusterPolicy or adding a new ClusterPolicy. By comparing two snapshots, the administrator could see how the configuration has changed.

kubectl exec -it -n projectsveltos sveltosctl-0 -- ./sveltosctl snapshot diff --snapshot=hourly --from-sample=2023-09-03:07:18:00 --to-sample=2023-09-03:07:23:00
+----------------------------------+--------------------------+-----------+-----------------------------+----------+--------------------------------+
|             CLUSTER              |      RESOURCE TYPE       | NAMESPACE |            NAME             |  ACTION  |            MESSAGE             |
+----------------------------------+--------------------------+-----------+-----------------------------+----------+--------------------------------+
| docker/capi--clusterapi-workload | kyverno.io/ClusterPolicy |           | disallow-empty-ingress-host | added    |                                |
| docker/capi--clusterapi-workload | kyverno.io/ClusterPolicy |           | disallow-latest-tag         | modified | use --raw-diff option to see   |
|                                  |                          |           |                             |          | diff                           |
| gke/sveltos--cluster1            | kyverno.io/ClusterPolicy |           | disallow-empty-ingress-host | added    |                                |
| gke/sveltos--cluster1            | kyverno.io/ClusterPolicy |           | disallow-latest-tag         | modified | use --raw-diff option to see   |
|                                  |                          |           |                             |          | diff                           |
| hetzner/capi--hetzner            | kyverno.io/ClusterPolicy |           | disallow-empty-ingress-host | added    |                                |
| hetzner/capi--hetzner            | kyverno.io/ClusterPolicy |           | disallow-latest-tag         | modified | use --raw-diff option to see   |
|                                  |                          |           |                             |          | diff                           |
+----------------------------------+--------------------------+-----------+-----------------------------+----------+--------------------------------+

kubectl exec -it -n projectsveltos sveltosctl-0 -- ./sveltosctl snapshot diff --snapshot=hourly --from-sample=2023-09-03:07:18:00 --to-sample=2023-09-03:07:23:00 --raw-diff --namespace=docker
--- kyverno.io/ClusterPolicy disallow-latest-tag from /collection/snapshot/hourly/2023-09-03:07:18:00
+++ kyverno.io/ClusterPolicy disallow-latest-tag from /collection/snapshot/hourly/2023-09-03:07:23:00
@@ -8,11 +8,6 @@
     policies.kyverno.io/minversion: 1.6.0
     policies.kyverno.io/severity: medium
     policies.kyverno.io/subject: Pod
-    policies.kyverno.io/description: >-
-      The ':latest' tag is mutable and can lead to unexpected errors if the
-      image changes. A best practice is to use an immutable tag that maps to
-      a specific version of an application Pod. This policy validates that the image
-      specifies a tag and that it is not called `latest`.
 spec:
   validationFailureAction: audit
   background: true

Here is a more detailed explanation of each of the features:

• Live snapshots: Live snapshots capture the current state of the Sveltos configuration at a point in time. This can be useful for debugging or troubleshooting problems.
• Recurring snapshots: Recurring snapshots can be scheduled to take place at regular intervals. This can be useful for tracking changes to the configuration over time.
• Versioned storage: Snapshots are stored in a versioned format, which means that each snapshot is assigned a unique identifier. This makes it easy to track changes to the configuration and to revert to a previous snapshot if necessary.
• Full viewing: Any snapshot configuration can be viewed in full, including the differences between snapshots. This makes it easy to see what changes have been made to the configuration.
• Rollback: Snapshots can be rolled back to any previous configuration snapshot, either fully or partially. This can be useful for undoing changes that have been made to the configuration.

👏 Support this project
If you enjoyed this article, please check out the Sveltos GitHub repo and star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with Sveltos. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.
If you have any feedback, bugs, or PRs, please feel free to contribute to the project. Your contributions will help make Sveltos even better.

And finally, please help spread the word about Sveltos by starring the repo or sharing it with your friends and colleagues. Thank you for reading!