Deploying add-ons to a fleet of Kubernetes cluster
Sveltos is a set of Kubernetes controllers that run in the management cluster. From the management cluster, Sveltos can manage add-ons and applications on a fleet of managed Kubernetes clusters. It is a declarative tool to ensure that the desired state of an application is always reflected in the actual state of the Kubernetes managed clusters.
In a management cluster, each individual Kubernetes cluster is represented by a dedicated resource. Labels can be attached to those resources.
Sveltos configuration utilises a concept called a cluster selector. This selector essentially acts like a filter based on Kubernetes labels. By defining specific labels or combinations of labels, you can create a subset of clusters that share those characteristics.
Automated Kubernetes Resource Cleanup
Cleaner is a Kubernetes controller that automates resource management, keeping your cluster clean and efficient. It proactively identifies and removes (or updates) unused resources, optimizing resource utilization. Powerful features include:
- Flexible Scheduling: Schedule cleanup tasks at regular intervals.
- Precise Targeting: Select resources based on labels or use Lua scripting for complex criteria.
- Dry Run Mode: Test cleanup actions before making permanent changes.
- Integrated Notifications: Receive alerts about resource cleanup via Slack, Webex, MS Teams, or Discord.
Deploying k8s-cleaner across Kubernetes Clusters with Sveltos
By combining k8s-cleaner with Sveltos, you gain a powerful solution for maintaining clean, efficient, and secure Kubernetes clusters across your entire fleet. This approach simplifies management, reduces operational overhead, and optimizes resource utilization.
Lab Setup
Management cluster is created using Kind cluster. Then two Civo clusters, all with label env=fv :
+------------------------+-------------+-------------------------------------+
| Cluster Name | Version | Comments |
+------------------------+-------------+-------------------------------------+
| cluster1 | v1.29.2+k3s1| Civo 3 Node - Medium Standard |
| cluster2 | v1.28.7+k3s1| Civo 3 Node - Medium Standard |
+------------------------+-------------+-------------------------------------+
Step 1: Install Sveltos on Managament Cluster
For this demonstration, we will install Sveltos in the management cluster. Sveltos installation details can be found here.
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.34.1/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.34.1/manifest/default-classifier.yaml
Step 2: Register Civo Cluster with Sveltos
Create two Kubernetes clusters using Civo UI. Download the Kubeconfigs, then:
kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=fv
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=fv
Verify your Civo were successfully registered:
kubectl get sveltoscluster -n civo
NAME READY VERSION
cluster1 true v1.29.2+k3s1
cluster2 true v1.28.7+k3s1
Step 3: Create Deployment Configuration
This step showcases deploying k8s-cleaner with Sveltos across all clusters (labeled env=fv) to optimize resource utilization. We’ll leverage pre-configured ConfigMaps provided by the projectsveltos/demos repository:
install-cleaner.yaml: contains K8s-cleaner resources.
unused-configmaps.yaml: contains a Cleaner instance configured to identify ConfigMaps that are not currently used by any Pods in the cluster. This helps optimize resource utilization by removing unused configuration data.
pod-with-outdated-secret-data.yaml: contains a Cleaner instance configured to detect Pods that are still using old data from mounted Secrets. This scenario might occur if Pods haven’t been restarted after a Secret update. Cleaner can trigger a restart or other actions to ensure Pods use the latest secret data.
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/install-cleaner.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/unused-configmaps.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/pod-with-outdated-secret-data.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/clusterprofile-deploy-cleaner.yaml
We’ll use a Sveltos ClusterProfile named deploy-cleaner to target these deployments to all clusters with the label env=fv. This ensures Cleaner runs in all relevant clusters.
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
name: deploy-cleaner
spec:
clusterSelector:
matchLabels:
env: fv
policyRefs:
- name: install-cleaner
namespace: default
kind: ConfigMap
- name: pod-with-outdated-secret
namespace: default
kind: ConfigMap
- name: unused-configmaps
namespace: default
kind: ConfigMap
Using sveltosctl we can verify all resources have been deployed:
sveltosctl show addons
+---------------+-----------------------------------------------+----------------+----------------------------------------+---------+--------------------------------+-------------------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | PROFILES |
+---------------+-----------------------------------------------+----------------+----------------------------------------+---------+--------------------------------+-------------------------------+
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRole | | k8s-cleaner-proxy-role | N/A | 2024-07-15 15:40:37 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRoleBinding | | k8s-cleaner-proxy-rolebinding | N/A | 2024-07-15 15:40:39 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | :Service | projectsveltos | k8s-cleaner-controller-metrics-service | N/A | 2024-07-15 15:40:40 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apps:Deployment | projectsveltos | k8s-cleaner-controller | N/A | 2024-07-15 15:40:41 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apps.projectsveltos.io:Cleaner | | unused-configmaps | N/A | 2024-07-15 15:40:43 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apiextensions.k8s.io:CustomResourceDefinition | | cleaners.apps.projectsveltos.io | N/A | 2024-07-15 15:40:31 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apiextensions.k8s.io:CustomResourceDefinition | | reports.apps.projectsveltos.io | N/A | 2024-07-15 15:40:32 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRole | | k8s-cleaner-metrics-reader | N/A | 2024-07-15 15:40:36 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRoleBinding | | k8s-cleaner-controller-rolebinding | N/A | 2024-07-15 15:40:38 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | apps.projectsveltos.io:Cleaner | | list-pods-with-outdated-secret-data | N/A | 2024-07-15 15:40:42 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | :Namespace | | projectsveltos | N/A | 2024-07-15 15:40:30 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | :ServiceAccount | projectsveltos | k8s-cleaner-controller | N/A | 2024-07-15 15:40:33 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster1 | rbac.authorization.k8s.io:ClusterRole | | k8s-cleaner-controller-role | N/A | 2024-07-15 15:40:35 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | :Service | projectsveltos | k8s-cleaner-controller-metrics-service | N/A | 2024-07-15 15:40:27 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apps:Deployment | projectsveltos | k8s-cleaner-controller | N/A | 2024-07-15 15:40:28 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apiextensions.k8s.io:CustomResourceDefinition | | cleaners.apps.projectsveltos.io | N/A | 2024-07-15 15:40:19 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apiextensions.k8s.io:CustomResourceDefinition | | reports.apps.projectsveltos.io | N/A | 2024-07-15 15:40:20 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | :ServiceAccount | projectsveltos | k8s-cleaner-controller | N/A | 2024-07-15 15:40:22 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRole | | k8s-cleaner-metrics-reader | N/A | 2024-07-15 15:40:23 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRole | | k8s-cleaner-proxy-role | N/A | 2024-07-15 15:40:24 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRoleBinding | | k8s-cleaner-controller-rolebinding | N/A | 2024-07-15 15:40:25 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apps.projectsveltos.io:Cleaner | | unused-configmaps | N/A | 2024-07-15 15:40:30 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | :Namespace | | projectsveltos | N/A | 2024-07-15 15:40:19 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRole | | k8s-cleaner-controller-role | N/A | 2024-07-15 15:40:23 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | rbac.authorization.k8s.io:ClusterRoleBinding | | k8s-cleaner-proxy-rolebinding | N/A | 2024-07-15 15:40:26 +0200 CEST | ClusterProfile/deploy-cleaner |
| civo/cluster2 | apps.projectsveltos.io:Cleaner | | list-pods-with-outdated-secret-data | N/A | 2024-07-15 15:40:29 +0200 CEST | ClusterProfile/deploy-cleaner |
+---------------+-----------------------------------------------+----------------+----------------------------------------+---------+--------------------------------+-------------------------------+
Step 4: Create Deployment Configuration
We will use Kustomize with configMapGenerator to deploy an Nginx application.
git clone git@github.com:projectsveltos/demos.git
cd demos/k8s-cleaner/nginx
And deploy to both our Civo clusters:
kustomize build |KUBECONFIG=<CIVO CLUSTER1 KUBECONFIG> kubectl apply -f -
kustomize build |KUBECONFIG=<CIVO CLUSTER2 KUBECONFIG> kubectl apply -f -
Check if the deployments are running successfully on both clusters using kubectl get deployments -n k3s-home with the appropriate kubeconfig file for each cluster.
When we update any configuration files (index.html), the configMap created by configMapGenerator will have a new hash, and applying new configuration will ask the deployment to use the new configmap, but the old configmap (with the old hash string) will not removed. You can read more about this here.
Let’s we change the content of index.html file and redeploy kustomize output only to cluster2:
vim index.html
kustomize build |KUBECONFIG=<CIVO CLUSTER2 KUBECONFIG> kubectl apply -f -
Step 5: Visualize the unused ConfigMaps
In this step, we’ll leverage Sveltos, your cluster management platform, to identify unused ConfigMaps across your managed clusters. While Sveltos takes center stage for visualization, it’s important to understand the supporting role played by k8s-cleaner.This tool, deployed in each cluster, actively scans for unused ConfigMaps and generates reports listing these potential cleanup candidates.
Let’s configure Sveltos to automatically collect these reports from each managed cluster.
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/k8s-cleaner/clusterhealthcheck.yaml
Then using sveltosctl we can see which ConfigMap is unused:
sveltosctl show resources
+---------------+----------------------------------+-----------+-------------------+------------------------------------------+
| CLUSTER | GVK | NAMESPACE | NAME | MESSAGE |
+---------------+----------------------------------+-----------+-------------------+------------------------------------------+
| civo/cluster2 | apps.projectsveltos.io/v1alpha1, | | unused-configmaps | Affected |
| | Kind=Report | | | Resources:k3s-home/nginx-html-kkkff7h9h5 |
| | | | | |
+---------------+----------------------------------+-----------+-------------------+------------------------------------------+
Conclusion: Automated Cluster Cleanup at Scale with Sveltos and k8s-cleaner
This guide demonstrated how to leverage Sveltos and k8s-cleaner to achieve automated Kubernetes cluster cleanup across a fleet of clusters. By combining these tools, you gain a centralized and efficient solution for:
Optimizing resource utilization: k8s-cleaner proactively identifies and removes unused resources like ConfigMaps, freeing up valuable cluster resources.
Simplified management: Sveltos provides a central dashboard for visualizing unused resources across all clusters, eliminating the need to manage each cluster individually.
👏 Support those projects
Every contribution counts! If you enjoyed this article, check out:
You can star 🌟 the project if you find it helpful.
The GitHub repos are a great resource for getting started with the project. It contains the code, documentation, and many more examples.
Thanks for reading!
Contact Information
If you have questions or would like to have a friendly chat, just feel free to add me to your LinkedIn network!
Top comments (0)