Working with many different Kubernetes add-on deployments, the actual deployment and management of those across different clusters, on-prem and in the Cloud, can be challenging and sometimes frustrating.
Projectsveltos is a Kubernetes add-on controller that simplifies the deployment and management of different add-ons and applications across multiple clusters (on-prem, Cloud). Sveltos runs in a management cluster and programmatically deploys and manages add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports many add-on formats, including Helm charts, raw YAML/JSON, Kustomize, Carvel ytt, and Jsonnet.
In this blog post, we will demonstrate how easy and fast it is to deploy Sveltos on an RKE2 cluster with the help of ArgoCD, register two RKE2 Cluster API (CAPI) clusters and create a ClusterProfile to deploy Prometheus and Grafana Helm charts down the managed CAPI clusters.
Diagram
Prerequisites
For this demonstration, I have already installed ArgoCD on a central cluster. If you would like to learn more about the ArgoCD installation, go through the official documentation found here. If you would like to follow along, below you can find the lab details used.
- - - - - -+ - - - - - - - - - - - + - - - - - - - - - - -+
| Cluster Name | Type | Version |
+ - - - - - - -+ - - - - - - - - - - + - - - - - - - - - -+
| cluster04 | Management Cluster | RKE2 v1.26.11+rke2r1 |
| cluster12 | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1 |
| cluster13 | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1 |
+ - - - - - -+ - - - - - - - - - + - - - - - - - - - - - -+
Step 1: Deploy Sveltos as a Helm Chart cluster04
Sveltos can be deployed either as a manifest or as a Helm chart . For more information about the different installation options, check the link here. In my case, I chose to follow the GitOps approach and let ArgoCD deal with the comparison and synchronisation of the Git repository where the code to deploy Sveltos is stored, with the actual running application.
If you are unsure how to deploy Helm charts with ArgoCD, have a look here.
Verification
After we deploy Sveltos, we want to ensure everything is in a working and fully functional state. This can be achieved either from the ArgoCD UI or from the management cluster itself.
$ kubectl get pods -n projectsveltos
NAME READY STATUS RESTARTS AGE
access-manager-77c7c64477-ns8ml 2/2 Running 0 70s
addon-compliance-manager-7f449d884c-6kgqr 2/2 Running 0 69s
addon-controller-55d7d848ff-ps8l8 2/2 Running 0 70s
classifier-manager-67d6f67d5b-cgpr7 2/2 Running 0 70s
event-manager-69db45b65d-htz5l 2/2 Running 0 70s
hc-manager-5679c69dcc-z6s48 2/2 Running 0 70s
sc-manager-84dbd64fb4-6hwpf 2/2 Running 0 70s
shard-controller-56678bcf8c-zjbvc 2/2 Running 0 70s
Step 2: Install the Sveltosctl
The Sveltosctl, is the command-line interface (CLI) for Sveltos. This is an available option to query Sveltos resources and it is available as a Kubernetes pod or as a binary.
As I would like to register cluster12 and cluster13 to the Sveltos management cluster, the Sveltosctl as a binary will be used.
Step 3: Register CAPI Clusters with Sveltos
To register any cluster with Sveltos, you only need three things:
- A ServiceAccount for Sveltos and a kubeconfig associated with that account;
- A namespace where you want to register the external cluster;
- The Sveltosctl should point to the management cluster and then perform the 'sveltosctl register cluster' command.
Now, if you are unsure how to create a Service Account and an associated kubeconfig, do not worry. There is a script publicly available to create everything you need automatically.
Registration
$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster12 --kubeconfig=cluster12.yaml
$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster13 --kubeconfig=cluster13.yaml
From the commands above, we register cluster12 and cluster13 in the namespace projectsveltos. Of course, you can register the clusters to a namespace of your preference.
Verification
$ kubectl get sveltosclusters -n projectsveltos
NAME READY VERSION
cluster12 true v1.26.6+rke2r1
cluster13 true v1.26.6+rke2r1
Step 4: Cluster Labelling
To allow Sveltos to deploy and manage Kubernetes add-ons, the concept of ClusterProfile and cluster labelling comes into play. ClusterProfile is the CustomerResourceDefinition used to instruct Sveltos which add-ons to deploy on a set of clusters.
For this demonstration, we will set the label "env:prod" to both Sveltos clusters. The below commands are executed on the management cluster (cluster04).
$ kubectl get sveltosclusters -n projectsveltos --show-labels
NAME READY VERSION LABELS
cluster12 true v1.26.6+rke2r1 sveltos-agent=present
cluster13 true v1.26.6+rke2r1 sveltos-agent=present
$ kubectl label sveltosclusters cluster12 env=prod -n projectsveltos
$ kubectl label sveltosclusters cluster13 env=prod -n projectsveltos
$ kubectl get sveltosclusters -n projectsveltos --show-labels
NAME READY VERSION LABELS
cluster12 true v1.26.6+rke2r1 env=prod,sveltos-agent=present
cluster13 true v1.26.6+rke2r1 env=prod,sveltos-agent=present
Step 5: ClusterProfile for Grafana and Prometheus
The below ClusterProfile is an example of a Helm chart deployment of Grafana and Prometheus to Sveltos clusters with the label set to "env:prod".
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: prometheus-grafana
spec:
clusterSelector: env=prod
helmCharts:
- repositoryURL: https://prometheus-community.github.io/helm-charts
repositoryName: prometheus-community
chartName: prometheus-community/prometheus
chartVersion: 23.4.0
releaseName: prometheus
releaseNamespace: prometheus
helmChartAction: Install
- repositoryURL: https://grafana.github.io/helm-charts
repositoryName: grafana
chartName: grafana/grafana
chartVersion: 6.58.9
releaseName: grafana
releaseNamespace: grafana
helmChartAction: Install
Apply the ClusterProfile
$ kubectl apply -f "grafana_prometheus.yaml"
Once the ClusterProfile is applied to the management cluster, the expected result is to have the Grafana and the Prometheus deployment on both managed clusters.
Verification
$ sveltosctl show addons
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster12 | helm chart | prometheus | prometheus | 23.4.0 | 2023-12-17 11:25:20 +0100 CET | prometheus-grafana |
| projectsveltos/cluster12 | helm chart | grafana | grafana | 6.58.9 | 2023-12-17 11:25:23 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart | prometheus | prometheus | 23.4.0 | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart | grafana | grafana | 6.58.9 | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
Verification - Cluster12
$ kubectl get pods -n grafana
NAME READY STATUS RESTARTS AGE
pod/grafana-78764f9cd6-zsqdx 1/1 Running 0 81s
$ kubectl pods all -n prometheus
NAME READY STATUS RESTARTS AGE
pod/prometheus-alertmanager-0 1/1 Running 0 2m3s
pod/prometheus-kube-state-metrics-587bd996f6-l94zq 1/1 Running 0 2m3s
pod/prometheus-prometheus-node-exporter-khw75 1/1 Running 0 2m3s
pod/prometheus-prometheus-pushgateway-75986b9c9f-2ql7v 1/1 Running 0 2m3s
pod/prometheus-server-86c66b89c6-7xk9r 2/2 Running 0 2m3s
The same verification can be performed for cluster13.
Remove Label 'env:prod' cluster12
You might wonder what will happen if we remove the label 'env:prod' from either cluster12 or cluster13. The answer is that Sveltos will identify the missing label 'env:prod' and undeploy the Grafana and the Prometheus deployment from the cluster.
Let's have a look.
Remove Label
$ kubectl label sveltosclusters cluster12 env- -n projectsveltos
Verification
$ kubectl get sveltosclusters -n projectsveltos --show-labels
NAME READY VERSION LABELS
cluster12 true v1.26.6+rke2r1 sveltos-agent=present
cluster13 true v1.26.6+rke2r1 env=prod,sveltos-agent=present
sveltosctl show addons
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster13 | helm chart | grafana | grafana | 6.58.9 | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart | prometheus | prometheus | 23.4.0 | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
As expected, Sveltos removed the deployments. The same will happen if we register a new cluster and assign the label 'env:prod'. Sveltos will take care of the complete lifecycle of your Kubernetes deployments in a simple and straightforward manner.
👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you found it helpful.
The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.
Thanks for reading!
Top comments (0)