DEV Community: Eleni Grosdouli

5-Step Approach: ProjectSveltos Event Framework for Kubernetes Deployment with Cilium Gateway API

Eleni Grosdouli — Mon, 19 Feb 2024 15:39:44 +0000

Introduction

In the previous post, we demonstrated how easy it is to deploy ArgoCD with the Cilium Gateway API and move away from the Kubernetes Ingress. In today’s post, we will explore how the Projectsveltos Event Framework can be utilised to seamlessly perform the same deployment based on an event.

The Sveltos Event Framework was designed to automate the deployment of Kubernetes add-ons triggered on specific cluster events. This functionality proves invaluable in managing environments spanning across multiple clusters.

Sveltos supports an event-driven workflow with the below steps:

Define what the event is
Select the clusters to watch for such events
Define an event trigger. Which add-ons/applications to deploy when the events occur

Once we identify the event we want Sveltos to watch for, we can deploy the EventSource and EventTrigger Kubernetes CRDs (Custom Resource Definitions). More information about these resources can be found later on.

Diagram

Prerequisites

To follow along, it is strongly advised to take a look at a previous post detailing the deployment of Sveltos on a cluster. To install Sveltos, just follow the instructions outlined in steps 1 and 2 of the mentioned post. Additionally, as we will use the Cilium Gateway API, have a look at the previous post to enable this capability on an RKE2 cluster.

Lab Setup

— — — — — -+ — — — — — — — — — — + — — — — — — — — — — — — -+
| Cluster Name | Type | Version |
+ — — — — — — -+ — — — — — — — — — — + — — — — — — — — — — -+
| rke2-mgmt01 | Management Cluster | RKE2 v1.26.12+rke2r1 |
| rke2-sles-demo | Managed Cluster | RKE2 v1.26.12+rke2r1|
+ — — — — — -+ — — — — — — — — — — + — — — — — — — — — — — — -+

- — — — — — -+ — — — — -+
| Deployment | Version |
+ — — — — — — -+ — — — — -+
| ArgoCD | v2.9.3 |
| Cilium | v1.14.5 |
| GatewayAPI | v0.7.0 |
+ — — — — — — -+ — — — — -+

Step 1: Register and label the RKE2 cluster with Sveltos

Once the RKE2 cluster is in a “Running” state, we will use the sveltosctl to register it. For the registration, we need three things: a service account, a kubeconfig associated with that account and a namespace. If you are unsure how to create a Service Account and an associated kubeconfig, there is a script publicly available to help you out.

Registration

$ sveltosctl register cluster --namespace=projectsveltos --cluster=rke2-sles-demo --kubeconfig=rke2-sles-demo.yaml

Verification

$ kubectl get sveltoscluster -n projectsveltos
NAME               READY   VERSION
rke2-sles-demo   true    v1.26.12+rke2r1

Assing Label

$ kubectl label sveltoscluster rke2-sles-demo env=staging -n projectsveltos

Verification

$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME             READY   VERSION           LABELS
rke2-sles-demo   true    v1.26.12+rke2r1   env=staging,sveltos-agent=present

For the demonstration, we will use the label ‘env=staging’ to specify the cluster we want to deploy the Gateway and the HTTPRoute resources.

Step 2: Pre-Work (management cluster)

Before we move on with the Gateway API implementation, we need to create additional Kubernetes resources for the managed cluster (rke2-sles-demo).

ArgoCD namespace
ArgoCD TLS Secret
Cilium IP Pool
Cilium GatewayClass
Gateway
HTTPRoute

To utilise the power of Sveltos, we will create the above resources in the management cluster as a Secret and configMap resource and use the Sveltos ClusterProfile to deploy them to the managed cluster.

More information on how to create the above resources can be found here.

ArgoCD Namespace

If the namespace does not exist in the managed cluster, Sveltos will create it!

ArgoCD TLS Secret

apiVersion: v1
data:
  tls.crt: {base64 encoded .crt}
  tls.key: {base64 encoded .key}
kind: Secret
metadata:
  name: argocd-server-tls
  namespace: argocd
type: kubernetes.io/tls

$ kubectl create secret generic argocd-server-tls --from-file=argocd_server_tls.yaml --type=addons.projectsveltos.io/cluster-profile

Cilium IP Pool

---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "rke2-pool"
spec:
  cidrs:
  - cidr: "10.10.10.0/24"

$ kubectl create configmap ipampool --from-file=ipam-pool.yaml

Cilium GatewayClass

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller

$ kubectl create configmap gatewayclass --from-file=gatewayclass.yaml

Gateway

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: argocd
  namespace: argocd
spec:
  gatewayClassName: cilium
  listeners:
  - hostname: argocd.example.com
    name: argocd-example-com-http
    port: 80
    protocol: HTTP
  - hostname: argocd.example.com
    name: argocd-example-com-https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        name: argocd-server-tls

$ kubectl create configmap gateway --from-file=gateway.yaml

HTTPRoute

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  creationTimestamp: null
  name: argocd
  namespace: argocd
spec:
  hostnames:
  - argocd.example.com
  parentRefs:
  - name: argocd
  rules:
  - backendRefs:
    - name: argocd-server
      port: 80
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents: []

$ kubectl create configmap gateway --from-file=httproute.yaml

Verification

$ kubectl get secret
NAME                TYPE                                       DATA   AGE
argocd-server-tls   addons.projectsveltos.io/cluster-profile   1      20m

$ kubectl get cm
NAME               DATA   AGE
gateway            1      35s
gatewayclass       1      20m
httproute          1      3s
ipam-pool          1      22m
kube-root-ca.crt   1      141m

ClusterProfile

---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
 name: cilium-gateway-api
spec:
 clusterSelector: env=staging
 syncMode: Continuous
 policyRefs:
 - name: ipampool
   namespace: default
   kind: ConfigMap
 - name: gatewayclass
   namespace: default
   kind: ConfigMap
 - name: argocd-server-tls
   namespace: default
   kind: Secret

The ClusterProfile definition will match the managed cluster with the label selector set to ‘env=staging’ and deploy the Cilium IP Pool, the GatewayClass and the ArgoCD TLS secret.

Apply ClusterProfile

$ kubectl apply -f "clusterprofile.yaml"

Verification

$ sveltosctl show addons

+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|            CLUSTER            |             RESOURCE TYPE              | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/rke2-sles-demo | cilium.io:CiliumLoadBalancerIPPool     |           | rke2-pool         | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:GatewayClass |           | cilium            | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | :Secret                                | argocd    | argocd-server-tls | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+

Step 3: Define the Event Framework (management cluster)

As mentioned at the beginning of the post, we will use the EventSource to define which events Sveltos will watch for and the EventTrigger resource to perform actions based on an event.

EventSource

---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventSource
metadata:
 name: http-https-service
spec:
 collectResources: true
 resourceSelectors:
 - group: ""
   version: "v1"
   kind: "Service"
   namespace: argocd
   evaluate: |
     function evaluate()
       hs = {}
       hs.matching = false
       if obj.spec.ports ~= nil then
         for _,p in pairs(obj.spec.ports) do
           if p.port == 80 or p.port == 443 then
             hs.matching = true
           end
         end
       end
       return hs
     end

The EventSource uses the Lua language to search for any services with ports set to 80 or 443 in the ‘argocd’ namespace. More examples can be found here.

EventTrigger

---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventTrigger
metadata:
 name: service-network-policy
spec:
 sourceClusterSelector: env=staging
 eventSourceName: http-https-service
 oneForEvent: true
 policyRefs:
 - name: httproute
   namespace: default
   kind: ConfigMap
 - name: gateway  
   namespace: default
   kind: ConfigMap

Apply Resources

$ kubectl apply -f "eventsource.yaml"
eventsource.lib.projectsveltos.io/loadbalancer-service created

$ kubectl apply -f "eventtrigger.yaml"
eventtrigger.lib.projectsveltos.io/service-network-policy created

Once an event is spotted by Sveltos, the HTTPRoute and Gateway Kubernetes add-ons will get deployed on the managed cluster with the label selector set to ‘env=staging’. In our case, the cluster ‘rke2-sles-demo’ will match the ClusterSelector and will automatically get the specified resources.

Step 4: Deploy ArgoCD Manifests (managed cluster)

Now, it is time to deploy the ArgoCD manifests and wait Sveltos to notice the event and trigger the deployment of the HTTPRoute and Gateway resources in the ‘argocd’ namespace.

$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Step 5: Verification and Tests

As we have installed the ArgoCD manifests, the ‘argocd-server’ service should have been created in the ‘argocd’ namespace. That means the Sveltos EventSource should have been activated and triggered the deployment of the HTTPRoute and Gateway Kubernetes resources.

Sveltos Verification


$ sveltosctl show addons
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|            CLUSTER            |             RESOURCE TYPE              | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/rke2-sles-demo | cilium.io:CiliumLoadBalancerIPPool     |           | rke2-pool         | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:GatewayClass |           | cilium            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | :Secret                                | argocd    | argocd-server-tls | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:HTTPRoute    | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-1lbzxnp4cs96gpo38g8b |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:Gateway      | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-1lbzxnp4cs96gpo38g8b |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:HTTPRoute    | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-i0nmkq3ebsvpis8ubd3s |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:Gateway      | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-i0nmkq3ebsvpis8ubd3s |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+

$ sveltosctl show usage
+----------------+--------------------+------------------------------+-------------------------------+
| RESOURCE KIND  | RESOURCE NAMESPACE |        RESOURCE NAME         |           CLUSTERS            |
+----------------+--------------------+------------------------------+-------------------------------+
| ClusterProfile |                    | cilium-gateway-api           | projectsveltos/rke2-sles-demo |
| ClusterProfile |                    | sveltos-fvc3vr84yqusfzwpobcl | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | gateway                      | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | ipampool                     | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | gatewayclass                 | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | httproute                    | projectsveltos/rke2-sles-demo |
+----------------+--------------------+------------------------------+-------------------------------+

Managed Cluster Verification

$ kubectl get gateway,httproute -n argocd
NAME                                       CLASS    ADDRESS        PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/argocd   cilium   10.10.10.147   True         78s

NAME                                         HOSTNAMES                AGE
httproute.gateway.networking.k8s.io/argocd   ["argocd.example.com"]   78s

Test ArgoCD Accessibility

curl -ki https://argocd.example.com
HTTP/1.1 200 OK
accept-ranges: bytes
content-length: 788
content-security-policy: frame-ancestors 'self';
content-type: text/html; charset=utf-8
vary: Accept-Encoding
x-frame-options: sameorigin
x-xss-protection: 1
date: Sun, 11 Feb 2024 18:59:57 GMT
x-envoy-upstream-service-time: 0
server: envoy

<!doctype html><html lang="en"><head><meta charset="UTF-8"><title>Argo CD</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32"/><link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16"/><link href="assets/fonts.css" rel="stylesheet"><script defer="defer" src="main.9ecae91d8fd1deedf944.js"></script></head><body><noscript><p>Your browser does not support JavaScript. Please enable JavaScript to view the site. Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.</p></noscript><div id="app"></div></body><script defer="defer" src="extensions.js"></script></html>

Note: As we use a self-sign certificate the argocd-cmd-params-cm ConfigMap in the argocd namespace should include the following setting server.insecure: “true”.

As expected, Sveltos took care of the complete lifecycle of the different Kubernetes deployments in a simple and straightforward manner! The example might seem simplistic but think about it on a bigger scale with hundreds of clusters across a multicloud setup. Imagine the flexibility of setting Sveltos labels to a cluster, defining the event source to trigger an action and having a Kubernetes deployment on a cluster. In the next post, we will demonstrate the power of Events in a cross multi-cluster environment.

👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

ArgoCD Deployment on RKE2 with Cilium Gateway API

Eleni Grosdouli — Mon, 19 Feb 2024 15:24:34 +0000

Introduction

It has already been a couple of years since the Kubernetes Ingress was defined as a “frozen” feature while further development will be added to the Gateway API.

After initial exposure to the Cilium Gateway API docs and the interactive lab session, it sounded promising to move the ArgoCD deployment from the Kubernetes Ingress to the Cilium Gateway API. The purpose of the blog post is to illustrate how easy it is to move the ArgoCD installation to the Cilium Gateway API. For this demonstration, the Gateway and the HTTPRoute have been created in the argocd namespace.

Diagram

Lab Setup

- — — — — — — + — — — — — — — — — -+ — — — — — — — — —+
| Cluster Name | Type | Version |
+ — — — — — — — + — — — — — — — — — -+ — — — — — — — — +
| rke2-test01| Test Management Cluster| RKE2 v1.26.12+rke2r1 |
+ — — — — — — — + — — — — — — — — — -+ — — — — — — — — +

- — — — — — -+ — — — — -+
| Deployment | Version |
+ — — — — — — -+ — — — — -+
| ArgoCD | v2.9.3 |
| Cilium | v1.14.5 |
| GatewayAPI | v0.7.0 |
+ — — — — — — -+ — — — — -+

Step 1: Deploy RKE2 Cluster with Cilium CNI

Before diving in, it is a good idea to checkout the RKE2 official documentation on Kubernetes Networking and the Cilium documentation. Also, take a peek at the prerequisites for deploying the Gateway API deployment.

RKE2 Pre-work

$ cat /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: 0644
tls-san:
  - {Master Node hostname}
token: {Your Token}
cni: cilium
disable-kube-proxy: true
etcd-expose-metrics: false

$ cat /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    image:
      tag: v1.14.5
    kubeProxyReplacement: strict
    k8sServiceHost: 127.0.0.1
    k8sServicePort: 6443
    operator:
      replicas: 1
    gatewayAPI:
      enabled: true

According to the Cilium documentation, to enable the Gateway API, we need at least the 1.14.5 Cilium Helm chart with the kubeProxyReplacement value set to true and the gatewayAPI inside the Helm chart definition set to enabled: true.

Once the remaining steps for the RKE2 installation are complete, we will have a two node RKE2 cluster with Cilium as a CNI.

Since Kubernetes v.1.26.x does not come with the Gateway API CRDs included, we will need to deploy them manually and let the Cilium containers restart until everything is in a “Running” state.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

Note: According to the RKE2 documentation, RKE2 v1.26.12 does not officially support Cilium v1.14.5. The latest supported version is v1.14.4. However, during the demo setup, we did not encounter any issues.

Verification

$ kubectl get nodes -o wide
NAME            STATUS   ROLES                       AGE    VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                              KERNEL-VERSION              CONTAINER-RUNTIME
rke2-master01   Ready    control-plane,etcd,master   111m   v1.26.12+rke2r1                  <none>        SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.11-k3s2
rke2-worker01   Ready    <none>                      98m    v1.26.12+rke2r1                  <none>        SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.11-k3s2

$ kubectl get pods -n kube-system | grep -i cilium
cilium-k9vhf                                            1/1     Running     0              111m
cilium-lc7jn                                            1/1     Running     0              99m
cilium-operator-548958b5bf-nc95q                        1/1     Running     5 (108m ago)   111m
helm-install-rke2-cilium-tp5l6                          0/1     Completed   0              112m

$ kubectl -n kube-system get daemonset cilium -o jsonpath="{.spec.template.spec.containers[0].image}"
rancher/mirrored-cilium-cilium:v1.14.5

Step 2: Install ArgoCD

We will follow the official “Getting Started” guide found here, and use the manifest installation option.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The code above will create the argocd Kubernetes namespace and deploy the latest stable manifest. If you would like to install a specific manifest, have a look here.

Verification

$ kubectl get pods,svc -n argocd
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0                     1/1     Running   0          82m
pod/argocd-applicationset-controller-6b67b96c9f-7szsr   1/1     Running   0          82m
pod/argocd-dex-server-c9d4d46b5-mdf67                   1/1     Running   0          82m
pod/argocd-notifications-controller-6975bff68d-ltbkc    1/1     Running   0          82m
pod/argocd-redis-7d8d46cc7f-2br7f                       1/1     Running   0          82m
pod/argocd-repo-server-59f5479b7-dfg9x                  1/1     Running   0          82m
pod/argocd-server-547bf65466-68554                      1/1     Running   0          58m

NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
service/argocd-applicationset-controller          ClusterIP      10.43.98.162    <none>         7000/TCP,8080/TCP            82m
service/argocd-dex-server                         ClusterIP      10.43.71.44     <none>         5556/TCP,5557/TCP,5558/TCP   82m
service/argocd-metrics                            ClusterIP      10.43.162.177   <none>         8082/TCP                     82m
service/argocd-notifications-controller-metrics   ClusterIP      10.43.55.157    <none>         9001/TCP                     82m
service/argocd-redis                              ClusterIP      10.43.62.79     <none>         6379/TCP                     82m
service/argocd-repo-server                        ClusterIP      10.43.224.205   <none>         8081/TCP,8084/TCP            82m
service/argocd-server                             ClusterIP      10.43.166.25    <none>         80/TCP,443/TCP               82m
service/argocd-server-metrics                     ClusterIP      10.43.165.222   <none>         8083/TCP                     82m

Step 3: Pre-Work

Before we move on with the Gateway API implementation, we need to create additional Kubernetes resources.

Argocd TLS Secret

$ kubectl create secret tls argocd-server-tls -n argocd --key=argocd-key.pem --cert=argocd.example.com.pem

The above assumes that we have already created a private/public key pair via an available utility. Also, keep in mind that the TLS secret name should be argocd-server-tls as it will be used at a later point.

Cilium IP Pool

In our lab environment, we do not have a tool to hand over Loadbalancer IP addresses. Therefore, we will use the Cilium LoadBalancer IP Address Management (LB IPAM).

$ cat ipam-pool.yaml
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "rke2-pool"
spec:
  cidrs:
  - cidr: "10.10.10.0/24"

Verification

$ kubectl apply -f "ipam-pool.yaml"

$ kubectl get ippool
NAME        DISABLED   CONFLICTING   IPS AVAILABLE   AGE
rke2-pool   false      False         253             79m

Cilium GatewayClass

If the GatewayClass resource is not present in the cluster, we have to create one for Cilium. The resource will be used in a later step and while deploying the Gateway. The GatewayClass is a template that lets infrastructure providers offer different types of Gateways.

$ cat gatewayclass.yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller

Verification

$ kubectl apply -f "gatewayclass.yaml"

$ kubectl get gatewayclass
NAME     CONTROLLER                     ACCEPTED   AGE
cilium   io.cilium/gateway-controller   True       104m

Step 4: Create a Gateway and an HTTPRoute Resources

Gateway

The Gateway is an instance of the GatewayClass created above.

$ cat argocd_gateway.yaml

1 ---
2 apiVersion: gateway.networking.k8s.io/v1beta1
3 kind: Gateway
4 metadata:
5   name: argocd
6   namespace: argocd
7 spec:
8   gatewayClassName: cilium
9   listeners:
10   - hostname: argocd.example.com
11     name: argocd-example-com-http
12     port: 80
13     protocol: HTTP
14   - hostname: argocd.example.com
15     name: argocd-example-com-https
16     port: 443
17     protocol: HTTPS
18  tls:
19    certificateRefs:
20    - kind: Secret
21      name: argocd-server-tls

Line 3: We define the kind Resource to Gateway

Line 6: We set the namespace to argocd

Line 8: We use the name of the GatewayClass created in the previous step

Line 9: We define the listeners for the ArgoCD server

Line 21: We define the TLS secret name created in the previous step

Note: In the definition above we use the .example.com as the Domain however, the value should be replaced with a valid Domain name.

HTTP Route

The HTTPRoute is used to distribute multiple HTTP requests. For example, based on the PathPrefix.

$ cat argocd_http_route.yaml

1 ---
2 apiVersion: gateway.networking.k8s.io/v1beta1
3 kind: HTTPRoute
4 metadata:
5   creationTimestamp: null
6   name: argocd
7   namespace: argocd
8 spec:
9   hostnames:
10   - argocd.example.com
11   parentRefs:
12   - name: argocd
13   rules:
14   - backendRefs:
15     - name: argocd-server
16       port: 80
17     matches:
18     - path:
19         type: PathPrefix
20         value: /
21 status:
22   parents: []

Line 10: We set the hostname we want the ArgoCD Server to get exposed to

Line 15: We define the name of the ArgoCD server service

Apply the Kubernetes Resources

$ kubectl apply -f argocd_gateway.yaml,argocd_http_route.yaml


$ kubectl get gateway,httproute -n argocd
NAME                                       CLASS    ADDRESS        PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/argocd   cilium   10.10.10.173   True         9s

NAME                                         HOSTNAMES                AGE
httproute.gateway.networking.k8s.io/argocd   ["argocd.example.com"]   9s

Step 5: Test Time

We want to see if everything works as expected and whether we are able to access the ArgoCD server with the Cilium Gateway API. Let’s perform a CURL request.

$ curl -kv https://argocd.example.com
*   Trying 10.10.10.173:443...
* Connected to argocd.example.com (10.10.10.173) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: O=mkcert development certificate; OU=root@server
*  start date: Feb  2 07:11:49 2024 GMT
*  expire date: May  2 07:11:49 2026 GMT
*  issuer: O=mkcert development CA; OU=root@server; CN=mkcert root@server
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: argocd.example.com
> User-Agent: curl/8.0.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 307 Temporary Redirect
< content-type: text/html; charset=utf-8
< location: https://argocd.example.com/
< date: Fri, 02 Feb 2024 07:58:21 GMT
< content-length: 63
< x-envoy-upstream-service-time: 0
< server: envoy
< 
<a href="https://argocd.example.com/">Temporary Redirect</a>.

* Connection #0 to host argocd.example.com left intact

From the above, it is visible that we are experiencing a well-known issue with 307 redirects. To resolce this, we will need to disable the TLS on the API server. This involves modifying the argocd-cmd-params-cm ConfigMap in the argocd namespace and setting the server.insecure: “true”. More information can be found here.

Once the changes are performed we need to restart the argocd-server deployment for the changes to take effect.

$ kubectl rollout restart deploy argocd-server -n argocd

$ kubectl rollout status deploy argocd-server -n argocd

Waiting for deployment "argocd-server" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "argocd-server" rollout to finish: 1 old replicas are pending termination...
deployment "argocd-server" successfully rolled out

Let us try once again.

$ curl -ki https://argocd.example.com
HTTP/1.1 200 OK
accept-ranges: bytes
content-length: 788
content-security-policy: frame-ancestors 'self';
content-type: text/html; charset=utf-8
vary: Accept-Encoding
x-frame-options: sameorigin
x-xss-protection: 1
date: Fri, 02 Feb 2024 13:03:45 GMT
x-envoy-upstream-service-time: 0
server: envoy

<!doctype html><html lang="en"><head><meta charset="UTF-8"><title>Argo CD</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32"/><link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16"/><link href="assets/fonts.css" rel="stylesheet"><script defer="defer" src="main.f14bff1ed334a13aa8c2.js"></script></head><body><noscript><p>Your browser does not support JavaScript. Please enable JavaScript to view the site. Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.</p></noscript><div id="app"></div></body><script defer="defer" src="extensions.js"></script></html>

Great, we received a 200 OK status message!

Note: The -v short option in the CURL request stands for --verbose, the -k short option stands for --insecure, and the -i short option is for --include.

The next steps will be to test the above deployment with the latest Cilium version and the Gateway API v.1.0.0.

Resources

Cilium Gateway API Lab: https://isovalent.com/labs/gateway-api/
Cilium Advanced Gateway API Lab: https://isovalent.com/labs/advanced-gateway-api-use-cases/
Migrate from Ingress to Gateway: https://docs.cilium.io/en/v1.14/network/servicemesh/ingress-to-gateway/ingress-to-gateway/
RKE2 Installation Methods: https://docs.rke2.io/install/methods

Thanks for reading!

5-Step Approach: Dry Run Kubernetes Resources with ProjectSveltos

Eleni Grosdouli — Sat, 20 Jan 2024 18:26:22 +0000

Introduction

Independent if you are a DevOps engineer or a Kubernetes administrator, we have all been in a position where we had to update a Kubernetes deployment to a later version in a Production environment. Even if the deployment was tested multiple times in the Test/Staging environment, there was always a fear that something might break or go wrong during the rollout.

Fear no more! Today, we will explore the Dry Run capability of ProjectSveltos and how it can help engineers and Kubernetes administrators to update deployments in Production environments with more confidence. Kubectl offers a "dry run" functionality, which allows users to simulate the execution of the commands they want to apply. Sveltos takes it one step further. You can launch a simulation of all the operations you would normally execute in a live run. The best part, no actual changes will be made to the matching clusters!

For today's demonstration, we will update Kyverno on an RKE2 cluster. If you did not have the chance to read the previous post about the Projectsveltos, have a look before you continue with this post.

Diagram

Prerequisites

For this demonstration, I have already installed ArgoCD, deployed Sveltos to cluster04 and created an RKE2 cluster. For the first two, follow step 1 and step 2 from the previous post.

- - - - - -+ - - - - - - - - - - + - - - - - - - - - - -+
| Cluster Name | Type | Version |
+ - - - - - - -+ - - - - - - - - - - + - - - - - - - - -+
| cluster04 | Management Cluster  | RKE2 v1.26.11+rke2r1|
| cluster11 | Managed CAPI Cluster| RKE2 v1.26.11+rke2r1|
+ - - - - - -+ - - - - - - - - - - + - - - - - - - - - -+

Step 1: Register the RKE2 Cluster and Add Label

We will use the sveltosctl to register cluster11 with Sveltos. For the registration, we need three things: a service account, a kubeconfig associated with that account and a namespace. If you are unsure how to create a Service Account and an associated kubeconfig, there is a script publicly available to help you out.

Registration

$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster11 --kubeconfig=cluster11.yaml

Verification

$ kubectl get sveltosclusters -n projectsveltos
NAME        READY   VERSION
cluster11   true    v1.26.11+rke2r1

Cluster Labelling and Verification

To deploy and manage Kubernetes add-ons with the help of Sveltos, the concept of ClusterProfile and cluster labelling comes into play. ClusterProfile is the CustomerResourceDefinition used to instruct Sveltos which add-ons to deploy on a set of clusters.

For this demonstration, we will set the unique label "env=prod".

$ kubectl label sveltosclusters cluster11 env=prod -n projectsveltos

$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME        READY   VERSION           LABELS
cluster11   true    v1.26.11+rke2r1   env=prod,sveltos-agent=present

Step 2: Create Kyverno ClusterProfile

Kyverno Helm chart (v3.0.9) and a Kyverno policy to disallow deployments that use the "latest" tag will get deployed to cluster11. The Helm chart and the Kyverno policy are defined in the same ClusterProfile. To push the Kyverno policy to cluster11, we will have to save the policy as a Configmap and pass it to the ClusterProfile.

Note: The configuration is done on cluster04 as it is our Sveltos management cluster.

Kyverno Policy

$ wget https://raw.githubusercontent.com/kyverno/policies/main/best-practices/disallow-latest-tag/disallow-latest-tag.yaml

$ kubectl create configmap disallow-latest-tag --from-file=disallow-latest-tag.yaml

$ kubectl get cm
NAME                  DATA   AGE
disallow-latest-tag   1      4s

ClusterProfile

---
  apiVersion: config.projectsveltos.io/v1alpha1
  kind: ClusterProfile
  metadata:
    name: kyverno-disallow-latest-tag
  spec:
    clusterSelector: env=prod
    helmCharts:
    - repositoryURL: https://kyverno.github.io/kyverno/
      repositoryName: kyverno
      chartName: kyverno/kyverno
      chartVersion: v3.0.9
      releaseName: kyverno-latest
      releaseNamespace: kyverno
      helmChartAction:  Install
    policyRefs:
    - kind: ConfigMap
      name: disallow-latest-tag
      namespace: default

$ kubectl apply -f "clusterprofile_kyverno_disallow_latest.yaml"

$ sveltosctl show addons
+--------------------------+---------------+-----------+----------------+---------+-------------------------------+-----------------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE |      NAME      | VERSION |             TIME              |      CLUSTER PROFILES       |
+--------------------------+---------------+-----------+----------------+---------+-------------------------------+-----------------------------+
| projectsveltos/cluster11 | helm chart               | kyverno   | kyverno-latest      | 3.0.9   | 2024-01-20 17:20:21 +0000 UTC | kyverno-disallow-latest-tag |
| projectsveltos/cluster11 | kyverno.io:ClusterPolicy |           | disallow-latest-tag | N/A     | 2024-01-20 17:20:21 +0000 UTC | kyverno-disallow-latest-tag |
+--------------------------+---------------+-----------+----------------+---------+-------------------------------+-----------------------------+

Verification - Cluster11

$ kubectl get all -n kyverno
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/kyverno-admission-controller-65f76b4f47-gfrwp    1/1     Running   0          75s
pod/kyverno-background-controller-66d498dd5c-xlkcs   1/1     Running   0          75s
pod/kyverno-cleanup-controller-8689db777f-6nd72      1/1     Running   0          75s
pod/kyverno-reports-controller-84fd865d49-gcb2f      1/1     Running   0          75s

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kyverno-background-controller-metrics   ClusterIP   10.43.63.55     <none>        8000/TCP   75s
service/kyverno-cleanup-controller              ClusterIP   10.43.170.79    <none>        443/TCP    75s
service/kyverno-cleanup-controller-metrics      ClusterIP   10.43.228.47    <none>        8000/TCP   75s
service/kyverno-latest-svc                      ClusterIP   10.43.80.134    <none>        443/TCP    75s
service/kyverno-latest-svc-metrics              ClusterIP   10.43.184.235   <none>        8000/TCP   75s
service/kyverno-reports-controller-metrics      ClusterIP   10.43.41.181    <none>        8000/TCP   75s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kyverno-admission-controller    1/1     1            1           75s
deployment.apps/kyverno-background-controller   1/1     1            1           75s
deployment.apps/kyverno-cleanup-controller      1/1     1            1           75s
deployment.apps/kyverno-reports-controller      1/1     1            1           75s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kyverno-admission-controller-65f76b4f47    1         1         1       75s
replicaset.apps/kyverno-background-controller-66d498dd5c   1         1         1       75s
replicaset.apps/kyverno-cleanup-controller-8689db777f      1         1         1       75s
replicaset.apps/kyverno-reports-controller-84fd865d49      1         1         1       75s

NAME                                                      SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/kyverno-cleanup-admission-reports           */10 * * * *   False     0        <none>          75s
cronjob.batch/kyverno-cleanup-cluster-admission-reports   */10 * * * *   False     0        <none>          75s

Step 3: Update Kyverno to v3.1.4

Just imagine you want to update the Kyverno deployment to the latest version available. We will update the ClusterProfile above to enable the Dry Run capability.

Update ClusterProfile

---
  apiVersion: config.projectsveltos.io/v1alpha1
  kind: ClusterProfile
  metadata:
    name: kyverno-disallow-latest-tag
  spec:
    syncMode: DryRun
    clusterSelector: env=prod
    helmCharts:
    - repositoryURL: https://kyverno.github.io/kyverno/
      repositoryName: kyverno
      chartName: kyverno/kyverno
      chartVersion: v3.1.4
      releaseName: kyverno-latest
      releaseNamespace: kyverno
      helmChartAction:  Install
    policyRefs:
    - kind: ConfigMap
      name: disallow-latest-tag
      namespace: default

In the above definition, we added the syncMode: DryRun to activate Sveltos DryRun mode

$ kubectl apply -f "clusterprofile_kyverno_disallow_latest.yaml"

$ sveltosctl show dryrun
+--------------------------+--------------------------+-----------+---------------------+-----------+--------------------------------+-----------------------------+
|         CLUSTER          |      RESOURCE TYPE       | NAMESPACE |        NAME         |  ACTION   |            MESSAGE             |       CLUSTER PROFILE       |
+--------------------------+--------------------------+-----------+---------------------+-----------+--------------------------------+-----------------------------+
| projectsveltos/cluster11 | helm release             | kyverno   | kyverno-latest      | Upgrade   | Current version: "3.0.9".      | kyverno-disallow-latest-tag |
|                          |                          |           |                     |           | Would move to version:         |                             |
|                          |                          |           |                     |           | "v3.1.4"                       |                             |
| projectsveltos/cluster11 | kyverno.io:ClusterPolicy |           | disallow-latest-tag | No Action | Object already deployed.       | kyverno-disallow-latest-tag |
|                          |                          |           |                     |           | And policy referenced by       |                             |
|                          |                          |           |                     |           | ClusterProfile has not changed |                             |
|                          |                          |           |                     |           | since last deployment.         |                             |
+--------------------------+--------------------------+-----------+---------------------+-----------+--------------------------------+-----------------------------+

From the output above, we can observe that only the Kyverno deployment will get updated from v3.0.9 to v3.1.4 and nothing will happen to the already applied Kyverno policy.

Of course, this is a simplistic example to demonstrate the Dry Run capability. However, imagine having a very large deployment with multiple components and dependencies. Sveltos Dry Run feature can be handy.

Step 4: Deploy Kyverno v3.1.4 to Cluster11

Once we are happy with the changes to be performed on the cluster, it is time to deploy them. To do so, we will have to update the syncMode variable to Continuous. The ClusterProfile will look like the below YAML output.

---
  apiVersion: config.projectsveltos.io/v1alpha1
  kind: ClusterProfile
  metadata:
    name: kyverno-disallow-latest-tag
  spec:
    syncMode: Continuous
    clusterSelector: env=prod
    helmCharts:
    - repositoryURL: https://kyverno.github.io/kyverno/
      repositoryName: kyverno
      chartName: kyverno/kyverno
      chartVersion: v3.1.4
      releaseName: kyverno-latest
      releaseNamespace: kyverno
      helmChartAction:  Install
    policyRefs:
    - kind: ConfigMap
      name: disallow-latest-tag
      namespace: default

$ kubectl apply -f "clusterprofile_kyverno_disallow_latest.yaml"

$ sveltosctl show usage
+----------------+--------------------+-----------------------------+--------------------------+
| RESOURCE KIND  | RESOURCE NAMESPACE |        RESOURCE NAME        |         CLUSTERS         |
+----------------+--------------------+-----------------------------+--------------------------+
| ClusterProfile |                    | kyverno-disallow-latest-tag | projectsveltos/cluster11 |
| ConfigMap      | default            | disallow-latest-tag         | projectsveltos/cluster11 |
+----------------+--------------------+-----------------------------+--------------------------+

$ sveltosctl show addons
+--------------------------+--------------------------+-----------+---------------------+---------+-------------------------------+-----------------------------+
|         CLUSTER          |      RESOURCE TYPE       | NAMESPACE |        NAME         | VERSION |             TIME              |      CLUSTER PROFILES       |
+--------------------------+--------------------------+-----------+---------------------+---------+-------------------------------+-----------------------------+
| projectsveltos/cluster11 | helm chart               | kyverno   | kyverno-latest      | 3.1.4   | 2024-01-20 17:36:51 +0000 UTC | kyverno-disallow-latest-tag |
| projectsveltos/cluster11 | kyverno.io:ClusterPolicy |           | disallow-latest-tag | N/A     | 2024-01-20 17:36:49 +0000 UTC | kyverno-disallow-latest-tag |
+--------------------------+--------------------------+-----------+---------------------+---------+-------------------------------+-----------------------------+

Step 5: Verify the Kyverno Update

From Sveltos point of view we can clearly see that the Kyverno v3.1.4 has been deployed in the cluster. Let's confirm this is the case from a cluster11 point of view.

$ kubectl get all -n kyverno
NAME                                                           READY   STATUS      RESTARTS   AGE
pod/kyverno-admission-controller-69c4c65769-qdg4w              1/1     Running     0          8m10s
pod/kyverno-background-controller-857c7b7b79-ngkcl             1/1     Running     0          8m10s
pod/kyverno-cleanup-admission-reports-28429540-kkrt4           0/1     Completed   0          5m7s
pod/kyverno-cleanup-cluster-admission-reports-28429540-wjz44   0/1     Completed   0          5m7s
pod/kyverno-cleanup-controller-7c9f487ccd-lrz6j                1/1     Running     0          8m10s
pod/kyverno-reports-controller-7bb7db947-f4ff5                 1/1     Running     0          8m10s

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kyverno-background-controller-metrics   ClusterIP   10.43.63.55     <none>        8000/TCP   24m
service/kyverno-cleanup-controller              ClusterIP   10.43.170.79    <none>        443/TCP    24m
service/kyverno-cleanup-controller-metrics      ClusterIP   10.43.228.47    <none>        8000/TCP   24m
service/kyverno-latest-svc                      ClusterIP   10.43.80.134    <none>        443/TCP    24m
service/kyverno-latest-svc-metrics              ClusterIP   10.43.184.235   <none>        8000/TCP   24m
service/kyverno-reports-controller-metrics      ClusterIP   10.43.41.181    <none>        8000/TCP   24m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kyverno-admission-controller    1/1     1            1           24m
deployment.apps/kyverno-background-controller   1/1     1            1           24m
deployment.apps/kyverno-cleanup-controller      1/1     1            1           24m
deployment.apps/kyverno-reports-controller      1/1     1            1           24m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kyverno-admission-controller-65f76b4f47    0         0         0       24m
replicaset.apps/kyverno-admission-controller-69c4c65769    1         1         1       8m10s
replicaset.apps/kyverno-background-controller-66d498dd5c   0         0         0       24m
replicaset.apps/kyverno-background-controller-857c7b7b79   1         1         1       8m10s
replicaset.apps/kyverno-cleanup-controller-7c9f487ccd      1         1         1       8m10s
replicaset.apps/kyverno-cleanup-controller-8689db777f      0         0         0       24m
replicaset.apps/kyverno-reports-controller-7bb7db947       1         1         1       8m10s
replicaset.apps/kyverno-reports-controller-84fd865d49      0         0         0       24m

NAME                                                      SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/kyverno-cleanup-admission-reports           */10 * * * *   False     0        5m7s            24m
cronjob.batch/kyverno-cleanup-cluster-admission-reports   */10 * * * *   False     0        5m7s            24m

NAME                                                           COMPLETIONS   DURATION   AGE
job.batch/kyverno-cleanup-admission-reports-28429540           1/1           4s         5m7s
job.batch/kyverno-cleanup-cluster-admission-reports-28429540   1/1           4s         5m7s

$ kubectl get deploy kyverno-admission-controller -n kyverno -o yaml | grep -i image
        image: ghcr.io/kyverno/kyverno:v1.11.4
        imagePullPolicy: IfNotPresent
        image: ghcr.io/kyverno/kyvernopre:v1.11.4
        imagePullPolicy: IfNotPresent

Updates just become easier and less stressful with the Dry Run capability of Sveltos. Try it out now!

👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

5-Step Approach: Projectsveltos for Kubernetes Cluster Autoscaler Deployment on Hybrid-Cloud

Eleni Grosdouli — Fri, 05 Jan 2024 13:24:51 +0000

Introduction

In the previous post, we talked about the Projectsveltos, and how it can be installed and used as a way to deploy Kubernetes add-on deployments on different environments, independent if they are on-prem or in the Cloud. The example provided was based on the Rancher Kubernetes Engine Government (RKE2) infrastructure in an on-prem environment.

Today, we will take a slightly different approach. We will work with a hybrid cloud setup. The cluster04 will act as our Sveltos management cluster, and the Kubernetes Cluster Autoscaler will get deployed to a Linode Kubernetes Engine (LKE) cluster with the assistance of Sveltos.

Diagram

Prerequisites

For this demonstration, I have already installed ArgoCD, deployed Sveltos to cluster04 and created an LKE cluster. For the first two, follow step 1 and step 2 from my previous post. If you want to learn how to create an LKE cluster, check out the link.

Note: While creating the LKE cluster, the below Node pools created.

Lab Setup

- - - - - -+ - - - - - - - - - - + - - - - - - - - - - -+
| Cluster Name | Type | Version |
+ - - - - - - -+ - - - - - - - - - - + - - - - - - - - -+
| cluster04 | Management Cluster | RKE2 v1.26.11+rke2r1 |
| linode-autoscaler | Managed k8s Cluster | k8s v1.27.8 |
+ - - - - - -+ - - - - - - - - - - + - - - - - - - - - -+

Step 1: Register the LKE Cluster with Sveltos

Once the LKE cluster is in a "Running" state, we will use the sveltosctl to register it. For the registration, we need three things: a service account, a kubeconfig associated with that account and a namespace. If you are unsure how to create a Service Account and an associated kubeconfig, there is a script publicly available to help you out.

Registration

$ sveltosctl register cluster --namespace=projectsveltos --cluster=linode-autoscaler --kubeconfig=linode-autoscaler.yaml

Verification

$ kubectl get sveltoscluster -n projectsveltos
NAME                READY   VERSION
cluster01           true    v1.26.6+rke2r1
linode-autoscaler   true    v1.27.8

Step 2: Cluster Labelling

For this demonstration, we will set the unique label 'env=autoscaler' as we want to differentiate this cluster from the other already existing cluster01.

$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME                READY   VERSION          LABELS
cluster01           true    v1.26.6+rke2r1   env=test,sveltos-agent=present
linode-autoscaler   true    v1.27.8          sveltos-agent=present

Add labels

$ kubectl label sveltosclusters linode-autoscaler env=autoscaler -n projectsveltos

Verification

$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME                READY   VERSION          LABELS
cluster01           true    v1.26.6+rke2r1   env=test,sveltos-agent=present
linode-autoscaler   true    v1.27.8          env=autoscaler,sveltos-agent=present

Step 3: Kubernetes Cluster Autoscaler ClusterProfile

The Cluster Autoscaler will get deployed with the Helm chart. Apart from that, we need a Kubernetes secret to store the cloud-config for the Linode environment.

The whole deployment process is orchestrated and managed by Sveltos. The secret which contains the needed information alongside the Helm chart, will get deployed with the Sveltos ClusterProfile.

Note: The configuration is done on cluster04. Cluster04 is our Sveltos management cluster.

Secret

---
apiVersion: v1
kind: Secret
metadata:
  name: cluster-autoscaler-cloud-config
  namespace: autoscaler
type: Opaque
stringData:
  cloud-config: |-
    [global]
    linode-token={Your own Linode Token}
    lke-cluster-id=147978
    defaut-min-size-per-linode-type=1
    defaut-max-size-per-linode-type=2

    [nodegroup "g6-standard-1"]
    min-size=1
    max-size=2

    [nodegroup "g6-standard-2"]
    min-size=1
    max-size=2

"linode-token": This will get replaced with your own token. To generate a Linode token check the link here.
"lke-cluster-id": You can get the ID of the LKE cluster with the following CURL request, curl -H "Authorization: Bearer $TOKEN" https://api.linode.com/v4/lke/clusters
"nodegroup": As mentioned above, the existing LKE cluster has two node pools. The name of the node pools is defined as the nodegroups in the secret above.

Note: The min and max numbers of the nodegroups can be customised based on your needs. As this is a demo environment, I wanted to keep the cost of the deployment as low as possible.

Create Secret (cluster04)

Before we can create the secret in a Sveltos ClusterProfile, first we need to create the secret locally and add the type 'addons.projectsveltos.io/cluster-profile'.

$ kubectl create secret generic cluster-autoscaler-cloud-config --from-file=secret.yaml --type=addons.projectsveltos.io/cluster-profile

$ kubectl get secret
NAME                              TYPE                                       DATA   AGE
cluster-autoscaler-cloud-config   addons.projectsveltos.io/cluster-profile   1      2m25s

Create ClusterProfile

---
apiVersion: config.projectsveltos.io/v1alpha1
  kind: ClusterProfile
  metadata:
    name: cluster-autoscaler
  spec:
    clusterSelector: env=autoscaler
    syncMode: Continuous
    policyRefs:
    - deploymentType: Remote
      kind: Secret
      name: cluster-autoscaler-cloud-config
      namespace: default
    helmCharts:
    - chartName: autoscaler/cluster-autoscaler
      chartVersion: v9.34.1
      helmChartAction: Install
      releaseName: autoscaler-latest
      releaseNamespace: autoscaler
      repositoryName: autoscaler
      repositoryURL: https://kubernetes.github.io/autoscaler
      helmChartAction:  Install
      values: |
        autoDiscovery:
          clusterName: linode-autoscaler
        cloudProvider: linode
        extraVolumeSecrets:
          cluster-autoscaler-cloud-config:
            mountPath: /config
            name: cluster-autoscaler-cloud-config
        extraArgs:
          logtostderr: true
          stderrthreshold: info
          v: 2
          cloud-config: /config/cloud-config
        image:
          pullPolicy: IfNotPresent
          pullSecrets: []
          repository: registry.k8s.io/autoscaling/cluster-autoscaler
          tag: v1.28.2

"policyRefs": Will create the secret with the name "cluster-autoscaler-cloud-config" in the cluster with the Sveltos tag set to"env=autoscaler" and in the namespace "autoscaler".

Note: the 'namespace: default' refers to the management cluster. Where cluter04 should look for the secret. In this case, it is the default namespace.

"helmCharts": Install the latest Cluster Autoscaler Helm chart
"values": Overwrite the default Helm values with the ones required from the Linode cloud provider. More details can be found here.

Apply ClusterProfile (cluster04)

$ kubectl apply -f "cluster-autoscaler.yaml"

Verification

$ sveltosctl show addons
+----------------------------------+---------------+------------+---------------------------------+---------+-------------------------------+--------------------+
|             CLUSTER              | RESOURCE TYPE | NAMESPACE  |              NAME               | VERSION |             TIME              |  CLUSTER PROFILES  |
+----------------------------------+---------------+------------+---------------------------------+---------+-------------------------------+--------------------+
| projectsveltos/linode-autoscaler | helm chart    | autoscaler | autoscaler-latest               | 9.34.1  | 2024-01-05 12:15:54 +0000 UTC | cluster-autoscaler |
| projectsveltos/linode-autoscaler | :Secret       | autoscaler | cluster-autoscaler-cloud-config | N/A     | 2024-01-05 12:16:03 +0000 UTC | cluster-autoscaler |
+----------------------------------+---------------+------------+---------------------------------+---------+-------------------------------+--------------------+

Verification - linode-autoscaler

$ kubectl get pods -n autoscaler
NAME                                                           READY   STATUS    RESTARTS   AGE
autoscaler-latest-linode-cluster-autoscaler-5fd8bfbb6f-r5zf6   1/1     Running   0          5m25s

Step 4: Cluster Multi Replica Deployment

We will use Sveltos to create a busybox deployment on the LKE cluster with 500 replicas.

Create Deployment and Configmap Resource (cluster04)

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-workload
  namespace: default
  labels:
    app: busybox
spec:
  replicas: 500
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'echo Demo Workload ; sleep 600']

$ kubectl create configmap busybox --from-file=busybox.yaml

Create ClusterProfile for busybox Deployment (cluster04)

---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: busybox
spec:
  clusterSelector: env=autoscaler
  policyRefs:
  - kind: ConfigMap
    name: busybox
    namespace: default

$ kubectl apply -f "clusterprofile-busybox-autoscaler.yaml"

$ sveltosctl show addons --clusterprofile=busybox
+--------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
|         CLUSTER          |  RESOURCE TYPE  | NAMESPACE |       NAME       | VERSION |             TIME              | CLUSTER PROFILES |
+--------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
| projectsveltos/linode-autoscaler | apps:Deployment | default   | busybox-workload | N/A     | 2024-01-05 12:25:10 +0000 UTC | busybox          |
+--------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+

As the cluster has only two small nodes, it cannot keep up with the load of the 500 replicas. The autoscaler will kick in and add the required size of the nodes available from the pool.

Verification

$ kubernetes logs autoscaler-latest-linode-cluster-autoscaler-5fd8bfbb6f-r5zf6 -n autoscaler -f

I0103 17:39:49.638775       1 linode_manager.go:84] LKE node group after refresh:
I0103 17:39:49.638804       1 linode_manager.go:86] node group ID g6-standard-1 := min: 1, max: 2, LKEClusterID: 148988, poolOpts: {Count:1 Type:g6-standard-1 Disks:[]}, associated LKE pools: { poolID: 218632, count: 1, type: g6-standard-1, associated linodes: [ID: "218632-48499cbb0000", instanceID: 53677574] }
I0103 17:39:49.638826       1 linode_manager.go:86] node group ID g6-standard-2 := min: 1, max: 2, LKEClusterID: 148988, poolOpts: {Count:1 Type:g6-standard-2 Disks:[]}, associated LKE pools: { poolID: 218633, count: 1, type: g6-standard-2, associated linodes: [ID: "218633-4d4d521d0000", instanceID: 53677576] }
I0103 17:39:49.639930       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-f558s is unschedulable
I0103 17:39:49.639939       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-rfhqp is unschedulable
I0103 17:39:49.639943       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-wsfcw is unschedulable
I0103 17:39:49.639946       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-25s5c is unschedulable
I0103 17:39:49.639948       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-r7fv4 is unschedulable
I0103 17:39:49.639951       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-nmbxc is unschedulable
I0103 17:39:49.639954       1 klogx.go:87] Pod default/busybox-workload-5d94965f98-dbrh4 is unschedulable
E0103 17:39:49.640116       1 orchestrator.go:450] Couldn't get autoscaling options for ng: g6-standard-1
E0103 17:39:49.640137       1 orchestrator.go:450] Couldn't get autoscaling options for ng: g6-standard-2
I0103 17:39:49.640587       1 orchestrator.go:185] Best option to resize: g6-standard-1
I0103 17:39:49.641940       1 orchestrator.go:189] Estimated 1 nodes needed in g6-standard-1
I0103 17:39:49.641961       1 orchestrator.go:295] Final scale-up plan: [{g6-standard-1 1->2 (max: 2)}]
I0103 17:39:49.641972       1 executor.go:147] Scale-up: setting group g6-standard-1 size to 2

Step 5: Undeploy Busybox

Finally, once the tests are complete, it is time to remove the busybox deployment from the cluster and allow 10 good minutes for the autoscaler to scale down the unneeded nodes. For the undeploy process, we only need to delete the busybox ClusterProfile.

Undeploy (LKE)

$ kubectl delete -f "clusterprofile-busybox-autoscaler.yaml"

Verification

$ kubernetes logs autoscaler-latest-linode-cluster-autoscaler-5fd8bfbb6f-r5zf6 -n autoscaler -f

I0103 18:02:27.179597       1 linode_manager.go:84] LKE node group after refresh:
I0103 18:02:27.179813       1 linode_manager.go:86] node group ID g6-standard-1 := min: 1, max: 2, LKEClusterID: 148988, poolOpts: {Count:1 Type:g6-standard-1 Disks:[]}, associated LKE pools: { poolID: 218632, count: 1, type: g6-standard-1, associated linodes: [ID: "218632-48499cbb0000", instanceID: 53677574] }
I0103 18:02:27.179840       1 linode_manager.go:86] node group ID g6-standard-2 := min: 1, max: 2, LKEClusterID: 148988, poolOpts: {Count:1 Type:g6-standard-2 Disks:[]}, associated LKE pools: { poolID: 218633, count: 1, type: g6-standard-2, associated linodes: [ID: "218633-4d4d521d0000", instanceID: 53677576] }
I0103 18:02:27.181433       1 static_autoscaler.go:547] No unschedulable pods
I0103 18:02:27.181449       1 pre_filtering_processor.go:67] Skipping lke148988-218632-48499cbb0000 - node group min size reached (current: 1, min: 1)
I0103 18:02:27.181623       1 pre_filtering_processor.go:67] Skipping lke148988-218633-4d4d521d0000 - node group min size reached (current: 1, min: 1)
I0103 18:02:36.040964       1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
I0103 18:02:36.040995       1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.52µs
I0103 18:02:38.019810       1 linode_manager.go:84] LKE node group after refresh:
I0103 18:02:38.019983       1 linode_manager.go:86] node group ID g6-standard-2 := min: 1, max: 2, LKEClusterID: 148988, poolOpts: {Count:1 Type:g6-standard-2 Disks:[]}, associated LKE pools: { poolID: 218633, count: 1, type: g6-standard-2, associated linodes: [ID: "218633-4d4d521d0000", instanceID: 53677576] }
I0103 18:02:38.020015       1 linode_manager.go:86] node group ID g6-standard-1 := min: 1, max: 2, LKEClusterID: 148988, poolOpts: {Count:1 Type:g6-standard-1 Disks:[]}, associated LKE pools: { poolID: 218632, count: 1, type: g6-standard-1, associated linodes: [ID: "218632-48499cbb0000", instanceID: 53677574] }
I0103 18:02:38.021154       1 static_autoscaler.go:547] No unschedulable pods
I0103 18:02:38.021181       1 pre_filtering_processor.go:67] Skipping lke148988-218632-48499cbb0000 - node group min size reached (current: 1, min: 1)
I0103 18:02:38.021187       1 pre_filtering_processor.go:67] Skipping lke148988-218633-4d4d521d0000 - node group min size reached (current: 1, min: 1)

As expected, Sveltos took care of the complete lifecycle of the different Kubernetes deployments in a simple and straightforward manner!

👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Install ArgoCD on RKE2 with Nginx Ingress Controller

Eleni Grosdouli — Thu, 28 Dec 2023 10:03:05 +0000

When you start working with ArgoCD, and with Kubernetes in general, it is not clear what configuration to use to install ArgoCD on an RKE2 cluster with the Nginx Controller integrated. ArgoCD is a Kubernetes continuous delivery tool based on the GitOps principles. It can be used as a standalone installation or as part of a CI/CD workflow.

The blog post aims to provide readers with a step-by-step approach to install ArgoCD as a standalone installation, create an Ingress Kubernetes resource and access the ArgoCD UI locally.

Lab Setup

- - - - - -+ - - - - - - - - - - - + - - - - - - - - - - -+
| Cluster Name |      Type         |      Version         |
+ - - - - - - -+ - - - - - - - - - - + - - - - - - - - - -+
| cluster04 | Management Cluster   | RKE2 v1.26.11+rke2r1 |
+ - - - - - -+ - - - - - - - - - + - - - - - - - - - - - -+

- - - - - - - + - - - -+
| Deployment | Version |
+ - - - - - - + - - - -+
| ArgoCD     | v2.9.3 |
| Rancher    | v2.7.9 |
+ - - - - - - + - - - -+

Step 1: Install ArgoCD

Going through the official documentation, there are two ways to install ArgoCD on a cluster, either via the Helm chart or via the manifest files. In our case, we will follow the official "Getting Started" guide found here and we will use the manifests approach.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The code above will create the "argocd" Kubernetes namespace and deploy the latest stable manifest. If you would like to install a specific manifest, have a look here.

Validate

Let's validate if all the ArgoCD Kubernetes resource are in a "Running" state.

$ kubectl get all -n argocd
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0                     1/1     Running   0          15m
pod/argocd-applicationset-controller-5877955b59-2j8fj   1/1     Running   0          15m
pod/argocd-dex-server-6c87968c75-rdnck                  1/1     Running   0          15m
pod/argocd-notifications-controller-64bb8dcf46-6tgnd    1/1     Running   0          15m
pod/argocd-redis-7d8d46cc7f-j5mgj                       1/1     Running   0          15m
pod/argocd-repo-server-665d6b7b59-5qmhs                 1/1     Running   0          15m
pod/argocd-server-7bccc77dd8-v5j2s                      1/1     Running   0          2m52s

NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/argocd-applicationset-controller          ClusterIP      10.43.110.123   <none>        7000/TCP,8080/TCP            15m
service/argocd-dex-server                         ClusterIP      10.43.62.176    <none>        5556/TCP,5557/TCP,5558/TCP   15m
service/argocd-metrics                            ClusterIP      10.43.76.103    <none>        8082/TCP                     15m
service/argocd-notifications-controller-metrics   ClusterIP      10.43.129.111   <none>        9001/TCP                     15m
service/argocd-redis                              ClusterIP      10.43.34.24     <none>        6379/TCP                     15m
service/argocd-repo-server                        ClusterIP      10.43.71.49     <none>        8081/TCP,8084/TCP            15m
service/argocd-server                             LoadBalancer   10.43.4.100     x.x.x.x     80:31274/TCP,443:31258/TCP     15m
service/argocd-server-metrics                     ClusterIP      10.43.219.7     <none>        8083/TCP                     15m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-applicationset-controller   1/1     1            1           15m
deployment.apps/argocd-dex-server                  1/1     1            1           15m
deployment.apps/argocd-notifications-controller    1/1     1            1           15m
deployment.apps/argocd-redis                       1/1     1            1           15m
deployment.apps/argocd-repo-server                 1/1     1            1           15m
deployment.apps/argocd-server                      1/1     1            1           15m

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-applicationset-controller-5877955b59   1         1         1       15m
replicaset.apps/argocd-dex-server-6c87968c75                  1         1         1       15m
replicaset.apps/argocd-notifications-controller-64bb8dcf46    1         1         1       15m
replicaset.apps/argocd-redis-7d8d46cc7f                       1         1         1       15m
replicaset.apps/argocd-repo-server-665d6b7b59                 1         1         1       15m
replicaset.apps/argocd-server-5986f74c99                      0         0         0       15m
replicaset.apps/argocd-server-7bccc77dd8                      1         1         1       2m52s

NAME                                             READY   AGE
statefulset.apps/argocd-application-controller   1/1     15m

Step 2: Create an Ingress

An Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. An SSL-Pathtrough Ingress example can be found in the official documentation here.

Let's have a look at the most important configurations performed on the example file.

  1 apiVersion: networking.k8s.io/v1
  2 kind: Ingress
  3 metadata:
  4   name: argocd-server-ingress
  5   namespace: argocd
  6   annotations:
  7     nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  8     nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  9 spec:
 10   ingressClassName: nginx
 11   rules:
 12   - host: argocd-cluster04.{YOUR DOMAIN}
 13     http:
 14       paths:
 15       - path: /
 16         pathType: Prefix
 17         backend:
 18           service:
 19             name: argocd-server
 20             port:
 21               name: http
 22   tls:
 23   - hosts:
 24     - argocd-cluster04.{YOUR DOMAIN}
 25     secretName: argocd-server-tls # as expected by argocd-server

Line 4: The Ingress name is defined as "argocd-server-ingress". The name can be anything you want.

Line 5: The ArgoCD resources in Step 1 were created in the "argocd" namespace. If this is the case for your deployment, keep the Ingress resource in the same namespace.

Line 6–8: The annotations are used as a way to expose the ArgoCD API server as a single ingress rule and hostname.

The "nginx.ingress.kubernetes.io/ssl-passthrough" annotation, is used to terminate SSL/TLS traffic at the ArgoCD API server instead of the Nginx Ingress Controller
The "nginx.ingress.kubernetes.io/force-ssl-redirect: "true"" annotation tells the Nginx Ingress Controller to automatically redirect HTTP requests to HTTPS

For the second annotation to be functional, we need to add the argument " - enable-ssl-passthrough" to the Nginx Ingress Controller Daemonset. The Daemonset name on an RKE2 installation is "rke2-ingress-nginx-controller".

Update the Nginx Ingress Controller Daemonset

First Option: Edit the Daemonset

$ kubectl edit daemonset rke2-ingress-nginx-controller -n kube-system

  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: rke2-ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: rke2-ingress-nginx
        app.kubernetes.io/part-of: rke2-ingress-nginx
        app.kubernetes.io/version: 1.9.3
        helm.sh/chart: rke2-ingress-nginx-4.8.200
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=rke2-ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --watch-ingress-without-class=true

Add the argument " - enable-ssl-passthrough" at the end of the argument list. The output should look like the below.

Output

template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: rke2-ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: rke2-ingress-nginx
        app.kubernetes.io/part-of: rke2-ingress-nginx
        app.kubernetes.io/version: 1.9.3
        helm.sh/chart: rke2-ingress-nginx-4.8.200
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=rke2-ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --watch-ingress-without-class=true
        - --enable-ssl-passthrough

Second Option: Patch the Daemonset

$ kubectl patch daemonset rke2-ingress-nginx-controller -n kube-system --type='json' -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--enable-ssl-passthrough"}]'

  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: rke2-ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: rke2-ingress-nginx
        app.kubernetes.io/part-of: rke2-ingress-nginx
        app.kubernetes.io/version: 1.9.3
        helm.sh/chart: rke2-ingress-nginx-4.8.200
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=rke2-ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --watch-ingress-without-class=true
        - --enable-ssl-passthrough

Line 12: Set the FQDN of your application. As long as your Domain Name System (DNS) can resolve the name defined, it can be anything. Of course, you will have to own a domain for that purpose.

Line 22–25: The "argocd-server-tls" secret is a new self-signed certificated generated with the use of the OpenSSL utility.

Step 3: Validate the ArgoCD Deployment

In the first two steps, we installed all the needed Kubernetes resources for ArgoCD to be functional and created an Ingress resource to work with the Nginx Ingress Controller to allow ssl-passthrough. Now, it is the big moment, to check if the deployment is actually working.

Validate


$ kubectl get pods,svc,secret -n argocd
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0                     1/1     Running   0          14m
pod/argocd-applicationset-controller-5877955b59-2j8fj   1/1     Running   0          14m
pod/argocd-dex-server-6c87968c75-rdnck                  1/1     Running   0          14m
pod/argocd-notifications-controller-64bb8dcf46-6tgnd    1/1     Running   0          14m
pod/argocd-redis-7d8d46cc7f-j5mgj                       1/1     Running   0          14m
pod/argocd-repo-server-665d6b7b59-5qmhs                 1/1     Running   0          14m
pod/argocd-server-7bccc77dd8-v5j2s                      1/1     Running   0          103s

NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/argocd-applicationset-controller          ClusterIP      10.43.110.123   <none>        7000/TCP,8080/TCP            14m
service/argocd-dex-server                         ClusterIP      10.43.62.176    <none>        5556/TCP,5557/TCP,5558/TCP   14m
service/argocd-metrics                            ClusterIP      10.43.76.103    <none>        8082/TCP                     14m
service/argocd-notifications-controller-metrics   ClusterIP      10.43.129.111   <none>        9001/TCP                     14m
service/argocd-redis                              ClusterIP      10.43.34.24     <none>        6379/TCP                     14m
service/argocd-repo-server                        ClusterIP      10.43.71.49     <none>        8081/TCP,8084/TCP            14m
service/argocd-server                             LoadBalancer   10.43.4.100     x.x.x.x     80:31274/TCP,443:31258/TCP     14m
service/argocd-server-metrics                     ClusterIP      10.43.219.7     <none>        8083/TCP                     14m

NAME                                 TYPE                DATA   AGE
secret/argocd-initial-admin-secret   Opaque              1      13m
secret/argocd-notifications-secret   Opaque              0      14m
secret/argocd-secret                 Opaque              3      14m
secret/argocd-server-tls             kubernetes.io/tls   2      105s
$ kubectl get ingress -n argocd
NAME                    CLASS   HOSTS                                     ADDRESS                                                                     PORTS     AGE
argocd-server-ingress   nginx   argocd-cluster04.{YOUR DOMAIN}   cluster04-controller-1,cluster04-controller-2,cluster04-worker-1,cluster04-worker-2  80, 443   4m57s

Access the ArgoCD UI

As long as your DNS is correctly set up, you will resolve the FQDN set during the Ingress configuration. Keep in mind, that the ArgoCD deployment is exposed via the HTTPS protocol over port 443.

If you do not control the DNS deployment and you want to perform local testing, for Linux and MacOS-based systems, modify the "/etc/hosts" file and add the IP address of a Kubernetes worker node followed by the FQDN. For Windows-based systems, you can modify the "C:\Windows\System32\Drivers\etc\hosts" file.

URL: https://argocd-cluster04.{YOUR DOMAIN}

Note: If you use a self-sign certificate, your preferred browser will pop up a message about an untrusted connection to a server. In this case and as this is our test environment, you can skip the verification and proceed to the login page of ArgoCD.

5-Step Approach: Projectsveltos for Kubernetes add-on deployment and management on RKE2

Eleni Grosdouli — Mon, 18 Dec 2023 12:40:57 +0000

Working with many different Kubernetes add-on deployments, the actual deployment and management of those across different clusters, on-prem and in the Cloud, can be challenging and sometimes frustrating.

Projectsveltos is a Kubernetes add-on controller that simplifies the deployment and management of different add-ons and applications across multiple clusters (on-prem, Cloud). Sveltos runs in a management cluster and programmatically deploys and manages add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports many add-on formats, including Helm charts, raw YAML/JSON, Kustomize, Carvel ytt, and Jsonnet.

In this blog post, we will demonstrate how easy and fast it is to deploy Sveltos on an RKE2 cluster with the help of ArgoCD, register two RKE2 Cluster API (CAPI) clusters and create a ClusterProfile to deploy Prometheus and Grafana Helm charts down the managed CAPI clusters.

Diagram

Prerequisites

For this demonstration, I have already installed ArgoCD on a central cluster. If you would like to learn more about the ArgoCD installation, go through the official documentation found here. If you would like to follow along, below you can find the lab details used.

- - - - - -+ - - - - - - - - - - - + - - - - - - - - - - -+
| Cluster Name |      Type         |      Version         |
+ - - - - - - -+ - - - - - - - - - - + - - - - - - - - - -+
| cluster04 | Management Cluster   | RKE2 v1.26.11+rke2r1 |
| cluster12 | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1  |
| cluster13 | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1  |
+ - - - - - -+ - - - - - - - - - + - - - - - - - - - - - -+

Step 1: Deploy Sveltos as a Helm Chart cluster04

Sveltos can be deployed either as a manifest or as a Helm chart . For more information about the different installation options, check the link here. In my case, I chose to follow the GitOps approach and let ArgoCD deal with the comparison and synchronisation of the Git repository where the code to deploy Sveltos is stored, with the actual running application.

If you are unsure how to deploy Helm charts with ArgoCD, have a look here.

Verification

After we deploy Sveltos, we want to ensure everything is in a working and fully functional state. This can be achieved either from the ArgoCD UI or from the management cluster itself.

$ kubectl get pods -n projectsveltos

NAME                                        READY   STATUS    RESTARTS   AGE
access-manager-77c7c64477-ns8ml             2/2     Running   0          70s
addon-compliance-manager-7f449d884c-6kgqr   2/2     Running   0          69s
addon-controller-55d7d848ff-ps8l8           2/2     Running   0          70s
classifier-manager-67d6f67d5b-cgpr7         2/2     Running   0          70s
event-manager-69db45b65d-htz5l              2/2     Running   0          70s
hc-manager-5679c69dcc-z6s48                 2/2     Running   0          70s
sc-manager-84dbd64fb4-6hwpf                 2/2     Running   0          70s
shard-controller-56678bcf8c-zjbvc           2/2     Running   0          70s

Step 2: Install the Sveltosctl

The Sveltosctl, is the command-line interface (CLI) for Sveltos. This is an available option to query Sveltos resources and it is available as a Kubernetes pod or as a binary.

As I would like to register cluster12 and cluster13 to the Sveltos management cluster, the Sveltosctl as a binary will be used.

Step 3: Register CAPI Clusters with Sveltos

To register any cluster with Sveltos, you only need three things:

A ServiceAccount for Sveltos and a kubeconfig associated with that account;
A namespace where you want to register the external cluster;
The Sveltosctl should point to the management cluster and then perform the 'sveltosctl register cluster' command.

Now, if you are unsure how to create a Service Account and an associated kubeconfig, do not worry. There is a script publicly available to create everything you need automatically.

Registration

$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster12 --kubeconfig=cluster12.yaml

$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster13 --kubeconfig=cluster13.yaml

From the commands above, we register cluster12 and cluster13 in the namespace projectsveltos. Of course, you can register the clusters to a namespace of your preference.

Verification

$ kubectl get sveltosclusters -n projectsveltos

NAME        READY   VERSION
cluster12   true    v1.26.6+rke2r1
cluster13   true    v1.26.6+rke2r1

Step 4: Cluster Labelling

To allow Sveltos to deploy and manage Kubernetes add-ons, the concept of ClusterProfile and cluster labelling comes into play. ClusterProfile is the CustomerResourceDefinition used to instruct Sveltos which add-ons to deploy on a set of clusters.

For this demonstration, we will set the label "env:prod" to both Sveltos clusters. The below commands are executed on the management cluster (cluster04).

$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   sveltos-agent=present

$ kubectl label sveltosclusters cluster12 env=prod -n projectsveltos

$ kubectl label sveltosclusters cluster13 env=prod -n projectsveltos

$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present

Step 5: ClusterProfile for Grafana and Prometheus

The below ClusterProfile is an example of a Helm chart deployment of Grafana and Prometheus to Sveltos clusters with the label set to "env:prod".

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: env=prod
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install

Apply the ClusterProfile

$ kubectl apply -f "grafana_prometheus.yaml"

Once the ClusterProfile is applied to the management cluster, the expected result is to have the Grafana and the Prometheus deployment on both managed clusters.

Verification

$ sveltosctl show addons

+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |             TIME              |  CLUSTER PROFILES  |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster12 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:20 +0100 CET | prometheus-grafana |
| projectsveltos/cluster12 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:23 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+

Verification - Cluster12

$ kubectl get pods -n grafana

NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-78764f9cd6-zsqdx   1/1     Running   0          81s

$ kubectl pods all -n prometheus

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-0                            1/1     Running   0          2m3s
pod/prometheus-kube-state-metrics-587bd996f6-l94zq       1/1     Running   0          2m3s
pod/prometheus-prometheus-node-exporter-khw75            1/1     Running   0          2m3s
pod/prometheus-prometheus-pushgateway-75986b9c9f-2ql7v   1/1     Running   0          2m3s
pod/prometheus-server-86c66b89c6-7xk9r                   2/2     Running   0          2m3s

The same verification can be performed for cluster13.

Remove Label 'env:prod' cluster12

You might wonder what will happen if we remove the label 'env:prod' from either cluster12 or cluster13. The answer is that Sveltos will identify the missing label 'env:prod' and undeploy the Grafana and the Prometheus deployment from the cluster.

Let's have a look.

Remove Label

$ kubectl label sveltosclusters cluster12 env- -n projectsveltos

Verification

$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present

sveltosctl show addons

+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |             TIME              |  CLUSTER PROFILES  |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster13 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+

As expected, Sveltos removed the deployments. The same will happen if we register a new cluster and assign the label 'env:prod'. Sveltos will take care of the complete lifecycle of your Kubernetes deployments in a simple and straightforward manner.

👏 Support this project

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!