5-Step Approach: ProjectSveltos Event Framework for Kubernetes Deployment with Cilium Gateway API

Introduction

In the previous post, we demonstrated how easy it is to deploy ArgoCD with the Cilium Gateway API and move away from the Kubernetes Ingress. In today’s post, we will explore how the Projectsveltos Event Framework can be utilised to seamlessly perform the same deployment based on an event.

The Sveltos Event Framework was designed to automate the deployment of Kubernetes add-ons triggered on specific cluster events. This functionality proves invaluable in managing environments spanning across multiple clusters.

Sveltos supports an event-driven workflow with the below steps:

1. Define what the event is
2. Select the clusters to watch for such events
3. Define an event trigger. Which add-ons/applications to deploy when the events occur

Once we identify the event we want Sveltos to watch for, we can deploy the EventSource and EventTrigger Kubernetes CRDs (Custom Resource Definitions). More information about these resources can be found later on.

Diagram

Prerequisites

To follow along, it is strongly advised to take a look at a previous post detailing the deployment of Sveltos on a cluster. To install Sveltos, just follow the instructions outlined in steps 1 and 2 of the mentioned post. Additionally, as we will use the Cilium Gateway API, have a look at the previous post to enable this capability on an RKE2 cluster.

Lab Setup

— — — — — -+ — — — — — — — — — — + — — — — — — — — — — — — -+
| Cluster Name | Type | Version |
+ — — — — — — -+ — — — — — — — — — — + — — — — — — — — — — -+
| rke2-mgmt01 | Management Cluster | RKE2 v1.26.12+rke2r1 |
| rke2-sles-demo | Managed Cluster | RKE2 v1.26.12+rke2r1|
+ — — — — — -+ — — — — — — — — — — + — — — — — — — — — — — — -+

- — — — — — -+ — — — — -+
| Deployment | Version |
+ — — — — — — -+ — — — — -+
| ArgoCD | v2.9.3 |
| Cilium | v1.14.5 |
| GatewayAPI | v0.7.0 |
+ — — — — — — -+ — — — — -+

Step 1: Register and label the RKE2 cluster with Sveltos

Once the RKE2 cluster is in a “Running” state, we will use the sveltosctl to register it. For the registration, we need three things: a service account, a kubeconfig associated with that account and a namespace. If you are unsure how to create a Service Account and an associated kubeconfig, there is a script publicly available to help you out.

Registration

$ sveltosctl register cluster --namespace=projectsveltos --cluster=rke2-sles-demo --kubeconfig=rke2-sles-demo.yaml

Verification

$ kubectl get sveltoscluster -n projectsveltos
NAME               READY   VERSION
rke2-sles-demo   true    v1.26.12+rke2r1

Assing Label

$ kubectl label sveltoscluster rke2-sles-demo env=staging -n projectsveltos

Verification

$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME             READY   VERSION           LABELS
rke2-sles-demo   true    v1.26.12+rke2r1   env=staging,sveltos-agent=present

For the demonstration, we will use the label ‘env=staging’ to specify the cluster we want to deploy the Gateway and the HTTPRoute resources.

Step 2: Pre-Work (management cluster)

Before we move on with the Gateway API implementation, we need to create additional Kubernetes resources for the managed cluster (rke2-sles-demo).

• ArgoCD namespace
• ArgoCD TLS Secret
• Cilium IP Pool
• Cilium GatewayClass
• Gateway
• HTTPRoute

To utilise the power of Sveltos, we will create the above resources in the management cluster as a Secret and configMap resource and use the Sveltos ClusterProfile to deploy them to the managed cluster.

More information on how to create the above resources can be found here.

ArgoCD Namespace

If the namespace does not exist in the managed cluster, Sveltos will create it!

ArgoCD TLS Secret

apiVersion: v1
data:
  tls.crt: {base64 encoded .crt}
  tls.key: {base64 encoded .key}
kind: Secret
metadata:
  name: argocd-server-tls
  namespace: argocd
type: kubernetes.io/tls

$ kubectl create secret generic argocd-server-tls --from-file=argocd_server_tls.yaml --type=addons.projectsveltos.io/cluster-profile

Cilium IP Pool

---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "rke2-pool"
spec:
  cidrs:
  - cidr: "10.10.10.0/24"

$ kubectl create configmap ipampool --from-file=ipam-pool.yaml

Cilium GatewayClass

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller

$ kubectl create configmap gatewayclass --from-file=gatewayclass.yaml

Gateway

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: argocd
  namespace: argocd
spec:
  gatewayClassName: cilium
  listeners:
  - hostname: argocd.example.com
    name: argocd-example-com-http
    port: 80
    protocol: HTTP
  - hostname: argocd.example.com
    name: argocd-example-com-https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        name: argocd-server-tls

$ kubectl create configmap gateway --from-file=gateway.yaml

HTTPRoute

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  creationTimestamp: null
  name: argocd
  namespace: argocd
spec:
  hostnames:
  - argocd.example.com
  parentRefs:
  - name: argocd
  rules:
  - backendRefs:
    - name: argocd-server
      port: 80
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents: []

$ kubectl create configmap gateway --from-file=httproute.yaml

Verification

$ kubectl get secret
NAME                TYPE                                       DATA   AGE
argocd-server-tls   addons.projectsveltos.io/cluster-profile   1      20m

$ kubectl get cm
NAME               DATA   AGE
gateway            1      35s
gatewayclass       1      20m
httproute          1      3s
ipam-pool          1      22m
kube-root-ca.crt   1      141m

ClusterProfile

---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
 name: cilium-gateway-api
spec:
 clusterSelector: env=staging
 syncMode: Continuous
 policyRefs:
 - name: ipampool
   namespace: default
   kind: ConfigMap
 - name: gatewayclass
   namespace: default
   kind: ConfigMap
 - name: argocd-server-tls
   namespace: default
   kind: Secret

The ClusterProfile definition will match the managed cluster with the label selector set to ‘env=staging’ and deploy the Cilium IP Pool, the GatewayClass and the ArgoCD TLS secret.

Apply ClusterProfile

$ kubectl apply -f "clusterprofile.yaml"

Verification

$ sveltosctl show addons

+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|            CLUSTER            |             RESOURCE TYPE              | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/rke2-sles-demo | cilium.io:CiliumLoadBalancerIPPool     |           | rke2-pool         | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:GatewayClass |           | cilium            | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | :Secret                                | argocd    | argocd-server-tls | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+

Step 3: Define the Event Framework (management cluster)

As mentioned at the beginning of the post, we will use the EventSource to define which events Sveltos will watch for and the EventTrigger resource to perform actions based on an event.

EventSource

---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventSource
metadata:
 name: http-https-service
spec:
 collectResources: true
 resourceSelectors:
 - group: ""
   version: "v1"
   kind: "Service"
   namespace: argocd
   evaluate: |
     function evaluate()
       hs = {}
       hs.matching = false
       if obj.spec.ports ~= nil then
         for _,p in pairs(obj.spec.ports) do
           if p.port == 80 or p.port == 443 then
             hs.matching = true
           end
         end
       end
       return hs
     end

The EventSource uses the Lua language to search for any services with ports set to 80 or 443 in the ‘argocd’ namespace. More examples can be found here.

EventTrigger

---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventTrigger
metadata:
 name: service-network-policy
spec:
 sourceClusterSelector: env=staging
 eventSourceName: http-https-service
 oneForEvent: true
 policyRefs:
 - name: httproute
   namespace: default
   kind: ConfigMap
 - name: gateway  
   namespace: default
   kind: ConfigMap

Apply Resources

$ kubectl apply -f "eventsource.yaml"
eventsource.lib.projectsveltos.io/loadbalancer-service created

$ kubectl apply -f "eventtrigger.yaml"
eventtrigger.lib.projectsveltos.io/service-network-policy created

Once an event is spotted by Sveltos, the HTTPRoute and Gateway Kubernetes add-ons will get deployed on the managed cluster with the label selector set to ‘env=staging’. In our case, the cluster ‘rke2-sles-demo’ will match the ClusterSelector and will automatically get the specified resources.

Step 4: Deploy ArgoCD Manifests (managed cluster)

Now, it is time to deploy the ArgoCD manifests and wait Sveltos to notice the event and trigger the deployment of the HTTPRoute and Gateway resources in the ‘argocd’ namespace.

$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Step 5: Verification and Tests

As we have installed the ArgoCD manifests, the ‘argocd-server’ service should have been created in the ‘argocd’ namespace. That means the Sveltos EventSource should have been activated and triggered the deployment of the HTTPRoute and Gateway Kubernetes resources.

Sveltos Verification

$ sveltosctl show addons
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|            CLUSTER            |             RESOURCE TYPE              | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/rke2-sles-demo | cilium.io:CiliumLoadBalancerIPPool     |           | rke2-pool         | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:GatewayClass |           | cilium            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | :Secret                                | argocd    | argocd-server-tls | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:HTTPRoute    | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-1lbzxnp4cs96gpo38g8b |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:Gateway      | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-1lbzxnp4cs96gpo38g8b |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:HTTPRoute    | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-i0nmkq3ebsvpis8ubd3s |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:Gateway      | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-i0nmkq3ebsvpis8ubd3s |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+

$ sveltosctl show usage
+----------------+--------------------+------------------------------+-------------------------------+
| RESOURCE KIND  | RESOURCE NAMESPACE |        RESOURCE NAME         |           CLUSTERS            |
+----------------+--------------------+------------------------------+-------------------------------+
| ClusterProfile |                    | cilium-gateway-api           | projectsveltos/rke2-sles-demo |
| ClusterProfile |                    | sveltos-fvc3vr84yqusfzwpobcl | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | gateway                      | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | ipampool                     | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | gatewayclass                 | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | httproute                    | projectsveltos/rke2-sles-demo |
+----------------+--------------------+------------------------------+-------------------------------+

Managed Cluster Verification

$ kubectl get gateway,httproute -n argocd
NAME                                       CLASS    ADDRESS        PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/argocd   cilium   10.10.10.147   True         78s

NAME                                         HOSTNAMES                AGE
httproute.gateway.networking.k8s.io/argocd   ["argocd.example.com"]   78s

Test ArgoCD Accessibility

curl -ki https://argocd.example.com
HTTP/1.1 200 OK
accept-ranges: bytes
content-length: 788
content-security-policy: frame-ancestors 'self';
content-type: text/html; charset=utf-8
vary: Accept-Encoding
x-frame-options: sameorigin
x-xss-protection: 1
date: Sun, 11 Feb 2024 18:59:57 GMT
x-envoy-upstream-service-time: 0
server: envoy

<!doctype html><html lang="en"><head><meta charset="UTF-8"><title>Argo CD</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32"/><link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16"/><link href="assets/fonts.css" rel="stylesheet"><script defer="defer" src="main.9ecae91d8fd1deedf944.js"></script></head><body><noscript><p>Your browser does not support JavaScript. Please enable JavaScript to view the site. Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.</p></noscript><div id="app"></div></body><script defer="defer" src="extensions.js"></script></html>

Note: As we use a self-sign certificate the argocd-cmd-params-cm ConfigMap in the argocd namespace should include the following setting server.insecure: “true”.

As expected, Sveltos took care of the complete lifecycle of the different Kubernetes deployments in a simple and straightforward manner! The example might seem simplistic but think about it on a bigger scale with hundreds of clusters across a multicloud setup. Imagine the flexibility of setting Sveltos labels to a cluster, defining the event source to trigger an action and having a Kubernetes deployment on a cluster. In the next post, we will demonstrate the power of Events in a cross multi-cluster environment.

👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!