DEV Community: Gianluca

Stop Forking Your YAML

Gianluca — Mon, 23 Feb 2026 12:52:20 +0000

TL;DR: Managing multi-cluster YAML usually leads to Copy-Paste drift or Helm-chart bloat. This post demonstrates how to use a surgical post-processor to decouple global application intent from regional infrastructure needs using Late-Binding.

The Problem: The "Copy-Paste" Trap

In a traditional setup, when you have a core application that needs a slight tweak for a specific region (like our EU vs. US example), you generally have three bad options:

The Forking Nightmare: You create a deployment-us.yaml and a deployment-eu.yaml. This works for a week. Then, you update the container image in the US version but forget the EU version. Your clusters are now drifting, and your global consistency is broken.
The Helm "Value" Bloat: You try to make the Helm chart handle everything. You add if/else blocks for every possible regional variable. Your values.yaml becomes a 2,000-line monster that is impossible to read, and you are still limited by what the original chart author decided to expose.
Manual Intervention: You deploy the base app and then have a human (or a fragile script) go in and manually patch the environment variables. This is the opposite of GitOps and is prone to human error.

The core of the problem is that Location is an infrastructure property, but Configuration is an application property. Standard tools force you to mix them.

The Vision: A "Location-Aware" Delivery System

To solve the multi-cluster dilemma, we need to move away from traditional "template-and-deploy" models. A truly scalable solution should function like a Surgical Post-Processor - an intelligent layer that sits between your global source of truth and your local clusters.

A working solution must decouple What you are deploying from Where it is going. Here is how that ideal workflow functions in practice:

Centralized Intent
You define a single, global Intent. Instead of managing individual clusters, you manage logical groups. You simply state: "I want the Customer Portal application deployed to every cluster labeled app: portal." This keeps your configuration clean and centralized, regardless of whether you have ten clusters or ten thousand.
Dynamic Discovery
The system must be reactive. When a new cluster is provisioned - for example, in France - it should automatically be "discovered" by its metadata (e.g., location: europe). The deployment system sees this new destination and understands its unique identity without a human ever having to update a manual list.
Surgical Patching (The "DNA Splice")
This is the core of the solution. Instead of re-rendering an entire Helm chart with different values - which is often limited by what the chart developer allows - the system takes the final, fully-rendered YAML and "splices" in regional requirements.
Overcoming the "Missing Knob" Bottleneck
In traditional models, a Local SRE is often blocked if the Global App Architect didn't "expose a knob" (a parameter or variable) in the original template. With a Surgical Post-Processor, that dependency is broken:
- Global Team: Maintains a clean, generic Immutable Core. They focus on the "Gold Standard" manifest that works everywhere.
- Local SRE: If a regional requirement (like a specific security sidecar or a unique mounting point) isn't supported by the base template, the SRE creates an independent External Patch.
- The Result: The system injects these "missing knobs" directly into the final manifest at runtime. Local teams no longer have to wait for Global PR approvals to meet urgent regional compliance or infrastructure needs.
Late Binding
Localization details should not be hardcoded into the application code or the main deployment manifest. Instead, they should exist as independent Policy Packets that are only bound to the application at the final moment of delivery. This ensures that the application remains generic and portable, while the regional nuances are managed as separate, modular entities.

One Management Cluster. Infinite Possibilities. Total Control

Sveltos is a set of Kubernetes controllers that run in a central management cluster. From this single point of control, Sveltos can manage add-ons and applications across a vast fleet of managed Kubernetes clusters. It is a strictly declarative tool: you define the "Desired State" on the management cluster, and Sveltos ensures that state is reflected in every managed cluster, automatically correcting any drift.

In the management cluster, each individual managed Kubernetes cluster is represented by a dedicated resource. Because these are standard Kubernetes resources, you can attach Labels to them to categorize your fleet by region, purpose, or compliance tier.
Sveltos configuration utilizes a concept called a cluster selector. This selector acts like a dynamic filter based on those Kubernetes labels. By defining specific labels or combinations of labels, you can create a subset of clusters that share those characteristics - such as all clusters located in "Europe."

The Working Solution: Sveltos in Action

When you deploy our Customer Portal, Sveltos doesn't just push a static file. It follows a dynamic pipeline:

Identify: The Cluster Selector
Sveltos continuously monitors your fleet. It identifies a cluster because it matches a selector like env: prod and app: portal.
Contextualize: Discovery via Labels Sveltos looks at the cluster's metadata. If the cluster has the label location: europe, Sveltos uses that variable to dynamically "point" to your portal-patch-europe ConfigMap.
The "Surgical Splice": patchesFrom in Action
This is where the magic happens. Sveltos takes your generic "Vanilla" Deployment (the one that has no idea Europe exists) and your Generic ConfigMap, and performs two precise operations based on your YAML:
- The ConfigMap Splice: It injects the EU-specific HTML (<h1>Welcome to the EU Portal</h1>) into the regional-config object.
- The Deployment Splice: It adds a volume and a volumeMount to the customer-portal Deployment.
Late Binding: The Final Result
The application remains generic in your Git repository. The "EU-ness" is only bound to the application at the very last millisecond before the YAML is applied to the cluster.

The Concrete Demo

Let's build it. We will use a simple NGINX Deployment as our "Customer Portal" and inject regional identity.
A Kind cluster is used as management cluster. Then two extra Civo clusters all with label app=portal.

+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    site-us/portal      | v1.33.6-k3s1| Civo 3 Node - Medium Standard       |
|    site-eu/portal      | v1.32.5+k3s1| Civo 3 Node - Medium Standard       |
+------------------------+-------------+-------------------------------------+

Install Sveltos on Managament Cluster For this tutorial, we will install Sveltos in the management cluster. Sveltos installation details can be found here.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v1.5.1/manifest/manifest.yaml

And register the cluster with Sveltos

kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME     READY   VERSION        AGE     LABELS
site-eu     portal   true    v1.32.5+k3s1   3m43s   app=portal,location=europe
site-us     portal   true    v1.33.6+k3s1   4m20s   app=portal

Define the Global Application (The "What") This is our generic manifest stored in a ConfigMap on the management cluster. It has no regional logic and stays 100% standardized.

apiVersion: v1
kind: ConfigMap
metadata:
  name: portal-base-manifests
  namespace: default
data:
  deployment.yaml: |
    apiVersion: v1
    kind: Service
    metadata:
      name: portal-service
    spec:
      selector:
        app: portal
      ports:
      - protocol: TCP
        port: 80
        targetPort: 80
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: regional-config
    data: {} # Empty! No index.html yet.
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: customer-portal
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: portal
      template:
        metadata:
          labels:
            app: portal
        spec:
          volumes: []
          containers:
          - name: nginx
            image: nginx:latest
            volumeMounts: [] # Initialize here too
            env:
            - name: APP_MODE
              value: "production"

Create the Regional Policy Packet (The "Late Binding") On the management cluster, we define the "EU-specific" instructions.

apiVersion: v1
kind: ConfigMap
metadata:
  name: portal-patch-europe
  namespace: default
data:
  configmap-patch: |2
      target:
        kind: ConfigMap
        name: regional-config
      patch: |
        - op: add
          path: /data/index.html
          value: |
            <html><body><h1>Welcome to the EU Portal</h1></body></html>
  deployment-patch: |2
      target:
        kind: Deployment
        name: customer-portal
      patch: |
        - op: add
          path: /spec/template/spec/volumes/-
          value:
            name: html-volume
            configMap:
              name: regional-config
        - op: add
          path: /spec/template/spec/containers/0/volumeMounts/-
          value:
            name: html-volume
            mountPath: /usr/share/nginx/html/index.html
            subPath: index.html

The Orchestrator (The ClusterProfile) This tells Sveltos how to tie the "What" to the "Where" using the patchesFrom logic.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-portal-fleet
spec:
  clusterSelector:
    matchLabels:
      app: portal

  policyRefs:
  - name: portal-base-manifests
    namespace: default
    kind: ConfigMap

  # The Surgical Splice logic
  patchesFrom:
  - kind: ConfigMap
    # The Power of Templating:
    # This dynamically resolves to 'portal-patch-europe' for EU clusters
    name: "portal-patch-{{ index .Cluster.metadata.labels \"location\" }}"
    namespace: default
    optional: true

With the ClusterProfile active, Sveltos acts as the intelligent orchestrator. It identifies the clusters, evaluates their metadata, and applies the "Surgical Splice" only where the conditions are met.

Verification of Deployment

Running sveltosctl confirms that the global intent has been successfully realized across our fleet. Both the US and EU clusters received the base application, but their internal configurations differ based on their labels.

sveltosctl show addons
┌────────────────┬─────────────────┬───────────┬─────────────────┬─────────┬───────────────────────────────┬─────────────────┬────────────────────────────────────┐
│    CLUSTER     │  RESOURCE TYPE  │ NAMESPACE │      NAME       │ VERSION │             TIME              │ DEPLOYMENT TYPE │              PROFILES              │
├────────────────┼─────────────────┼───────────┼─────────────────┼─────────┼───────────────────────────────┼─────────────────┼────────────────────────────────────┤
│ site-eu/portal │ :Service        │ default   │ portal-service  │ N/A     │ 2026-02-21 13:52:53 +0100 CET │ Managed cluster │ ClusterProfile/deploy-portal-fleet │
│ site-eu/portal │ :ConfigMap      │ default   │ regional-config │ N/A     │ 2026-02-21 13:52:53 +0100 CET │ Managed cluster │ ClusterProfile/deploy-portal-fleet │
│ site-eu/portal │ apps:Deployment │ default   │ customer-portal │ N/A     │ 2026-02-21 13:52:54 +0100 CET │ Managed cluster │ ClusterProfile/deploy-portal-fleet │
│ site-us/portal │ apps:Deployment │ default   │ customer-portal │ N/A     │ 2026-02-21 13:52:44 +0100 CET │ Managed cluster │ ClusterProfile/deploy-portal-fleet │
│ site-us/portal │ :Service        │ default   │ portal-service  │ N/A     │ 2026-02-21 13:52:43 +0100 CET │ Managed cluster │ ClusterProfile/deploy-portal-fleet │
│ site-us/portal │ :ConfigMap      │ default   │ regional-config │ N/A     │ 2026-02-21 13:52:44 +0100 CET │ Managed cluster │ ClusterProfile/deploy-portal-fleet │
└────────────────┴─────────────────┴───────────┴─────────────────┴─────────┴───────────────────────────────┴─────────────────┴────────────────────────────────────┘

The power of this approach is visible when we inspect the regional-config ConfigMap in both environments.

Since the US cluster does not have a location: europe label, the patchesFrom directive (which was marked as optional: true) simply skips the patching process. The result is the exact "Gold Standard" manifest defined by the Global Team.

kubectl get configmap regional-config
NAME              DATA   AGE
regional-config   0      109s

In the EU cluster, Sveltos detected the location: europe label. It dynamically resolved the patch name to portal-patch-europe and performed a JSON Patch at the moment of delivery.

kubectl get configmap regional-config
NAME              DATA   AGE
regional-config   1      106s

The ultimate test of our Surgical Post-Processor is what the end-user sees. By simply changing a label on the cluster, Sveltos has dynamically altered the application's behavior at the edge.

Scaling Without the Friction

Multi-cluster management doesn't have to be a choice between "copy-paste" chaos and Helm-chart bloat. By shifting to a Surgical Post-Processing model with Sveltos, you decouple your global intent from local requirements.
This "late-binding" approach ensures that your core application remains immutable and easy to update, while regional nuances - from compliance headers to security sidecars - are injected dynamically at the edge. The result is a GitOps workflow that finally scales as fast as your infrastructure.
Stop forking your YAML and start orchestrating your intent.

Let’s Connect

Have questions or want to discuss the latest in tech? Instead of commenting here, let’s stay in touch on LinkedIn. I’m always happy to network and share insights!

Orchestrating Kubernetes Deployments Through Dependencies

Gianluca — Tue, 18 Mar 2025 13:11:31 +0000

In complex Kubernetes environments, deploying applications often involves intricate dependencies. Ensuring that components are installed in the correct order, and that required services are healthy before proceeding, is crucial for stability and reliability. Manually managing these dependencies becomes increasingly challenging as the number of applications and their interrelationships grow. This necessitates robust mechanisms for automating deployment sequences, verifying application health, and optimizing resource utilization to prevent redundant installations. Effective dependency management is vital for simplifying deployments, reducing operational overhead, and ensuring the smooth functioning of distributed applications within Kubernetes.

Sveltos Solution

Sveltos is a set of Kubernetes controllers operating within a management cluster. From this central point, Sveltos manages add-ons and applications across a fleet of managed Kubernetes clusters. It employs a declarative approach, ensuring that the desired state is consistently reflected across these managed environments.

A Sveltos ClusterProfile/Profile is a bundle of Helm charts and Kubernetes resources that Sveltos will deploy to every cluster that matches its selection criteria.

To simplify complex deployments, Sveltos allows you to create multiple profiles and specify a deployment order using the dependsOn field, ensuring all profile prerequisites are met.

Consider deploying admission policies. Here are the relevant ClusterProfile snippets:

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: kyverno
spec:
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     3.3.4
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    ...

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: admission-policies
spec:
  dependsOn:
  - kyverno
  clusterSelector:
    matchLabels:      
      env: production
  ...

Notice that the kyverno ClusterProfile lacks a clusterSelector, so it won't be deployed on its own. The admission-policies ClusterProfile, however, has a clusterSelector targeting production clusters and a dependsOn field referencing kyverno. When this profile is created, Sveltos resolves its dependency. Any time a production cluster matches the admission-policies selector, Sveltos will first deploy the Kyverno Helm chart and then apply the admission policies.

Beyond Order: Verifying Application Health

While deployment order is a fundamental aspect of application dependencies, it’s not always the sole requirement. In the Kyverno example, we prioritize deploying the CRDs before the custom resources, focusing on sequence. But consider a scenario where an application relies on a database. The database must not only be deployed first but also be fully initialized and ready to accept connections. In such cases, the state of the dependency becomes critical. Dependency management tools must accommodate these more complex state-based requirements.

Building on our Kyverno example, let’s address the need for state-based dependencies. We want to ensure that all Kyverno deployments are fully operational before deploying admission policies. To achieve this, we add a validateHealths section to the kyverno ClusterProfile.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: kyverno
spec:
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     3.3.4
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install
    values: |
      admissionController:
        replicas: 3
      backgroundController:
        replicas: 3
validateHealths:
- name: deployment-health
  featureID: Helm
  group: "apps"
  version: "v1"
  kind: "Deployment"
  namespace: kyverno
  script: |
    function evaluate()
      hs = {}
      hs.healthy = false
      hs.message = "available replicas not matching requested replicas"
      if obj.status ~= nil then
        if obj.status.availableReplicas ~= nil then
          if obj.status.availableReplicas == obj.spec.replicas then
            hs.healthy = true
          end
        end
      end
      return hs
    end

Recursive Resolution

One of the key strengths of Sveltos is its ability to handle complex dependency chains. Imagine an application whoami that relies on Traefik, which itself depends on cert-manager. With Sveltos, you only need to define the deployment of whoami. Sveltos will automatically resolve the entire dependency tree, ensuring cert-manager and Traefik are deployed in the correct order before whoami is deployed. This simplifies complex deployments by automating the resolution of multi-level dependencies.

This capability is especially valuable in environments where different administrators manage distinct components. For instance, the whoami application might be managed by the application administrator, Traefik by the networking administrator, and cert-manager by the security administrator. Sveltos eliminates the need for manual coordination, allowing each administrator to focus on their respective components while ensuring the overall deployment succeeds.

Sveltos Dashboard displays those dependencies:

Dependency Deduplication

Sveltos efficiently manages shared dependencies by ensuring they are deployed only once per cluster, even when multiple profiles rely on them. This optimizes resource utilization and prevents redundant deployments. Crucially, Sveltos maintains a dependency as long as any profile requiring it is active on the cluster.

Consider a 3-tier application scenario:

Database Layer (Profile postgresql): Deploys a PostgreSQL database.
Backend Layer (Profiles backend-service-1, backend-service-2): Two distinct microservices, each requiring the PostgreSQL database.
Frontend Layer (Profiles frontend-app-1, frontend-app-2): Two frontend applications, each dependent on a respective backend service.

Here’s how this translates to Sveltos profiles and dependencies:

Profile postgresql: Deploys the PostgreSQL database.
Profile backend-service-1: Depends on Profile postgresql.
Profile frontend-app-1: Depends on Profile backend-service-1.
Profile backend-service-2: Depends on Profile postgresql.
Profile frontend-app-2: Depends on Profile backend-service-2.

When frontend-app-1 is deployed, Sveltos first deploys postgresql and then backend-service-1, resolving the dependency chain.

Subsequently, when frontend-app-2 is deployed to the same cluster, Sveltos recognizes that postgresql is already present and avoids redeploying it.

If frontend-app-1 is then removed, backend-service-1 is also removed. However, postgresql persists because it remains a dependency of frontend-app-2.

Finally, only when frontend-app-2 is removed will Sveltos remove backend-service-2 and postgresql, as they are no longer required by any active profile on the cluster.

Contact Information

If you have some questions, would like to have a friendly chat or just network to not miss any topics, then don’t use the comment function at medium, just feel free to add me to your LinkedIn network!

👏 Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.

Automating Kro Deployments Across Kubernetes Fleets

Gianluca — Fri, 14 Mar 2025 12:43:20 +0000

Deploying and managing complex applications across multiple Kubernetes clusters can be a daunting task. This tutorial demonstrates how to simplify this process using Kro, the Kube Resource Orchestrator, and Sveltos, a powerful multi-cluster management tool. We’ll walk through a practical example, showing you how to automate Kro deployments across your entire Kubernetes fleet.

Kro

Kro, or Kube Resource Orchestrator, is an open-source Kubernetes project designed to simplify the deployment and management of complex applications on Kubernetes.

Kro’s installation introduces the ResourceGraphDefinition CustomResourceDefinition, enabling teams to build custom APIs. By defining custom resources within the RGD, teams can create abstractions like ApplicationStack, which encapsulate complex deployments. This simplifies Kubernetes for developers, who can then interact with high-level APIs instead of managing low-level resources.

In the above example, the Platform Team has created a Resource Group named Application Stack that encapsulates the necessary resources, along with any additional logic, abstractions, and security best practices.

When the ResourceGraphDefinition instance is applied to the cluster, a new API of kind ApplicationStack is created and available for Developer to interact with. The Developers no longer need to directly manage the underlying infrastructure complexities, as the custom API handles the deployment and configuration of the required resources.

Using kro, the Platform Team can enable Developer teams to quickly deploy and manage applications and their dependencies as one unit, handling the entire lifecycle from deployment to maintenance.

Managing a Fleet of Clusters

Sveltos automates and scales Kro deployments across multiple clusters.

Platform and security teams configure Kro and ResourceGroupDefinitions through ClusterProfiles, deferring target cluster selection because at this stage they don’t know where the dependent applications will be deployed, and importantly, because Kro should only be deployed on clusters where it’s actually needed.

Application admins, who possess the knowledge of application deployment locations, then create deployment-specific profiles, selecting target clusters and declaring dependencies on the pre-defined ResourceGroupDefinitions.

Sveltos automates the process, identifying matching clusters, resolving dependencies, and deploying Kro, ResourceGroupDefinitions, and applications in the correct order across the matching clusters.

Step 1: Install Sveltos on Management Cluster

Sveltos installation details can be found here.

For this demo we will use helm, so while pointing to the management cluster:

helm repo add projectsveltos https://projectsveltos.github.io/helm-charts
helm repo update
helm install projectsveltos projectsveltos/projectsveltos -n projectsveltos - create-namespace

Step 2: Register Managed Clusters with Sveltos

For this tutorial, I’m using a GKE cluster and an EKS cluster as production environments, along with two Civo clusters for staging.

To register these clusters for Sveltos management, you’ll need their respective Kubeconfigs. Use the following commands, adjusting namespace, cluster name, Kubeconfig path, and labels accordingly:

sveltosctl register cluster --namespace=<namespace> --cluster=<cluster name> \
    --kubeconfig=<path to file with Kubeconfig> \
    --labels=env=production

sveltosctl register cluster --namespace=<namespace> --cluster=<cluster name> \
    --kubeconfig=<path to file with Kubeconfig> \
    --labels=env=staging

Verify all clusters are correctly registered:

kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME       READY   VERSION               LABELS
civo        cluster1   true    v1.30.5+k3s1          env=staging,projectsveltos.io/k8s-version=v1.30.5,sveltos-agent=present
civo        cluster2   true    v1.29.8+k3s1          env=staging,projectsveltos.io/k8s-version=v1.29.8,sveltos-agent=present
eks         cluster    true    v1.31.6-eks-bc803b4   env=production,projectsveltos.io/k8s-version=v1.31.6,sveltos-agent=present
gke         cluster    true    v1.31.5-gke.1233000   env=production,projectsveltos.io/k8s-version=v1.31.5,sveltos-agent=present
mgmt        mgmt       true    v1.32.2               projectsveltos.io/k8s-version=v1.32.2,sveltos-agent=present

Step 3: Configuring Kro Deployment with a ClusterProfile

This ClusterProfile configures Sveltos for Kro deployment, but intentionally omits a clusterSelector. The platform administrator is deferring cluster selection until the specific need for Kro on managed clusters becomes clear.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-kro
spec:
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    oci://ghcr.io/kro-run/kro
    repositoryName:   kro
    chartName:        kro
    chartVersion:     0.2.1
    releaseName:      kro
    releaseNamespace: kro
    helmChartAction:  Install

To apply this ClusterProfile to your management cluster, use the following command:

kubectl apply -f https://raw.githubusercontent.com/gianlucam76/devops-tutorial/refs/heads/main/kro/clusterprofile-deploy-kro.yaml

Step 4: Configuring Kro ResourceGroupDefinition with a ClusterProfile

This ClusterProfile configures Sveltos to deploy the Kro ResourceGraphDefinition.

It intentionally omits a clusterSelector, as the target clusters are determined later when Kro’s necessity on managed clusters is known.

Crucially, this profile also declares a dependency on the deploy-kro ClusterProfile, ensuring Kro is installed before the ResourceGraphDefinition.

To apply this ClusterProfile to your management cluster, use the following command:

kubectl apply -f https://raw.githubusercontent.com/gianlucam76/devops-tutorial/refs/heads/main/kro/clusterprofile-deploy-resource-graph-defintion.yaml

Step 5: Configuring Kro Application with a ClusterProfile

In this step, we’ll deploy a Kro application specifically to our staging clusters using a ClusterProfile.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-kro-application-staging
spec:
  dependsOn:
  - deploy-kro-resource-graph-defintion
  clusterSelector:
    matchLabels:
      env: staging

As you can see, the clusterSelector field targets clusters with the env: staging label, ensuring the application is deployed only to those environments.

The dependsOn field specifies that the deploy-kro-resource-graph-defintion ClusterProfile must be applied first, guaranteeing the necessary ResourceGraphDefinition is in place.

Sveltos handles the dependency resolution automatically, creating a Directed Acyclic Graph (DAG) to ensure all required profiles are deployed in the correct order to the selected staging clusters.

To apply this ClusterProfile to your management cluster, execute the following command:

kubectl apply -f https://raw.githubusercontent.com/gianlucam76/devops-tutorial/refs/heads/main/kro/clusterprofile-deploy-application-staging.yaml

After applying the ClusterProfile, you can verify the deployment using sveltosctl show addons. The output should resemble the following:

sveltosctl show addons
+---------------+---------------------------------+-----------+-------------------------+---------+-------------------------------+----------------------------------------------------+
|    CLUSTER    |          RESOURCE TYPE          | NAMESPACE |          NAME           | VERSION |             TIME              |                      PROFILES                      |
+---------------+---------------------------------+-----------+-------------------------+---------+-------------------------------+----------------------------------------------------+
| civo/cluster1 | helm chart                      | kro       | kro                     | 0.2.1   | 2025-03-12 13:49:42 +0100 CET | ClusterProfile/deploy-kro                          |
| civo/cluster1 | kro.run:ResourceGraphDefinition |           | my-application          | N/A     | 2025-03-12 13:49:50 +0100 CET | ClusterProfile/deploy-kro-resource-graph-defintion |
| civo/cluster1 | kro.run:Application             | default   | my-application-instance | N/A     | 2025-03-12 13:50:07 +0100 CET | ClusterProfile/deploy-kro-application-staging      |
| civo/cluster2 | helm chart                      | kro       | kro                     | 0.2.1   | 2025-03-12 13:49:42 +0100 CET | ClusterProfile/deploy-kro                          |
| civo/cluster2 | kro.run:Application             | default   | my-application-instance | N/A     | 2025-03-12 13:49:56 +0100 CET | ClusterProfile/deploy-kro-application-staging      |
| civo/cluster2 | kro.run:ResourceGraphDefinition |           | my-application          | N/A     | 2025-03-12 13:49:50 +0100 CET | ClusterProfile/deploy-kro-resource-graph-defintion |
+---------------+---------------------------------+-----------+-------------------------+---------+-------------------------------+----------------------------------------------------+

This output confirms that Kro, the ResourceGraphDefinition, and the application have been successfully deployed to both civo/cluster1 and civo/cluster2, our staging clusters.

Step 6: Deploy Kro Application to production clusters

Now, let’s extend our deployment to the production environments. We’ll use another ClusterProfile to deploy the Kro application to our production clusters.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-kro-application-production
spec:
  dependsOn:
  - deploy-kro-resource-graph-defintion
  clusterSelector:
    matchLabels:
      env: production

This ClusterProfile, much like the staging one, uses a clusterSelector. Here, it targets clusters with the label env: production, ensuring the application is deployed only to your production clusters.

The dependsOn field again ensures that the deploy-kro-resource-graph-defintion ClusterProfile is deployed first, maintaining the correct deployment order.

To apply this ClusterProfile to your management cluster, use the following command:

kubectl apply -f https://raw.githubusercontent.com/gianlucam76/devops-tutorial/refs/heads/main/kro/clusterprofile-deploy-application-production.yaml

After applying the ClusterProfile, verify the deployment using sveltosctl show addons. You should see the following output, confirming the application’s deployment to your production clusters:

sveltosctl show addons
+---------------+---------------------------------+-----------+-------------------------+---------+-------------------------------+----------------------------------------------------+
|    CLUSTER    |          RESOURCE TYPE          | NAMESPACE |          NAME           | VERSION |             TIME              |                      PROFILES                      |
+---------------+---------------------------------+-----------+-------------------------+---------+-------------------------------+----------------------------------------------------+
| civo/cluster1 | helm chart                      | kro       | kro                     | 0.2.1   | 2025-03-12 13:49:42 +0100 CET | ClusterProfile/deploy-kro                          |
| civo/cluster1 | kro.run:Application             | default   | my-application-instance | N/A     | 2025-03-12 13:50:07 +0100 CET | ClusterProfile/deploy-kro-application-staging      |
| civo/cluster1 | kro.run:ResourceGraphDefinition |           | my-application          | N/A     | 2025-03-12 13:49:50 +0100 CET | ClusterProfile/deploy-kro-resource-graph-defintion |
| civo/cluster2 | helm chart                      | kro       | kro                     | 0.2.1   | 2025-03-12 13:49:42 +0100 CET | ClusterProfile/deploy-kro                          |
| civo/cluster2 | kro.run:Application             | default   | my-application-instance | N/A     | 2025-03-12 13:49:56 +0100 CET | ClusterProfile/deploy-kro-application-staging      |
| civo/cluster2 | kro.run:ResourceGraphDefinition |           | my-application          | N/A     | 2025-03-12 13:49:50 +0100 CET | ClusterProfile/deploy-kro-resource-graph-defintion |
| eks/cluster   | helm chart                      | kro       | kro                     | 0.2.1   | 2025-03-12 13:52:43 +0100 CET | ClusterProfile/deploy-kro                          |
| eks/cluster   | kro.run:Application             | default   | my-application-instance | N/A     | 2025-03-12 13:53:04 +0100 CET | ClusterProfile/deploy-kro-application-production   |
| eks/cluster   | kro.run:ResourceGraphDefinition |           | my-application          | N/A     | 2025-03-12 13:52:51 +0100 CET | ClusterProfile/deploy-kro-resource-graph-defintion |
| gke/cluster   | helm chart                      | kro       | kro                     | 0.2.1   | 2025-03-12 13:52:43 +0100 CET | ClusterProfile/deploy-kro                          |
| gke/cluster   | kro.run:ResourceGraphDefinition |           | my-application          | N/A     | 2025-03-12 13:52:50 +0100 CET | ClusterProfile/deploy-kro-resource-graph-defintion |
| gke/cluster   | kro.run:Application             | default   | my-application-instance | N/A     | 2025-03-12 13:53:04 +0100 CET | ClusterProfile/deploy-kro-application-production   |
+---------------+---------------------------------+-----------+-------------------------+---------+-------------------------------+----------------------------------------------------+

As you can see, the application has now been deployed to both eks/cluster and gke/cluster, our production environments, in addition to the existing staging deployments.

If you have also deployed Sveltos dashboard, you can see it to display all managed clusters and all resources deployed in each managed cluster:

Conclusion

In this tutorial, we’ve demonstrated how Sveltos simplifies the deployment and management of Kro applications across a fleet of Kubernetes clusters. By leveraging ClusterProfiles, we were able to deploy Kro and its associated resources in a staged and controlled manner, first setting up the foundational components and then deploying applications to specific environments based on their requirements.

Sveltos’s ability to defer target cluster selection and resolve dependencies allowed us to efficiently manage Kro deployments, ensuring that resources were only deployed to the clusters where they were needed. We showcased how to target staging and production environments using clusterSelector and how to maintain the correct deployment order using dependsOn.

Beyond what we’ve covered here, Sveltos offers even greater flexibility. It can also be used to deploy Kro applications in response to events occurring in the managed clusters.

This event-driven capability allows for dynamic and automated deployments, enabling you to react to changes in your environment in real-time. This opens up possibilities for sophisticated automation,such as scaling applications based on resource utilization or deploying specific configurations in response to security alerts.

By combining the power of Kro for application orchestration with Sveltos for fleet management and event-driven automation, you can significantly streamline your Kubernetes operations and achieve greater efficiency and reliability.

Contact Information

👏 Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.

Click-to-Cluster: GitOps EKS Provisioning

Gianluca — Tue, 04 Mar 2025 13:49:25 +0000

Imagine a scenario where you need to provide dedicated Kubernetes environments to individual users or teams on demand. Manually creating and managing these clusters can be time-consuming and error-prone. In this tutorial, we'll demonstrate how to automate this process using a powerful combination of ArgoCD, Sveltos, and ClusterAPI. We'll set up a GitOps workflow where adding a new user via a pull request triggers the automatic provisioning of a dedicated EKS cluster. ArgoCD will maintain the desired state of our management cluster, Sveltos will detect changes and orchestrate cluster creation, and ClusterAPI will handle the provisioning of the EKS clusters. This tutorial will provide a hands-on experience in building a robust and scalable cluster provisioning pipeline.

Argo

Argo CD is an open-source, continuous delivery (CD) tool for Kubernetes that automates the deployment, monitoring, and rollback of applications. It is a declarative tool that uses GitOps principles to ensure that the desired state of an application is always reflected in the actual state of the Kubernetes Management or Workload cluster.

Sveltos

ClusterAPI

ClusterAPI (CAPI) is a Kubernetes sub-project that brings declarative, Kubernetes-style APIs to cluster creation, configuration, and management. It allows you to manage the lifecycle of Kubernetes clusters using the same declarative approach you use for managing applications.

Step 1: Install ClusterAPI on Management Cluster

We'll use a Kind cluster as our management cluster. To begin, we need to deploy ClusterAPI with the AWS infrastructure provider.
First, configure your AWS credentials:

export AWS_ACCESS_KEY_ID=<YOUR ACCESS KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR SECRET ACCESS KEY>
clusterawsadm bootstrap iam create-cloudformation-stack --region us-east-1
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

Then, initialize ClusterAPI:

export AWS_REGION=us-east-1
export EKS=true
export EXP_MACHINE_POOL=true
export CAPA_EKS_IAM=true
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=capi-eks
clusterctl init --infrastructure aws

Step 2: Install ArgoCD on Management Cluster

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl config set-context --current --namespace=argocd
kubectl port-forward svc/argocd-server -n argocd 8080:443

The admin password to connect to ArgoCD dashboard is in the Secret named argocd-initial-admin-secret in the argocd namespace.

Step 3: Deploy Sveltos

We will leverage Argo CD to automate the deployment of Sveltos onto your management Kubernetes cluster.

argocd app create sveltos --repo https://github.com/projectsveltos/helm-charts.git --path charts/projectsveltos --dest-server https://kubernetes.default.svc --dest-namespace projectsveltos

This directs Argo CD to create an application named sveltos, retrieving the Sveltos Helm chart from the specified GitHub repository and deploying it to the projectsveltos namespace within the management cluster. Next, we'll deploy the Sveltos configurations using Argo CD.

argocd app create sveltos-configuration --repo https://github.com/gianlucam76/devops-tutorial.git --path argocd-sveltos-clusterapi-eks --dest-server https://kubernetes.default.svc

The Sveltos configuration we're deploying establishes a dynamic cluster creation process. First, it monitors a ConfigMap named existing-users within the default namespace. This ConfigMap maintains a list of users, each entry formatted as user-id: cluster-type, where cluster-type designates either production or staging. Subsequently, whenever a new user entry is detected in this ConfigMap, Sveltos triggers the deployment of a ClusterAPI configuration, which in turn initiates the creation of a new EKS cluster. Importantly, each newly created cluster will be labeled with type:cluster-type, allowing Sveltos to immediately apply the appropriate configuration, whether for production or staging environments, based on the cluster's label.

Before we proceed further, label the management cluster with type: mgmt

kubectl label sveltoscluster -n mgmt mgmt type=mgmt

Step 4: Push a PR to add a new user

To add a new user and trigger the creation of a corresponding EKS cluster, we will implement a straightforward GitOps workflow. First, we'll submit a pull request (PR) that modifies the existing-users.yaml ConfigMap within our repository. Specifically, this PR will introduce a new user entry, user1: production, within the data section of the ConfigMap, as shown in the provided diff.

diff --git a/argocd-sveltos-clusterapi-eks/existing-users.yaml b/argocd-sveltos-clusterapi-eks/existing-users.yaml
index ab0d862..10987b3 100644
--- a/argocd-sveltos-clusterapi-eks/existing-users.yaml
+++ b/argocd-sveltos-clusterapi-eks/existing-users.yaml
@@ -3,3 +3,5 @@ kind: ConfigMap
 metadata:
   name: existing-users
   namespace: default
+data:
+  user1: production

Once the PR is merged, we'll then instruct Argo CD to synchronize these changes to the management cluster. This synchronization will apply the updated ConfigMap, thereby signaling Sveltos to initiate the ClusterAPI-driven provisioning of a new production EKS cluster for user1.

Step 5: Sveltos triggers creation of a new EKS cluster

Sveltos will immediately detect the new user entry, user1: production. This detection triggers Sveltos to deploy all the necessary ClusterAPI resources to the management cluster, effectively orchestrating the creation of a new EKS cluster. Executing sveltosctl show addons will then reveal the deployed resources, as shown below:

sveltosctl show addons
+-----------+-------------------------------------------------------+-----------+------------------------------+---------+-------------------------------+---------------------------------+
|  CLUSTER  |                     RESOURCE TYPE                     | NAMESPACE |             NAME             | VERSION |             TIME              |            PROFILES             |
+-----------+-------------------------------------------------------+-----------+------------------------------+---------+-------------------------------+---------------------------------+
| mgmt/mgmt | cluster.x-k8s.io:Cluster                              | user1     | capi-eks-user1               | N/A     | 2025-03-04 12:10:02 +0100 CET | ClusterProfile/deploy-resources |
| mgmt/mgmt | infrastructure.cluster.x-k8s.io:AWSManagedCluster     | user1     | capi-eks-user1               | N/A     | 2025-03-04 12:10:02 +0100 CET | ClusterProfile/deploy-resources |
| mgmt/mgmt | controlplane.cluster.x-k8s.io:AWSManagedControlPlane  | user1     | capi-eks-control-plane-user1 | N/A     | 2025-03-04 12:10:02 +0100 CET | ClusterProfile/deploy-resources |
| mgmt/mgmt | cluster.x-k8s.io:MachinePool                          | user1     | capi-eks-pool-0-user1        | N/A     | 2025-03-04 12:10:02 +0100 CET | ClusterProfile/deploy-resources |
| mgmt/mgmt | infrastructure.cluster.x-k8s.io:AWSManagedMachinePool | user1     | capi-eks-pool-0-user1        | N/A     | 2025-03-04 12:10:02 +0100 CET | ClusterProfile/deploy-resources |
+-----------+-------------------------------------------------------+-----------+------------------------------+---------+-------------------------------+---------------------------------+

Sveltos dashboard can also be used to see what Sveltos has deployed to the management cluster:

Step 6: Sveltos deploys production policies

Once ClusterAPI completes the provisioning of the new EKS cluster, Sveltos will automatically deploy the add-ons configured for a production environment. This deployment includes:

The cert-manager Helm chart, to manage and issue TLS certificates.
The kyverno Helm chart, for policy-based control of Kubernetes resources.
The prometheus and grafana Helm charts, to provide monitoring and visualization capabilities.
A kyverno admission policy specifically configured to disallow-latest-tag, enforcing the use of explicit image tags for container deployments.

sveltosctl show addons --namespace=user1
+----------------------+--------------------------+--------------+---------------------+---------+-------------------------------+------------------------------------------+
|       CLUSTER        |      RESOURCE TYPE       |  NAMESPACE   |        NAME         | VERSION |             TIME              |                 PROFILES                 |
+----------------------+--------------------------+--------------+---------------------+---------+-------------------------------+------------------------------------------+
| user1/capi-eks-user1 | helm chart               | cert-manager | cert-manager        | v1.16.3 | 2025-03-04 12:28:00 +0100 CET | ClusterProfile/deploy-cert-manager       |
| user1/capi-eks-user1 | helm chart               | prometheus   | prometheus          | 26.0.0  | 2025-03-04 12:29:20 +0100 CET | ClusterProfile/prometheus-grafana        |
| user1/capi-eks-user1 | helm chart               | grafana      | grafana             | 8.6.4   | 2025-03-04 12:29:29 +0100 CET | ClusterProfile/prometheus-grafana        |
| user1/capi-eks-user1 | helm chart               | kyverno      | kyverno-latest      | 3.3.4   | 2025-03-04 12:28:50 +0100 CET | ClusterProfile/deploy-kyverno-production |
| user1/capi-eks-user1 | kyverno.io:ClusterPolicy |              | disallow-latest-tag | N/A     | 2025-03-04 12:29:19 +0100 CET | ClusterProfile/deploy-kyverno-resources  |
+----------------------+--------------------------+--------------+---------------------+---------+-------------------------------+------------------------------------------+

You can also view deployed Helm charts within the Sveltos dashboard.

Step 7: Removing user

Just as easily as we created a cluster by adding a user, we can remove a user and, consequently, delete their associated EKS cluster. This is achieved through the same GitOps workflow, but in reverse.

To remove a user, we simply submit a pull request (PR) that removes the user's entry from the existing-users.yaml ConfigMap. For example, to remove user1, we would revert the changes we made in Step 4. Once this PR is merged and Argo CD synchronizes the changes to the management cluster, Sveltos detects the removal of the user from the ConfigMap.

Sveltos, upon detecting this change, proceeds to delete all the ClusterAPI resources it previously deployed for that user. This includes the Cluster, AWSManagedCluster, AWSManagedControlPlane, MachinePool, and AWSManagedMachinePool resources within the user's namespace (in this case, "user1").

Because ClusterAPI operates declaratively, deleting these ClusterAPI resources triggers the deletion of the underlying EKS cluster. ClusterAPI's AWS infrastructure provider interprets the removal of these resources as a request to terminate the corresponding cloud resources.

Therefore, by simply removing the user entry from the existing-users.yaml ConfigMap and allowing Argo CD and Sveltos to synchronize the changes, we effectively automate the entire lifecycle of the EKS cluster, from creation to deletion. This ensures that resources are efficiently managed and that orphaned clusters are avoided, maintaining a clean and cost-effective environment.

Conclusion

In this tutorial, we've demonstrated how to build a fully automated, GitOps-driven pipeline for provisioning dedicated EKS clusters on demand. By leveraging the power of Argo CD, Sveltos, and ClusterAPI, we've established a robust and scalable solution that eliminates the manual effort and potential errors associated with traditional cluster management.
We've seen how:

Argo CD maintains the desired state of our management cluster, ensuring that all configurations are synchronized with our Git repository.
Sveltos acts as a dynamic orchestrator, detecting changes in our user configuration and triggering the creation of new EKS clusters.
ClusterAPI handles the heavy lifting of provisioning and managing the EKS clusters themselves, providing a consistent and declarative approach.

Contact Information

If you have some questions, would like to have a friendly chat or just network to not miss any topics, then don't use the comment function at medium, just feel free to add me to your LinkedIn network!

👏 Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.

Orchestrating Secrets Across Kubernetes Clusters: Precision and Automation

Gianluca — Mon, 03 Mar 2025 08:38:10 +0000

Secrets, like API keys, passwords, and certificates, are critical confidential data. Centralized secret management across Kubernetes clusters offers key benefits: improved visibility, simplified updates, and enhanced security. By managing secrets in one location, you gain a clear overview, automate updates to all clusters, and can quickly isolate compromised clusters by revoking their access through label changes.

What is Sveltos?

Sveltos is a set of Kubernetes controllers that run in the management cluster. From the management cluster, Sveltos can manage add-ons and applications on a fleet of managed Kubernetes clusters. It is a declarative tool to ensure that the desired state of an application is always reflected in the actual state of the Kubernetes managed clusters.

Sveltos installation details can be found here.

Distrubute Secret to all Production clusters

Sveltos is used to manage four Kubernetes clusters: EKS, GKE, Civo, and an on-premises deployment. The EKS, GKE, and Civo clusters share the label env: production.

A secret created in the management cluster is then distributed by Sveltos to the production clusters.

Create Secret

kubectl create secret generic -n default login-credential --from-literal=username=admin --from-literal=password=mypassword

Post the Sveltos configuration:

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector: # Deploy to all cluster with label env: production
    matchLabels:
      env: production
  templateResourceRefs:
  - resource: # Fetch the Secret from the management cluster
      apiVersion: v1
      kind: Secret
      name: login-credentials
      namespace: default
    identifier: Credentials
  policyRefs: # Use this ConfigMap as a template and deploy its content, populating it with data from the referenced Secret.
  - kind: ConfigMap
    name: info
    namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: info
  namespace: default
  annotations:
    projectsveltos.io/template: "true"  # add annotation to indicate Sveltos content is a template
data:
  secret.yaml: |
    {{ copy "Credentials" }}

This YAML configuration tells Sveltos to perform three actions. First, it fetches the login-credentials Secret from the default namespace. Second, it takes the info ConfigMap from the default namespace, uses it as a template, and deploys the instantiated content using data from the login-credentials Secret. Finally, it deploys the login-credentials Secret to all Kubernetes clusters with the env: production label.

Automating Secret Deployment with Event-Based Triggers

Imagine you have a multi-cluster Kubernetes environment where different teams or applications need access to specific secrets. Instead of manually deploying these secrets to each cluster or pre-provisioning them, you want a more dynamic and automated approach.

By using Sveltos to monitor for events like the creation of namespaces with certain labels (e.g., credentials: required), you can automatically trigger the deployment of relevant secrets.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: requiring-credentials
spec:
  collectResources: true
  resourceSelectors:
  - group: ""
    version: "v1"
    kind: "Namespace"
    labelFilters:
    - key: credentials
      operation: Equal
      value: required
---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: distribute-credentials
spec:
  sourceClusterSelector:
    matchLabels:
      env: production
  eventSourceName: requiring-credentials
  templateResourceRefs:
  - resource:
      apiVersion: v1
      kind: Secret
      name: login-credentials
      namespace: default
    identifier: Credentials
  - resource: # This refers to the resource that Sveltos dynamically generates using ConfigMapGenerator.
      apiVersion: v1
      kind: ConfigMap
      name: "{{ .Cluster.metadata.namespace }}-{{ .Cluster.metadata.name }}-namespaces"
      namespace: projectsveltos
    identifier: Namespaces
  configMapGenerator:
  - name: namespaces
    namespace: default
    nameFormat: "{{ .Cluster.metadata.namespace }}-{{ .Cluster.metadata.name }}-namespaces"
  policyRefs:
  - kind: ConfigMap
    name: info
    namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: namespaces 
  namespace: default
  annotations:
    projectsveltos.io/instantiate: "true"
data:
  namespaces: |
    {{- range $v := .MatchingResources }}
       {{ $v.Name }}: "ok"
    {{- end }} 
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: info
  namespace: default
  annotations:
    projectsveltos.io/template: "true"  # add annotation to indicate Sveltos content is a template
data:
  secret.yaml: |
    {{ $namespaces := ( ( index (getResource "Namespaces").data "namespaces" ) | fromYaml ) }}
    {{- range $key, $value := $namespaces }}
        apiVersion: v1
        kind: Secret
        metadata:
          namespace: {{ $key }}
          name: {{ (getResource "Credentials").metadata.name }}
        data:
          {{- range $secretKey, $secretValue := (getResource "Credentials").data }}
            {{ $secretKey }} : {{ $secretValue }}
          {{- end }}
    ---
    {{- end }}

This Sveltos configuration automates the targeted deployment of the login-credentials secret. It monitors your production Kubernetes clusters for the creation of new namespaces labeled credentials: required. Upon detection, Sveltos automatically deploys the login-credentials secret into those specific namespaces.

This allows for dynamic, targeted, and automated secret distribution, ensuring that secrets are only deployed where and when they are needed.

Contact Information

👏 Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.

Thank you for reading!

Kubernetes on Autopilot: Event-Driven Automation Across Clusters

Gianluca — Thu, 06 Feb 2025 15:16:51 +0000

In today’s dynamic cloud environments, managing Kubernetes resources across multiple clusters can be a complex task. Traditional methods often lack the agility and event-driven architecture needed to respond quickly to changes and automate resource provisioning. This article explores how Sveltos, in conjunction with NATS and JetStream, simplifies multi-cluster Kubernetes management through event-driven automation, streamlining operations and improving responsiveness.

What is Sveltos

Within the management cluster, each managed Kubernetes cluster is represented by a dedicated resource, to which labels can be attached. Sveltos leverages cluster selectors, essentially label-based filters, to target specific groups of managed clusters for configuration.

What is NATS

NATS is a lightweight, high-performance messaging system optimized for speed and scalability. It excels at publish/subscribe communication. JetStream enhances NATS with robust streaming and data management features, including message persistence, flow control, and ordered delivery, creating a powerful platform for modern distributed systems.

Sveltos Event Framework: Expanding Beyond Kubernetes with NATS

Sveltos offers an event-driven workflow: define events, select target clusters, and specify the add-ons/applications to deploy when those events occur. (see the documentation and video by Colin for more details.)

Initially, Sveltos’s event framework focused on monitoring Kubernetes resources within the managed clusters, enabling automated responses to changes in those resources. However, with its NATS integration, Sveltos now supports a significantly expanded event horizon. It can now also react to events originating outside the Kubernetes environment. This allows Sveltos to respond to a much wider range of triggers, including events from external systems, applications, and even manual interactions, dramatically increasing its automation and integration capabilities.

Sveltos can connect to a NATS server and listen for CloudEvents published on NATS subjects. When a CloudEvent is received, Sveltos can automatically deploy or manage other Kubernetes resources in response to. Consider user authentication: a successful login (published as a CloudEvent) can trigger Sveltos to automatically create a user-specific namespace. Similarly, a logout event can initiate automated cleanup within the cluster.

Lab Setup

A Kind cluster is used as management cluster. Then two extra Civo clusters all with label env=production.

+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    civo/cluster1       | v1.29.8+k3s1| Civo 3 Node - Medium Standard       |
|    civo/cluster2       | v1.30.5+k3s1| Civo 3 Node - Medium Standard       |
+------------------------+-------------+-------------------------------------+

Step 1: Install Sveltos on Managament Cluster

For this tutorial, we will install Sveltos in the management cluster. Sveltos installation details can be found here.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.46.1/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.46.1/manifest/default-classifier.yaml

Step 2: Register Clusters with Sveltos

Using Civo UI, download the Kubeconfigs, then:

kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=production
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=production

Verify your Civo clusters were successfully registered:

kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME       READY   VERSION        LABELS
civo        cluster1   true    v1.29.8+k3s1   env=production,projectsveltos.io/k8s-version=v1.29.8,sveltos-agent=present
civo        cluster2   true    v1.30.5+k3s1   env=production,projectsveltos.io/k8s-version=v1.30.5,sveltos-agent=present
mgmt        mgmt       true    v1.31.2        projectsveltos.io/k8s-version=v1.31.2,sveltos-agent=present

Step 3: Deploy NATS in every production cluster

Use a Sveltos ClusterProfile to deploy NATS (version 1.2.9) to all production clusters.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: nats
spec:
  clusterSelector:
    matchLabels:
      env: production
  helmCharts:
  - chartName: nats/nats
    chartVersion: 1.2.9
    helmChartAction: Install
    releaseName: nats
    releaseNamespace: nats
    repositoryName: nats
    repositoryURL: https://nats-io.github.io/k8s/helm/charts/
    values: |-
      config:
        merge:
          authorization:
            default_permissions:
              publish: [">"]
              subscribe:  [">"]
            users:
            - user: "admin"
              password: "my-password"
  syncMode: Continuous

Step 4: Instruct Sveltos to connect to NATS server

A separate ClusterProfile then configures Sveltos to establish a connection to the deployed NATS server.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-deploy-sveltos-nats-secret
spec:
  dependsOn:
  - nats
  clusterSelector:
    matchLabels:      
      env: production
  policyRefs:
  - name: deploy-sveltos-nats-secret
    namespace: default
    kind: Secret
---
apiVersion: v1
data:
  sveltos-nats: YXBpVmVyc2lvbjogdjEKZGF0YToKICBzdmVsdG9zLW5hdHM6IGV3b2dJQ0p1WVhSeklqb0tJQ0FnZXdvZ0lDQWdJQ0pqYjI1bWFXZDFjbUYwYVc5dUlqb0tDWHNLQ1NBZ0lDQWlkWEpzSWpvZ0ltNWhkSE02THk5dVlYUnpMbTVoZEhNdWMzWmpMbU5zZFhOMFpYSXViRzlqWVd3Nk5ESXlNaUlzQ2drZ0lDQWdJbk4xWW1wbFkzUnpJam9nV3dvSkNTSjFjMlZ5TFc5d1pYSmhkR2x2YmlJS0NTQWdJQ0JkTEFvSklDQWdJQ0poZFhSb2IzSnBlbUYwYVc5dUlqb2dld29KQ1NKMWMyVnlJam9nZXdvSkNTQWdJQ0FpZFhObGNpSTZJQ0poWkcxcGJpSXNDZ2tKSUNBZ0lDSndZWE56ZDI5eVpDSTZJQ0p0ZVMxd1lYTnpkMjl5WkNJS0NRbDlDZ2tnSUNBZ2ZRb0pmUW9nSUNCOUNuMEsKa2luZDogU2VjcmV0Cm1ldGFkYXRhOgogIG5hbWU6IHN2ZWx0b3MtbmF0cwogIG5hbWVzcGFjZTogcHJvamVjdHN2ZWx0b3MKdHlwZTogT3BhcXVlCg==
kind: Secret
metadata:
  name: deploy-sveltos-nats-secret
  namespace: default
type: addons.projectsveltos.io/cluster-profile

Step 5: Automate namespace creation on user login

Sveltos is configured to listen on the user-operation NATS subject. When a user login CloudEvent is received, Sveltos automatically creates a Kubernetes namespace in the originating cluster. A user logout CloudEvent triggers the deletion of that namespace.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: user-operation
spec:
  messagingMatchCriteria:
  - subject: "user-operation"
    cloudEventSource: "auth.example.com"
---
apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: manage-namespace
spec:
  sourceClusterSelector:
    matchLabels:
      env: production
  eventSourceName: user-operation
  oneForEvent: true
  syncMode: ContinuousWithDriftDetection
  cloudEventAction: '{{if eq .CloudEvent.type "auth.example.com.logout"}}Delete{{else}}Create{{end}}'
  policyRefs:
  - name: namespace
    namespace: default
    kind: ConfigMap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: namespace
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  namespace.yaml: |
    kind: Namespace
    apiVersion: v1
    metadata:
      name: {{ .CloudEvent.subject }}

Step 6: Submit a CloudEvent for user login

Trigger the user login event by publishing the following CloudEvent to the user-operation NATS subject:

CLOUDEVENT_JSON=$(cat << EOF                                                                                     
{
  "specversion": "1.0",
  "type": "auth.example.com.login",
  "source": "auth.example.com",
  "id": "10001",
  "subject": "mgianluc",
  "datacontenttype": "application/json",
  "data": {
    "message": "User mgianluc login"
  }
}
EOF
)

KUBECONFIG=<production cluster kubeconfig> kubectl exec -it deployment/nats-box -n nats -- nats pub user-operation $CLOUDEVENT_JSON --user=admin --password=my-password

Confirm the namespace mgianluc has been created using:

sveltosctl show addons                                                    
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
|           CLUSTER           | RESOURCE TYPE |   NAMESPACE    |     NAME     | VERSION |             TIME              |                     PROFILES                     |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
| default/clusterapi-workload | helm chart    | nats           | nats         | 1.2.9   | 2025-02-04 14:06:14 +0100 CET | ClusterProfile/nats                              |
| default/clusterapi-workload | :Secret       | projectsveltos | sveltos-nats | N/A     | 2025-02-04 14:06:36 +0100 CET | ClusterProfile/deploy-deploy-sveltos-nats-secret |
| default/clusterapi-workload | :Namespace    |                | mgianluc     | N/A     | 2025-02-04 14:12:03 +0100 CET | ClusterProfile/sveltos-gbv99bcdsk1aa04jkdzv      |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+

Step 7: Submit a CloudEvent representing user logout

Publish the user logout CloudEvent:

CLOUDEVENT_JSON=$(cat << EOF                                                                                     
{
  "specversion": "1.0",
  "type": "auth.example.com.logout",
  "source": "auth.example.com",
  "id": "10001",
  "subject": "mgianluc",
  "datacontenttype": "application/json",
  "data": {
    "message": "User mgianluc logout"
  }
}
EOF
)

KUBECONFIG=<production cluster kubeconfig> kubectl exec -it deployment/nats-box -n nats -- nats pub user-operation $CLOUDEVENT_JSON --user=admin --password=my-password

Verify namespace deletion:

sveltosctl show addons                                                                                                                        
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
|           CLUSTER           | RESOURCE TYPE |   NAMESPACE    |     NAME     | VERSION |             TIME              |                     PROFILES                     |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+
| default/clusterapi-workload | helm chart    | nats           | nats         | 1.2.9   | 2025-02-04 14:06:14 +0100 CET | ClusterProfile/nats                              |
| default/clusterapi-workload | :Secret       | projectsveltos | sveltos-nats | N/A     | 2025-02-04 14:06:36 +0100 CET | ClusterProfile/deploy-deploy-sveltos-nats-secret |
+-----------------------------+---------------+----------------+--------------+---------+-------------------------------+--------------------------------------------------+

Conclusion

This article demonstrates how Sveltos can use NATS to react to external events and automate Kubernetes resource provisioning (in this case, namespaces) across a fleet of managed clusters. This provides a powerful, event-driven approach to multi-cluster Kubernetes management.

Contact Information

Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.

Automate vCluster Management in EKS with Sveltos and Helm

Gianluca — Wed, 15 Jan 2025 11:13:36 +0000

This video demonstrates how to fully automate the deployment and management of remote vClusters running in EKS using the Sveltos event framework and Helm.

In this video, each developer gets an isolated Kubernetes environment that can be managed independently. The process involves using a load balancer service as an event source, which triggers the creation of vClusters for each developer automatically.

How to Create Event-Driven Resources in Kubernetes

Gianluca — Thu, 26 Dec 2024 15:42:42 +0000

In this video, Colin introduces the Sveltos Events Framework, which allows cluster admins to create event listeners and handlers for K8s resource creation. When one resource is created, you can automatically respond by creating another resources or helm charts!

If you haven't checked first video, here is the link.

If you want to know more about ProjectSveltos check out the GitHub repo

Kubernetes Fleet Resource Orchestration

Gianluca — Wed, 18 Dec 2024 12:58:04 +0000

In this video Colin Lacy covers the core features of cluster configuration and add-on management.

If you want to know more about ProjectSveltos check out the GitHub repo

Simplifying Secret Distribution Across Kubernetes Clusters

Gianluca — Mon, 25 Nov 2024 07:37:59 +0000

A secret, in the context of software, is any piece of sensitive information that you want to keep confidential. This could include API keys, passwords, certificates, or SSH keys.

Now, imagine managing a fleet of Kubernetes clusters, each requiring access to the same secret. The traditional approach often involves manually creating and distributing the secret to each cluster, a time-consuming and error-prone process.

To streamline this process and enhance security, you need a solution that allows you to:

Centralize Secret Storage: Store the secret in a single, secure location.
Automate Secret Distribution: Automatically deploy the secret to all target clusters.

In the following sections, we’ll explore how Sveltos can help you achieve these goals.

What is Sveltos

In a management cluster, each individual Kubernetes cluster is represented by a dedicated resource. Labels can be attached to those resources.

Sveltos configuration utilises a concept called a cluster selector. This selector essentially acts like a filter based on Kubernetes labels. By defining specific labels or combinations of labels, you can create a subset of clusters that share those characteristics.

Lab Setup

A Kind cluster is used as management cluster. Then two extra Civo clusters and a GKE cluster all with label env=prod.

+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    civo/cluster1       | v1.29.8+k3s1| Civo 3 Node - Medium Standard       |
|    civo/cluster2       | v1.30.5+k3s1| Civo 3 Node - Medium Standard       |
+------------------------+-------------+-------------------------------------+

Step 1: Install Sveltos on Managament Cluster

For this tutorial, we will install Sveltos in the management cluster. Sveltos installation details can be found here.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.42.0/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.42.0/manifest/default-classifier.yaml

Step 2: Register Clusters with Sveltos

Using Civo UI, download the Kubeconfigs, then:

kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=production
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=production

Verify your Civo clusters were successfully registered:

kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME       READY   VERSION        LABELS
civo        cluster1   true    v1.29.8+k3s1   env=production,projectsveltos.io/k8s-version=v1.29.8,sveltos-agent=present
civo        cluster2   true    v1.30.5+k3s1   env=production,projectsveltos.io/k8s-version=v1.30.5,sveltos-agent=present
mgmt        mgmt       true    v1.31.2        projectsveltos.io/k8s-version=v1.31.2,sveltos-agent=present

Step 3: Deploy Sveltos configuration

First create a Secret of type kubernetes.io/dockerconfigjson to authenticate with a container registry to pull a private image. This will create a Secret regcred in the default namespace on the management cluster.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/refs/heads/main/propagate-secret/secret.yaml

We can now configure Sveltos to automatically copy the regcred Secret to namespaces that require it. This configuration will instruct Sveltos to watch for any new namespaces in your production clusters that have the label imagepullsecret: required.

When Sveltos detects a new namespace with this label, it will automatically copy the regcred Secret from the management cluster to the newly created namespace.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/refs/heads/main/propagate-secret/config.yaml

Step 4: Create namespaces within Civo clusters

Create the coke namespace in Civo cluster 1:

KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl create namespace coke
KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl label namespace coke imagepullsecret=required

To confirm that the regcred secret has been successfully propagated to the coke namespace:

sveltosctl show addons  
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE |  NAME   | VERSION |             TIME              |                  PROFILES                   |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
| civo/cluster1 | :Secret       | coke      | regcred | N/A     | 2024-11-20 14:43:05 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+

You can follow the same steps to create additional namespaces in Civo cluster 1 (e.g., pepsi) and Civo cluster 2 (e.g., bar), ensuring they inherit the necessary image pull secret.

KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl create namespace pepsi
KUBECONFIG=<CIVO cluster1 kubeconfig> kubectl label namespace pepsiimagepullsecret=required

KUBECONFIG=<CIVO cluster2 kubeconfig> kubectl create namespace bar
KUBECONFIG=<CIVO cluster2 kubeconfig> kubectl label namespace bar imagepullsecret=required

Verify Sveltos has propagated the Secret to all namespaces:

sveltosctl show addons                                                                           
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE |  NAME   | VERSION |             TIME              |                  PROFILES                   |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
| civo/cluster1 | :Secret       | coke      | regcred | N/A     | 2024-11-20 14:44:41 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster1 | :Secret       | pepsi     | regcred | N/A     | 2024-11-20 14:44:42 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster2 | :Secret       | bar       | regcred | N/A     | 2024-11-20 14:45:03 +0100 CET | ClusterProfile/sveltos-3ef2he7v7fklm1s03coa |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+

Step 5: Update Secret

While pointing to the management cluster, update the regcred Secret:

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/refs/heads/main/propagate-secret/update-secret.yaml

Verify Secret has been updated to all production clusters:

sveltosctl show addons                                                                                   
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE |  NAME   | VERSION |             TIME              |                  PROFILES                   |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+
| civo/cluster1 | :Secret       | coke      | regcred | N/A     | 2024-11-20 14:54:14 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster1 | :Secret       | pepsi     | regcred | N/A     | 2024-11-20 14:54:15 +0100 CET | ClusterProfile/sveltos-t4q7bhsrkhul3fkh6v0k |
| civo/cluster2 | :Secret       | bar       | regcred | N/A     | 2024-11-20 14:54:15 +0100 CET | ClusterProfile/sveltos-3ef2he7v7fklm1s03coa |
+---------------+---------------+-----------+---------+---------+-------------------------------+---------------------------------------------+

Step 5: External Secrets Operator

External Secrets Operator is an open source Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, and many more. The goal of External Secrets Operator is to synchronize secrets from external APIs into Kubernetes. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret. If the secret from the external API changes, the controller will reconcile the state in the cluster and update the secrets accordingly.

When managing a multitude of Kubernetes clusters, External Secrets Operator can be deployed in the management cluster. Sveltos can be used to distribute the secrets to the managed clusters.

Conclusion

This guide demonstrated how Sveltos simplifies the process of propagating secrets to all your production clusters.

Contact Information

Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo. You can also star 🌟 the project if you found it helpful.
The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.
Thank you for reading!

CIS Benchmark Compliance Across Multiple Kubernetes Clusters

Gianluca — Thu, 26 Sep 2024 11:48:17 +0000

CIS Kubernetes Benchmarks are a set of security best practices and recommendations developed by the Center for Internet Security (CIS) specifically for Kubernetes environments. These benchmarks provide a comprehensive guide to secure configuration, helping organizations reduce the risk of common vulnerabilities and attacks.

kube-bench is a tool that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark. Tests are configured with YAML files, making this tool easy to update as test specifications evolve.

This article demonstrates how Sveltos can deploy Kube-bench in each managed cluster, and finally leverage event mechanisms to post-process and collect kube-bench from each managed cluster.

Lab Setup

A Kind cluster is used as management cluster. Then two extra Civo clusters and a GKE cluster all with label env=production.

+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    civo/cluster1       | v1.28.7+k3s1| Civo 3 Node - Medium Standard       |
|    civo/cluster2       | v1.29.2+k3s1| Civo 3 Node - Medium Standard       |
|    gke/cluster         | v1.30.3-gke | GKE 3 Node                          |
+------------------------+-------------+-------------------------------------+

Step 1: Install Sveltos on Managament Cluster

For this tutorial, we will install Sveltos in the management cluster. Sveltos installation details can be found here.

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.38.3/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.38.3/manifest/default-classifier.yaml

Step 2: Register Clusters with Sveltos

Using Civo UI, download the Kubeconfigs, then:

kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=production
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=production

Using sveltosctl, pointing to the GKE cluster:

sveltosctl generate kubeconfig --create --expirationSeconds=86400 > gke_kubeconfig
sveltosctl register cluster --namespace=gke --cluster=cluster --kubeconfig=gke_kubeconfig --labels=env=production

Verify your Civo and GKE clusters were successfully registered:

kubectl get sveltoscluster -A --show-labels                                                               
NAMESPACE   NAME       READY   VERSION               LABELS
civo        cluster1   true    v1.28.7+k3s1          env=production,projectsveltos.io/k8s-version=v1.28.7,sveltos-agent=present
civo        cluster2   true    v1.29.2+k3s1          env=production,projectsveltos.io/k8s-version=v1.29.2,sveltos-agent=present
gke         cluster    true    v1.30.3-gke.1969001   env=production,projectsveltos.io/k8s-version=v1.30.3,sveltos-agent=present
mgmt        mgmt       true    v1.31.0               projectsveltos.io/k8s-version=v1.31.0,sveltos-agent=present

Step 3: Deploy kube-bench

Sveltos is used to deploy kube-bench in all production clusters:

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/cis-scan/kube-bench.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/cis-scan/clusterprofile-deploy-kube-bench.yaml

Above creates a ConfigMap in the management cluster containing kube-bench in its data section. And a ClusterProfile resource that targets all production clusters. The ClusterProfile deploys kube-bench.

Verify all resources are deployed:

sveltosctl show addons
+---------------+---------------+------------+------------+---------+--------------------------------+----------------------------------+
|    CLUSTER    | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |              TIME              |             PROFILES             |
+---------------+---------------+------------+------------+---------+--------------------------------+----------------------------------+
| civo/cluster1 | batch:Job     | kube-bench | kube-bench | N/A     | 2024-09-24 14:33:27 +0200 CEST | ClusterProfile/deploy-kube-bench |
| civo/cluster2 | batch:Job     | kube-bench | kube-bench | N/A     | 2024-09-24 14:33:49 +0200 CEST | ClusterProfile/deploy-kube-bench |
| gke/cluster   | batch:Job     | kube-bench | kube-bench | N/A     | 2024-09-24 14:33:37 +0200 CEST | ClusterProfile/deploy-kube-bench |
+---------------+---------------+------------+------------+---------+--------------------------------+-----------

Step 4: Use Event Frameworks to Collect Kube-bench Results

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/cis-scan/clusterprofile-deploy-kube-bench.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/cis-scan/events.yaml

Kube-bench results are initially stored in the kube-bench logs. To facilitate analysis and reporting, this process deploys a Job in each managed cluster. This Job post-processes the kube-bench logs, extracting and storing all failure information in a ConfigMap. These ConfigMaps are then automatically collected by Sveltos and sent to the management cluster for centralized review and action.

Step 5: Use sveltosctl to display all failures

sveltosctl provides a comprehensive overview of issues identified by kube-bench across our managed production clusters.

sveltosctl show resources --kind=configmap 
+---------------+---------------------+------------+---------------------+--------------------------------+
|    CLUSTER    |         GVK         | NAMESPACE  |        NAME         |            MESSAGE             |
+---------------+---------------------+------------+---------------------+--------------------------------+
| civo/cluster1 | /v1, Kind=ConfigMap | kube-bench | kube-bench-failures | [FAIL] 4.1.1 Ensure that       |
|               |                     |            |                     | the kubelet service file       |
|               |                     |            |                     | permissions are set to 600 or  |
|               |                     |            |                     | more restrictive (Automated)   |
|               |                     |            |                     | [FAIL] 4.1.5 Ensure that the   |
|               |                     |            |                     | --kubeconfig kubelet.conf      |
|               |                     |            |                     | file permissions are set       |
|               |                     |            |                     | to 600 or more restrictive     |
|               |                     |            |                     | (Automated) [FAIL] 4.1.6       |
|               |                     |            |                     | Ensure that the --kubeconfig   |
|               |                     |            |                     | kubelet.conf file ownership is |
|               |                     |            |                     | set to root:root (Automated)   |
|               |                     |            |                     | [FAIL] 4.1.9 If the kubelet    |
|               |                     |            |                     | config.yaml configuration      |
|               |                     |            |                     | file is being used validate    |
|               |                     |            |                     | permissions set to 600 or      |
|               |                     |            |                     | more restrictive (Automated)   |
|               |                     |            |                     | [FAIL] 4.1.10 If the kubelet   |
|               |                     |            |                     | config.yaml configuration      |
|               |                     |            |                     | file is being used validate    |
|               |                     |            |                     | file ownership is set          |
|               |                     |            |                     | to root:root (Automated)       |
|               |                     |            |                     | [FAIL] 4.2.1 Ensure that the   |
|               |                     |            |                     | --anonymous-auth argument      |
|               |                     |            |                     | is set to false (Automated)    |
|               |                     |            |                     | [FAIL] 4.2.2 Ensure that       |
|               |                     |            |                     | the --authorization-mode       |
|               |                     |            |                     | argument is not set to         |
|               |                     |            |                     | AlwaysAllow (Automated)        |
|               |                     |            |                     | [FAIL] 4.2.3 Ensure that the   |
|               |                     |            |                     | --client-ca-file argument is   |
|               |                     |            |                     | set as appropriate (Automated) |
|               |                     |            |                     | [FAIL] 4.2.6 Ensure that the   |
|               |                     |            |                     | --make-iptables-util-chains    |
|               |                     |            |                     | argument is set to             |
|               |                     |            |                     | true (Automated) [FAIL]        |
|               |                     |            |                     | 4.2.10 Ensure that the         |
|               |                     |            |                     | --rotate-certificates          |
|               |                     |            |                     | argument is not set to false   |
|               |                     |            |                     | (Automated) [FAIL] 4.3.1       |
|               |                     |            |                     | Ensure that the kube-proxy     |
|               |                     |            |                     | metrics service is bound       |
|               |                     |            |                     | to localhost (Automated)       |
|               |                     |            |                     | [FAIL] 5.1.1 Ensure that       |
|               |                     |            |                     | the cluster-admin role is      |
|               |                     |            |                     | only used where required       |
|               |                     |            |                     | (Automated) [FAIL] 5.1.2       |
|               |                     |            |                     | Minimize access to secrets     |
|               |                     |            |                     | (Automated) [FAIL] 5.1.3       |
|               |                     |            |                     | Minimize wildcard use in Roles |
|               |                     |            |                     | and ClusterRoles (Automated)   |
|               |                     |            |                     | [FAIL] 5.1.4 Minimize access   |
|               |                     |            |                     | to create pods (Automated)     |
|               |                     |            |                     | [FAIL] 5.1.5 Ensure that       |
|               |                     |            |                     | default service accounts are   |
|               |                     |            |                     | not actively used. (Automated) |
|               |                     |            |                     | [FAIL] 5.1.6 Ensure that       |
|               |                     |            |                     | Service Account Tokens are     |
|               |                     |            |                     | only mounted where necessary   |
|               |                     |            |                     | (Automated)                    |
| civo/cluster2 |                     | kube-bench | kube-bench-failures | [FAIL] 4.1.1 Ensure that       |
|               |                     |            |                     | the kubelet service file       |
|               |                     |            |                     | permissions are set to 600 or  |
|               |                     |            |                     | more restrictive (Automated)   |
|               |                     |            |                     | [FAIL] 4.1.5 Ensure that the   |
|               |                     |            |                     | --kubeconfig kubelet.conf      |
|               |                     |            |                     | file permissions are set       |
|               |                     |            |                     | to 600 or more restrictive     |
|               |                     |            |                     | (Automated) [FAIL] 4.1.6       |
|               |                     |            |                     | Ensure that the --kubeconfig   |
|               |                     |            |                     | kubelet.conf file ownership is |
|               |                     |            |                     | set to root:root (Automated)   |
|               |                     |            |                     | [FAIL] 4.1.9 If the kubelet    |
|               |                     |            |                     | config.yaml configuration      |
|               |                     |            |                     | file is being used validate    |
|               |                     |            |                     | permissions set to 600 or      |
|               |                     |            |                     | more restrictive (Automated)   |
|               |                     |            |                     | [FAIL] 4.1.10 If the kubelet   |
|               |                     |            |                     | config.yaml configuration      |
|               |                     |            |                     | file is being used validate    |
|               |                     |            |                     | file ownership is set          |
|               |                     |            |                     | to root:root (Automated)       |
|               |                     |            |                     | [FAIL] 4.2.1 Ensure that the   |
|               |                     |            |                     | --anonymous-auth argument      |
|               |                     |            |                     | is set to false (Automated)    |
|               |                     |            |                     | [FAIL] 4.2.2 Ensure that       |
|               |                     |            |                     | the --authorization-mode       |
|               |                     |            |                     | argument is not set to         |
|               |                     |            |                     | AlwaysAllow (Automated)        |
|               |                     |            |                     | [FAIL] 4.2.3 Ensure that the   |
|               |                     |            |                     | --client-ca-file argument is   |
|               |                     |            |                     | set as appropriate (Automated) |
|               |                     |            |                     | [FAIL] 4.2.6 Ensure that the   |
|               |                     |            |                     | --make-iptables-util-chains    |
|               |                     |            |                     | argument is set to             |
|               |                     |            |                     | true (Automated) [FAIL]        |
|               |                     |            |                     | 4.2.10 Ensure that the         |
|               |                     |            |                     | --rotate-certificates          |
|               |                     |            |                     | argument is not set to false   |
|               |                     |            |                     | (Automated) [FAIL] 4.3.1       |
|               |                     |            |                     | Ensure that the kube-proxy     |
|               |                     |            |                     | metrics service is bound       |
|               |                     |            |                     | to localhost (Automated)       |
|               |                     |            |                     | [FAIL] 5.1.1 Ensure that       |
|               |                     |            |                     | the cluster-admin role is      |
|               |                     |            |                     | only used where required       |
|               |                     |            |                     | (Automated) [FAIL] 5.1.2       |
|               |                     |            |                     | Minimize access to secrets     |
|               |                     |            |                     | (Automated) [FAIL] 5.1.3       |
|               |                     |            |                     | Minimize wildcard use in Roles |
|               |                     |            |                     | and ClusterRoles (Automated)   |
|               |                     |            |                     | [FAIL] 5.1.4 Minimize access   |
|               |                     |            |                     | to create pods (Automated)     |
|               |                     |            |                     | [FAIL] 5.1.5 Ensure that       |
|               |                     |            |                     | default service accounts are   |
|               |                     |            |                     | not actively used. (Automated) |
|               |                     |            |                     | [FAIL] 5.1.6 Ensure that       |
|               |                     |            |                     | Service Account Tokens are     |
|               |                     |            |                     | only mounted where necessary   |
|               |                     |            |                     | (Automated)                    |
| gke/cluster   |                     | kube-bench | kube-bench-failures | [FAIL] 3.2.1 Ensure that the   |
|               |                     |            |                     | --anonymous-auth argument      |
|               |                     |            |                     | is set to false (Automated)    |
|               |                     |            |                     | [FAIL] 3.2.2 Ensure that       |
|               |                     |            |                     | the --authorization-mode       |
|               |                     |            |                     | argument is not set to         |
|               |                     |            |                     | AlwaysAllow (Automated)        |
|               |                     |            |                     | [FAIL] 3.2.3 Ensure that the   |
|               |                     |            |                     | --client-ca-file argument is   |
|               |                     |            |                     | set as appropriate (Automated) |
|               |                     |            |                     | [FAIL] 3.2.6 Ensure that the   |
|               |                     |            |                     | --protect-kernel-defaults      |
|               |                     |            |                     | argument is set to true        |
|               |                     |            |                     | (Manual) [FAIL] 3.2.9 Ensure   |
|               |                     |            |                     | that the --event-qps argument  |
|               |                     |            |                     | is set to 0 or a level which   |
|               |                     |            |                     | ensures appropriate event      |
|               |                     |            |                     | capture (Automated) [FAIL]     |
|               |                     |            |                     | 3.2.12 Ensure that the         |
|               |                     |            |                     | RotateKubeletServerCertificate |
|               |                     |            |                     | argument is set to true        |
|               |                     |            |                     | (Automated)                    |
+---------------+---------------------+------------+---------------------+--------------------------------+

Conclusion

This article has demonstrated how Sveltos can be leveraged to achieve centralized and automated security auditing of Kubernetes clusters using CIS Benchmarks and kube-bench.
By utilizing Sveltos and kube-bench together, organizations can gain a comprehensive overview of the security posture of their Kubernetes clusters and take necessary steps to address any identified vulnerabilities.

Contact Information

👏 Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo.

You can also star 🌟 the project if you found it helpful.

Thank you for reading!

Gateway API: Exposing managed services with Kong and Sveltos

Gianluca — Thu, 08 Aug 2024 13:20:28 +0000

Kubernetes Services act as a stable access point for our microservices, abstracting away the complexity of managing multiple instances or their locations. Building on this, the Gateway API serves as a unified entry point for multiple Kubernetes Services.

This article demonstrates how Sveltos can deploy Kong as an API Gateway in the management cluster, subsequently deploy nginx services to managed clusters, and finally leverage templating and event mechanisms to expose these services through the API Gateway in the management cluster.

Lab Setup

A Civo cluster is used as management cluster. Then two extra Civo clusters, all with label env=fv:



+------------------------+-------------+-------------------------------------+
|    Cluster Name        |   Version   |             Comments                |
+------------------------+-------------+-------------------------------------+
|    cluster1            | v1.29.2+k3s1| Civo 3 Node - Medium Standard       |
|    cluster2            | v1.28.7+k3s1| Civo 3 Node - Medium Standard       |
+------------------------+-------------+-------------------------------------+

For this demo, Sveltos needs extra permissions.



kubectl edit clusterroles addon-controller-role-extra

and add following permissions



- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  verbs:
  - "*"
- apiGroups:
  - "gateway.networking.k8s.io"
  resources:
  - httproutes
  verbs:
  - "*"

Step 1: Install Sveltos on Managament Cluster

For this demonstration, we will install Sveltos in the management cluster. Sveltos installation details can be found here.



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.35.0/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/v0.35.0/manifest/default-classifier.yaml

Step 2: Register Civo Cluster with Sveltos

Create two Kubernetes clusters using Civo UI. Download the Kubeconfigs, then using sveltosctl:



kubectl create ns civo
sveltosctl register cluster --namespace=civo --cluster=cluster1 --kubeconfig=civo-cluster1-kubeconfig --labels=env=fv
sveltosctl register cluster --namespace=civo --cluster=cluster2 --kubeconfig=civo-cluster2-kubeconfig --labels=env=fv

Verify your Civo were successfully registered:



kubectl get sveltoscluster -n civo
NAME       READY   VERSION
cluster1   true    v1.28.7+k3s1
cluster2   true    v1.29.2+k3s1

Step 3: Deploy Kong as API Gateway

Sveltos is used to deploy Kong as an API Gateway in the management cluster.

First, we label the management cluster with type: mgmt.



kubectl label sveltoscluster -n mgmt mgmt type=mgmt

Next, we’ll deploy the necessary Gateway API resources. This includes the core Custom Resource Definitions (CRDs), a GatewayClass to specify the API gateway implementation (Kong in this case), and a Gateway instance to represent the actual API gateway.



wget https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/standard-install.yaml 
kubectl create configmap apigateway-crds --from-file=standard-install.yaml

wget https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/gateway.yaml 
kubectl create configmap gateway --from-file=gateway.yaml

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/clusterprofile-apigateway.yaml

Finally we will deploy Kong Helm chart with Sveltos:



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/clusterprofile-kong.yaml

ClusterProfile configuration involves selecting the management cluster using clusterSelector and deploying the Kong Helm chart



apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: kong
spec:
  dependsOn:
  - apigateway 
  clusterSelector:
    matchLabels:
      type: mgmt
  helmCharts:
  - repositoryURL:    https://charts.konghq.com
    repositoryName:   kong
    chartName:        kong/ingress
    chartVersion:     0.13.1
    releaseName:      kong
    releaseNamespace: kong
    helmChartAction:  Install

Verify all resources are deployed:



+-----------+-----------------------------------------------+-----------+-------------------------------------------+---------+--------------------------------+---------------------------+
|  CLUSTER  |                 RESOURCE TYPE                 | NAMESPACE |                   NAME                    | VERSION |              TIME              |         PROFILES          |
+-----------+-----------------------------------------------+-----------+-------------------------------------------+---------+--------------------------------+---------------------------+
| mgmt/mgmt | helm chart                                    | kong      | kong                                      | 0.13.1  | 2024-08-08 10:15:33 +0200 CEST | ClusterProfile/kong       |
| mgmt/mgmt | apiextensions.k8s.io:CustomResourceDefinition |           | gatewayclasses.gateway.networking.k8s.io  | N/A     | 2024-08-08 10:15:08 +0200 CEST | ClusterProfile/apigateway |
| mgmt/mgmt | apiextensions.k8s.io:CustomResourceDefinition |           | gateways.gateway.networking.k8s.io        | N/A     | 2024-08-08 10:15:08 +0200 CEST | ClusterProfile/apigateway |
| mgmt/mgmt | apiextensions.k8s.io:CustomResourceDefinition |           | grpcroutes.gateway.networking.k8s.io      | N/A     | 2024-08-08 10:15:08 +0200 CEST | ClusterProfile/apigateway |
| mgmt/mgmt | apiextensions.k8s.io:CustomResourceDefinition |           | httproutes.gateway.networking.k8s.io      | N/A     | 2024-08-08 10:15:09 +0200 CEST | ClusterProfile/apigateway |
| mgmt/mgmt | apiextensions.k8s.io:CustomResourceDefinition |           | referencegrants.gateway.networking.k8s.io | N/A     | 2024-08-08 10:15:09 +0200 CEST | ClusterProfile/apigateway |
| mgmt/mgmt | gateway.networking.k8s.io:GatewayClass        |           | kong-class                                | N/A     | 2024-08-08 10:15:10 +0200 CEST | ClusterProfile/apigateway |
| mgmt/mgmt | gateway.networking.k8s.io:Gateway             | kong      | kong-gateway                              | N/A     | 2024-08-08 10:15:10 +0200 CEST | ClusterProfile/apigateway |
+-----------+-----------------------------------------------+-----------+-------------------------------------------+---------+--------------------------------+---------------------------+

Step 4: Provision Nginx on Each Managed Cluster

Sveltos deploys Nginx to each managed cluster, exposing it via a LoadBalancer Service accessible only from the management cluster.

Following creates a ConfigMap with nginx resources:



wget https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/production-nginx.yaml
kubectl create configmap production-nginx --from-file=production-nginx.yaml
kubectl annotate configmap production-nginx projectsveltos.io/template=ok

Then use a ClusterProfile to deploy those resources to all managed clusters:



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/clusterprofile-production-nginx.yaml

ClusterProfile selects managed clusters labeled env: fv and deploys resources defined in a referenced ConfigMap. This ConfigMap contains template-based Deployment and Service definitions, which are instantiated using information from the kong-gateway-proxy Service within the kong namespace of the management cluster.



apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: production-nginx
spec:
  clusterSelector:
    matchLabels:      
      env: fv
  templateResourceRefs:
  - resource:
      apiVersion: v1
      kind: Service
      name: kong-gateway-proxy
      namespace: kong
    identifier: KongGatewayProxy
  policyRefs:
  - name: production-nginx
    namespace: default
    kind: ConfigMap

Verify that nginx is successfully deployed on managed clusters:



sveltosctl show addons --namespace=civo
+---------------+-----------------+-----------+------------------+---------+--------------------------------+---------------------------------+
|    CLUSTER    |  RESOURCE TYPE  | NAMESPACE |       NAME       | VERSION |              TIME              |            PROFILES             |
+---------------+-----------------+-----------+------------------+---------+--------------------------------+---------------------------------+
| civo/cluster2 | :ConfigMap      | default   | nginx-config     | N/A     | 2024-08-08 10:17:56 +0200 CEST | ClusterProfile/production-nginx |
| civo/cluster2 | apps:Deployment | default   | nginx-deployment | N/A     | 2024-08-08 10:17:56 +0200 CEST | ClusterProfile/production-nginx |
| civo/cluster2 | :Service        | default   | nginx-service    | N/A     | 2024-08-08 10:17:56 +0200 CEST | ClusterProfile/production-nginx |
| civo/cluster1 | :ConfigMap      | default   | nginx-config     | N/A     | 2024-08-08 10:17:56 +0200 CEST | ClusterProfile/production-nginx |
| civo/cluster1 | apps:Deployment | default   | nginx-deployment | N/A     | 2024-08-08 10:17:56 +0200 CEST | ClusterProfile/production-nginx |
| civo/cluster1 | :Service        | default   | nginx-service    | N/A     | 2024-08-08 10:17:56 +0200 CEST | ClusterProfile/production-nginx |
+---------------+-----------------+-----------+------------------+---------+--------------------------------+---------------------------------+

Step 5: Configure HTTPRoute for each service

Sveltos simplifies access to the Nginx services running in your managed clusters. It achieves this by leveraging the Gateway API:

Discovery: Sveltos identifies the IP address and port of each LoadBalancer service sitting in front of Nginx instances within your managed clusters.
Service Creation: It creates a headless Service (without a cluster IP) in the management cluster. This Service represents the Nginx instance and references the discovered IP:port combination through an Endpoint resource.
HTTPRoute Generation: Finally, Sveltos creates an HTTPRoute object for each Nginx instance. This HTTPRoute defines how traffic reaches the Nginx service.

To achieve all of above just post:



kubectl apply -f https://raw.githubusercontent.com/projectsveltos/demos/main/kong-apigateway/fetch-production-nginx.yaml

Two HTTPRoutes instances were created by Sveltos on the management cluster:



kubectl get httproutes -A                
NAMESPACE   NAME                     HOSTNAMES        AGE
civo        cluster2-nginx-service   ["worlds.com"]   22s
civo        cluster1-nginx-service   ["worlds.com"]   12s

Looking at one of those HTTPRoute instances:



apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: cluster1-nginx-service
  namespace: civo
spec:
  hostnames:
  - worlds.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong-gateway
    namespace: kong
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: cluster1-nginx-service
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /cluster1/nginx-service

it has a rule that redirects requests to the path /cluster1/nginx-service. The rule points to a headless Service named cluster1-nginx-service on port 80. This Service acts as a bridge to the actual Nginx instance behind the LoadBalancer in your managed cluster.

Step 6: Reach the Nginx services via Gateway API

Get the IP:port of the LoadBalancer service exposing the Gateway API



kubectl get services -n kong kong-gateway-proxy 
NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
kong-gateway-proxy   LoadBalancer   10.43.126.228   212.2.242.138   80:32155/TCP,443:31584/TCP   19m

verify both Nginx services can be reached via Gateway API:



curl -H "Host: worlds.com" http://212.2.242.138:80/cluster1/nginx-service 
<html>
<h2>Hello world. We are in cluster1 cluster!</h2>
</html>



curl -H "Host: worlds.com" http://212.2.242.138:80/cluster2/nginx-service
<html>
<h2>Hello world. We are in cluster2 cluster!</h2>
</html>

Step 7: Add another managed cluster

When you register a new cluster (e.g., cluster3), Sveltos takes care of everything for you:

Automatic Nginx Deployment: Sveltos automatically deploys an Nginx instance within the new cluster (cluster3).
Gateway API Integration: It creates an HTTPRoute object to manage traffic routing for the new Nginx instance.
Seamless Service Access: Clients can now access the Nginx service in cluster3 via Gateway API on the management cluster.



sveltosctl register cluster --namespace=civo --cluster=cluster3 --kubeconfig=civo-cluster3-kubeconfig --labels=env=fv

New HTTPRoute is created by Sveltos:



kubectl get httproutes -n civo                                                                                                                                   
NAME                     HOSTNAMES        AGE
cluster2-nginx-service   ["worlds.com"]   19m
cluster1-nginx-service   ["worlds.com"]   18m
cluster3-nginx-service   ["worlds.com"]   20s

And reaching the Nginx instance on cluster3



curl -H "Host: worlds.com" http://212.2.242.138:80/cluster3/nginx-service


<html>

<h2>Hello world. We are in cluster3 cluster!</h2>

</html>

Contact Information

If you have some questions, would like to have a friendly chat or just network to not miss any topics, then don’t use the comment function at dev.to, just feel free to add me to your LinkedIn network!