DEV Community: Lukas Gentele

Ephemeral PR environment using vCluster

Lukas Gentele — Mon, 24 Feb 2025 06:12:13 +0000

In a fast-paced development environment, having an isolated and ephemeral environment to test changes for every pull request (PR) is a game-changer. In this blog, I’ll walk you through setting up ephemeral PR environments using vCluster, enabling seamless testing of your application in a Kubernetes environment. We'll also leverage GitHub Actions for automation, ensuring every labeled PR dynamically creates a vCluster, deploys the application, and cleans up upon merging or label removal.

Let’s dive into the step-by-step guide.

What is vCluster?

vCluster is a technology that allows you to create lightweight, isolated Kubernetes clusters within a host cluster. These virtual clusters offer full Kubernetes functionality while being resource-efficient, making them ideal for scenarios like PR testing environments.

Why Ephemeral PR Environments?

Ephemeral environments allow:

Testing pull request changes in an isolated environment
Quick validation without interfering with the main cluster
Automatic cleanup post-testing

By leveraging vCluster and GitHub Actions, you can automate this workflow and ensure every PR gets its own dedicated environment.

Prerequisites:

Kubernetes cluster

You need to have a Kubernetes cluster, in this case I am using a DigitalOcean Kubernetes cluster but any should work. I am creating a realistic production scenario so for that I used a cluster that can create service type: LoadBalancer.

Command:

kubectl get nodes

Output:

kubectl get nodes
NAME              STATUS   ROLES    AGE   VERSION
live-demo-e0is0   Ready    <none>   19d   v1.31.1
live-demo-e0is1   Ready    <none>   19d   v1.31.1
live-demo-e0isz   Ready    <none>   19d   v1.31.1

Deploying Ingress controller

Command

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.4/deploy/static/provider/cloud/deploy.yaml

Output

kubectl get po,svc -n ingress-nginx
NAME                                           READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-lcb85       0/1     Completed   0          19d
pod/ingress-nginx-admission-patch-xl2fk        0/1     Completed   0          19d
pod/ingress-nginx-controller-79fcc99b4-7f7ls   1/1     Running     0          19d

Getting the LoadBalancer IP for the ingress controller:

Command:

kubectl get svc -n ingress-nginx

Output:

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.109.28.126   209.38.160.229   80:31228/TCP,443:30435/TCP   19d
service/ingress-nginx-controller-admission   ClusterIP      10.109.15.162   <none>           443/TCP                      19d

Domain mapping:

For our application we need dynamic ingress for testing so what we have done here is added the loadbalancer IP of the ingress controller as the A record to the Domain.

Connect the Kubernetes cluster to the platform

We will enable vCluster Pro in order to use templates and create the clusters. For simplicity, I am using my vcluster.cloud account and then creating the access key to login. In this way I don’t have to run any agent on the current cluster. You can either run vcluster platform start or sign up on vCluster cloud and once you login, you should be able to go to access keys and create a short lived access key for the demo (remember to delete the key post demo for security reasons).

Command:

vcluster platform login https://saiyam.vcluster.cloud --access-key <your-access-key>

Output:

Create a template under vCluster templates in the vCluster cloud platform instance.

sync:
  fromHost:
    ingressClasses:
      enabled: true
  toHost:
    ingresses:
      enabled: true
external:
  platform:
    autoSleep:
      afterInactivity: 3600  # Automatically sleep after 1 hour of inactivity

Until now we have a Kubernetes cluster with ingress controller installed and the Public IP of the nginx controller pointed to our domain.

We also have logged into the platform using the access keys created using vcluster.cloud. Now let’s see the demo application that we have.

Demo Application

The scenario we are trying to achieve here involves a sample application deployed onto a Kubernetes cluster. Often, in organizations, new features or bug fixes need to be deployed and tested before being merged into the main branch. In this case, a developer raises a pull request and adds a label to test it. Based on GitHub Actions, the application is built, and then a deployment, service, and ingress Kubernetes object file are generated and pushed to a new branch. A virtual cluster is created, and the new deployment file is applied, allowing the developer to test and verify the new application deployment.

Let’s see how this looks in practice.

GitHub repo - https://github.com/saiyam1814/vcluster-demo

The application for this demo is a simple Go-based HTTP server:

package main
import (
    "fmt"
    "net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hellooo, World for blog!!")
}
func main() {
    http.HandleFunc("/", handler)
    fmt.Println("Starting server on :8080")
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        panic(err)
    }
}

Step 1: Setting Up the Deployment Template

The application is packaged as a Kubernetes deployment and exposed via a service and ingress. The deployment uses Jinja2 templating to inject dynamic values like the image tag and ingress host.

tmpl/deploy.j2:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: {{ image_deploy_tag }}
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: hello-world
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
spec:
  ingressClassName: nginx
  rules:
  - host: {{ ingress_tag }}
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: hello-world
            port:
              number: 80

Step 2: Automating with GitHub Actions

GitHub Actions handles the workflow from building the application to deploying it on a vCluster.

PR Workflow

File: .github/workflows/build-and-deploy.yml This workflow:

Builds the application with the latest changes made by the developer using ko
Pushes the container image to docker hub account(credentials for which should be set in the Actions secret as described previously)
Creates a deployment manifest using Jinja2 - The action will replace the ingress host and the deployment image variables mentioned in the jinja template and then push to a new feature branch.
Creates a vCluster.
Deploys the application to the vCluster.
Exposes it via ingress for testing.

name: Build and Deploy with vCluster

on:
pull_request:
types: [labeled]

jobs:
build-and-deploy:
if: ${{ github.event.label.name == 'test' }}
runs-on: ubuntu-latest

steps:
# Step 1: Checkout PR Code
- name: Checkout PR Code
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}

# Step 2: Set up Go
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.22.5'

# Step 3: Set up ko
- name: Set up ko
uses: ko-build/setup-ko@v0.6
with:
version: v0.14.1

# Step 4: Log in to Docker Hub
- name: Log in to Docker Hub
env:
KO_DOCKER_REPO: docker.io/saiyam911
run: |
echo "${{ secrets.DOCKER_PASSWORD }}" | ko login docker.io --username ${{ secrets.DOCKER_USERNAME }} --password-stdin

# Step 5: Build and Push Image
- name: Build and Push Image
env:
KO_DOCKER_REPO: docker.io/saiyam911/vcluster-demo
run: |
cd app
export IMAGE_TAG=sha-$(git rev-parse --short HEAD)
echo "image_deploy_tag=docker.io/saiyam911/vcluster-demo:$IMAGE_TAG" >> $GITHUB_ENV
ko build --bare -t $IMAGE_TAG

# Step 6: Generate Deployment Manifest
- name: Generate Deployment Manifest
uses: cuchi/jinja2-action@v1.1.0
with:
template: tmpl/deploy.j2
output_file: deploy/deployment.yaml
strict: true
variables: |
image_deploy_tag=${{ env.image_deploy_tag }}
ingress_tag=pr${{ github.event.pull_request.number }}.vcluster.tech

# Step 7: Install vCluster CLI
- name: Install vCluster CLI
uses: loft-sh/setup-vcluster@main

# Step 8: Login to vCluster Platform
- name: Login to vCluster Platform instance
env:
LOFT_URL: ${{ secrets.VCLUSTER_PLATFORM_URL }}
ACCESS_KEY: ${{ secrets.VCLUSTER_ACCESS_KEY }}
run: |
vcluster platform login $LOFT_URL --access-key $ACCESS_KEY

# Step 9: Create vCluster for the PR
- name: Create A vCluster
env:
NAME: pr-${{ github.event.pull_request.number }}
run: |
vcluster platform create vcluster $NAME --project default --template my-template --link "Preview=http://pr${{ github.event.pull_request.number }}.vcluster.tech"

# Step 10: Deploy to vCluster
- name: Deploy Application to vCluster
run: |
kubectl apply -Rf deploy/

# Step 11: Test Application with curl
- name: Test Application
run: |
sleep 10
curl --retry 5 --retry-delay 10 http://pr${{ github.event.pull_request.number }}.vcluster.tech

Step 3: Cleanup Workflow

Once the PR is merged or the label is removed, the ephemeral vCluster is deleted.

File: .github/workflows/cleanup.yml

name: Clean Up vCluster

on:
  pull_request:
    types: [closed, unlabeled]

jobs:
  cleanup:
    if: (github.event.action == 'closed' && github.event.pull_request.merged == true) || github.event.label.name == 'test'
    runs-on: ubuntu-latest

    steps:
      # Step 1: Install vCluster CLI
      - name: Install vCluster CLI
        uses: loft-sh/setup-vcluster@main


      # Step 2: Login to vCluster Platform
      - name: Login to vCluster Platform instance
        env:
          LOFT_URL: ${{ secrets.VCLUSTER_PLATFORM_URL }}
          ACCESS_KEY: ${{ secrets.VCLUSTER_ACCESS_KEY }}
        run: |
          vcluster platform login $LOFT_URL --access-key $ACCESS_KEY


      # Step 3: Delete vCluster
      - name: Delete vCluster
        env:
          NAME: pr-${{ github.event.pull_request.number }}
        run: |
          vcluster platform delete vcluster $NAME --project default

How It Works

A developer creates a PR to do the feature changes.

‎With a small change the developer has raised a PR and now needs to add a test label.

As soon as the label is added the GitHub actions kicks off

In the vCluster platform cloud instance you will be able to see the cluster getting created and the application will be deployed.

The Action is completed and pr14.vcluster.tech is created as part of ingress.

The application is accessible at http://pr.vcluster.tech.

As you can see the latest changes made by the developer are deployed.

Cleanup:

Upon PR merge or label removal, the ephemeral vCluster is automatically deleted.

After merging, the cleanup action is triggered, which will clear the virtual cluster.

Conclusion

Ephemeral PR environments using vCluster simplify testing, reduce resource usage, and provide a seamless developer experience. By combining vCluster with GitHub Actions, you can achieve an automated and efficient workflow for testing PRs.

Check out the demo repository and give it a try! 🚀

Let me know your thoughts or if you face any challenges while implementing this.

We will be doing this as part of our workshop happening on 20th March 2025.

Seamless Kubernetes Multi-Tenancy with vCluster and a Shared Platform Stack

Lukas Gentele — Fri, 21 Feb 2025 05:49:14 +0000

Multi-tenancy in Kubernetes is not a new cooncept; it simply refers to creating isolated spaces for different users, teams, or projects. Many organizations begin by using namespaces to isolate workloads, teams, or projects. However, they soon encounter limitations, such as challenges with custom resource deployments, security, and per-tenant configurations.

In short, multi-tenancy is essential for cost savings, enabling multiple tenants to share a single host cluster, preventing cluster sprawl, and reducing the maintenance efforts associated with managing multiple Kubernetes clusters.

With this in mind, the easiest and most effective way to implement multi-tenancy is by creating virtual Kubernetes clusters using vCluster. Instead of taking the "one cluster per tenant" approach, we recommend the "control plane per tenant" model, where each virtual cluster has its own isolated control plane.

One of the key things we want to highlight in this post is the concept of a shared platform stack. Let’s first understand what that means.

A major challenge in Kubernetes multi-tenancy, or Kubernetes in general, is the duplication of applications installed across multiple Kubernetes clusters.

Let’s break this down with an example:

Imagine three teams: A, B, and C, each needing their own Kubernetes cluster. As administrators, we create three separate Kubernetes clusters. By default, a newly created cluster only runs the essential components needed for Kubernetes itself, such as the control plane components, the cloud controller manager etc.

Now, if all three teams need to deploy applications with HTTPS support, the typical approach is to install an Ingress Controller and cert-manager. Each team then creates Deployments, Services, Ingress, and Certificate objects. However, since these components need to be installed on every cluster separately, this results in duplicate resources.

This duplication problem also exists in multi-tenancy. One of the biggest challenges in Kubernetes multi-tenancy is the shared platform stack. Ideally, we should be able to reuse resources from the host cluster instead of installing cert-manager and an Ingress Controller in every new cluster.

The easiest way to solve this problem is by using virtual clusters (vClusters). With vCluster, you can define in the cluster configuration file which resources should be synced from the host cluster, allowing multiple tenants to share platform resources. This optimizes resource utilization and eliminates unnecessary duplication.

This concept of a shared platform stack in a multi-tenant Kubernetes environment using virtual clusters helps organizations efficiently manage resources.

Now, let’s see this in action with an end-to-end example where we install cert-manager and an Nginx Ingress Controller on the host cluster and then create a vCluster that reuses these host cluster resources.

We will have a Kubernetes cluster with some tools installed on the host cluster, as mentioned in the image above, and then use those inside virtual clusters.

1. Prerequisites

Before we dive in, ensure you have the following:

A Kubernetes cluster with admin access
The latest version of the vCluster CLI(v0.22+) installed
cert-manager installed in the host cluster
Nginx ingress controlled installed in the host cluster
Basic understanding of Kubernetes resources like Ingress and Services

Tools to Install:

Install vCluster CLI on a Linux system follow the below command. If on a different system, refer to the docs.

Command:

curl -LO https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64
chmod +x vcluster-linux-amd64
sudo mv vcluster-linux-amd64 /usr/local/bin/vcluster

Install cert-manager on the host cluster:

Command:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

Install nginx ingress controller on the host cluster:

Command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.4/deploy/static/provider/cloud/deploy.yaml

2. Setting Up vCluster

We will configure a vCluster with cert-manager integration enabled.

vCluster Configuration

Create a vcluster.yaml file:

integrations:
  certManager:
    enabled: true
sync:
  ingresses:
    enabled: true

Enable vCluster Pro in order to use this feature: For simplicity, I am using my vcluster.cloud account and then creating the access key to login and enable pro features. In this way I don’t have to run any agent on the current cluster. You can either run vcluster platform start or sign up on vCluster cloud and once you login, you should be able to go to access keys and create a short lived access key for the demo (Remember to delete the key post demo for security reasons)

Command:

vcluster platformlogin https://saiyam.vcluster.cloud --access-key <your-access-key>

Output:

Create the vCluster

Run the following command to create the vCluster:

Command:

vcluster create democert -f vcluster.yaml

Output:

Once the vCluster is created, verify it is running:

vcluster list
  
      NAME   |     NAMESPACE     | STATUS  | VERSION | CONNECTED |  AGE    
  -----------+-------------------+---------+---------+-----------+---------
    democert | vcluster-democert | Running | 0.22.1  | True      | 3h3m1s

Export the vCluster kubeconfig:

You need to make sure for the next steps to be done, you have switched the context to the virtual cluster.

kubectl config current-context
vcluster_democert_vcluster-democert_do-nyc1-demo

3. Configuring cert-manager Integration

Create an Issuer

Create an Issuer in the virtual cluster that references cert-manager in the host cluster. With the cert-manager integration, the namespaced Issuers and Certificates are synced from the virtual cluster to the host cluster.

Create a file issuer.yaml with below configuration:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt
  namespace: default
spec:
  acme:
    email: saiyam-test@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx

Apply the Issuer inside the virtual cluster:

Command:

kubectl apply -f issuer.yaml

Output:

kubectl get issuer
NAME                  READY   AGE
letsencrypt-staging   True    3h34m

4. Deploying an Application with Ingress

Deploy a Sample NGINX Application

Create a file app.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Apply the file: Apply this on the virtual cluster

kubectl apply -f app.yaml

Output:

kubectl get pod,svc      
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7769f8f85b-pmt2n   1/1     Running   0          3h34m
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.245.238.188   <none>        443/TCP   3h35m
service/nginx        ClusterIP   10.245.212.192   <none>        80/TCP    3h34m

5. Configure Ingress and TLS

Create an Ingress

Create a file ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - cert.<YOUR-EXTERNAL-IP>.nip.io
    secretName: example-cert-tls
  rules:
  - host: cert.<YOUR-EXTERNAL-IP>.nip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

In above yaml file, the IP is the external IP of the nginx ingress controller manager running inside the host cluster.

Apply the file: Apply this inside the virtual cluster.

Command:

kubectl apply -f ingress.yaml

Output:

kubectl get ing
NAME              CLASS   HOSTS                       ADDRESS         PORTS     AGE
example-ingress   nginx   cert.24.199.67.197.nip.io   24.199.67.197   80, 443   3h36m

6. Request a Certificate

Create a Certificate Resource

Create a file certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-cert
  namespace: default
spec:
  dnsNames:
  - cert.<YOUR-EXTERNAL-IP>.nip.io
  issuerRef:
    name: letsencrypt
    kind: Issuer
  secretName: example-cert-tls

Apply the file: Apply this inside the virtual cluster.

Command:

kubectl apply -f certificate.yaml

Output:

kubectl get certificate
NAME           READY   SECRET             AGE
example-cert   True    example-cert-tls   3h36m

7. Testing the Setup

Verify that the https curl command is working as expected

Command:

curl https://cert.24.199.67.197.nip.io

Output:

curl https://cert.24.199.67.197.nip.io
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Conclusion

vCluster allows you to reuse the shared platform stack, enabling resources installed in your host Kubernetes cluster to be shared with virtual clusters. The virtual clusters can then leverage these host cluster resources efficiently. We will be conducting a hands-on workshop on March 6th, where we will demonstrate this in action, and you'll have the opportunity to try it out alongside us.

Join the vCluster Slack to stay updated!

How to Assign vCluster to Specific Nodes Using Node Selectors

Lukas Gentele — Thu, 16 Jan 2025 12:45:09 +0000

When deploying a vCluster, you might need to ensure it runs on specific nodes, such as GPU-enabled nodes or production-specific nodes. This can be achieved using node selectors, which limit the scheduling of the vCluster control plane pods to nodes with specific labels.

Example Configuration

To schedule your vCluster control plane on nodes labeled environment=GPU, use the following configuration in your Helm chart:

controlPlane:
  statefulSet:
    scheduling:
      nodeSelector:
        environment: GPU

This ensures that the vCluster control plane only runs on nodes labeled with environment=GPU.

Why Use Node Selectors?

Resource Optimization: Assign vCluster workloads to nodes with specific resources (e.g., GPUs).
Isolation: Keep vCluster workloads separate from other applications.
Environment Control: Deploy to specific environments, such as production or staging.

Let’s see this in Action

Step 1: Open Killercoda playground

You can go to the Killercoda Kubernetes playground

Step 2: Install the Vcluster CLI

Command:

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

Step 3: Create the demo.yaml configuration file

Command:

cat <<EOF > demo.yaml
controlPlane:
  statefulSet:
    scheduling:
      nodeSelector:
        environment: GPU
EOF

Step 4: Label the controlplane node for Killercoda

Let’s Label the node

Command:

kubectl label node controlplane environment=GPU

Output:

Step 5: Create vCluster

Command:

vcluster create demo -f demo.yaml

Output:

Let’s Verify

Command:

kubectl config use-context kubernetes-admin@kubernetes
kubectl get pods -n vcluster-demo -owide

Output:

As you can see above, the stateful set for the vCluster created landed on the node names controlplane and this was the node where we set up the label. This is how you can assign vCluster to a specific node and if you want to do that for the pods within the vCluster too, you can follow the documentation here.

Chaos Engineering with Chaos Mesh and vCluster: Testing Close to Production

Lukas Gentele — Fri, 10 Jan 2025 15:30:06 +0000

Pioneered at Netflix over a decade ago, chaos engineering is a term used to describe the organized and intentional introduction of failures into systems in pre-production or production to understand their outcomes better and plan effectively for these failures.

In a previous articles we have highlighted tools which can help you perform chaos engineering. In this article, we will go one step further and take a look at how vCluster can help you test as close to production by using Litmus.

The Importance of Testing Close to Production

Before diving deeper into the technicalities, it's essential to understand why testing close to production is critical. Traditional pre-production environments often require setting up a separate staging area, deploying the application, and running various tests such as load testing or failover simulations.

In Kubernetes environments, this typically means spinning up a separate cluster, often with limited resources to cut costs. While this setup is practical, it rarely mirrors the true production environment. As Netflix put it, "Learn with real scale, not toy models."

This is where vCluster shines. Instead of spinning up a separate environment, vCluster lets you leverage underlying host resources. You can create virtual Kubernetes clusters within a single physical cluster. This enables you to test applications at a near-production scale without the overhead of managing a full cluster.

For operations teams, this means monitoring whether Horizontal Pod Autoscaler (HPA) rules kick in as expected or validating the effectiveness of network policies. For developers, it provides insights into application behavior under real-world scenarios.

What is Chaos Mesh?

Chaos Mesh is an end-to-end chaos engineering platform for cloud-native infrastructure and applications. It is a CNCF incubating project and aims to help both developers and SREs gain meaningful insights from their applications from their applications.

Prerequisites

This tutorial assumes some familiarity with Kubernetes; additionally, you will need the following installed locally to follow along.

Create a virtual cluster

Begin by creating a virtual cluster. For this demonstration, let's consider that the infrastructure team is starting to introduce chaos mesh into your pre-prod cluster. Later in this tutorial, you will use Horizontal Pod Autoscaler (HPA). Because of this, you need to explicitly allow vCluster to sync metrics from the host cluster into your virtual cluster.

Create a custom vCluster config:

cat <<EOF > values.yaml
integrations:
  metricsServer:
    enabled: true
    nodes: true
    pods: true
EOF

Create a virtual cluster:

vcluster create pre-prod -n pre-prod -f values.yaml

Output is similar to:

11:48:27 warn There is a newer version of vcluster: v0.21.1. Run `vcluster upgrade` to upgrade to the newest version. 11:48:27 info Creating namespace pre-prod 11:48:27 info Detected local kubernetes cluster minikube. Will deploy vcluster with a NodePort & sync real nodes 11:48:27 info Create vcluster pre-prod... 11:48:27 info execute command: helm upgrade pre-prod /var/folders/gw/gd4m32rs5k5chjbf761w3zrw0000gn/T/vcluster-0.20.1.tgz-1751275844 --create-namespace --kubeconfig /var/folders/gw/gd4m32rs5k5chjbf761w3zrw0000gn/T/689082436 --namespace pre-prod --install --repository-config='' --values /var/folders/gw/gd4m32rs5k5chjbf761w3zrw0000gn/T/754764801 --values values.yaml 11:48:28 done Successfully created virtual cluster pre-prod in namespace pre-prod 11:48:30 info Waiting for vcluster to come up... 11:48:50 done vCluster is up and running 11:48:50 info Starting proxy container... 11:48:52 done Switched active kube context to vcluster_pre-prod_pre-prod_minikube - Use `vcluster disconnect` to return to your previous kube context - Use `kubectl get namespaces` to access the vcluster

Install Chaos Mesh

With a virtual cluster created, the next step is to install chaos mesh inside your virtual cluster. You can do this by using the helm chart:

helm repo add chaos-mesh https://charts.chaos-mesh.org

helm install chaos-mesh chaos-mesh/chaos-mesh -n=chaos-mesh --set chaosDaemon.runtime=docker --set chaosDaemon.socketPath=/var/run/docker.sock --version 2.7.0 --create-namespace

Chaos Mesh requires access to the container runtime socket, which varies depending on your Kubernetes cluster. This tutorial was written using Minikube, where the runtime is Docker, and the socket path is /var/run/docker.sock.

Most Kubernetes clusters use Containerd by default. For such environments, you should use:

helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh \  
  --set chaosDaemon.runtime=containerd \  
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock \  
  --version 2.7.0

For clusters like K3s, MicroK8s, or those using CRI-O, refer to this section of the documentation.

Verify your installation

kubectl get po -n chaos-mesh

Output is similar to:

NAME                                        READY   STATUS    RESTARTS        AGE
chaos-controller-manager-867f657d95-8pcx6   1/1     Running   0               13m
chaos-controller-manager-867f657d95-d4hbs   1/1     Running   0               13m
chaos-controller-manager-867f657d95-h9fwv   1/1     Running   0               13m
chaos-daemon-2rgcj                          1/1     Running   0               13m
chaos-dashboard-7c66c9f685-h9sq6            1/1     Running   0               13m
chaos-dns-server-69dd8595bf-zsm2l           1/1     Running   0               13m

Your First Chaos Experiment

With Chaos Mesh installed, you’re almost ready to write your first experiment. Before diving in, let’s clarify some terminology. In Chaos Mesh, an experiment refers to a test designed to introduce controlled failures into your system. Experiments are divided into two categories:

Kubernetes-level experiments: These target resources like pods, deployments, or services within your Kubernetes cluster.
Physical node-level experiments: These focus on the underlying infrastructure, such as CPU stress or network disruptions on the nodes themselves.

In this tutorial, you’ll run two experiments—one targeting Kubernetes resources and another targeting a physical node. Each experiment will be created using a Kubernetes Custom Resource (CR) based on Chaos Mesh’s Custom Resource Definitions (CRDs).

PodChaos

One of the easiest experiments to write is PodChaos. This experiment randomly terminates a pod at a predefined interval, allowing you to simulate unexpected failures in your application. This can be particularly useful when testing consensus-based applications such as databases, message brokers, or distributed storage systems.

By using the PodChaos experiment in a virtual cluster, you can evaluate how well your application maintains quorum, the minimum number of nodes required to make progress. For example, you can test if your application continues to handle reads and writes effectively when a pod is unexpectedly terminated.

Begin by creating new deployment:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80 
EOF

This will create a new deployment called whoami in the default namespace. Verify all pods are running correctly using:

kubectl get pods

Output is similar to:

whoami-76c9859cfc-lsxkm   1/1     Running             0          4s

Create a PodChaos experiment:

Next, apply the following manifest to create a PodChaos experiment:

kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-failure-example
  namespace: default
spec:
  action: pod-failure
  mode: one
  duration: '120s'
  selector:
    labelSelectors:
      app: whoami
EOF

The manifest above defines a PodChaos experiment that runs for two minutes(120s) which is specified using the duration field, the mode is also set to one which will terminate one pod at a time for the duration of the experiment. Finally using selectors, specify the target, which are pods with the label app=whoami .

You can immediately verify the experiment is active by running:

kubectl get pods -w

Adding the -w flag will watch the namespace for any events, such as pod restarts. Output is similar to:

‎Slowly but surely you should see pods start to get restarted which means the test is active.

Accessing Test Results

While you could view the test was performed through the CLI, this is by no means ideal. Chaos Mesh provides a dashboard for viewing and analyzing all test results.

To access the dashboard, you first need to generate credentials. The following manifest creates a ServiceAccount, a ClusterRole with the required permissions, and a ClusterRoleBinding to link the two. Apply the manifest using the command below:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: account-cluster-manager
  namespace: default

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: role-cluster-manager
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["chaos-mesh.org"]
  resources: ["*"]
  verbs: ["get", "list", "watch", "create", "delete", "patch", "update"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bind-cluster-manager
subjects:
- kind: ServiceAccount
  name: account-cluster-manager
  namespace: default
roleRef:
  kind: ClusterRole
  name: role-cluster-manager
  apiGroup: rbac.authorization.k8s.io
EOF

Generating a Token

Once the manifest is applied, generate a token for the ServiceAccount using the following command:

kubectl create token account-cluster-manager -n default

This should output a token for the service account you just created. Save this to a secure location and run the following command to expose the dashboard:

kubectl port-forward svc/chaos-dashboard -n chaos-mesh 2333:2333

Navigating to http://localhost:2333 , should return a page like this:

Enter the credentials you just created, and you will be logged in.

StressChaos

The StressChaos CRD allows you to stress-test your application’s CPU or memory, simulating resource-intensive conditions. This is particularly useful for verifying metric-based scaling mechanisms, such as Kubernetes Horizontal Pod Autoscalers (HPA), to ensure they trigger under the correct conditions

Verify or Install Metrics Server

The metrics-server component is critical for collecting resource metrics like CPU and memory usage, which are required for HPA to function. To check if it’s installed, run the following command on your host cluster:

kubectl get deployment metrics-server -n kube-system

If the deployment is not found on your host cluster, run the following commands after disconnecting from your virtual cluster:

vcluster disconnect

Install metrics-server:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups: [""]
  resources: ["nodes/metrics", "pods", "nodes"]
  verbs: ["get", "list", "watch"]

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: registry.k8s.io/metrics-server/metrics-server:v0.7.2
        args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-insecure-tls
        - --metric-resolution=15s
EOF

Reconnect to your virtual cluster:

vcluster connect pre-prod

Configure Horizontal Pod Autoscaler

Next, apply a HPA to scale the whoami deployment based on memory usage. The following manifest scales the pods between 1 and 8 replicas when the average memory usage exceeds 300Mi:

kubectl apply -f - <<EOF
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: whoami-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: whoami
  minReplicas: 1
  maxReplicas: 8
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 300Mi
EOF

Apply the StressChaos Experiment

Finally, create a StressChaos experiment to simulate memory pressure:

kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: mem-stress
  namespace: default
spec:
  mode: all
  selector:
    namespaces:
    - default
  stressors:
    memory:
      workers: 3
      size: 600MiB
      options: []  # Use an empty list for options
  duration: '120s'
EOF

The manifest above defines a StressChaos experiment that runs for 2 minutes (120s), as specified by the duration field. This experiment is configured in mode: all, meaning it will apply the stress test to all pods within the selected namespace(s).

Using the selector field, the experiment targets all pods in the default namespace. The stressors section specifies a memory stress test with the following configuration:

Workers: 3 — Three concurrent workers will be deployed to simulate the memory load.
Size: 600MiB — Each worker will attempt to allocate 600 MiB of memory.

As soon as you apply the manifest, you should see the HPA kick in. Verify this using:

kubectl get pods

Output is similar to:

You can also check on the HPA directly:

kubectl get hpa

Output is similar to:

NAME         REFERENCE           TARGETS                   MINPODS   MAXPODS   REPLICAS   AGE
whoami-hpa   Deployment/whoami   memory: 639655936/300Mi   1         8         1          9h

Finally, you can take a look at more detailed results of the experiments by navigating to http://localhost:2333/#/experiments , be sure to expose the dashboard if you haven’t already:

kubectl port-forward svc/chaos-dashboard -n chaos-mesh 2333:2333

Select the mem-stress experiment, and you will be greeted with a more detailed view of the experiment:‎

‎This will show information such as the duration, namespace . Kind of experiment that was run ,events that occurred during the experiment and the manifest used. For the memory stress test, it displays what namespace it was run in, along with the number of workers(6 in this case) how much memory each worker used(600MiB) and the duration 1200

Conclusion

Chaos Engineering forces us to embrace failures and treat them as inevitable instead of hoping for the best outcome. With vCluster, teams can take this further and perform these tests as they would occur in production with no additional infrastructure costs.

While this blog covered only two experiments, Chaos Mesh has an extensive library of tests that you can leverage. Here are some suggestions for more chaos engineering content.

Don’t forget to check out the vCluster documentation for more details! Join our Slack here.

vcluster Exploded in 2022

Lukas Gentele — Tue, 27 Dec 2022 10:02:53 +0000

By Rich Burroughs

2022 was a very exciting year for vcluster. If you’re unfamiliar with vcluster, it’s an open source tool for creating and managing virtual Kubernetes clusters. Virtual clusters are lightweight and run in a namespace of an underlying host cluster but appear to the users as if they’re full-blown clusters. If you’d like more details, there’s an extended explanation in the docs.

Let’s take a look at the massive growth of the project and some of the new features that showed up in 2022.

Highlights

First, some numbers: vcluster now has more than 2,200 stars on GitHub, and the Docker images have been downloaded more than 26 million times from Docker Hub. While I don’t put a lot of faith in GitHub stars as a metric on its own, there’s a lot of other evidence that the project has grown a lot this year.

One of the highlights for vcluster was KubeCon + CloudNativeCon 2022 North America, which was held in Detroit. vcluster was featured in a keynote by Whitney Lee and Mauricio Salatino, which illustrated how platform teams can better serve developers, as well as a talk by Joseph Sandoval and Dan Garfield about how Adobe’s new CI/CD platform that uses vcluster and Argo CD. And Mike Tougeron from Adobe spoke more about their use of vcluster at GitOps Con North America in the buildup to KubeCon.

vcluster was featured on VMware’s TGIK stream, Whitney Lee’s Elightning show, and Bret Fisher’s DevOps and Docker stream. We also saw written content about vcluster from the community, like this intro tutorial by Pavan Kumar, a blog post from Mauricio Salatino on building dev platforms with vcluster and Crossplane, and this super cool post from Jason Andress about building honeypots with vcluster. The honeypot use case had never occurred to me. I love seeing what creative uses people come up with for vcluster.

New Features

There wouldn’t be so much excitement around vcluster if not for the work of the maintainers and contributors. It’s great to see so many new features being added, and they often come out of feedback from people in the community. Here’s a look at some of the things that shipped in 2022.

Deploying Helm charts and manifests on startup: This is one of my favorite features that’s been added to vcluster. It’s one thing to provision a bare cluster and another thing for that to become a useful environment. With this feature, you can apply Helm charts (public or private) or Kubernetes YAML manifests as the virtual cluster spins up for the first time. It’s super helpful. (Read the docs here)

Distros: Initially, vcluster was built on top of k3s, but after a while, people started asking if we could support other Kubernetes distributions. Now vcluster also supports k0s, EKS, and standard Kubernetes. This allows you to use a virtual cluster more like your production environment. (Docs)

Isolated mode: As more people started using vcluster, we received lots of questions and feedback about how much isolation the virtual clusters had. With isolated mode, vcluster now adds some additional Kubernetes security features to the virtual clusters as they’re provisioned: a Pod Security Standard, a resource quota, a limit range, and a network policy. Isolated mode is optional and can be invoked with the --isolated flag. (Docs)

Plugins: With the complexity of Kubernetes, it became clear that people would need the ability to customize and extend vcluster to fit their workflows. Enter vcluster plugins and the vcluster SDK. Plugins are written in Go and allow users to customize the behavior of vcluster’s syncer to do all kinds of things, like sharing resources between host and virtual clusters. (Docs)

Pausing and resuming virtual clusters: This handy feature can scale down workloads in your virtual clusters when they’re not being used. That’s done by setting the number of replicas to zero for StatefulSets and Deployments. Resuming the virtual cluster sets the replicas back to their original value, and then the scheduler spins the pods back up. This allows users to suspend the workloads while keeping configurations in the virtual cluster in place. (Docs)

This is just a handful of the many improvements made to vcluster in 2022.

Thank you

I want to thank the vcluster maintainers, the contributors, and everyone who used vcluster in 2022. As I mentioned, many great ideas for improving the project come from folks in the community through GitHub issues, pull requests, or even feedback in our community Slack.

We’re excited that so many people have found vcluster useful for their work. If you’ve read this far and haven’t tried vcluster yet yourself, there’s an easy quickstart that takes just a few minutes.

I’m looking forward to seeing how the project grows in 2023.

Kubernetes Multitenancy: Why Namespaces aren’t Good Enough

Lukas Gentele — Mon, 21 Nov 2022 19:32:30 +0000

By Jason Bloomberg

Multitenancy has long been a core capability of cloud computing, and indeed, for virtualization in general.
With multitenancy, everybody has their own sandbox to play in, isolated from everybody else’s sandbox, even though beneath the covers, they share common infrastructure.
Kubernetes offers its own kind of multitenancy as well, via the use of namespaces. Namespaces provide a mechanism for organizing clusters into virtual sub-clusters that serve as logically separated tenants.
Relying upon namespaces to provide all the advantages of true multitenancy, however, is a mistake. Namespaces are for cloud native teams that don’t want to step on each other’s toes – but who are all colleagues who trust each other.
True multitenancy, in contrast, isn’t for colleagues. It’s for strangers – where no one knows whether the owner of the next tenant over is up to no good. Kubernetes namespaces don’t provide this level of multitenancy.

Multitenancy: A Quick Primer

Multitenancy is most familiar as a core part of how cloud computing operates.
Everybody’s cloud account is its own separate world, complete in itself and separate from everybody else’s. You get your own login, configuration, services, and environments. Meanwhile, everybody else has the same experience, even though under the covers, the cloud provider runs each of these tenants on shared infrastructure.

There are different flavors of multitenancy, depending upon just what infrastructure they share beneath this abstraction.

IaaS tenants, aka instances or nodes, share hypervisors that abstract the underlying hardware and physical networking. Meanwhile, SaaS tenants, for example, Salesforce or ServiceNow accounts, might share database infrastructure, common services, or other application elements.

Either way, each tenant is isolated from all the others. Isolation, in fact, is one of the most important characteristics of multitenancy, because it protects one tenant from the actions of another.

To be effective, the infrastructure must enforce isolation at the network layer. Any network traffic from one tenant that is destined for another must leave the tenant via its public interfaces, traverse the network external to the tenants, and then enter the destination tenant through the same interface that any other external traffic would enter.

Even when the infrastructure provider decides to offer a shortcut for such traffic, avoiding the hairpin to improve performance, it’s important for such shortcuts to comply with the same network security policies as any other traffic.

Using Kubernetes Namespaces for Multitenancy

Namespaces have been around for years, largely as a way to keep different developers working on the same project from inadvertently declaring identical variable or method names.

Providing a scope for names is also a benefit of Kubernetes namespaces, but naming alone isn’t their entire purpose. Kubernetes namespaces are also virtual clusters within the name physical cluster.

This notion of virtual clusters sounds like individual tenants in a multitenant cluster, but in the case of Kubernetes namespaces, they have markedly different properties.

Kubernetes logically separates namespaces within a cluster but allows for them to communicate with each other within the cluster. By default, Kubernetes doesn’t offer any security for such interactions, although it does allow for role-based access control (RBAC) in order to limit users and processes to individual namespaces.

Such RBAC, however, does not provide the network isolation that is essential to true multitenancy. Furthermore, Kubernetes doesn’t implement any privilege separation, instead delegating such control to a dedicated authorization plugin.

Furthermore, Kubernetes defines cluster roles and their associated bindings, thus empowering certain individuals to have access and control over all the namespaces within the cluster. Not only do such roles open the door for insider attacks, but they also allow for misconfigurations of the cluster roles that would leave the door open between namespaces.

If cluster roles weren’t bad enough, Kubernetes also allows for privileged pods within a cluster. Depending upon how admins have configured such pods, they can access any node-level capabilities for the node hosting the cluster. For example, the privileged pod might be able to access file system, network, or Linux process capabilities.

In other words, a privileged pod can impersonate the node that hosts its cluster – regardless of what namespaces run on that cluster or how they’re configured.

‘True’ Kubernetes Multitenancy that Provides Isolation between Tenants

In order to implement secure Kubernetes multitenancy, it’s essential to use a tool like Loft Labs to implement virtual clusters within Kubernetes clusters that act just like real clusters.

With this ‘true’ multitenancy, traffic from one virtual cluster to another must go through the same access controls as any cluster-to-cluster traffic would – because fundamentally, Loft Labs handles traffic between virtual clusters just as Kubernetes would handle traffic between clusters.

One of the primary benefits of this approach to Kubernetes multitenancy is that virtual clusters support namespaces just as clusters do – not necessarily for isolation (as namespace isolation is inadequate), but for the name scoping that namespaces are most familiar for.

Loft Labs’ multitenancy provides other benefits that namespaces cannot, for example, the ability to spin down individual virtual clusters for better cost efficiency.

The Intellyx Take

The best way to think about multitenancy options for Kubernetes is this: namespaces are for friends, while Loft Labs’ multitenancy is also for strangers.

Using namespaces for multitenancy works best when securing traffic between tenants is a non-issue, say, when all the developers using the cluster are on the same team and actively collaborating.

True multitenancy of the sort Loft provides, in contrast, provides virtual clusters that separate teams can use – even if those teams aren’t collaborating, or indeed, don’t know or trust each other at all.

This zero-trust approach to sharing resources is fundamental to modern cloud native computing, even in situations where people are all working for the same company.

Not only does such isolation add security, but it also enforces GitOps-style best practices for how multiple teams can work on the same codebase in parallel.

Making Self-Service Clusters Ready for DevOps Adoption

Lukas Gentele — Mon, 21 Nov 2022 19:32:26 +0000

By Jason English

History is littered with cautionary tales of software delivery tools that were technically ahead of their time, yet were ultimately unsuccessful because of a lack of end user adoption.

In the past, the success of developer tooling vendors depended upon the rise and fall of the major competitive platforms around them. An upstart vendor could still break through to grab market share from dominant players in a space by delivering a superior user experience (or UX) and partnering with a leader, until such time as they were acquired.

A great UX generally includes an intuitive UI design based on human factors, especially important in consumer-facing applications. Human factors are still important in software development tooling, however, the UX focus is on whether the tools will readily deliver value to the organization, by empowering developers to efficiently deliver better software.

Kubernetes (or k8s) arose from the open source foundations of Linux, containerization, and a contributed project from Google. A global community of contributors turned the enterprise space inside out, by abstracting away the details of deploying and managing infrastructure as code.

Finally, development and operations teams could freely download non-proprietary tooling and orchestrate highly scalable cloud native software architecture. So what was holding early K8s adopters back from widespread use in their DevOps lifecycles?

The challenge: empowering developers

A core tenet of the DevOps movement is self-service automation. Key stakeholders should be fully empowered to collaborate freely with access to the tools and resources they need.

Instead of provisioning through the approval process of an IT administrative control board, DevOps encourages the establishment of an agile platform team (in smaller companies, this may be one platform manager). The platform team should provide developers with a self-service stack of approved on-demand tooling and environments, without requesting an exhaustive procurement process or ITIL review cycles.

At first glance, Kubernetes, with its declarative abstraction of infrastructure, seems like a perfect fit for orchestrating these environments. But much like an early sci-fi spaceship where wires are left hanging behind the lights of control panels, many specifics of integration, data movement, networking and security were intentionally left up to the open source community to build out, rather than locking in design assumptions in these key areas.

Because the creation and configuration of Kubernetes clusters comes with a unique set of difficulties, the platform team may try to reduce rework by offering a one-size-fits-all approach. Sometimes, this may not meet the needs of all developers, and may exceed the needs of other teams with excess allocation and cloud cost.

You can easily tell if an organization’s DevOps initiative is off track if it simply shifts the provisioning bottleneck from IT to a platform team that is backlogged and struggling to deploy k8s clusters for the right people at the correct specifications.

Handing over the keys

The ability to usurp the limitations of physical networks and IP addressing is the secret weapon of Kubernetes. With configuration defined as code, teams can call for namespaces and clusters that truly fit the semantics and dimensions of the application.

The inherent flexibility of k8s produces an additional set of concerns around role-based access controls (RBAC) that must be solved in order to scale without undue risk.

In today’s cloudy and distributed ecosystem, engineering organizations are composed differently than the siloed Dev and Ops teams in traditional IT organizations. Various teams may need to access certain clusters or pods within as part of their developmental or operational duties on specific projects.

Even with automated provisioning, a request would by default generate a cluster with one ‘front door’ key for an administrator, who may share this key among project team members. Permissioned individuals can step on each other’s work in the environment, inadvertently break the cluster, or even allow their credentials to get exposed to the outside world.

To accelerate delivery without risk, least-privilege rights should be built into the provisioning system by policy and leverage the company’s single-sign-on (SSO) backend for resource access across an entire domain, rather than being manually doled out by an admin.

In a self-service solution, multiple people can get their own keys with access to specific clusters and pods, or assign other team members to get them. These permissions can lean on the organization’s authorization tools of choice for access control, without requiring admins to write any custom policies to prevent inadvertent conflicts.

A self-service Kubernetes storefront

We already know the cost of not getting self-service right. Unsatisfied developers will sneak around procurement to provision their own rogue clusters, creating costly cloud sprawl and lots of lost and forgotten systems with possible vulnerabilities.

As consumers, we’re acclimated to using e-commerce websites and app stores on our personal devices. At work, we can use a credit card to buy apps, plugins and tooling from marketplaces provided by a SaaS vendor or public cloud.

The storefront model offers a good paradigm for self-service cluster provisioning. One vendor, Loft Labs, offers a Kubernetes control plane built upon the open source DevSpace tool for standing up stacks. An intuitive interface allows domain-level administrators to navigate automated deployments and track usage.

More importantly, developers can use their own filtered view of Loft Labs as a storefront for provisioning all available and approved K8s cluster images into new or existing namespaces. Or they can make the provisioning requests via a CLI and drill down into each cluster’s details with the kubectl prompt.

The system provides guardrails for developers to provision Kubernetes clusters and namespaces in the mode they prefer, without consuming excess resources or making configuration mistakes.

The Intellyx Take

Quite a few vendors are already offering comprehensive ‘Kubernetes-as-a-Service’ management platforms that gloss over much of the complexity of provisioning and access to clusters, when what is really needed is transparency and portability.

Engineers will avoid waiting on procurement boards, and they hate writing repetitive commands, whether that is launching 100 pods at a time for autoscaling or bringing them down when they are no longer required. But they do still want to directly address kubectl for a single pod, look at the logs for that pod and analyze what is going on.

The platform team’s holy grail is to provide a self-service Kubernetes storefront that works with the company’s authorization regimes to entitle the right users and allow project management, tracking and auditing, while giving experienced engineers the engineering interfaces they need.

Next up in this series, we’ll be covering the challenges of multi-tenancy and cost control!

\
*© 2022, Intellyx, LLC. Intellyx is solely responsible for the content of this article. At the time of writing, Loft Labs is an Intellyx customer. Image sources: Maps, Unsplash. Screenshot, Loft Labs.*

Interview with KubeCon Keynote Speaker Mauricio Salatino from VMware

Lukas Gentele — Thu, 27 Oct 2022 13:16:49 +0000

By Rich Burroughs

Loft Labs is excited to welcome Kubecon North America 2022 keynote speaker Mauricio “Salaboy” Salatino for an exclusive interview where we dive into the struggles facing platform engineers with the CNCF ecosystem. Mauricio and his co-speaker Whitney Lee will be presenting a demo for their keynote presentation focused on the provisioning of virtual clusters with Crossplane, vcluster, and Knative, to develop an internal development platform.

Rich: How did you learn about vcluster?

Mauricio: I had heard about vcluster in the context of multi-tenancy. While delivering training for LearnK8s and working for different companies, I’ve repeatedly seen teams struggling to answer a very simple question: One cluster or multiple clusters? I’ve seen teams starting simple with namespaces inside a single Kubernetes Cluster and then struggling to move to use multiple clusters when the isolation levels of namespaces are not enough. And that is precisely where I see vcluster as a better alternative because it provides, from the get-go, the separation into different Kubernetes API Servers. In today’s world, where every organization is building its internal development platform, I see tools like vcluster as critical components of these platforms.

Rich: In the demo for your KubeCon keynote you provisioned virtual clusters with vcluster and Crossplane. Can you explain how that works? And what’s your experience been like using those two tools together?

Mauricio: The demo presented at the KubeCon keynote session uses Crossplane, vcluster, and Knative, all working together around this concept of building internal development platforms. Crossplane is used to abstract where cloud resources are created. With Crossplane, we can declaratively create Kubernetes clusters in all major cloud providers, but creating these clusters is expensive, and it is a process that takes time. This is where using vcluster can save you time and money. Because, to my surprise, creating a vcluster is just installing a Helm chart into your existing cluster. We can use the Crossplane Helm Provider to create vcluster from inside a Crossplane Composition, and that is exactly what my demo is doing. But vcluster doesn’t stop there, because with vcluster Plugins you can share tools between the host cluster and the virtual clusters. The demo shows how you can enable your vclusters with tools like Knative Serving (for dynamic autoscaling and advanced traffic management) without installing Knative Serving in each cluster. In a scenario where you have 10 teams all working in different vclusters and using tools like Knative Serving you save on installing and running 9 Knative Serving installations.

Rich: Your demo focused on provisioning environments for developers. What do you think makes vcluster a great tool for dev environments?

Mauricio: vclusters are better than namespaces because they provide more isolation and are cheaper than fully fledged clusters. Suppose you have developers working with cluster-wide resources such as CRDs and tools that need to be installed to do their work. In that case, vclusters will give them the freedom to work against a dedicated Kubernetes API Server, where they will have total freedom to do what they need. By using vcluster you can give your development teams full access to their dedicated API servers without the need of creating, maintaining and paying for full-fledged Kubernetes Control Planes.

Rich: There are so many new tools in the Kubernetes space and more arriving all the time. The CNCF landscape continues to grow. What do you look for when you evaluate new open source, cloud native tools?

Mauricio: I evaluate their (project) community, and in the last two years, I’ve been very interested in tools that fit into the story of building internal developer platforms, as I’ve been in that space for a long time. Most of my work in the open source space is around helping developers to be more productive in building their software.

If you are evaluating Open Source / CNCF projects, check their maturity level, who (which company or companies) are sponsoring the project, and how healthy their community is. Looking at which companies are active in their community forums or slack channels is a great validation to see if other companies in your industry are trying to solve the same problems that you are facing.

Rich: There’s been a lot of focus on the role of platform engineer the last few years. What do you think are the big challenges facing platform engineers today?

Mauricio: I’ve been writing about these topics lately in my blog, you can read more about this here -> The challenges of building platforms on top of Kubernetes Part 1, Part 2, Part 3, Part 4

But the big challenge nowadays is to keep up with all that is happening in the Cloud Native space, so you need to look for tools that:

Be ready to pivot: look for tools that provide you with the right abstractions to be able to pivot if things change, Crossplane is a big player in this space, but there are other projects that are worth looking at. If you are really into platform building you should check a project that I am hoping gets donated to the CNCF called Kratix. These folks are working on providing the right abstractions to build platforms allowing platform teams to focus on deciding which projects they want to use and how those projects will work together.
Reuse instead of build: try to identify the problem that you are trying to solve into two different buckets: 1) Generic problem that every company will have 2) very specific challenge that is specific to your company. If you are looking for tools to solve problems in the first bucket, then make sure that you don’t build an in-house solution. If you are looking for tools to solve problems in the second bucket you need to focus your search on a combination of an existing tools can do the work to make sure that you don’t spend time and effort reinventing the wheel just because the tools available doesn’t match 100% your requirements.
Ecosystem integrations: When you look at a specific tool make sure that it integrates well with the other tools in the ecosystem. Don’t be tricked by the fact that they are all Kubernetes projects. Depending on how you want tools to work together your platform team might need to spend a considerable amount of time to make these tools work for your specific use case.
Tailored Developer Experiences are the best way to promote your platform: you need to spend a lot of time and effort into building amazing developer experiences for your teams to have the right tools to do their work. These developer experiences are enabled by the tools that the platform team chooses to use and by always keep improving how application development teams interact with the platform.

If you are into building platforms, I am currently writing a book titled ”Continuous Delivery for Kubernetes” where I cover tools like Tekton, Crossplane, vcluster, Keptn, Knative, ArgoCD, Helm, among others, to build platforms that are focused on delivering more software more efficiently.

Rich: Thanks for your time Mauricio.

If you want to learn more about vcluster, check our vcluster.com for links to the docs, the GitHub repo, and our community Slack. You can find Mauricio on Twitter at @salaboy, and Rich at @richburroughs.

3 Ways to Manage Kubernetes on AWS and How to Get Started

Lukas Gentele — Fri, 30 Sep 2022 19:58:49 +0000

By Talha Khalid

AWS is one of the most popular choices for container orchestration due to its reliability and efficiency. In this article, we will look at some of the most popular tools and ways to manage Kubernetes on AWS.

Containers With AWS

There are various tools that are of great help to developers. Today, one of the most important tools is a container, since applications can be implemented and packaged through them. Because they are lightweight, they provide a consistent software environment. Also, they very easily run applications that can be scaled to any location.

Containers are used to build and deploy microservices, run batch jobs, learn applications, and port previously existing apps to the cloud.

AWS offers different types of container services that help in managing Kubernetes. Let’s look at some of them.

1. Amazon Elastic Container Service (ECS)

Amazon Elastic Container Service, or ECS, is a container control service that the client fully manages. Once Amazon ECS is installed, you don't need to install or run your Kubernetes container control software, manage or scale clusters, or schedule virtual machine containers.

Amazon ECS was designed to be integrated with the entire AWS platform, which means you can count on all of its services. Furthermore, Amazon ECS is a very reliable solution that offers a native AWS API experience for Kubernetes, much like EC2 offers for virtual machines.

Benefits of Amazon ECS

There are several advantages to why using Amazon ECS to manage Kubernetes is an excellent decision. These are some of the most relevant:

Run Containers Without Provisioning Servers

Through AWS Fargate, which Amazon ECS offers, you can deploy and manage containers without the need to provision or manage servers. Also, Fargate gives you the freedom to focus on building and running applications.

Containerize Everything

You can easily create all containerized applications and migrate Linux or Windows apps from on-premises environments to the cloud. Then, you can run them as Amazon ECS containerized applications.

Security

Amazon ECS offers a high level of isolation. This allows you to create secure and reliable applications because Amazon ECS has its own Amazon VPC. Through this virtual private cloud, it launches its containers. This then allows the use of VPC network security groups and ACLs.

Performance at Scale

Amazon ECS is based on technology developed from a solid track record in the execution of services.

Usability With Other AWS Services

Amazon ECS integrates with various AWS services, such as Amazon VPC, IAM, Batch, CloudFormation, and more.

2. Elastic Kubernetes Service on AWS

Amazon's Elastic Kubernetes Service, or EKS, is a service that helps make it easy to deploy and run Kubernetes on AWS. That means you can do so without being an expert. Amazon EKS fully manages the solution and Kubernetes control plane scalability for each cluster.

It is in charge of automatically performing each cluster operation, its updates, the scale of the masters, and the persistence layer. Additionally, it detects and replaces problem masters.

Moreover, Amazon EKS also integrates with a variety of AWS services. This way, it can provide security and scalability for your applications. These include Elastic Load Balancing for load balancing, IAM for authentication, and AWS CloudTrail, which is the keeper of the record.

EKS always runs the latest version of Kubernetes, so you can use all of its plugins, consequently making it possible to migrate any standard Kubernetes application to Amazon EKS without any code modifications.

Benefits of Amazon EKS

There are several benefits of using Amazon EKS.

High Availability Fully Managed Service

Amazon EKS makes it easy to run highly available Kubernetes clusters by automatically running and managing three master nodes spread across three zones for each cluster.

Security

Among other things, Amazon EKS integrates IAM with Kubernetes, allowing you to register IAM entities with the native authentication system. You can also use PrivateLink to access Kubernetes masters from your Amazon VPC.

Kubernetes Community Tools

As Amazon EKS typically runs the latest version of Kubernetes software, all existing features, plugins, and applications are supported.

3. AWS Fargate

Fargate is a service that deploys and manages containers without the need to manage the underlying infrastructure. You don't have to provision, scale, or configure clusters for virtual machines to run containers. In other words, AWS Fargate allows you to focus only on building and running your application without worrying about your infrastructure.

Available for Amazon ECS and Amazon EKS

Amazon ECS and EKS have two modes: Fargate and EC2 launch types.

In the Fargate launch type, you only need to containerize the applications, specify the CPU and memory requirements, define the IAM access policies, and launch the application.

In the EC2 release type, you can have more granular and server-level control over the application infrastructure, which is grouped on that server. Also, with this type of launch (EC2), it is possible to use ECS and EKS to manage a cluster of servers and schedule the placement of containers.

Running Kubernetes on AWS Fargate - AWS Online Tech Talks

Both ECS and EKS monitor the CPU, memory, and all other resources of the cluster. Additionally, they find the best server for running a container. ECS and EKS handle provisioning, scaling, and patching of clusters. Also, they can decide what kind of server to use, which applications to use, and how many containers they should run in a cluster to optimize their use and decide when to add or remove servers.

The EC2 release offers more control over clustering and a greater range of customization. In turn, this allows for meeting requirements in specific applications or in compliance with government regulations.

Benefits of AWS Fargate

Here are some of the benefits of AWS Fargate.

No Need to Manage Clusters

You can focus on containers and on building and running your application.

Seamless Scalability

Scaling your applications will be much easier, as you will not need to provide resources for your applications since AWS manages everything.

Integration With Amazon ECS and EKS

Fargate integrates seamlessly with Amazon ECS. Moreover, since 2018, it has also integrated with Amazon EKS.

Useful Tools for Managing Kubernetes on AWS

There are some useful tools for managing Kubernetes on AWS that are worth discussing too.

Amazon Elastic Container Registry (ECR)

ECR is a fully managed Docker container registry. With ECR, it's easy to store, manage, and deploy images to those containers using it. Amazon ECR integrates with ECS, making it easier to develop workflows. Using Amazon ECR automatically hosts your images on a highly available and scalable architecture, giving you the freedom to deploy reliable containers for your applications. Additionally, it also integrates with AWS Identity and Access Management, which provides resource controls for each repository. Amazon ECR costs are calculated per amount of data stored and per amount of data received, so there are no predefined quotas.

Amazon CodePipeline

AWS CodePipeline is a continuous integration and delivery service that executes application and infrastructure updates quickly and reliably. Also, it can be used with Kubernetes to create a continuous deployment pipeline.

CodePipeline is in charge of compiling, testing, and deploying the code each time there is a change, as long as it complies with the processing models previously defined in the publication. Besides, it also enables fast and reliable delivery of features and updates.

Using CodePipeline, you can easily create an end-to-end solution using third-party plugins like GitHub or by integrating your plugins at any release stage. With AWS CodePipeline, you pay only for your use, with no up-front fees or commitments.

AWS CloudWatch Logs

CloudWatch Logs is the functionality of CloudWatch. It allows us to consolidate and analyze logs of the execution of your Kubernetes containers. Thus, it becomes an essential tool for recording execution data since containers are stateless and will not store information locally.

Final Words

All in all, AWS provides several different ways to manage Kubernetes containers. The choice of which service to opt for comes down to the specific requirements of your project, your budget, the dev team's experience, and other variables.

This post was written by Talha Khalid. Talha is a full-stack developer and data scientist who loves to make the cold and hard topics exciting and easy to understand.

How to Create and Manage Kubernetes Secrets: A Tutorial

Lukas Gentele — Fri, 30 Sep 2022 19:53:37 +0000

By Mercy Kibet

When developing applications, it's common to have sensitive information you would not want exposed to unauthorized personnel. Unlike other objects in your cluster, such as those used in a Pod specification, a Secret can be created and stored independently of its associated Pods. This eliminates the risk of exposing the data to the public during the workflow of editing, creating, and viewing Pods. This post will show you how to create and manage Kubernetes Secrets.

What Are Kubernetes Secrets?

Kubernetes applications are often set up with Secrets. They're a way to insulate sensitive information from the rest of your environment, so it's harder for malicious actors to find and modify it. Secrets have different methods for controlling access, like having an administrator create them and then assign access via role-based access control (RBAC). It can also be done with a policy file, which we'll cover in this tutorial.

How to Define Kubernetes Secrets

Like any other object in Kubernetes, Secrets must be defined or created with a spec with a name, Secret, and other fields. Below is a short example of the required information for the spec. All you need to do is assign values for those three fields and you'll have a working Kubernetes Secret.

apiVersion: v1

kind: Secret

metadata:

  name: my-secret

spec:

  selector: ingress.kubernetes.io/node-role.---/master, ingress.kubernetes.io/service-role.---/persistent-volume-controller, ingress.kubernetes.io/auth-provider, ingress.kubernetes.io/ingress-from, ingress.kubernetes.io/ingress-label, ingress.kubernetes....

  type: Opaque

  data: [base64 encoded value]

Let's break down this spec so you can see what's happening here. The metadata field is just a place to store your object's name, Secret, and other information. I chose:\
\

ingress.kubernetes.io/node-role.---/master\
ingress.kubernetes.io/service-role.---/persistent-volume-controller \
and ingress.kubernetes.io/auth-provider

because they're the service accounts I use across my Kubernetes environment for various components (names will vary based on your setup).

data: [base64 encoded value] tells the object to be stored as an opaque binary. Opaque is one of the options for dataType, which determines how the data is versioned and stored. Here, we're using a base64 encoded value, which will be stored in the database as a byte array.

selector is used to determine where to inject the Secret. Here, we're saying that the Secret should only exist on nodes with a role of master or nodes that are service controllers (those with instance roles). The label selector option allows you to select by multiple labels at a time, so I've chosen several from my cluster configuration. In this case, I chose ingress.kubernetes.io/ingress-from (name of the ingress), ingress.kubernetes.io/ingress-label (the label for the ingress), and ingress.kubernetes.

You can see there are a lot of options for defining Secrets in Kubernetes, but that's not necessarily a bad thing. You should be able to get started with some defaults and extend it later throughout your cluster, as long as you keep the constructs as YAML and never write to raw database fields, which would be a really unfortunate mistake.

Defining a Secret Under the Namespace

Let's assume we have a Secret to store sensitive data and want to deploy it on our cluster nodes. We won't need the application, so we'll define the Secret under our Kubernetes cluster namespace (i.e., the default). First, we'll create a namespace. We can do that with the following command:

$ kubectl create ns mysecret

Now that we have a namespace, we can proceed to define our Secret. We'll create a .yaml file with the content below:

apiVersion : v1 
kind : Secret 
metadata : 
   name : mysecretns 
data :
   mysecret : YWRtaW4= 
type : Opaque

Then we can instruct kubectl to create the Secret file:

$ kubectl apply -f mysecret.yaml

Verify that the Secret is in place using the below:

$ kubectl get secrets NAME TYPE DATA AGE mysecretns Opaque 8 1s 
$ kubectl describe secrets/mysecretns Name: mysecretns Namespace: default Labels: <none> Annotations: <none> Data ==== mysecret: 8 byte string (binary)

And now we have successfully created a Secret file containing the information we need. We can use this information to set environment variables on instances in our cluster. We could also create a deployment and access the Secret values as environment variables with a container inside the deployment.

As shown above, this is a straightforward example of deploying a Secret. Secrets can be used to store all sorts of information, from database passwords to private keys.

Managing Kubernetes Secrets

From your terminal, you can use the kubectl command and subcommands such as create, delete, describe, and apply to manage and view your Kubernetes Secrets. First, we'll look at examples of each type with their corresponding command-line interface (CLI) output. Then, we'll examine how to use them in more detail.

Viewing Kubernetes Secrets

To view Kubernetes Secrets, first use cat or id <name> to check if there are any Secrets in your cluster. Next, use describe <name> to get more information about a specific Secret.

Use create -f file.yaml to add a new Secret and replace file.yaml with the name of your file that contains your Secret information.

Use delete to remove a specific Secret.

Finally, apply -f filename.yaml is used to apply changes that you have made locally without overwriting the Secret on the cluster (for example, if you want to test your changes).

Deleting a Kubernetes Secret Using kubectl delete

To delete a Secret, first, use cat or id <name> to check if there are any Secrets in your cluster. Next, use describe <name> to get more information about a specific Secret.

You delete Kubernetes Secrets using the kubectl delete command.

The following example deletes a Kubernetes Secret named my-secret-name-1:

kubectl delete secret my-secret-name-1

Modifying a Secret's Properties

To modify the properties of an existing Kubernetes Secret you can use ``kubectl edit.

The following command shows how you could modify a Kubernetes Secret named production-secret:

kubectl edit secret production-secret

To find the original Kubernetes Secret that your new one replaced, use kubectl get <name> to see if the key pair exists in the cluster. If it does, you can use** kubectl get secrets <name>** to see the original Secrets.

Encrypting Kubernetes Secrets

You can use Kubernetes to provide sensitive data only to people who require access to it. This can reduce your exposure to security problems, typically from storing sensitive data on a public server. To do this, you must generate an encryption key pair called a kubecrypt key store and keep it private. To encrypt Secrets with a kubecrypt key store, you'll first have to create the key pair. You can learn more with kubecrypt.

To create a key pair, run the following:

` kubectl create -f kubernetes.yaml `

This should generate your new key pair. You should see something like this after running the command:

` token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9 `

Secrets Alternatives

Although Secrets allow you to store encrypted information, they only store the key and not the actual data. This limits security, as Kubernetes does not encrypt the actual data before storing it. There are two alternatives:

Sealed Secrets: This is a special type of Kubernetes Secret that encrypts and stores your sensitive information in its database.
HashiCorp Vault: This digital vault supports multi-algorithm encryption for over 20 blockchain technologies, authentication providers, and storage methods while also being able to maintain user permissions and privilege escalation policies on Secrets decryption keys using RBAC.

Conclusion

Kubernetes Secrets are a great way to manage sensitive information for your applications. They can be used to store things like database passwords, API keys, and other sensitive data. It's impossible to keep your most precious Secrets safe from prying eyes because of the sheer number of potential attack vectors available to hackers. Luckily, there are many better options for securing your Secrets (and backups), including Sealed Secrets and HashiCorp. If you're looking for an easy way to manage Kubernetes Secrets, check out Loft. Loft provides a simple and easy-to-use web interface for managing Kubernetes Secrets.

This post was written by Mercy Kibet. Mercy is a full-stack developer with a knack for learning and writing about new and intriguing tech stacks.

Kubernetes DaemonSet: What It Is and 5 Ways to Use It

Lukas Gentele — Wed, 28 Sep 2022 14:15:19 +0000

By Mercy Kibet

Kubernetes DaemonSet is a powerful tool for managing persistent workloads on your system. It ensures that each instance of an application running on your system will always be running. When starting a Kubernetes cluster, you must configure the services or applications you want running in your cluster.

In this post, we'll explain the problem it solves and five ways to use it.

What Is a DaemonSet in Kubernetes?

A DaemonSet ensures that a specified collection of pods runs on the specified nodes. DaemonSet makes sure one pod exists per node.

Kubernetes DaemonSets can be used for various applications, including key-value stores, caches, and servers that require high availability, like messaging apps. A DaemonSet will allow you to specify how many instances your app should run in the cluster and guarantee a consistent state among all pods running in the cluster.

Kubernetes DaemonSet can run Docker containers without managing the underlying infrastructure. You don't have to worry about scaling your clusters or where the container data is stored.

What Is the Difference Between DaemonSet and Deployment?

DaemonSet manages the number of pod copies to run in a node. However, a deployment manages the number of pods and where they should be on nodes. Deployment selects nodes to place replicas using labels and other functions (e.g., tolerations).

A DaemonSet doesn't need external resources like IP addresses or port numbers. However, you must provide them if you create a deployment.

Additionally, a DaemonSet doesn't need to know the number of nodes in your Kubernetes cluster. However, you must provide it if you create a deployment.

Last, a DaemonSet doesn't need runtime state information, like the number of pods currently running on each node or the number of replicas running on each pod (although these things can still be specified).

Ways to Use a DaemonSet

Running a backup job—You can have a DaemonSet running on every node in your cluster responsible for running backups of your etcd, MySQL data files, and PostgreSQL data files. If one pod fails for any reason, the other backup pods in the set will take over.
Logging—Another use case is to install an agent such as Sysdig on each node and launch a DaemonSet to manage all of these agents in a cluster-ready state.
Enforcing network policy—If you have a multi-tenant cluster and wish to lock down each tenant to its range of IPs, you can create a DaemonSet for each tenant that ensures every node is running the pods for that tenant's range of IPs, which is defined as part of the pod spec.
Docker registry—Maintaining a highly available and scalable Docker registry in Kubernetes is an example of another use case for DaemonSet. Ensure your Docker registry always has two replicas running in a cluster-ready state.
Log aggregation—If you want to aggregate your container logs and ship them off to a centralized logging tool like Logz.io, you can create a DaemonSet that launches a tail-logs pod and replicates it many times as required.

Creating a DaemonSet Is Simple With an Example of Nginx

First, we need to create a Manifest file which will contain all of the necessary configuration information for our DaemonSet.

apiVersion: extensions/v1beta1 
kind: DaemonSet 
metadata: 
  name: nginx-example 
spec: template: metadata : labels : app : nginx spec : nodeSelector : role : web containers : - name : nginx image : nginx ports : - containerPort : 80

Specifying the node selector allows you to control which nodes the pod will run on. You could spread your pods across multiple nodes for load balancing, high availability, and more. You can use the labels for pod selection and service discovery.

In this example, we created a DaemonSet with all nodes in the cluster.

From this example, we can use multiple containers in a DaemonSet. You could use CoreDNS (a DNS server) to spin up on every node in your cluster, or even an image library for processing images on every node for faster processing of your containers.

The “spec” section tells Kubernetes what the pod is supposed to do; the “node selector” is how you choose which machines the pod is running on.

Creating and Managing Kubelet DaemonSet

If you add a node to your cluster, then you can create a new DaemonSet and add it to the node. If you want to ensure the required number of pods run on a subset of nodes, you can create multiple DaemonSets. Once created, they need to be managed by an operator.

Managing Sysdig Ops Agent with Kubernetes DaemonSet

Here, these steps explain how to use Kubernetes to manage your cluster's kubelet agent (Sysdig) and how it compares with creating a deployment.

Create the DaemonSet object in Kubernetes.

curl -s create-daemonset.sh -H "Content-Type: application/json" -d '{"kind":"DaemonSet", "apiVersion":"v1", "metadata": {"name": "sysdig"}}' | kubectl
 --kubeconfig=/etc/kubernetes/admin.kubeconfig apply -f-

The following is created in your Kubernetes cluster:

$ kubectl --kubeconfig=/etc/kubernetes/admin.kubeconfig get svc/sysdig-k8s-agent

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

sysdig-k8s-agent ClusterIP 10.157.194.75 <none> 80:31185/TCP,443:31186/TCP 1m

Check the pod status to see how many pods are running in a cluster-ready state on your node.

curl -s http://10.157.194.75:31186/api/v1/namespaces/system:pod/pods | jq '.status.containerStatus'

"active" "running" "stopped-for-preview"

Use the kubectl top command to list all DaemonSets you've created, then check the status of your DaemonSet in a cluster-ready state for a pod (Sysdig pod).
Use the kubectl delete command to remove your DaemonSet after using it successfully.
To view the pod status of your DaemonSet in a cluster-ready state, use the kubectl get command.
To launch a DaemonSet, use the command below:

kubectl create -f daemonset.yaml --namespace=system/daemonset

How to Use a Kubernetes DaemonSet for Log Collection

DaemonSet is a collection of log files available to your application. You use it for collecting performance, usage, and error logs.

DaemonSet contains a few components—a daemon, a set of clients (logs and other activity watchers), and a central log handler. It all comes together through the standard protocol described below, allowing seamless integration with your application's ongoing activity.

DaemonSet collects logs, errors, and other system statistics and doesn't interfere with your application's behavior.

The handler receives logs from clients and does the work of processing and sending them to the desired destination. You can have multiple handlers in a single DaemonSet.

DaemonSet uses a flexible messaging format over TCP connection (by default, on port 9000) that allows for sending various messages to the handler(s).

Conclusion

Kubernetes DaemonSet is a great way to manage and deploy applications in a clustered environment. It's easy to use and has a wide range of features, making it an ideal choice for managing applications in a production environment.

You can use DaemonSet to run a cluster storage, log collection, and node monitoring demon on each node. Now that you know how to spin up a DaemonSet, check out Loft, a platform that gives you autonomy to fully leverage DaemonSet's capabilities.

This post was written by Mercy Kibet. Mercy is a full-stack developer with a knack for learning and writing about new and intriguing tech stacks.

Kubernetes Service Account: What It Is and How to Use It

Lukas Gentele — Mon, 19 Sep 2022 14:38:44 +0000

By Dawid Ziolkowski

Kubernetes provides a few authentication and authorization methods. It comes with a built-in account management solution, but it should be integrated with your own user management system like Active Directory or LDAP. User management is one thing, but there is also a whole additional layer of non-human access. Think about CI/CD access to the cluster, pods-to-pods communication, or pods to Kubernetes API authentication. For these use cases, Kubernetes offers so-called "service accounts," and in this post, you'll learn what they are and how to use them.

What Are Kubernetes Service Accounts?

Whenever you access your Kubernetes cluster with kubectl, you are authenticated by Kubernetes with your user account. User accounts are meant to be used by humans. But when a pod running in the cluster wants to access the Kubernetes API server, it needs to use a service account instead. Service accounts are just like user accounts but for non-humans.

Why Do Kubernetes Service Accounts Exist?

Why do Kubernetes Service Accounts exist? The simple answer is because pods are not humans, so it's good to have a distinction from user accounts. It's especially important for security reasons. Also, once you start using an external user management system with Kubernetes, it becomes even more important since all your users will probably follow typical firstname.lastname@your-company.com usernames.

But, you may wonder, why would pods inside the Kubernetes cluster need to connect to the Kubernetes API at all? Well, there are multiple use cases for it. The most common one is when you have a CI/CD pipeline agent deploying your applications to the same cluster. Many cloud-native tools also need access to your Kubernetes API to do their jobs, such as logging or monitoring applications.

Default Service Account

You now know what service accounts are and why would you need one. Let's discuss how to use them. The first thing you need to know is that you have probably already used service accounts even if you never configured any. That's because Kubernetes comes with a predefined service account called "default." And by default, every created pod has that service account assigned to it. Let's validate that. I'll create a simple nginx deployment:

$ kubectl create deployment nginx1 --image=nginx
deployment.apps/nginx1 created

Now, let's see the details of the deployed pod:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
nginx1-585f98d7bf-84rxg   1/1     Running   0          12s

$ kubectl get pod nginx1-585f98d7bf-84rxg -o yaml
apiVersion: v1
kind: Pod
metadata:
  (...)
spec:
  containers:
  - image: nginx
    (...)
  serviceAccount: default
  serviceAccountName: default

OK, What Does It Mean?

So, it turns out that your pods have the default service account assigned even when you don't ask for it. This is because every pod in the cluster needs to have one (and only one) service account assigned to it. What can your pod do with that service account? Well, pretty much nothing. That default service account doesn't have any permissions assigned to it.

We can validate that as well. Let's get into our freshly deployed nginx pod and try to connect to a Kubernetes API from there. For that, we'll need to export a few environment variables and then use the curl command to send an HTTP request to the Kubernetes API.

# Export the internal Kubernetes API server hostname
$ APISERVER=https://kubernetes.default.svc

# Export the path to ServiceAccount mount directory
$ SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read the ServiceAccount bearer token
$ TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal Kubernetes certificate authority (CA)
$ CACERT=${SERVICEACCOUNT}/ca.crt

# Make a call to the Kubernetes API with TOKEN
$ curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/default/pods
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}#

As you can see, the default service account indeed doesn't have many permissions. It's really only there to fulfil the requirement that each pod has a service account assigned to it.

Creating Your Own Service Accounts

So, if you want your pod to actually be able to talk to the Kubernetes API and do something, you have two options. You either need to assign some permissions to the default service account, or you need to create a new service account. The first option is not recommended. In fact, you shouldn't use the default service account for anything. Let's choose the recommended option then, which is creating dedicated service accounts. It's also worth mentioning here that, just like with user access, you should create separate service accounts for separate needs.

The easiest way to create a service account is by executing the kubectl create serviceaccount command followed by a desired service account name.

$ kubectl create serviceaccount nginx-serviceaccount
serviceaccount/nginx-serviceaccount created

Just like with anything else in Kubernetes, it's worth knowing how to create one using the YAML definition file. In the case of service accounts, it's actually really simple and looks like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-serviceaccount

I'll save it as nginx-sa.yaml and apply that simple YAML file using kubectl apply -f nginx-sa.yaml:

$ kubectl apply -f nginx-sa.yaml
serviceaccount/nginx-serviceaccount created

You can see from the output above that a service account was created, but you can double-check that with kubectl get serviceaccounts, or kubectl get as for short.

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         3h14m
nginx-serviceaccount   1         72s

$ kubectl get sa
NAME                   SECRETS   AGE
default                1         3h14m
nginx-serviceaccount   1         112s

Assigning Permissions to a Service Account

OK, you created a new service account for your pod, but by default, it won't do much more than the default service account (called "default") that you saw previously. To change that, you can use the standard Kubernetes role-based access control mechanism. This means that you can either use existing role or create a new one (or ClusterRole) and then use RoleBinding to bind a role with your new ServiceAccount.

For the purpose of the demo, you can assign a built-in Kubernetes ClusterRole called "view" that allows viewing all resources. You then need to create a RoleBinding for your Service Account.

kubectl create rolebinding nginx-sa-readonly \
  --clusterrole=view \
  --serviceaccount=default:nginx-serviceaccount \
  --namespace=default

Now, any pod using the new ServiceAccount should be able to view all resources in the default namespace. To validate that, you can perform the same test you did earlier, but first you need to assign that ServiceAccount to your nginx pod.

Specifying ServiceAccount For Your Pod

As mentioned previously, if you don't specify any service account for your pod, it will be assigned a "default" service account. You just created a new service account for your needs, so you'll want to use that one instead. For that, you need to pass the service account name as the value for serviceAccountName key in a spec section of your deployment definition file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx1
  labels:
    app: nginx1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx1
  template:
    metadata:
      labels:
        app: nginx1
    spec:
      serviceAccountName: nginx-serviceaccount
      containers:
      - name: nginx1
        image: nginx
        ports:
        - containerPort: 80

That's it. Now, after applying this definition, you can try to perform the same test as before from within the pod.

$ curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/default/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "52233"
  },
  "items": [
    {
      "metadata": {
        "name": "nginx1-65448895f9-5j6b6",
        "generateName": "nginx1-65448895f9-",
        "namespace": "default",
        "uid": "b09bfa93-a388-4cd9-9495-131f620613d0",
        "resourceVersion": "49536",
(...)

And as expected, now it works fine. If, for any reason, your pods need access to a Kubernetes API, you create a new ServiceAccount for it, then assign a role or ClusterRole to it via RoleBinding, then specify the ServiceAccount name in your deployment.

Summary

Service accounts are extremely useful. They provide a way for your pods to access the Kubernetes API. Many Kubernetes-native tools rely on this mechanism. Therefore, it's good to know what service accounts are and how they access the Kubernetes API.

However, you also need to be careful because a misconfigured service account can be a security risk. If, for example, to save time, you decide to increase the permission for a default service account (instead of creating a new one), you'll make it possible for any pod on the cluster to access the Kubernetes API. If that access is read-only, it can lead to data exposure, and if it's read-write, the damage can be even great. Fortunately, as you learned in this post, creating a service account is easy.

If you want to learn more about Kubernetes, take a look at our other blog posts.

This post was written by Dawid Ziolkowski. Dawid has 10 years of experience as a Network/System Engineer at the beginning, DevOps in between, Cloud Native Engineer recently. He’s worked for an IT outsourcing company, a research institute, telco, a hosting company, and a consultancy company, so he’s gathered a lot of knowledge from different perspectives. Nowadays he’s helping companies move to cloud and/or redesign their infrastructure for a more Cloud Native approach.