DEV Community: LoftLabs

Reimagining Local Kubernetes: Replacing Kind with vind — A Deep Dive

vCluster — Mon, 02 Mar 2026 17:12:16 +0000

Kubernetes developers including myself have long relied on tools like KinD, aka, kind (Kubernetes in Docker) to spin up disposable clusters locally for development, testing, and CI/CD workflows. I love the product and have used it many times but there were certain limitations which were a bit annoying like not being able to use service type LoadBalancer, accessing the homelab kind clusters from the web, using the pull through cache, adding a GPU node to your local kind cluster.

Introducing vind (vCluster in Docker) - An open source alternative to kind or you can say kind on steroids. vind enables Kubernetes clusters as first-class Docker containers — offering improved performance, modern features, and a more enhanced developer experience.

In this post, we'll explore what vind is, how it compares to kind, and walk through real-world usage with examples from the vind repository.

What is vind?

At its core, vind is a way to run Kubernetes clusters directly as Docker containers. Which you can also do using kind and other tooling, so why vind and why should you try it? Well, here is the thing:

vind gives you

FREE vCluster platform UI which you can connect to your local cluster from anywhere in the world — a perfect use case is when you have to give a homelab demo at a conference
Native service type LoadBalancer support
Pull-through image cache via Docker daemon
Easy multi node Kubernetes cluster creation
Attaching external nodes from an EC2 instance or a GPU node to your local cluster via vCluster VPN and no additional tooling required
Flexible CNI choices
Sleep and wakeup your cluster — you can pause the clusters to save resources, resume instantly
Snapshot and backup your cluster is also coming in the next release

In essence, vind is "a better kind" — bringing modern features that matter for real developer workflows.

vind vs Kind

Let's see how vind stacks up against Kind:

Feature	vind	Kind
Built-in UI	✅ via vCluster Platform	❌
Sleep/Wake	✅ Native	❌ requires delete & recreate
Load Balancers	✅ Automatic	❌ Manual setup
Image Caching	✅ Via Docker cache	❌ External registries
External Nodes	✅ Supported	❌ Local only
CNI / CSI Options	✅ Flexible	Limited
Snapshots	🕐 Coming soon	❌

Kind is useful for several use cases, but vind expands these capabilities to support modern tooling like vCluster and Docker drivers providing a richer multi-cloud developer experience.

Installing and Setting Up vind

The docs that I have created are updated with the instructions needed, so we will just follow that and then attach an external node to my local cluster.

Before we dive into examples, let's get vind installed.

1. Prerequisites

Ensure you have:

Docker installed and running
vCluster CLI (v0.31.0 or later)

2. Install / Update vCluster CLI

If you have never installed vCluster CLI then you can install from here.

If you already have the CLI then:

vcluster upgrade --version v0.31.0
vcluster use driver docker

3. Start Optional UI

The vCluster Platform UI gives a nice web interface:

vcluster platform start

Once booted, you can manage clusters visually — something kind doesn't provide out of the box.

You will get login details that you can use to login to the UI and manage your clusters via the UI.

Create Your First vind Cluster

Once set up, creating your first Kubernetes cluster using vind is simple:

vcluster create my-cluster

Then, validate the cluster:

kubectl get nodes
kubectl get namespaces

You now have a Kubernetes cluster running inside Docker, ready for development.

Cluster with multiple nodes?

Let's try to create a cluster with multiple nodes.

You can use one of the examples — multi-node-cluster.yaml

experimental:
  docker:
    # Network configuration
    network: "vind-multi-node"

    # Load balancer and registry proxy are enabled by default

    # Additional worker nodes
    nodes:
      - name: worker-1
        env:
          - "CUSTOM_VAR=value1"

      - name: worker-2
        env:
          - "CUSTOM_VAR=value2"

      - name: worker-3
        env:
          - "CUSTOM_VAR=value3"
privateNodes:
  enabled: true
  vpn:
    enabled: true
    nodeToNode:
      enabled: true

And run the command:

vcluster create my-vcluster -f multi-node-cluster.yaml --upgrade

Boom!! You have a multi node Kubernetes cluster!

Adding external node

This is my favourite one!

In case you want to attach a GPU or external node to your local cluster, you can do that using vCluster VPN that comes with vCluster Free tier.

Add below to the multi-node yaml manifest and recreate the cluster:

privateNodes:
  enabled: true
  vpn:
    enabled: true
    nodeToNode:
      enabled: true

Create the token:

vcluster token createdemo ~ vcluster token create

curl -fsSLk "https://25punio.loft.host/kubernetes/project/default/virtualcluster/my-cluster/node/join?token=eerawx.dwets9a8adfw52gz" | sh -

Now you can create an instance in any cloud provider — I have created an instance in Google Cloud:

gcloud compute instances create my-vm \
  --zone=us-central1-a \
  --machine-type=e2-micro \
  --image-family=ubuntu-2204-lts \
  --image-project=ubuntu-os-cloud

Next ssh into the instance:

gcloud compute ssh my-vm --zone=us-central1-a

Run the curl command:

saiyam@my-vm:~$ sudo su 
root@my-vm:/home/saiyam# curl -fsSLk "https://25punio.loft.host/kubernetes/project/default/virtualcluster/my-cluster/node/join?token=eerawx.hdnueyd9a8myfr52gz" | sh -
Detected OS: ubuntu
Preparing node for Kubernetes installation...
Kubernetes version: v1.34.0
Installing Kubernetes binaries...
Downloading Kubernetes binaries from https://github.com/loft-sh/kubernetes/releases/download...
Loading bridge and br_netfilter modules...
insmod /lib/modules/6.8.0-1046-gcp/kernel/net/llc/llc.ko 
insmod /lib/modules/6.8.0-1046-gcp/kernel/net/802/stp.ko 
insmod /lib/modules/6.8.0-1046-gcp/kernel/net/bridge/bridge.ko 
insmod /lib/modules/6.8.0-1046-gcp/kernel/net/bridge/br_netfilter.ko 
Activating ip_forward...
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
Resetting node...
Ensuring kubelet is stopped...
kubelet service not found
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
sysctl: setting key "net.ipv4.conf.all.accept_source_route": Invalid argument
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/60-gce-network-security.conf ...
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
kernel.randomize_va_space = 2
kernel.panic = 10
* Applying /etc/sysctl.d/99-cloudimg-ipv6.conf ...
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.use_tempaddr = 0
* Applying /etc/sysctl.d/99-gce-strict-reverse-path-filtering.conf ...
* Applying /usr/lib/sysctl.d/99-protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/99-tailscale.conf ...
* Applying /etc/sysctl.conf ...
Starting vcluster-vpn...
Created symlink /etc/systemd/system/multi-user.target.wants/vcluster-vpn.service → /etc/systemd/system/vcluster-vpn.service.
Waiting for vcluster-vpn to be ready...
Waiting for vcluster-vpn to be ready...
Configuring node to node vpn...
Waiting for a tailscale ip...
Starting containerd...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service.
Importing pause image...
registry.k8s.io/pause:3.10               saved 
application/vnd.oci.image.manifest.v1+json sha256:a883b8d67f5fe8ae50f857fb4c11c789913d31edff664135b9d4df44d3cb85cb
Importing elapsed: 0.2 s total:   0.0 B (0.0 B/s) 
Starting kubelet...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
Installation successful!
Joining node into cluster...
[preflight] Running pre-flight checks
W0209 16:03:28.474612    1993 file.go:102] [discovery] Could not access the cluster-info ConfigMap for refreshing the cluster-info information, but the TLS cert is valid so proceeding...
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
W0209 16:03:30.253845    1993 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.96.0.10]; the provided value is: [10.109.18.131]
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.501478601s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:

Certificate signing request was sent to apiserver and a response was received.
The Kubelet was informed of the new secure connection details.

Run kubectl get nodes on the control-plane to see this node join the cluster.

How cool is this!

kubectl get po -A
NAMESPACE            NAME                                      READY   STATUS    RESTARTS   AGE
kube-flannel         kube-flannel-ds-nfb6g                     1/1     Running   0          14m
kube-flannel         kube-flannel-ds-nkxfq                     1/1     Running   0          15m
kube-flannel         kube-flannel-ds-r28k7                     1/1     Running   0          42s
kube-flannel         kube-flannel-ds-tr9cm                     1/1     Running   0          14m
kube-flannel         kube-flannel-ds-xx7p7                     1/1     Running   0          14m
kube-system          coredns-75bb76df-5tbdc                    1/1     Running   0          15m
kube-system          kube-proxy-hfgw7                          1/1     Running   0          15m
kube-system          kube-proxy-l86w5                          1/1     Running   0          42s
kube-system          kube-proxy-lqprr                          1/1     Running   0          14m
kube-system          kube-proxy-sz6f2                          1/1     Running   0          14m
kube-system          kube-proxy-wrkzl                          1/1     Running   0          14m
local-path-storage   local-path-provisioner-6f6fd5d9d9-4tpvj   1/1     Running   0          15m

There are many other features like Sleep Mode, where you can sleep and wakeup of a vind Kubernetes cluster. Accessing your clusters from the UI and attaching external nodes along with working of LoadBalancer service without any additional third party tooling is just amazing!

So I have replaced my kind setup.

For teams focused on:

Fast local spins
CI/CD pipelines
Kubernetes testing
Hybrid workflows
Efficient resource usage

…vind is worth serious consideration.

Summary

Aspect	Kind	vind
Local cluster creation	✅	✅
Ease of management	⚠️ CLI only	✅ UI available
Resource efficiency	⚠️ Must recreate	✅ Sleep/Wake
Networking & LB	⚠️ requires plugins	✅ automatic
Developer experience	Good	Great

If you're evaluating a modern replacement for kind in your tooling — especially for team and CI/Devflow use cases — vind puts a compelling stake in the ground.

⭐ Star the repo

Start creating issues and let's discuss to make it even better!

💬 Join our Slack

vCluster Free: Enterprise Kubernetes Features at No Cost

vCluster — Mon, 23 Feb 2026 13:01:22 +0000

We had such an amazing year 2025 and our commercial offering is gaining a ton of traction with large enterprises, neoclouds and AI factories alike. We doubled our team size and almost tripled our revenue in 2025. This success is rooted in our strong open source project and in our vCluster community, so to kick off 2026, we wanted to make sure to give back to our strongest supporters in the community and we thought hard about how to do this. The result of this are two additions to vCluster that we’ve been working on over the past few months and we’re announcing both of them today:

vind - vCluster in Docker (100% open source, no strings attached)
vCluster Free - Our New Free Tier (makes many Enterprise feature available for free)

To learn more about vind, check out our livestream today on Youtube or LinkedIn. To learn more about vCluster Free, we will run a separate livestream next week (Youtube, LinkedIn) but of course, we’ll show some of it off today. For details about vind and vCluster Free outside of these livestreams, keep reading.

What is vCluster Free?

vCluster Free gives anyone access to many of our enterprise features at no cost. The following features are included in this free tier:

CRD Sync
Sync Patches
Namespace 1:1 Syncing
Custom DNS Entries
Embedded etcd
Private Nodes
Auto Nodes
Standalone
vCluster Platform core features enabling self-service: Templates, CRDs, User Management, RBAC, etc.

It's quite a long list, isn't it? All of these features were previously part of our Enterprise plan, but now you can use them for free.

Why not open source?

We're an open source company deeply committed to ensuring that vCluster thrives as an open source project. To ensure this long-term, we need to build a successful business around it. That's the best way to make open source sustainable.

If we open sourced all of the features above, many enterprises would build around the open source project instead of purchasing our commercial offering, which would make it difficult to sustain the business and continue investing in vCluster. This is already happening today—vCluster provides significant value and enables teams to build real platforms on top of the open source project. Even commercial vendors like SpectroCloud, Rafay, Taikun Cloud, Uffizzi, and others have integrated open source vCluster into their products.

There's clearly a lot of value in vCluster open source. We're continuing to add more features, including our announcement of vind today, which allows you to run vCluster in Docker similar to KinD—entirely available in our open source project. Watch our livestream to learn more about vind. We will continue investing in our open source project and building features valuable to our open source community.

However, to make more of our enterprise features available to the community in a way that doesn't cannibalize our Enterprise offering, enable competitors, or reduce large enterprises' interest in our commercial offering, we decided to launch vCluster Free—an offering between vCluster Open Source and vCluster Enterprise. We designed this free tier to be actually useful and set the limits intentionally high so you can get significant value from it.

What are the limits of vCluster Free?

This free tier is not a trial but our attempt to make more enterprise features permanently available to the community. There is no time limit on how long you can use this free tier. Instead, it's limited by infrastructure size:

Up to 64 vCPU cores
Up to 32 GPUs

As long as you stay below these numbers, you can use any of the features above and you get:

Unlimited Users
Unlimited Host Clusters
Unlimited Virtual Clusters (max 1 HA) We wanted this free tier to be actually useful and enable anyone who loves vCluster to access more of the features we're building for our enterprise customers.

What can I do with vCluster Free?

vCluster Free lets anyone use many of our most advanced vCluster Enterprise features without speaking to our sales team or providing a credit card. We believe the limits above are high enough to cover many use cases entirely. Such use cases include:

vCluster as dev environments in startups or R&D teams within larger orgs
Ephemeral CI environments for running e2e tests (especially for operators/CRDs)
Simulating cluster changes (upgrading platform components)
Running vendor software in a more isolated way (e.g., GitLab runners, etc.)
Ephemeral clusters for demos (product demos, conference talks, etc.)
Running a small production cluster
Home labs and personal test environments

There are probably many more use cases. Many of them can already be addressed by our open source project, but with the new vCluster Free offering, you get access to even more functionality, a great UI, plus all the CRDs and controllers that come with vCluster Platform.

I hope you enjoy this new offering. You can try it today by signing up for vCluster Cloud or spinning up vCluster Platform using any of the commands below:

# Run in Docker (for local testing)
vcluster platform start --docker
# Run in Kubernetes
vcluster platform start

We’re excited to see what you build with vCluster Free.

Watch the liverstream demo here.

Originally published at https://www.vcluster.com.

One giant Kubernetes cluster for everything

Nicolas Fränkel — Thu, 20 Mar 2025 09:02:00 +0000

The ideal size of your Kubernetes clusters is a day 0 question and demands a definite answer.

You find one giant cluster on one end of the spectrum and many small-sized ones on the other, with every combination in between. This decision will impact your organization for years to come. Worse, if you decide to change your topology, you're in for a time-wasting and expensive ride.

I want to list each approach's pros and cons in this post. Then, I'll settle the discussion once and for all and argue why selecting the giant cluster option is better.

The one giant cluster approach

Deciding on a single giant cluster has great benefits.

Better resource utilization

Kubernetes was designed to handle large-scale deployments, initially focusing on managing thousands of nodes to support extensive and complex containerized applications. This scalability was a key feature from its inception, enabling it to orchestrate resources across vast, distributed systems efficiently.

Thus, a Kubernetes cluster is a scheduler at its core: it knows how to run workloads on nodes according to constraints. Without constraints, it will happily balance workloads across its available nodes. If you split the cluster into multiple clusters, you lose this benefit. You can have situations where one cluster is idle while another cluster is close to resource starvation and must cancel workloads and kill pods.

Lower operational overhead

Good Kubernetes practices mandate that you back up your etcd data, monitor your cluster metrics, log your cluster events, provide security-related tools, etc. Size aside, it stands to reason that it's more time-effective to operate fewer clusters.

For example, regarding metrics, you'd set up a single Prometheus instance, potentially clustered to handle additional traffic, and be done with it. Automation can mitigate the repetitive aspect of installing and maintaining an instance for each cluster, but you'll still end up with as many instances as you have clusters (or more). Prometheus is just one example because many cluster admins have a long list of tools they run in every cluster.

Straightforward networking and service communication

Service-to-service communication inside a single cluster is straightforward. Point to <service-name>.<namespace>.svc.cluster.local and be done with it. Even better, you only need the service name part inside the same namespace.

You'll need a tool to help you with inter-cluster communication, from the simplicity of External DNS with LoadBalancer to the complexity of a full-fledged solution like Istio—both ends of the spectrum mandate time and operation costs.

Simplified governance

Since every object is part of the same cluster inside a single cluster, one can enforce a centralized set of policies with a standardized approach. For example, you can create a namespace per team and environment, restricting access to only that team's members.

Once you start having multiple clusters, even if you take the same approach, you'll duplicate the policy rules across clusters, with potential differences that will drift further with time.

Cost efficiency

A single cluster means a single control plane, simplifying management and reducing overhead. While a control plane is essential for orchestration, its value comes from enabling efficient workload execution rather than directly running business applications.

Additionally, many of the points above tie into cost optimization. With a single cluster, you only need to configure monitoring (e.g., Prometheus), logging, and security tools once, reducing duplication. In-place automation streamlines operations, helping manage costs without adding unnecessary infrastructure expenses.

Downsides of a one giant cluster approach

Unfortunately, the giant cluster option is not only unicorns and rainbows; there are definite downsides.

Larger blast radius

The larger the cluster, the more teams will use it. Unfortunately, this means that if something bad happens to the cluster, it will wreak havoc on the work of more teams. It occurs regardless of whether the outage results from a malicious actor, a wrong configuration, or resource starvation.

When an actual malicious actor does indeed breach access, a larger cluster exposes more workloads, and the actor can compromise more of them.

Even without malicious actors, every maintenance operation and upgrade on a cluster can affect its users; the bigger the cluster, the larger the potential impact. When planning an upgrade for a single cluster, you need to conduct an impact analysis that encompasses all users and teams in the organization.

Complex multi-tenancy management

Even within a single organization, multiple teams will use the Kubernetes cluster. Even if every team member behaves professionally, we must set strict policies to avoid issues. If the cluster resembles a building, you'd still put locks on your apartment even if you have friendly neighbors. Likewise, the cluster administrator must enforce strict rules to make sharing a cluster acceptable. At the very least, we need strict namespace isolation to avoid unnecessary access and resource quotas and enforce fairness across teams. The problems caused by using a single cluster by teams across a single organization multiply a hundredfold if the cluster is multi-tenant and shared across several organizations.

Scalability limits

Regardless of how exceptional Kubernetes is, it's still a physically based system with physical limits. For example, some objects have a clear limit, but even if you never reach them, getting close to them will require excellent system administration skills with some fine-tuning.

Even then, the more load you put on a Kubernetes API server, the more sluggish your system will be. If you're lucky, it will degrade linearly, but chances are it will hit some system limit and degrade all at once.

Cluster-wide objects

Kubernetes objects are namespace-scoped, but a couple of them are cluster-scoped. A cluster-scoped object can only have a single instance across the whole cluster. For example, a Custom Resource Definition is cluster-scoped.

It means that if a team wants to use v1 of a CRD, then every team on the same cluster is stuck with v1 if they wish to use this CRD. Worse, if any team wants to upgrade to v2, they must coordinate across all teams using the CRD to synchronize the upgrade.

What's the ideal size, then?

I could describe the pros and cons of very granular clusters, but they mirror the opposite of what we have just seen. For example, very granular clusters allow each team to work on their version of a CRD without stepping on another team's toes. For this reason, I'll avoid repeating myself.

Most, if not all, articles evaluating the pros and cons of each end of the spectrum advise a meet-in-the-middle approach: "a couple" of clusters to mitigate the worst aspects of each extreme approach. It's all well and good, but none of them, at least none I've read, tell precisely how many "a couple" is. Is it a cluster per environment, i.e., production, staging, or development? Is it a cluster per team?

I'll take a risk and advertise for two clusters: one for production and the other for everything else. How would you manage the cons mentioned above? Read on.

vCluster

vCluster is an Open Source product that allows creating of so-called virtual clusters. vCluster is part of the CNCF landscape, specifically a certified Kubernetes distribution. Being a certified distro means a virtual cluster offers every Kubernetes API you can expect, and you can deploy any application to it just like any other Kubernetes cluster.

vCluster operates by creating the virtual cluster in a dedicated namespace. You can specify the latter, or vCluster will infer it from the virtual cluster's name. By default, it creates a control plane using the vanilla k8s distribution, but you can choose another one, such as k3s. Likewise, by default, it stores its configuration in an SQLite database, which works particularly well for temporary and pre-production clusters, such as those you create for a pull request. Alternatively, you can rely on a regular etcd or even external databases such as MySQL or Postgres as a data store for more permanent usage and better resilience and scalability of the virtual cluster.

Once you've created a virtual cluster via the CLI or the Helm chart, you can connect to it. The client-side CLI creates a dedicated reusable kubeconfig context. From within a virtual cluster, users see no other virtual clusters.

If you need to access the host cluster resources from the virtual cluster or vice versa, vCluster uses a so-called syncer that syncs objects back and forth according to a configuration file. This way, you can set up an Ingress Controller on the host cluster and define your Ingress objects in the virtual cluster(s).

How vCluster mitigates the downsides of a giant cluster

Let's review each downside of a giant cluster and how vCluster handles it.

Larger blast radius: When using a virtual cluster, the blast radius is automatically contained inside its boundaries. If you want to be conservative, aim for a small granularity approach, such as a cluster per team and environment.
Complex multi-tenancy management: Gone are multi-tenancy problems since your tenants don't see each other and are isolated inside their respective virtual clusters.
Scalability limits: While the limits are still there, the chances of reaching them decrease with the number of virtual clusters. If your giant cluster had 100k services, they are now spread throughout all virtual clusters. Even if the distribution is not even (and won't be), it gives you fresh air.
Upgrades and maintenance risks: Upgrade and maintenance tasks are limited to the scope of a single virtual cluster. You can do them in turn, and they will only affect the virtual clusters you target.
Cluster-wide objects: Finally, with virtual clusters, every team can install their version of a CRD, and its virtual cluster binds the CRD. It allows each team to be entirely independent of each other regarding the version of a CRD they use.

But I need different clusters!

While a single giant cluster provides compelling advantages, there are contexts in which a multi-cluster approach is justified. The most common reason is geographic distribution—specific applications require clusters in multiple regions to meet compliance requirements, reduce latency, or provide disaster recovery. For example, companies operating under GDPR or financial regulations may need strict data residency enforcement, which requires region-specific clusters. Similarly, organizations with stringent security postures may enforce complete isolation between environments or business units, making separate clusters a hard requirement.

However, even in these cases, vCluster remains relevant. It allows for minimizing the number of physical clusters while still enabling workload separation at a virtual level. Instead of creating a sprawling landscape of Kubernetes clusters, teams can deploy regional virtual clusters within a single host cluster, balancing isolation and operation complexity.

Conclusion

Kubernetes cluster topology decisions are critical and long-lasting. While many advocate for a middle-ground approach between a single cluster and many small ones, they rarely specify the exact setup. Instead of guessing how many clusters to create, consolidating everything into a single, well-managed giant cluster makes more sense. The benefits—better resource utilization, lower operational overhead, simplified networking, centralized governance, and cost efficiency—outweigh the downsides.

That said, the traditional downsides of a giant cluster, such as a larger blast radius, multi-tenancy complexities, scalability limits, upgrade challenges, and cluster-wide object constraints, are valid concerns. This is where vCluster changes the game. By using virtual clusters, you retain all the advantages of a single giant cluster while mitigating its worst drawbacks. vCluster isolates workloads, reduces operational risk, scales dynamically, simplifies upgrades, and removes conflicts over cluster-wide objects.

Enhanced with vCluster, one cluster for production and one giant cluster for everything else, is the best approach for long-term scalability, efficiency, and ease of operations.

WebAssembly on Kubernetes

Nicolas Fränkel — Thu, 06 Mar 2025 09:02:00 +0000

Like a couple of innovative technologies, different people have different viewpoints on where WebAssembly fits the technology landscape.

WebAssembly (also called Wasm) is certainly the subject of much hype right now. But what is it? Is it the JavaScript Killer? Is it a new programming language for the web? Is it (as we like to say) the next wave of cloud compute? We’ve heard it called many things: a better eBPF, the alternative to RISC V, a competitor to Java (or Flash), a performance booster for browsers, a replacement for Docker.

-- How to think about WebAssembly

In this post, I'll stay away from these debates and focus solely on how to use WebAssembly on Kubernetes.

My approach and the use case

Unlike regular programming languages, you don't write WebAssembly directly: you write code that generates WebAssembly. At the moment, Go and Rust are the main source languages. I know Kotlin and Python are working toward this objective. There might be other languages I'm not aware of.

I've settled on Rust for this post because of my familiarity with the language. In particular, I'll keep the same code across three different architectures:

Regular Rust-to-native code as the baseline
Rust-to-WebAssembly using a WasmEdge embedded runtime
Rust-to-WebAssembly using an external runtime

Don't worry; I'll explain the difference between the two last approaches later.

The use case should be more advanced than Hello World to highlight the capabilities of WebAssembly. I've implemented an HTTP server mimicking a single endpoint of the excellent httpbin API testing utility. The code itself is not essential as the post is not about Rust, but in case you're interested, you can find it on GitHub. I add a field to the response to explicitly return the underlying approach, respectively native, embed, or runtime.

Baseline: regular Rust-to-native

For the regular native compilation, I'm using a multistage Docker file:

FROM rust:1.84-slim AS build                                             #1

RUN <<EOB                                                                #2
  apt-get update
  apt-get install -y musl-tools musl-dev
  rustup target add aarch64-unknown-linux-musl                           #3
EOB

WORKDIR /native

COPY native/Cargo.toml Cargo.toml

WORKDIR /

COPY src src

WORKDIR /native

RUN RUSTFLAGS="-C target-feature=+crt-static" cargo build --target aarch64-unknown-linux-musl --release #4

FROM gcr.io/distroless/static                                            #5

COPY --from=build /native/target/aarch64-unknown-linux-musl/release/httpbin httpbin #6

ENTRYPOINT ["./httpbin"]

Start from the latest Rust image
Heredocs for the win
Install the necessary toolchain to cross-compile
Statically compile
I could potentially use FROM scratch, but after reading this, I prefer to use distroless
Copy the executable from the previous compilation phase

The final wasm-kubernetes:native image weighs 8.71M, with its base image distroless/static taking 6.03M of them.

Adapting to WebAssembly

The main idea behind WebAssembly is that it's secure because it can't access the host system. However, we must open a socket to listen to incoming requests to run an HTTP server. WebAssembly can't do that. We need a runtime that provides this feature and other system-dependent capabilities. It's the goal of WASI.

The WebAssembly System Interface (WASI) is a group of standards-track API specifications for software compiled to the W3C WebAssembly (Wasm) standard. WASI is designed to provide a secure standard interface for applications that can be compiled to Wasm from any language, and that may run anywhere—from browsers to clouds to embedded devices.

-- Introduction to WASI

The specification v0.2 defines the following system interfaces:

Clocks
Random
Filesystem
Sockets
CLI
HTTP

A couple of runtimes already implement the specification:

Wasmtime, developed by the Bytecode Alliance
Wasmer
Wazero, Go-based
WasmEdge, designed for cloud, edge computing, and AI applications
Spin for serverless workloads

I had to choose without being an expert in any of these. I finally decided on WasmEdge because of its focus on the Cloud.

We must intercept code that calls with system APIs and redirect them to the runtime. Instead of runtime interception, the Rust ecosystem provides a patch mechanism: we replace code that calls system APIs with code that calls WASI APIs. We must know which dependency calls which system API and hope a patch exists for our dependency version.

[patch.crates-io]
tokio = { git = "https://github.com/second-state/wasi_tokio.git", branch = "v1.36.x" }  #1-2
socket2 = { git = "https://github.com/second-state/socket2.git", branch = "v0.5.x" }    #1

[dependencies]
tokio = { version = "1.36", features = ["rt", "macros", "net", "time", "io-util"] }     #2
axum = "0.8"
serde = { version = "1.0.217", features = ["derive"] }

Patch the tokio and socket2 crates with WASI-related calls
The latest tokio crate is 1.43, but the latest (and only) patch v1.36. We can't use the latest version because there's no patch.

We must change the Dockerfile to compiler WebAssembly code instead of native:

FROM --platform=$BUILDPLATFORM rust:1.84-slim AS build

RUN <<EOT bash
    set -ex
    apt-get update
    apt-get install -y git clang
    rustup target add wasm32-wasip1                                      #1
EOT

WORKDIR /wasm

COPY wasm/Cargo.toml Cargo.toml

WORKDIR /

COPY src src

WORKDIR /wasm

RUN RUSTFLAGS="--cfg wasmedge --cfg tokio_unstable" cargo build --target wasm32-wasip1 --release #2-3

Install the WASM target
Compile to WASM
We must activate the wasmedge flag, as well as the tokio_unstable one, to successfully compile to WebAssembly

At this stage, we have two options for the second stage:

Use the WasmEdge runtime as a base image:

FROM --platform=$BUILDPLATFORM wasmedge/slim-runtime:0.13.5

COPY --from=build /wasm/target/wasm32-wasip1/release/httpbin.wasm /httpbin.wasm

CMD ["wasmedge", "--dir", ".:/", "/httpbin.wasm"]

From a usage perspective, it's pretty similar to the native approach.

Copy the WebAssembly file and make it a runtime responsibility:

FROM scratch

COPY --from=build /wasm/target/wasm32-wasip1/release/httpbin.wasm /httpbin.wasm

ENTRYPOINT ["/httpbin.wasm"]

It's where things get interesting.

The native approach is slightly better than the embed one, but the runtime is the leanest since it contains only a single Webassembly file.

Running the Wasm image on Docker

Not all Docker runtimes are equal, and to run Wasm workloads, we need to delve a bit into the Docker name. While Docker, the company, created Docker as the product, the current reality is that containers have evolved beyond Docker and now answer to specifications.

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Established in June 2015 by Docker and other leaders in the container industry, the OCI currently contains three specifications: the Runtime Specification (runtime-spec), the Image Specification (image-spec) and the Distribution Specification (distribution-spec). The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. At a high-level an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

-- Open Container Initiative

From then on, I'll use the proper terminology for OCI images and containers. Not all OCI runtimes are equal, and far from all of them can run Wasm workloads: OrbStack, my current OCI runtime, can't, but Docker Desktop can, as an experimental feature. As per the documentation, we must:

Use containerd for pulling and storing images
Enable Wasm

Finally, we can run the above OCI image containing the Wasm file by selecting a Wasm runtime, Wasmedge, in my case. Let's do it:

docker run --rm -p3000:3000 --runtime=io.containerd.wasmedge.v1 ghcr.io/ajavageek/wasm-kubernetes:runtime

io.containerd.wasmedge.v1 is the current version of the Wasmedge runtime. You must be authenticated with GitHub if you want to try it out.

curl localhost:3000/get\?foo=bar | jq

The result is the same as for the native version:

{
  "flavor": "runtime",
  "args": {
    "foo": "bar"
  },
  "headers": {
    "accept": "*/*",
    "host": "localhost:3000",
    "user-agent": "curl/8.7.1"
  },
  "url": "/get?foo=bar"
}

Wasi on Docker Desktop allows you to spin up an HTTP server that behaves like a regular native image! Even better, the image size is as tiny as the WebAssembly file it contains:

Repository	Tag	Size (Mb)
ghcr.io/ajavageek/wasm-kubernetes	runtime	1.15
ghcr.io/ajavageek/wasm-kubernetes	embed	12.4
ghcr.io/ajavageek/wasm-kubernetes	native	8.7

Running the Wasm image on Kubernetes

Now comes the fun part: your favorite Cloud provider(s) isn't using Docker Desktop. Despite this, we can still run WebAssembly workloads on Kubernetes. For this, we need to understand a bit about the not-too-low levels of what happens when you run a container, regardless of whether it's from an OCI runtime or Kubernetes.

The latter executes a process; in our case, it's containerd. Yet, containerd is only an orchestrator of other container processes. It detects the "flavor" of the container and calls the relevant executable. For example, for "regular" containers, it calls runc via a shim. The good thing is that we can install other shims dedicated to other container types, such as Wasm. The following illustration, taken from the Wasmedge website, summarizes the flow:

Despite some of the mainstream Cloud providers offering Wasm integration, none of them provide such a low-level one. I'll continue on my laptop, but Docker Desktop doesn't offer a direct integration either: it's time to be creative. For example, minikube is a full-fledged Kubernetes distribution that creates an intermediate Linux virtual machine within a Docker environment. We can SSH into the VM and configure it to our heart's content. Let's start by installing minikube.

brew install minikube

Now, we start minikube with the containerd driver and specify a profile to enable differently configured VMs. We unimaginatively call this profile wasm.

minikube start --driver=docker --container-runtime=containerd -p=wasm

Depending on whether you have already installed minikube and whether it has already downloaded its images, starting can take a few seconds to dozens of minutes. Be patient. The output should be something akin to:

😄  [wasm] minikube v1.35.0 on Darwin 15.1.1 (arm64)
✨  Using the docker driver based on user configuration
📌  Using Docker Desktop driver with root privileges
👍  Starting "wasm" primary control-plane node in "wasm" cluster
🚜  Pulling base image v0.0.46 ...
❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.46, but successfully downloaded docker.io/kicbase/stable:v0.0.46@sha256:fd2d445ddcc33ebc5c6b68a17e6219ea207ce63c005095ea1525296da2d1a279 as a fallback image
🔥  Creating docker container (CPUs=2, Memory=12200MB) ...
📦  Preparing Kubernetes v1.32.0 on containerd 1.7.24 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "wasm" cluster and "default" namespace by default

At this point, our goal is to install on the underlying VM:

Wasmedge to run Wasm workloads
A shim to bridge between containerd and wasmedge

minikube ssh -p wasm

We can install Wasmedge, but I found nowhere to download the shim. In the next step, we will build both. We first need to install Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

The script likely complains that it can't execute the downloaded binary:

Cannot execute /tmp/tmp.NXPz8utAQx/rustup-init (likely because of mounting /tmp as noexec).
Please copy the file to a location where you can execute binaries and run ./rustup-init.

Follow the instructions:

cp /tmp/tmp.NXPz8utAQx/rustup-init .
./rustup-init

Proceed with the default installation by pressing the ENTER button. When it's finished, source your current shell.

. "$HOME/.cargo/env"

The system is ready to build Wasmedge and the shim.

sudo apt-get update
sudo apt-get install -y git

git clone https://github.com/containerd/runwasi.git

cd runwasi
./scripts/setup-linux.sh

make build-wasmedge
INSTALL="sudo install" LN="sudo ln -sf" make install-wasmedge

The last step requires configuring the containerd process with the shim. Insert the following snippet in the [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] section of the /etc/containerd/config.toml file:

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedgev1]
          runtime_type = "io.containerd.wasmedge.v1"

Restart containerd to load the new config.

sudo systemctl restart containerd

Our system is finally ready to accept Webassembly workloads. Users can deploy a Wasmedge pod with the following manifest:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge                                                         #1
handler: wasmedgev1                                                      #2
---
apiVersion: v1
kind: Pod
metadata:
  name: runtime
  labels:
    arch: runtime
spec:
  containers:
    - name: runtime
      image: ghcr.io/ajavageek/wasm-kubernetes:runtime
  runtimeClassName: wasmedge                                             #3

Wasmedge workloads should use this name
Handler to use. It should be the last segment of the section added in the TOML file, i.e., containerd.runtimes.wasmedgev2
Point to the runtime class name we defined just above

I used a single Pod instead of a full-fledged Deployment to keep things simple.

Notice the many levels of indirection:

The pod refers to the wasmedge runtime class name
The wasmedge runtime class points to the wasmedgev1 handler
The wasmedgev1 handler in the TOML file specifies the io.containerd.wasmedge.v1 runtime type

Final steps

To compare the approaches and test our work, we can use the minikube ingress addon and vCluster. The former offers a single access point for all three workloads, native, embed, and runtime, while vCluster isolates workloads from each other in their virtual cluster.

Let's start by installing the addon:

minikube -p wasm addons enable ingress

It deploys an Nginx Ingress Controller in the ingress-nginx namespace:

💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
💡  After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.11.3
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

We must create a dedicated virtual cluster to deploy the Pod later.

helm upgrade --install runtime vcluster/vcluster --namespace runtime --create-namespace  --values vcluster.yaml

We will define the Ingress, the Service, and their related Pod in each virtual cluster. We need vCluster to synchronize the Ingress with the Ingress Controller. Here's the configuration to achieve this:

sync:
  toHost:
    ingresses:
      enabled: true

The output should be similar to:

Release "runtime" does not exist. Installing it now.
NAME: runtime
LAST DEPLOYED: Thu Jan 30 11:53:14 2025
NAMESPACE: runtime
STATUS: deployed
REVISION: 1
TEST SUITE: None

We can amend the above manifest with the Service and Ingress to expose the Pod:

apiVersion: v1
kind: Service
metadata:
  name: runtime
spec:
  type: ClusterIP                                                        #1
  ports:
    - port: 3000                                                         #1
  selector:
    arch: runtime
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: runtime
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"                        #2
    nginx.ingress.kubernetes.io/rewrite-target: /$2                      #2
spec:
  ingressClassName: nginx
  rules:
    - host: localhost
      http:
        paths:
          - path: /runtime(/|$)(.*)                                      #4
            pathType: ImplementationSpecific                             #4
            backend:
              service:
                name: runtime
                port:
                  number: 3000

Expose the Pod inside the cluster
Nginx-specific annotations to handle path regular expression and rewrite it
Regex path

Nginx will forward all requests starting with /runtime to the runtime service, removing the prefix. To apply the manifest, we first connect to the previously created virtual cluster:

vcluster connect runtime

11:53:21 info Waiting for vcluster to come up...
11:53:39 done vCluster is up and running
11:53:39 info Starting background proxy container...
11:53:39 done Switched active kube context to vcluster_embed_embed_vcluster_runtime_runtime_wasm
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

Now apply the manifest:

kubectl apply -f runtime.yaml

We do the same with the embed and the native pods, barring the runtimeClassName as they are "regular" images.

The final deployment diagram is the following:

The final touch is to tunnel to expose services:

minikube -p wasm tunnel

✅  Tunnel successfully started

📌  NOTE: Please do not close this terminal as this process must stay alive for the tunnel to be accessible ...

❗  The service/ingress runtime-x-default-x-runtime requires privileged ports to be exposed: [80 443]
🔑  sudo permission will be asked for it.
🏃  Starting tunnel for service runtime-x-default-x-runtime.
Password:

Let's request the lightweight container that uses the Wasmedge runtime:

curl localhost/runtime/get\?foo=bar | jq

We get the expected output:

{
  "flavor": "runtime",
  "args": {
    "foo": "bar"
  },
  "headers": {
    "user-agent": "curl/8.7.1",
    "x-forwarded-host": "localhost",
    "x-request-id": "dcbdfde4715fbfc163c7c9098cbdf077",
    "x-scheme": "http",
    "x-forwarded-for": "10.244.0.1",
    "x-forwarded-scheme": "http",
    "accept": "*/*",
    "x-real-ip": "10.244.0.1",
    "x-forwarded-proto": "http",
    "host": "localhost",
    "x-forwarded-port": "80"
  },
  "url": "/get?foo=bar"
}

We should get similar results with the other approaches, with different flavor values.

Conclusion

In this post, I showed how to use Webassembly on Kubernetes with the Wasmedge runtime. I created three flavors for comparison purposes: native, embed, and runtime. The first two are "regular" Docker images, while the latter contains only a single Wasm file, which makes it very lightweight and secure. However, we need a dedicated runtime to run it.

Regular managed Kubernetes services don't allow configuring an additional shim, such as the Wasmedge shim. Even on my laptop, I had to be creative to make it happen. I had to use Minikube and put much effort into configuring its intermediate virtual machine to run Wasm workloads on Kubernetes. Yet, I managed to run all three images inside their virtual cluster, exposed outside the cluster by an Nginx Ingress Controller.

Now, it's up to you to decide whether the extra effort is worth the 10x reduction of image size and the improved security. I hope the future will improve the support so that the pros outweigh the cons.

The complete source code for this post can be found on GitHub:

ajavageek / wasm-kubernetes

helm upgrade --install runtime vcluster/vcluster --namespace runtime --create-namespace  --values vcluster.yaml
helm upgrade --install embed vcluster/vcluster --namespace embed --create-namespace  --values vcluster.yaml
helm upgrade --install native vcluster/vcluster --namespace native --create-namespace  --values vcluster.yaml

vcluster connect runtime
kubectl apply -f runtime.yaml
vcluster connect embed
kubectl apply -f embed.yaml
vcluster connect native
kubectl apply -f native.yaml

View on GitHub

Go further:

Technical Guide: Syncing Ingress Resources from various Virtual Cluster on GKE with vCluster

Hrittik Roy — Mon, 03 Mar 2025 13:43:35 +0000

Kubernetes Ingress is the most widely used Kubernetes resource for exposing an application to the outside world. Understanding the concepts and Layer-7 load balancing may sound difficult, but with this article, it won’t be.

This article uses Google Kubernetes Engine (GKE) as the host Kubernetes cluster, where you will install the Nginx Ingress controller and cert-manager to get TLS certificates for your web apps. With that, you can run an application with proper TLS.

This article touches on how to create a virtual cluster using vCluster to reuse the host cluster ingress controller and cert-manager to create ingress. This approach allows your virtual clusters to reuse the ingress controller running on the host cluster, GKE in our case, and the cert-manager.

If you are hearing about virtual clusters for the first time, then read more here.

Prerequisites

Ensure these are installed:

Google Cloud SDK (gcloud) – To interact (create/delete) with GKE. Make sure to have a project and a billing account linked and authenticated. Install [Note: You can use the UI if you prefer]
Kubernetes CLI (kubectl) – To manage Kubernetes clusters. Install
vCluster CLI – To create virtual clusters within a single Kubernetes cluster. Install
A Domain Name – This is used to configure A Records, DNS, and TLS.
Basic Kubernetes Knowledge – Familiarity with Deployments, Ingress, and Services.

Setting Up GKE Cluster

Once you have completed the prerequisites, you can proceed to the next step: creating your Kubernetes cluster. This cluster will serve as the environment where you deploy your controllers, deployments, services as well as your virtual clusters.

With gcloud installed, you can create your cluster using the command gcloud container clusters create, along with a few parameters to specify the project and location of the deployment:

Command:

gcloud container clusters create test  --project=hrittik-project --zone asia-southeast1-c

After you apply the above command, your gcloud SDK will begin creating the cluster. This process will take a few minutes, and once completed, a kubeconfig entry will be generated that allows you to interact with the cluster using kubectl.

Successful cluster creation will look something similar:

Creating cluster test in asia-southeast1-c... Cluster is being health-checked (Kubernetes Control Plane is healthy)...done.                                                                     
Created [https://container.googleapis.com/v1/projects/hrittik-project/zones/asia-southeast1-c/clusters/test].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/asia-southeast1-c/test?project=hrittik-project
kubeconfig entry generated for test
NAME  LOCATION           MASTER_VERSION      MASTER_IP       MACHINE_TYPE  NODE_VERSION        NUM_NODES  STATUS
test  asia-southeast1-c  1.30.8-gke.1051000  34.124.254.219  e2-medium     1.30.8-gke.1051000  3          RUNNING

Setting Up the Ingress Controller

Once your GKE cluster is set up and operational, the next step is to install an Ingress Controller to handle the routing of external traffic to your services. In this section, we will use NGINX as our Ingress Controller. NGINX is a popular and reliable choice for Kubernetes ingress management, renowned for its flexibility and performance.

Ingress will act as a reverse proxy to route traffic outside the cluster to specific Kubernetes Services based on the Ingress Rules.

Cluster Admin Permissions

To configure Ingress on GKE, the first step is to provide the user cluster-admin permissions to carry out the operational tasks:

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

Success in this step will create a cluster-admin-bindingin your cluster like below:

clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created

Ingress Controller Deployment

With the appropriate permissions configured, the next step is to deploy the controller which will manage all of the routing logic. The step’s as simple as running the command in your cluster:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0/deploy/static/provider/cloud/deploy.yaml

Success will create resources in an ingress-nginx namespace as shown below:

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

Testing Ingress Controller

The key point to focus on is that the ingress-controller will be exposed as a LoadBalancer by default, which we will explore later in the configuring A Record Section. For now, to verify the installation, check if your pods are running by using the command below:

kubectl get pods -n ingress-nginx

The Status as Running means everything is configured correctly:

~ ❯ kubectl get pods -n ingress-nginx                                  
NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-bh7nl       0/1     Completed   0          100s
ingress-nginx-admission-patch-bvlhf        0/1     Completed   0          100s
ingress-nginx-controller-cbb88bdbc-5dkxt   1/1     Running     0          101s

Installing Cert-Manager on the Host Cluster

Now that your Ingress Controller is set up and running, the next step is to install Cert-Manager to automate the management and issuance of TLS certificates. Cert-Manager is a powerful Kubernetes project that simplifies obtaining and renewing SSL/TLS certificates from various certificate authorities (CAs) like for eg: Let's Encrypt.

This tutorial uses Cert Manager to create and store certificates as Kubernetes Secrets automatically.

Installation of cert-manager

The installation process is straightforward; simply execute the command below on your cluster:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml

Successful installation will look similar to the screenshot below:

Verifying cert-manager Installation

Once installed, you can verify that Cert-Manager is running by checking the pods in the cert-manager namespace:

kubectl get pods --namespace cert-manager

Successful creation will display a manager pod, a cainjector pod, and a webhook pod to verify domain authority within that namespace, as shown below:

~ ❯ kubectl get pods --namespace cert-manager                                                                                                

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6c7fdcbcd5-6lp94              1/1     Running   0          2m11s
cert-manager-cainjector-64d77f8498-f6p7d   1/1     Running   0          2m12s
cert-manager-webhook-68796f6795-59sqq      1/1     Running   0          2m11s

Creating a Certificate Issuer CRD

A Certificate Issuer instructs the manager on how to obtain certificates. There are two primary types of issuers:

1. ClusterIssuer: This issuer is available throughout the entire cluster and is recommended for most cases.

2. Issuer: This issuer is limited to a specific namespace.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
 namespace: cert-manager
spec:
 acme:
   # The ACME server URL
   server: https://acme-v02.api.letsencrypt.org/directory 
   # Email address used for ACME registration
   email: test-hrittik@example.com # Replace with your email
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-prod
   # Enable the HTTP-01 challenge provider
   solvers:
   - http01:
       ingress:
         class: nginx
EOF

In this example, we will use a ClusterIssuer with Let's Encrypt as the verification server. Don’t forget to update your email address in the YAML below before applying:

Success will look similar:

clusterissuer.cert-manager.io/letsencrypt-prod created

Verify the Cluster Issuer

To verify successful installation, use the below command to get the clusterissuer CRD:

kubectl get clusterissuer letsencrypt-prod

Successful output will be similar to:

NAME AGE 
letsencrypt-prod 3m

Configuring A Records for Domain

Now that you have set up your NGINX Ingress Controller and are issuing certificates with Cert-Manager, it’s time to configure the A Records **for your domain. In Google Kubernetes Engine (GKE), A Records **are used for mapping.

An A Record (Address Record) in DNS points a domain or subdomain to a specific IP address. In this case, you need to direct your domain (for example, hrittikhere.live) to the external IP address of your NGINX Ingress Controller. This will allow external traffic to be properly routed to your cluster.

Getting the External-IP

After installing the NGINX Ingress Controller, a LoadBalancer type service will be created that automatically allocates an external IP address. To find the allocated IP, use the following command:

 kubectl get svc -A | grep -E "NAME|ingress-nginx-controller"

The output will display a list of services, from which you should select the one that has an external IP. Keep the IP address readily available:

Configuring A Name Record on Domain Provider

The next step is to log in to your domain provider and navigate to the DNS Record section. Depending on the provider, the UI might change but you just need to configure four three things once you click on Create New Record:

Name/Host: * # Wildcard to capture all Hosts

Type: A # A record for GCP

Value: 34.142.255.42 # External IP address of your NGINX Ingress Controller

TTL: 300 # Use the default value.

On the Dashboard, this will look something similar to below:

Click on Add Record and you will be all ready for the next step!

Creating an Application on the Host Cluster

Now, you have all of the moving pieces in place: domain to serve your application, ingress controller for Load Balancing and Cert-Manager for TLS. The only missing piece is application so the next step is to deploy one simple game to your cluster.

The following YAML deployed three things together:

Service: Exposes the game application via a Kubernetes Service.

Deployment: Deploys the game application.

Ingress: Defines routing rules for the Ingress Controller to route external traffic to the Service.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: game-2048
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-2048
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-2048
  template:
    metadata:
      labels:
        app: game-2048
    spec:
      containers:
        - name: backend
          image: alexwhen/docker-2048
          ports:
            - name: http
              containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: game-2048-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: game.hrittikhere.live
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: game-2048
                port:
                  number: 80
  tls:
  - hosts:
    - game.hrittikhere.live
    secretName: letsencrypt-prod
EOF

Success will create three objects in your cluster:

service/game-2048 created
deployment.apps/game-2048 created
ingress.networking.k8s.io/game-2048-ingress created

Verification of Ingress

To find the ingress Host you can use the following command:

kubectl get ingress

Success will look something like this:

~/Code/ingress ❯ kubectl get ingress                                                                                               
NAME                CLASS    HOSTS                   ADDRESS         PORTS     AGE

game-2048-ingress   nginx    game.hrittikhere.live   34.142.255.42   80, 443   51s

If you go to the following URL, you will find your application running on the host:

If you check the network tab, the Remote Address corresponds to the LoadBalancer assigned by the Ingress Controller. In simple terms, the Ingress Controller assigns the subdomain and serves the service smoothly.

Syncing Ingress Resources with vCluster between your Virtual Clusters

After completing this tutorial, you will see that it’s a bit complicated to set up the backbone to create ingress resources. However, imagine the IT headache if you’re operating 100s of Clusters and managing all these certificates and secrets while making sure your domains are linked well. Sounds Challenging?

vCluster saves you from these and many similar problems. What vCluster does is create virtual clusters on top of your host clusters in specific namespaces. This will help you create a virtual cluster for each of your teams with full admin privileges in seconds instead of the hours it takes to create a normal cluster and configure all the required resources.

You just define a virtual cluster and what host cluster features you want in your virtual cluster. For example, if we want to ingressClasses from our Host Cluster and sync ingresses back to our host cluster from the virtual cluster to back to your host it’s as simple as defying it in the vcluster.yaml file like below:

sync:
  fromHost:
    ingressClasses:
      enabled: true
  toHost:
    ingresses:
      enabled: true

With this, whatever application you’re running on your virtual cluster can request and run an ingress smoothly. Let’s see a demo but the first step is to create the virtual cluster using the vCluster CLI [Installation Step]:

 vcluster create my-vcluster -f vcluster.yaml

Success will look something like this:

11:35:20 done Successfully created virtual cluster my-vcluster in namespace vcluster-my-vcluster

With that, make sure that you’re in the** virtual cluster context** and you can apply the same application again with a new subdomain path:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: game-2048
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-2048
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-2048
  template:
    metadata:
      labels:
        app: game-2048
    spec:
      containers:
        - name: backend
          image: alexwhen/docker-2048
          ports:
            - name: http
              containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: game-2048-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: game1.hrittikhere.live
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: game-2048
                port:
                  number: 80
  tls:
  - hosts:
    - game1.hrittikhere.live
    secretName: letsencrypt-prod
EOF

The success will look like this again when you have three things configured:

service/game-2048 created
deployment.apps/game-2048 created
ingress.networking.k8s.io/game-2048-ingress created

On the CLI, if you do kubectl get ingress you will see a new ingress resource created with the new specified host: \

Now, if you check game1.hrittikhere.live, you will find a similar game running alongside. With just one vCluster configuration, you can create multiple new clusters, and all of them will have ingress functioning seamlessly.

The IP is also again the same as your host cluster, ensuring all the networking is working smoothly:

Clean Up

With everything working the cleaning steps are very simple. You can remove the DNS Record from your service provider through the UI by clicking on delete record and for the GKE Cluster use the following command with your own parameters:

gcloud container clusters delete test  --project=hrittik-project --zone asia-southeast1-c

Successful deletion will look something like this:

With that, you have cleared all the resources, including your virtual clusters.

Final Thoughts

With this guide, we’ve explored how to configure an Ingress Controller using NGINX in a GKE (Google Kubernetes Engine) environment and to set up TLS certificates with Cert-Manager. We’ve also seen how to configure DNS A Records to point to a Kubernetes cluster’s external IP and route traffic efficiently to our services.

In addition to all of this, you’ve learned how powerful vCluster can be, especially when dealing with multi-tenant Kubernetes environments. If you're managing several teams or clusters, using vCluster allows you to create isolated, lightweight Kubernetes clusters (virtual clusters) within a larger host cluster, providing the flexibility of separate clusters without needing to pay for the control plane cost of each cluster.

More Questions? Join our Slack to talk to the team behind vCluster!

Pull request testing on Kubernetes: vCluster for isolation and costs control

Nicolas Fränkel — Thu, 27 Feb 2025 09:02:00 +0000

This week's post is the third and final in my series about running tests on Kubernetes for each pull request. In the first post, I described the app and how to test locally using Testcontainers and in a GitHub workflow. The second post focused on setting up the target environment and running end-to-end tests on Kubernetes.

I concluded the latter by mentioning a significant quandary. Creating a dedicated cluster for each workflow significantly impacts the time it takes to run. On GKE, it took between 5 and 7 minutes to spin off a new cluster. If you create a GKE instance upstream, you face two issues:

Since the instance is always up, it raises costs. While they are reasonable, they may become a decision factor if you are already struggling. In any case, we can leverage the built-in Cloud autoscaler. Also, note that the costs mainly come from the workloads; the control plane costs are marginal.
Worse, some changes affect the whole cluster, e.g., CRD version changes. CRDs are cluster-wide resources. In this case, we need a dedicated cluster to avoid incompatible changes. From an engineering point of view, it requires identifying which PR can run on a shared cluster and which one needs a dedicated one. Such complexity hinders the delivery speed.

In this post, I'll show how to benefit from the best of both worlds with vCluster: a single cluster with testing from each PR in complete isolation from others.

Virtual clusters are fully functional Kubernetes clusters nested inside a physical host cluster providing better isolation and flexibility to support multi-tenancy. Multiple teams can operate independently within the same physical infrastructure while minimizing conflicts, maximizing autonomy, and reducing costs.

-- What are virtual clusters?

With virtual clusters, we can have our cake—a single physical cluster for limited costs—and eat it with fully isolated virtual clusters.

Weaving vCluster into the GitHub workflow

Weaving vCluster into the GitHub workflow is a three-step process:

Install vCluster
Create a virtual cluster
Connect to the virtual cluster

      - name: Install vCluster
        uses: loft-sh/setup-vcluster@main                                    #1
        with:
          kubectl-install: false
      - name: Create a vCluster
        id: vcluster                                                         #2
        run: time vcluster create vcluster-pipeline-${{github.run_id}}       #3
      - name: Connect to the vCluster
        run: vcluster connect vcluster-pipeline-${{github.run_id}}           #4

Install vCluster. By default, the action installs the latest available version. You can override it.
Step IDs are not necessary unless you want to reference them in later steps. We are going to need it
Create the virtual cluster. To avoid collisions, we name it with the workflow name suffixed with the GitHub run ID
Connect to the virtual cluster

The output is along the following lines:

> Run time vcluster create vcluster-pipeline-12632713145

12:44:13 info Creating namespace vcluster-vcluster-pipeline-12632713145
12:44:13 info Create vcluster vcluster-pipeline-12632713145...
12:44:13 info execute command: helm upgrade vcluster-pipeline-12632713145 /tmp/vcluster-0.22.0.tgz-2721862840 --create-namespace --kubeconfig /tmp/3273578530 --namespace vcluster-vcluster-pipeline-12632713145 --install --repository-config='' --values /tmp/3458157332
12:44:19 done Successfully created virtual cluster vcluster-pipeline-12632713145 in namespace vcluster-vcluster-pipeline-12632713145
12:44:23 info Waiting for vcluster to come up...
12:44:35 info vcluster is waiting, because vcluster pod vcluster-pipeline-12632713145-0 has status: Init:1/3
12:45:03 done vCluster is up and running
12:45:04 info Starting background proxy container...
12:45:11 done Switched active kube context to vcluster_vcluster-pipeline-12632713145_vcluster-vcluster-pipeline-12632713145_gke_vcluster-pipeline_europe-west9_minimal-cluster
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

real    1m2.947s
user    0m0.828s
sys 0m0.187s

> Run vcluster connect vcluster-pipeline-12632713145

12:45:13 done vCluster is up and running
12:45:13 info Starting background proxy container...
12:45:16 done Switched active kube context to vcluster_vcluster-pipeline-12632713145_vcluster-vcluster-pipeline-12632713145_gke_vcluster-pipeline_europe-west9_minimal-cluster
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

For fairness' sake, I used the time command to measure the creation time of a virtual cluster precisely. I measure other steps by looking at the GitHub workflow log.

Installing vCluster and connecting to the virtual cluster take around one second. The creation of a virtual cluster takes about one minute; the creation of a full-fledged GKE instance takes at least five times more.

Changes to the workflow

Here comes the great news: there's absolutely no change to any of the workflow steps. We can keep using the same steps because a virtual cluster has the same interface as a regular Kubernetes cluster.

It includes:

Installing the PostgreSQL Helm Chart
Creating the PostgreSQL connection parameters ConfigMap
Creating the GitHub registry Secret
Applying the Kustomized manifest
And retrieving the external IP from the LoadBalancer!

If you are already using Kubernetes, and you probably are because you read this post, introducing vCluster in our daily work does not require any breaking changes.

Cleaning up

So far, we haven't cleaned up any objects we created. It means pods with our app and PostgreSQL keep piling up in the cluster, not to mention Service objects, making available ports a scarce resource. It was not an oversight: the reason was that it was a lot of overload to delete each object individually. I could have deployed all objects of a workflow run into a dedicated namespace and deleted that namespace. Unfortunately, I've been bitten by namespaces stuck in the Terminating state before.

On the opposite, deleting a virtual cluster is a breeze. Let's add the last step to our workflow definition:

      - name: Delete the vCluster
        run: vcluster delete vcluster-pipeline-${{github.run_id}}

There still is one issue: if a step of a GitHub workflow fails, i.e., returns a non-0 exit code, the job fails immediately, and GitHub skips executing subsequent steps. Hence, the above cleanup won't happen if the end-to-end tests fail. For example, it might be on purpose to keep the cluster's state if things go wrong. In this case, you should rely on observability instead for this purpose, like you do in production. I encourage you to delete your environment in every case.

GitHub provides an if attribute to run a step depending on conditions. For example, it offers a if: always(); GitHub runs the step regardless of the success or failure of previous steps. It would be redundant since we don't want to delete the virtual cluster unless it has been created in a prior step. We should delete it only if the creation is successful:

      - name: Delete the vCluster
        if: ${{ !cancelled() && steps.vcluster.conclusion == 'success' }}    #1
        run: vcluster delete vcluster-pipeline-${{github.run_id}}

Run if the job wasn't canceled and if the vcluster step (defined above) was successful. The job cancellation guard isn't necessary, but it allows you to keep the cluster up anyway.

The above setup allows each Pull Request to run in its sandbox, avoiding conflicts while controlling costs. By leveraging this approach, you can simplify your workflows, reduce risks, and focus on delivering features without worrying about breaking shared environments.

Conclusion

This post concludes our series on testing Pull Requests on Kubernetes. In the first post, we ran unit tests with Testcontainers locally and set up the foundations of the GitHub workflow. We also leveraged GitHub Service Containers in our pipeline. In the second post, we created a GKE instance, deployed our app and its PostgreSQL database, got the Service URL, and ran the end-to-end tests. In this post, we used vCluster to isolate each PR and manage the costs.

While I couldn't cover every possible option, the series provides a solid foundation for starting your journey on end-to-end testing PRs on Kubernetes.

The complete source code for this post can be found on GitHub:

ajavageek / vcluster-pipeline

To go further:

Remote Development made simple with DevPod

Nicolas Fränkel — Thu, 06 Feb 2025 09:02:00 +0000

I come relatively late to the subject of Remote Development Environments (also known as Cloud Development Environments). The main reason is that I haven't worked in a development team for over six years. However, I'm now working for Loft Labs, and we have a RDE product: DevPod. I wanted to understand our value proposition as I'll be at FOSDEM operating the DevPod booth.

The problem

As a former developer, I vividly remember the pain of setting up each developer's development environment. At the beginning of my career, the architect had to configure my development machine painfully, so it was similar to his setup. Later on, I did the same for my team members repeatedly. The scope of possible discrepancies impacting development is virtually endless: Operating System, of course, version and flavour of the SDKs, e.g., Java's Eclipse Temurin vs SapMachine, git hooks, etc. It was sweat, toil, and blood on every project.

Over the years, I saw some interesting approaches to reproducing development environments. In the beginning, they stemmed from VMs, then from containers. I think Vagrant was the first tool that caught my attention: I attended a talk in 2012 where the speaker mentioned he used it to set up machines before his training sessions.

App architectures have evolved significantly over the years, becoming more complex and sophisticated. Years ago, chances were that the only infrastructure dependency was a SQL database. In the JVM ecosystem, we were lucky to have JDBC, an API that would work across all SQL databases. All you needed to do was write standard SQL, and you could configure the database instance at runtime. With embedded databases such as Apache Derby and H2, you didn't need a dedicated Oracle instance for each developer.

Times have changed. It's not uncommon for apps to need a SQL database, a NoSQL database, a Kafka cluster, and a few additional application services. Organizations that develop such apps are already using some container-related technology, e.g., Docker or Kubernetes, to manage this complexity.

It doesn't solve the initial issue, though: how do you align the IDE, its plugins, the SDK(s), the git hooks, and everything else? You probably guessed it from the title-Remote Development Environments.

Development Containers

In the introduction, I mentioned that RDEs are called Cloud Development Environments. The main idea behind RDEs is to store all you can in a Cloud and share it with all developers. In addition, you want them to work across the most widespread Cloud providers and the most commonly used IDEs. When such a need appears, it's time for industry actors to gather around a standard. Microsoft initiated the Development Container standard for their VS Code Remove development plugin for this exact purpose.

A development container (or dev container for short) allows you to use a container as a full-featured development environment. It can be used to run an application, to separate tools, libraries, or runtimes needed for working with a codebase, and to aid in continuous integration and testing. Dev containers can be run locally or remotely, in a private or public cloud, in a variety of supporting tools and editors.

The Development Container Specification seeks to find ways to enrich existing formats with common development specific settings, tools, and configuration while still providing a simplified, un-orchestrated single container option – so that they can be used as coding environments or for continuous integration and testing. Beyond the specification's core metadata, the spec also enables developers to quickly share and reuse container setup steps through Features and Templates.

-- What are Development Containers?

The configuration file is devcontainer.json. You can find the schema reference here. VS Code, Visual Studio, and IntelliJ products can leverage a devcontainer.json file. On the provider side, GitHub Codespaces, CodeSandbox, and DevPod suport it.

Introducing DevPod

DevPod is a solution that leverages devcontainer.json. It implements three main properties:

Open Source: No vendor lock-in. 100% free and open source built by developers for developers.
Client Only: No server side setup needed. Download the desktop app or the CLI to get started.
Unopinionated: Repeatable dev environment for any infra, any IDE, and any programming language.

DevPod is designed to be user-friendly and straightforward, making it a breeze to use. I decided to write this post because I was impressed with the product and to get my thoughts in order.

The first step is to install DevPod itself. I'm on Mac; there's a Homebrew recipe.

brew install devpod

Once installed, you can launch it from the CLI or the GUI. I favour GUIs, in the beginning, to help understand the available options.

DevPod offers providers: locations where to run the containers. The default is Docker. You can add additional providers, including Cloud Providers and Kubernetes clusters.

For this post, I'll keep Docker—I'm using OrbStack. Now, onto the meat. Let's go to the workspaces menu item. If you already have created workspaces, they should appear here. Since it's our first visit, we shall create one. Click on the btn:[Create workspace] button. Let's try one of the quickstart examples, i.e., Rust. My IDE of choice is IntelliJ IDEA, but you can choose yours. Once you've selected an image, an IDE, and a provider, click Create Workspace.

At this point, DevPod will download the image and open the project running in OrbStack in IntelliJ.

From now on, we can happily start working on our Rust project, confident that every team member uses the same Rust version.

Note that the first time you use this setup, DevPod will download the JetBrains client as well. It's a one-time download delay, though.

The same holds for Git pre-commit hooks, for example. If you prefer to develop within another IDE, select it at launch time, and you're good. When done with your day's work, stop the container. If you're running in the Cloud, it saves money. On the next day, resume the container and continue your work.

Conclusion

DevPod is a nice tool around your toolbelt that allows your development team(s) to share the same machine configuration without hassle. In this introductory blog post, I showed a small fraction of what you can do. I encourage you to leverage its power if faced with heterogeneous development environments.

To go further:

A solution to the problem of cluster-wide CRDs

Nicolas Fränkel — Thu, 19 Dec 2024 09:02:00 +0000

I'm an average Reddit user, scrolling much more than reading or interacting. Sometimes, however, a post rings a giant red bell. When I stumbled upon If you could add one feature to K8s, what would it be?, I knew the content would be worth it. The most voted answer is:

Namespace scoped CRDs

A short intro to CRDs

Kubernetes comes packed with existing objects, such as Pod, Service, DaemonSet, etc., but you can create your own: the latter are called Custom Resource Definitions. Most of the time, CRDs are paired with a custom controller called an operator. An operator subscribes to the lifecycle events of CRD(s). When you act upon a CRD by creating, updating, or deleting it, Kubernetes changes its status, and the operator gets notified. What it does depends on the nature of the CRD.

For example, the Prometheus operator subscribes to the lifecycles of a couple of different CRDs: Prometheus, Alertmanager, ServiceMonitor, etc., to make operating Prometheus easier. In particular, it will create a Prometheus instance when it detects a new Prometheus CR. It will configure the instance according to the CR's manifest.

The issue with cluster-wide CRDs

CRDs have a cluster-wide scope; that is, you install a CRD for an entire cluster. Note that while the definition is cluster-wide, the CR's scope is either Cluster or Namespaced depending on the CRD.

I noticed the problem of cluster-wide CRDs when I worked with Apache APISIX, an API gateway. Routing in Kubernetes has evolved across several steps: NodePort, LoadBalancer, and IngressController, each trying to fix the limitations of its predecessor. The latest step is the Gateway API.

At the time of this writing, the Gateway API is still an add-on and not part of the Kubernetes distro. You need to install it explicitly as a CRD. The Gateway API went through several versions. If team A were a precursor and installed version v1alpha2, every other team would need to use the same version because the Gateway API is a CRD. Of course, team B can try to convince team A to upgrade, but if you've been in such a situation, you know how painful it can be.

I mentioned above that the magic happened via an operator. The Gateway API doesn't come with an out-of-the-box operator. Instead, different vendors provide their own. For example, Apache APISIX has one, Traefik has one, etc. Of course, they are more or less advanced. At the time, the APISIX operator only worked with version 0.5.0 of the Gateway API CRD.

So now, it gets worse. Team A installed v0.5.0 to work with APISIX; team B comes later and wants to use Traefik, which fully supports the latest and greatest. Unfortunately, they can't because it would require the latest CRD.

Don't get me wrong; I'm all for a lean architectural landscape that limits the number of different technologies. However, it should be a deliberate choice, not a technical limitation. The above also prevents rolling upgrades. Imagine that we decided on Apache APISIX early on. Yet, it hasn't progressed toward supporting the latest Gateway API versions. We should be able to migrate from APISIX to Traefik (or any other) team by team.

The cluster-wide CRD doesn't allow it, or at least makes it very hard: we should find a Traefik that handles v0.5.0, if there's one and it's still maintained, migrate all APISIX CR to Traefik at once, and then proceed to upgrade. This approach requires expensive coordination, the cost of which grows exponentially with the number of teams involved.

The separate clusters approach

The obvious solution is to have one cluster per team. If you have been operating clusters, you know this approach doesn't scale.

Each cluster requires a primary node and a control plane. These are just "administrative" costs of running a cluster: they don't bring anything to the table.

On top of that, every cluster needs a complete monitoring solution. It includes at least metrics and logging, possibly distributed tracing. Whatever your architecture, it's again an additional burden with no business value. You can generalize the above over every support feature of a cluster, authentication, authorization, etc.

All in all, lots of clusters mean lots of additional operational costs.

vCluster, a sensible alternative

The ideal situation, as the initial quote of this post states, would be to have namespace-scoped CRDs. Unfortunately, it's not the path that Kubernetes chose. The next best thing would be to add a virtual cluster on top of the real one to partition it: that's the promise of vCluster.

What are virtual clusters?

Virtual clusters are a Kubernetes concept that enables isolated clusters to be run within a single physical Kubernetes cluster. Each cluster has its own API server, which makes them better isolated than namespaces and more affordable than separate Kubernetes clusters.

vCluster isolates each virtual cluster. Hence, with a single control plane, you can deploy a v1.0 CRD in one cluster and a v1.2 in another without trouble.

Imagine two teams working with different Gateway API providers, each requiring a different CRD version. Let's create a virtual cluster for each of them with vCluster so each can work independently from the other team. I'll assume you already have the vcluster CLI installed; if not, look at the documentation, for we provide a couple of different installation options depending on your platform and your tastes.

We can now create our virtual clusters.

vcluster create teamx

The output should be similar to the following:

08:01:02 info Creating namespace vcluster-teamx
08:01:02 info Detected local kubernetes cluster orbstack. Will deploy vcluster with a NodePort & sync real nodes
08:01:02 info Chart not embedded: "open chart/vcluster-0.21.1.tgz: file does not exist", pulling from helm repository.
08:01:02 info Create vcluster teamx...
08:01:02 info execute command: helm upgrade teamx https://charts.loft.sh/charts/vcluster-0.21.1.tgz --create-namespace --kubeconfig /var/folders/kb/g075x6tx36360yvwjrb1x6yr0000gn/T/83460322 --namespace vcluster-teamx --install --repository-config='' --values /var/folders/kb/g075x6tx36360yvwjrb1x6yr0000gn/T/1777816672
08:01:03 done Successfully created virtual cluster teamx in namespace vcluster-teamx
08:01:07 info Waiting for vcluster to come up...
08:01:32 done vCluster is up and running

Because we didn't specify any namespace, vcluster created one with the same name as the virtual cluster. If you prefer to set a specific namespace, use the -n option, .e.g., vcluster create mycluster -n mynamespace.

Note that you can customize each virtual cluster via a values.yaml configuration file. In the context of this post, we will keep the default options.

We use the vcluster connect command to connect to a virtual cluster. However, we are already connected because we used the vcluster create command.

At this point, it's as if we were in a separate Kubernetes cluster. Team X can install the CRDs using the version that they require.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

The output is:

customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created

Team Y can do the same with their version. Because we are both teams X and Y, we need to disconnect first from the virtual cluster.

vcluster disconnect

You should see the result of the operation:

08:05:29 info Successfully disconnected and switched back to the original context: orbstack

Let's impersonate team Y, create the virtual cluster, and install another version of the CRDs:

vcluster create teamy
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

The output of the second command is the following:

customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created

Version 1.2 has a new GRPC route that is not found in version 1.0. This way, team X can now install their Gateway API provider that works with v1.0 and team Y the one that works with 1.2.

CRDs are cluster-wide resources, but there's no conflict since the virtual clusters behave like isolated clusters. Each team can happily use the version they need without forcing others to use it.

Conclusion

In this post, we touched on the problem of some Kubernetes objects: they are cluster-wide and lock all teams working on the same cluster to use the same version. Running a Kubernetes cluster incurs costs; managing lots of them requires mature and organized automation.

vCluster allows an organization to get the best of both worlds: limit the number of clusters while preventing teams from stepping on each others' toes.

To go further:

vcluster Exploded in 2022

Lukas Gentele — Tue, 27 Dec 2022 10:02:53 +0000

By Rich Burroughs

2022 was a very exciting year for vcluster. If you’re unfamiliar with vcluster, it’s an open source tool for creating and managing virtual Kubernetes clusters. Virtual clusters are lightweight and run in a namespace of an underlying host cluster but appear to the users as if they’re full-blown clusters. If you’d like more details, there’s an extended explanation in the docs.

Let’s take a look at the massive growth of the project and some of the new features that showed up in 2022.

Highlights

First, some numbers: vcluster now has more than 2,200 stars on GitHub, and the Docker images have been downloaded more than 26 million times from Docker Hub. While I don’t put a lot of faith in GitHub stars as a metric on its own, there’s a lot of other evidence that the project has grown a lot this year.

One of the highlights for vcluster was KubeCon + CloudNativeCon 2022 North America, which was held in Detroit. vcluster was featured in a keynote by Whitney Lee and Mauricio Salatino, which illustrated how platform teams can better serve developers, as well as a talk by Joseph Sandoval and Dan Garfield about how Adobe’s new CI/CD platform that uses vcluster and Argo CD. And Mike Tougeron from Adobe spoke more about their use of vcluster at GitOps Con North America in the buildup to KubeCon.

vcluster was featured on VMware’s TGIK stream, Whitney Lee’s Elightning show, and Bret Fisher’s DevOps and Docker stream. We also saw written content about vcluster from the community, like this intro tutorial by Pavan Kumar, a blog post from Mauricio Salatino on building dev platforms with vcluster and Crossplane, and this super cool post from Jason Andress about building honeypots with vcluster. The honeypot use case had never occurred to me. I love seeing what creative uses people come up with for vcluster.

New Features

There wouldn’t be so much excitement around vcluster if not for the work of the maintainers and contributors. It’s great to see so many new features being added, and they often come out of feedback from people in the community. Here’s a look at some of the things that shipped in 2022.

Deploying Helm charts and manifests on startup: This is one of my favorite features that’s been added to vcluster. It’s one thing to provision a bare cluster and another thing for that to become a useful environment. With this feature, you can apply Helm charts (public or private) or Kubernetes YAML manifests as the virtual cluster spins up for the first time. It’s super helpful. (Read the docs here)

Distros: Initially, vcluster was built on top of k3s, but after a while, people started asking if we could support other Kubernetes distributions. Now vcluster also supports k0s, EKS, and standard Kubernetes. This allows you to use a virtual cluster more like your production environment. (Docs)

Isolated mode: As more people started using vcluster, we received lots of questions and feedback about how much isolation the virtual clusters had. With isolated mode, vcluster now adds some additional Kubernetes security features to the virtual clusters as they’re provisioned: a Pod Security Standard, a resource quota, a limit range, and a network policy. Isolated mode is optional and can be invoked with the --isolated flag. (Docs)

Plugins: With the complexity of Kubernetes, it became clear that people would need the ability to customize and extend vcluster to fit their workflows. Enter vcluster plugins and the vcluster SDK. Plugins are written in Go and allow users to customize the behavior of vcluster’s syncer to do all kinds of things, like sharing resources between host and virtual clusters. (Docs)

Pausing and resuming virtual clusters: This handy feature can scale down workloads in your virtual clusters when they’re not being used. That’s done by setting the number of replicas to zero for StatefulSets and Deployments. Resuming the virtual cluster sets the replicas back to their original value, and then the scheduler spins the pods back up. This allows users to suspend the workloads while keeping configurations in the virtual cluster in place. (Docs)

This is just a handful of the many improvements made to vcluster in 2022.

Thank you

I want to thank the vcluster maintainers, the contributors, and everyone who used vcluster in 2022. As I mentioned, many great ideas for improving the project come from folks in the community through GitHub issues, pull requests, or even feedback in our community Slack.

We’re excited that so many people have found vcluster useful for their work. If you’ve read this far and haven’t tried vcluster yet yourself, there’s an easy quickstart that takes just a few minutes.

I’m looking forward to seeing how the project grows in 2023.

Kubernetes Multitenancy: Why Namespaces aren’t Good Enough

Lukas Gentele — Mon, 21 Nov 2022 19:32:30 +0000

By Jason Bloomberg

Multitenancy has long been a core capability of cloud computing, and indeed, for virtualization in general.
With multitenancy, everybody has their own sandbox to play in, isolated from everybody else’s sandbox, even though beneath the covers, they share common infrastructure.
Kubernetes offers its own kind of multitenancy as well, via the use of namespaces. Namespaces provide a mechanism for organizing clusters into virtual sub-clusters that serve as logically separated tenants.
Relying upon namespaces to provide all the advantages of true multitenancy, however, is a mistake. Namespaces are for cloud native teams that don’t want to step on each other’s toes – but who are all colleagues who trust each other.
True multitenancy, in contrast, isn’t for colleagues. It’s for strangers – where no one knows whether the owner of the next tenant over is up to no good. Kubernetes namespaces don’t provide this level of multitenancy.

Multitenancy: A Quick Primer

Multitenancy is most familiar as a core part of how cloud computing operates.
Everybody’s cloud account is its own separate world, complete in itself and separate from everybody else’s. You get your own login, configuration, services, and environments. Meanwhile, everybody else has the same experience, even though under the covers, the cloud provider runs each of these tenants on shared infrastructure.

There are different flavors of multitenancy, depending upon just what infrastructure they share beneath this abstraction.

IaaS tenants, aka instances or nodes, share hypervisors that abstract the underlying hardware and physical networking. Meanwhile, SaaS tenants, for example, Salesforce or ServiceNow accounts, might share database infrastructure, common services, or other application elements.

Either way, each tenant is isolated from all the others. Isolation, in fact, is one of the most important characteristics of multitenancy, because it protects one tenant from the actions of another.

To be effective, the infrastructure must enforce isolation at the network layer. Any network traffic from one tenant that is destined for another must leave the tenant via its public interfaces, traverse the network external to the tenants, and then enter the destination tenant through the same interface that any other external traffic would enter.

Even when the infrastructure provider decides to offer a shortcut for such traffic, avoiding the hairpin to improve performance, it’s important for such shortcuts to comply with the same network security policies as any other traffic.

Using Kubernetes Namespaces for Multitenancy

Namespaces have been around for years, largely as a way to keep different developers working on the same project from inadvertently declaring identical variable or method names.

Providing a scope for names is also a benefit of Kubernetes namespaces, but naming alone isn’t their entire purpose. Kubernetes namespaces are also virtual clusters within the name physical cluster.

This notion of virtual clusters sounds like individual tenants in a multitenant cluster, but in the case of Kubernetes namespaces, they have markedly different properties.

Kubernetes logically separates namespaces within a cluster but allows for them to communicate with each other within the cluster. By default, Kubernetes doesn’t offer any security for such interactions, although it does allow for role-based access control (RBAC) in order to limit users and processes to individual namespaces.

Such RBAC, however, does not provide the network isolation that is essential to true multitenancy. Furthermore, Kubernetes doesn’t implement any privilege separation, instead delegating such control to a dedicated authorization plugin.

Furthermore, Kubernetes defines cluster roles and their associated bindings, thus empowering certain individuals to have access and control over all the namespaces within the cluster. Not only do such roles open the door for insider attacks, but they also allow for misconfigurations of the cluster roles that would leave the door open between namespaces.

If cluster roles weren’t bad enough, Kubernetes also allows for privileged pods within a cluster. Depending upon how admins have configured such pods, they can access any node-level capabilities for the node hosting the cluster. For example, the privileged pod might be able to access file system, network, or Linux process capabilities.

In other words, a privileged pod can impersonate the node that hosts its cluster – regardless of what namespaces run on that cluster or how they’re configured.

‘True’ Kubernetes Multitenancy that Provides Isolation between Tenants

In order to implement secure Kubernetes multitenancy, it’s essential to use a tool like Loft Labs to implement virtual clusters within Kubernetes clusters that act just like real clusters.

With this ‘true’ multitenancy, traffic from one virtual cluster to another must go through the same access controls as any cluster-to-cluster traffic would – because fundamentally, Loft Labs handles traffic between virtual clusters just as Kubernetes would handle traffic between clusters.

One of the primary benefits of this approach to Kubernetes multitenancy is that virtual clusters support namespaces just as clusters do – not necessarily for isolation (as namespace isolation is inadequate), but for the name scoping that namespaces are most familiar for.

Loft Labs’ multitenancy provides other benefits that namespaces cannot, for example, the ability to spin down individual virtual clusters for better cost efficiency.

The Intellyx Take

The best way to think about multitenancy options for Kubernetes is this: namespaces are for friends, while Loft Labs’ multitenancy is also for strangers.

Using namespaces for multitenancy works best when securing traffic between tenants is a non-issue, say, when all the developers using the cluster are on the same team and actively collaborating.

True multitenancy of the sort Loft provides, in contrast, provides virtual clusters that separate teams can use – even if those teams aren’t collaborating, or indeed, don’t know or trust each other at all.

This zero-trust approach to sharing resources is fundamental to modern cloud native computing, even in situations where people are all working for the same company.

Not only does such isolation add security, but it also enforces GitOps-style best practices for how multiple teams can work on the same codebase in parallel.

Making Self-Service Clusters Ready for DevOps Adoption

Lukas Gentele — Mon, 21 Nov 2022 19:32:26 +0000

By Jason English

History is littered with cautionary tales of software delivery tools that were technically ahead of their time, yet were ultimately unsuccessful because of a lack of end user adoption.

In the past, the success of developer tooling vendors depended upon the rise and fall of the major competitive platforms around them. An upstart vendor could still break through to grab market share from dominant players in a space by delivering a superior user experience (or UX) and partnering with a leader, until such time as they were acquired.

A great UX generally includes an intuitive UI design based on human factors, especially important in consumer-facing applications. Human factors are still important in software development tooling, however, the UX focus is on whether the tools will readily deliver value to the organization, by empowering developers to efficiently deliver better software.

Kubernetes (or k8s) arose from the open source foundations of Linux, containerization, and a contributed project from Google. A global community of contributors turned the enterprise space inside out, by abstracting away the details of deploying and managing infrastructure as code.

Finally, development and operations teams could freely download non-proprietary tooling and orchestrate highly scalable cloud native software architecture. So what was holding early K8s adopters back from widespread use in their DevOps lifecycles?

The challenge: empowering developers

A core tenet of the DevOps movement is self-service automation. Key stakeholders should be fully empowered to collaborate freely with access to the tools and resources they need.

Instead of provisioning through the approval process of an IT administrative control board, DevOps encourages the establishment of an agile platform team (in smaller companies, this may be one platform manager). The platform team should provide developers with a self-service stack of approved on-demand tooling and environments, without requesting an exhaustive procurement process or ITIL review cycles.

At first glance, Kubernetes, with its declarative abstraction of infrastructure, seems like a perfect fit for orchestrating these environments. But much like an early sci-fi spaceship where wires are left hanging behind the lights of control panels, many specifics of integration, data movement, networking and security were intentionally left up to the open source community to build out, rather than locking in design assumptions in these key areas.

Because the creation and configuration of Kubernetes clusters comes with a unique set of difficulties, the platform team may try to reduce rework by offering a one-size-fits-all approach. Sometimes, this may not meet the needs of all developers, and may exceed the needs of other teams with excess allocation and cloud cost.

You can easily tell if an organization’s DevOps initiative is off track if it simply shifts the provisioning bottleneck from IT to a platform team that is backlogged and struggling to deploy k8s clusters for the right people at the correct specifications.

Handing over the keys

The ability to usurp the limitations of physical networks and IP addressing is the secret weapon of Kubernetes. With configuration defined as code, teams can call for namespaces and clusters that truly fit the semantics and dimensions of the application.

The inherent flexibility of k8s produces an additional set of concerns around role-based access controls (RBAC) that must be solved in order to scale without undue risk.

In today’s cloudy and distributed ecosystem, engineering organizations are composed differently than the siloed Dev and Ops teams in traditional IT organizations. Various teams may need to access certain clusters or pods within as part of their developmental or operational duties on specific projects.

Even with automated provisioning, a request would by default generate a cluster with one ‘front door’ key for an administrator, who may share this key among project team members. Permissioned individuals can step on each other’s work in the environment, inadvertently break the cluster, or even allow their credentials to get exposed to the outside world.

To accelerate delivery without risk, least-privilege rights should be built into the provisioning system by policy and leverage the company’s single-sign-on (SSO) backend for resource access across an entire domain, rather than being manually doled out by an admin.

In a self-service solution, multiple people can get their own keys with access to specific clusters and pods, or assign other team members to get them. These permissions can lean on the organization’s authorization tools of choice for access control, without requiring admins to write any custom policies to prevent inadvertent conflicts.

A self-service Kubernetes storefront

We already know the cost of not getting self-service right. Unsatisfied developers will sneak around procurement to provision their own rogue clusters, creating costly cloud sprawl and lots of lost and forgotten systems with possible vulnerabilities.

As consumers, we’re acclimated to using e-commerce websites and app stores on our personal devices. At work, we can use a credit card to buy apps, plugins and tooling from marketplaces provided by a SaaS vendor or public cloud.

The storefront model offers a good paradigm for self-service cluster provisioning. One vendor, Loft Labs, offers a Kubernetes control plane built upon the open source DevSpace tool for standing up stacks. An intuitive interface allows domain-level administrators to navigate automated deployments and track usage.

More importantly, developers can use their own filtered view of Loft Labs as a storefront for provisioning all available and approved K8s cluster images into new or existing namespaces. Or they can make the provisioning requests via a CLI and drill down into each cluster’s details with the kubectl prompt.

The system provides guardrails for developers to provision Kubernetes clusters and namespaces in the mode they prefer, without consuming excess resources or making configuration mistakes.

The Intellyx Take

Quite a few vendors are already offering comprehensive ‘Kubernetes-as-a-Service’ management platforms that gloss over much of the complexity of provisioning and access to clusters, when what is really needed is transparency and portability.

Engineers will avoid waiting on procurement boards, and they hate writing repetitive commands, whether that is launching 100 pods at a time for autoscaling or bringing them down when they are no longer required. But they do still want to directly address kubectl for a single pod, look at the logs for that pod and analyze what is going on.

The platform team’s holy grail is to provide a self-service Kubernetes storefront that works with the company’s authorization regimes to entitle the right users and allow project management, tracking and auditing, while giving experienced engineers the engineering interfaces they need.

Next up in this series, we’ll be covering the challenges of multi-tenancy and cost control!

\
*© 2022, Intellyx, LLC. Intellyx is solely responsible for the content of this article. At the time of writing, Loft Labs is an Intellyx customer. Image sources: Maps, Unsplash. Screenshot, Loft Labs.*

Interview with KubeCon Keynote Speaker Mauricio Salatino from VMware

Lukas Gentele — Thu, 27 Oct 2022 13:16:49 +0000

By Rich Burroughs

Loft Labs is excited to welcome Kubecon North America 2022 keynote speaker Mauricio “Salaboy” Salatino for an exclusive interview where we dive into the struggles facing platform engineers with the CNCF ecosystem. Mauricio and his co-speaker Whitney Lee will be presenting a demo for their keynote presentation focused on the provisioning of virtual clusters with Crossplane, vcluster, and Knative, to develop an internal development platform.

Rich: How did you learn about vcluster?

Mauricio: I had heard about vcluster in the context of multi-tenancy. While delivering training for LearnK8s and working for different companies, I’ve repeatedly seen teams struggling to answer a very simple question: One cluster or multiple clusters? I’ve seen teams starting simple with namespaces inside a single Kubernetes Cluster and then struggling to move to use multiple clusters when the isolation levels of namespaces are not enough. And that is precisely where I see vcluster as a better alternative because it provides, from the get-go, the separation into different Kubernetes API Servers. In today’s world, where every organization is building its internal development platform, I see tools like vcluster as critical components of these platforms.

Rich: In the demo for your KubeCon keynote you provisioned virtual clusters with vcluster and Crossplane. Can you explain how that works? And what’s your experience been like using those two tools together?

Mauricio: The demo presented at the KubeCon keynote session uses Crossplane, vcluster, and Knative, all working together around this concept of building internal development platforms. Crossplane is used to abstract where cloud resources are created. With Crossplane, we can declaratively create Kubernetes clusters in all major cloud providers, but creating these clusters is expensive, and it is a process that takes time. This is where using vcluster can save you time and money. Because, to my surprise, creating a vcluster is just installing a Helm chart into your existing cluster. We can use the Crossplane Helm Provider to create vcluster from inside a Crossplane Composition, and that is exactly what my demo is doing. But vcluster doesn’t stop there, because with vcluster Plugins you can share tools between the host cluster and the virtual clusters. The demo shows how you can enable your vclusters with tools like Knative Serving (for dynamic autoscaling and advanced traffic management) without installing Knative Serving in each cluster. In a scenario where you have 10 teams all working in different vclusters and using tools like Knative Serving you save on installing and running 9 Knative Serving installations.

Rich: Your demo focused on provisioning environments for developers. What do you think makes vcluster a great tool for dev environments?

Mauricio: vclusters are better than namespaces because they provide more isolation and are cheaper than fully fledged clusters. Suppose you have developers working with cluster-wide resources such as CRDs and tools that need to be installed to do their work. In that case, vclusters will give them the freedom to work against a dedicated Kubernetes API Server, where they will have total freedom to do what they need. By using vcluster you can give your development teams full access to their dedicated API servers without the need of creating, maintaining and paying for full-fledged Kubernetes Control Planes.

Rich: There are so many new tools in the Kubernetes space and more arriving all the time. The CNCF landscape continues to grow. What do you look for when you evaluate new open source, cloud native tools?

Mauricio: I evaluate their (project) community, and in the last two years, I’ve been very interested in tools that fit into the story of building internal developer platforms, as I’ve been in that space for a long time. Most of my work in the open source space is around helping developers to be more productive in building their software.

If you are evaluating Open Source / CNCF projects, check their maturity level, who (which company or companies) are sponsoring the project, and how healthy their community is. Looking at which companies are active in their community forums or slack channels is a great validation to see if other companies in your industry are trying to solve the same problems that you are facing.

Rich: There’s been a lot of focus on the role of platform engineer the last few years. What do you think are the big challenges facing platform engineers today?

Mauricio: I’ve been writing about these topics lately in my blog, you can read more about this here -> The challenges of building platforms on top of Kubernetes Part 1, Part 2, Part 3, Part 4

But the big challenge nowadays is to keep up with all that is happening in the Cloud Native space, so you need to look for tools that:

Be ready to pivot: look for tools that provide you with the right abstractions to be able to pivot if things change, Crossplane is a big player in this space, but there are other projects that are worth looking at. If you are really into platform building you should check a project that I am hoping gets donated to the CNCF called Kratix. These folks are working on providing the right abstractions to build platforms allowing platform teams to focus on deciding which projects they want to use and how those projects will work together.
Reuse instead of build: try to identify the problem that you are trying to solve into two different buckets: 1) Generic problem that every company will have 2) very specific challenge that is specific to your company. If you are looking for tools to solve problems in the first bucket, then make sure that you don’t build an in-house solution. If you are looking for tools to solve problems in the second bucket you need to focus your search on a combination of an existing tools can do the work to make sure that you don’t spend time and effort reinventing the wheel just because the tools available doesn’t match 100% your requirements.
Ecosystem integrations: When you look at a specific tool make sure that it integrates well with the other tools in the ecosystem. Don’t be tricked by the fact that they are all Kubernetes projects. Depending on how you want tools to work together your platform team might need to spend a considerable amount of time to make these tools work for your specific use case.
Tailored Developer Experiences are the best way to promote your platform: you need to spend a lot of time and effort into building amazing developer experiences for your teams to have the right tools to do their work. These developer experiences are enabled by the tools that the platform team chooses to use and by always keep improving how application development teams interact with the platform.

If you are into building platforms, I am currently writing a book titled ”Continuous Delivery for Kubernetes” where I cover tools like Tekton, Crossplane, vcluster, Keptn, Knative, ArgoCD, Helm, among others, to build platforms that are focused on delivering more software more efficiently.

Rich: Thanks for your time Mauricio.

If you want to learn more about vcluster, check our vcluster.com for links to the docs, the GitHub repo, and our community Slack. You can find Mauricio on Twitter at @salaboy, and Rich at @richburroughs.