Cluster-level multitenancy with vCluster

What are they?

Virtual clusters are fully working Kubernetes clusters that run on top of other Kubernetes clusters. Compared to fully separate "real" clusters, virtual clusters reuse worker nodes and networking of the host cluster. They have their own control plane and schedule all workloads into a single namespace of the host cluster. Like virtual machines, virtual clusters partition a single physical cluster into multiple separate ones.
Official documentation

Why do we need them?

To partition the cluster into multiple virtual clusters which can be provided to the tenant teams for isolation. They provide following advantages:

• Ease of use - each tenant gets a cluster of their own!
• Customization - ability to install and use (and test) different cluster level resources (CRDs)
• Isolation - much stricter isolation than traditional namespace based multi tenancy
• Manageability - operators need not handle (too) many clusters
• Lightweight and full-fledged - based on the popular k3s distro and sqlite for DB (instead of etcd)

---

Architecture

---

How to use them

1. Download vcluster cli
   brew install vcluster

2. Install the cluster under a namespace team-abc
   

❯ time vcluster create -n team-abc vcluster-abc
info   Detected local kubernetes cluster kind. Will deploy vcluster with a NodePort & sync real nodes
info   Create vcluster vcluster-abc...
done √ Successfully created virtual cluster vcluster-abc in namespace team-abc
info   Waiting for vcluster to come up...
warn   vcluster is waiting, because vcluster pod vcluster-abc-0 has status: ContainerCreating
warn   vcluster is waiting, because vcluster pod vcluster-abc-0 has status: ContainerCreating
warn   vcluster is waiting, because vcluster pod vcluster-abc-0 has 
...
info   Starting proxy container...
done √ Switched active kube context to vcluster_vcluster-abc_team-abc_kind-macbook
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
vcluster create -n team-abc vcluster-abc  0.57s user 0.49s system 0% cpu 2:18.09 total

1. Check the new cluster

❯ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:10754
CoreDNS is running at https://127.0.0.1:10754/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
❯ kubectl ns
default
kube-system
kube-public
kube-node-lease

1. Deploy workloads

❯ kubectl create ns test-ns
namespace/test-ns created
❯ kubectl create deployment test-dep --image=nginx --replicas=2 -n test-ns
deployment.apps/test-dep created
❯ kubectl get all -n test-ns
NAME                            READY   STATUS              RESTARTS   AGE
pod/test-dep-574f5c6754-x4l45   0/1     ContainerCreating   0          51s
pod/test-dep-574f5c6754-7tlr5   0/1     ContainerCreating   0          51s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test-dep   0/2     2            0           51s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/test-dep-574f5c6754   2         2         0       51s

1. Check back from host cluster

❯ vcluster disconnect
❯ k get pods -n team-abc
NAME                                                    READY   STATUS    RESTARTS   AGE
coredns-5df468b6b7-dtmn6-x-kube-system-x-vcluster-abc   1/1     Running   0          7m38s
test-dep-574f5c6754-7tlr5-x-test-ns-x-vcluster-abc      1/1     Running   0          2m47s
test-dep-574f5c6754-x4l45-x-test-ns-x-vcluster-abc      1/1     Running   0          2m47s
vcluster-abc-0                                          2/2     Running   0          9m30s

House keeping

❯ vcluster list
 NAME           NAMESPACE   STATUS    CONNECTED   CREATED                         AGE
 vcluster-abc   team-abc    Running               2022-06-26 21:38:38 +0530 IST   10m44s
❯ vcluster delete vcluster-abc
info   Stopping docker proxy...
info   Delete vcluster vcluster-abc...
done √ Successfully deleted virtual cluster vcluster-abc in namespace team-abc
done √ Successfully deleted virtual cluster pvc data-vcluster-abc-0 in namespace team-abc

Get the kubeconfig of a vcluster

❯ vcluster list
 NAME           NAMESPACE   STATUS    CONNECTED   CREATED                         AGE
 vcluster-abc   team-abc    Running   True        2022-06-26 21:51:11 +0530 IST   5m29s
❯ vcluster disconnect
# Look for a secret named vc-<cluster-name> in the <namespace> used
❯ kubectl get secret vc-vcluster-abc -n team-abc --template={{.data.config}} | base64 -D
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTlRZeU5qQTFNalV3SGhjTk1qSXdOakkyTVRZeU1qQTFXaGNOTXpJd05qSXpNVFl5TWpBMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTlRZeU5qQTFNalV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTVThuZGdKaWlNTFFHTEI0VmxSSjlQNm9LdTdGbkU4ZTM2OXl0eDhUYnUKeFJxNTg3NVE4SE1zRU9SMjl4a2puUmN5dUNrcVpOTTRHeDYzLzM1bW5TSjBvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVU5PKzlHb1M5Z1lib0NMWjU3ck9GCmdndFFDZlF3Q2dZSUtv=
...

TODO

There are a lot more features to be explored and will do in another post

• pause/resume
• applying manifests on init
• passing down limits and quotas
• extensions/plugins
• using different distributions

What is impressive

• Tailor made for testing CRDs
• Probably well suited for CI use-cases to create clusters on the fly
• Very useful in creating throw-away clusters (for DEV/R&D purposes)