Welcome to the launch of my new blog series, "Into The Depths of Kubernetes," where we are peeling back the abstraction layers to explore the architectural patterns, runtime mechanics, and production realities that truly matter for software and platform engineers. Whether you are scaling microservices, optimizing infrastructure costs, or hardening cluster security, this series is designed to give you actionable insights and deep technical clarity. We are kicking things off with a fundamental challenge every growing organization faces: Multi-Tenancy in Kubernetes. In this inaugural post, we will dive into isolating workloads, managing shared cluster resources, enforcing strict security boundary conditions with Namespaces and Network Policies, and balancing cost efficiency against robust tenant isolation.
Part 1 — Exploding the Control Plane: Concurrency, CRD Collisions, and Virtual Slicing
If you have spent enough time operating Kubernetes, you have probably heard some variation of the same advice whenever the conversation turns toward multi-tenancy: “Just give every team its own namespace.” It sounds reasonable because namespaces are one of the most visible boundaries Kubernetes gives us. Team A gets team-a, Team B gets team-b, RBAC prevents them from touching each other's workloads, ResourceQuotas limit how much they can consume, and NetworkPolicies can restrict communication between them. On a whiteboard, it looks like isolation. In a production cluster, however, that picture is incomplete. A namespace is a logical boundary, not an independent Kubernetes environment, and that distinction becomes extremely important as soon as one tenant becomes noisy, misconfigured, or simply operates at a scale that was never anticipated.
A namespace does not create a new API server. It does not create a new etcd instance. It does not create an independent controller-manager. It does not create a separate CRD registry, and it certainly does not give a tenant its own Linux kernel. Multiple tenants can therefore appear isolated from the perspective of their Kubernetes objects while continuing to compete for several of the same underlying control-plane resources. A badly behaved internal platform team does not need to be malicious to create problems for everyone else; a runaway reconciliation loop, an aggressive CI system, an incorrectly configured operator, or a simple shell script repeatedly querying the API can generate enough pressure to make the shared control plane everyone's problem.
This is where Kubernetes multi-tenancy becomes much more interesting than creating namespaces and writing a few RBAC rules. In this first part of Into The Depths of Kubernetes, we're going below the namespace abstraction and looking at the control plane itself: how API requests compete for finite processing capacity, how API Priority and Fairness prevents one class of traffic from monopolizing that capacity, why cluster-scoped CRDs create a very different type of tenant collision, and why virtual control planes such as vCluster exist when logical isolation starts reaching its limits.
The Namespace Illusion
Let's start with the architecture most teams actually deploy. From the application team's perspective, namespaces provide a clean way to divide ownership: one team works inside team-a, another works inside team-b, and a third operates inside team-c. Kubernetes then gives administrators familiar mechanisms such as RBAC, ResourceQuotas and NetworkPolicies to control what each team can access or consume. The problem is that these mechanisms sit above a number of shared control-plane components. The namespace changes the scope of many Kubernetes objects, but it doesn't transform the underlying cluster into three independent control planes.
Kubernetes Cluster
│
┌──────────┴──────────┐
│ kube-apiserver │
└──────────┬──────────┘
│
┌────────────────┼────────────────┐
│ │ │
Team A Team B Team C
namespace namespace namespace
│ │ │
Pods Pods Pods
This is the first mental model I want to establish: namespaces provide logical isolation, but they do not magically partition the Kubernetes control plane. That difference matters because isolation is not one problem. There are several different resources and trust boundaries involved, including namespace objects, authorization, network traffic, API-server concurrency, cluster-scoped APIs, worker nodes, operating-system resources, and the control plane itself. A namespace addresses some of these concerns extremely well, but expecting it to solve all of them is where multi-tenant Kubernetes architectures usually start to become fragile.
Consider what happens when Tenant A begins generating an unusually high volume of API traffic. Tenant A may have no permission to access Tenant B's objects, and its pods may be completely isolated from Tenant B's workloads, but both tenants still depend on the same API-server infrastructure. The request originating from Tenant A therefore doesn't need permission to touch Tenant B in order to affect Tenant B. It only needs to consume enough shared control-plane capacity that requests from Tenant B start waiting longer. That is the essence of a noisy-neighbor problem at the API layer.
The API Server Is a Shared Resource
Let's look at an intentionally simple example:
while true; do
kubectl get pods -A
done
One shell loop isn't particularly interesting. Now imagine that command being executed by a CI job, a badly written automation script, or hundreds of processes at the same time. The individual kubectl operation may be perfectly valid, and Kubernetes may happily authorize every request, but the fact that the request is valid does not mean the API server has infinite capacity to process it. Every API operation still travels through authentication, authorization, request classification, admission where applicable, request handling, serialization and downstream processing, with some operations ultimately interacting with storage or triggering additional work elsewhere in the control plane.
The important point here is that kubectl itself isn't dangerous. The problem is request volume and concurrency. A Kubernetes API server is a highly concurrent Go application, and requests are handled using goroutines and a collection of internal mechanisms rather than a simplistic one-thread-per-request model. Goroutines are considerably cheaper than operating-system threads, but they still consume CPU, memory, network resources and downstream capacity. More importantly, the API server operates with bounded concurrency rather than allowing an unlimited number of requests to execute simultaneously. Once enough traffic arrives, requests begin competing for that finite capacity.
Historically, Kubernetes exposed mechanisms such as:
--max-requests-inflight
--max-mutating-requests-inflight
to limit request concurrency. Modern Kubernetes uses API Priority and Fairness (APF) to provide a more structured mechanism for deciding which requests should receive concurrency and how competing request flows should be isolated. Kubernetes documents APF specifically as a way of protecting important API traffic from being overwhelmed by other traffic classes. The important engineering lesson is therefore not to think of the API server as a limitless HTTP endpoint, but as a shared control-plane resource whose processing capacity has to be deliberately managed.
One useful way to visualize the problem is this:
Tenant A
│
│ 1000 requests/sec
▼
┌─────────────────────┐
│ kube-apiserver │
│ │
│ concurrency = C │
└─────────┬───────────┘
│
▼
processing
Now add four other teams:
Tenant A ─────┐
Tenant B ─────┤
Tenant C ─────┼──► kube-apiserver ──► etcd/controllers
Tenant D ─────┤
Tenant E ─────┘
The problem becomes obvious. If Tenant A generates enough traffic, the effect is no longer confined to Tenant A. Tenant B might still have plenty of CPU and memory allocated to its workloads, and its pods may all be healthy, but its requests to the Kubernetes API are entering the same shared control-plane environment. This is why API-server saturation can be particularly deceptive during an incident: the worker nodes can look completely normal while kubectl get pods starts taking seconds, then tens of seconds, and eventually timing out.
The cluster isn't necessarily "down" in the traditional sense. The control plane may simply be struggling to keep up with the amount of work being presented to it.
The API-Starvation Experiment
Let's make the problem a little more formal. Imagine that the API server has an effective concurrency capacity of C for a particular class of requests. Tenant A generates traffic at a rate of λA, while the system is capable of completing those requests at a rate of μA. When the arrival rate remains comfortably below the service rate, the system can keep up and queues remain small. As the arrival rate approaches the service rate, latency begins increasing because the system has less spare capacity available to absorb bursts.
λA < μA → system keeps up
λA → μA → latency increases
λA > μA → requests accumulate
This is basic queueing behavior, but it becomes extremely important when the queue belongs to a shared control plane. If Tenant A's request stream is allowed to consume essentially all of the available processing capacity, other request classes begin waiting behind work they did not generate. A platform controller trying to update a critical resource, a developer running kubectl, or another tenant attempting to scale an application can all experience increased latency even though none of them caused the original traffic spike.
A simplified model looks like this:
Tenant A
│
│ 1000 requests/sec
▼
┌─────────────────────┐
│ kube-apiserver │
│ │
│ concurrency = C │
└─────────┬───────────┘
│
▼
processing
The interesting part is what happens when several tenants are introduced:
Tenant A ─────┐
Tenant B ─────┤
Tenant C ─────┼──► kube-apiserver ──► etcd/controllers
Tenant D ─────┤
Tenant E ─────┘
Without an appropriate fairness mechanism, the control plane has to deal with a competition problem. The tenant generating the most work is not necessarily the tenant whose work is most important, and the request arriving first isn't necessarily the request that should be allowed to consume the next available unit of concurrency. Kubernetes therefore needs a mechanism that understands classes of traffic, assigns them different levels of importance, and prevents a noisy request stream from becoming an effective denial-of-service against unrelated control-plane operations.
This is precisely the problem APF was designed to address.
This Is Where APF Becomes Interesting
API Priority and Fairness exists because the statement "every request gets equal treatment" isn't particularly useful when operating a shared Kubernetes control plane. A request generated by a critical control-plane component may need very different treatment from a burst of interactive kubectl queries generated by a tenant. APF provides a mechanism to classify requests into different priority levels and flows, and then control how concurrency is distributed between those classes.
At a high level, the request path can be visualized like this:
Incoming API Requests
│
▼
┌─────────────┐
│ FlowSchema │
└──────┬──────┘
│
┌─────────┴─────────┐
▼ ▼
Priority Level A Priority Level B
│ │
queues queues
│ │
▼ ▼
execution execution
A FlowSchema determines which requests belong to a particular traffic class and associates those requests with a PriorityLevelConfiguration. The priority level then controls the concurrency and queueing behavior for that class. This gives Kubernetes something much more useful than a single global request queue: it can distinguish platform traffic, tenant traffic, monitoring traffic, leader-election traffic, or other categories that need different treatment.
For a multi-tenant platform, you can conceptually imagine a design like this:
Platform controllers
│
▼
high-priority
Tenant A
│
▼
tenant-workloads
Tenant B
│
▼
tenant-workloads
Monitoring
│
▼
monitoring
Kubernetes already ships with APF configurations intended to protect important classes of control-plane traffic, including leader-election and built-in controller traffic. The exact defaults have evolved across Kubernetes releases, so production administrators should inspect the APF configuration of the Kubernetes version they actually operate rather than assuming that an example from an older cluster behaves identically today.
The important architectural idea remains the same: not all API traffic needs to compete in one undifferentiated pool. Once requests are classified, Kubernetes can give different categories different concurrency behavior, which means a noisy tenant can be prevented from turning the entire API server into its personal work queue.
The Math Behind API Capacity
Let's make the model concrete. Suppose, purely for illustration, that the API server has a concurrency budget of:
Total concurrency = 100
Now imagine that the platform administrator establishes three priority levels with nominal allocations resembling:
Platform = 40
Tenant traffic = 50
Monitoring = 10
Conceptually, that gives us:
100 concurrent requests
│
├── 40 → Platform
├── 50 → Tenants
└── 10 → Monitoring
The useful mental model is approximately:
PL concurrency
≈
Total API concurrency ×
(nominal level limit / total nominal limits)
So a nominal 40% allocation against a concurrency budget of 100 corresponds to roughly 40 units of concurrency under the basic proportional model. However, this should not be interpreted as a permanent physical wall around exactly 40 requests. APF has borrowing and lending behavior, and its actual scheduling behavior depends on the complete PriorityLevelConfiguration. The purpose of the calculation is to understand the relative capacity allocation rather than to reduce APF to a simple static partitioning mechanism.
That distinction becomes important when designing real clusters. If the platform has a 40% nominal share, it doesn't necessarily mean 40 requests are permanently reserved and will sit idle whenever platform traffic is quiet. APF is designed to use capacity efficiently while maintaining fairness and protecting important traffic classes. The configuration therefore has to be evaluated against real workload behavior instead of being treated like a conventional static CPU reservation.
The numbers themselves should also come from measurements rather than guesswork. A cluster hosting hundreds of microservices with aggressive controllers, large watch sets, admission webhooks and frequent CI activity has a very different API workload from a small development cluster. API-server sizing, etcd performance, controller behavior, watch traffic, admission latency and tenant request patterns all contribute to the actual capacity required. The goal is not to find a magical percentage that works everywhere; the goal is to deliberately establish which traffic must remain responsive when another traffic class becomes noisy.
FlowSchemas: Separating the Noisy Neighbors
Now consider a slightly different problem. Suppose Tenant A and Tenant B both belong to the same priority level. We have successfully separated tenant traffic from critical platform traffic, but Tenant A can still generate a disproportionate amount of work within that tenant priority level. We therefore need another layer of separation: the ability to distinguish individual request flows so that one noisy tenant does not dominate the queueing behavior experienced by other tenants.
This is where FlowSchema and its flow distinguisher become particularly useful.
A FlowSchema can classify requests using characteristics such as:
- User
- Group
- ServiceAccount
- Verb
- API group
- Resource
- Namespace
Conceptually, we might create a classification around a tenant's ServiceAccount:
FlowSchema
│
├── ServiceAccount: team-a
│
└── PriorityLevel: tenant-workloads
That can then result in separate flows such as:
tenant-workloads
│
├── flow(team-a)
├── flow(team-b)
├── flow(team-c)
└── flow(team-d)
This distinction is important because priority level and flow are not the same thing. The priority level answers something like, "How important is this class of traffic, and how much concurrency should this class receive?" The flow answers something closer to, "Which requests should compete with one another inside that class?" That separation allows the system to protect both high-level control-plane priorities and fairness between individual request sources.
In a multi-tenant platform, this can be particularly valuable when tenants have very different request patterns. One tenant may run a highly automated GitOps platform that continuously watches and reconciles resources, while another may only make a handful of API calls during deployments. If both are treated as one undifferentiated flow, the heavy requester can influence the experience of the lighter requester. Flow-level separation gives APF more information with which to distribute the pressure.
Shuffle Sharding: Don't Let One Tenant Jack the Queue
This is one of the more interesting pieces of APF because it addresses the queue-collision problem without requiring a completely dedicated queue for every possible flow. Imagine that the API server has a collection of queues:
64 queues
If every request from every tenant could land anywhere, a particularly noisy tenant could eventually occupy a large portion of the queueing system. On the other hand, assigning one permanent queue to every possible tenant would be wasteful and difficult to manage when the number of flows changes dynamically.
Shuffle sharding takes a different approach.
The basic idea is that each flow receives a deterministic-looking subset of candidate queues, sometimes described as its hand, and requests from that flow are then placed among those candidates according to queue state. Kubernetes documents handSize as the number of queues considered for a flow, and its documented configuration uses values such as 64 queues with a hand size of 8.
64 queues
┌─────────────────────────────────────────┐
│ Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 ... Q64 │
└─────────────────────────────────────────┘
Tenant A
│
├── hash
│
└── candidate queues:
Q4, Q17, Q29, Q33, Q41, Q46, Q52, Q61
Tenant B
│
└── different candidate queues
The practical benefit is that two unrelated flows will generally have different subsets of queues. Tenant A therefore doesn't automatically gain access to every queue that Tenant B might need, while the system also avoids dedicating an entire permanently isolated queue to every tenant. If Tenant A becomes extremely noisy, the impact tends to remain concentrated around the subset of queues associated with its flow rather than allowing that traffic to contaminate every queue in the priority level.
This is why shuffle sharding is more interesting than simply saying "APF has multiple queues." The engineering goal is not merely to create more queues; it is to reduce the probability that two unrelated noisy flows collide across the same queueing resources. It provides a probabilistic form of isolation that becomes increasingly valuable as the number of request flows grows.
Designing APF for Tenants
A production design might therefore look conceptually like this:
kube-apiserver
│
API Priority
& Fairness
│
┌───────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Platform Tenant API Observability
│ │ │
30%+ 50% 20%
│
┌────────────┼────────────┐
▼ ▼ ▼
tenant-a tenant-b tenant-c
These numbers are deliberately illustrative rather than recommended defaults. A production administrator should not copy a 30/50/20 split into a cluster simply because it looks sensible on a diagram. The correct configuration depends on the actual API workload, number of tenants, controller behavior, admission webhooks, cluster size, etcd performance, API-server sizing and the operational SLOs that need to be protected.
There is also a subtle but important operational point here: APF should be observed, not merely configured. When an API request is classified, Kubernetes provides diagnostic information that can help operators understand which FlowSchema and PriorityLevelConfiguration handled it. That becomes extremely useful during an incident because it lets you move from the vague statement "the API server is slow" to a much more specific question: "Which traffic class is consuming the available concurrency, and where are requests waiting?"
For a serious multi-tenant platform, API capacity should therefore be treated as a shared resource just like CPU, memory or network bandwidth. You should know which classes of traffic are important, what happens when they exceed their expected rates, and whether critical control-plane operations continue making progress when one tenant becomes unexpectedly noisy.
The CRD Problem Nobody Sees Coming
API starvation is only one side of the multi-tenancy problem. The second problem is more architectural because it has nothing to do with request volume. It has to do with the fact that some Kubernetes APIs are cluster-scoped primitives, and Custom Resource Definitions are one of the most important examples.
A namespace can contain namespaced resources such as Pods, Deployments and Services, but a CRD defines an API resource for the Kubernetes cluster itself. That means the CRD is not independently created inside every namespace. If several teams share one physical Kubernetes cluster, they ultimately participate in the same cluster-level CRD definitions.
Consider:
Cluster
│
├── Namespace: team-a
│
├── Namespace: team-b
│
├── Namespace: team-c
│
└── CRD: certificates.cert-manager.io
The CRD isn't:
team-a/certificates.cert-manager.io
team-b/certificates.cert-manager.io
There is one cluster-level definition that establishes how the Kubernetes API understands that resource.
This distinction becomes particularly important when different tenants want to operate different versions of an operator or API ecosystem. Kubernetes CRDs can support multiple served versions and conversion between versions, so the problem is not as simplistic as saying that changing v1alpha1 to v1beta1 automatically breaks every other tenant. The real problem is that tenants sharing one physical CRD are still sharing the same API contract, schema, conversion behavior and lifecycle. They do not independently own that API simply because their workloads happen to live in different namespaces.
A CRD Collision Scenario
Imagine Tenant A operates an application platform that depends on an operator exposing:
Certificate
v1alpha1
At the same time, Tenant B has workloads built around a newer operator ecosystem expecting:
Certificate
v1beta1
Both tenants may believe they own their respective environments because all of their application objects are namespaced. However, if they are installing and managing the same cluster-scoped CRD, they are actually sharing a critical part of the Kubernetes API surface.
Now Tenant A performs an operator upgrade. That upgrade may modify the CRD's schema, served versions, storage version, conversion configuration, validation rules or related cluster-scoped components. Whether Tenant B actually breaks depends on the exact CRD versions, conversion strategy, operator behavior and resource definitions involved, but the architectural problem already exists before anything fails: two independently managed tenants have incompatible expectations about the same cluster-level API contract.
That is a very different failure mode from an ordinary namespace permission issue. RBAC can prevent Tenant A from modifying Tenant B's Deployment, but RBAC does not transform a cluster-scoped CRD into a tenant-scoped object. Similarly, a ResourceQuota can control how many Pods Tenant A creates, but it cannot create a second independent definition of certificates.cert-manager.io.
And this problem becomes even more significant with operators because installing an operator can involve much more than creating a Deployment in a namespace. Operators commonly install CRDs, ClusterRoles, ClusterRoleBindings, admission webhooks and other cluster-scoped resources. What looked like a namespace-level application deployment can therefore become an operation that changes shared cluster infrastructure.
The moment multiple independent teams are allowed to manage those components, the namespace abstraction starts showing its limits.
This Is the Point Where Namespaces Stop Being Enough
At this point, we have two fundamentally different multi-tenancy problems.
The first is API contention:
Tenant A
│
├── massive API traffic
│
▼
Shared kube-apiserver
│
├── Tenant B
├── Tenant C
└── Platform controllers
APF is designed to help here. It provides mechanisms for classifying requests, assigning them to priority levels, separating flows, queueing them fairly and reducing the ability of one traffic class to consume all available API-server capacity.
The second is cluster-scoped API collision:
Tenant A ──┐
├── same CRD
Tenant B ──┤
│
Tenant C ──┘
APF cannot solve this problem because the problem isn't request scheduling. ResourceQuotas cannot solve it because the problem isn't resource consumption. NetworkPolicies cannot solve it because the problem isn't network connectivity. Even RBAC only partially addresses it because preventing tenants from modifying cluster-scoped objects doesn't create independent cluster-scoped objects for every tenant.
At some point, the question changes from:
"How do we better isolate tenants inside one Kubernetes API?"
to:
"Should these tenants actually share the same Kubernetes API in the first place?"
That is the point where virtual control planes become interesting.
Enter vCluster
Instead of giving every tenant another namespace inside the same Kubernetes API, we can give them something that looks much closer to a Kubernetes cluster while still running that control plane on top of an existing Kubernetes cluster.
This is the basic idea behind vCluster.
A vCluster provides a virtualized Kubernetes control plane running inside a host Kubernetes cluster. The virtual control plane has its own API server and control-plane components, with a datastore and synchronization layer that connect the virtual environment to the underlying physical cluster.
Conceptually:
Host Kubernetes Cluster
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
vCluster A vCluster B vCluster C
│ │ │
API Server API Server API Server
Controller Controller Controller
Data Store Data Store Data Store
│ │ │
└──────────────┼──────────────┘
│
Syncer
│
▼
Host Kubernetes API
The important architectural difference is that the tenant is no longer interacting directly with the host Kubernetes API as its primary API surface. Instead, it talks to a virtual control plane. The virtual cluster can maintain its own Kubernetes objects, CRDs, RBAC configuration and control-plane state while selected workloads and resources are synchronized to the underlying host cluster.
This introduces another API boundary between the tenant and the physical cluster. Instead of trying to make one API server behave like several independent Kubernetes environments, we create multiple logical Kubernetes control planes that happen to share underlying infrastructure.
Virtual API vs Namespace
The difference becomes easier to see when the two models are placed next to each other.
Namespace tenancy
HOST API SERVER
│
┌──────────────┼──────────────┐
│ │ │
Namespace A Namespace B Namespace C
│ │ │
Apps Apps Apps
Every tenant is still talking to the same host API server. They may have different RBAC permissions and different namespaces, but their requests ultimately enter the same physical control-plane environment.
Virtual cluster tenancy
HOST CLUSTER
│
┌───────────────┼───────────────┐
│ │ │
vCluster A vCluster B vCluster C
│ │ │
API server API server API server
│ │ │
tenant tenant tenant
objects objects objects
Now each tenant has another API boundary. Tenant A can maintain its own API ecosystem without registering every tenant-facing CRD directly into the same host API surface used by Tenant B. That distinction can be extremely valuable for platform teams that need to give users a Kubernetes-like environment without handing each user an entire physical cluster.
However, this should not be misunderstood as magic isolation. The virtual control planes still run on the underlying Kubernetes infrastructure, and their workloads may still share worker nodes and the underlying operating-system kernel. The architecture therefore improves control-plane and API isolation without automatically providing the same security boundary as a physically separate cluster or a dedicated virtual machine.
But Where Do the Pods Actually Go?
This is where the synchronization layer becomes particularly interesting.
Suppose a tenant runs:
kubectl apply -f deployment.yaml
inside its virtual cluster. From the tenant's perspective, it is simply interacting with Kubernetes. Behind the scenes, however, the virtual control plane processes the request and the syncer determines which resources need to be represented on the underlying host cluster.
The simplified flow looks like this:
Tenant API
kubectl apply -f deployment.yaml
│
▼
vCluster API server
│
▼
Syncer
│
▼
Host Kubernetes API
│
▼
Namespace: tenant-a
│
▼
Pod
The tenant therefore sees a Kubernetes object in its virtual environment while the physical cluster sees the synchronized representation required to actually run the workload. Depending on the vCluster configuration, resources such as Pods, Services, ConfigMaps and Secrets can be synchronized between the virtual and host environments.
This creates an important abstraction:
Tenant abstraction
│
▼
Virtual Kubernetes API
│
▼
Synchronization layer
│
▼
Physical Kubernetes infrastructure
The tenant gets a Kubernetes API that is logically separate from the host API, while the infrastructure operator retains control over the physical cluster. This is why the model is often described as a virtual control plane rather than simply another namespace-management mechanism.
Why This Changes the CRD Problem
Let's return to our earlier CRD collision.
With ordinary namespace-based tenancy, the model looks like this:
Tenant A
│
└── shared CRD
Tenant B
│
└── shared CRD
Both tenants are ultimately participating in the same physical Kubernetes API definition.
With virtual clusters, the conceptual model changes:
vCluster A
│
└── CRD: certificates.cert-manager.io
vCluster B
│
└── CRD: certificates.cert-manager.io
The two tenants can now maintain their own virtual API definitions without requiring both definitions to coexist as independent versions of the same CRD in the host cluster's tenant-facing API. That is a major architectural improvement for teams that need Kubernetes-level API independence rather than simply namespace-level resource separation.
But there is still an important caveat. The physical cluster has not disappeared. The worker nodes, host API server, network, storage infrastructure and Linux kernel remain shared according to the deployment architecture. A virtual control plane therefore solves a particular class of isolation problems—it doesn't automatically solve every multi-tenancy problem.
That distinction is important because it prevents another common mistake: assuming that virtual Kubernetes equals virtual machine isolation. It doesn't. If two virtual clusters eventually schedule workloads onto the same worker node, those workloads are still ultimately sharing the same operating-system kernel and physical resources. Stronger isolation at that layer is a separate engineering problem, and that is exactly where the next part of this series will go.
The Multi-Tenancy Spectrum
At this point, it helps to stop thinking about multi-tenancy as a binary decision where a cluster is either "multi-tenant" or "not multi-tenant." Kubernetes provides a spectrum of isolation mechanisms, and the correct choice depends heavily on the trust relationship between tenants, the resources they need to control, the consequences of compromise, and the operational cost the platform team is willing to accept.
LOWER ISOLATION
│
▼
┌──────────────────────────────┐
│ Namespace │
│ RBAC │
│ ResourceQuota │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Namespace + NetworkPolicy │
│ + APF + admission policies │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Virtual Control Plane │
│ vCluster │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Dedicated Kubernetes Cluster │
└──────────────────────────────┘
│
▼
HIGHER ISOLATION
A development environment shared by trusted teams may require nothing more than namespaces, RBAC, quotas and network policies. A large internal platform serving dozens of independent engineering teams may need APF, stronger admission controls and carefully managed cluster-scoped resources. A platform offering Kubernetes environments to external or untrusted users may require virtual control planes, dedicated nodes, sandboxed runtimes or even completely separate physical clusters.
There is no universal "correct" tenancy model because the isolation requirement comes from the threat model. The important thing is to understand exactly which boundary each technology provides. Namespace isolation is useful, APF is useful, virtual control planes are useful, and dedicated clusters are useful—but they solve different problems and carry very different operational costs.
The Bigger Lesson
The mistake isn't using namespaces.
Namespaces are one of the most useful abstractions in Kubernetes, and for many environments they are exactly the right starting point. The mistake is assuming that a namespace represents more isolation than it actually provides. Once you operate Kubernetes at scale, the more useful questions are not simply "Which namespace does this application belong to?" but rather "Which underlying resources and control-plane components does this tenant still share with everyone else?"
Ask who shares the API server. Ask who shares API-server concurrency. Ask who can install or modify CRDs. Ask who controls admission webhooks. Ask who can create operators. Ask which workloads share worker nodes. Ask which tenants share storage and network infrastructure. And eventually, ask the question that becomes unavoidable when the trust boundary becomes weaker: what happens when the tenant is not trusted?
Those questions expose the actual architecture underneath the Kubernetes abstractions.
Multi-tenancy is therefore not a Kubernetes object that you enable with a single flag. It is a collection of isolation decisions made across several layers of the stack. Kubernetes gives you the primitives, but the platform engineer has to decide where each boundary belongs and what happens when one of those boundaries is stressed or deliberately attacked.
Part 1 Takeaway
Kubernetes gives you several layers of isolation, but they are not interchangeable. A namespace gives you a logical resource boundary. RBAC gives you an authorization boundary. NetworkPolicy provides a network-level control boundary. ResourceQuota constrains resource consumption within its scope. APF gives you a mechanism for protecting API-server concurrency between different request classes, while virtual control planes introduce another Kubernetes API boundary when namespace-level isolation no longer provides the separation you actually need.
The architecture can be summarized like this:
Multi-Tenancy
│
┌────────────┼────────────┐
▼ ▼ ▼
API isolation API fairness Resource isolation
│ │ │
vCluster APF Quotas / Policies
│
▼
Control-plane
separation
The deeper you go into Kubernetes, the more obvious the pattern becomes: multi-tenancy isn't a Kubernetes object; it is an architecture. The namespace is simply one layer in that architecture. Once you understand what remains shared underneath it, you can make a much more deliberate decision about whether your environment needs stronger API fairness, tighter control over cluster-scoped resources, virtual control planes, dedicated nodes, or completely separate clusters.
And this is only the first layer.
What's Coming in Part 2
In the next part we move past basic ResourceQuotas and focuses on the underlying Linux kernel, exploring container breakouts, kernel-level resource starvation,m and the absolute engineering trade-offsof running sandboxed runtimes. Stay tuned.
Top comments (0)