Welcome back to my blog series, "Into The Depths of Kubernetes," where we are peeling back the abstraction layers to explore the architectural patterns, runtime mechanics, and production realities that truly matter for software and platform engineers. Whether you are scaling microservices, optimizing infrastructure costs, or hardening cluster security, this series is designed to give you actionable insights and deep technical clarity. We are kicking things off with a fundamental challenge every growing organization faces: Multi-Tenancy in Kubernetes. In this inaugural post, we will dive into isolating workloads, managing shared cluster resources, enforcing strict security boundary conditions with Namespaces and Network Policies, and balancing cost efficiency against robust tenant isolation.
Part 2: Kernel Contention & Hard Sandboxing — Cgroups, OOM Kills, and Runtime Escapes
Containers do not exist.
At least, not in the way most Kubernetes users imagine them.
Underneath the YAML, Pods, Deployments, Services, and container images, there are still ordinary Linux processes running on a shared Linux kernel.
That distinction becomes extremely important when multiple organizations, teams, customers, or workloads are sharing the same Kubernetes cluster.
In Part 1 of our series, "Into The Depths of Kubernetes: Multi-Tenancy," we looked beyond the basic "just use namespaces" advice to expose the architectural limits of logical isolation. We explored how shared control-plane components—like the API server, etcd, and controller managers—remain vulnerable to noisy-neighbor workloads and concurrency bottlenecks. The post examined how API Priority and Fairness mitigates request saturation, why cluster-scoped CRDs introduce critical tenant collisions, and how virtual control planes like vCluster provide true control-plane isolation. If you missed it or need a refresher, check out Part 1 to see how we laid the groundwork for hard multi-tenancy.
Part 2 takes us one layer deeper.
We are leaving the Kubernetes API and entering the Linux kernel.
Because a namespace does not create a separate machine. A ResourceQuota does not create a separate CPU. A Pod security policy does not magically create a second kernel. And a container boundary, by itself, does not mean that a tenant has been isolated from every other process on the node.
What Kubernetes gives you is a carefully constructed set of abstractions around Linux primitives such as namespaces, cgroups, capabilities, seccomp, LSMs, and container runtimes.
The uncomfortable question is what happens when those boundaries are pushed hard.
What happens when Tenant A consumes memory faster than the node can provide it? What happens when one workload generates enormous amounts of filesystem writeback? What happens when the kernel has to kill something? And perhaps the most important question for genuinely untrusted workloads:
What happens when a process inside a container finds a way to attack the kernel itself?
That is where Kubernetes multi-tenancy stops being primarily a scheduling problem and becomes a kernel-isolation problem.
1. The Container Boundary Is Not the Kernel Boundary
A conventional container runtime such as containerd or CRI-O does not boot a miniature operating system for every container.
Instead, the container process runs directly on the host kernel.
Linux namespaces make the process believe it has its own process tree, network interfaces, mounts, hostname, and other resources. Cgroups control how much CPU, memory, and other resources the process can consume. Capabilities reduce what privileged operations it can perform, while seccomp can restrict which system calls it is allowed to invoke.
But there is still one kernel underneath all of it.
Kubernetes Node
┌─────────────────────────────────────────────────────────────┐
│ │
│ Linux Kernel │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────┐ │
│ │ Tenant A │ │ Tenant B │ │ Tenant C │ │
│ │ │ │ │ │ │ │
│ │ Container │ │ Container │ │ Container │ │
│ │ │ │ │ │ │ │
│ └───────┬────────┘ └───────┬────────┘ └─────┬──────┘ │
│ │ │ │ │
│ namespaces namespaces namespaces │
│ cgroups cgroups cgroups │
│ seccomp seccomp seccomp │
│ │ │ │ │
│ └─────────────────────┼───────────────────┘ │
│ │ │
│ Shared Kernel │
│ │ │
└─────────────────────────────────┼───────────────────────────┘
│
Physical Hardware
This architecture is one of the reasons containers are so efficient. There is no guest kernel per workload, so container startup is fast and resource overhead is relatively small.
It is also the reason the host kernel becomes part of your multi-tenancy threat model.
A vulnerability in an application is normally an application problem. A vulnerability in a container runtime may become a node problem. But a vulnerability in a kernel subsystem can potentially become a cluster isolation problem, because every ordinary container on that node ultimately interacts with the same kernel.
That is the fundamental trade-off we are going to explore in this part.
2. Cgroups: Where Kubernetes Resource Limits Actually Become Real
Kubernetes users normally interact with resources through something familiar:
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"
It is easy to think that Kubernetes itself is continuously watching the process and enforcing these limits.
It isn't.
The Kubelet translates Kubernetes resource configuration into Linux control-group configuration. The kernel then becomes responsible for enforcing those controls.
The important path looks roughly like this:
Pod YAML
│
▼
Kubernetes API
│
▼
Scheduler
│
▼
Kubelet
│
▼
Container Runtime
│
▼
Linux cgroup configuration
│
▼
Linux Kernel
│
├── CPU accounting / throttling
├── Memory accounting
├── Memory reclaim
├── OOM handling
└── I/O control
That last layer is where multi-tenancy becomes interesting.
3. cgroups v1 vs cgroups v2: The Hierarchy Problem
For years, Linux resource control was built around cgroups v1.
The problem with v1 was not that it could not limit resources. It could. The problem was that different controllers were organized around separate hierarchies. CPU could have one hierarchy. Memory could have another. Block I/O could have another. And this created increasingly complicated behavior as workloads became more sophisticated.
cgroups v1
┌─────────────────┐
│ CPU hierarchy │
└─────────────────┘
│
Tenant A/B/C
┌─────────────────┐
│ Memory hierarchy│
└─────────────────┘
│
Tenant A/B/C
┌─────────────────┐
│ I/O hierarchy │
└─────────────────┘
│
Tenant A/B/C
The result was not simply an administrative inconvenience. Resource accounting itself could become complicated, especially around filesystem I/O.
One particularly nasty area was buffered writeback.
A process can write data to a filesystem without immediately forcing all of that data to the physical storage device. Linux can temporarily hold dirty pages in memory and flush them later through writeback mechanisms.
Historically, cgroups v1 had limitations around consistently attributing this writeback activity to the workload that caused it.
That matters in a multi-tenant environment.
Imagine Tenant A running a workload that continuously generates enormous amounts of filesystem writes while Tenant B operates a latency-sensitive service on the same node.
Tenant A may not simply consume "disk bandwidth" in the obvious way. The actual work can involve page cache, dirty memory, asynchronous writeback, filesystem activity, block-layer processing, and device queues.
The performance cost can therefore appear somewhere other than the original application process.
That is exactly the kind of problem that makes noisy-neighbor analysis painful.
4. cgroups v2: One Hierarchy, Better Accounting
cgroups v2 was designed around a unified hierarchy.
Instead of treating resource controllers as completely independent trees, cgroups v2 provides a common hierarchy through which controllers can participate in a consistent resource-management model.
cgroups v2
│
Unified Tree
│
┌──────────┼──────────┐
│ │ │
Tenant A Tenant B Tenant C
│ │ │
┌──┼──┐ ┌──┼──┐ ┌──┼──┐
│ │ │ │ │ │ │ │ │
CPU MEM IO CPU MEM IO CPU MEM IO
This does not magically eliminate noisy neighbors.
It does something more important: it gives Linux a more coherent model for managing resources across a hierarchy.
Memory accounting becomes more consistent. CPU controls become integrated into the same tree. I/O controls can participate in the hierarchy, and writeback attribution has been substantially improved compared with the older cgroup model.
For Kubernetes operators, this matters because the container boundary is only as strong as the kernel's ability to accurately account for the work performed by that container.
If the kernel cannot correctly associate resource consumption with the workload that caused it, the cluster administrator eventually ends up debugging symptoms rather than causes.
That is one reason modern Kubernetes environments increasingly care about the underlying cgroup version rather than treating it as an invisible implementation detail.
5. CPU Limits: The Throttling Trap
CPU limits are another place where Kubernetes abstractions eventually become kernel mechanics.
Suppose a container has:
limits:
cpu: "1"
This does not mean the process has been assigned a dedicated CPU core.
It means the workload is constrained by Linux CPU control mechanisms, historically through CFS quota and period semantics and, depending on the environment, newer cgroup CPU controls.
If the workload attempts to consume more CPU than its configured allowance, it can be throttled.
That distinction becomes important when tenants run workloads with very different CPU behavior.
A batch-processing tenant may happily consume CPU whenever available. A latency-sensitive API may only require a fraction of a core but becomes extremely sensitive to scheduling delays.
Both workloads can technically remain within their configured resource boundaries while producing very different application-level consequences.
Tenant A
CPU-heavy batch workload
│
▼
┌───────────────┐
│ CPU pressure │
└───────┬───────┘
│
▼
Kernel scheduler / cgroup controls
│
├──────────────► Tenant A throttled
│
└──────────────► Shared CPU contention
│
▼
Tenant B latency rises
This is why simply saying "both tenants have CPU limits" does not completely describe isolation. Resource limits control consumption. They do not guarantee identical performance.
6. Memory Is Different: Eventually Somebody Has to Die
CPU contention can result in throttling. Memory is less forgiving.
When a node runs short of memory, Linux can reclaim memory through several mechanisms, but eventually there may be insufficient reclaimable memory to satisfy a new allocation.
At that point, the kernel's Out-Of-Memory killer becomes involved. This is one of the most misunderstood parts of Kubernetes resource management.
People often say:
"The container exceeded its memory limit, so Kubernetes killed it."
That is an incomplete description.
The actual behavior depends on whether the OOM occurs within a constrained cgroup, whether the cgroup is configured for OOM handling, whether the node itself is under memory pressure, and how the kernel evaluates possible victims.
The kernel does not think in terms of Kubernetes Pods. It thinks in terms of processes and memory pressure.
7. Anatomy of an OOM Kill
At a high level, when Linux cannot satisfy a memory allocation after reclaim attempts, it enters an OOM path and evaluates processes as potential victims.
One of the values involved is:
oom_score
and one of the most important modifiers is:
oom_score_adj
The adjustment allows userspace to influence how attractive a process is as an OOM victim. Kubernetes uses this mechanism as part of its QoS model.
The simplified relationship looks like this:
Node Memory Pressure
│
▼
Kernel OOM Decision
│
┌──────────┴──────────┐
│ │
oom_score_adj Memory footprint
│ │
└──────────┬──────────┘
▼
Candidate process
│
▼
KILL
For Kubernetes workloads, the important point is that QoS class influences OOM protection.
8. Guaranteed, Burstable, and BestEffort
Kubernetes classifies Pods into QoS categories based on their resource configuration.
At the broadest level:
| QoS Class | Resource Configuration | OOM Protection |
|---|---|---|
| Guaranteed | CPU and memory requests equal limits for every container | Strongest |
| Burstable | Some resource requests/limits are defined but don't meet Guaranteed criteria | Intermediate |
| BestEffort | No CPU/memory requests or limits | Weakest |
For a Guaranteed Pod, Kubernetes assigns an oom_score_adj close to the most protected end of the scale, traditionally -997.
BestEffort workloads sit at the opposite extreme, with an oom_score_adj of 1000.
Burstable workloads sit between these extremes, with the adjustment derived from their requested memory relative to node capacity, subject to Kubernetes' implementation rules.
That middle category is where things get interesting. Because "Burstable" sounds safer than it actually is.
9. The Burstable Trap
Consider two tenants.
Tenant A:
resources:
requests:
memory: "256Mi"
limits:
memory: "8Gi"
Tenant B:
resources:
requests:
memory: "4Gi"
limits:
memory: "4Gi"
Tenant A is Burstable.
Tenant B may qualify as Guaranteed, assuming its CPU configuration also satisfies the requirements.
Tenant A has effectively said:
"Schedule me as though I need 256 MiB, but permit me to consume substantially more."
That is perfectly legitimate Kubernetes behavior. It is also where multi-tenant systems can get into trouble.
If a large number of tenants make aggressive use of the gap between requests and limits, the scheduler may place workloads based on relatively modest requested resources while the node can experience substantially higher real memory consumption.
Scheduler view:
Tenant A → Request: 256 MiB
Tenant B → Request: 256 MiB
Tenant C → Request: 512 MiB
Tenant D → Request: 1 GiB
↓
Node appears schedulable
↓
Actual runtime consumption:
A → 5 GiB
B → 3 GiB
C → 4 GiB
D → 6 GiB
↓
Memory pressure
↓
OOM / reclaim
This is why Burstable is not synonymous with "safe isolation."
A tenant can be perfectly valid according to Kubernetes scheduling rules while still generating significant memory pressure at runtime.
And when that pressure reaches the node, the consequences are shared.
10. The OOM Killer Doesn't Understand Your Business
This is one of the most important operational realities in shared Kubernetes clusters.
The kernel does not know that:
- Tenant A owns the payment API.
- Tenant B owns a development environment.
- Tenant C is running a background batch job.
- Tenant D is running a critical production database.
It knows about processes, memory consumption, and OOM scoring. Your business priorities exist above the kernel. Your OOM decision exists below them. That means Kubernetes operators must translate business criticality into enforceable resource configuration.
QoS classes, requests, limits, node pools, eviction thresholds, priority classes, and workload placement are not independent features. Together, they form the practical memory-isolation strategy of the cluster.
11. When a Container Becomes a Kernel Problem
So far, everything we've discussed has been about accidental contention.
But multi-tenancy has another class of failure:
malicious contention.
A normal container shares the host kernel.
Every application eventually performs operations through system calls:
Application
│
│ system calls
▼
┌─────────────────────┐
│ Linux system calls │
└──────────┬──────────┘
│
▼
Host Kernel
│
┌──────┼──────┐
▼ ▼ ▼
Network Files Memory
A container process may execute operations such as:
open()
read()
write()
mmap()
clone()
socket()
ioctl()
mount()
setns()
The exact set available to an ordinary container depends on its security configuration, capabilities, seccomp profile, user namespaces, LSM policy, and runtime behavior.
But the important architectural fact remains: the syscall eventually enters the host kernel.
12. The Syscall Attack Surface
A Linux kernel is enormous. It handles process scheduling, virtual memory, networking, filesystems, storage, namespaces, IPC, security mechanisms, and much more. That makes the kernel one of the largest pieces of privileged software in the entire platform.
A vulnerability in an application may compromise that application. A kernel vulnerability can potentially cross the isolation boundary.
The attack pattern can look conceptually like this:
Untrusted Tenant
│
▼
Malicious process
│
▼
Crafted system call
│
▼
Kernel subsystem
│
▼
Kernel vulnerability
│
▼
Privilege escalation
│
▼
Host compromise
│
▼
Other tenants / node
This is what makes container escape fundamentally different from an ordinary application exploit. The attacker is not necessarily trying to break the application. They are trying to break the assumption that the container boundary is sufficient.
13. containerd and CRI-O Do Not Give You a Second Kernel
containerd and CRI-O are responsible for managing containers and integrating with Kubernetes through the CRI ecosystem.
They provide an important abstraction layer between Kubernetes and the low-level runtime. But neither one changes the fundamental architecture of conventional Linux containers.
Kubernetes
│
▼
Kubelet
│
▼
CRI
│
┌───┴────┐
│ │
containerd CRI-O
│ │
▼ ▼
OCI Runtime
│
▼
Linux Container
│
▼
Shared Host Kernel
If an attacker can exploit a vulnerable kernel interface that is reachable from inside the container, changing from containerd to CRI-O does not magically turn the workload into a hardware-isolated virtual machine.
The runtime can significantly improve the security boundary through seccomp, capabilities, namespaces, AppArmor/SELinux, user namespaces, and other controls. But the kernel remains shared.
That distinction is the foundation for understanding hard sandboxing.
14. The Engineering Question: How Much Isolation Do You Actually Need?
There is no universally correct answer.
If you are running your own trusted microservices, a conventional container runtime with a hardened configuration may be entirely appropriate.
If you are running arbitrary customer code, CI jobs submitted by unknown users, plugin systems, serverless functions, or workloads that intentionally execute untrusted binaries, the threat model changes.
Now the question becomes:
"How much of the host kernel do I want an untrusted workload to directly interact with?"
And this is where sandboxed runtimes enter the architecture.
Three technologies are particularly interesting:
- gVisor
- Kata Containers
- Firecracker
They all try to improve isolation. They simply do it at different layers.
15. gVisor: Put a Kernel-Like Boundary in User Space
gVisor takes a fundamentally different approach from a traditional VM.
Instead of allowing the application to interact directly with the host kernel for every supported operation, gVisor introduces a user-space component called Sentry, commonly used through the runsc runtime.
Conceptually:
Tenant Workload
│
▼
Application
│
▼
System Calls
│
▼
┌─────────────────┐
│ gVisor Sentry │
│ User Space │
└────────┬────────┘
│
▼
Host Kernel
The Sentry implements a substantial portion of Linux's kernel interfaces in user space. The application therefore interacts primarily with the Sentry rather than directly exposing the full host kernel surface.
This reduces the amount of host-kernel functionality directly exposed to the workload. The trade-off is obvious once you look at the architecture. You have introduced another compatibility and performance layer.
Applications that expect unusual Linux kernel behavior may encounter compatibility limitations, and workloads that perform heavily syscall-intensive operations can experience measurable overhead.
But for workloads where security isolation is more important than squeezing every last percentage point of performance from a conventional container, that trade-off can make sense.
16. Kata Containers: Put the Workload Behind a MicroVM
Kata Containers takes the isolation boundary further. Rather than building a user-space kernel-like layer, Kata launches workloads inside lightweight virtual machines.
Kubernetes Pod
│
▼
Kata Runtime
│
▼
MicroVM
┌─────────────────────┐
│ Guest Kernel │
│ │
│ Container Workload │
└──────────┬──────────┘
│
Virtual Hardware
│
▼
Host Kernel
Now the workload does not share the host kernel in the same way as a conventional Linux container. The guest kernel becomes part of the isolation boundary.
That gives Kata a much stronger boundary against classes of attacks that rely on exploiting the host kernel through normal container syscall access.
But virtualization has a cost. There is a guest kernel to manage. There is virtual hardware. There are additional memory structures. There is additional startup and runtime overhead.
Modern hardware virtualization makes this significantly lighter than traditional heavyweight virtual machines, but "lighter VM" does not mean "zero overhead."
17. Firecracker: MicroVMs for Fast, Strong Isolation
Firecracker takes the microVM concept and strips it down aggressively.
Originally developed for serverless workloads, Firecracker uses KVM-based virtualization while exposing a deliberately minimal virtual device model.
The design goal is simple:
Traditional VM
────────────────────────
Many virtual devices
Large attack surface
Heavy guest environment
Firecracker
────────────────────────
Minimal virtual hardware
Small device model
Fast startup
Strong hardware boundary
A simplified architecture looks like this:
Tenant Workload
│
▼
Guest Kernel
│
▼
┌────────────────┐
│ Firecracker VM │
│ │
│ Minimal VMM │
└───────┬────────┘
│
KVM
│
▼
Host Kernel
The critical difference is that Firecracker is not simply "another container runtime."
It is a virtual machine monitor designed around a deliberately narrow device model.
That makes it attractive for environments where workloads are genuinely untrusted and the security boundary needs to be stronger than ordinary containers provide.
18. gVisor vs Kata vs Firecracker
The three approaches can be understood as three different answers to the same question.
| Technology | Isolation Mechanism | Host Kernel Exposure | Overhead | Typical Use |
|---|---|---|---|---|
| Standard container | Linux namespaces + cgroups | High | Very low | Trusted workloads |
| gVisor | User-space kernel boundary | Reduced | Low–moderate | Untrusted applications |
| Kata Containers | Lightweight VM | Much lower | Moderate | Strong workload isolation |
| Firecracker | Minimal KVM microVM | Much lower | Moderate | Strong isolation / sandboxing |
The architecture matters more than the product name. A conventional container asks the host kernel to isolate processes. gVisor places an additional kernel-like boundary between the workload and host kernel.
Kata and Firecracker move toward hardware-assisted virtualization, giving the workload a separate guest kernel.
19. The Overhead Question
This is where architecture discussions often become misleading.
You will frequently see claims such as:
"Runtime X adds exactly 5% overhead."
That number is almost meaningless without knowing the workload. A syscall-heavy workload can behave very differently from a CPU-bound workload. A memory-heavy workload behaves differently from a network-heavy workload. Startup latency differs from steady-state throughput.
So rather than pretending there is one universal number, think about the relative shape of the trade-off.
Relative Resource / Isolation Trade-off
Isolation
▲
│
│ Firecracker
│ ┌───────────────
│ Kata ─┤
│
│ gVisor ─────┤
│
│ Standard Container ─┤
│
└────────────────────────────────────►
Runtime Overhead
A rough engineering model looks like this:
| Runtime | CPU Overhead | Memory Overhead | Startup | Isolation |
|---|---|---|---|---|
| Standard container | Lowest | Lowest | Extremely fast | Process/kernel boundary |
| gVisor | Low–moderate | Moderate | Fast | Stronger syscall isolation |
| Kata | Moderate | Moderate–higher | Fast, but above containers | VM boundary |
| Firecracker | Moderate | Moderate | Very fast for a VM | Hardware-assisted VM boundary |
These should be treated as relative architectural expectations, not benchmark guarantees.
The actual result needs to be measured against your workload.
For example, a workload performing millions of system calls per second may expose gVisor's additional syscall path more clearly than a CPU-bound application.
Likewise, a workload requiring large amounts of guest memory will make the memory cost of VM-based isolation much more visible than a tiny service.
20. The Real Trade-Off: Density vs Isolation
This is ultimately not a technology-selection argument. It is an engineering trade-off. Consider a cluster with 500 tenants.
If every workload is trusted internal software, putting every Pod inside a microVM may introduce unnecessary operational and resource costs.
But imagine the cluster runs:
- customer-submitted code,
- CI workloads,
- build environments,
- arbitrary binaries,
- third-party plugins,
- serverless functions,
- educational code execution,
- multi-user development environments.
The threat model is completely different.
In that environment, the question isn't:
"Can we run more Pods per node?"
The question becomes:
"What is the cost of allowing an untrusted process to share a kernel with everyone else?"
That is a much more meaningful question.
21. A Practical Isolation Spectrum
There is no single "multi-tenancy architecture."
There is a spectrum.
Increasing Isolation
│
▼
┌─────────────────────────────────────────────┐
│ Namespace-based tenancy │
│ │
│ Shared kernel │
└─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ Hardened containers │
│ │
│ seccomp + capabilities + LSM + cgroups │
└─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ Sandboxed containers │
│ │
│ gVisor / similar isolation │
└─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ MicroVM-backed workloads │
│ │
│ Kata / Firecracker │
└─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ Dedicated node / cluster │
│ │
│ Physical infrastructure boundary │
└─────────────────────────────────────────────┘
Moving downward generally increases isolation.
It can also increase cost, operational complexity, startup time, memory consumption, and debugging complexity. There is no free isolation. Every stronger boundary introduces another engineering layer.
22. What Should a Platform Engineer Actually Do?
The answer should not be "deploy Kata everywhere." Start by defining the threat model. Ask what kind of tenants are sharing the cluster.
- Are they internal teams?
- Are they different business units?
- Are they external customers?
- Can they execute arbitrary code?
- Can they submit containers?
- Can they control Linux capabilities?
- Can they mount host paths?
- Can they access devices?
- Can they create privileged Pods?
- The answers completely change the architecture.
For conventional internal workloads, a strong baseline usually involves:
Namespaces
+
ResourceQuota
+
LimitRange
+
Requests / Limits
+
Pod Security controls
+
Seccomp
+
Capabilities dropped
+
AppArmor / SELinux
+
NetworkPolicy
+
Hardened node configuration
+
Regular kernel/runtime patching
For genuinely untrusted workloads, add a stronger execution boundary.
That might mean gVisor, Kata, Firecracker-based infrastructure, dedicated node pools, or even separate clusters depending on the threat model.
23. The Most Dangerous Assumption
The most dangerous assumption in Kubernetes multi-tenancy is:
"It's a container, so it's isolated."
It isn't.
It is isolated to a degree.
That degree depends on how the container is configured, how the runtime is configured, what the kernel exposes, what capabilities the process has, what security policies are applied, and whether the underlying kernel and runtime are patched.
The same statement applies to resource isolation.
A memory limit is not the same thing as guaranteed memory availability. A CPU limit is not the same thing as dedicated CPU performance. A namespace is not a machine. A QoS class is not a business-criticality guarantee. And a container runtime is not necessarily a security boundary equivalent to a virtual machine.
Once you understand those distinctions, Kubernetes multi-tenancy becomes much easier to reason about.
24. The Platform Engineer's Mental Model
When debugging multi-tenant failures, stop looking at Kubernetes alone.
Trace the entire stack:
Tenant
│
▼
Pod
│
▼
Container
│
▼
Runtime
│
▼
cgroup
│
▼
Linux namespace
│
▼
Linux kernel
│
▼
Scheduler / MM / VFS / Network stack
│
▼
Hardware
A Kubernetes incident may begin as a Pod problem but end up being a kernel scheduling problem. A storage latency problem may begin as a tenant workload but manifest as filesystem writeback contention.
A memory incident may look like a Kubernetes deployment failure but ultimately be an OOM decision made by the Linux kernel. And a security incident may begin with a seemingly harmless container but end with a host kernel vulnerability.
The abstraction layers are useful. But during an incident, they can also hide the actual failure.
25. Conclusion: Kubernetes Can Isolate Workloads, But the Kernel Still Has the Final Word
Kubernetes multi-tenancy becomes considerably more complicated once you move below the API server.
At the control-plane layer, tenants compete for API resources, controllers, CRDs, and scheduling capacity. At the node layer, they compete for something much more fundamental: CPU cycles, memory pages, filesystem activity, network resources, and ultimately access to the Linux kernel itself.
cgroups provide the foundation for controlling that competition, but understanding them requires going beyond resources.requests and resources.limits. The differences between cgroups v1 and v2, memory accounting, writeback behavior, CPU throttling, and OOM handling can directly influence how predictable a shared node actually is.
The OOM killer is a particularly good reminder that Kubernetes abstractions eventually terminate at the kernel. The kernel does not understand tenants, applications, SLAs, or business priorities. It evaluates processes and memory pressure. Kubernetes' QoS classes and oom_score_adj mechanisms help influence that decision, but they do not turn an oversubscribed node into an isolated machine.
Security pushes the problem even further.
A standard container still shares the host kernel. That gives us excellent density and startup performance, but it also means that the host kernel remains part of the attack surface. For trusted workloads, hardened containers can provide a very strong practical boundary. For workloads executing genuinely untrusted code, however, the architecture may need another layer.
That is where gVisor, Kata Containers, and Firecracker become interesting. They represent different points on the isolation spectrum, from syscall interception and user-space kernel mechanisms to hardware-assisted microVM boundaries. None of them is universally "better." They simply move the security boundary and accept different performance, compatibility, memory, and operational costs in exchange.
And that is the uncomfortable engineering truth about hard multi-tenancy:
Isolation is never free.
You either pay with reduced density, additional memory, virtualization overhead, operational complexity, or engineering effort—or you accept more shared infrastructure and therefore a larger blast radius.
The job of a platform engineer is not to eliminate that trade-off. It is to understand it well enough to place the boundary deliberately.
Because once multiple tenants share a Kubernetes node, the kernel becomes part of your tenancy architecture whether you planned for it or not.
Coming in Part 3: Encrypting the Tenant Boundary
We have now moved from the Kubernetes control plane down into the Linux kernel. But there is still one major boundary left. The data itself.
Even if Tenant A cannot consume all of Tenant B's memory, cannot escape its container, and cannot compromise the host kernel, we still have to answer a much harder question:
Can Tenant A observe, intercept, modify, or recover Tenant B's data while it is moving through the cluster or sitting on storage?
In Part 3 — the final chapter of this multi-tenancy series, we will move into the networking and storage layers and examine how to lock down data-in-transit and data-at-rest across tenants.
We will go beyond basic NetworkPolicies and Kubernetes Secrets and look at the architecture behind encrypted east-west traffic, service-mesh mTLS, identity-aware communication, storage encryption, key management, CSI integration, and the cryptographic boundaries required when tenants share the same underlying infrastructure.
Because isolating the process is only half the problem. The final question is whether you can isolate the data.
See you in Part 3. Into The Depths of Kubernetes' continues...
Top comments (0)