DEV Community: Kubernetes with Naveen

Who Owns Kubernetes on Day 2? Defining the Operational Boundary

Kubernetes with Naveen — Wed, 07 Oct 2026 12:00:47 +0000

Day 1 of a Kubernetes deployment feels like a victory. Day 2 is where the organizational cracks show up.

The cluster is provisioned. Nodes are healthy. Ingress is routing traffic. CoreDNS is responding. Monitoring is green. The application team gets a namespace, deploys its workloads, and everyone moves on to the next project.

Then reality arrives.

A deployment starts getting OOMKilled. A rollout hangs because of a restrictive PodDisruptionBudget. Someone changes a production manifest directly with kubectl. Three weeks later, the Git repository says one thing while the cluster is running something else. A node pool needs an upgrade, but nobody wants to touch it because six teams have workloads running there.

Then comes the inevitable question: Who owns this?

That question sounds simple, but it exposes one of the most important design decisions in Kubernetes: the operational boundary between the platform team and the application team. If that boundary is too loose, Kubernetes becomes the Wild West. Developers get excessive privileges, production state changes outside Git, and platform engineers spend their days cleaning up application mistakes.

If that boundary is too restrictive, Kubernetes becomes Ticket-Ops. Developers cannot change a ConfigMap without filing a Jira ticket, cannot restart their own workload, and cannot inspect production logs without asking the platform team.

Neither model scales.

The goal is not to decide who owns Kubernetes as a single system. The goal is to decide who owns each operational responsibility after the cluster becomes a production platform.

The Day 2 Reality Check: Why Healthy Clusters Still Host Broken Applications

A Kubernetes cluster can be completely healthy while the application running inside it is completely broken. That distinction sounds obvious, but many organizations blur it.

Suppose every node reports Ready, the Kubernetes API server is responsive, the CNI is functioning, CoreDNS is healthy, and the ingress controller has no errors. From a platform perspective, the cluster looks excellent. Now imagine an application deployment has a memory limit of 256Mi, but its actual working set regularly reaches 400Mi.

The container gets terminated with:

OOMKilled

Kubernetes is doing exactly what it was configured to do. The platform team did not break the application. The application team configured a workload that cannot operate within its declared resource boundary.

The same distinction appears in networking. The cluster network can be healthy while an application has an incorrect Service selector. The ingress controller can be perfectly operational while an application has configured the wrong backend port. The scheduler can be functioning correctly while a workload cannot be placed because its resource requests are unrealistic. This is why Day 2 operations need explicit ownership boundaries.

A useful mental model is:

The platform team owns the environment in which workloads run. Application teams own the workloads themselves.

That sounds simple until you start defining the edges.

Who owns RBAC?
Who owns secrets?
Who owns resource quotas?
Who owns node upgrades?
Who owns a failed deployment?
Who owns configuration drift?

The answers need to be designed before the first serious incident, not negotiated during one.

The Core Philosophy: Platform as a Product vs. Infrastructure Monopoly

The strongest Kubernetes operating models treat the internal platform as a product. The platform team is not simply the group that manages Kubernetes. It provides a product consumed by engineering teams.

That product includes the Kubernetes API, namespaces, networking capabilities, identity integration, observability, deployment mechanisms, security policies, storage primitives, upgrade processes, and paved roads for deploying applications.

The platform team owns the API and the paved road. Application teams own what they run on top of that road.

This distinction is important because platform engineering is not about taking operational responsibility away from developers. It is about removing unnecessary infrastructure complexity while preserving application ownership.

A good platform might provide an application team with:

Application Repository
        |
        v
GitOps Configuration
        |
        v
Deployment Pipeline
        |
        v
Kubernetes Namespace
        |
        +---- Service
        +---- Deployment
        +---- Ingress
        +---- ConfigMap
        +---- HPA
        +---- Resource Requests/Limits

The platform team controls the guardrails around this system. The application team controls the application-specific configuration within those guardrails. That is Platform as a Product.

The platform should answer:

"How do we safely run workloads?"

The application team should answer:

"What workload should we run, and how should it behave?"

When those two questions become mixed together, operational ownership becomes unclear.

The Responsibility Breakdown: The Operational Matrix

There is no universal ownership matrix that fits every company, but the following model works well for organizations operating shared Kubernetes platforms.

Operational Domain	Platform / SRE	Application Team	Shared Boundary
Control plane	Own	Consume	Platform health
Node pools	Own	Validate workloads	Upgrade coordination
Kubernetes upgrades	Own	Test applications	Compatibility
Namespace creation	Own / automate	Request/consume	Standard templates
RBAC platform roles	Own	Consume	Least privilege
Application RBAC	Guardrails	Own	Policy
Application deployment	Provide platform	Own	GitOps
ConfigMaps	Guardrails	Own	GitOps
Secrets	Secret platform	Own application secrets	Security policy
Resource quotas	Set boundaries	Operate within them	Capacity planning
CPU/memory requests	Validate/policy	Own	Platform defaults
Application health	Infrastructure signals	Own	Observability
Configuration drift	GitOps platform	Own desired state	Reconciliation
Cluster incidents	Own	Support	Incident coordination
Application incidents	Support platform	Own	Escalation
Security policies	Own guardrails	Comply	Exceptions
Cost optimization	Infrastructure efficiency	Workload efficiency	Shared

The important part is not the exact table. It is the principle behind it:

Platform owns the boundaries. Application teams own the workload behavior inside those boundaries.

Cluster Infrastructure & Upgrades: Platform/SRE Owns the Floor

The Kubernetes control plane and worker infrastructure should normally belong to the platform or SRE organization.

That includes:

Kubernetes version upgrades
Control plane lifecycle
Node pools
Container runtime configuration
CNI
CSI infrastructure
CoreDNS
Ingress infrastructure
Cluster-wide observability
Cluster-wide security controls
Autoscaling infrastructure
Node replacement
Cluster capacity

Application developers should not need to understand how the control plane is upgraded in order to deploy an application. That is exactly what the platform exists to abstract. But abstraction does not mean isolation. A cluster upgrade can still break an application.

Imagine the platform team wants to replace a node pool. They cordon nodes and begin draining them. Kubernetes attempts to evict workloads. Then the drain gets stuck.

The reason?

An application has a single replica and an overly restrictive PodDisruptionBudget:

spec:
  minAvailable: 1

The platform team owns the node lifecycle. The application team owns the workload availability configuration. This is where operational boundaries become shared interfaces rather than rigid walls.

Pod Disruption Budgets Are a Contract

A PodDisruptionBudget tells the platform:

"This is how much voluntary disruption this workload can tolerate."

The platform team must respect it. The application team must configure it correctly. Neither side can treat it as someone else’s problem. The same applies to resource requests and limits.

If a workload declares:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

the scheduler and kubelet use those values to make placement and enforcement decisions. The platform provides the scheduling environment. The application team provides realistic workload requirements.

A production node upgrade therefore becomes a coordination problem, not a ticket handoff.

Workload Lifecycle & Health: Application Teams Own What They Deploy

Once the platform provides a namespace and the required primitives, the application team should own the lifecycle of its workload.

That includes:

Deployment manifests
StatefulSets
Jobs
CronJobs
Services
Ingress configuration
ConfigMaps
Application-level RBAC
Resource requests and limits
Horizontal Pod Autoscalers
Probes
PodDisruptionBudgets
Application dependencies
Deployment strategies
Application-level dashboards and alerts

This does not mean developers need unrestricted access to the cluster. They should have enough access to operate their applications without requiring the platform team to perform routine actions for them.

A developer should generally be able to answer:

Is my deployment running?
Are my pods ready?
Why did my pod restart?
What does the application log say?
What image version is deployed?
Is my rollout progressing?
Are my resources being throttled?

If answering these questions requires opening a Jira ticket, the platform has created unnecessary friction. The solution is not cluster-admin. The solution is better scoped access.

Access Control, Secrets & RBAC Boundaries

One of the easiest ways to create a dangerous Kubernetes environment is to give developers cluster-admin because it makes everything easier. It also eliminates the need to design access properly. That is precisely why it should be avoided.

A developer working on an application normally does not need permission to:

delete nodes
modify cluster-wide RBAC
change admission policies
modify CRDs
inspect unrelated namespaces
change networking infrastructure
delete persistent volumes belonging to another team

Their access should normally be scoped around their namespace and the resources they actually operate.

For example:

Cluster
|
+-- platform-system
|
+-- monitoring
|
+-- team-a
|   +-- Deployment
|   +-- Service
|   +-- ConfigMap
|   +-- Secret
|   +-- Pods
|
+-- team-b
    +-- Deployment
    +-- Service
    +-- ConfigMap
    +-- Secret
    +-- Pods

Team A should not need administrative access to Team B’s namespace. This is where Kubernetes Role and RoleBinding become operational boundaries rather than just security objects.

Production `kubectl` Access Is a Design Smell

Direct production access through kubectl is not automatically forbidden. Emergency debugging sometimes requires it. The problem is treating unrestricted interactive access as the normal deployment mechanism.

If engineers routinely run:

kubectl edit deployment production-api

or:

kubectl set image deployment/api api=myimage:latest

you have created a second configuration system. Git says one thing. The cluster says another. The next GitOps reconciliation may overwrite the manual change. Or worse, the manual change becomes an undocumented production configuration that nobody remembers making.

A healthier model is:

Git defines desired state. Kubernetes executes that state. GitOps continuously reconciles the two.

Managing Configuration Drift via GitOps

Configuration drift is one of the biggest Day 2 problems because it is often invisible until something fails.

Consider this sequence:

Monday:
Git says replicas = 3

Tuesday:
Developer changes replicas to 5 with kubectl

Wednesday:
Platform engineer upgrades the cluster

Thursday:
GitOps reconciliation changes replicas back to 3

Friday:
Traffic increases and the application behaves differently

Nobody necessarily made a mistake on Friday. The system simply exposed a configuration ownership problem that started on Tuesday. GitOps tools such as Argo CD and Flux solve part of this problem by turning Git into the declared source of truth.

A simplified model looks like this:

                Git Repository
                     |
                     v
             Desired State
                     |
                     v
              GitOps Controller
                     |
                     v
              Kubernetes API
                     |
                     v
             Running Workload
                     |
                     |
              Drift Detection
                     |
                     +------> Reconcile

The critical organizational decision is not merely adopting Argo CD or Flux. It is deciding which repository owns which resources.

For example:

platform-config/
├── ingress-controller/
├── cert-manager/
├── monitoring/
└── policies/

application-config/
├── payments/
├── checkout/
└── catalog/

The platform repository owns platform components. The application repository owns application configuration. This creates an explicit ownership contract. If a developer wants to change an application deployment, they change the application repository. If the platform team wants to change the ingress controller, it changes the platform repository. Neither team needs to manually edit live resources.

Git Becomes More Than Version Control

In a mature GitOps environment, Git becomes an operational contract.

It answers:

Who owns this resource?
Who approved the change?
What changed?
When did it change?
What should the cluster look like?
Who can modify the configuration?
Can we reproduce the environment?

That is far more useful than simply saying, "Everything is managed through Git." The real value is clear ownership of desired state.

Access Should Be Self-Service, Not Permissionless

There is an important distinction between self-service and unrestricted access. Self-service means a developer can perform an approved operation without opening a ticket. It does not mean the developer can bypass every control.

For example, a platform can provide an application template:

service:
  name: payments

resources:
  requests:
    cpu: 250m
    memory: 256Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10

The developer controls application-specific values.

The platform enforces constraints such as:

Maximum CPU request
Maximum memory request
Required securityContext
Required probes
Allowed container registries
Required labels
Allowed ingress classes
Approved storage classes

This is where admission policies, resource quotas, namespace templates, and policy engines become useful. The platform team defines the guardrails. The developer gets the steering wheel.

Resource Management & Cost: Who Owns the OOMKilled Pod?

This is where ownership models often become emotional.

A pod gets:

OOMKilled

The application team says:

"Kubernetes killed our application."

The platform team says:

"You configured a 256Mi memory limit."

Both statements describe part of the situation. But operational ownership should still be clear. The application team owns the workload’s resource requirements. The platform team owns the cluster’s capacity and enforcement mechanisms.

A useful distinction is:

Application Team Owns

CPU requests
Memory requests
CPU limits where appropriate
Memory limits
Autoscaling configuration
Application efficiency
Capacity requirements
Workload behavior under load

Platform Team Owns

Node capacity
Cluster autoscaling
ResourceQuota enforcement
LimitRange defaults
Scheduling infrastructure
Node sizing
Cluster-level capacity planning
Cost visibility at the infrastructure layer

If a Java application genuinely requires 2Gi but is configured with a 512Mi memory limit, the platform team should not be expected to permanently compensate by adding larger nodes.

That is hiding an application configuration problem inside infrastructure spending. At the same time, if dozens of workloads are correctly requesting resources and the cluster consistently runs at capacity, the platform team cannot tell developers to "optimize their pods" forever.

That becomes a platform capacity problem.

Resource ownership is therefore shared at the boundary, but accountability remains specific.

Failure Modes of Bad Ownership: Ticket-Ops vs. The Wild West

There are two predictable extremes. Both are bad.

Failure Mode #1: Ticket-Ops

The organization tries to maintain control by putting the platform team in the middle of everything. A developer needs to change an environment variable. They create a Jira ticket. They need to increase replicas. Another ticket. They need to restart a deployment. Another ticket. They need logs from production. Another ticket. They need a namespace. Another ticket.

Eventually the platform team becomes an infrastructure help desk.

The consequences are predictable:

Developer
   |
   v
Jira Ticket
   |
   v
Platform Engineer
   |
   v
Manual kubectl
   |
   v
Production

This does not create governance. It creates a bottleneck. Worse, it encourages developers to find workarounds.

Failure Mode #2: The Wild West

The opposite model gives every engineering team broad production access.

Everyone can run:

kubectl delete pod
kubectl edit deployment
kubectl apply -f production.yaml

It feels productive at first. Then someone changes a production deployment manually. Another engineer deletes a resource they thought was unused. A developer modifies a cluster-wide object while troubleshooting. Nobody knows who owns the change. Auditability disappears. The platform team eventually responds by introducing even more controls. The organization swings from Wild West to Ticket-Ops.

This cycle happens repeatedly when teams treat permissions as the ownership model. They are not the same thing.

The Better Model: Guardrails, Not Gates

A mature platform does not ask:

"How can we stop developers from changing things?"

It asks:

"How can developers safely change the things they own without being able to damage the things they do not?"

That leads to a much better architecture.

                 Platform Team
                       |
       +---------------+----------------+
       |               |                |
   Guardrails       Paved Roads      Platform APIs
       |               |                |
       +---------------+----------------+
                       |
                 Kubernetes
                       |
        +--------------+--------------+
        |              |              |
     Team A          Team B         Team C
        |              |              |
     Workloads      Workloads      Workloads
        |              |              |
      GitOps         GitOps         GitOps

The platform team does not become the operator of every workload.

Instead, it provides a safe operating environment.

That means:

Namespace-level RBAC
Standard workload templates
GitOps
Policy enforcement
Resource quotas
Observability
Self-service deployment
Automated namespace provisioning
Standard secrets management
Documented escalation paths
Automated cluster upgrades
Clear ownership metadata

The result is a platform where developers can move quickly without requiring administrative access to the entire cluster.

Pragmatic Recommendations for Mid-to-Senior Engineers

The ownership model does not need to be perfect on day one. It needs to be explicit. Start by documenting the operational boundary.

1. Write the Ownership Contract

For every major Kubernetes resource, answer:

Who creates it?
Who modifies it?
Who approves changes?
Who monitors it?
Who gets paged?
Who can delete it?

If the answer is "everyone," the boundary is probably not defined well enough.

2. Keep Cluster-Admin Extremely Small

cluster-admin should be an exceptional privilege. Platform engineers who maintain the cluster may need it. Most application developers do not. Build namespace-scoped roles that provide exactly what teams need. The objective is not zero access. The objective is appropriate access.

3. Make Git the Default Change Path

If production configuration is represented in Git, developers should be able to submit pull requests rather than tickets.

A good flow looks like:

Developer
   |
   v
Pull Request
   |
   v
Validation / Policy Checks
   |
   v
Review
   |
   v
GitOps
   |
   v
Kubernetes

This gives developers autonomy while preserving review, history, and rollback.

4. Automate the Boring Requests

If developers frequently ask the platform team for:

Namespace creation
Standard RBAC
Service accounts
Basic monitoring
Resource quotas
Application templates
Standard ingress
Deployment scaffolding

then those are candidates for platform automation. A platform engineer should not manually create the same namespace structure twenty times.

5. Define Upgrade Contracts

Before upgrading Kubernetes, application teams should know:

Which Kubernetes version is being targeted
What APIs are deprecated
When workloads will be disrupted
What PDB behavior is expected
Which applications need compatibility testing
How rollback or remediation works

The platform team owns the upgrade. Application teams own application compatibility.

6. Make Resource Requests a First-Class Engineering Concern

Do not treat CPU and memory values as YAML decoration.

They affect:

Scheduling
Cluster capacity
Autoscaling
Cost
Reliability
Eviction behavior
Upgrade safety

Every production workload should have a reasonable resource profile. If the platform automatically rejects workloads without resource requests, that is not bureaucracy. It is preventing undefined scheduling behavior from becoming a production incident.

7. Give Developers Production Visibility Without Production Control

This is one of the most useful boundaries an organization can establish.

Developers should usually be able to see:

Logs
Metrics
Traces
Events
Deployment status
Pod status
Resource consumption
Rollout history

without automatically receiving permission to modify:

Nodes
Cluster RBAC
Admission policies
Other teams’ namespaces
Cluster networking
Control-plane resources

Read access and write access do not need to be coupled.

This single distinction can eliminate a surprising amount of unnecessary cluster-admin usage.

8. Build Incident Ownership Around the Failure Domain

When something breaks, do not ask:

"Which team owns Kubernetes?"

Ask:

"Where is the failure?"

If the API server is unavailable, platform owns the incident. If nodes are failing, platform owns the incident. If an application’s readiness probe is broken, the application team owns the incident. If the ingress controller is failing globally, platform owns it. If one application’s ingress rule is incorrect, the application team owns it.

If both sides contributed to the failure, establish a shared incident channel and resolve the immediate problem first. Conduct the ownership discussion during the follow-up, not while production traffic is burning.

That is what mature operations looks like.

A Practical Day 2 Operating Model

A healthy Kubernetes organization should eventually look something like this:

                    PLATFORM TEAM
                         |
       +-----------------+------------------+
       |                 |                  |
   Kubernetes         Guardrails        Paved Roads
   Infrastructure       |                  |
       |            RBAC / Policy       GitOps
       |            Quotas / Security   Templates
       |                                  |
       +----------------+-----------------+
                        |
                   Kubernetes API
                        |
       +----------------+----------------+
       |                |                |
    Team A            Team B           Team C
       |                |                |
   Application       Application      Application
   Ownership         Ownership        Ownership
       |                |                |
      Git              Git              Git
       |                |                |
     GitOps           GitOps           GitOps

This architecture creates a useful separation:

Platform engineering is responsible for making the platform safe and usable.

Application engineering is responsible for making the application reliable.

Those responsibilities overlap, but they should not be confused.

3 Key Takeaways

1. Platform owns the platform. Application teams own the workloads.

The platform team should own Kubernetes infrastructure, upgrades, node lifecycle, cluster-wide security, networking primitives, and platform reliability.

Application teams should own deployments, configuration, resource requirements, application health, and application-level operational behavior. The boundary should be explicit.

2. Self-service beats both Ticket-Ops and unrestricted access.

Developers should not need a Jira ticket to make routine changes to workloads they own. They also should not need cluster-admin to do it.

Use namespace-scoped RBAC, GitOps, policy enforcement, templates, and automated workflows to give developers autonomy inside well-defined boundaries. The goal is not to remove control.

The goal is to move control into automation and policy instead of human gatekeeping.

3. Day 2 ownership is an architectural decision, not an org-chart decision.

Kubernetes does not automatically create operational boundaries. Your platform architecture does. Git repositories, RBAC roles, namespaces, resource quotas, admission policies, GitOps controllers, observability systems, and incident processes should all reinforce the same ownership model.

When they do, the question changes from:

"Who owns Kubernetes?"

to the much more useful question:

"Who owns this layer of the platform, and what does the other team need from them?"

That is the boundary worth designing. Because a Kubernetes cluster is not the product. The platform is the product.

And on Day 2, the quality of that platform is measured less by whether the cluster is running and more by whether engineers can safely operate what is running on it.

Into The Depths of Kubernetes: Multi-Tenancy Part 3

Kubernetes with Naveen — Mon, 28 Sep 2026 14:26:19 +0000

Welcome back to my blog series, "Into The Depths of Kubernetes," where we are peeling back the abstraction layers to explore the architectural patterns, runtime mechanics, and production realities that truly matter for software and platform engineers. Whether you are scaling microservices, optimizing infrastructure costs, or hardening cluster security, this series is designed to give you actionable insights and deep technical clarity. We are kicking things off with a fundamental challenge every growing organization faces: Multi-Tenancy in Kubernetes. In this inaugural post, we will dive into isolating workloads, managing shared cluster resources, enforcing strict security boundary conditions with Namespaces and Network Policies, and balancing cost efficiency against robust tenant isolation.

The Data Plane & Storage Subsystem: eBPF Firewalls, CSI Scoping, and Cryptographic Isolation

Multi-tenancy in Kubernetes does not end when the API server rejects an unauthorized request.

In the first two installments of our Into The Depths of Kubernetes multi-tenancy series, we moved beyond basic logical namespace isolation to dissect the underlying control-plane and node-level bottlenecks. In Part 1 exploded the Kubernetes control plane, examining API request concurrency, how API Priority and Fairness (APF) prevents request saturation, CRD name-collision risks, and the case for virtual control planes like vCluster. Part 2 took us a layer deeper into the Linux kernel, demystifying hard workload sandboxing through cgroups v1 vs. v2 hierarchies, CPU throttling traps, OOM killer mechanics, and secure runtime boundaries like gVisor and Kata Containers. Together, these posts laid the architectural and kernel-level foundations required to truly understand high-density, multi-tenant cluster design.

That is where things get considerably more interesting.

A compromised container does not necessarily care about your carefully designed RBAC policies. It cares about what it can reach over the network, what it can discover through DNS, what interfaces it can access, and what data it can read from mounted storage. If Tenant A can reach Tenant B's backend service, or if a compromised pod can escape its volume boundary and read another tenant's persistent data, the isolation model has already failed regardless of how well the Kubernetes API is protected.

This final part of the series moves beneath the Kubernetes API and looks at the infrastructure where tenant workloads actually exchange packets and persist data.

The objective is simple: make tenant boundaries hold even when the workload itself is no longer trustworthy.

The Data Plane Is Where Multi-Tenancy Gets Real

There is a common mistake in multi-tenant Kubernetes design: treating namespaces as if they were security boundaries by themselves.

They are not.

A namespace gives Kubernetes a logical scope for resources. RBAC can control who can interact with that namespace. ResourceQuota can control how much a tenant consumes. Admission policies can restrict what workloads are allowed to create. But none of those mechanisms automatically prevents a running workload from communicating with another workload or accessing data exposed through a poorly isolated storage subsystem.

Consider a cluster hosting three tenants:

                    Kubernetes Cluster
                           │
          ┌────────────────┼────────────────┐
          │                │                │
       Tenant A         Tenant B         Tenant C
          │                │                │
      Web/API Pods     Web/API Pods     Web/API Pods
          │                │                │
          └────────── Network ──────────────┘
                           │
                       Storage
                           │
                 Shared Infrastructure

If Tenant A compromises one of its containers, the attack does not stop at the container boundary. The attacker will typically start looking sideways.

Can I resolve another tenant's service?
Can I connect to its database?
Can I access an internal metrics endpoint?
Can I reach the Kubernetes API?
Can I discover mounted files?
Can I access another tenant's persistent volume?

The answer to those questions is determined primarily by the data plane, not the control plane.

That is why serious multi-tenancy requires two independent assumptions:

Network isolation must survive a compromised workload.

and

Storage isolation must survive access to the underlying storage infrastructure.

The first requires strong packet-level enforcement. The second requires storage boundaries that do not depend solely on directory permissions or mount configuration.

This is where eBPF networking and cryptographic storage isolation become particularly useful.

Bypassing IPTables: The eBPF Networking Revolution

For years, Kubernetes networking security has been heavily associated with iptables.

The basic model is straightforward. A Kubernetes NetworkPolicy is translated by the networking implementation into rules that eventually influence packet filtering in the Linux networking stack. As workloads and policies grow, however, the amount of state that the node has to maintain can become significant.

The problem is not that iptables suddenly becomes insecure. The problem is scale and enforcement efficiency.

Traditional iptables processing is fundamentally rule-oriented. Packets encounter chains containing rules that must be evaluated until a matching condition determines the result. In a small cluster this is rarely something you lose sleep over. In a large multi-tenant environment with thousands of workloads, services, endpoints, and policy relationships, the number of rules and the amount of churn can become substantial.

Every workload event can potentially change the effective networking state. A new pod appears. An endpoint changes. A service is updated. A NetworkPolicy changes. A node receives more workloads. The networking layer must continuously reconcile these changes.

With enough tenants and enough policy relationships, this becomes another scalability problem in the node data plane.

eBPF Changes Where the Enforcement Happens

eBPF approaches the problem from a different direction.

Instead of relying exclusively on large rule chains, an eBPF-based networking implementation can load programs directly into the Linux kernel and attach them at strategic points in the networking path. The program can inspect packet metadata, identities, sockets, ports, protocols, and other context and make an enforcement decision without requiring every packet to walk through a massive iptables rule set.

This is one reason projects such as Cilium have become important in Kubernetes networking.

A simplified conceptual path looks like this:

Traditional:

Pod
 │
 ▼
Linux Networking Stack
 │
 ▼
iptables chains
 │
 ├── Rule 1
 ├── Rule 2
 ├── Rule 3
 ├── Rule 4
 ├── ...
 └── Rule N
 │
 ▼
Destination


eBPF-oriented:

Pod
 │
 ▼
eBPF Hook
 │
 ├── Identity lookup
 ├── Policy lookup
 ├── Security decision
 │
 ├── DROP
 └── ALLOW
 │
 ▼
Destination

The important part is not simply that eBPF is "faster."

The more interesting property for multi-tenancy is that the networking layer can reason about workload identity and enforce policy closer to the point where traffic enters the kernel networking path.

That gives the platform considerably more control over lateral movement.

Instead of thinking only in terms of:

Source IP → Destination IP → Port

modern Kubernetes networking can reason more naturally about:

Tenant
  ↓
Namespace
  ↓
Workload identity
  ↓
Service identity
  ↓
Policy

That shift is extremely useful in environments where IP addresses are ephemeral and workloads are continuously recreated.

Identity Matters More Than IP Addresses

IP-based security rules are awkward in Kubernetes because IP addresses are temporary implementation details.

A pod disappears. Another pod replaces it. The replacement receives a different address. The security relationship, however, has not changed.

If the security model is "Tenant A's frontend can talk to Tenant A's backend," you do not actually care whether the backend currently has 10.42.4.18 or 10.42.7.31. You care about who owns that workload and what role it plays. This is where identity-aware networking becomes valuable.

A policy can conceptually express:

Allow:
tenant-a/frontend
        ↓
tenant-a/backend:8080

while denying:

tenant-a/frontend
        X
tenant-b/backend:8080

The distinction becomes particularly important when tenants share nodes. Two pods belonging to completely different customers may be running side by side on the same Linux host, and the network policy needs to maintain that boundary regardless of their physical placement.

That is the data-plane version of multi-tenancy.

Default-Deny-All Is the Starting Point

If tenants are supposed to be isolated, the safest starting assumption is that nothing should communicate unless it has been explicitly permitted.

A default-deny policy is therefore one of the most important controls in a multi-tenant cluster.

A basic Kubernetes NetworkPolicy can establish this boundary:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

This does something intentionally boring.

It says:

Every pod in tenant-a starts with no allowed ingress or egress traffic.

That is exactly what we want from a zero-trust starting point. But there is an immediate problem. A completely isolated namespace cannot do much useful work. Applications still need DNS. They may need access to an internal database. They may need to communicate with an ingress gateway. They may need access to a monitoring endpoint.

The solution is not to abandon default deny. The solution is to add small, explicit exceptions.

Allowing DNS Without Opening the Door

DNS is one of the easiest places to accidentally weaken a supposedly strict NetworkPolicy.

Kubernetes workloads commonly use CoreDNS through the cluster DNS service. If DNS traffic is blocked, applications begin failing in ways that look completely unrelated to networking policy.

A production policy therefore normally needs an explicit DNS exception.

For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

The important detail is that DNS is being allowed to the DNS workload, rather than allowing unrestricted traffic to the entire kube-system namespace.

That distinction matters.

This:

tenant-a → kube-system

is broad.

This:

tenant-a → CoreDNS:53

is narrow. The principle should always be:

Allow the dependency, not the infrastructure namespace.

The same model applies to other shared services. If tenants need access to a central observability endpoint, expose only the required destination and port. If they need an ingress gateway, allow the gateway identity rather than the entire ingress namespace.

Building a Tenant Network Boundary

Once DNS is handled, the tenant network can become deliberately restrictive.

A simplified architecture might look like this:

                     Cluster
                        │
        ┌───────────────┼────────────────┐
        │               │                │
    Tenant A         Tenant B         Tenant C
        │               │                │
   ┌────┴────┐     ┌────┴────┐     ┌────┴────┐
   │ Frontend│     │ Frontend│     │ Frontend│
   └────┬────┘     └────┬────┘     └────┬────┘
        │               │                │
   Backend A       Backend B        Backend C
        │               │                │
        X               X                X
        │               │                │
        └────── Cross-Tenant DENY ───────┘

Within the tenant, you can then explicitly define allowed paths:

Frontend
   │
   ├── DNS
   │
   └── Backend:8080

Backend
   │
   ├── DNS
   │
   └── Database:5432

Everything else remains denied. This is much easier to reason about than trying to enumerate every possible unwanted connection. The security model becomes:

Default = DENY

Explicitly required dependency = ALLOW

That is a much stronger foundation for multi-tenancy.

The Storage Loophole

Network isolation solves only half of the data-plane problem. The other half is storage. And storage is where many seemingly secure Kubernetes environments have a surprisingly large hole.

Imagine two tenants:

Tenant A → PVC-A
Tenant B → PVC-B

At the Kubernetes API level, these appear completely independent. But eventually those PVCs map to something underneath:

PVC-A ──► PV-A ──► Storage Backend
PVC-B ──► PV-B ──► Storage Backend

The Kubernetes abstraction can make those volumes look isolated while the underlying implementation may not be.

This becomes especially dangerous when operators use hostPath, shared NFS exports, custom CSI drivers, or storage backends that were not designed with hostile tenants in mind.

CSI Is a Security Boundary, Not Just a Storage Plugin

The Container Storage Interface provides a standard mechanism through which Kubernetes interacts with storage systems.

That standardization is useful, but CSI itself does not magically make storage multi-tenant safe. The security depends on how the driver provisions, mounts, and exposes the underlying storage.

Consider a simplistic hostPath-style arrangement:

Node
│
└── /data
    ├── tenant-a/
    │   ├── database.db
    │   └── secrets/
    │
    └── tenant-b/
        ├── database.db
        └── secrets/

A pod might receive:

/data/tenant-a

as its mounted filesystem. The assumption is that the container can only see that directory. But if the underlying hostPath configuration, privileged permissions, mount propagation, or driver implementation is flawed, that boundary can become surprisingly weak.

A compromised workload should never be trusted simply because Kubernetes mounted a particular directory into the container.

Directory Traversal Is Only One Failure Mode

The more serious problem is that storage isolation can fail at several layers.

You can have:

Container filesystem isolation
        ↓
Kubernetes volume isolation
        ↓
CSI driver isolation
        ↓
Node filesystem isolation
        ↓
Storage backend isolation
        ↓
Physical/cloud storage

A weakness at any layer can undermine the layers above it. A shared NFS server provides another example. Suppose multiple tenants receive directories from the same export:

NFS Export
│
├── /tenant-a
├── /tenant-b
└── /tenant-c

If permissions, UID/GID mapping, root squashing, export configuration, or CSI mount behavior is incorrectly configured, a compromised workload may be able to interact with storage that belongs to another tenant.

The critical point is that filesystem permissions are not the same thing as cryptographic isolation.

If two tenants ultimately depend on the same encryption key, a compromised storage credential, snapshot, backup, or underlying storage access path may still expose both datasets.

Zero-Trust Storage: Encryption Per Tenant

This leads to a stronger model.

Instead of asking:

"Can Tenant A access Tenant B's directory?"

ask:

"Even if Tenant A somehow obtains the underlying storage bytes, can it decrypt Tenant B's data?"

That is a much stronger security boundary.

The architecture becomes:

                Kubernetes
                    │
             Tenant Namespace
                    │
                   PVC
                    │
                  CSI
                    │
          ┌─────────┴─────────┐
          │                   │
     Volume Provisioning   Key Request
          │                   │
          ▼                   ▼
    Cloud Block Storage      KMS
                              │
                    ┌─────────┴─────────┐
                    │                   │
               Tenant-A Key        Tenant-B Key

The critical property is that the storage encryption key becomes part of the tenant isolation model. Tenant A does not simply receive a different directory. Tenant A receives a volume encrypted under a key that Tenant B cannot use.

CSI + KMS: The Architecture

A production implementation can integrate a CSI driver with a Key Management Service.

Depending on the environment, the KMS might be:

AWS KMS
HashiCorp Vault
Azure Key Vault
Google Cloud KMS
Another enterprise key-management platform

The workflow looks roughly like this:

1. Tenant creates PVC
          │
          ▼
2. Kubernetes requests volume
          │
          ▼
3. CSI driver provisions storage
          │
          ▼
4. CSI requests encryption material
          │
          ▼
5. KMS authorizes tenant-specific key
          │
          ▼
6. Volume is encrypted
          │
          ▼
7. Pod receives mounted volume

The important part is step five. The KMS authorization layer should understand which tenant is requesting the key.

For example:

Tenant A
   │
   └──► kms/tenant-a-key

Tenant B
   │
   └──► kms/tenant-b-key

Tenant A should not have permission to request or unwrap Tenant B's encryption key. This creates a security boundary outside Kubernetes itself.

Why Encryption Changes the Threat Model

Consider an attacker who somehow obtains the raw storage device or snapshot.

Without encryption:

Attacker
   │
   ▼
Storage Snapshot
   │
   ▼
Read Database Files

With tenant-specific encryption:

Attacker
   │
   ▼
Storage Snapshot
   │
   ▼
Encrypted Bytes
   │
   X
   │
Missing Tenant-B Key

The data is still physically present. The attacker simply cannot turn those bytes into meaningful information without the cryptographic material required to decrypt them.

That distinction becomes extremely important for cloud environments, snapshots, backups, storage migrations, and infrastructure operators who may have access to the storage layer but should not automatically have access to tenant data.

KMS Policy Becomes Part of Kubernetes Security

This is where the architecture becomes more interesting. You now have security controls distributed across multiple layers:

Kubernetes RBAC
       │
       ▼
Namespace Boundary
       │
       ▼
NetworkPolicy / eBPF
       │
       ▼
CSI Provisioning
       │
       ▼
KMS Authorization
       │
       ▼
Encrypted Storage

The tenant's identity must therefore propagate through the infrastructure correctly. A Kubernetes ServiceAccount might identify the workload.

The CSI driver provisions the volume. The cloud identity layer determines which KMS operation the workload or infrastructure component can perform.

The KMS policy then determines which encryption key can actually be used. This produces something much closer to a true zero-trust storage model.

A Practical Multi-Tenant Storage Model

For a serious multi-tenant platform, I would think about storage in terms of three independent boundaries.

Boundary 1: Provisioning

A tenant should only be able to request storage through approved StorageClasses and CSI drivers. Avoid giving tenants arbitrary access to hostPath volumes. Avoid allowing workloads to manipulate mounts. Avoid exposing privileged storage operations to ordinary application pods.

Boundary 2: Physical Isolation

Where practical, separate tenants at the storage backend.

For example:

Tenant A → Volume Group A
Tenant B → Volume Group B
Tenant C → Volume Group C

This does not necessarily mean dedicated physical hardware. It can mean separate logical storage resources, access policies, projects, accounts, or backend namespaces depending on the storage platform.

The goal is to reduce the blast radius of a storage-layer compromise.

Boundary 3: Cryptographic Isolation

Finally:

Tenant A → Key A
Tenant B → Key B
Tenant C → Key C

Even if the physical storage boundary is bypassed, the cryptographic boundary remains.

This is the strongest layer because it does not depend on directory names, Linux permissions, or Kubernetes object ownership.

The Real Architecture

Put everything together and the multi-tenant data plane begins to look like this:

                         Kubernetes Cluster
                                │
              ┌─────────────────┴─────────────────┐
              │                                   │
         CONTROL PLANE                         DATA PLANE
              │                                   │
        RBAC / Admission                     eBPF / CNI
              │                                   │
        Namespace Policy                 Default Deny
              │                                   │
              │                         Explicit Allow
              │                                   │
              │                         Tenant Identity
              │                                   │
              │                                   ▼
              │                            Network Boundary
              │
              └──────────────────────┬────────────────────
                                     │
                                    CSI
                                     │
                            Storage Provisioning
                                     │
                                     ▼
                                   KMS
                                     │
                    ┌────────────────┼────────────────┐
                    │                │                │
                 Key A            Key B             Key C
                    │                │                │
                    ▼                ▼                ▼
                 Tenant A         Tenant B         Tenant C
                  Volume           Volume           Volume

There is no single magical Kubernetes feature providing this isolation. It is the composition of several independent security boundaries. That is the important lesson. ## What Happens When One Layer Fails?

This is perhaps the most useful way to evaluate a multi-tenant platform. Assume Tenant A's application is compromised.

Scenario 1: Network compromise

The attacker attempts:

Tenant A → Tenant B

The eBPF or NetworkPolicy layer rejects the traffic.

Scenario 2: Service discovery abuse

The attacker attempts to discover internal services. DNS may still work, because DNS is explicitly required. But DNS access does not automatically provide network access to the discovered service.

That distinction is important.

DNS lookup = ALLOWED

Connection to Tenant B backend = DENIED

Scenario 3: Storage compromise

The attacker somehow gains access to another tenant's storage bytes. The volume data is encrypted. Without the appropriate KMS key, the raw bytes are useless.

Scenario 4: Node compromise

This is where things become significantly more difficult.

If an attacker obtains full root access to a Kubernetes worker node, many workload-level security assumptions are already under pressure. This is why node hardening, confidential computing where appropriate, restricted privileges, kernel security, runtime isolation, and careful separation of highly sensitive workloads remain important.

Multi-tenancy is therefore not about creating an absolute wall.

It is about creating multiple independent barriers so that compromising one layer does not automatically compromise everything above or below it.

The Platform Engineer's Final Checklist

Before calling a Kubernetes cluster genuinely multi-tenant, I would want to answer these questions clearly.

Networking

Is every tenant namespace default-deny?
Are cross-namespace connections explicitly controlled?
Is DNS allowed only to the required CoreDNS endpoints?
Are policies identity-aware where possible?
Can a compromised workload reach node-local services?
Can workloads bypass the CNI?
Are privileged pods restricted?

Storage

Are hostPath volumes prohibited for ordinary tenants?
Is the CSI driver trusted and correctly configured?
Can a tenant access another tenant's mount?
Is shared NFS configured with strong isolation?
Are volume snapshots isolated?
Are backups protected by the same tenant boundary?
Can storage credentials be reused across tenants?

Cryptography

Does each tenant have an independent encryption boundary?
Can Tenant A request Tenant B's key?
Are KMS permissions tied to workload or tenant identity?
Are encryption keys rotated?
Are snapshots encrypted?
Are backup copies encrypted?
What happens when a tenant is deleted?

That final question is often forgotten. Deleting a namespace does not necessarily mean deleting its data. A mature platform must define what happens to:

PVC
 ↓
Volume
 ↓
Snapshot
 ↓
Backup
 ↓
Encryption Key

Tenant offboarding is therefore also a cryptographic lifecycle problem.

The Final Layer of Multi-Tenancy

Multi-tenancy is often introduced as a Kubernetes namespace problem.

It is not.

Namespaces provide a useful logical boundary, but serious isolation requires the platform to enforce boundaries across the entire workload lifecycle.

The control plane decides who is allowed to do what. The scheduler decides where workloads can run. The network layer decides who can communicate with whom. The storage layer decides who can access which data.

And the cryptographic layer decides whether that data is useful even when the underlying storage boundary has been breached.

That is why eBPF, CSI architecture, and KMS integration matter so much in advanced Kubernetes environments. They push tenant isolation below the Kubernetes object model and into the infrastructure that actually carries packets and stores bytes.

A useful way to think about the entire series is:

             Multi-Tenant Kubernetes
                      │
       ┌──────────────┼──────────────┐
       │              │              │
   Control Plane    Workloads      Data Plane
       │              │              │
      RBAC         Runtime        eBPF/CNI
    Admission      Security       NetworkPolicy
    Quotas         Scheduling         │
       │              │              │
       └──────────────┼──────────────┘
                      │
                     CSI
                      │
                     KMS
                      │
             Cryptographic Boundary

And that is ultimately the difference between running multiple tenants on Kubernetes and building Kubernetes as a multi-tenant platform.

The second requires you to assume that eventually, something will fail.

A container will be compromised. A credential will leak. A policy will be misconfigured. A node will behave unexpectedly. A storage snapshot will escape the normal workflow.

The question is not whether you can prevent every possible failure. The question is whether one failure gives an attacker the keys to everything else.

A well-designed multi-tenant Kubernetes platform says no.

Network boundaries stop lateral movement. CSI boundaries restrict storage access. KMS-backed encryption protects data even when lower-level assumptions fail. And eBPF gives the data plane a way to enforce those decisions efficiently as the number of workloads and tenants grows.

That is where multi-tenancy becomes more than namespace isolation. It becomes an exercise in containing failure, limiting blast radius, and making every layer assume that the layer beneath it might eventually be compromised.

That is the real depth of Kubernetes multi-tenancy.

Into The Depths of Kubernetes: Multi-Tenancy Part 2

Kubernetes with Naveen — Fri, 25 Sep 2026 12:00:53 +0000

Part 2: Kernel Contention & Hard Sandboxing — Cgroups, OOM Kills, and Runtime Escapes

Containers do not exist.

At least, not in the way most Kubernetes users imagine them.

Underneath the YAML, Pods, Deployments, Services, and container images, there are still ordinary Linux processes running on a shared Linux kernel.

That distinction becomes extremely important when multiple organizations, teams, customers, or workloads are sharing the same Kubernetes cluster.

In Part 1 of our series, "Into The Depths of Kubernetes: Multi-Tenancy," we looked beyond the basic "just use namespaces" advice to expose the architectural limits of logical isolation. We explored how shared control-plane components—like the API server, etcd, and controller managers—remain vulnerable to noisy-neighbor workloads and concurrency bottlenecks. The post examined how API Priority and Fairness mitigates request saturation, why cluster-scoped CRDs introduce critical tenant collisions, and how virtual control planes like vCluster provide true control-plane isolation. If you missed it or need a refresher, check out Part 1 to see how we laid the groundwork for hard multi-tenancy.

Part 2 takes us one layer deeper.

We are leaving the Kubernetes API and entering the Linux kernel.

Because a namespace does not create a separate machine. A ResourceQuota does not create a separate CPU. A Pod security policy does not magically create a second kernel. And a container boundary, by itself, does not mean that a tenant has been isolated from every other process on the node.

What Kubernetes gives you is a carefully constructed set of abstractions around Linux primitives such as namespaces, cgroups, capabilities, seccomp, LSMs, and container runtimes.

The uncomfortable question is what happens when those boundaries are pushed hard.

What happens when Tenant A consumes memory faster than the node can provide it? What happens when one workload generates enormous amounts of filesystem writeback? What happens when the kernel has to kill something? And perhaps the most important question for genuinely untrusted workloads:

What happens when a process inside a container finds a way to attack the kernel itself?

That is where Kubernetes multi-tenancy stops being primarily a scheduling problem and becomes a kernel-isolation problem.

1. The Container Boundary Is Not the Kernel Boundary

A conventional container runtime such as containerd or CRI-O does not boot a miniature operating system for every container.

Instead, the container process runs directly on the host kernel.

Linux namespaces make the process believe it has its own process tree, network interfaces, mounts, hostname, and other resources. Cgroups control how much CPU, memory, and other resources the process can consume. Capabilities reduce what privileged operations it can perform, while seccomp can restrict which system calls it is allowed to invoke.

But there is still one kernel underneath all of it.

                    Kubernetes Node
┌─────────────────────────────────────────────────────────────┐
│                                                             │
│                    Linux Kernel                              │
│                                                             │
│   ┌────────────────┐   ┌────────────────┐   ┌────────────┐ │
│   │    Tenant A    │   │    Tenant B    │   │  Tenant C  │ │
│   │                │   │                │   │            │ │
│   │   Container    │   │   Container    │   │ Container  │ │
│   │                │   │                │   │            │ │
│   └───────┬────────┘   └───────┬────────┘   └─────┬──────┘ │
│           │                     │                   │        │
│      namespaces             namespaces          namespaces  │
│      cgroups                cgroups             cgroups     │
│      seccomp                seccomp             seccomp     │
│           │                     │                   │        │
│           └─────────────────────┼───────────────────┘        │
│                                 │                            │
│                         Shared Kernel                        │
│                                 │                            │
└─────────────────────────────────┼───────────────────────────┘
                                  │
                          Physical Hardware

This architecture is one of the reasons containers are so efficient. There is no guest kernel per workload, so container startup is fast and resource overhead is relatively small.

It is also the reason the host kernel becomes part of your multi-tenancy threat model.

A vulnerability in an application is normally an application problem. A vulnerability in a container runtime may become a node problem. But a vulnerability in a kernel subsystem can potentially become a cluster isolation problem, because every ordinary container on that node ultimately interacts with the same kernel.

That is the fundamental trade-off we are going to explore in this part.

2. Cgroups: Where Kubernetes Resource Limits Actually Become Real

Kubernetes users normally interact with resources through something familiar:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"

  limits:
    cpu: "1"
    memory: "1Gi"

It is easy to think that Kubernetes itself is continuously watching the process and enforcing these limits.

It isn't.

The Kubelet translates Kubernetes resource configuration into Linux control-group configuration. The kernel then becomes responsible for enforcing those controls.

The important path looks roughly like this:

Pod YAML
   │
   ▼
Kubernetes API
   │
   ▼
Scheduler
   │
   ▼
Kubelet
   │
   ▼
Container Runtime
   │
   ▼
Linux cgroup configuration
   │
   ▼
Linux Kernel
   │
   ├── CPU accounting / throttling
   ├── Memory accounting
   ├── Memory reclaim
   ├── OOM handling
   └── I/O control

That last layer is where multi-tenancy becomes interesting.

3. cgroups v1 vs cgroups v2: The Hierarchy Problem

For years, Linux resource control was built around cgroups v1.

The problem with v1 was not that it could not limit resources. It could. The problem was that different controllers were organized around separate hierarchies. CPU could have one hierarchy. Memory could have another. Block I/O could have another. And this created increasingly complicated behavior as workloads became more sophisticated.

                    cgroups v1

              ┌─────────────────┐
              │  CPU hierarchy  │
              └─────────────────┘
                       │
                 Tenant A/B/C

              ┌─────────────────┐
              │ Memory hierarchy│
              └─────────────────┘
                       │
                 Tenant A/B/C

              ┌─────────────────┐
              │  I/O hierarchy  │
              └─────────────────┘
                       │
                 Tenant A/B/C

The result was not simply an administrative inconvenience. Resource accounting itself could become complicated, especially around filesystem I/O.

One particularly nasty area was buffered writeback.

A process can write data to a filesystem without immediately forcing all of that data to the physical storage device. Linux can temporarily hold dirty pages in memory and flush them later through writeback mechanisms.

Historically, cgroups v1 had limitations around consistently attributing this writeback activity to the workload that caused it.

That matters in a multi-tenant environment.

Imagine Tenant A running a workload that continuously generates enormous amounts of filesystem writes while Tenant B operates a latency-sensitive service on the same node.

Tenant A may not simply consume "disk bandwidth" in the obvious way. The actual work can involve page cache, dirty memory, asynchronous writeback, filesystem activity, block-layer processing, and device queues.

The performance cost can therefore appear somewhere other than the original application process.

That is exactly the kind of problem that makes noisy-neighbor analysis painful.

4. cgroups v2: One Hierarchy, Better Accounting

cgroups v2 was designed around a unified hierarchy.

Instead of treating resource controllers as completely independent trees, cgroups v2 provides a common hierarchy through which controllers can participate in a consistent resource-management model.

                     cgroups v2
                         │
                    Unified Tree
                         │
              ┌──────────┼──────────┐
              │          │          │
           Tenant A   Tenant B   Tenant C
              │          │          │
           ┌──┼──┐    ┌──┼──┐    ┌──┼──┐
           │  │  │    │  │  │    │  │  │
          CPU MEM IO  CPU MEM IO  CPU MEM IO

This does not magically eliminate noisy neighbors.

It does something more important: it gives Linux a more coherent model for managing resources across a hierarchy.

Memory accounting becomes more consistent. CPU controls become integrated into the same tree. I/O controls can participate in the hierarchy, and writeback attribution has been substantially improved compared with the older cgroup model.

For Kubernetes operators, this matters because the container boundary is only as strong as the kernel's ability to accurately account for the work performed by that container.

If the kernel cannot correctly associate resource consumption with the workload that caused it, the cluster administrator eventually ends up debugging symptoms rather than causes.

That is one reason modern Kubernetes environments increasingly care about the underlying cgroup version rather than treating it as an invisible implementation detail.

5. CPU Limits: The Throttling Trap

CPU limits are another place where Kubernetes abstractions eventually become kernel mechanics.

Suppose a container has:

limits:
  cpu: "1"

This does not mean the process has been assigned a dedicated CPU core.

It means the workload is constrained by Linux CPU control mechanisms, historically through CFS quota and period semantics and, depending on the environment, newer cgroup CPU controls.

If the workload attempts to consume more CPU than its configured allowance, it can be throttled.

That distinction becomes important when tenants run workloads with very different CPU behavior.

A batch-processing tenant may happily consume CPU whenever available. A latency-sensitive API may only require a fraction of a core but becomes extremely sensitive to scheduling delays.

Both workloads can technically remain within their configured resource boundaries while producing very different application-level consequences.

Tenant A
CPU-heavy batch workload
        │
        ▼
 ┌───────────────┐
 │ CPU pressure  │
 └───────┬───────┘
         │
         ▼
Kernel scheduler / cgroup controls
         │
         ├──────────────► Tenant A throttled
         │
         └──────────────► Shared CPU contention
                              │
                              ▼
                       Tenant B latency rises

This is why simply saying "both tenants have CPU limits" does not completely describe isolation. Resource limits control consumption. They do not guarantee identical performance.

6. Memory Is Different: Eventually Somebody Has to Die

CPU contention can result in throttling. Memory is less forgiving.

When a node runs short of memory, Linux can reclaim memory through several mechanisms, but eventually there may be insufficient reclaimable memory to satisfy a new allocation.

At that point, the kernel's Out-Of-Memory killer becomes involved. This is one of the most misunderstood parts of Kubernetes resource management.

People often say:

"The container exceeded its memory limit, so Kubernetes killed it."

That is an incomplete description.

The actual behavior depends on whether the OOM occurs within a constrained cgroup, whether the cgroup is configured for OOM handling, whether the node itself is under memory pressure, and how the kernel evaluates possible victims.

The kernel does not think in terms of Kubernetes Pods. It thinks in terms of processes and memory pressure.

7. Anatomy of an OOM Kill

At a high level, when Linux cannot satisfy a memory allocation after reclaim attempts, it enters an OOM path and evaluates processes as potential victims.

One of the values involved is:

oom_score

and one of the most important modifiers is:

oom_score_adj

The adjustment allows userspace to influence how attractive a process is as an OOM victim. Kubernetes uses this mechanism as part of its QoS model.

The simplified relationship looks like this:

                 Node Memory Pressure
                         │
                         ▼
                 Kernel OOM Decision
                         │
              ┌──────────┴──────────┐
              │                     │
       oom_score_adj          Memory footprint
              │                     │
              └──────────┬──────────┘
                         ▼
                  Candidate process
                         │
                         ▼
                       KILL

For Kubernetes workloads, the important point is that QoS class influences OOM protection.

8. Guaranteed, Burstable, and BestEffort

Kubernetes classifies Pods into QoS categories based on their resource configuration.

At the broadest level:

QoS Class	Resource Configuration	OOM Protection
Guaranteed	CPU and memory requests equal limits for every container	Strongest
Burstable	Some resource requests/limits are defined but don't meet Guaranteed criteria	Intermediate
BestEffort	No CPU/memory requests or limits	Weakest

For a Guaranteed Pod, Kubernetes assigns an oom_score_adj close to the most protected end of the scale, traditionally -997.

BestEffort workloads sit at the opposite extreme, with an oom_score_adj of 1000.

Burstable workloads sit between these extremes, with the adjustment derived from their requested memory relative to node capacity, subject to Kubernetes' implementation rules.

That middle category is where things get interesting. Because "Burstable" sounds safer than it actually is.

9. The Burstable Trap

Consider two tenants.

Tenant A:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "8Gi"

Tenant B:

resources:
  requests:
    memory: "4Gi"
  limits:
    memory: "4Gi"

Tenant A is Burstable.

Tenant B may qualify as Guaranteed, assuming its CPU configuration also satisfies the requirements.

Tenant A has effectively said:

"Schedule me as though I need 256 MiB, but permit me to consume substantially more."

That is perfectly legitimate Kubernetes behavior. It is also where multi-tenant systems can get into trouble.

If a large number of tenants make aggressive use of the gap between requests and limits, the scheduler may place workloads based on relatively modest requested resources while the node can experience substantially higher real memory consumption.

Scheduler view:

Tenant A → Request: 256 MiB
Tenant B → Request: 256 MiB
Tenant C → Request: 512 MiB
Tenant D → Request: 1 GiB

              ↓

Node appears schedulable

              ↓

Actual runtime consumption:

A → 5 GiB
B → 3 GiB
C → 4 GiB
D → 6 GiB

              ↓

Memory pressure
              ↓
        OOM / reclaim

This is why Burstable is not synonymous with "safe isolation."

A tenant can be perfectly valid according to Kubernetes scheduling rules while still generating significant memory pressure at runtime.

And when that pressure reaches the node, the consequences are shared.

10. The OOM Killer Doesn't Understand Your Business

This is one of the most important operational realities in shared Kubernetes clusters.

The kernel does not know that:

Tenant A owns the payment API.
Tenant B owns a development environment.
Tenant C is running a background batch job.
Tenant D is running a critical production database.

It knows about processes, memory consumption, and OOM scoring. Your business priorities exist above the kernel. Your OOM decision exists below them. That means Kubernetes operators must translate business criticality into enforceable resource configuration.

QoS classes, requests, limits, node pools, eviction thresholds, priority classes, and workload placement are not independent features. Together, they form the practical memory-isolation strategy of the cluster.

11. When a Container Becomes a Kernel Problem

So far, everything we've discussed has been about accidental contention.

But multi-tenancy has another class of failure:

malicious contention.

A normal container shares the host kernel.

Every application eventually performs operations through system calls:

Application
    │
    │ system calls
    ▼
┌─────────────────────┐
│ Linux system calls  │
└──────────┬──────────┘
           │
           ▼
     Host Kernel
           │
    ┌──────┼──────┐
    ▼      ▼      ▼
 Network  Files  Memory

A container process may execute operations such as:

open()
read()
write()
mmap()
clone()
socket()
ioctl()
mount()
setns()

The exact set available to an ordinary container depends on its security configuration, capabilities, seccomp profile, user namespaces, LSM policy, and runtime behavior.

But the important architectural fact remains: the syscall eventually enters the host kernel.

12. The Syscall Attack Surface

A Linux kernel is enormous. It handles process scheduling, virtual memory, networking, filesystems, storage, namespaces, IPC, security mechanisms, and much more. That makes the kernel one of the largest pieces of privileged software in the entire platform.

A vulnerability in an application may compromise that application. A kernel vulnerability can potentially cross the isolation boundary.

The attack pattern can look conceptually like this:

Untrusted Tenant
       │
       ▼
Malicious process
       │
       ▼
Crafted system call
       │
       ▼
Kernel subsystem
       │
       ▼
Kernel vulnerability
       │
       ▼
Privilege escalation
       │
       ▼
Host compromise
       │
       ▼
Other tenants / node

This is what makes container escape fundamentally different from an ordinary application exploit. The attacker is not necessarily trying to break the application. They are trying to break the assumption that the container boundary is sufficient.

13. containerd and CRI-O Do Not Give You a Second Kernel

containerd and CRI-O are responsible for managing containers and integrating with Kubernetes through the CRI ecosystem.

They provide an important abstraction layer between Kubernetes and the low-level runtime. But neither one changes the fundamental architecture of conventional Linux containers.

Kubernetes
     │
     ▼
Kubelet
     │
     ▼
CRI
     │
 ┌───┴────┐
 │        │
containerd   CRI-O
 │        │
 ▼        ▼
OCI Runtime
 │
 ▼
Linux Container
 │
 ▼
Shared Host Kernel

If an attacker can exploit a vulnerable kernel interface that is reachable from inside the container, changing from containerd to CRI-O does not magically turn the workload into a hardware-isolated virtual machine.

The runtime can significantly improve the security boundary through seccomp, capabilities, namespaces, AppArmor/SELinux, user namespaces, and other controls. But the kernel remains shared.

That distinction is the foundation for understanding hard sandboxing.

14. The Engineering Question: How Much Isolation Do You Actually Need?

There is no universally correct answer.

If you are running your own trusted microservices, a conventional container runtime with a hardened configuration may be entirely appropriate.

If you are running arbitrary customer code, CI jobs submitted by unknown users, plugin systems, serverless functions, or workloads that intentionally execute untrusted binaries, the threat model changes.

Now the question becomes:

"How much of the host kernel do I want an untrusted workload to directly interact with?"

And this is where sandboxed runtimes enter the architecture.

Three technologies are particularly interesting:

gVisor
Kata Containers
Firecracker

They all try to improve isolation. They simply do it at different layers.

15. gVisor: Put a Kernel-Like Boundary in User Space

gVisor takes a fundamentally different approach from a traditional VM.

Instead of allowing the application to interact directly with the host kernel for every supported operation, gVisor introduces a user-space component called Sentry, commonly used through the runsc runtime.

Conceptually:

             Tenant Workload
                    │
                    ▼
             Application
                    │
                    ▼
             System Calls
                    │
                    ▼
          ┌─────────────────┐
          │ gVisor Sentry   │
          │  User Space     │
          └────────┬────────┘
                   │
                   ▼
            Host Kernel

The Sentry implements a substantial portion of Linux's kernel interfaces in user space. The application therefore interacts primarily with the Sentry rather than directly exposing the full host kernel surface.

This reduces the amount of host-kernel functionality directly exposed to the workload. The trade-off is obvious once you look at the architecture. You have introduced another compatibility and performance layer.

Applications that expect unusual Linux kernel behavior may encounter compatibility limitations, and workloads that perform heavily syscall-intensive operations can experience measurable overhead.

But for workloads where security isolation is more important than squeezing every last percentage point of performance from a conventional container, that trade-off can make sense.

16. Kata Containers: Put the Workload Behind a MicroVM

Kata Containers takes the isolation boundary further. Rather than building a user-space kernel-like layer, Kata launches workloads inside lightweight virtual machines.

             Kubernetes Pod
                   │
                   ▼
            Kata Runtime
                   │
                   ▼
              MicroVM
       ┌─────────────────────┐
       │ Guest Kernel        │
       │                     │
       │ Container Workload  │
       └──────────┬──────────┘
                  │
            Virtual Hardware
                  │
                  ▼
             Host Kernel

Now the workload does not share the host kernel in the same way as a conventional Linux container. The guest kernel becomes part of the isolation boundary.

That gives Kata a much stronger boundary against classes of attacks that rely on exploiting the host kernel through normal container syscall access.

But virtualization has a cost. There is a guest kernel to manage. There is virtual hardware. There are additional memory structures. There is additional startup and runtime overhead.

Modern hardware virtualization makes this significantly lighter than traditional heavyweight virtual machines, but "lighter VM" does not mean "zero overhead."

17. Firecracker: MicroVMs for Fast, Strong Isolation

Firecracker takes the microVM concept and strips it down aggressively.

Originally developed for serverless workloads, Firecracker uses KVM-based virtualization while exposing a deliberately minimal virtual device model.

The design goal is simple:

Traditional VM
────────────────────────
Many virtual devices
Large attack surface
Heavy guest environment


Firecracker
────────────────────────
Minimal virtual hardware
Small device model
Fast startup
Strong hardware boundary

A simplified architecture looks like this:

          Tenant Workload
                 │
                 ▼
            Guest Kernel
                 │
                 ▼
        ┌────────────────┐
        │ Firecracker VM │
        │                │
        │ Minimal VMM    │
        └───────┬────────┘
                │
               KVM
                │
                ▼
          Host Kernel

The critical difference is that Firecracker is not simply "another container runtime."

It is a virtual machine monitor designed around a deliberately narrow device model.

That makes it attractive for environments where workloads are genuinely untrusted and the security boundary needs to be stronger than ordinary containers provide.

18. gVisor vs Kata vs Firecracker

The three approaches can be understood as three different answers to the same question.

Technology	Isolation Mechanism	Host Kernel Exposure	Overhead	Typical Use
Standard container	Linux namespaces + cgroups	High	Very low	Trusted workloads
gVisor	User-space kernel boundary	Reduced	Low–moderate	Untrusted applications
Kata Containers	Lightweight VM	Much lower	Moderate	Strong workload isolation
Firecracker	Minimal KVM microVM	Much lower	Moderate	Strong isolation / sandboxing

The architecture matters more than the product name. A conventional container asks the host kernel to isolate processes. gVisor places an additional kernel-like boundary between the workload and host kernel.

Kata and Firecracker move toward hardware-assisted virtualization, giving the workload a separate guest kernel.

19. The Overhead Question

This is where architecture discussions often become misleading.

You will frequently see claims such as:

"Runtime X adds exactly 5% overhead."

That number is almost meaningless without knowing the workload. A syscall-heavy workload can behave very differently from a CPU-bound workload. A memory-heavy workload behaves differently from a network-heavy workload. Startup latency differs from steady-state throughput.

So rather than pretending there is one universal number, think about the relative shape of the trade-off.

Relative Resource / Isolation Trade-off

Isolation
  ▲
  │
  │                         Firecracker
  │                    ┌───────────────
  │              Kata ─┤
  │
  │        gVisor ─────┤
  │
  │ Standard Container ─┤
  │
  └────────────────────────────────────►
             Runtime Overhead

A rough engineering model looks like this:

Runtime	CPU Overhead	Memory Overhead	Startup	Isolation
Standard container	Lowest	Lowest	Extremely fast	Process/kernel boundary
gVisor	Low–moderate	Moderate	Fast	Stronger syscall isolation
Kata	Moderate	Moderate–higher	Fast, but above containers	VM boundary
Firecracker	Moderate	Moderate	Very fast for a VM	Hardware-assisted VM boundary

These should be treated as relative architectural expectations, not benchmark guarantees.

The actual result needs to be measured against your workload.

For example, a workload performing millions of system calls per second may expose gVisor's additional syscall path more clearly than a CPU-bound application.

Likewise, a workload requiring large amounts of guest memory will make the memory cost of VM-based isolation much more visible than a tiny service.

20. The Real Trade-Off: Density vs Isolation

This is ultimately not a technology-selection argument. It is an engineering trade-off. Consider a cluster with 500 tenants.

If every workload is trusted internal software, putting every Pod inside a microVM may introduce unnecessary operational and resource costs.

But imagine the cluster runs:

customer-submitted code,
CI workloads,
build environments,
arbitrary binaries,
third-party plugins,
serverless functions,
educational code execution,
multi-user development environments.

The threat model is completely different.

In that environment, the question isn't:

"Can we run more Pods per node?"

The question becomes:

"What is the cost of allowing an untrusted process to share a kernel with everyone else?"

That is a much more meaningful question.

21. A Practical Isolation Spectrum

There is no single "multi-tenancy architecture."

There is a spectrum.

          Increasing Isolation
                  │
                  ▼

┌─────────────────────────────────────────────┐
│ Namespace-based tenancy                     │
│                                             │
│ Shared kernel                               │
└─────────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────┐
│ Hardened containers                         │
│                                             │
│ seccomp + capabilities + LSM + cgroups     │
└─────────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────┐
│ Sandboxed containers                        │
│                                             │
│ gVisor / similar isolation                  │
└─────────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────┐
│ MicroVM-backed workloads                    │
│                                             │
│ Kata / Firecracker                         │
└─────────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────┐
│ Dedicated node / cluster                   │
│                                             │
│ Physical infrastructure boundary            │
└─────────────────────────────────────────────┘

Moving downward generally increases isolation.

It can also increase cost, operational complexity, startup time, memory consumption, and debugging complexity. There is no free isolation. Every stronger boundary introduces another engineering layer.

22. What Should a Platform Engineer Actually Do?

The answer should not be "deploy Kata everywhere." Start by defining the threat model. Ask what kind of tenants are sharing the cluster.

Are they internal teams?
Are they different business units?
Are they external customers?
Can they execute arbitrary code?
Can they submit containers?
Can they control Linux capabilities?
Can they mount host paths?
Can they access devices?
Can they create privileged Pods?
The answers completely change the architecture.

For conventional internal workloads, a strong baseline usually involves:

Namespaces
    +
ResourceQuota
    +
LimitRange
    +
Requests / Limits
    +
Pod Security controls
    +
Seccomp
    +
Capabilities dropped
    +
AppArmor / SELinux
    +
NetworkPolicy
    +
Hardened node configuration
    +
Regular kernel/runtime patching

For genuinely untrusted workloads, add a stronger execution boundary.

That might mean gVisor, Kata, Firecracker-based infrastructure, dedicated node pools, or even separate clusters depending on the threat model.

23. The Most Dangerous Assumption

The most dangerous assumption in Kubernetes multi-tenancy is:

"It's a container, so it's isolated."

It isn't.

It is isolated to a degree.

That degree depends on how the container is configured, how the runtime is configured, what the kernel exposes, what capabilities the process has, what security policies are applied, and whether the underlying kernel and runtime are patched.

The same statement applies to resource isolation.

A memory limit is not the same thing as guaranteed memory availability. A CPU limit is not the same thing as dedicated CPU performance. A namespace is not a machine. A QoS class is not a business-criticality guarantee. And a container runtime is not necessarily a security boundary equivalent to a virtual machine.

Once you understand those distinctions, Kubernetes multi-tenancy becomes much easier to reason about.

24. The Platform Engineer's Mental Model

When debugging multi-tenant failures, stop looking at Kubernetes alone.

Trace the entire stack:

Tenant
  │
  ▼
Pod
  │
  ▼
Container
  │
  ▼
Runtime
  │
  ▼
cgroup
  │
  ▼
Linux namespace
  │
  ▼
Linux kernel
  │
  ▼
Scheduler / MM / VFS / Network stack
  │
  ▼
Hardware

A Kubernetes incident may begin as a Pod problem but end up being a kernel scheduling problem. A storage latency problem may begin as a tenant workload but manifest as filesystem writeback contention.

A memory incident may look like a Kubernetes deployment failure but ultimately be an OOM decision made by the Linux kernel. And a security incident may begin with a seemingly harmless container but end with a host kernel vulnerability.

The abstraction layers are useful. But during an incident, they can also hide the actual failure.

25. Conclusion: Kubernetes Can Isolate Workloads, But the Kernel Still Has the Final Word

Kubernetes multi-tenancy becomes considerably more complicated once you move below the API server.

At the control-plane layer, tenants compete for API resources, controllers, CRDs, and scheduling capacity. At the node layer, they compete for something much more fundamental: CPU cycles, memory pages, filesystem activity, network resources, and ultimately access to the Linux kernel itself.

cgroups provide the foundation for controlling that competition, but understanding them requires going beyond resources.requests and resources.limits. The differences between cgroups v1 and v2, memory accounting, writeback behavior, CPU throttling, and OOM handling can directly influence how predictable a shared node actually is.

The OOM killer is a particularly good reminder that Kubernetes abstractions eventually terminate at the kernel. The kernel does not understand tenants, applications, SLAs, or business priorities. It evaluates processes and memory pressure. Kubernetes' QoS classes and oom_score_adj mechanisms help influence that decision, but they do not turn an oversubscribed node into an isolated machine.

Security pushes the problem even further.

A standard container still shares the host kernel. That gives us excellent density and startup performance, but it also means that the host kernel remains part of the attack surface. For trusted workloads, hardened containers can provide a very strong practical boundary. For workloads executing genuinely untrusted code, however, the architecture may need another layer.

That is where gVisor, Kata Containers, and Firecracker become interesting. They represent different points on the isolation spectrum, from syscall interception and user-space kernel mechanisms to hardware-assisted microVM boundaries. None of them is universally "better." They simply move the security boundary and accept different performance, compatibility, memory, and operational costs in exchange.

And that is the uncomfortable engineering truth about hard multi-tenancy:

Isolation is never free.

You either pay with reduced density, additional memory, virtualization overhead, operational complexity, or engineering effort—or you accept more shared infrastructure and therefore a larger blast radius.

The job of a platform engineer is not to eliminate that trade-off. It is to understand it well enough to place the boundary deliberately.

Because once multiple tenants share a Kubernetes node, the kernel becomes part of your tenancy architecture whether you planned for it or not.

Coming in Part 3: Encrypting the Tenant Boundary

We have now moved from the Kubernetes control plane down into the Linux kernel. But there is still one major boundary left. The data itself.

Even if Tenant A cannot consume all of Tenant B's memory, cannot escape its container, and cannot compromise the host kernel, we still have to answer a much harder question:

Can Tenant A observe, intercept, modify, or recover Tenant B's data while it is moving through the cluster or sitting on storage?

In Part 3 — the final chapter of this multi-tenancy series, we will move into the networking and storage layers and examine how to lock down data-in-transit and data-at-rest across tenants.

We will go beyond basic NetworkPolicies and Kubernetes Secrets and look at the architecture behind encrypted east-west traffic, service-mesh mTLS, identity-aware communication, storage encryption, key management, CSI integration, and the cryptographic boundaries required when tenants share the same underlying infrastructure.

Because isolating the process is only half the problem. The final question is whether you can isolate the data.

See you in Part 3. Into The Depths of Kubernetes' continues...

Into The Depths of Kubernetes: Multi-Tenancy Part 1

Kubernetes with Naveen — Sat, 19 Sep 2026 09:03:17 +0000

Welcome to the launch of my new blog series, "Into The Depths of Kubernetes," where we are peeling back the abstraction layers to explore the architectural patterns, runtime mechanics, and production realities that truly matter for software and platform engineers. Whether you are scaling microservices, optimizing infrastructure costs, or hardening cluster security, this series is designed to give you actionable insights and deep technical clarity. We are kicking things off with a fundamental challenge every growing organization faces: Multi-Tenancy in Kubernetes. In this inaugural post, we will dive into isolating workloads, managing shared cluster resources, enforcing strict security boundary conditions with Namespaces and Network Policies, and balancing cost efficiency against robust tenant isolation.

Part 1 — Exploding the Control Plane: Concurrency, CRD Collisions, and Virtual Slicing

If you have spent enough time operating Kubernetes, you have probably heard some variation of the same advice whenever the conversation turns toward multi-tenancy: “Just give every team its own namespace.” It sounds reasonable because namespaces are one of the most visible boundaries Kubernetes gives us. Team A gets team-a, Team B gets team-b, RBAC prevents them from touching each other's workloads, ResourceQuotas limit how much they can consume, and NetworkPolicies can restrict communication between them. On a whiteboard, it looks like isolation. In a production cluster, however, that picture is incomplete. A namespace is a logical boundary, not an independent Kubernetes environment, and that distinction becomes extremely important as soon as one tenant becomes noisy, misconfigured, or simply operates at a scale that was never anticipated.

A namespace does not create a new API server. It does not create a new etcd instance. It does not create an independent controller-manager. It does not create a separate CRD registry, and it certainly does not give a tenant its own Linux kernel. Multiple tenants can therefore appear isolated from the perspective of their Kubernetes objects while continuing to compete for several of the same underlying control-plane resources. A badly behaved internal platform team does not need to be malicious to create problems for everyone else; a runaway reconciliation loop, an aggressive CI system, an incorrectly configured operator, or a simple shell script repeatedly querying the API can generate enough pressure to make the shared control plane everyone's problem.

This is where Kubernetes multi-tenancy becomes much more interesting than creating namespaces and writing a few RBAC rules. In this first part of Into The Depths of Kubernetes, we're going below the namespace abstraction and looking at the control plane itself: how API requests compete for finite processing capacity, how API Priority and Fairness prevents one class of traffic from monopolizing that capacity, why cluster-scoped CRDs create a very different type of tenant collision, and why virtual control planes such as vCluster exist when logical isolation starts reaching its limits.

The Namespace Illusion

Let's start with the architecture most teams actually deploy. From the application team's perspective, namespaces provide a clean way to divide ownership: one team works inside team-a, another works inside team-b, and a third operates inside team-c. Kubernetes then gives administrators familiar mechanisms such as RBAC, ResourceQuotas and NetworkPolicies to control what each team can access or consume. The problem is that these mechanisms sit above a number of shared control-plane components. The namespace changes the scope of many Kubernetes objects, but it doesn't transform the underlying cluster into three independent control planes.

                    Kubernetes Cluster
                           │
                ┌──────────┴──────────┐
                │     kube-apiserver   │
                └──────────┬──────────┘
                           │
          ┌────────────────┼────────────────┐
          │                │                │
       Team A           Team B           Team C
      namespace         namespace         namespace
          │                │                │
        Pods             Pods             Pods

This is the first mental model I want to establish: namespaces provide logical isolation, but they do not magically partition the Kubernetes control plane. That difference matters because isolation is not one problem. There are several different resources and trust boundaries involved, including namespace objects, authorization, network traffic, API-server concurrency, cluster-scoped APIs, worker nodes, operating-system resources, and the control plane itself. A namespace addresses some of these concerns extremely well, but expecting it to solve all of them is where multi-tenant Kubernetes architectures usually start to become fragile.

Consider what happens when Tenant A begins generating an unusually high volume of API traffic. Tenant A may have no permission to access Tenant B's objects, and its pods may be completely isolated from Tenant B's workloads, but both tenants still depend on the same API-server infrastructure. The request originating from Tenant A therefore doesn't need permission to touch Tenant B in order to affect Tenant B. It only needs to consume enough shared control-plane capacity that requests from Tenant B start waiting longer. That is the essence of a noisy-neighbor problem at the API layer.

The API Server Is a Shared Resource

Let's look at an intentionally simple example:

while true; do
    kubectl get pods -A
done

One shell loop isn't particularly interesting. Now imagine that command being executed by a CI job, a badly written automation script, or hundreds of processes at the same time. The individual kubectl operation may be perfectly valid, and Kubernetes may happily authorize every request, but the fact that the request is valid does not mean the API server has infinite capacity to process it. Every API operation still travels through authentication, authorization, request classification, admission where applicable, request handling, serialization and downstream processing, with some operations ultimately interacting with storage or triggering additional work elsewhere in the control plane.

The important point here is that kubectl itself isn't dangerous. The problem is request volume and concurrency. A Kubernetes API server is a highly concurrent Go application, and requests are handled using goroutines and a collection of internal mechanisms rather than a simplistic one-thread-per-request model. Goroutines are considerably cheaper than operating-system threads, but they still consume CPU, memory, network resources and downstream capacity. More importantly, the API server operates with bounded concurrency rather than allowing an unlimited number of requests to execute simultaneously. Once enough traffic arrives, requests begin competing for that finite capacity.

Historically, Kubernetes exposed mechanisms such as:

--max-requests-inflight
--max-mutating-requests-inflight

to limit request concurrency. Modern Kubernetes uses API Priority and Fairness (APF) to provide a more structured mechanism for deciding which requests should receive concurrency and how competing request flows should be isolated. Kubernetes documents APF specifically as a way of protecting important API traffic from being overwhelmed by other traffic classes. The important engineering lesson is therefore not to think of the API server as a limitless HTTP endpoint, but as a shared control-plane resource whose processing capacity has to be deliberately managed.

One useful way to visualize the problem is this:

Tenant A
   │
   │ 1000 requests/sec
   ▼
┌─────────────────────┐
│    kube-apiserver   │
│                     │
│   concurrency = C   │
└─────────┬───────────┘
          │
          ▼
     processing

Now add four other teams:

Tenant A ─────┐
Tenant B ─────┤
Tenant C ─────┼──► kube-apiserver ──► etcd/controllers
Tenant D ─────┤
Tenant E ─────┘

The problem becomes obvious. If Tenant A generates enough traffic, the effect is no longer confined to Tenant A. Tenant B might still have plenty of CPU and memory allocated to its workloads, and its pods may all be healthy, but its requests to the Kubernetes API are entering the same shared control-plane environment. This is why API-server saturation can be particularly deceptive during an incident: the worker nodes can look completely normal while kubectl get pods starts taking seconds, then tens of seconds, and eventually timing out.

The cluster isn't necessarily "down" in the traditional sense. The control plane may simply be struggling to keep up with the amount of work being presented to it.

The API-Starvation Experiment

Let's make the problem a little more formal. Imagine that the API server has an effective concurrency capacity of C for a particular class of requests. Tenant A generates traffic at a rate of λA, while the system is capable of completing those requests at a rate of μA. When the arrival rate remains comfortably below the service rate, the system can keep up and queues remain small. As the arrival rate approaches the service rate, latency begins increasing because the system has less spare capacity available to absorb bursts.

λA < μA     → system keeps up

λA → μA     → latency increases

λA > μA     → requests accumulate

This is basic queueing behavior, but it becomes extremely important when the queue belongs to a shared control plane. If Tenant A's request stream is allowed to consume essentially all of the available processing capacity, other request classes begin waiting behind work they did not generate. A platform controller trying to update a critical resource, a developer running kubectl, or another tenant attempting to scale an application can all experience increased latency even though none of them caused the original traffic spike.

A simplified model looks like this:

Tenant A
   │
   │ 1000 requests/sec
   ▼
┌─────────────────────┐
│    kube-apiserver   │
│                     │
│   concurrency = C   │
└─────────┬───────────┘
          │
          ▼
     processing

The interesting part is what happens when several tenants are introduced:

Tenant A ─────┐
Tenant B ─────┤
Tenant C ─────┼──► kube-apiserver ──► etcd/controllers
Tenant D ─────┤
Tenant E ─────┘

Without an appropriate fairness mechanism, the control plane has to deal with a competition problem. The tenant generating the most work is not necessarily the tenant whose work is most important, and the request arriving first isn't necessarily the request that should be allowed to consume the next available unit of concurrency. Kubernetes therefore needs a mechanism that understands classes of traffic, assigns them different levels of importance, and prevents a noisy request stream from becoming an effective denial-of-service against unrelated control-plane operations.

This is precisely the problem APF was designed to address.

This Is Where APF Becomes Interesting

API Priority and Fairness exists because the statement "every request gets equal treatment" isn't particularly useful when operating a shared Kubernetes control plane. A request generated by a critical control-plane component may need very different treatment from a burst of interactive kubectl queries generated by a tenant. APF provides a mechanism to classify requests into different priority levels and flows, and then control how concurrency is distributed between those classes.

At a high level, the request path can be visualized like this:

                     Incoming API Requests
                              │
                              ▼
                       ┌─────────────┐
                       │ FlowSchema  │
                       └──────┬──────┘
                              │
                    ┌─────────┴─────────┐
                    ▼                   ▼
             Priority Level A     Priority Level B
                    │                   │
                 queues              queues
                    │                   │
                    ▼                   ▼
                execution           execution

A FlowSchema determines which requests belong to a particular traffic class and associates those requests with a PriorityLevelConfiguration. The priority level then controls the concurrency and queueing behavior for that class. This gives Kubernetes something much more useful than a single global request queue: it can distinguish platform traffic, tenant traffic, monitoring traffic, leader-election traffic, or other categories that need different treatment.

For a multi-tenant platform, you can conceptually imagine a design like this:

Platform controllers
        │
        ▼
   high-priority

Tenant A
        │
        ▼
   tenant-workloads

Tenant B
        │
        ▼
   tenant-workloads

Monitoring
        │
        ▼
   monitoring

Kubernetes already ships with APF configurations intended to protect important classes of control-plane traffic, including leader-election and built-in controller traffic. The exact defaults have evolved across Kubernetes releases, so production administrators should inspect the APF configuration of the Kubernetes version they actually operate rather than assuming that an example from an older cluster behaves identically today.

The important architectural idea remains the same: not all API traffic needs to compete in one undifferentiated pool. Once requests are classified, Kubernetes can give different categories different concurrency behavior, which means a noisy tenant can be prevented from turning the entire API server into its personal work queue.

The Math Behind API Capacity

Let's make the model concrete. Suppose, purely for illustration, that the API server has a concurrency budget of:

Total concurrency = 100

Now imagine that the platform administrator establishes three priority levels with nominal allocations resembling:

Platform       = 40
Tenant traffic = 50
Monitoring     = 10

Conceptually, that gives us:

100 concurrent requests
│
├── 40 → Platform
├── 50 → Tenants
└── 10 → Monitoring

The useful mental model is approximately:

PL concurrency
≈
Total API concurrency ×
(nominal level limit / total nominal limits)

So a nominal 40% allocation against a concurrency budget of 100 corresponds to roughly 40 units of concurrency under the basic proportional model. However, this should not be interpreted as a permanent physical wall around exactly 40 requests. APF has borrowing and lending behavior, and its actual scheduling behavior depends on the complete PriorityLevelConfiguration. The purpose of the calculation is to understand the relative capacity allocation rather than to reduce APF to a simple static partitioning mechanism.

That distinction becomes important when designing real clusters. If the platform has a 40% nominal share, it doesn't necessarily mean 40 requests are permanently reserved and will sit idle whenever platform traffic is quiet. APF is designed to use capacity efficiently while maintaining fairness and protecting important traffic classes. The configuration therefore has to be evaluated against real workload behavior instead of being treated like a conventional static CPU reservation.

The numbers themselves should also come from measurements rather than guesswork. A cluster hosting hundreds of microservices with aggressive controllers, large watch sets, admission webhooks and frequent CI activity has a very different API workload from a small development cluster. API-server sizing, etcd performance, controller behavior, watch traffic, admission latency and tenant request patterns all contribute to the actual capacity required. The goal is not to find a magical percentage that works everywhere; the goal is to deliberately establish which traffic must remain responsive when another traffic class becomes noisy.

FlowSchemas: Separating the Noisy Neighbors

Now consider a slightly different problem. Suppose Tenant A and Tenant B both belong to the same priority level. We have successfully separated tenant traffic from critical platform traffic, but Tenant A can still generate a disproportionate amount of work within that tenant priority level. We therefore need another layer of separation: the ability to distinguish individual request flows so that one noisy tenant does not dominate the queueing behavior experienced by other tenants.

This is where FlowSchema and its flow distinguisher become particularly useful.

A FlowSchema can classify requests using characteristics such as:

User
Group
ServiceAccount
Verb
API group
Resource
Namespace

Conceptually, we might create a classification around a tenant's ServiceAccount:

FlowSchema
    │
    ├── ServiceAccount: team-a
    │
    └── PriorityLevel: tenant-workloads

That can then result in separate flows such as:

tenant-workloads
│
├── flow(team-a)
├── flow(team-b)
├── flow(team-c)
└── flow(team-d)

This distinction is important because priority level and flow are not the same thing. The priority level answers something like, "How important is this class of traffic, and how much concurrency should this class receive?" The flow answers something closer to, "Which requests should compete with one another inside that class?" That separation allows the system to protect both high-level control-plane priorities and fairness between individual request sources.

In a multi-tenant platform, this can be particularly valuable when tenants have very different request patterns. One tenant may run a highly automated GitOps platform that continuously watches and reconciles resources, while another may only make a handful of API calls during deployments. If both are treated as one undifferentiated flow, the heavy requester can influence the experience of the lighter requester. Flow-level separation gives APF more information with which to distribute the pressure.

Shuffle Sharding: Don't Let One Tenant Jack the Queue

This is one of the more interesting pieces of APF because it addresses the queue-collision problem without requiring a completely dedicated queue for every possible flow. Imagine that the API server has a collection of queues:

64 queues

If every request from every tenant could land anywhere, a particularly noisy tenant could eventually occupy a large portion of the queueing system. On the other hand, assigning one permanent queue to every possible tenant would be wasteful and difficult to manage when the number of flows changes dynamically.

Shuffle sharding takes a different approach.

The basic idea is that each flow receives a deterministic-looking subset of candidate queues, sometimes described as its hand, and requests from that flow are then placed among those candidates according to queue state. Kubernetes documents handSize as the number of queues considered for a flow, and its documented configuration uses values such as 64 queues with a hand size of 8.

                     64 queues
 ┌─────────────────────────────────────────┐
 │ Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 ... Q64        │
 └─────────────────────────────────────────┘

 Tenant A
    │
    ├── hash
    │
    └── candidate queues:
         Q4, Q17, Q29, Q33, Q41, Q46, Q52, Q61

 Tenant B
    │
    └── different candidate queues

The practical benefit is that two unrelated flows will generally have different subsets of queues. Tenant A therefore doesn't automatically gain access to every queue that Tenant B might need, while the system also avoids dedicating an entire permanently isolated queue to every tenant. If Tenant A becomes extremely noisy, the impact tends to remain concentrated around the subset of queues associated with its flow rather than allowing that traffic to contaminate every queue in the priority level.

This is why shuffle sharding is more interesting than simply saying "APF has multiple queues." The engineering goal is not merely to create more queues; it is to reduce the probability that two unrelated noisy flows collide across the same queueing resources. It provides a probabilistic form of isolation that becomes increasingly valuable as the number of request flows grows.

Designing APF for Tenants

A production design might therefore look conceptually like this:

                    kube-apiserver
                          │
                    API Priority
                    & Fairness
                          │
          ┌───────────────┼────────────────┐
          │               │                │
          ▼               ▼                ▼
      Platform         Tenant API       Observability
          │               │                │
       30%+              50%               20%
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
          tenant-a     tenant-b     tenant-c

These numbers are deliberately illustrative rather than recommended defaults. A production administrator should not copy a 30/50/20 split into a cluster simply because it looks sensible on a diagram. The correct configuration depends on the actual API workload, number of tenants, controller behavior, admission webhooks, cluster size, etcd performance, API-server sizing and the operational SLOs that need to be protected.

There is also a subtle but important operational point here: APF should be observed, not merely configured. When an API request is classified, Kubernetes provides diagnostic information that can help operators understand which FlowSchema and PriorityLevelConfiguration handled it. That becomes extremely useful during an incident because it lets you move from the vague statement "the API server is slow" to a much more specific question: "Which traffic class is consuming the available concurrency, and where are requests waiting?"

For a serious multi-tenant platform, API capacity should therefore be treated as a shared resource just like CPU, memory or network bandwidth. You should know which classes of traffic are important, what happens when they exceed their expected rates, and whether critical control-plane operations continue making progress when one tenant becomes unexpectedly noisy.

The CRD Problem Nobody Sees Coming

API starvation is only one side of the multi-tenancy problem. The second problem is more architectural because it has nothing to do with request volume. It has to do with the fact that some Kubernetes APIs are cluster-scoped primitives, and Custom Resource Definitions are one of the most important examples.

A namespace can contain namespaced resources such as Pods, Deployments and Services, but a CRD defines an API resource for the Kubernetes cluster itself. That means the CRD is not independently created inside every namespace. If several teams share one physical Kubernetes cluster, they ultimately participate in the same cluster-level CRD definitions.

Consider:

Cluster
│
├── Namespace: team-a
│
├── Namespace: team-b
│
├── Namespace: team-c
│
└── CRD: certificates.cert-manager.io

The CRD isn't:

team-a/certificates.cert-manager.io
team-b/certificates.cert-manager.io

There is one cluster-level definition that establishes how the Kubernetes API understands that resource.

This distinction becomes particularly important when different tenants want to operate different versions of an operator or API ecosystem. Kubernetes CRDs can support multiple served versions and conversion between versions, so the problem is not as simplistic as saying that changing v1alpha1 to v1beta1 automatically breaks every other tenant. The real problem is that tenants sharing one physical CRD are still sharing the same API contract, schema, conversion behavior and lifecycle. They do not independently own that API simply because their workloads happen to live in different namespaces.

A CRD Collision Scenario

Imagine Tenant A operates an application platform that depends on an operator exposing:

Certificate
v1alpha1

At the same time, Tenant B has workloads built around a newer operator ecosystem expecting:

Certificate
v1beta1

Both tenants may believe they own their respective environments because all of their application objects are namespaced. However, if they are installing and managing the same cluster-scoped CRD, they are actually sharing a critical part of the Kubernetes API surface.

Now Tenant A performs an operator upgrade. That upgrade may modify the CRD's schema, served versions, storage version, conversion configuration, validation rules or related cluster-scoped components. Whether Tenant B actually breaks depends on the exact CRD versions, conversion strategy, operator behavior and resource definitions involved, but the architectural problem already exists before anything fails: two independently managed tenants have incompatible expectations about the same cluster-level API contract.

That is a very different failure mode from an ordinary namespace permission issue. RBAC can prevent Tenant A from modifying Tenant B's Deployment, but RBAC does not transform a cluster-scoped CRD into a tenant-scoped object. Similarly, a ResourceQuota can control how many Pods Tenant A creates, but it cannot create a second independent definition of certificates.cert-manager.io.

And this problem becomes even more significant with operators because installing an operator can involve much more than creating a Deployment in a namespace. Operators commonly install CRDs, ClusterRoles, ClusterRoleBindings, admission webhooks and other cluster-scoped resources. What looked like a namespace-level application deployment can therefore become an operation that changes shared cluster infrastructure.

The moment multiple independent teams are allowed to manage those components, the namespace abstraction starts showing its limits.

This Is the Point Where Namespaces Stop Being Enough

At this point, we have two fundamentally different multi-tenancy problems.

The first is API contention:

Tenant A
   │
   ├── massive API traffic
   │
   ▼
Shared kube-apiserver
   │
   ├── Tenant B
   ├── Tenant C
   └── Platform controllers

APF is designed to help here. It provides mechanisms for classifying requests, assigning them to priority levels, separating flows, queueing them fairly and reducing the ability of one traffic class to consume all available API-server capacity.

The second is cluster-scoped API collision:

Tenant A ──┐
            ├── same CRD
Tenant B ──┤
            │
Tenant C ──┘

APF cannot solve this problem because the problem isn't request scheduling. ResourceQuotas cannot solve it because the problem isn't resource consumption. NetworkPolicies cannot solve it because the problem isn't network connectivity. Even RBAC only partially addresses it because preventing tenants from modifying cluster-scoped objects doesn't create independent cluster-scoped objects for every tenant.

At some point, the question changes from:

"How do we better isolate tenants inside one Kubernetes API?"

to:

"Should these tenants actually share the same Kubernetes API in the first place?"

That is the point where virtual control planes become interesting.

Enter vCluster

Instead of giving every tenant another namespace inside the same Kubernetes API, we can give them something that looks much closer to a Kubernetes cluster while still running that control plane on top of an existing Kubernetes cluster.

This is the basic idea behind vCluster.

A vCluster provides a virtualized Kubernetes control plane running inside a host Kubernetes cluster. The virtual control plane has its own API server and control-plane components, with a datastore and synchronization layer that connect the virtual environment to the underlying physical cluster.

Conceptually:

                 Host Kubernetes Cluster
                         │
          ┌──────────────┼──────────────┐
          │              │              │
          ▼              ▼              ▼
      vCluster A      vCluster B      vCluster C
          │              │              │
       API Server      API Server      API Server
       Controller      Controller      Controller
       Data Store      Data Store      Data Store
          │              │              │
          └──────────────┼──────────────┘
                         │
                       Syncer
                         │
                         ▼
                Host Kubernetes API

The important architectural difference is that the tenant is no longer interacting directly with the host Kubernetes API as its primary API surface. Instead, it talks to a virtual control plane. The virtual cluster can maintain its own Kubernetes objects, CRDs, RBAC configuration and control-plane state while selected workloads and resources are synchronized to the underlying host cluster.

This introduces another API boundary between the tenant and the physical cluster. Instead of trying to make one API server behave like several independent Kubernetes environments, we create multiple logical Kubernetes control planes that happen to share underlying infrastructure.

Virtual API vs Namespace

The difference becomes easier to see when the two models are placed next to each other.

Namespace tenancy

                HOST API SERVER
                      │
       ┌──────────────┼──────────────┐
       │              │              │
   Namespace A    Namespace B    Namespace C
       │              │              │
      Apps           Apps           Apps

Every tenant is still talking to the same host API server. They may have different RBAC permissions and different namespaces, but their requests ultimately enter the same physical control-plane environment.

Virtual cluster tenancy

                  HOST CLUSTER
                       │
       ┌───────────────┼───────────────┐
       │               │               │
   vCluster A      vCluster B      vCluster C
       │               │               │
    API server       API server       API server
       │               │               │
     tenant          tenant          tenant
     objects         objects         objects

Now each tenant has another API boundary. Tenant A can maintain its own API ecosystem without registering every tenant-facing CRD directly into the same host API surface used by Tenant B. That distinction can be extremely valuable for platform teams that need to give users a Kubernetes-like environment without handing each user an entire physical cluster.

However, this should not be misunderstood as magic isolation. The virtual control planes still run on the underlying Kubernetes infrastructure, and their workloads may still share worker nodes and the underlying operating-system kernel. The architecture therefore improves control-plane and API isolation without automatically providing the same security boundary as a physically separate cluster or a dedicated virtual machine.

But Where Do the Pods Actually Go?

This is where the synchronization layer becomes particularly interesting.

Suppose a tenant runs:

kubectl apply -f deployment.yaml

inside its virtual cluster. From the tenant's perspective, it is simply interacting with Kubernetes. Behind the scenes, however, the virtual control plane processes the request and the syncer determines which resources need to be represented on the underlying host cluster.

The simplified flow looks like this:

Tenant API

    kubectl apply -f deployment.yaml
                  │
                  ▼
          vCluster API server
                  │
                  ▼
             Syncer
                  │
                  ▼
          Host Kubernetes API
                  │
                  ▼
       Namespace: tenant-a
                  │
                  ▼
              Pod

The tenant therefore sees a Kubernetes object in its virtual environment while the physical cluster sees the synchronized representation required to actually run the workload. Depending on the vCluster configuration, resources such as Pods, Services, ConfigMaps and Secrets can be synchronized between the virtual and host environments.

This creates an important abstraction:

Tenant abstraction
        │
        ▼
Virtual Kubernetes API
        │
        ▼
Synchronization layer
        │
        ▼
Physical Kubernetes infrastructure

The tenant gets a Kubernetes API that is logically separate from the host API, while the infrastructure operator retains control over the physical cluster. This is why the model is often described as a virtual control plane rather than simply another namespace-management mechanism.

Why This Changes the CRD Problem

Let's return to our earlier CRD collision.

With ordinary namespace-based tenancy, the model looks like this:

Tenant A
   │
   └── shared CRD

Tenant B
   │
   └── shared CRD

Both tenants are ultimately participating in the same physical Kubernetes API definition.

With virtual clusters, the conceptual model changes:

vCluster A
   │
   └── CRD: certificates.cert-manager.io

vCluster B
   │
   └── CRD: certificates.cert-manager.io

The two tenants can now maintain their own virtual API definitions without requiring both definitions to coexist as independent versions of the same CRD in the host cluster's tenant-facing API. That is a major architectural improvement for teams that need Kubernetes-level API independence rather than simply namespace-level resource separation.

But there is still an important caveat. The physical cluster has not disappeared. The worker nodes, host API server, network, storage infrastructure and Linux kernel remain shared according to the deployment architecture. A virtual control plane therefore solves a particular class of isolation problems—it doesn't automatically solve every multi-tenancy problem.

That distinction is important because it prevents another common mistake: assuming that virtual Kubernetes equals virtual machine isolation. It doesn't. If two virtual clusters eventually schedule workloads onto the same worker node, those workloads are still ultimately sharing the same operating-system kernel and physical resources. Stronger isolation at that layer is a separate engineering problem, and that is exactly where the next part of this series will go.

The Multi-Tenancy Spectrum

At this point, it helps to stop thinking about multi-tenancy as a binary decision where a cluster is either "multi-tenant" or "not multi-tenant." Kubernetes provides a spectrum of isolation mechanisms, and the correct choice depends heavily on the trust relationship between tenants, the resources they need to control, the consequences of compromise, and the operational cost the platform team is willing to accept.

LOWER ISOLATION
       │
       ▼
┌──────────────────────────────┐
│ Namespace                     │
│ RBAC                          │
│ ResourceQuota                 │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│ Namespace + NetworkPolicy     │
│ + APF + admission policies    │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│ Virtual Control Plane         │
│ vCluster                      │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│ Dedicated Kubernetes Cluster  │
└──────────────────────────────┘
       │
       ▼
HIGHER ISOLATION

A development environment shared by trusted teams may require nothing more than namespaces, RBAC, quotas and network policies. A large internal platform serving dozens of independent engineering teams may need APF, stronger admission controls and carefully managed cluster-scoped resources. A platform offering Kubernetes environments to external or untrusted users may require virtual control planes, dedicated nodes, sandboxed runtimes or even completely separate physical clusters.

There is no universal "correct" tenancy model because the isolation requirement comes from the threat model. The important thing is to understand exactly which boundary each technology provides. Namespace isolation is useful, APF is useful, virtual control planes are useful, and dedicated clusters are useful—but they solve different problems and carry very different operational costs.

The Bigger Lesson

The mistake isn't using namespaces.

Namespaces are one of the most useful abstractions in Kubernetes, and for many environments they are exactly the right starting point. The mistake is assuming that a namespace represents more isolation than it actually provides. Once you operate Kubernetes at scale, the more useful questions are not simply "Which namespace does this application belong to?" but rather "Which underlying resources and control-plane components does this tenant still share with everyone else?"

Ask who shares the API server. Ask who shares API-server concurrency. Ask who can install or modify CRDs. Ask who controls admission webhooks. Ask who can create operators. Ask which workloads share worker nodes. Ask which tenants share storage and network infrastructure. And eventually, ask the question that becomes unavoidable when the trust boundary becomes weaker: what happens when the tenant is not trusted?

Those questions expose the actual architecture underneath the Kubernetes abstractions.

Multi-tenancy is therefore not a Kubernetes object that you enable with a single flag. It is a collection of isolation decisions made across several layers of the stack. Kubernetes gives you the primitives, but the platform engineer has to decide where each boundary belongs and what happens when one of those boundaries is stressed or deliberately attacked.

Part 1 Takeaway

Kubernetes gives you several layers of isolation, but they are not interchangeable. A namespace gives you a logical resource boundary. RBAC gives you an authorization boundary. NetworkPolicy provides a network-level control boundary. ResourceQuota constrains resource consumption within its scope. APF gives you a mechanism for protecting API-server concurrency between different request classes, while virtual control planes introduce another Kubernetes API boundary when namespace-level isolation no longer provides the separation you actually need.

The architecture can be summarized like this:

                Multi-Tenancy
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
   API isolation  API fairness  Resource isolation
        │            │            │
   vCluster          APF       Quotas / Policies
        │
        ▼
   Control-plane
     separation

The deeper you go into Kubernetes, the more obvious the pattern becomes: multi-tenancy isn't a Kubernetes object; it is an architecture. The namespace is simply one layer in that architecture. Once you understand what remains shared underneath it, you can make a much more deliberate decision about whether your environment needs stronger API fairness, tighter control over cluster-scoped resources, virtual control planes, dedicated nodes, or completely separate clusters.

And this is only the first layer.

What's Coming in Part 2

In the next part we move past basic ResourceQuotas and focuses on the underlying Linux kernel, exploring container breakouts, kernel-level resource starvation,m and the absolute engineering trade-offsof running sandboxed runtimes. Stay tuned.

Pod is Healthy, Service is Green, Users See 502: The Anatomy of a Kubernetes Ghost Outage

Kubernetes with Naveen — Tue, 15 Sep 2026 12:15:48 +0000

At 2:07 AM, the page goes off.

Users are reporting that the application is returning 502 Bad Gateway. You open Grafana expecting the usual disaster: crashed Pods, failing readiness probes, CPU at 100%, or a node running out of memory.

Nothing.

Every Pod says Running. Readiness and liveness probes have been green for hours. The Service has healthy endpoints. CPU is normal. Memory is normal. Even the ingress controller looks healthy.

You refresh the application. 502. Refresh again. 200 OK. Again. 502. This is the kind of incident that makes Kubernetes look broken.

Usually, Kubernetes isn't broken. The problem is that your dashboards are looking at the wrong layer.

Kubernetes can tell you that a Pod exists and that an HTTP health endpoint responds. It cannot, by itself, tell you that a particular TCP connection sitting inside an ingress controller's connection pool is still usable, that the application's listen() queue isn't overflowing, or that a terminating process just closed a socket while traffic was still being routed toward it.

To understand these outages, you have to go below Pods and Services.

You have to look at processes, file descriptors, sockets, TCP state, kernel queues, and syscalls.

The Nightmare Scenario: All Pods Are Green, Probes Pass, but Ingress Returns 502

Consider a simple request path:

Client
   |
   v
Load Balancer
   |
   v
Ingress / NGINX / Envoy
   |
   v
Kubernetes Service
   |
   v
Pod
   |
   v
Application process
   |
   v
Linux TCP socket

A request does not magically travel from an Ingress to a Pod.

Eventually, somebody has to perform operations equivalent to:

socket()
connect()
send()
recv()
close()

On the backend, the application eventually does something equivalent to:

socket()
bind()
listen()
accept()
recv()
send()
close()

Between those system calls sits the Linux networking stack, TCP state machines, queues, connection buffers, and the application runtime.

That is where many "healthy Pod, mysterious 502" incidents actually live. A Pod can be perfectly healthy according to Kubernetes while one particular connection between the proxy and application is completely broken.

**What an HTTP 502 Really Means at the Network Layer**

A 502 Bad Gateway generally means the component acting as a gateway or reverse proxy could not obtain a valid response from its upstream backend. The important word is proxy.

Suppose NGINX receives:

GET /api/users HTTP/1.1
Host: example.com

NGINX selects a backend:

Service
   |
   +---- Pod A
   +---- Pod B
   +---- Pod C

It then needs to communicate with one of those Pods.

A simplified successful exchange looks like this:

NGINX Backend
  | |
  | -------- TCP SYN ----------> |
  | <------- SYN + ACK --------- |
  | -------- ACK --------------> |
  | |
  | -------- HTTP request -----> |
  | |
  | <------- HTTP response ----- |
  | |

If the backend closes the connection unexpectedly, resets it, fails to accept the connection, or sends something the proxy cannot interpret as a valid HTTP response, the proxy may return:

HTTP/1.1 502 Bad Gateway

The critical detail is that the 502 can be generated by the proxy. The application itself may never have generated a 502.

For instance:

Client
  |
  | HTTP request
  v
NGINX
  |
  | connection reset
  X
Backend

NGINX can only tell the client:

"I couldn't successfully talk to my upstream."

So the first question during a 502 incident should not be:

"Why is my Pod unhealthy?"

It should be:

"What happened between the proxy and the upstream socket?"

That change in thinking saves a lot of debugging time.

Root Cause Breakdown

1. Race Condition: The HTTP Keep-Alive Timeout Mismatch

This is one of the nastiest causes because everything can look perfectly healthy. Imagine your ingress controller maintains a pool of persistent TCP connections to your application.

Instead of doing this for every request:

Request 1 -> TCP connection -> close
Request 2 -> TCP connection -> close
Request 3 -> TCP connection -> close

it does this:

TCP connection
      |
      +--> Request 1
      +--> Request 2
      +--> Request 3
      +--> Request 4

That's HTTP keep-alive. It reduces TCP connection setup overhead and is normal in production. But now imagine the proxy and application disagree about how long an idle connection should remain alive.

For instance:

NGINX upstream keep-alive: 60 seconds
Node.js server timeout: 30 seconds

The application is allowed to close an idle connection after 30 seconds. NGINX believes the connection can remain available for 60 seconds. Now the race looks like this:

Time →

0s Request completes
         |
         | connection becomes idle
         |
30s Application closes socket
         |
         X
         |
60s NGINX still thinks connection is reusable
         |
         | next request
         v
      stale socket
         |
         X ECONNRESET / closed connection
         |
         v
       502

This can produce an extremely confusing pattern:

The Pod is still running. The process is still running. The readiness probe still succeeds. But a connection in the proxy's connection pool is stale.

Why `ECONNRESET` matters

At the application or proxy level, you may see something similar to:

ECONNRESET
upstream prematurely closed connection
connection reset by peer
upstream timed out

ECONNRESET means the TCP connection was reset rather than completing normally. It doesn't automatically prove a keep-alive mismatch, but it is an important clue.

The fix

The basic rule is:

Don't let the proxy assume an idle upstream connection will survive longer than the backend actually keeps it alive.

For example, if the backend closes idle connections after 30 seconds, configure the proxy's reusable connection lifetime appropriately below that boundary. The exact settings depend on the proxy and application runtime.

For an NGINX-style configuration, you might encounter settings such as:

upstream backend {
    server app:8080;

    keepalive 32;
    keepalive_timeout 25s;
}

On the application side, the corresponding timeout might be configured around:

30s

The exact numbers are not universal. The important thing is that you understand which component owns each timeout. Do not blindly set everything to five minutes because "more keep-alive is better."

Longer idle connections mean more persistent sockets and more resources.

2. Kernel Queues: Sockets, `SOMAXCONN`, and TCP Backlog Drops

Now let's move lower. Suppose traffic suddenly increases. Your application normally receives: 100 connections/sec. Then an event sends: 5,000 connections/sec. Your Pod does not necessarily fail immediately.

Instead, connections begin accumulating in queues. When a server calls:

listen(fd, backlog);

it tells the kernel that the socket is a listening socket and provides a requested backlog.

For instance:

listen(server_fd, 128);

The actual behavior depends on the operating system and runtime, and Linux also has system-wide limits such as:

cat /proc/sys/net/core/somaxconn

You might see: 4096

SOMAXCONN places an upper bound relevant to the listen backlog requested by applications. The important mental model is:

Incoming connections
        |
        v
+----------------------+
| Linux TCP structures |
| and connection queues |
+----------------------+
        |
        v
     accept()
        |
        v
Application

If connections arrive faster than the application can accept them, queues can fill.

What happens when the application cannot keep up?

Imagine:

Traffic:
5000 connections/sec

Application accepts:
1000 connections/sec

The difference has to go somewhere. As queues fill, connection attempts can experience delays, retransmissions, or failures depending on the exact TCP state, kernel settings, workload, and application behavior.

From the proxy's perspective, that can eventually become:

connect() failed
connection timed out
connection refused
upstream unavailable

And the client may see: 502 Bad Gateway.

Again, Kubernetes might still report:

Pod: Running
Ready: True

because the process hasn't crashed. The process is simply struggling at the socket boundary.

Inspecting the socket state

Start with:

ss -lntp

For more detail:

ss -lnt

You might see:

State Recv-Q Send-Q Local Address:Port
LISTEN 128 0 0.0.0.0:8080

For a listening socket, the queue-related values can give you clues about whether connections are accumulating. You can also inspect established connections:

ss -ant

Or filter by port:

ss -ant '( sport = :8080 or dport = :8080 )'

During an incident, look for unusual growth in:

SYN-SENT
SYN-RECV
ESTAB
CLOSE-WAIT
TIME-WAIT

Each state tells a different story. For instance, a large number of:

CLOSE-WAIT

can indicate that the remote side has closed the connection but the application has not closed its local socket. That is very different from a TCP backlog problem. This is why simply looking at "number of connections" isn't enough. You want to know what state those connections are actually in.

Kernel-level tuning

You may encounter:

sysctl net.core.somaxconn

and:

sysctl net.ipv4.tcp_max_syn_backlog

These control different parts of the connection-handling path. A common mistake is to increase them blindly:

sysctl -w net.core.somaxconn=65535

and declare victory. That doesn't magically make an application capable of accepting 65,535 connections.

If the application has a slow event loop, blocked workers, exhausted file descriptors, or insufficient CPU, a larger queue can simply allow more work to accumulate.

Tune the kernel alongside the application. Also remember that Kubernetes does not make node-level kernel settings disappear. The network stack is still Linux underneath.

3. The Termination Gap: `SIGTERM`, Endpoint Propagation, and Inflight Traffic Drops

This is another classic source of intermittent 502s. Suppose Kubernetes wants to terminate a Pod. The lifecycle looks roughly like:

Pod termination begins
        |
        v
SIGTERM sent to process
        |
        +----------------------+
        | |
        v v
Endpoint removal Application shutdown
        | |
        v v
Traffic stops Socket closes
eventually

The problem is that these operations are not one instantaneous atomic event. There is propagation time. Imagine a request arrives just as a Pod starts terminating.

A proxy or load-balancing path may still have traffic associated with that Pod while Kubernetes networking components and endpoint consumers are converging on the new state.

Meanwhile, your application receives:

SIGTERM

and immediately does:

close(listening_socket);
exit(0);

Now imagine the timing:

T0 Pod is serving traffic

T1 Kubernetes starts termination

T2 Application receives SIGTERM

T3 Application closes listening socket

T4 Some traffic still reaches old Pod path

T5 Proxy attempts connection

T6 Connection fails/reset

T7 Proxy returns 502

The exact packet path depends on the Kubernetes networking implementation, kube-proxy mode, CNI, service topology, proxy behavior, and timing. But the underlying problem remains:

traffic draining and process shutdown must be coordinated.

Don't treat `SIGTERM` as "exit immediately"

A production application should interpret SIGTERM as:

"Stop accepting new work and gracefully finish what you're already doing."

Not:

"Die now."

A typical graceful shutdown sequence is:

SIGTERM
   |
   v
Stop accepting new work
   |
   v
Allow existing requests to finish
   |
   v
Close keep-alive connections
   |
   v
Close listening socket
   |
   v
Exit

Kubernetes gives you mechanisms to help with this.

For instancee:

spec:
  terminationGracePeriodSeconds: 30

You can also use a preStop lifecycle hook.

A simple example:

lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - "sleep 10"

The purpose isn't to randomly sleep because Kubernetes likes sleeping Pods. The purpose is to give the system time to drain traffic and propagate endpoint changes before the process disappears.

The correct duration depends on your environment. Ten seconds isn't a magic number. Thirty seconds isn't a magic number either. Measure the actual propagation and request-drain behavior in your cluster.

Beyond Kubernetes Abstractions: Are Syscalls and Sockets Enough?

If you are debugging this class of outage, yes, you need to get comfortable with syscalls and sockets. But raw syscalls alone aren't enough.

You need to correlate four layers:

Layer 1: Kubernetes
    Pod / Service / EndpointSlice
              |
Layer 2: Proxy
    NGINX / Envoy / HAProxy
              |
Layer 3: Linux
    sockets / TCP / queues
              |
Layer 4: Application
    Node.js / Go / Python / Java

Kubernetes tells you where traffic should go. The proxy tells you what it experienced. Linux tells you what happened to the connection. The application tells you why it behaved that way. You need all four when the failure is subtle.

Start with `ss`

First inspect listening sockets:

ss -lntp

Then established connections:

ss -antp

For a specific port:

ss -antp | grep ':8080'

Look for patterns rather than individual connections.

For instance:

many CLOSE-WAIT
many SYN-RECV
rapid connection churn
unexpected connection resets

These patterns can tell you whether you are dealing with application cleanup, connection pressure, or connection-establishment problems.

Use `strace` to See What the Process Actually Does

When the application is behaving strangely, strace can expose the system calls underneath the runtime.

For instance:

strace -f -e trace=network -p <PID>

You may see operations resembling:

socket(...)
bind(...)
listen(...)
accept4(...)
recvfrom(...)
sendto(...)
close(...)

Or something like:

accept4(...) = -1 EMFILE

That is a very different incident. EMFILE means the process has exhausted its file descriptors. The Pod can still be running but the application cannot accept new sockets.

Now the Kubernetes health dashboard suddenly makes a lot more sense: the Pod isn't dead; the application is resource-starved.

You can inspect limits with:

ulimit -n

and:

cat /proc/<PID>/limits

This is one of the reasons SRE debugging eventually moves beyond container-level metrics.

Use `tcpdump` When You Need the Truth

When logs disagree with reality, capture packets. Inside a suitable network namespace or node:

tcpdump -ni any host <POD_IP> and port 8080

Or:

tcpdump -ni any tcp port 8080

Now you can observe things such as:

SYN
SYN-ACK
ACK
PSH
ACK
FIN
RST

A TCP reset (RST) is especially interesting.

For instance:

Proxy Pod
  | |
  | -------- SYN ------------> |
  | <------- SYN/ACK --------- |
  | -------- ACK ------------> |
  | |
  | -------- HTTP ------------>|
  | |
  | <--------- RST ------------|
  | |
  v
502

Now you have something much more useful than:

Pod: Healthy

You have evidence that a TCP connection was reset. The next question becomes:

Who sent the reset, and why?

That's a question you can investigate.

How to Fix It Permanently

The goal isn't to make the dashboard green. The goal is to make the connection lifecycle deterministic.

Step 1: Align Proxy and Application Keep-Alive Behavior

Inventory the relevant timeouts.

For instance:

Client idle timeout
        |
Load balancer timeout
        |
Ingress keep-alive timeout
        |
Proxy upstream keep-alive
        |
Application keep-alive
        |
Application request timeout

Write the actual values down. Don't assume them.

For instance:

Proxy idle connection: 60s
Application idle timeout: 30s

That's a potential mismatch. Adjust the configuration so the proxy does not retain reusable upstream connections beyond the backend's effective lifetime. Then test it under real keep-alive traffic.

A useful test is to deliberately leave connections idle and reuse them later. You want to prove that:

idle connection
      |
      v
backend timeout
      |
      v
proxy reuse

doesn't result in a stale connection being handed to a request.

Step 2: Inspect and Tune Socket Queues

Check:

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog

Inspect the application's listener:

ss -lntp

Then check the application's file descriptor limits:

ulimit -n

and:

cat /proc/<PID>/limits

If the application allows configuring its listen backlog, make sure it is appropriate for the workload.

For instance, a Go application may explicitly configure a listener:

listener, err := net.Listen("tcp", ":8080")
if err != nil {
    log.Fatal(err)
}

The runtime and operating system then determine the actual socket behavior. For other runtimes, the configuration may expose backlog directly.

The important part is to avoid treating:

SOMAXCONN

as a magic performance knob. If the application cannot process connections fast enough, increasing the queue only changes where the pressure accumulates.

Also consider:

CPU saturation
file descriptor exhaustion
worker/thread exhaustion
event-loop blocking
connection pool exhaustion

These often appear alongside socket pressure.

Step 3: Build a Real Graceful Shutdown Path

A better Kubernetes deployment might look like:

spec:
  terminationGracePeriodSeconds: 30

  containers:
    - name: app
      image: example/app:1.0

      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - "sleep 10"

But don't stop there. The application itself should handle SIGTERM.

For instance:

SIGTERM
   |
   v
Set "shutting down" state
   |
   v
Fail readiness
   |
   v
Stop accepting new requests
   |
   v
Drain existing requests
   |
   v
Close listeners
   |
   v
Exit

This is much stronger than simply adding:

preStop:
  exec:
    command: ["sleep", "10"]

and hoping everything works. The sleep is a drain window, not a replacement for graceful shutdown.

Step 4: Verify Endpoint Propagation

During a controlled rollout, watch:

kubectl get pods -w

and:

kubectl get endpointslices -A -w

You want to understand the sequence:

Pod begins termination
        |
        v
Readiness changes
        |
        v
EndpointSlice changes
        |
        v
Traffic drains
        |
        v
Application exits

The exact ordering and timing depend on your Kubernetes version and networking stack, so measure it rather than assuming there is a fixed delay.

If you routinely see requests reaching terminating Pods, investigate:

readiness behavior
endpoint propagation
ingress connection reuse
load balancer draining
application shutdown handling
preStop
terminationGracePeriodSeconds

Step 5: Make the Failure Observable

A generic metric like:

Pod availability = 100%

is not enough. Track the failure boundary. For the ingress layer, useful signals include:

HTTP 502 count
upstream connection errors
upstream resets
upstream timeouts
active connections
connection reuse

At the application layer:

request latency
active connections
open file descriptors
connection errors
graceful shutdown duration
request rejection

At Linux level:

TCP retransmissions
socket states
listen queue pressure
file descriptor usage
network drops

Now you can distinguish:

Pod is unhealthy

from:

Pod is healthy, but proxy-to-Pod TCP connections are failing

Those are completely different incidents.

A Practical Debugging Sequence

When users report intermittent 502s while Kubernetes says everything is healthy, don't randomly restart Pods.

Walk down the stack.

1. Confirm where the 502 originates

Check ingress/proxy logs.

Look for:

upstream reset
upstream prematurely closed connection
connect() failed
connection refused
timeout
ECONNRESET

2. Identify the backend Pod

Determine which Pod the proxy attempted to reach.

kubectl get pods -o wide

3. Check the application directly

From inside the cluster:

curl -v http://<POD_IP>:8080/health

Then test the actual application endpoint:

curl -v http://<POD_IP>:8080/api/users

A healthy /health endpoint doesn't prove that the application can successfully process every real request.

4. Inspect sockets

ss -antp
ss -lntp

5. Check file descriptors

ulimit -n
cat /proc/<PID>/limits

6. Trace system calls when necessary

strace -f -e trace=network -p <PID>

7. Capture packets

tcpdump -ni any tcp port 8080

8. Compare timeouts

Write down the values for:

Ingress
Load Balancer
Application
TCP keep-alive
Request timeout
Idle timeout

Don't rely on memory.

9. Reproduce under load

Use a controlled load test. A failure that occurs once every 20,000 requests is difficult to diagnose manually. A reproducible failure under controlled traffic is much easier to understand.

Why Kubernetes Health Checks Miss This

A readiness probe might execute:

GET /health

and receive:

200 OK

That proves something useful. It proves that at that moment, the kubelet could successfully perform that health check according to the configured probe mechanism.

It does not prove:

Every existing TCP connection is healthy.
Every proxy connection is fresh.
The listen queue isn't saturated.
The process isn't close to its FD limit.
No connection will reset.
The application won't terminate during an inflight request.
The proxy's connection pool contains no stale sockets.

This is the gap between health and correctness under traffic.

A Pod can be alive. A Pod can be ready. A Service can have endpoints. And a specific TCP connection can still be dead. That's not contradictory. They're different layers observing different things.

The Mental Model to Keep

When you see:

502 Bad Gateway

think:

Client
  |
  | request
  v
Proxy
  |
  | "I couldn't successfully talk to upstream"
  |
  X
Backend connection
  |
  +-- TCP failure?
  +-- stale keep-alive?
  +-- backlog pressure?
  +-- application closed socket?
  +-- graceful shutdown race?
  +-- FD exhaustion?
  +-- timeout?

Then move downward:

Ingress logs
      ↓
Service / EndpointSlice
      ↓
Pod IP
      ↓
Application process
      ↓
Linux socket
      ↓
TCP state
      ↓
Packets

That's how you turn a "ghost outage" into a concrete failure.

3 Key Takeaways

A green Pod does not mean every network connection to that Pod is healthy. A 502 usually means the proxy failed to get a valid upstream response, and the failure may exist entirely below the Kubernetes health-check layer.
Keep-alive mismatches, socket queue pressure, and termination races are connection-lifecycle problems. Fix them by aligning proxy/application timeouts, understanding listen() backlogs and kernel limits, and giving terminating Pods enough time to drain traffic.
When Kubernetes abstractions stop explaining the outage, inspect Linux. ss shows socket state, strace shows what the process is asking the kernel to do, and tcpdump shows what actually crossed the network. Together, they can expose the failure that a green dashboard cannot.

Kubernetes Networking Was Easy — Until Production Got Real

Kubernetes with Naveen — Fri, 04 Sep 2026 11:32:38 +0000

When I first started managing microservices at scale, Kubernetes felt like magic — until it didn’t. Pods restarted, requests vanished into the void, and tracing cross-namespace traffic became a weekend sport.

At first, the networking model seemed almost too simple.

A Pod gets an IP. A Kubernetes Service gives that Pod a stable virtual endpoint. CoreDNS handles service discovery. The CNI provides pod-to-pod connectivity. kube-proxy handles service traffic using mechanisms such as iptables or IPVS.

For a small cluster, that model works remarkably well. Then you have 150 services. Then 500.

Then teams start deploying independently, services communicate across namespaces, traffic crosses availability zones, a few workloads become extremely chatty, and somebody introduces three layers of retries because the network is sometimes flaky.

That's when the networking stops being infrastructure you can ignore. You start asking questions that Kubernetes Services alone don't really answer:

Which service is calling this endpoint?
Why did this request get a 503?
Which version received it?
Is the connection failing, or is the application returning the error?
Why did latency jump only for traffic crossing namespaces?
Is this workload actually talking to the service it thinks it is?

This is where a service mesh starts making sense. And this is also where you discover that a service mesh isn't free.

The Architectural Shift: From Kubernetes Services to a Service Mesh

Kubernetes gives you the basic network primitives. The CNI establishes the network between Pods. A Service provides a stable virtual IP and load-balancing abstraction. kube-proxy programs the node networking rules required to direct Service traffic toward backend Pods.

That's enough to answer:

How does service-a reach service-b?

But production systems eventually ask:

How should service-a reach service-b?

Those are very different questions. Suppose payments has two versions:

payments-v1 → 90%
payments-v2 → 10%

Kubernetes Services don't natively give you application-aware traffic splitting based on HTTP headers, cookies, weights, or request properties. You can create separate Services, manipulate Deployments, introduce an ingress controller, or build application-level routing logic. But now routing logic starts leaking into multiple layers. Istio moves much of that policy into the networking layer.

The basic architecture becomes:

Istio Control Plane

                               │
                  Configuration / Certificates
                               │
                               ▼
                        ┌─────────────┐
                        │    Envoy    │
                        │   Proxies   │
                        └─────────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        ▼                      ▼                      ▼
   service-a               service-b               service-c
   + Envoy                 + Envoy                 + Envoy

The important distinction is between the control plane and data plane. The Istio control plane manages configuration and security material. The Envoy proxies sit in the traffic path and actually handle requests. That distinction matters enormously when debugging.

The control plane can be struggling while existing Envoy configuration continues serving traffic perfectly well. Understanding that separation saves a lot of unnecessary panic during incidents.

The Kubernetes Layer Still Matters

One mistake I see repeatedly is treating Istio as if it replaces Kubernetes networking. It doesn't. Your underlying Kubernetes networking still needs to work.

A typical request might look like:

Pod A
  │
  ▼
Envoy sidecar
  │
  ▼
ClusterIP
  │
  ▼
Kubernetes networking
  │
  ▼
Envoy sidecar
  │
  ▼
Pod B

The mesh adds another layer of behavior on top of Kubernetes networking. That means there are now more places where things can go wrong.

DNS → Service → CNI → routing → Envoy listener → Envoy cluster → application

When someone says:

The service is reachable.

That statement is almost meaningless without knowing from where, through what path, and at which layer.

Ingress Isn't Service-to-Service Networking

Another production lesson is keeping north-south and east-west traffic conceptually separate. North-south traffic is traffic entering or leaving the cluster. East-west traffic is traffic between workloads inside the cluster.

An Istio Ingress Gateway handles the former.

For instance:

Internet
   │
   ▼
Load Balancer
   │
   ▼
Istio Ingress Gateway
   │
   ▼
VirtualService
   │
   ▼
service-a

Inside the cluster:

service-a
   │
   ▼
Envoy
   │
   ▼
service-b
   │
   ▼
Envoy

These are different traffic-management problems. The ingress gateway is your controlled entry point. The sidecars or ambient data plane handle service-to-service communication. Mixing those responsibilities makes architecture and troubleshooting unnecessarily difficult.

1. Securing the Perimeter: Ingress and Egress

A production cluster shouldn't have every workload freely reaching the internet. Direct Pod egress looks convenient:

Pod
 │
 ├── api.example.com
 ├── payment-provider.com
 ├── random-third-party.com
 └── anything-else

The problem isn't simply security. It's control.

When something goes wrong, you want to know:

Which workload made the connection?
Where did it connect?
Was the destination approved?
What protocol was used?
Can we block it centrally?
Can we observe the traffic?

An Istio Egress Gateway gives you a controlled exit point:

Pod
 │
 ▼
Envoy
 │
 ▼
Egress Gateway
 │
 ▼
External Service

Now outbound traffic can be governed at a predictable boundary. That doesn't mean every organization needs to force every packet through an egress gateway. You pay for centralized inspection with additional hops, infrastructure, configuration, and failure modes.

The right question isn't:

Can we put everything through the egress gateway?

It's:

Which external traffic actually needs centralized policy and visibility?

That distinction matters.

2. mTLS: Zero Trust Without Breaking Everything

One of Istio's strongest capabilities is mutual TLS.

Instead of:

service-a ───── HTTP ─────> service-b

you can have:

service-a
   │
 Envoy
   │
   │ mTLS
   ▼
 Envoy
   │
service-b

The application doesn't necessarily need to manage certificates itself. The mesh handles identity and encryption between workloads. But switching an existing production environment directly to:

mode: STRICT

can turn a quiet Tuesday into a very long night. Why? Because not everything is necessarily inside the mesh.

You might have:

mesh workload
     │
     ▼
legacy service
     │
     X
   TLS required

The legacy workload doesn't have an Envoy sidecar and therefore cannot participate in mesh mTLS in the same way.

This is where PeerAuthentication modes matter.

PERMISSIVE

Accept both plaintext and mTLS. Useful during migration.

mTLS ────────┐
             ├──> workload
plaintext ───┘
STRICT

Require mTLS.

plaintext ───> rejected
mTLS ────────> accepted

A safer migration looks like:

Phase 1
PERMISSIVE
   ↓
Inject sidecars
   ↓
Verify workload communication
   ↓
Identify legacy clients
   ↓
Migrate dependencies
   ↓
STRICT

Don't turn on STRICT because the architecture diagram says everything is meshed. Production traffic doesn't care what the architecture diagram says.

3. Traffic Shifting Without DNS Games

Canary deployments are another area where Istio becomes extremely useful. Without mesh-level routing, teams sometimes create:

payments-v1.example.com
payments-v2.example.com

and manipulate DNS or load balancers. DNS isn't designed to provide precise request-level traffic control. Caching, TTLs, resolvers, client behavior, and connection reuse all get involved. Istio lets you shift traffic directly at the request-routing layer.

A VirtualService can express something conceptually like:

payments
   │
   ├── v1 → 90%
   │
   └── v2 → 10%

Then:

90% → payments-v1
10% → payments-v2

You can gradually move:

The interesting part isn't the YAML. It's what you can do without changing application code or DNS. But traffic splitting introduces another responsibility: knowing what you're actually measuring. If v2 receives 10% of traffic but happens to receive the most expensive customer requests, raw request percentages can become misleading.

Traffic management is easy. Traffic management with meaningful telemetry is the real engineering problem.

DestinationRules: Where Routing Gets Interesting

A VirtualService describes how requests should be routed. A DestinationRule describes policies applied to traffic going toward a destination.

This is where things such as subsets, connection pools, circuit breakers, outlier detection, and TLS behavior start becoming relevant.

For instance:

payments
   │
   ├── subset: v1
   │
   └── subset: v2

VirtualService:

route 90% → v1
route 10% → v2

DestinationRule:

v1 → connection policy
v2 → connection policy

That separation becomes valuable once routing rules grow beyond simple send traffic here. It also becomes a source of configuration complexity. We'll get to that.

The Battle Scars: The Istio Tax

Let's talk about the part nobody gets excited about during the architecture presentation. Every sidecar consumes resources. One proxy doesn't sound like much. Now multiply it.

500 workloads
×
1 Envoy proxy
=
500 additional processes

At low traffic, this can look harmless. Under high concurrency, it isn't. Envoy maintains connections, buffers data, processes HTTP, performs TLS operations, tracks metrics, handles filters, and maintains configuration.

Memory consumption can become particularly painful. Imagine a cluster where application Pods were sized carefully:

Application:
500Mi memory

Envoy:
250Mi memory

Suddenly your 500 MiB application workload isn't a 500 MiB workload anymore. It's closer to: 750MiB+. And that changes: node packing, autoscaling, eviction pressure, cluster cost, and pod startup behavior

CPU can also spike under heavy request rates, TLS operations, logging, or complex filters. This is why blindly enabling sidecars everywhere is dangerous.

Control the scope

Istio's Sidecar resource can be used to constrain the configuration visibility available to workloads. That's important in larger environments.

A proxy doesn't necessarily need configuration for every service in the cluster. If a workload can only communicate with payments, orders, and identity there's little reason for its Envoy to carry unnecessary configuration for hundreds of unrelated services.

Reducing configuration scope can improve both resource usage and operational clarity.

The exact optimization depends on the cluster, but the principle is simple:

Don't make every proxy understand the entire universe if it only needs to understand three services.

The Battle Scars: The Danger of Naive Retries

his one has caused some spectacular incidents. Let's imagine service-a → service-b, service-b becomes slow. So service-a retries.

Now imagine 100 clients doing the same thing. Then each client has three retries. You can accidentally transform 1,000 requests into 4,000 requests when the dependency is already struggling.

That's a retry storm. And it gets uglier with multiple layers.

Frontend
  ↓ retry × 3
Service A
  ↓ retry × 3
Service B
  ↓ retry × 3
Service C

One failed request can explode into a ridiculous number of downstream attempts. The service mesh makes retries easy to configure. That doesn't mean you should configure them everywhere.

Retries should be deliberate. So use bounded retry counts, exponential backoff, jitter, appropriate timeout budgets, retry budgets, circuit breaking, and outlier detection

The idea behind a retry budget is particularly important: retries should consume only a controlled fraction of normal traffic rather than being allowed to multiply without bound. And never retry operations blindly. Retrying a failed GET may be reasonable. Retrying a payment operation without understanding idempotency can be a financial incident.

Circuit Breaking and Outlier Ejection

A service that's returning failures shouldn't necessarily continue receiving traffic indefinitely. DestinationRules can define connection-pool and outlier-detection behavior.

Conceptually:

service-b
   │
   ├── healthy instance
   ├── healthy instance
   └── failing instance
             │
             ▼
       outlier detection
             │
             ▼
          ejected

The unhealthy endpoint can temporarily be removed from load balancing. This can stop one broken instance from poisoning every caller.

But again, thresholds matter. Make them too aggressive and healthy instances can get ejected during normal traffic variation. Make them too lenient and the protection arrives too late.

There is no magical production value. You need to understand your workload.

The Battle Scars: Debugging the Ghost 503s

Few things are more irritating than: The application is healthy. while users are receiving:

503 Service Unavailable

This is where Envoy's response flags become incredibly useful. One example you'll eventually encounter is: 503 NR (NR means No Route). The request reached Envoy, but Envoy couldn't find a valid route for it.

That is very different from:

application returned HTTP 503

The HTTP status looks similar from the outside. The failure is completely different. When debugging, I want to know who generated the 503? Then:

Did Envoy have a route?
Did it have a cluster?
Did the cluster have endpoints?
Could it establish a connection?
Did TLS negotiation work?
Did the upstream respond?
Did the application return the status?

That's the difference between randomly restarting Pods and actually debugging the system.

Read the Envoy Access Logs

A useful Envoy access log can tell you considerably more than an application log. You want fields such as downstream address, request method, request path, response code, response flags, upstream host, request duration, and upstream service time.

Suppose you see:

HTTP 503
response_flags=NR

Start looking at routing configuration. Check:

VirtualService
DestinationRule
Service
ServiceEntry
Gateway

If instead you see something indicating an upstream connection failure, your investigation moves toward:

endpoint health
network connectivity
TLS
connection limits
upstream availability

And if Envoy successfully reaches the upstream and the application itself returns 503, stop blaming the mesh. The application is returning the error. This sounds obvious. During an outage, it's surprisingly easy to forget.

Configuration Drift Is a Silent Killer

Another lesson from large clusters: The configuration you think exists and the configuration Envoy is actually running are not necessarily the same thing. Istio configuration flows through the control plane before reaching the proxies.

When configuration changes, you need to know What was applied? What did Istio accept? What configuration did Envoy receive? and What configuration is Envoy actually using? This is why operational tooling matters.

Commands such as:

istioctl proxy-status

and:

istioctl proxy-config

become part of your everyday debugging toolkit.

For instance:

istioctl proxy-status

can help identify proxies that aren't synchronized correctly.

Then you can inspect specific configuration categories on a problematic workload rather than staring at YAML for an hour.

The important lesson is:

Source configuration is not the same thing as runtime configuration.

That's true in Kubernetes generally. Istio just gives you another layer where drift can happen.

Sidecars vs. Ambient Mesh

Sidecar-based Istio is the model most engineers first encounter:

Pod
├── Application
└── Envoy

It works, it's mature, and it's conceptually straightforward. But it also means every workload carries proxy overhead. Istio's ambient mesh changes the data-plane architecture by moving away from requiring an Envoy sidecar in every Pod.

The important point isn't that ambient is "better." It's that it changes the operational trade-offs.

With sidecars:

Every workload
      ↓
Envoy sidecar

With Ambient:

Workloads
    │
    ▼
Node / shared mesh components
    │
    ▼
Optional higher-level L7 processing

That can reduce some per-Pod overhead and simplify certain adoption scenarios. But it introduces a different set of components and operational concepts. If you're already operating a mature sidecar mesh successfully, moving to ambient isn't automatically an upgrade worth doing.

Infrastructure decisions should be driven by an actual problem, not architectural fashion.

The Control Plane Is Part of Your Production Dependency Chain

There's another uncomfortable reality. Once you depend heavily on Istio, the control plane becomes part of your platform's operational surface.

Upgrading it isn't:

kubectl apply

You need to think about:

API compatibility
proxy compatibility
configuration changes
CRD behavior
gateway behavior
certificate management
control-plane resource usage
rollout sequencing
rollback strategy

And don't assume that because the control plane upgrade succeeded, every proxy is healthy. Check proxy synchronization, check gateways, check workloads, and check actual traffic.

A control-plane upgrade that looks green in Kubernetes can still leave a subset of proxies running unexpected configuration.

What I Would Actually Do?

If I were building a new Kubernetes platform today, I wouldn't start by installing Istio everywhere. I'd start with Kubernetes networking and make sure that CNI works, DNS works, Services work, NetworkPolicies are understood, Ingress is predictable. Observability exists before adding another layer. Then I'd identify the problems that justify a mesh.

If the organization needs:

workload-to-workload mTLS
consistent service identity
advanced traffic splitting
request-level telemetry
standardized retries and circuit breaking
controlled east-west traffic
centralized egress policy

then Istio starts earning its operational cost. But if you have 20 services, 3 engineers, low traffic, simple architecture and the biggest production problem is that nobody understands Kubernetes Services yet, installing a service mesh probably isn't going to save you.

It may actually make things worse.

The Platform Engineer's Verdict

Istio is one of those tools that becomes extremely valuable once your Kubernetes environment has problems that Kubernetes itself wasn't designed to solve. When you're operating hundreds of services, managing service-to-service encryption, running canary deployments, debugging cross-service latency, and trying to understand why a request disappeared somewhere between two namespaces, having a consistent networking layer can make an enormous difference. mTLS, traffic splitting, circuit breaking, outlier detection, service identity, and detailed Envoy telemetry are not just nice features when you're dealing with a large production platform. They can turn an otherwise opaque failure into something you can actually investigate.

But I've also seen teams introduce Istio far earlier than they needed it and end up spending more time debugging the mesh than solving their original networking problems. Every sidecar consumes CPU and memory. Every VirtualService and DestinationRule becomes another piece of configuration that someone has to understand. Every control-plane upgrade needs planning and validation. A badly configured retry policy can turn a small downstream failure into a cluster-wide incident, and a single routing mistake can produce a wall of mysterious 503s. The mesh gives you more control, but it also gives you more ways to shoot yourself in the foot.

My rule is fairly simple: don't adopt Istio because your architecture diagram looks more impressive with it. Adopt it because you have a networking problem that justifies the operational cost. Start with solid Kubernetes networking, clear service ownership, sensible observability, and NetworkPolicies. Introduce the mesh gradually around the workloads that benefit from it, keep routing policies simple, put strict limits around retries and resources, and make Envoy debugging part of your team's operational knowledge. If you do that, Istio becomes a useful platform capability rather than another layer of infrastructure that everyone is afraid to touch when production breaks.

What Happens When You Create a Pod in Kubernetes?

Kubernetes with Naveen — Mon, 17 Aug 2026 11:03:15 +0000

So, like when you type:

kubectl apply -f pod.yaml

A second later, Kubernetes might respond with:

pod/nginx created

It is tempting to think that Kubernetes simply read your YAML and immediately started an NGINX container somewhere in the cluster. That's not what happened. Behind that single command, several Kubernetes components have already started working together. The API server receives your request, authenticates you, checks your permissions, runs admission logic, and stores the desired state. The scheduler then finds a suitable node, the kubelet on that node takes responsibility for the Pod, and the container runtime eventually creates the actual container.

For the CKA, this sequence is worth understanding because Kubernetes troubleshooting becomes much easier once you know which component is responsible for which part of the journey. If a Pod is Pending, you should be thinking about scheduling. If it is stuck at ContainerCreating, you start looking toward the kubelet, runtime, networking, or storage. If it is in ImagePullBackOff, you don't waste time debugging the scheduler because scheduling has already happened.

Let's walk through the entire journey as if we're standing in front of a whiteboard.

The Story Begins: kubectl to kube-apiserver

Let's start with a simple Pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.27
      ports:
        - containerPort: 80

Now you run:

kubectl apply -f pod.yaml

he first thing to understand is that kubectl does not create the container. It doesn't connect directly to containerd, it doesn't SSH into a worker node, and it doesn't write anything directly into etcd. kubectl is primarily a client for the Kubernetes API. It takes the configuration you provide, constructs an API request, and sends that request to the kube-apiserver.

You can think of the API server as the front door of the Kubernetes cluster. Almost every major Kubernetes component communicates through this API: users, controllers, the scheduler, kubelets, and many other components. So when you execute kubectl apply, your request first enters the control plane through the API server. At this point, Kubernetes hasn't created a container and hasn't even selected a worker node. You've simply submitted a request saying, "This is the state I want Kubernetes to maintain."

The journey starts roughly like this:

kubectl
   |
   v
kube-apiserver

From here, the API server has several questions to answer before it accepts your request.

Step 1: Authentication — "Who Are You?"

Before Kubernetes can decide what you're allowed to do, it needs to know who is making the request. This is the job of authentication. When you use kubectl, your client configuration normally contains credentials or instructions for obtaining credentials. Depending on the cluster, Kubernetes might use client certificates, bearer tokens, OIDC, cloud-provider identity mechanisms, or another authentication method.

You can inspect your current Kubernetes configuration with:

kubectl config view

The important thing for the CKA is to keep authentication and authorization separate in your head. Authentication establishes your identity; it doesn't determine what you're allowed to do. If Kubernetes determines that the request came from a user called naveen, it has answered the first question: Who are you? It still needs to answer the second question: Are you allowed to create this Pod?

Step 2: Authorization — Are You Allowed to Do This?

Now the API server checks authorization. Kubernetes needs to determine whether the authenticated identity has permission to perform the requested operation on the requested resource in the requested namespace. In most Kubernetes environments, this is handled using RBAC — Role-Based Access Control.

For example, a user might be allowed to read Pods:

get pods *list pods
watch pods

but not create or delete them:

create pods
delete pods

You can test permissions directly with kubectl:

kubectl auth can-i create pods

You can also check against a specific namespace:

kubectl auth can-i create pods -n default

If Kubernetes responds with: yes the request can continue. If it responds with: no

the API server rejects the operation. This is one of those small distinctions that frequently appears in interviews and CKA questions: authentication answers Who are you? while authorization answers What are you allowed to do?

Step 3: Admission — Should We Allow This Request?

Passing authentication and authorization doesn't automatically mean the Pod will be accepted. The request now goes through the admission control stage. Admission controllers get an opportunity to inspect the API request after authentication and authorization but before the object is persisted.

There are two important concepts here: mutating admission and validating admission. A mutating admission controller can modify the object before it is stored. For example, an admission webhook might inject configuration, add labels, modify security settings, or inject a sidecar container. A validating admission controller doesn't modify the object; instead, it checks whether the object satisfies a particular policy and can reject it if it doesn't.

Custom admission webhooks are particularly common in production Kubernetes environments. A MutatingAdmissionWebhook might modify your Pod, while a ValidatingAdmissionWebhook might reject it because the image comes from an unapproved registry or because the workload violates an organizational security policy.

The request now looks something like this:

kubectl
   |
   v
kube-apiserver
   |
   +--> Authentication
   |
   +--> Authorization / RBAC
   |
   +--> Admission
            |
            +--> Mutating
            |
            +--> Validating
   |
   v
accepted

This stage is also useful when troubleshooting. If kubectl apply immediately returns an error saying that the request was forbidden or rejected by a policy, you don't need to start inspecting kubelet logs on worker nodes. The request never got that far.

The State Is Written: kube-apiserver to etcd

Assume our Pod passes authentication, authorization, admission, and API validation. The API server now has an accepted Pod object that needs to become part of the cluster's state. This is where etcd enters the picture.

etcd is the distributed key-value store used by Kubernetes to persist cluster state. For our Pod, the API server stores the Pod's desired configuration there. But notice something important: the Pod has been accepted and stored, but it is not necessarily running yet. At this point Kubernetes knows what you want, but nobody has necessarily decided where the Pod should run.

You can think of etcd as the cluster's persistent source of truth. If Kubernetes needs to know about an object such as a Pod, Deployment, Service, ConfigMap, Secret, or many other API resources, the API server manages that state. The Pod object can exist in the cluster even while there is no running container behind it. This distinction is fundamental to understanding Kubernetes: creating the Kubernetes object and starting the workload are different events.

Conceptually, the state currently looks like this:

Pod: nginx
Image: nginx:1.27
Desired state: exists
Node: not assigned yet
Container: not running yet

The Pod is now waiting for the next component in the chain.

Why Doesn't Everything Talk Directly to etcd?

One important Kubernetes architecture rule is that normal Kubernetes components don't simply connect directly to etcd whenever they want cluster information. The kube-apiserver acts as the central API gateway between Kubernetes components and the cluster's persistent state.

The simplified communication pattern looks like this:

kube-scheduler ---> kube-apiserver ---> etcd
kubelet ---------> kube-apiserver
kubectl ----------> kube-apiserver
controllers ------> kube-apiserver

Instead of every component having its own direct database connection, the API server provides a consistent interface for reading and modifying Kubernetes objects. This also gives Kubernetes a central place for authentication, authorization, admission, validation, auditing, concurrency handling, and API semantics.

For the CKA, remember this simple mental model:

etcd stores the cluster state. The API server is the gateway to that state.

That single distinction prevents a lot of architectural conf

The Matchmaker: kube-scheduler Finds a Node

Our Pod now exists in the cluster, but it still has a problem: where should it run? This is the responsibility of the kube-scheduler. The scheduler watches the Kubernetes API for newly created Pods that don't yet have a node assigned.

Kubernetes is largely event-driven. The scheduler doesn't repeatedly ask the API server, "Do you have a Pod for me?" Instead, it maintains a watch and reacts when relevant objects change. When our nginx Pod appears without a node assignment, the scheduler sees it and begins the scheduling process.

The flow now looks roughly like:

Pod created
     |
     v
kube-apiserver
     |
     v
etcd
     |
     v
kube-scheduler
     |
     v
Find a suitable node

The scheduler's job is not to start the container. Its job is to make a scheduling decision.

Step 4: Filtering — Which Nodes Can Run the Pod?

Imagine our cluster contains three worker nodes:

worker-1
worker-2
worker-3

The scheduler can't simply choose the first node it sees. It evaluates the Pod's requirements against the available nodes. The first major phase is filtering, where Kubernetes eliminates nodes that cannot satisfy the Pod's requirements.

A node might be filtered out because it doesn't have enough CPU or memory, because the Pod's nodeSelector doesn't match, because node affinity rules don't match, because a taint isn't tolerated, or because topology and other scheduling constraints cannot be satisfied.

For example:

worker-1   Not enough memory       ❌
worker-2   Taint doesn't match     ❌
worker-3   Suitable                ✅

This gives us a useful CKA mental model:

Filtering = eliminate unsuitable nodes
Scoring = rank suitable nodes

You may also encounter older Kubernetes documentation that uses terms such as predicates and priorities. Modern Kubernetes scheduling uses the scheduling framework with filtering and scoring plugins, but understanding the older terminology is still useful when reading older tutorials or answering interview questions.

Step 5: Scoring — Which Suitable Node Is Better?

What happens if several nodes pass the filtering stage? Kubernetes still needs to choose the best candidate. That's where scoring comes in. The scheduler evaluates the remaining nodes using its scheduling plugins and gives them scores based on the scheduling rules and preferences.

You can influence scheduling with mechanisms such as nodeSelector, node affinity, taints and tolerations, topology constraints, and other scheduling configuration.

For example:

spec:
  nodeSelector:
    disktype: ssd

Now the scheduler knows that the Pod should only be considered for nodes carrying the disktype=ssd label.

You can inspect node labels with:

kubectl get nodes --show-labels

And when a Pod is stuck in Pending, one of your first troubleshooting commands should be:

kubectl describe pod nginx

Pay particular attention to the Events section. Kubernetes will often tell you exactly why a Pod couldn't be scheduled, such as insufficient CPU, an untolerated taint, an affinity mismatch, or another scheduling constraint.

The Scheduler Doesn't Start the Container

This is a point worth emphasizing because it is a common misunderstanding when people are new to Kubernetes. The scheduler doesn't SSH into a worker node, execute docker run, or directly invoke containerd. It makes a decision and records that decision through the Kubernetes API.

Conceptually, the scheduler says:

This Pod should run on worker-3.

It then updates the Pod's assignment through the API server:

kube-scheduler
      |
      | Bind Pod to worker-3
      v
kube-apiserver
      |
      v
Pod object updated

Now you can see the assigned node with:

kubectl get pod nginx -o wide

You might see:

NAME    READY   STATUS    RESTARTS   AGE   IP            NODE
nginx   1/1     Running   0          30s   10.244.1.10   worker-3

The scheduler has completed its part of the story. The responsibility now moves to the worker node.

Handing Off to the Node: The kubelet Takes Over

Every Kubernetes worker node runs a kubelet. If the scheduler is the component deciding where the Pod belongs, the kubelet is the component on the node that works to make that decision become reality.

The kubelet watches the API server for Pods assigned to its node. When it sees that nginx has been assigned to worker-3, it begins reconciling the desired state with the actual state of the node. The desired state says, "There should be an nginx Pod running here." The actual state says, "There isn't one yet." The kubelet's job is to close that gap.

This reconciliation model is one of the most important ideas in Kubernetes:

Desired state:
nginx Pod should be running

Actual state:
nginx Pod doesn't exist

        ↓

kubelet reconciles

        ↓

nginx Pod gets created

This is why Kubernetes isn't simply a collection of commands that run once. Components continuously watch state and take action when reality doesn't match the desired configuration.

The Engine Room: CRI and containerd

The kubelet doesn't normally implement all of the low-level container operations itself. Instead, it communicates with the container runtime through the Container Runtime Interface (CRI).

A common runtime you'll encounter is containerd.

The simplified relationship is:

kubelet
   |
   | CRI
   v
containerd
   |
   v
container runtime
   |
   v
containers

The kubelet asks the runtime to create the necessary Pod sandbox and containers. The runtime is then responsible for the lower-level container lifecycle operations. In a CKA environment, understanding this boundary is useful because it tells you where to look when Kubernetes knows about a Pod but the runtime isn't successfully creating the containers.

On a node where you have access, you can inspect the CRI runtime using commands such as:

crictl info

You can list running and stopped containers with:

crictl ps -a

And you can inspect Pod sandboxes with:

crictl pods

If something looks wrong at the node level, these commands can reveal information that isn't always obvious from kubectl get pods.

CNI Enters the Picture: Giving the Pod a Network

A container also needs networking, and this is where CNI — Container Network Interface comes into the picture. Kubernetes itself defines the networking expectations, but the actual networking implementation is provided by a CNI plugin. Depending on your cluster, that could be Cilium, Calico, or another networking implementation.

The simplified flow looks like this:

kubelet
   |
   v
container runtime
   |
   v
CNI plugin
   |
   +--> create/configure network namespace
   |
   +--> assign Pod IP
   |
   +--> create network interfaces
   |
   +--> connect Pod to node network

A common Linux networking pattern uses a virtual Ethernet pair:

Pod network namespace
        |
       veth
        |
       veth
        |
Node network namespace

The exact implementation depends on the CNI plugin, but the general idea remains the same: the Pod needs its network namespace configured, an IP address assigned, and connectivity established with the rest of the cluster.

This is why a Pod can successfully pass scheduling and still have networking problems. If the CNI layer is broken, the Pod might be assigned to a node but fail during network setup. For troubleshooting, remember that the Pod exists does not automatically mean the Pod has working networking.

The Engine Starts: Image Pulling, Storage, and Containers

Now the runtime needs the image requested by our Pod:

nginx:1.27

If the image isn't already available on the node, the runtime pulls it from the configured container registry.

The simplified process looks like:

containerd
    |
    v
Container Registry
    |
    v
nginx:1.27

If the image is successfully pulled, container creation can continue. If the image cannot be pulled because the name is wrong, the registry is unavailable, credentials are missing, or networking is broken, the Pod may enter states such as: ErrImagePull or: ImagePullBackOff

This is where kubectl describe becomes extremely useful:

kubectl describe pod nginx

Look at the Events section. You might see:

Failed to pull image nginx:1.27 or: Back-off pulling image nginx:1.27

At this point, you know that scheduling probably isn't your problem. The scheduler already selected the node. Your investigation has moved further down the chain toward the kubelet and container runtime.

Where Storage Fits: CSI

Now suppose the Pod also requests persistent storage. Kubernetes may need to work with CSI — Container Storage Interface components to make that storage available to the Pod.

The simplified relationship looks like:

kubelet
   |
   v
CSI components
   |
   v
Storage system

Depending on the storage configuration, Kubernetes may need to identify the volume, attach it to the node, mount it, and make the resulting filesystem available to the Pod.

This gives you another useful troubleshooting boundary. A Pod can be successfully scheduled to a node but still fail to start because its required volume cannot be attached or mounted. When that happens, you might investigate the Pod's events, PVC/PV status, StorageClass configuration, and CSI components.

Again, don't treat: Scheduled as equivalent to: Running

Scheduling only tells you that Kubernetes has selected a node. There is still plenty of work left before the application can actually start.

Init Containers Run Before the Application

If the Pod contains init containers, those containers must complete successfully before the main application containers are started.

For example:

spec:
  initContainers:
    - name: setup
      image: busybox
      command:
        - sh
        - -c
        - echo "Preparing..."

  containers:
    - name: nginx
      image: nginx:1.27

The simplified sequence becomes:

Pod sandbox
     |
     v
Init container
     |
     v
Init container completes
     |
     v
Main application container starts

If the init container fails repeatedly, the main application container doesn't move forward normally. This can make a Pod appear stuck even though the image and main application configuration look perfectly fine.

You can inspect the Pod:

kubectl describe pod nginx

And view logs from a specific init container:

kubectl logs nginx -c setup

When troubleshooting a Pod that isn't progressing, always check whether init containers are part of the Pod specification.

Finally: The Main Application Container Starts

After the image is available, networking has been configured, required storage has been prepared, and any init containers have completed, the container runtime can finally start the main application container.

At this point, the journey that started with one kubectl command has crossed almost the entire Kubernetes architecture:

kubectl
   |
   v
kube-apiserver
   |
   +--> Authentication
   |
   +--> Authorization / RBAC
   |
   +--> Admission
   |
   v
etcd
   |
   v
kube-scheduler
   |
   +--> Filter
   |
   +--> Score
   |
   +--> Bind
   |
   v
kube-apiserver
   |
   v
kubelet
   |
   v
CRI
   |
   v
containerd
   |
   +--> Pull image
   |
   +--> Configure networking through CNI
   |
   +--> Prepare storage through CSI
   |
   +--> Run init containers
   |
   v
Main application container

The application process is now running inside its container. Kubernetes may report the Pod as:

Running

But there's still another distinction worth understanding: a running container isn't necessarily a ready application.

Running Doesn't Always Mean Ready

Imagine your application starts its process successfully but needs another 20 seconds to initialize. The container itself is technically running, but the application isn't ready to accept traffic yet.

That's where a readiness probe becomes useful:

readinessProbe:
  httpGet:
    path: /
    port: 80

Kubernetes can use the result of that probe to determine whether the application should receive traffic through a Service.

A liveness probe answers a different question. It helps Kubernetes determine whether the application is still functioning and whether the container should be restarted.

So you should remember another important distinction:

Container Running
       ≠
Application Ready

This becomes especially important when you're debugging Deployments and Services. You can have Pods that are technically running but still have zero ready endpoints because their readiness checks are failing.

The Complete Journey on One Whiteboard

If I were explaining this during a CKA study session, this is the diagram I'd want on the board:

                    CONTROL PLANE
┌──────────────────────────────────────────────┐
│                                              │
│  kubectl                                     │
│     │                                        │
│     ▼                                        │
│  kube-apiserver                              │
│     │                                        │
│     ├── Authentication                       │
│     ├── Authorization / RBAC                 │
│     ├── Admission Controllers / Webhooks     │
│     │                                        │
│     ▼                                        │
│    etcd                                      │
│     ▲                                        │
│     │                                        │
│  kube-scheduler                              │
│     │                                        │
│     ├── Filter nodes                         │
│     ├── Score nodes                          │
│     └── Bind Pod                             │
│                                              │
└──────────────────────┬───────────────────────┘
                       │
                       │ API
                       ▼
                    WORKER NODE
┌──────────────────────────────────────────────┐
│                                              │
│  kubelet                                     │
│     │                                        │
│     ▼                                        │
│  CRI                                         │
│     │                                        │
│     ▼                                        │
│  containerd                                  │
│     │                                        │
│     ├── Pull image                            │
│     ├── Create Pod sandbox                   │
│     ├── Start init containers                │
│     └── Start application container          │
│                                              │
│     ├──────────► CNI                         │
│     │             └── Pod networking         │
│     │                                        │
│     └──────────► CSI                         │
│                   └── Storage                 │
│                                              │
└──────────────────────────────────────────────┘

The important thing to notice is that there isn't one Kubernetes component sitting in the middle doing everything. The API server handles the API interaction and persistence path, the scheduler makes the placement decision, and the kubelet takes responsibility for making the Pod actually exist on the selected node. The container runtime handles container lifecycle operations, while CNI and CSI handle networking and storage concerns.

Once you understand those boundaries, Kubernetes becomes much easier to reason about. Instead of asking, Why isn't my Pod working?, you can ask a much better question: Which stage of the Pod lifecycle has failed?

Mentor's Closing: Follow the Handoff

This is the mental model I want you to carry into the CKA exam and into production troubleshooting. When a Pod fails, don't immediately start throwing random kubectl commands at it. Walk through the same path that the Pod walked during creation and identify the first stage where reality diverged from the expected state.

If kubectl apply itself fails, investigate the API layer first: authentication, RBAC authorization, admission, validation, or resource-related restrictions. If the Pod object exists but remains Pending, investigate scheduling using kubectl describe pod and look at the Events section. If the Pod has a node but remains stuck in ContainerCreating, start thinking about the kubelet, CRI, container runtime, CNI networking, image pulling, or volume mounting.

If you see ImagePullBackOff, think about the image name, registry access, credentials, or network connectivity. If you see CrashLoopBackOff, the container is starting and then repeatedly exiting, so inspect the application logs and previous container logs:

kubectl logs <pod-name>
kubectl logs <pod-name> --previous

If networking isn't working, investigate the CNI, DNS, NetworkPolicies, Services, or the application itself. If storage isn't mounting, investigate the PVC, PV, StorageClass, CSI components, and volume events. And if the Pod is running but not receiving traffic, check readiness probes and the Service's endpoints.

The entire lifecycle can therefore be reduced to one mental model:

kubectl
  ↓
API Server
  ↓
Authentication
  ↓
Authorization / RBAC
  ↓
Admission
  ↓
etcd
  ↓
Scheduler
  ↓
Filter + Score
  ↓
Bind to Node
  ↓
kubelet
  ↓
CRI / containerd
  ↓
CNI + CSI
  ↓
Init Containers
  ↓
Application Container
  ↓
Readiness
  ↓
Running + Ready

Once this flow becomes second nature, Kubernetes troubleshooting stops feeling like a collection of unrelated commands. Every command has a purpose because you know which component you're investigating and what should have happened before you got there.

That's the real advantage of understanding Kubernetes internals. When a Pod gets stuck, you're no longer asking What command should I try next? You're asking the much more useful question:

Which handoff failed?

And that is the mental model that makes you considerably better at both the CKA exam and real-world Kubernetes troubleshooting.

Building Self-Healing Microservices with Kubernetes and Service Mesh

Kubernetes with Naveen — Fri, 31 Jul 2026 10:15:54 +0000

A container crashing is usually the easiest failure you'll deal with.

The harder ones are when every pod is healthy, every readiness probe is passing, CPU usage looks normal, and users are still getting timeouts because an upstream dependency has quietly fallen apart. Kubernetes sees healthy processes. Your customers see a broken application.

That's the gap many teams discover after their first serious production incident.

Kubernetes is exceptionally good at keeping containers alive. It was never designed to understand application behavior, dependency health, or whether sending another request to a struggling service is making the situation worse. That's where a service mesh enters the picture—not as another shiny platform component, but as the second layer of resilience that Kubernetes intentionally leaves unsolved.

The most reliable production systems don't rely on a single "self-healing" mechanism. They combine infrastructure recovery with traffic-aware recovery, while making sure neither layer accidentally amplifies an outage.

The Myth of the "Self-Healing" Container

Self-healing has become one of Kubernetes' most frequently repeated selling points, but it often creates the wrong expectation. Kubernetes absolutely knows how to recover from infrastructure failures. If a process crashes, it restarts the container. If a node disappears, it reschedules Pods elsewhere. If a replica dies, another one replaces it. Those are infrastructure failures, and Kubernetes handles them exceptionally well. The problem is that most production incidents don't begin with crashed containers. They begin with applications that are technically alive but no longer capable of serving users correctly. A service waiting eight seconds for a database response still passes a liveness probe. A JVM stuck behind exhausted thread pools is still running. A Go application returning request timeouts because an upstream dependency is overloaded is still a healthy Linux process. Kubernetes only sees containers and processes, while users experience application behavior. That distinction is where many so-called self-healing architectures fall apart.

Consider a payment service that depends on an external fraud detection API. The API becomes slow, but it never actually stops responding. Every incoming request now waits several seconds before timing out. Connection pools begin filling, request queues grow longer, CPU usage rises because more goroutines or threads remain active, and latency spreads into completely unrelated services. From Kubernetes' perspective, nothing appears broken because every Pod is still running and every readiness endpoint continues returning HTTP 200. Restarting those Pods simply replaces healthy processes with freshly started healthy processes while the real bottleneck continues to exist. Recovering from logical failures requires understanding traffic patterns, dependency health, and request behavior—not simply replacing containers.

Layer 1: Infrastructure Resilience (Kubernetes Native)

Kubernetes should be responsible for maintaining healthy infrastructure, not making application-level traffic decisions. Its built-in primitives have proven themselves across thousands of production clusters, provided they are used for the problems they were actually designed to solve. Liveness Probes answer one question: should this process continue running? If an application has deadlocked, exhausted memory, or entered an unrecoverable state, restarting it is entirely appropriate. Problems begin when teams connect liveness probes to remote dependencies. If every probe executes a database query or calls another microservice, a temporary slowdown in that dependency suddenly appears as application failure. Kubernetes faithfully restarts perfectly healthy Pods, increasing cold starts, opening new database connections, and placing even more pressure on the already struggling dependency.

Readiness Probes serve a completely different purpose. Rather than determining whether a process should exist, readiness determines whether the application should receive production traffic. During cache warm-up, background initialization, rolling deployments, or temporary dependency failures, failing readiness allows Kubernetes to stop routing new requests without destroying the Pod. Startup Probes provide another layer of protection for applications with slow initialization, particularly JVM workloads, AI inference services, or applications loading large datasets into memory. Without them, Kubernetes may repeatedly restart applications that simply haven't finished booting yet. Equally important is graceful termination. Production workloads should stop accepting new traffic, complete in-flight requests, flush telemetry, close open connections, and exit cleanly before the termination grace period expires. Otherwise, every deployment introduces avoidable request failures.

Infrastructure resilience also depends on protecting availability during normal cluster operations. Pod Disruption Budgets (PDBs) ensure maintenance events, node upgrades, or cluster autoscaling never remove too many replicas simultaneously. Meanwhile, the Horizontal Pod Autoscaler (HPA) provides elasticity by increasing capacity when workloads experience sustained demand. That elasticity has limits, however. HPA cannot compensate for an overloaded database, a saturated message broker, or an external API with strict rate limits. Adding more frontend replicas against an already constrained backend frequently accelerates resource exhaustion rather than solving it. Kubernetes can create more capacity, but it cannot create capacity where none actually exists.

Layer 2: Traffic & Application Resilience (Service Mesh)

Once Kubernetes has established stable infrastructure, the next challenge becomes communication between services. This is where a Service Mesh such as Istio or Linkerd provides capabilities Kubernetes intentionally leaves out. Instead of requiring every engineering team to implement retries, circuit breakers, connection pools, and traffic management differently inside application code, the mesh standardizes those behaviors within sidecar proxies or equivalent data-plane components. Developers continue writing business logic while platform teams define consistent resilience policies across the entire service estate.

One of the most valuable capabilities is Circuit Breaking. When an upstream dependency begins failing consistently, continuing to send requests simply wastes resources and increases queue lengths on an already unhealthy service. Circuit breakers recognize repeated failures and temporarily stop forwarding requests, allowing downstream systems time to recover instead of drowning them under additional traffic. Outlier Detection extends this concept further by recognizing that failures are rarely uniform across every replica. One Pod may suffer from memory pressure or a noisy neighbor while the remaining instances continue operating normally. Rather than removing the entire service from rotation, the mesh temporarily ejects only the unhealthy endpoint and shifts traffic toward healthier replicas until Kubernetes replaces the failing instance.

Equally important is disciplined retry behavior. Retries absolutely improve resilience when failures are genuinely transient, such as brief packet loss or short-lived network interruptions. Problems arise when retries become unlimited or synchronized. Production-grade Service Meshes should combine retry budgets, exponential backoff, and random jitter so retry traffic spreads naturally over time rather than arriving simultaneously. Finally, rate limiting provides an essential safety mechanism during overload conditions. Instead of allowing one malfunctioning service or noisy client to consume every available connection and thread, rate limiting rejects excess traffic early, preserving capacity for higher-priority workloads and preventing failures from propagating deeper into the platform.

The Dark Side: How Automated Healing Breaks Production

Automation is often blamed for outages not because it malfunctioned, but because it behaved exactly as configured. Many large-scale incidents begin with a relatively small problem that recovery mechanisms unintentionally amplify into something far worse. Retry storms are one of the classic examples. A downstream service experiences a brief slowdown, clients immediately retry every failed request, those retries generate additional retries higher in the call chain, and request volume grows exponentially. Before long, the dependency spends more CPU processing duplicated requests than genuine customer traffic. The original incident may have been recoverable, but the retry policy transformed it into a full-scale outage. Sensible retry budgets, bounded retry counts, exponential backoff, and randomized jitter exist specifically to prevent this feedback loop.

Another common failure mode involves overly aggressive liveness probes. Under sustained CPU saturation, applications naturally respond more slowly. Poorly configured probes interpret that latency as application failure and restart Pods that were actually making forward progress. Those restarts erase caches, create additional cold starts, increase CPU consumption, and reduce the amount of available serving capacity precisely when demand is highest. The platform begins attacking its own recovery efforts. Similar instability appears with flapping services that repeatedly transition between healthy and unhealthy states. Continuous readiness changes force load balancers, autoscalers, monitoring systems, and deployment controllers into constant adjustment, generating noisy alerts while masking the true root cause. The situation becomes even more dangerous when Kubernetes, the Service Mesh, client libraries, autoscalers, and cloud load balancers all attempt recovery simultaneously. Each layer behaves correctly in isolation, yet together they multiply request volume, infrastructure churn, and operational complexity. Effective resilience is less about adding more automation and more about ensuring each recovery mechanism understands its boundaries.

The Engineering Verdict

The strongest production platforms are built on the understanding that Kubernetes and a Service Mesh solve fundamentally different problems. Kubernetes provides infrastructure resilience through health probes, graceful termination, Pod Disruption Budgets, and intelligent autoscaling, ensuring workloads remain available despite hardware failures or process crashes. A Service Mesh operates at an entirely different layer, protecting communication through circuit breaking, outlier detection, carefully controlled retries, exponential backoff, jitter, and rate limiting. Neither technology replaces the other because each observes a different part of the system. Together they create a layered recovery model capable of handling both infrastructure failures and application-level degradation.

The final lesson, however, has little to do with technology and everything to do with restraint. Mature platforms recover gracefully because they avoid making bad situations worse. Conservative probe thresholds prevent unnecessary restarts, bounded retry budgets stop transient glitches from becoming retry storms, sensible circuit breaker settings isolate failures before they spread, and coordinated autoscaling avoids overwhelming already constrained dependencies. The objective is not to build a platform that reacts to every symptom immediately. It is to build one that understands when intervention helps and when patience is the better engineering decision. The difference between a resilient platform and a fragile one is rarely the number of recovery mechanisms it contains; it is whether those mechanisms work together instead of competing during the moments that matter most.

Stop the Context-Switching: Correlating Logs, Metrics, and Traces in a Single View

Kubernetes with Naveen — Fri, 10 Jul 2026 09:48:02 +0000

It usually starts the same way. A PagerDuty alert goes off, Slack lights up with messages, and someone posts a screenshot showing HTTP 500 errors climbing while customers begin reporting failed requests. Within minutes, everyone is staring at different dashboards. One engineer is looking at Prometheus graphs, another is digging through logs, someone else is opening distributed traces, and a fourth person is convinced Kubernetes is the problem because a few pods restarted twenty minutes ago. Nobody is wrong, but nobody has the complete picture either.

This is the hidden cost of modern observability. Most engineering teams
have invested heavily in collecting telemetry, yet incident response
still feels like detective work. The problem isn't a lack of data. Quite the opposite. We have more metrics, logs, and traces than ever before, but they often live in separate places, forcing engineers to mentally connect information that should already be linked. Every context switch steals a few more seconds, and during an outage those seconds quickly become minutes. Improving Mean Time to Resolution (MTTR) is rarely about collecting more telemetry; it's about making the telemetry work together so engineers spend less time searching and more time solving.

The Cost of the Three Panes of Glass

For years, observability platforms evolved as separate ecosystems.
Metrics were collected by Prometheus, logs flowed into systems like Loki or Elasticsearch, and distributed traces landed in Tempo, Jaeger, or Zipkin. Each tool solved a different problem extremely well, but incident response became an exercise in jumping between multiple browser tabs.

Imagine investigating an outage where your payment service suddenly
begins returning HTTP 500 responses. Your first stop is usually a
Prometheus dashboard. The graphs immediately tell you something is
wrong: request latency has increased dramatically, error rates are
climbing, and request throughput is beginning to fall. Metrics are
excellent at telling you what is happening across the system, but
they deliberately sacrifice detail for aggregation. They cannot explain
why one specific request failed or which dependency introduced the
delay.

Naturally, the next step is opening your logging platform. Now you're
searching around the same timestamp, filtering by Kubernetes namespace,
pod name, container name, or service labels. Hundreds or even thousands
of log entries appear, many of which have nothing to do with the
incident. You narrow the time window, adjust the filters, search for
"ERROR," and hope the application logged something useful. Sometimes it
did. Sometimes it didn't. Even when you find an interesting error
message, you're still guessing whether it belongs to the requests
responsible for the spike you saw on the metrics dashboard.

Distributed tracing becomes the third stop. Now you're trying to locate
the trace that corresponds to the failed request you observed earlier.
If your services process thousands of requests per second, finding the
right trace using only timestamps can feel like searching for a specific grain of sand on a beach. Every manual lookup increases cognitive load because your brain is acting as the integration layer between three independent systems.

This is what many engineers jokingly call the "three panes of glass" problem. The telemetry exists. The answers exist. What doesn't exist is a natural path between them. During a production incident, engineers shouldn't have to remember timestamps, copy request IDs between tools, or manually correlate unrelated datasets. That work should already be done before the alert even arrives.

The Glue: How OpenTelemetry and Prometheus Actually Correlate

The good news is that modern observability doesn't require replacing
your entire stack. Instead, it relies on common standards that allow
metrics, traces, and logs to reference one another automatically.
OpenTelemetry has become the foundation for this approach by providing a consistent way to generate, enrich, and transport telemetry across
distributed systems.

At the center of this architecture sits the OpenTelemetry Collector. Rather than every application exporting telemetry directly to multiple destinations, applications send metrics, traces, and logs to the Collector. From there, the Collector can enrich telemetry with
Kubernetes metadata, normalize attributes using OpenTelemetry semantic
conventions, batch data efficiently, sample traces when necessary, and
forward telemetry to systems such as Prometheus-compatible backends,
Grafana Loki, Tempo, Jaeger, or other storage platforms.

The first piece of correlation comes from W3C Trace Context.

traceparent:
00-4bf92f3577b34da6a3ce929d0e0e4736

Applications inject the active trace_id into every log entry.

{
  "service.name":"payment-service",
  "level":"ERROR",
  "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
  "message":"Database connection timeout"
}

Prometheus Exemplars bridge aggregated metrics to individual traces.

http_server_request_duration_seconds

Value: 5.8 seconds

Exemplar:
trace_id=4bf92f3577b34da6a3ce929d0e0e4736

Anatomy of a Failure: Following the Telemetry Instead of Guessing

It's a normal weekday afternoon when your payment service begins
failing. Within a minute, Prometheus fires an alert because HTTP 500
responses have increased from less than one percent to nearly thirty
percent. Customer support reports failed checkouts, and engineers start
assembling in the incident channel.

The metrics dashboard confirms the impact. Request latency has increased from around 200 milliseconds to almost six seconds, throughput is declining, and the error rate continues climbing. Instead of searching through logs, an engineer clicks a Prometheus Exemplar attached to the latency graph.

The distributed trace shows the request flowing through the API Gateway, the payment service, inventory validation, and finally spending nearly six seconds waiting on PostgreSQL.

Gateway
   │
   ▼
Payment Service
   │
   ▼
Inventory Service
   │
   ▼
PostgreSQL

Database Span:
Duration: 5.8 seconds
Status: ERROR
Exception: Connection Pool Timeout

Because every application log contains the same trace_id, the
observability platform automatically filters the logs associated with
that request.

ERROR

Unable to acquire database connection

Pool Size: 100
Active Connections: 100
Waiting Requests: 243
Connection Timeout: 30 seconds

A few lines later:

WARN

Long-running transaction detected

Transaction Duration: 94 seconds

The root cause is immediately clear. A long-running transaction
exhausted the PostgreSQL connection pool, causing new requests to wait
until they timed out. Metrics revealed what changed, the trace
showed where the delay occurred, and the logs explained why it
happened---all without switching tools.

Wrap-up

Engineering teams have largely solved the problem of collecting
telemetry. The harder problem is making those signals tell a single,
coherent story during an incident.

Technologies like W3C Trace Context, Prometheus Exemplars,
trace_id log injection, OpenTelemetry semantic conventions,
and the OpenTelemetry Collector transform isolated telemetry into
connected evidence that follows every request across your system.

When the next 3 AM alert arrives, you shouldn't have to play detective
across half a dozen browser tabs. You should be able to click a graph,
open the trace responsible for the anomaly, inspect the logs attached to that request, identify the root cause, and get back to sleep.

Observing AI Microservices with OpenTelemetry & Grafana

Kubernetes with Naveen — Sun, 05 Jul 2026 12:12:30 +0000

Modern AI workloads introduce new observability
challenges. Learn how OpenTelemetry and Grafana help uncover latency,
failures, and security risks.

The 2 AM Page

It's 2:07 AM and your phone starts vibrating. Checkout latency has
suddenly jumped from a few hundred milliseconds to well over ten
seconds. The Kubernetes cluster looks perfectly healthy, CPU usage is
normal, memory pressure is low, and none of the infrastructure
dashboards indicate anything unusual. Yet customers are abandoning their carts because requests are hanging somewhere inside the platform.

After nearly an hour of investigation, the real culprit finally emerges. An AI recommendation service is waiting on an external LLM provider that has begun throttling requests. Since several downstream services depend on those recommendations, request queues begin growing across the platform. Retry mechanisms unintentionally multiply traffic, circuit breakers start opening, and what was originally considered a nice-to-have AI feature suddenly becomes responsible for bringing an entire production workload to its knees.

Without distributed tracing, every service simply reports another
timeout. Every dashboard shows elevated latency. Every team assumes the
problem belongs to someone else. At that point, debugging turns into
educated guessing rather than engineering.

That is exactly why observability has moved from being an operational
luxury to a production requirement.

The Architecture Shift

Microservices were already challenging to observe long before AI entered the picture. Instead of debugging a single monolithic application, engineers now spend their time following requests across dozens or even hundreds of independently deployed services communicating over HTTP, gRPC, Kafka, RabbitMQ, and various asynchronous messaging systems. Every additional network hop introduces another possible failure point, making end-to-end visibility increasingly valuable.

AI fundamentally changes this architecture again because request
execution is no longer completely deterministic. A traditional request
usually follows a predictable path through authentication, business
logic, and persistence before returning a response. AI agents rarely
behave that way. A single customer request might invoke multiple LLMs,
perform several vector database searches, retrieve historical memory,
call external APIs, execute internal tools, and stream results back to
the client. The exact execution path often depends on decisions made
dynamically by the agent itself.

This introduces an entirely new class of operational problems. Response
times become inconsistent because LLM inference is naturally variable.
Tool execution paths differ between requests. Prompt sizes fluctuate
dramatically, directly affecting latency and infrastructure costs.
Streaming responses remain active for much longer than conventional REST APIs, while external model providers can introduce unpredictable delays that are completely outside your infrastructure boundary. Understanding these systems requires visibility into every component participating in the request lifecycle rather than simply monitoring the health of Kubernetes pods or virtual machines.

The Deep Dive (OTel + Grafana)

This is where OpenTelemetry begins to earn its place in modern
production environments. Rather than forcing engineering teams to adopt
proprietary instrumentation libraries for every observability vendor,
OpenTelemetry provides a common standard for collecting metrics, logs,
and distributed traces. Applications generate telemetry once, the
OpenTelemetry Collector processes and enriches that data, and
organizations remain free to export it to whichever backend best fits
their platform strategy. That separation between instrumentation and
storage has become one of OpenTelemetry's greatest strengths.

The initial setup is rarely effortless. Configuring Collectors, deciding on sampling strategies, keeping SDK versions aligned, and learning evolving semantic conventions can be frustrating, particularly for teams introducing observability into an existing platform. Nevertheless, the investment quickly pays for itself. Once the telemetry pipeline exists, adding meaningful observability to newly deployed services becomes significantly easier, and engineering teams spend far less time modifying application code whenever their monitoring stack evolves.

Distributed tracing is where this ecosystem delivers the greatest value. Instead of investigating isolated log entries from individual services, engineers can follow a single request as it moves through API gateways, authentication services, business logic, vector databases, caching layers, AI agents, and external LLM providers. When a request suddenly takes eight seconds to complete, the trace immediately reveals whether the delay occurred inside your application, during a vector similarity search, while waiting for an embedding model, or because an external AI provider responded slowly. The investigation shifts from speculation to evidence within seconds.

Modern AI workloads also require telemetry that extends beyond
conventional infrastructure metrics. Engineering teams increasingly need visibility into model selection, prompt size, token consumption, tool execution, streaming duration, provider latency, completion status, and response quality. This is exactly why OpenTelemetry has been expanding its semantic conventions for generative AI workloads. Rather than treating an LLM request as just another HTTP call, telemetry can describe the interaction in terms that are meaningful to engineers operating AI systems.

The Grafana LGTM stack complements OpenTelemetry by giving engineers a
unified environment for exploring this telemetry. Loki centralizes logs, Tempo stores distributed traces, Mimir provides highly scalable
Prometheus-compatible metrics, and Grafana brings everything together
through dashboards, alerting, and cross-signal correlation.

Top 3 Key Takeaways

The first priority should be complete instrumentation across every
dependency involved in a request. Observing only internally developed
microservices leaves critical blind spots because modern production
systems increasingly depend on external LLM providers, embedding
services, vector databases, caches, and messaging platforms.

Secondly, adopt OpenTelemetry semantic conventions as early as possible
instead of creating custom telemetry formats for every team. Consistent
naming, attributes, and resource metadata make dashboards reusable
across hundreds of services and significantly reduce operational
complexity.

Finally, treat observability as part of the application architecture
rather than something added after deployment. Teams that design
telemetry alongside APIs, infrastructure, and deployment pipelines
consistently resolve production incidents faster because the necessary
diagnostic information already exists before failures occur.

A Pragmatic Conclusion

Most engineering organizations don't need another dashboard. They need
confidence that when production behaves unexpectedly, they can explain
exactly why.

Start with one critical production workload. Instrument it with
OpenTelemetry, deploy an OpenTelemetry Collector, connect it to the
Grafana LGTM stack, and ensure a single request can be followed from
your API gateway through internal microservices, vector databases, AI
agents, and external LLM providers. Once that visibility exists, every
additional service becomes easier to understand, troubleshoot, and
secure.

The Microservice-to-Engineer Ratio (MTR): Why Too Many Microservices Slow Down Engineering Teams

Kubernetes with Naveen — Wed, 03 Jun 2026 14:38:31 +0000

Discover the Microservice-to-Engineer Ratio (MTR), a powerful architectural metric that reveals when microservices begin hurting engineering productivity. Learn the ideal MTR range, warning signs of service sprawl, and practical strategies to reduce operational complexity.

Top 3 Key Takeaways

A growing number of microservices does not necessarily indicate architectural maturity; in many cases, it signals increasing operational complexity.
The biggest cost of a high MTR is not infrastructure spending but the cognitive load imposed on engineers.
High-performing engineering organizations focus on ownership, simplicity, governance, and platform engineering to maintain a healthy MTR.

The Problem Nobody Talks About

Imagine a team of five engineers responsible for maintaining forty microservices.

On paper, the architecture looks modern. The organization proudly claims to have embraced cloud-native development. The system is containerized, deployed on Kubernetes, monitored through a sophisticated observability stack, and supported by automated CI/CD pipelines.

Yet the day-to-day reality tells a very different story.

Engineers spend their mornings investigating failed deployment pipelines. Afternoons disappear into debugging service-to-service communication failures. Sprint planning meetings are filled with discussions about infrastructure upgrades rather than customer-facing improvements. Production incidents frequently originate from unexpected interactions between services that were supposed to be independent.

Weeks pass without meaningful product innovation because the engineering team is trapped in an endless cycle of maintaining the machinery required to keep the architecture running.

Many organizations find themselves in exactly this situation. They adopted microservices hoping to achieve greater agility, independent deployments, and faster innovation. Instead, they discovered that microservices can create an entirely new category of complexity that gradually consumes engineering capacity.

The uncomfortable truth is that many teams spend years optimizing the architecture while slowly losing the ability to efficiently build products.

This is where a surprisingly simple metric becomes incredibly valuable: the Microservice-to-Engineer Ratio, commonly referred to as MTR.

Although rarely discussed in architecture conferences or engineering leadership meetings, MTR often reveals more about the long-term health of an engineering organization than many traditional productivity metrics.

What Is the Microservice-to-Engineer Ratio?

The Microservice-to-Engineer Ratio measures the relationship between the number of microservices an organization operates and the number of engineers responsible for building, maintaining, and supporting them.

The formula is straightforward:

MTR = Number of Microservices ÷ Number of Engineers

If an organization operates 50 microservices and employs 25 engineers, its MTR is 2.0.

At first glance, this may appear overly simplistic. Experienced engineers are often skeptical of metrics that attempt to reduce complex systems into a single number. However, the power of MTR lies not in mathematical precision but in its ability to expose organizational patterns that are otherwise difficult to see.

Every microservice introduces an operational responsibility. It requires source code management, deployment automation, observability, monitoring, security controls, documentation, runtime upgrades, dependency maintenance, and long-term ownership. While each individual service may appear manageable, the cumulative effect of dozens or hundreds of services can become overwhelming.

As the number of services increases, engineers are required to understand more deployment pipelines, more APIs, more infrastructure components, and more failure modes. Eventually, the operational burden begins to compete with product development for engineering attention.

MTR helps organizations identify when that balance starts shifting in the wrong direction.

Why MTR Matters More Than Most Engineering Metrics

Modern engineering organizations track countless measurements. Leadership teams monitor deployment frequency, lead time, incident counts, uptime percentages, cloud spending, and DORA metrics. These measurements are valuable, but they often describe symptoms rather than underlying causes.

MTR provides insight into structural complexity.

Think about the lifecycle of a single microservice. It starts as a seemingly harmless architectural decision. A team extracts a small component from a larger system to improve modularity. Initially, the benefits are clear. The service can be deployed independently and maintained by a dedicated team.

However, the service also requires its own repository, build process, deployment configuration, monitoring dashboards, alerting rules, security policies, documentation, and operational support model. These responsibilities persist indefinitely.

When an organization repeats this process dozens of times, complexity accumulates silently. Each service adds another moving piece to the ecosystem. Engineers eventually find themselves spending more time managing interactions between services than building the functionality those services were intended to deliver.

This is why MTR matters. It highlights whether the architectural complexity introduced by microservices remains sustainable relative to the engineering capacity available to manage it.

Understanding the Golden Ratio of MTR

There is no universally accepted perfect MTR. Every organization operates under different constraints, team structures, and business requirements.

However, after years of observing enterprise systems across industries, certain patterns consistently emerge. These patterns allow us to define three broad MTR zones that help explain the relationship between service count and organizational effectiveness.

MTR Below 0.5: The Healthy Service Era

An MTR below 0.5 generally indicates that engineers are responsible for relatively few services. For example, a team of twenty engineers managing eight microservices would have an MTR of 0.4.

Many engineers assume this represents an immature architecture. In reality, some of the most effective engineering organizations intentionally operate within this range.

The reason is simple: simplicity scales remarkably well.

When engineers are responsible for fewer services, they can maintain a clearer mental model of the overall system. Understanding how requests flow through the platform becomes easier. Debugging incidents requires less detective work. Onboarding new team members becomes faster because there are fewer moving parts to learn.

Perhaps most importantly, engineering effort remains focused on solving business problems rather than managing infrastructure complexity.

Organizations in this range often benefit from strong modular boundaries without excessive operational fragmentation. Teams can evolve systems confidently because they understand how components interact. Architectural discussions tend to focus on customer outcomes rather than service orchestration.

That said, an extremely low MTR is not automatically ideal. Large monolithic systems can eventually become difficult to scale, deploy, and maintain. If service boundaries are ignored entirely, organizations may encounter a different set of challenges involving release coordination, ownership ambiguity, and scalability constraints.

The goal is not to minimize service count at all costs. The goal is to achieve the lowest level of complexity necessary to support business objectives.

MTR Between 0.5 and 1.5: The Sweet Spot

This range is where many mature engineering organizations operate most effectively.

Consider a company with thirty engineers maintaining twenty-eight microservices. Its MTR would be approximately 0.93, placing it comfortably within the sweet spot.

At this stage, services are often aligned with meaningful business domains rather than arbitrary technical boundaries. Teams enjoy the benefits of independent deployment and ownership without becoming overwhelmed by operational overhead.

One of the defining characteristics of healthy organizations in this range is that teams own domains rather than individual services.

This distinction may appear subtle, but it fundamentally changes how architecture evolves. When engineers think in terms of domains such as payments, customer identity, inventory, or order management, architectural decisions become guided by business needs. Services become implementation details rather than organizational units.

Another characteristic of organizations in this range is strong platform support. Engineers are not expected to become experts in every infrastructure technology. Internal platforms provide standardized deployment pipelines, observability tooling, security controls, and operational workflows. This dramatically reduces the cost of maintaining multiple services.

Perhaps most importantly, organizations in the sweet spot treat the creation of new services as a deliberate architectural decision rather than a default response to every design challenge.

Before introducing a new service, mature teams ask difficult questions. Does the proposed service represent a true bounded context? Does it simplify ownership? Does it provide meaningful deployment independence? Does it solve a real business problem?

These questions help prevent unnecessary service proliferation.

MTR Above 2.0: The Danger Zone

Once MTR exceeds 2.0, warning signs typically begin appearing across the organization.

Imagine a company with fifteen engineers responsible for forty-five microservices. The architecture may look impressive from a distance, but engineers inside the organization often experience a very different reality.

Small feature requests suddenly require modifications across multiple repositories. Deployment pipelines multiply. Runtime dependencies become increasingly difficult to manage. Engineers spend significant amounts of time coordinating changes between teams.

Over time, the architecture begins consuming more energy than the product itself.

One of the first symptoms is reduced development velocity. A change that previously required modifications to a single codebase now involves multiple services, API contracts, deployment pipelines, and validation processes. Delivery slows not because engineers are less capable but because the system itself has become more difficult to navigate.

Onboarding new engineers becomes increasingly challenging. Understanding the platform requires learning dozens of services, countless integration points, and years of accumulated tribal knowledge. Engineers often spend months developing enough context to contribute effectively.

Observability presents another challenge. More services generate more logs, traces, dashboards, and alerts. While visibility theoretically improves, the volume of telemetry frequently overwhelms teams. Important signals become buried beneath operational noise.

Eventually, ownership begins to erode. Everyone owns pieces of the system, but nobody fully understands the whole. This is often when serious reliability issues emerge.

Why the MTR Explodes

Organizations rarely wake up one morning and intentionally decide to create an unsustainable architecture.

Instead, MTR tends to grow gradually through a series of individually reasonable decisions.

One common cause is what many architects jokingly refer to as "resume-driven architecture." Engineers sometimes pursue architectural patterns because they are fashionable rather than necessary. Microservices, event-driven systems, and distributed architectures can appear sophisticated, but sophistication is not the same as effectiveness.

Another major contributor is the tendency to imitate large technology companies without understanding their context.

Organizations frequently study the engineering practices of industry giants and attempt to replicate them. What they overlook is that companies operating at global scale face challenges fundamentally different from those encountered by smaller teams. Architectural decisions that make sense for thousands of engineers may be entirely inappropriate for dozens.

Premature domain decomposition also plays a significant role. Teams often attempt to define perfect service boundaries before they fully understand the business domain. As a result, services become fragmented around assumptions rather than actual organizational needs.

Fear of monoliths contributes as well. Over the past decade, the software industry has developed an almost reflexive aversion to monolithic architectures. While poorly designed monoliths certainly create problems, well-structured modular monoliths remain highly effective solutions for many organizations.

Finally, weak architectural governance allows service creation to proceed unchecked. Without clear standards and review processes, every team develops its own interpretation of microservices. The result is an ecosystem of inconsistent patterns, technologies, and operational models that become increasingly difficult to manage.

The Real Cost of a High MTR

The most damaging consequences of a high MTR rarely appear on financial reports.

Instead, they manifest through human limitations.

Engineering organizations often focus heavily on infrastructure costs, but infrastructure is rarely the primary problem. The true expense of excessive service fragmentation is cognitive load.

Every engineer has a limited capacity to understand complexity. As the number of services grows, engineers must track more dependencies, more deployment workflows, more runtime behaviors, and more potential failure scenarios. Eventually, the system exceeds what individuals can reasonably comprehend.

When cognitive load becomes excessive, decision quality deteriorates. Engineers become hesitant to make changes because they fear unintended consequences. Innovation slows because understanding the system requires enormous effort. Incidents take longer to resolve because diagnosing failures involves navigating an increasingly complex web of interactions.

Infrastructure overhead compounds the problem. Each service requires compute resources, deployment pipelines, monitoring systems, networking configurations, and security controls. Cloud spending rises, but more importantly, operational workload increases.

The organization eventually reaches a point where maintaining the architecture consumes a significant portion of engineering capacity.

The Distributed Monolith: The Worst of Both Worlds

Perhaps the most dangerous outcome of an unhealthy MTR is the emergence of a distributed monolith.

A distributed monolith is a system that looks like a microservices architecture but behaves like a tightly coupled monolith.

Services depend heavily on one another. Deployments require coordination. Failures cascade across boundaries. Independent releases become nearly impossible.

In this scenario, organizations inherit all the complexity associated with distributed systems without receiving the benefits that microservices are supposed to provide.

Network latency becomes a concern. Observability becomes more difficult. Failure modes multiply. Yet teams still lack true independence.

This architectural state is surprisingly common and extraordinarily expensive.

Many organizations spend years attempting to optimize distributed monoliths when the real solution is architectural simplification.

How to Fix a Broken MTR

Recovering from an unhealthy MTR requires discipline, not heroics.

The first step is right-sizing the architecture. Mature engineering organizations periodically evaluate whether existing services still justify their existence. Services that provide little architectural value and create significant operational burden should be consolidated when appropriate.

Contrary to popular belief, merging services is often a sign of architectural maturity rather than failure. Experienced engineers understand that simplicity frequently produces better outcomes than excessive decomposition.

The second step involves investing in platform engineering. A strong platform team reduces the operational burden placed on product engineers by providing standardized deployment mechanisms, observability tooling, security controls, and self-service infrastructure capabilities. This allows teams to focus on business functionality rather than infrastructure management.

Governance is equally important. Organizations need clear criteria for creating new services. Architectural reviews should evaluate not only technical feasibility but also long-term operational impact. Every new service should have a compelling justification supported by measurable benefits.

Finally, engineering leaders must actively manage cognitive load. Architecture exists to help humans solve problems. When a system becomes too difficult for engineers to understand, no amount of technological sophistication can compensate for the resulting productivity loss.

The Most Important Lesson About MTR

The most mature engineers eventually discover a simple truth.

Microservices are not the goal.

The goal is delivering value to customers efficiently, reliably, and sustainably.

Microservices are merely one possible tool for achieving that outcome.

When architectural decisions become disconnected from business objectives, organizations risk optimizing for complexity rather than effectiveness. Teams become trapped maintaining elaborate systems that provide little competitive advantage.

The best architectures are rarely the most complicated. More often, they are the ones that remain understandable as organizations grow.

A healthy MTR helps preserve that understanding.

Conclusion

The Microservice-to-Engineer Ratio is not a perfect metric, nor should it be treated as a rigid rule. However, it provides a valuable lens through which engineering leaders can evaluate architectural sustainability.

When MTR remains within a healthy range, engineers spend more time solving customer problems and less time wrestling with operational complexity. Ownership remains clear, onboarding stays manageable, and teams retain the ability to move quickly.

When MTR grows unchecked, complexity accumulates faster than organizations can manage it. Cognitive load increases, delivery slows, operational overhead expands, and distributed monoliths emerge.

The organizations that thrive over the long term are not necessarily the ones operating the most microservices. They are the ones that maintain the right balance between architectural flexibility and human understanding.

In the end, architecture should serve engineers, not the other way around.

Because while infrastructure can scale almost infinitely, human attention cannot. And every successful architecture is ultimately built upon the limited but incredibly valuable cognitive capacity of the engineers who maintain it.

OpenTelemetry: The Foundation of Modern Cloud-Native Observability — Traces, Metrics, Logs, and the Future of Observability

Kubernetes with Naveen — Thu, 28 May 2026 12:19:54 +0000

Discover how OpenTelemetry became the industry standard for cloud-native observability. Learn how it collects, processes, and exports traces, metrics, and logs across distributed systems, why organizations are adopting it at scale, and how it serves as foundational infrastructure for modern platform engineering teams.

OpenTelemetry: The Foundation of Modern Cloud-Native Observability

Modern software systems have become increasingly distributed, dynamic, and complex. Applications are no longer monolithic programs running on a single server. Instead, they span containers, Kubernetes clusters, serverless functions, APIs, service meshes, databases, message queues, and third-party services spread across multiple cloud environments.

While this architectural evolution has enabled organizations to build highly scalable and resilient systems, it has also introduced a significant challenge: understanding what is actually happening inside these systems when things go wrong.

A customer-facing API slowdown may originate from a database query. A payment failure might be caused by a downstream dependency. A latency spike could be the result of resource contention in a Kubernetes cluster. In modern distributed environments, identifying root causes quickly requires comprehensive visibility across every layer of the stack. This is where observability becomes essential.

Over the last few years, one technology has emerged as the de facto standard for collecting observability data across cloud-native environments: OpenTelemetry.

What started as an open-source initiative to standardize telemetry collection has evolved into one of the most widely adopted pieces of infrastructure in modern software engineering. Today, OpenTelemetry serves as the backbone of observability strategies for startups, enterprises, hyperscalers, and platform engineering teams worldwide.

Why Observability Needed a Standard

Before OpenTelemetry, organizations faced a fragmented observability landscape.

Every monitoring vendor typically provided its own SDKs, instrumentation libraries, agents, and data collection mechanisms. Development teams often found themselves tightly coupled to specific observability platforms. Migrating from one vendor to another frequently required substantial code changes, extensive re-instrumentation efforts, and operational overhead.

This fragmentation created several challenges:

Vendor lock-in
Inconsistent telemetry formats
Duplicate instrumentation efforts
Increased operational complexity
Difficulty correlating data across tools
Limited interoperability between observability ecosystems

As cloud-native adoption accelerated, the industry recognized the need for a common observability language—a universal framework capable of collecting telemetry data once and sending it anywhere.

OpenTelemetry emerged as the answer to that problem.

What Is OpenTelemetry?

OpenTelemetry (often abbreviated as OTel) is an open-source observability framework designed to generate, collect, process, and export telemetry data from applications and infrastructure.

It provides a vendor-neutral approach for instrumenting software systems and capturing operational insights through three primary telemetry signals:

Distributed Traces
Metrics
Logs

Rather than functioning as a monitoring platform itself, OpenTelemetry acts as the telemetry pipeline that sits between applications and observability backends.

Think of OpenTelemetry as the universal data collection layer for observability.

Applications generate telemetry data using OpenTelemetry instrumentation libraries. The data is then collected, processed, enriched, and exported to monitoring platforms such as:

Grafana Labs ecosystem
Datadog
New Relic
Dynatrace
Splunk
Elastic
Custom data lakes and analytics systems

This separation between instrumentation and backend systems gives organizations unprecedented flexibility in how they manage observability.

The Three Pillars of OpenTelemetry

The core value of OpenTelemetry lies in its ability to collect multiple telemetry signals consistently across distributed systems.

1. Distributed Traces: Following Requests Across Services

Distributed tracing is arguably one of OpenTelemetry's most transformative capabilities.

In modern microservice architectures, a single user request may traverse dozens of services before returning a response.

For example:

API Gateway receives request
Authentication service validates credentials
User service retrieves profile data
Recommendation engine generates suggestions
Database processes queries
External payment service validates transaction
Response returns to the client

Without tracing, understanding the journey of that request becomes extremely difficult.

OpenTelemetry captures this journey through traces composed of spans.

Each span represents a unit of work within a service and records information such as:

Start time
End time
Duration
Errors
Metadata
Parent-child relationships

By linking spans together, OpenTelemetry creates an end-to-end transaction view that allows engineers to identify:

Latency bottlenecks
Failed dependencies
Service communication issues
Slow database operations
Cascading failures

For platform teams managing large microservice environments, distributed tracing has become indispensable for troubleshooting production incidents.

2. Metrics: Measuring System Health at Scale

Metrics provide numerical measurements that describe system behavior over time.

These measurements help answer questions such as:

What is the CPU utilization of a service?
How many requests are being processed?
What is the error rate?
How much memory is being consumed?
What is the average request latency?

OpenTelemetry supports various metric types, including:

Counters

Track continuously increasing values.

Examples:

Total requests processed
Orders completed
Login attempts
Gauges

Represent current values at a specific point in time.

Examples:

Memory usage
Active connections
Queue depth

Histograms

Capture value distributions.

Examples:

Request duration
Database query latency
API response times

These metrics enable dashboards, service-level indicators (SLIs), service-level objectives (SLOs), and alerting systems that help organizations maintain reliability and performance.

For Site Reliability Engineering (SRE) and platform teams, metrics remain the first line of defense against operational issues.

3. Logs: Capturing Detailed Operational Context

Logs have long been the most familiar observability signal.

They provide detailed event records describing what occurred inside an application or infrastructure component.

Examples include:

Application startup events
Authentication failures
Database connection errors
Business transactions
Security events
Configuration changes

Historically, logs existed separately from traces and metrics.

This separation often forced engineers to switch between tools when investigating incidents.

OpenTelemetry's logging initiatives aim to create stronger relationships between all telemetry signals by introducing common context and correlation mechanisms.

As a result, engineers can more easily move from:

Metrics showing abnormal behavior
To traces revealing request paths
To logs explaining the precise failure

This unified observability experience significantly reduces troubleshooting time.

The OpenTelemetry Architecture

One reason for OpenTelemetry's rapid adoption is its flexible architecture. The framework consists of several major components that work together to create a complete telemetry pipeline.

Instrumentation

Instrumentation represents the process of generating telemetry data from applications. OpenTelemetry supports both:

Automatic Instrumentation

Telemetry collection occurs without significant code modifications.

Examples include:

Java agents
.NET auto-instrumentation
Python instrumentation libraries
Kubernetes integrations

Manual Instrumentation

Developers explicitly define spans, metrics, and attributes within application code. Manual instrumentation enables richer business-level observability, including:

Customer workflows
Checkout processes
Inventory transactions
Internal business operations

OpenTelemetry SDKs

The SDK layer provides language-specific implementations for generating telemetry data.

OpenTelemetry currently supports major programming languages including Java, Go, Python, JavaScript, Node.js, .NET, Rust, C++, PHP, Ruby

This broad language support allows organizations to instrument diverse technology stacks consistently.

OpenTelemetry Collector

The OpenTelemetry Collector is widely considered the most important operational component of the ecosystem.

The Collector functions as a vendor-neutral telemetry processing pipeline. Instead of applications sending data directly to observability platforms, telemetry is routed through collectors that can:

Receive data
Transform records
Filter telemetry
Perform sampling
Enrich metadata
Batch requests
Export to multiple destinations

This architecture provides significant operational benefits. Teams can modify telemetry routing and processing without changing application code.

They can also send the same telemetry data simultaneously to multiple backends, enabling migration strategies and multi-platform observability architectures.

Why Platform Engineering Teams Love OpenTelemetry

OpenTelemetry's popularity extends far beyond application developers. Platform engineering organizations increasingly treat OpenTelemetry as a foundational infrastructure component. There are several reasons for this shift:

Standardized Instrumentation

Instead of every team implementing observability differently, OpenTelemetry establishes a common instrumentation standard across the organization.

This consistency improves operational efficiency and reduces onboarding complexity.

Reduced Vendor Lock-In

One of OpenTelemetry's strongest value propositions is backend independence.

Organizations can change observability vendors, they can adopt new monitoring platforms, and they cab operate hybrid observability architectures

without re-instrumenting applications.

For large enterprises, this flexibility can translate into substantial cost savings and reduced migration risk.

Kubernetes-Native Design

OpenTelemetry integrates naturally with cloud-native infrastructure. It works seamlessly alongside technologies such as:

Kubernetes
Prometheus
Grafana
Service meshes
Cloud provider platforms

This compatibility makes OpenTelemetry particularly attractive within modern platform engineering ecosystems.

Scalability

Organizations operating thousands of services require telemetry systems capable of handling enormous data volumes. This compatibility makes OpenTelemetry particularly attractive within modern platform engineering ecosystems.

The OpenTelemetry Collector architecture supports:

Horizontal scaling
Distributed processing
Load balancing
High-throughput telemetry ingestion

This enables observability pipelines to grow alongside application ecosystems.

OpenTelemetry as Foundational Infrastructure

Perhaps the most significant evolution of OpenTelemetry is the role it now plays inside organizations. Initially viewed as a developer instrumentation framework, OpenTelemetry has increasingly become infrastructure in its own right. Today, many organizations deploy OpenTelemetry Collectors as platform-managed services.

Application teams simply emit telemetry while platform teams manage:

Collection pipelines
Sampling strategies
Data governance
Security controls
Routing policies
Backend integrations

This separation of concerns mirrors the broader platform engineering movement, where internal platforms abstract operational complexity away from development teams.

In many cloud-native organizations, OpenTelemetry now sits alongside Kubernetes, service meshes, ingress controllers, and CI/CD systems as core platform infrastructure. It is no longer just an observability tool—it is part of the operational fabric of modern software delivery.

The Growing Ecosystem Around OpenTelemetry

The success of OpenTelemetry extends beyond its technical capabilities. Its ecosystem has become one of the strongest examples of industry-wide collaboration in cloud-native computing. Major cloud providers, observability vendors, and open-source communities actively contribute to its development.

This widespread support has accelerated:

Standard adoption
Ecosystem integrations
Tooling maturity
Language support
Operational best practices

As organizations continue modernizing their application architectures, OpenTelemetry increasingly serves as the common observability layer connecting diverse technologies and platforms.

Looking Ahead: The Future of OpenTelemetry

The observability landscape continues to evolve rapidly.

Emerging technologies such as AI-powered operations, platform engineering, cloud-native security, and large-scale distributed systems require increasingly sophisticated telemetry strategies. OpenTelemetry is uniquely positioned to support this future.

Its open standards, vendor-neutral philosophy, and broad ecosystem adoption provide a foundation upon which next-generation observability platforms can innovate.

As telemetry data becomes more critical for automation, reliability engineering, capacity planning, security monitoring, and operational intelligence, OpenTelemetry's role will likely become even more central to modern infrastructure.

The question is no longer whether organizations should adopt OpenTelemetry.

The conversation has shifted toward how effectively they can leverage OpenTelemetry as a strategic platform capability.

Top 3 Key Takeaways

1. OpenTelemetry Has Become the Industry Standard for Observability

OpenTelemetry provides a unified, vendor-neutral framework for collecting traces, metrics, and logs across modern distributed systems, making it one of the most widely adopted cloud-native technologies today.

2. It Powers End-to-End Visibility Across Distributed Architectures

Through standardized instrumentation, SDKs, and the OpenTelemetry Collector, organizations gain comprehensive insights into application performance, system health, and operational behavior across complex microservice environments.

3. OpenTelemetry Is Now Foundational Platform Infrastructure

Beyond telemetry collection, OpenTelemetry has evolved into a core platform engineering capability that enables scalable observability, reduces vendor lock-in, and supports the operational needs of modern cloud-native organizations.

Closing Thoughts

Observability has become a prerequisite for operating reliable distributed systems, and OpenTelemetry has emerged as the connective tissue that makes modern observability possible. By standardizing telemetry generation, collection, and export across traces, metrics, and logs, it eliminates fragmentation while empowering organizations with greater flexibility, portability, and operational insight. As cloud-native architectures continue to expand in scale and complexity,.

OpenTelemetry is not merely another open-source project—it is the foundational observability infrastructure shaping how the next generation of software systems will be built, monitored, and operated.

DEV Community: Kubernetes with Naveen

Who Owns Kubernetes on Day 2? Defining the Operational Boundary

The Day 2 Reality Check: Why Healthy Clusters Still Host Broken Applications

The Core Philosophy: Platform as a Product vs. Infrastructure Monopoly

The Responsibility Breakdown: The Operational Matrix

Cluster Infrastructure & Upgrades: Platform/SRE Owns the Floor

Pod Disruption Budgets Are a Contract

Workload Lifecycle & Health: Application Teams Own What They Deploy

Access Control, Secrets & RBAC Boundaries

Production kubectl Access Is a Design Smell

Managing Configuration Drift via GitOps

Git Becomes More Than Version Control

Access Should Be Self-Service, Not Permissionless

Resource Management & Cost: Who Owns the OOMKilled Pod?

Application Team Owns

Platform Team Owns

Failure Modes of Bad Ownership: Ticket-Ops vs. The Wild West

Failure Mode #1: Ticket-Ops

Failure Mode #2: The Wild West

The Better Model: Guardrails, Not Gates

Pragmatic Recommendations for Mid-to-Senior Engineers

1. Write the Ownership Contract

2. Keep Cluster-Admin Extremely Small

3. Make Git the Default Change Path

4. Automate the Boring Requests

5. Define Upgrade Contracts

6. Make Resource Requests a First-Class Engineering Concern

7. Give Developers Production Visibility Without Production Control

8. Build Incident Ownership Around the Failure Domain

A Practical Day 2 Operating Model

3 Key Takeaways

1. Platform owns the platform. Application teams own the workloads.

2. Self-service beats both Ticket-Ops and unrestricted access.

3. Day 2 ownership is an architectural decision, not an org-chart decision.

Into The Depths of Kubernetes: Multi-Tenancy Part 3

The Data Plane & Storage Subsystem: eBPF Firewalls, CSI Scoping, and Cryptographic Isolation

The Data Plane Is Where Multi-Tenancy Gets Real

Bypassing IPTables: The eBPF Networking Revolution

eBPF Changes Where the Enforcement Happens

Identity Matters More Than IP Addresses

Default-Deny-All Is the Starting Point

Allowing DNS Without Opening the Door

Building a Tenant Network Boundary

The Storage Loophole

CSI Is a Security Boundary, Not Just a Storage Plugin

Directory Traversal Is Only One Failure Mode

Zero-Trust Storage: Encryption Per Tenant

CSI + KMS: The Architecture

Why Encryption Changes the Threat Model

KMS Policy Becomes Part of Kubernetes Security

A Practical Multi-Tenant Storage Model

Boundary 1: Provisioning

Boundary 2: Physical Isolation

Boundary 3: Cryptographic Isolation

The Real Architecture

Scenario 1: Network compromise

Scenario 2: Service discovery abuse

Scenario 3: Storage compromise

Scenario 4: Node compromise

The Platform Engineer's Final Checklist

Networking

Storage

Cryptography

The Final Layer of Multi-Tenancy

Into The Depths of Kubernetes: Multi-Tenancy Part 2

Part 2: Kernel Contention & Hard Sandboxing — Cgroups, OOM Kills, and Runtime Escapes

1. The Container Boundary Is Not the Kernel Boundary

2. Cgroups: Where Kubernetes Resource Limits Actually Become Real

3. cgroups v1 vs cgroups v2: The Hierarchy Problem

4. cgroups v2: One Hierarchy, Better Accounting

5. CPU Limits: The Throttling Trap

6. Memory Is Different: Eventually Somebody Has to Die

7. Anatomy of an OOM Kill

8. Guaranteed, Burstable, and BestEffort

9. The Burstable Trap

10. The OOM Killer Doesn't Understand Your Business

11. When a Container Becomes a Kernel Problem

12. The Syscall Attack Surface

13. containerd and CRI-O Do Not Give You a Second Kernel

14. The Engineering Question: How Much Isolation Do You Actually Need?

Production `kubectl` Access Is a Design Smell

**What an HTTP 502 Really Means at the Network Layer**

Why `ECONNRESET` matters

2. Kernel Queues: Sockets, `SOMAXCONN`, and TCP Backlog Drops

3. The Termination Gap: `SIGTERM`, Endpoint Propagation, and Inflight Traffic Drops

Don't treat `SIGTERM` as "exit immediately"

Start with `ss`

Use `strace` to See What the Process Actually Does

Use `tcpdump` When You Need the Truth