Kubernetes Architecture Deep Dive: From Resource Limits to Custom Operators

Resource Management (Requests & Limits)

Resource management prevents Kubernetes workloads from depleting node hardware or causing resource contention among containers.

Resource Requests

The absolute minimum CPU and Memory guaranteed for a Pod to start. The Kubernetes Scheduler uses requests to determine node placement. If a node cannot fulfill the requested resources, the Pod will not be scheduled on that node.

Architectural Takeaway: Omitting requests leads to poor scheduling decisions, resulting in unbalanced cluster distribution and potential node starvation.

Resource Limits

The maximum ceiling of CPU and Memory a Pod is allowed to consume.

• Memory (Non-Compressible Resource): If a process (e.g., a memory-heavy Pandas pipeline) exceeds its memory limit by even 1 MB, the Linux Kernel terminates the container with an OOMKilled (Out Of Memory Killed) exit code. This acts as a critical safety circuit breaker to constrain the blast radius and protect co-located services.

• CPU (Compressible Resource): Unlike memory, exceeding CPU limits does not terminate the pod. Instead, the Linux Completely Fair Scheduler (CFS) enforces CPU Throttling. This constrains CPU usage, keeping the application alive but causing severe latency spikes during heavy traffic.

Architectural Takeaway: Memory limits protect nodes from crashing due to leaks, while improperly tuned CPU limits risk performance degradation via throttling even when the host node has idle CPU capacity.

Advanced Architectural Concepts

Beyond basic resource allocation, managing production Kubernetes clusters requires an understanding of hardware overcommitment, Linux Kernel throttling mechanics, and implicit Quality of Service (QoS) eviction hierarchies.

+-----------------------------------------------------------------------------------+
|                            Node Hardware Capacity                                 |
|                                                                                   |
|  +---------------------------+  +---------------------------+  +---------------+  |
|  | Guaranteed Pod            |  | Burstable Pod             |  | BestEffort    |  |
|  | Requests == Limits        |  | Requests < Limits         |  | No Req/Limits |  |
|  | (Lowest OOM Kill Priority)|  | (Medium Eviction Risk)    |  | (First Killed)|  |
|  +---------------------------+  +---------------------------+  +---------------+  |
+-----------------------------------------------------------------------------------+

Resource Overcommit

Overcommit occurs when the sum of all container resource limits on a node exceeds the node's actual physical hardware capacity, while the sum of resource requests remains within capacity bounds.

• The Economics: In cloud infrastructure, running compute nodes at 20% average utilization is an expensive waste. Overcommit allows engineering teams to pack more workloads onto fewer nodes by betting that not all pods will hit their maximum resource limits simultaneously.

• The Overcommit Trade-off:

  • CPU Overcommit: Safe and manageable. CPU is a compressible resource; if demand exceeds capacity, execution speed slows down across pods.
  • Memory Overcommit: High risk. Memory is non-compressible. If multiple pods suddenly spike toward their memory limits concurrently, the node runs out of physical RAM and swap. The Linux Kernel triggers OOMKilled events to forcibly terminate containers and reclaim memory.

• Architectural Takeaway: Calculate the Overcommit Ratio (Sum of Limits) / (Physical Capacity) carefully. Overcommit CPU aggressively to save money, but keep memory overcommit conservative to avoid cascading application crashes.

The Linux CFS Quota & CPU Throttling Trap

Setting CPU limits relies on the Linux Kernel Completely Fair Scheduler (CFS) using cgroup enforcement. The Kernel evaluates CPU usage in enforced time windows, typically every 100ms (the CFS Period).

• How the Trap Works: If a pod with a CPU limit of 1 vCPU (1000m) executes a multi-threaded operation that consumes 100ms worth of CPU processing time within the first 20ms of a period, the Kernel locks out the pod's CPU access for the remaining 80ms of that window.

100ms CFS Period Window
┌────────────────────────┬─────────────────────────────────────────┐
│  Multi-Threaded Burst  │          CPU Throttled / Locked         │
│   (Consumes 100ms CPU) │           (Latency Spikes Hit)          │
└────────────────────────┴─────────────────────────────────────────┘
 0ms                    20ms                                     100ms

• The Idle Node Paradox: A pod can experience severe CPU Throttling (causing 500ms+ latency spikes in HTTP services) even when the underlying host worker node shows 80% idle CPU capacity.

• Architectural Takeaway: Many enterprise SRE teams disable CPU limits entirely (Limits: Unset) for latency-sensitive microservices, relying strictly on well-tuned CPU Requests paired with Horizontal Pod Autoscaler (HPA) to handle traffic spikes safely.

Quality of Service (QoS) Classes & Eviction Order

Kubernetes automatically assigns every pod a QoS Class based on how its container requests and limits are configured. When a worker node experiences memory pressure, the Linux Kernel and Kubernetes Kubelet use the QoS class to determine eviction priority via the oom_score_adj metric.

Node Resource Exhaustion (Eviction Sequence)
────────────────────────────────────────────────────────────────────────►
[BestEffort Pods]   ───►   [Burstable Pods]   ───►   [Guaranteed Pods]
 (Terminated First)         (Terminated Second)       (Protected / Last)

• Guaranteed (Highest Priority / Protected)

  • Condition: Every container in the pod must explicitly specify both CPU and Memory, and Requests == Limits for all resources.
  • Behavior: Highly stable. Granted an oom_score_adj of -997. These pods are the absolute last to be evicted or terminated during node memory starvation. Ideal for databases, core stateful sets, and critical payment services.

• Burstable (Medium Priority / Standard)

  • Condition: At least one container specifies a request or limit, but Requests != Limits (or CPU has a limit while Memory does not).
  • Behavior: Allowed to burst beyond baseline when capacity allows. Evicted after all BestEffort pods are killed if memory pressure persists. Ideal for web APIs, background workers, and standard web applications.

• BestEffort (Lowest Priority / Disposable)

  • Condition: No requests or limits are defined for any container in the pod.
  • Behavior: Assigned an oom_score_adj of 1000. Gets access to unallocated node resources, but is the first target for termination during node memory pressure. Ideal for non-critical batch processing, dev/test pods, or temporary log collectors.

Security and Observability

The control mechanisms that keep the system resilient before everything crashes or when a breach occurs.

RBAC (Role-Based Access Control)

The identity card and permission firewall for your code. It strictly limits what a Pod (or a user) can execute within the system.

• ServiceAccount: The identity assigned to a Pod.
• Role/ClusterRole: The explicit list of allowed permissions.
• RoleBinding/ClusterRoleBinding: The bridge that staples an identity to a set of permissions. Even if a vulnerability leaks into the application code, RBAC prevents lateral movement and protects the underlying infrastructure.

Monitoring & Alerting

Kubernetes knows whether an application is running, but it is blind to the question: “Is the business logic actually behaving correctly?” To solve this:

• Prometheus: Continuously scrapes and collects every metric (CPU, API response times, queue lengths, etc.) from the system.
• Grafana: Transforms raw time-series metrics into visual, customizable dashboards.
• Alertmanager: Triggers notifications (Slack, PagerDuty, Email) when response latency spikes from 200ms to 4 seconds, or when OOMKilled errors start popping up.

Service Mesh (Istio/Linkerd)

An infrastructure highway that manages, encrypts, and observes inter-service communication (East-West Traffic).

• Sidecar Proxy: A lightweight proxy (like Envoy or NGINX) injected alongside your main application container inside the same Pod. It intercepts all inbound and outbound traffic completely transparently to your application code.
• mTLS (Mutual TLS): Automatically encrypts network traffic between microservices in transit. Even if an attacker intercepts the network traffic, the payload remains unreadable.
• Traffic Splitting: Allows you to route a portion of live traffic (e.g., 10%) to a new application version (Canary deployment) without altering a single line of application code.

Pod Lifecycle and Probes

Kubernetes evaluates the true health and readiness of an application inside a Pod using three distinct types of Probes.

Startup Probe

Ensures Kubernetes remains patient while an application boots up (for instance, legacy Java applications that might take over a minute to initialize). No other probes run until the Startup Probe succeeds. If this probe fails, Kubernetes assumes the application is stuck during startup and immediately restarts the Pod.

Liveness Probe

Answers the question: “Is this Pod alive?” If the application enters a deadlock or infinite loop, Kubernetes detects the stall via the probe, terminates the unresponsive container, and restarts it according to the restart policy.

Readiness Probe

Answers the question: “This Pod is alive, but is it ready to accept incoming user requests?” For example, the application process might be running, but it’s still establishing connection to a database. If this probe fails, Kubernetes does NOT kill or restart the Pod; it simply removes the Pod’s IP from the Service endpoints to stop routing traffic to it. Once the connection is established and the probe passes, traffic automatically resumes.

Networking Architecture: Ingress vs. Service

The networking layer that defines how traffic flows within the cluster and how external users access internal workloads.

ClusterIP (Service)

An internal extension line that allows Pods to communicate with one another within the cluster. It is completely isolated from the outside world.

NodePort (Service)

Exposes a specific port on every Node directly to the public internet. It is insecure and inefficient, making it rarely suitable for production environments.

LoadBalancer (Service)

Provisions a dedicated, paid external Cloud Load Balancer from your cloud provider (e.g., AWS ALB/NLB). Provisioning a separate cloud load balancer for every single microservice drastically inflates your cloud bill.

Ingress

A single Smart Traffic Router (Reverse Proxy) positioned at the cluster gateway. It terminates traffic from a single external Load Balancer and intelligently routes incoming requests to internal services based on domain names or URL path rules (/api, /auth). It minimizes infrastructure costs while providing a centralized point for TLS termination and traffic management.

Smart Placement

It is the set of rules that decides which applications run on which physical/virtual servers.

Taints

A security barrier (or label) applied to a Node. It is a node’s way of saying: “I am an expensive, GPU-heavy server. Standard web APIs without the right **tolerance* should stay away from me.”*

Tolerations

A specification written inside a Pod’s configuration. It is a Pod’s way of saying: “Yes, that server has a GPU taint, but I have the tolerance for it. You can place me there.”

Node Affinity

A specific requirement whispered by a Pod to the Kubernetes Control Plane. It is a Pod’s way of saying: “Do not place me randomly; run me strictly (or preferably) on Memory-Optimized nodes.”

• Hard Constraint (required...): “I MUST have this node type, or do NOT deploy me at all.” (Strict rule)
• Soft Constraint (preferred...): “I WOULD LIKE this node type, but if it’s not available, just put me anywhere.” (Preference)

Scaling and Autonomy

A set of mechanisms used to balance performance and infrastructure costs as workload demand increases or decreases.

HPA (Horizontal Pod Autoscaler)

Monitors CPU and RAM usage of Pods. If a Pod is under heavy load (heats up), HPA increases the replica count; as load drops (cools down), it scales back down. It is inherently sluggish because waiting for CPU metrics to spike takes time.

KEDA (Kubernetes Event-Driven Autoscaling)

Grants Kubernetes the ability to listen to external event sources (such as Redpanda, Kafka, S3, RabbitMQ, etc.). If a queue in Redpanda is empty, KEDA can scale the Pod replicas down to zero (Scale-to-Zero) to completely eliminate compute costs. If 10,000 messages suddenly hit the queue, it immediately spins up 50 Pods within seconds without waiting for CPU usage to rise.

Data and Storage

The layer that overcomes the ephemeral (temporary) nature of containers to manage persistent data.

CSI (Container Storage Interface)

The standard interface Kubernetes uses to communicate with cloud providers regarding storage. When a Pod needs a 50 GB disk for a database or AI model weights, the CSI driver dynamically provisions that disk from the cloud and physically attaches it to the node hosting the Pod. If the Pod dies and reschedules onto another node, CSI detaches the disk and reattaches it to the new node.

PV (Persistent Volume)

The physical/actual disk allocated to the cluster by the infrastructure administrator. It is a cluster-level resource independent of Pods, representing a declaration like: “I have a 100 GB SSD-backed disk available in AWS.”

PVC (Persistent Volume Claim)

A request for storage made by a developer (or application) to Kubernetes. It is a Pod’s way of saying: “I urgently need a 20 GB high-speed read/write disk.”

How PV and PVC Interact

When a developer creates a PVC, Kubernetes acts like a matchmaker. It scans the pool of available PVs. If it finds a PV that meets the requested capacity and access modes, it binds that PVC to the PV. The Pod only knows the name of the PVC and doesn’t care about the underlying storage implementation.

SC (StorageClass)

The set of instructions given to the CSI driver. When a developer specifies storageClassName: gp3 inside a PVC, Kubernetes checks if a matching PV exists. If no pre-provisioned PV is available, it immediately triggers the CSI driver. The CSI driver then provisions a 20 GB volume on AWS, registers it in Kubernetes as a new PV, and binds it to the PVC. This entire automated workflow is called Dynamic Provisioning.

Teaching K8s a New Language (Extending Kubernetes)

The capability that transforms Kubernetes from a simple application runner into a customizable platform builder.

CRD (Custom Resource Definition)

The process of extending Kubernetes’ built-in dictionary (which natively includes objects like Pods, Services, and Deployments) with your own custom resources. You define and register brand-new objects—such as ModelDeployment or PostgresDatabase—directly into the Kubernetes API database (etcd).

Operator Pattern

Kubernetes natively recognizes custom resources like ModelDeployment, but it doesn’t know what actions to take when one is created. An Operator is a custom controller written in languages like Go or Python that hooks into the Kubernetes control loop. When someone submits a ModelDeployment YAML manifest, the Operator wakes up and says: “Got it! I need to pull the AI model weights from S3, schedule a GPU-enabled node, and expose a REST API.” It then automatically orchestrates all the underlying standard Kubernetes objects on your behalf.