DEV Community: Rajesh Deshpande

Kubernetes Myth #12: K8s Service Account Pulls Images

Rajesh Deshpande — Thu, 13 Mar 2025 04:21:27 +0000

🔍 Myth: "A Kubernetes ServiceAccount is responsible for pulling container images."

💡 Reality: The Container Runtime on the worker node—not the ServiceAccount—pulls container images from the registry!

This misconception often leads to confusion when working with private registries and authentication. Let’s dive into the real mechanism.

🚀 Deep Dive: How Kubernetes Pulls Images

When a Pod is scheduled, Kubernetes ensures that the required container images are available on the node. But Kubelet itself does not pull images—it delegates this task to the container runtime (containerd, CRI-O, Docker).

Step-by-Step Image Pulling Process

Pod Creation & Scheduling:

A user applies a Pod spec with an image (nginx:latest).
The Kubernetes scheduler assigns the Pod to a node.

Kubelet Contacts the Container Runtime:

Kubelet detects that a new Pod needs to run on its node.
It checks if the required image is already present.
If not, Kubelet instructs the container runtime to pull the image via the Container Runtime Interface (CRI).

Container Runtime Handles Authentication & Pulling:

The container runtime (e.g., containerd, CRI-O) authenticates with the container registry.
It checks for imagePullSecrets (for private registries).
It fetches the image from the specified container registry (Docker Hub, ECR, GCR, etc.).
Once downloaded, the image is stored locally on the node.

Container is Started:

The container runtime unpacks the image and starts the container.
Kubelet monitors the container's lifecycle.

🛑 ServiceAccount is NOT involved in this process! It does not fetch images or interact with the container registry.

🔑 What is a Kubernetes ServiceAccount Actually For?
A ServiceAccount in Kubernetes is used for Pod-to-API-Server authentication, not for pulling images. Here’s what it does:

It provides a secure identity for Pods to interact with the Kubernetes API.
It is automatically mounted into Pods to authenticate API requests.
It can be used to manage RBAC permissions for Pods.
It has NOTHING to do with pulling images from a registry!

🔥 Why Do People Think ServiceAccounts Pull Images?
This myth exists because:

ImagePullSecrets can be linked to a ServiceAccount, making it seem like the ServiceAccount is responsible for authentication.
ServiceAccount tokens are automatically mounted into Pods, leading to confusion about their purpose.
Both ServiceAccounts and imagePullSecrets handle authentication, but for different things (API access vs. registry authentication).

✅ How to Pull Images Correctly in Kubernetes?

If your images are public, Kubernetes pulls them without any authentication. But for private registries, you must provide credentials explicitly using imagePullSecrets:

Option 1: Define imagePullSecrets in the Pod

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
    - name: app
      image: my-private-registry.com/my-app:v1
  imagePullSecrets:
    - name: my-registry-secret

Option 2: Attach imagePullSecrets to a ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-sa
imagePullSecrets:
  - name: my-registry-secret

Then reference the ServiceAccount in your Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  serviceAccountName: my-sa
  containers:
    - name: app
      image: my-private-registry.com/my-app:v1

Even though we attach imagePullSecrets to a ServiceAccount, it's still the Container Runtime that pulls the image—not the ServiceAccount!

🎯 Key Takeaways – Myth Busted!
🚫 Kubernetes ServiceAccounts DO NOT pull container images.
✅ Kubelet pulls images using the container runtime.
✅ Use imagePullSecrets for private registries.
✅ ServiceAccounts handle API authentication, not image pulling.

Next time someone says, “ServiceAccount pulls images”, you’ll know the truth!

Kubernetes Myth #11: CPU Requests Guarantee Reserved CPU

Rajesh Deshpande — Wed, 12 Mar 2025 13:57:02 +0000

❌ Myth: If I set cpu: 500m in my pod’s requests, Kubernetes reserves 0.5 CPU exclusively for my pod.

✅ Reality: Kubernetes does not reserve CPU like it does for memory. CPU is a compressible resource, meaning:

Your pod gets priority access to 500m CPU but can use more if available.
Other pods can borrow unused CPU from your pod if it's not using the full 500m.
If the node is under high load, the kernel’s CPU throttling kicks in, limiting your pod’s CPU usage.

📌 Example:

🔹 You set cpu: 100m request, but your pod only needs 50m most of the time.
🔹 Another pod on the same node borrows the remaining 50m CPU.
🔹 Later, when your pod suddenly needs 100m CPU, it may get throttled if the node is already overloaded.

💡 What to Do?
✅ Set realistic requests—don’t go too high or too low.
✅ Use CPU limits if you want to cap a pod’s max CPU usage.
✅ Monitor throttling using 'kubectl top pods' and Prometheus metrics.

Kubernetes Myth #10: Kube-Proxy Assigns IP Addresses to Pods

Rajesh Deshpande — Wed, 12 Mar 2025 13:54:47 +0000

Many believe that Kube-Proxy is responsible for assigning IP addresses to Pods. But is that really the case?

❌ Myth: Kube-Proxy assigns IP addresses to Pods.

✅ Reality: The CNI (Container Network Interface) plugin handles Pod IP address allocation!

🔍 Breaking it Down

When a Pod is created, it needs a unique IP address to communicate within the cluster. However:
Kube-Proxy is responsible for service-level networking and load balancing across Pods.
CNI Plugins (like Flannel, Calico, Cilium) handle Pod networking, including assigning IPs to Pods and setting up routes.

⚙️ How Pod Networking Actually Works

1️⃣ When a Pod is scheduled, Kubelet requests an IP for it.
2️⃣ The installed CNI plugin assigns an IP and configures network routes.
3️⃣ The Pod uses this IP to communicate within the cluster.
4️⃣ Kube-Proxy does not interfere with this process—it only manages routing for Kubernetes Services (e.g., ClusterIP, NodePort, LoadBalancer).

📌 Why Does This Matter?

If a Pod isn’t getting an IP, the issue is likely with the CNI plugin, not Kube-Proxy.

Misconfiguring the CNI plugin can lead to network failures in your cluster.

Understanding the difference helps in debugging Kubernetes networking issues more efficiently.

Kubernetes Myth #09: ClusterIP Services Always Use Round-Robin Load Balancing

Rajesh Deshpande — Wed, 12 Mar 2025 13:52:45 +0000

Myth: Kubernetes ClusterIP Services always distribute traffic using a round-robin algorithm.

Reality: The default iptables mode of kube-proxy does not use round-robin. Instead, it uses random probability-based selection for load balancing.

🔍 How It Actually Works:

kube-proxy sets up NAT rules using iptables.
When a request hits a ClusterIP, iptables randomly selects one of the backend pods.
The selection is not true round-robin but statistically distributed.

🔥 Want True Round-Robin?

✅ Use kube-proxy in IPVS mode:

Supports round-robin (rr), least connections (lc), and other scheduling methods.

Enable it with:

kube-proxy --proxy-mode=ipvs

✅ Use an external load balancer (e.g., NGINX, HAProxy) if strict round-robin is needed.

🚀 Kubernetes is full of surprises—what other myths have you encountered? Drop them in the comments!

Kubernetes Myth #08: Kubelet runs only on worker nodes!

Rajesh Deshpande — Wed, 12 Mar 2025 13:50:35 +0000

🧐 If Kubelet is only for worker nodes, then who runs the control plane components like the API server, scheduler, and controller-manager?

Reality: ❌ Kubelet runs on both worker and control plane nodes—but their roles differ!

✅ On Worker Nodes → Kubelet manages application pods, ensuring they run as per the scheduler’s instructions.
✅ On Control Plane Nodes → Kubelet manages static pods (like API server, scheduler, and controller-manager) by reading manifest files from /etc/kubernetes/manifests/.

🔹 The real twist? Control plane components are NOT scheduled like regular pods! Instead, Kubelet directly runs them as static pods, bypassing the API server entirely.

💡 Understanding this helps in debugging control plane failures and troubleshooting cluster issues. Next time someone says, "Kubelet is only for worker nodes," you’ll know how to bust that myth!

Have you ever had to debug Kubelet behavior on a control plane node? Share your experience in the comments! 👇

Kubernetes Myth #07: K8s Uses Limits and Requests for Scheduling

Rajesh Deshpande — Wed, 12 Mar 2025 13:42:37 +0000

💡 Reality: The Kubernetes scheduler only considers resource requests when making scheduling decisions. Limits do NOT impact scheduling, nor does actual CPU/memory usage!

🔍 How It Actually Works:

✅ Requests define scheduling – The scheduler ensures a node has enough reserved capacity before placing a pod.
❌ Limits do NOT impact scheduling – They only affect runtime behavior (CPU throttling, OOMKills).
❌ Actual resource usage is ignored – Even if a node has free CPU/memory, if requests are already "reserved," a new pod won’t be scheduled there.

🛠️ Why Does Kubernetes Work This Way?

Ensures predictable resource allocation
Prevents nodes from getting overloaded later
Avoids constant pod rescheduling, ensuring stability

🔥 Myth busted! Kubernetes scheduling is based on requests, not limits or actual usage. Misunderstanding this can lead to inefficient resource planning and unexpected scheduling issues.

Kubernetes Myth #06: Kubernetes Pods Always Need a Service Account

Rajesh Deshpande — Wed, 12 Mar 2025 13:23:37 +0000

🛑 Myth: "Every pod in Kubernetes must need a service account to function."

✅ Reality: A service account is only needed when a pod needs to communicate with the Kubernetes API server or requires an identity for authentication.

But, Kubernetes assigns one to every pod by default.

🔍 Why does Kubernetes do this?

Some workloads need to interact with the API server (e.g., retrieving secrets, managing resources, scaling applications).
Kubernetes enforces a secure-by-default approach, ensuring every pod has an identity—even if it never uses it.
It follows RBAC (Role-Based Access Control) best practices, restricting what workloads can do in the cluster.

⚠️ But what if my pod doesn’t need API access?

Even if your pod doesn’t interact with the Kubernetes API, it still gets a default service account. You can’t remove it, but you can strip its power to improve security.

You can disable token mount at two levels, Pod level and SA level.

🔍Which One to Use?

Use pod-level: When you only want to restrict specific pods.

Use SA-level: When you want to enforce the restriction namespace-wide for all pods using that SA.

Tip: If both are set, the pod-level setting takes precedence.

💡 Bottom Line: You can’t remove the service account itself, but you can make it powerless by removing its token.
This is a simple yet effective way to reduce unnecessary attack surfaces in your cluster.

Kubernetes Myth #05: ClusterIP is Only for Internal Communication

Rajesh Deshpande — Wed, 12 Mar 2025 13:22:54 +0000

🛑 The Myth:
"A ClusterIP service in Kubernetes is only for internal communication."

✅ The Reality:
Yes, a pure ClusterIP service is internal. But… even NodePort and LoadBalancer services rely on ClusterIP!

💡 How It Actually Works:
1️⃣ Every Kubernetes service (NodePort, LoadBalancer) has a ClusterIP behind the scenes.
2️⃣ External traffic first hits the NodePort (on a node) or a LoadBalancer (via a cloud provider).
3️⃣ Kubernetes routes that traffic through ClusterIP to distribute requests across pods.

🔍 Breakdown of How Services Work:
🔹 ClusterIP: Internal communication only.
🔹 NodePort: Exposes a node’s port externally, but still forwards traffic through ClusterIP.
🔹 LoadBalancer: Cloud-managed external access, but traffic ultimately flows via ClusterIP.

📌 Bottom Line: ClusterIP isn’t just for internal traffic—it’s the core of Kubernetes networking, even for external services.

💬 Have you encountered this myth before? Let’s discuss in the comments! 👇

Kubernetes Myth #04: K8s Can Work Without a CNI Plugin

Rajesh Deshpande — Wed, 12 Mar 2025 13:22:38 +0000

Think you can run Kubernetes without a CNI plugin? 🤔 Think again!

🔮 Myth: Kubernetes can function without a CNI (Container Network Interface) plugin.

✅ Reality: Kubernetes needs a CNI (Container Network Interface) plugin to function properly. Without one:
❌ Pods can’t communicate across nodes.
❌ kubectl exec and kubectl logs may stop working.
❌ Pods might get stuck in ContainerCreating.

Think of it like setting up a phone system without any network provider—sure, you have the hardware, but no way to actually connect!

A CNI (Container Network Interface) plugin is essential for:
✅ Pod-to-pod communication across nodes
✅ Service discovery & load balancing
✅ IP address management for pods

Unless you rely on host networking (which isn’t scalable), a CNI plugin like Calico, Flannel, or Cilium is a must!

💡 Moral of the story? Don’t overlook networking—without a CNI, your Kubernetes cluster is just a bunch of isolated containers.

Which CNI do you prefer? Drop your thoughts below! ⬇️

Kubernetes Myth #03: Kube-Proxy is a Must-Have

Rajesh Deshpande — Wed, 12 Mar 2025 13:21:31 +0000

You’ve been lied to. Kube-Proxy is NOT mandatory in Kubernetes.

🚫 Myth: Kubernetes needs Kube-Proxy to function.

✅ Reality: Kubernetes can run just fine without Kube-Proxy if a CNI plugin like Cilium takes over its job using eBPF-based networking.

The Shocking Truth

Kube-Proxy traditionally manages service-to-pod communication using iptables/ipvs, but that comes with overhead. What if you could eliminate it entirely?

That’s where Cilium steps in. Instead of routing packets through layers of iptables, Cilium hooks directly into the kernel using eBPF—handling traffic dynamically, efficiently, and at lightning speed.

Why Should You Care?

⚡ Lower Latency – No more iptables slowdowns.
📈 Better Scalability – Handles massive clusters with ease.
🔒 Stronger Security – Enables fine-grained network policies & deep observability.

So next time someone tells you Kube-Proxy is mandatory, challenge them. Kubernetes can be even more powerful without it.

Have you tried Kubernetes without Kube-Proxy? Drop your thoughts below! ⬇️

Kubernetes Myth #02: All Pods Are Created Using the API Server and Scheduler

Rajesh Deshpande — Wed, 12 Mar 2025 12:59:57 +0000

❌ Myth: Every Pod in Kubernetes is created through the API server and scheduled by the Kubernetes scheduler.

✅ Reality: Not all Pods follow this path! There’s a special type of Pod that completely bypasses the API server and scheduler—Static Pods.

Here’s how they break the rules:
🔹 No API Server Involvement – Static Pods are launched directly by the kubelet from YAML files stored in "/etc/kubernetes/manifests/".
🔹 No Scheduler Needed – The kubelet binds them to a specific node instead of waiting for the Kubernetes scheduler to assign them.
🔹 Mirror Pods in the API Server – They appear in kubectl get pods, but deleting them via kubectl delete pod won’t work—you have to remove their YAML file!
🔹 Critical for Kubernetes Itself – The API server, scheduler, and controller manager in self-managed clusters are all Static Pods running outside the normal scheduling flow.

💡 Fun Fact: Since Static Pods start before the CNI plugin is ready, they initially have no network! Only after the CNI kicks in do they get an IP.

Would you ever use a Static Pod outside of the control plane? Let’s discuss below!

Kubernetes Myth #01: You Need 3 Control Plane Nodes!

Rajesh Deshpande — Wed, 12 Mar 2025 12:47:40 +0000

We've all heard this:
"Kubernetes requires at least 3 control plane nodes for high availability!"

But here’s the reality—that’s not always true.

Take AWS EKS, for example. It runs with only 2 API server nodes, not 3. However, it still ensures high availability.

So, what’s going on? 🤔

🔹 The real 3-node requirement comes from etcd, not Kubernetes itself. etcd, which stores the cluster state, follows Raft consensus and needs a quorum to function. That’s why AWS EKS runs 3 etcd nodes—to tolerate failures while maintaining consistency.

🔹 But the Kubernetes API Server, Scheduler, and Controller Manager don’t require quorum. That’s why AWS EKS can run just 2 API server nodes while keeping the control plane available.

💡 Takeaway?
Not every Kubernetes cluster follows the same HA model. Don’t blindly apply rules—understand why they exist.