Kubernetes tutorial - Static Pods Explained

Are you ready for some Kubernetes fun? You know a few minutes on a Kubernetes tutorial wouldn't be a waste.

Alright, let's start with a Kubernetes buzzword - Pods.

Pods are the most granular resource you can manage in a Kubernetes cluster by using Kubernetes control plane. Pods can contain one or more containers, where one will deploy applications. You may also mount storage to your pods. And the pods have their own ephemeral IPs. And more. But let's not get lost in the details of a Pod definition. Our goal is to understand what Static Pods are.

First, let's take a look at a simple example of creating a pod using kubectl.

root:/home/ubuntu/pods# cat test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test-pod
  name: test-pod
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: test-pod

root:/home/ubuntu/pods# kubectl apply -f test-pod.yaml
pod/test-pod created

root:/home/ubuntu/pods# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          20s

Nothing fancy going on here - a pod resource was created using a simple yaml file which has the pod specifications.

But what is not obvious is that kubectl has interacted with Kubernetes API Server (also referred to as Kube-apiserver) which in turn communicates with a kube-scheduler component to find the best node to create the pod resource on and then instructs a kubelet agent sitting on that particular node to create the pod. Then all the task details are stored in an etcd database. This is a typical workflow that happens mostly.

By the way, a couple of ways to validate me ;)

Subscribe to my YouTube channel

Connect with me on LinkedIn

The components like kube-apiserver, kube-scheduler, kube-controller-manager, and etcd database together form the control plane of a Kubernetes cluster.

Source: Kubernetes Components | Kubernetes

While the (normal) pods are created in response to the instruction from kube-apiserver by kubelet, kubelet itself can create and manage pods on its own. The type of pods that are managed by a kubelet is called static pods.

Kubelet Service

To understand further, we will start by looking at how a kubelet service is configured. Then we will see how to get a kubelet agent to create a static pod.

My test/lab Kubernetes cluster runs on Ubuntu operating system. I believe any linux distribution should look pretty similar to what we are going to see here. Just to be clear, we are observing one of my worker nodes, where Kubernetes resources are commonly created.

root:/home/ubuntu# ps -ef|grep kubelet
root       820     1  2 May21 ?        00:12:57 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=node1 --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2

/var/lib/kubelet/config.yaml file is particular important because it contains a line that will help us to create static pods. More specifically, a key-value pair. I am referring to staticPodPath highlighted below.

root:/home/ubuntu# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

One way to create a static pod

We can simply create a pod yaml file, which has the static pod definition, in the directory '/etc/kubernetes/manifests'. Kubelet will create the static pod shortly. Let's try that.

# list the pods prior to creating a pod yaml file
root:/etc/kubernetes/manifests# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
test-pod           1/1     Running   0          21m

# this is the file I created in the directory we found before.
root:/etc/kubernetes/manifests# cat static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: static-pod
  name: static-pod
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: static-pod

# static pod gets created. kubelet appends the node name to the pod name. Interesting.
root:/etc/kubernetes/manifests# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
static-pod-node1   1/1     Running   0          26s
test-pod           1/1     Running   0          21m

What if we removed the yaml file in the manifests directory?

root:/etc/kubernetes/manifests# rm static-pod.yaml

root:/etc/kubernetes/manifests# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
static-pod-node1   1/1     Running   0          8m51s
test-pod           1/1     Running   0          30m

root:/etc/kubernetes/manifests# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          30m

Well, the static pod gets removed automatically.

Another way to create a static pod

Alternatively, we can create the pod yaml file in a different directory as shown below.

root:/etc/kubelet.d# cat static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: static-pod
  name: static-pod
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: static-pod

If we observe the status of kubelet service using a systemd command, we will see an additional configuration file being used from a Drop-In directory. I think of this directory as a place that contains the override configuration of particularly kubelet service.

root@ip-172-31-27-42:/etc/kubelet.d# systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sat 2022-05-21 19:02:44 UTC; 10h ago
     Docs: https://kubernetes.io/docs/home/

Let's dig this file - /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

root:/etc/kubelet.d# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Let's add an extra argument to "/etc/default/kubelet" file such that kubelet will be started with a new manifest path. And restart kubelet service. You will see 'static-pod-node1' gets created.

#Do this on a worker node
#Add new pod manifest path
root:/etc/kubelet.d# cat /etc/default/kubelet
KUBELET_EXTRA_ARGS="--pod-manifest-path=/etc/kubelet.d/"

#list pods
root:/etc/kubelet.d# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          47m

#restart kubelet service
root:/etc/kubelet.d# systemctl restart kubelet.service

root:/etc/kubelet.d# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
static-pod-node1   1/1     Running   0          14s
test-pod           1/1     Running   0          52m

#Observe kubelet was restarted with the new argument you added.
root:/etc/kubelet.d# ps -ef|grep kubelet
root     20820     1 13 06:13 ?        00:00:00 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=node1 --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2 --pod-manifest-path=/etc/kubelet.d/

By the way, do you know the control plane components we looked at before run as pods as well in a namespace called kube-system.

root:/home/ubuntu# kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS         AGE
etcd-master                               1/1     Running   8 (5d11h ago)    53d
kube-apiserver-master                     1/1     Running   8 (5d11h ago)    53d
kube-controller-manager-master            1/1     Running   8 (5d11h ago)    53d
kube-scheduler-master                     1/1     Running   8 (5d11h ago)    53d

At this point we understand the normal pods are created by kubelet as per the instruction from kube-apiserver. But then who told kubelet to create kube-apiserver pod? Obviously it would not have been kube-apiserver, because it didn't exist.

Well, kube-apiserver and other control plane components' pods are indeed created as static pods by kubelet. Because during the installation, the YAML files of these components are created in /etc/kubernetes/manifests directory. Now you know why we need static pods. :)

root:/home/ubuntu/pods# ls -l /etc/kubernetes/manifests/
total 16
-rw------- 1 root root 2209 Mar 29 14:49 etcd.yaml
-rw------- 1 root root 3854 Mar 29 14:49 kube-apiserver.yaml
-rw------- 1 root root 3251 Mar 29 14:49 kube-controller-manager.yaml
-rw------- 1 root root 1435 Mar 29 14:49 kube-scheduler.yaml

Comments

[Marcelo Andrade]

Great article on the operational tidbits of static pods.

I'd just give an emphasys on the fact that, to make a way for people to actually know that static pods exists, Kubelet communicates to their Apiserver the fact, so it shows when you give kubectl commands.

The important part is to comprehend that it is purely informational - as the documentation mentions, iirc, it is a "mirror image of the Pod".

As such, if you try kubectl delete on them, you're instructing APIServer to remove the mirror image, and it most definitely won't restart the real static pod.

It might not feel all that relevant, but if you deploy something that is referenced by the manifest (like certificates ou service account signing keys) it won't reload just because you deleted the mirror image, you'd have to restart/reload it on the node.