DEV Community

Sina Tavakkol
Sina Tavakkol

Posted on

40 Days Of Kubernetes (14/40)

#kubernetes #40daysofkubernetes

Day 14/40

Taints and Tolerations in Kubernetes

Video Link
@piyushsachdeva
Git Repository
My Git Repo

We're going to look at taint and toleration. While a node has a label it means it has taint for scheduling a workload with that specific label and doesn't tolerate other workloads to be scheduling on itself.

We taint node and tell a pod to tolerate that taint to be scheduled on that node.
Image description
(Photo from the video)

"Tolerations are applied to pods. Tolerations allow the scheduler to schedule pods with matching taints. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also evaluates other parameters as part of its function.

Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints."
Note There are two special cases:

  • An empty key with operator Exists matches all keys, values and effects which means this will tolerate everything.

  • An empty effect matches all effects with key key1.

The allowed values for the effect field are:

  • NoExecute > for newer and existing pods
  • NoSchedule > for newer pods
  • PreferNoSchedule > No Guaranteed

source

A toleration is essentially the counter to a taint, allowing a pod to ignore taints applied to a node. A toleration is defined in the pod specification and must match the key, value, and effect of the taint it intends to tolerate.
Toleration Operators: While matching taints, tolerations can use operators like Equal and Exists.
The Equal operator requires an exact match of key, value, and effect , whereas the Exists operator matches a taint based on the key alone, disregarding the value.

For instance:

tolerations:
- key: "key"
  operator: "Equal"
  value: "value"
  effect: "NoSchedule"
Enter fullscreen mode Exit fullscreen mode

source

1. Taint the node 

root@localhost:~# kubectl get nodes
NAME                       STATUS   ROLES           AGE   VERSION
lucky-luke-control-plane   Ready    control-plane   8d    v1.30.0
lucky-luke-worker          Ready    <none>          8d    v1.30.0
lucky-luke-worker2         Ready    <none>          8d    v1.30.0
root@localhost:~# kubectl taint node lucky-luke-worker gpu=true:NoSchedule
node/lucky-luke-worker tainted
root@localhost:~# kubectl taint node lucky-luke-worker2 gpu=true:NoSchedule
node/lucky-luke-worker2 tainted
root@localhost:~# kubectl describe node lucky-luke-worker | grep -i taints
Taints:             gpu=true:NoSchedule
Enter fullscreen mode Exit fullscreen mode
  • Let's schedule a pod 
root@localhost:~# kubectl run nginx --image=nginx
pod/nginx created
root@localhost:~# kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   0/1     Pending   0          6s
Enter fullscreen mode Exit fullscreen mode

It says it's in Pending status, so let's see the error message of the pod:

root@localhost:~# kubectl describe pod nginx
Name:             nginx
Namespace:        default
Priority:         0
Service Account:  default
Node:             <none>
Labels:           run=nginx
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4xh8p (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  kube-api-access-4xh8p:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  89s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {gpu: true}. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Enter fullscreen mode Exit fullscreen mode

And the message is clear to us :)

0/3 nodes are available.
1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }
2 node(s) had untolerated taint {gpu: true}
0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Enter fullscreen mode Exit fullscreen mode
  • We need create toleration on a pod to be scheduled 
root@localhost:~# kubectl run redis --image=redis --dry-run=client -o yaml > redis_day14.yaml
root@localhost:~# vim redis_day14.yaml
Enter fullscreen mode Exit fullscreen mode

Adding tolerations to yaml file:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: redis
  name: redis
spec:
  containers:
  - image: redis
    name: redis
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
status: {}
Enter fullscreen mode Exit fullscreen mode

Apply the file:

root@localhost:~# kubectl apply -f redis_day14.yaml
pod/redis created
root@localhost:~# kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   0/1     Pending   0          10m
redis   1/1     Running   0          5s
root@localhost:~# kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
nginx   0/1     Pending   0          10m   <none>        <none>               <none>           <none>
redis   1/1     Running   0          17s   10.244.2.12   lucky-luke-worker2   <none>           <none>
Enter fullscreen mode Exit fullscreen mode

Let's delete the taint of one node and see what will happen to our pending pod:

root@localhost:~# kubectl taint node lucky-luke-worker gpu=true:NoSchedule-
node/lucky-luke-worker untainted
root@localhost:~# kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          22m   10.244.1.14   lucky-luke-worker    <none>           <none>
redis   1/1     Running   0          11m   10.244.2.12   lucky-luke-worker2   <none>           <none>
Enter fullscreen mode Exit fullscreen mode
  • By default, the control-plane node has taint NoSchedule 
root@localhost:~# kubectl get nodes
NAME                       STATUS   ROLES           AGE   VERSION
lucky-luke-control-plane   Ready    control-plane   8d    v1.30.0
lucky-luke-worker          Ready    <none>          8d    v1.30.0
lucky-luke-worker2         Ready    <none>          8d    v1.30.0
root@localhost:~# kubectl describe node lucky-luke-control-plane | grep Taint
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Enter fullscreen mode Exit fullscreen mode

Selector

Instead a node can decide which type pod to accept, it will give the decision to a pod to which node can deployed on.

  • Let's try: 
root@localhost:~# kubectl run nginx2 --image=nginx --dry-run=client -o yaml > nginx2-day14.yaml
root@localhost:~# vim nginx2-day14.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx2
  name: nginx2
spec:
  containers:
  - image: nginx
    name: nginx2
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  nodeSelector:
    gpu: "false"
status: {}
Enter fullscreen mode Exit fullscreen mode 
root@localhost:~# kubectl apply -f nginx2-day14.yaml
pod/nginx2 created
root@localhost:~# kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
nginx    1/1     Running   0          46m   10.244.1.14   lucky-luke-worker    <none>           <none>
nginx2   0/1     Pending   0          8s    <none>        <none>               <none>           <none>
redis    1/1     Running   0          35m   10.244.2.12   lucky-luke-worker2   <none>           <none>
Enter fullscreen mode Exit fullscreen mode
  • Label one node and let's see what will happen: 
root@localhost:~# kubectl label node lucky-luke-worker gpu="false"
node/lucky-luke-worker labeled
root@localhost:~# kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE     IP            NODE                 NOMINATED NODE   READINESS GATES
nginx    1/1     Running   0          49m     10.244.1.14   lucky-luke-worker    <none>           <none>
nginx2   1/1     Running   0          3m21s   10.244.1.15   lucky-luke-worker    <none>           <none>
redis    1/1     Running   0          38m     10.244.2.12   lucky-luke-worker2   <none>           <none>
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

seifrajhi profile image

Simplify Kubernetes Resource Management with KRO by AWS

saifeddine Rajhi -

sayanide profile image

LitmusChaos is joining Kubecon + CloudNativeCon North America 2024!

Sayan Mondal -

atixag profile image

Scaling Applications to Zero with Kubernetes and KEDA

ATIXAG -

thenjdevopsguy profile image

Why WASM: Quickstart

Michael Levan -