When Nodes Start Rejecting Pods: Taints and Tolerations

TL;DR

• Taints go on nodes, tolerations go on pods. A tainted node rejects pods by default, and a toleration is what lets a specific pod back in — the opposite of node selectors and affinity, where the pod was the one doing the choosing.
• Three effects: NoSchedule, PreferNoSchedule, NoExecute.
• NoExecute is the odd one out. It can evict pods that are already running, not just block new ones from landing.
• Tolerating a taint doesn't mean a pod prefers that node. It just means the taint won't stop it from scheduling.

Skip around if you already know the basics:

• The problem this actually solves
• Taints, tolerations, and the three effects
• Where this actually shows up
• What the scheduler's doing under the hood
• Taints and tolerations vs node affinity
• What goes wrong, and why
• Try it yourself

WHY: the problem

Node selectors and affinity, from Part 2, are opt-in. A pod decides which nodes it wants, and the scheduler tries to honor that.

Sometimes you need the opposite. A node that rejects almost everything by default, unless a pod specifically says it's fine with that. Control-plane nodes are the classic example, you don't want regular application pods landing there and competing with the control plane for resources.

That's what taints and tolerations are for. Instead of a pod pulling toward a node, a node pushes pods away, and only pods that explicitly tolerate that push get scheduled.

WHAT: taints, tolerations, and effects

A taint lives on a node. It's a key, a value, and an effect, written as key=value:effect.

A toleration lives on a pod. It lists the key, value, and effect the pod is willing to tolerate. Match, and the taint doesn't block that pod.

Here's a way to picture it: a taint is bug repellent sprayed on a node. A toleration is a bug that's immune to that specific spray.

The three effects behave differently, and the analogy maps onto all of them:

• NoSchedule — full-strength spray. Immune bugs land fine, everything else won't come near.
• PreferNoSchedule — a lighter dose. Bugs would rather avoid it, but if every other spot's taken, they'll land there anyway.
• NoExecute — once sprayed bugs without immunity get driven out, even the ones that landed before the spray was ever there. And new bugs won't come near.

Setting a taint on a node:

kubectl taint nodes node-1 dedicated=gpu:NoSchedule

Tolerating it from a pod:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: nginx

operator can also be Exists instead of Equal, if a pod should tolerate a key regardless of its value:

tolerations:
- key: "dedicated"
  operator: "Exists"
  effect: "NoSchedule"

One more piece worth knowing: tolerationSeconds. It only applies to NoExecute, and it's a grace period, how long a pod can stay bound to a node after that taint shows up, before it actually gets evicted.

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoExecute"
  tolerationSeconds: 3600

📖 Docs: Taints and tolerations

WHEN: where this actually shows up

A few real situations where taints and tolerations show up:

• Control-plane nodes, tainted by default so regular workloads stay off them
• Dedicated GPU or other specialized-hardware nodes, so only the workloads that need them land there
• Kubernetes itself uses this mechanism internally. When a node goes unreachable or stops responding, the node controller taints it automatically.

HOW: what's actually happening

Taints get checked during the same filtering phase from Part 1, just inverted. Normally filtering asks "can this node run the pod?" Here it's more like "has this node been told to reject the pod?" if yes and No matching toleration on pod, then node gets filtered out, same as any other hard requirement.

PreferNoSchedule doesn't touch filtering at all. It works through scoring, same as preferred node affinity in Part 2, nudging the scheduler away from tainted nodes without ruling them out.

What happens on a node that's already running pods matters here, and the three effects don't behave the same way:

• Taint a live node with NoSchedule or PreferNoSchedule, and nothing happens to pods already running there. These only affect future scheduling decisions, not the pods sitting there right now.
• Taint it with NoExecute, and any pod without a matching toleration gets evicted, immediately, or after the grace period if tolerationSeconds is set. This is the one real exception to "scheduling is a one-time decision" from Part 1, worth closing that loop here.

DIFFERENCE: taints and tolerations vs node affinity

Node affinity and taints solve a similar-sounding problem from opposite directions.

Affinity is a pod pulling toward a node it wants. Taints are a node pushing pods away by default, and a toleration is what lets a specific pod back in.

That difference matters for what each one actually guarantees. Tolerating a taint doesn't mean a pod prefers that node, or that it'll get scheduled there. It just means the taint won't block it. If the goal is a pod landing on GPU nodes specifically, not just being allowed there, tolerations usually get paired with node affinity on top. The taint keeps everyone else out, the affinity pulls the right pod in.

FAILURE: what goes wrong

The tolerate-doesn't-mean-prefer mix-up is the most common one. A pod with the right toleration can still land on an untainted node it has nothing to do with, because a toleration only removes a restriction, it doesn't express a preference.

The second one is more of a hazard than a misconception. Adding a NoExecute taint to a node that's already serving traffic can evict running pods with no warning, if that wasn't accounted for. Worth checking what's already running and what it tolerates before tainting a node people are relying on.

One more thing worth flagging now and covering properly later: DaemonSet pods tolerate several of these built-in taints automatically, which is part of why they keep running in places regular pods can't. More on that in Part 5.

PRACTICE: taint, tolerate, check

Taint a node:

kubectl taint nodes node-1 dedicated=gpu:NoSchedule

Deploy the toleration pod from the WHAT section, plus a plain one with no toleration at all:

apiVersion: v1
kind: Pod
metadata:
  name: plain-pod
spec:
  containers:
  - name: nginx
    image: nginx

kubectl apply -f gpu-pod.yaml
kubectl apply -f plain-pod.yaml
kubectl get pods -o wide

plain-pod should avoid node-1 entirely. Confirm the taint is actually there:

kubectl describe node node-1

Look for a Taints: line near the top of the output.

To remove the taint later:

kubectl taint nodes node-1 dedicated=gpu:NoSchedule-

That trailing - is easy to miss and just as easy to forget, so it's worth calling out on its own.

Next up, Part 4: pod affinity and anti-affinity, where instead of a pod caring about a node, it starts caring about other pods.

If anything here didn't land, or you'd explain it differently, drop a comment. Doubts and pushback are both welcome.