DEV Community

Cover image for Node Selectors and Node Affinity: Telling the Scheduler Where Pods Can Run
Siddharth More
Siddharth More

Posted on AI-assisted

Node Selectors and Node Affinity: Telling the Scheduler Where Pods Can Run

TL;DR

  • nodeSelector matches a pod to a node using labels. Simple, exact match, nothing fancier.
  • Node affinity does the same job with more expressive rules: In, NotIn, Exists, DoesNotExist, Gt, Lt.
  • Affinity comes in two flavors: required (hard rule) and preferred (soft nudge, weighted).
  • IgnoredDuringExecution means once the pod's running, changes to the node's labels don't affect it. No eviction, no re-check.

Skip around if you already know the basics:

WHY: the problem

In Part 1, we saw kube-scheduler filter out nodes that can't run a pod, then score whatever's left. That's default. Most of the time, you don't need to touch it.

But sometimes the scheduler doesn't know something you know. Maybe only some of your nodes have GPUs. Maybe a database needs SSD-backed storage, and a spinning disk will tank its performance. Nothing in a plain pod spec tells the scheduler to care about that.

Node selectors and node affinity exist to close that gap. They let you attach the missing context yourself.

WHAT: the two mechanisms

Both work off the same idea: label your nodes, then tell the pod which labels to look for.

nodeSelector is the simple version. Give it one or more key-value pairs, and the pod only lands on a node that has every one of them.

apiVersion: v1
kind: Pod
metadata:
  name: nodeselector-pod
spec:
  nodeSelector:
    disktype: ssd
  containers:
  - name: nginx
    image: nginx
Enter fullscreen mode Exit fullscreen mode

Node affinity does the same job, but with real operators instead of one flat match: In, NotIn, Exists, DoesNotExist, Gt, Lt.

Worth noting: there's no separate "node anti-affinity" field - using NotIn or DoesNotExist as the operator is what gets you that behavior, keeping a pod away from nodes that match instead of drawing it toward them.

It also splits into two types:

  • requiredDuringSchedulingIgnoredDuringExecution - a hard rule, functionally like nodeSelector but more expressive
  • preferredDuringSchedulingIgnoredDuringExecution - a soft rule with a weight (1-100) that nudges the scheduler's scoring, without blocking the pod if nothing matches
apiVersion: v1
kind: Pod
metadata:
  name: affinity-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - hdd
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - us-east-1a
  containers:
  - name: nginx
    image: nginx
Enter fullscreen mode Exit fullscreen mode

Read that as: this pod MUST land on an HDD node, and it would prefer us-east-1a, but it won't sit Pending for the zone to match.

One thing that trips people up: nodeSelectorTerms doesn't just take matchExpressions. It also takes matchFields, and the two aren't interchangeable.

matchExpressions matches against node labels, the same labels you set earlier. matchFields matches against node fields instead, things like the node's actual name in the API, not something someone labeled.

matchFields lets node affinity match against supported node fields rather than labels. A common example is matching metadata.name to target a particular node.

apiVersion: v1
kind: Pod
metadata:
  name: pinned-to-node-1
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - node-1
  containers:
  - name: nginx
    image: nginx
Enter fullscreen mode Exit fullscreen mode

Worth being honest about this one: it's rarely the right call. Hard-coding a node name defeats the whole point of a label-based system, and if that node ever gets replaced, your pod just stops scheduling. Reach for matchExpressions and labels first.

There's one more thing worth nailing down: how multiple expressions and values actually combine. It's not obvious just from looking at the YAML.

Inside one matchExpressions list, every entry has to match. That's an AND.

Inside one expression's values list, matching any single value is enough. That's an OR.

And across separate entries in nodeSelectorTerms itself, only one term has to fully match. That's another OR, one level up.

Stacked together, it looks like this:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: zone
            operator: In
            values:
            - us-east-1a
            - us-east-1b
          - key: disktype
            operator: In
            values:
            - ssd
        - matchExpressions:
          - key: zone
            operator: In
            values:
            - us-west-2a
Enter fullscreen mode Exit fullscreen mode

Read that as:
(zone is (us-east-1a OR us-east-1b), AND disktype is ssd)
OR
(
zone is us-west-2a).

It looks like a lot stacked up like that, but it's really just three separate rules, each with its own logic, nested inside each other.

You can also use both nodeSelector and nodeAffinity,in that case a node must satisfy both before the Pod can be scheduled.

๐Ÿ“– Docs: nodeSelector ยท node affinity

WHEN: where node selection actually matters

A few real situations where controlling Pod placement makes sense, regardless of which mechanism you reach for:

  • GPU workloads that should only run on GPU-equipped nodes
  • Databases or caches that need SSD-backed storage
  • Workloads you want kept in a specific zone, without hard-blocking if that zone's full
  • Compliance or licensing rules that pin certain workloads to certain hardware

If none of that applies, the scheduler's default behavior is usually fine on its own.

HOW: what's actually happening

nodeSelector and the required affinity rule both get evaluated during the filtering phase from Part 1. Any node that doesn't match gets thrown out before scoring even starts.

The preferred affinity rule works differently. It doesn't touch filtering at all. Once the feasible set is decided, the scheduler adds the rule's weight to a node's score if that node matches. Highest score wins, same as any other scoring factor.

Both mechanisms share one behavior worth calling out: they're ignored during execution. The rule only gets checked once, at scheduling time. If someone relabels the node an hour later, nothing happens to the pod already running there.

TRADE-OFFS: nodeSelector vs affinity

nodeSelector is less YAML and easier to read. If the requirement really is "must have this one label," there's no reason to reach for affinity instead.

The moment you need OR logic, negation, or a "prefer but don't require," nodeSelector can't do it and affinity can. That flexibility costs a bit more nesting, but there's no real downside beyond that.

One thing neither of these solves: pod-to-pod placement, like "run this next to that" or "spread these apart." That's pod affinity and anti-affinity, coming in Part 4. And if the goal is nodes actively rejecting pods rather than pods choosing nodes, that's taints and tolerations, next up in Part 3.

FAILURE: what goes wrong

The most common failure: you set a required rule, no node matches it, and the pod remains Pending until a suitable node becomes available or the scheduling constraints change. There's no obvious error, just a pod that never starts. kubectl describe pod, specifically its Events section, is where you'll actually see the scheduling failure reason.

The second one catches people off guard: because these rules are IgnoredDuringExecution, relabeling a node does nothing to pods already scheduled there. If you're expecting a label change to trigger a reschedule, it won't. You'd have to delete and recreate the pod yourself.

PRACTICE: label, deploy, check

Label two nodes differently:

kubectl label node node-1 disktype=ssd
kubectl label node node-2 disktype=hdd
kubectl get nodes --show-labels
Enter fullscreen mode Exit fullscreen mode

Save the two pod specs from the WHAT section as nodeselector-pod.yaml and affinity-pod.yaml, then deploy both:

kubectl apply -f nodeselector-pod.yaml
kubectl apply -f affinity-pod.yaml
kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

For the nodeSelector pod, describe shows it directly:

kubectl describe pod nodeselector-pod
Enter fullscreen mode Exit fullscreen mode

Look for a Node-Selectors: line. It'll show disktype=ssd.

kubectl describe pod is useful for inspecting scheduling information and events, but it isn't the best way to inspect the complete affinity configuration. To see the exact affinity rules attached to the Pod, inspect the Pod spec directly:

kubectl get pod affinity-pod -o yaml | grep -A 16 affinity
Enter fullscreen mode Exit fullscreen mode

That's the only reliable way to confirm what's actually attached to a running pod.

Next up, Part 3: taints and tolerations, where instead of pods choosing nodes, nodes start rejecting pods.

For those who read till end, I appreciate your patience since this was bit complex, and I too had to struggle to shape this article.

If anything here didn't land, or you'd explain it differently, drop a comment. Doubts and pushback are both welcome.

Top comments (0)