K8s: Topology Spread

Following the previous article, where we were introduced to the affinity club, we concluded that if:

1. We apply pod anti-affinity to our critical service.
2. We have only two worker nodes.

We are “safe” if one node fails, but we cannot scale beyond two replicas during a traffic spike or another period of distress.

This is where topology spread constraints come into play.

Before continuing, it helps to remember what the Kubernetes scheduler does: whenever a new Pod needs a home, it filters out unsuitable nodes and then chooses among the remaining candidates.

Topology spread constraints participate in that decision; they do not move Pods that are already running.

In our web Deployment, we add the following constraint:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    nodeAffinityPolicy: Honor
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app: web

Let’s dissect it one field at a time.

maxSkew: 1

maxSkew defines the largest permitted difference in the number of matching Pods between topology domains. In our case, each worker node is a topology domain.

For two worker nodes, the possible distributions look like this:

worker	worker2	Allowed?
1	1	✅
2	1	✅
2	2	✅
3	1	❌
3	0	❌

If the difference is greater than 1, the distribution is not allowed. A new Pod that would violate this rule remains Pending.

topologyKey: kubernetes.io/hostname

The topology key tells the scheduler how to divide the cluster into topology domains.

Because kubernetes.io/hostname normally has a unique value on each node, the scheduler spreads matching Pods across individual nodes. In a larger cluster, we could instead spread across zones by using a label such as topology.kubernetes.io/zone.

Node inclusion policies

nodeAffinityPolicy: Honor
nodeTaintsPolicy: Honor

These fields control whether node affinity and taints are respected when the scheduler decides which topology domains participate in the skew calculation.

nodeAffinityPolicy: Honor has no practical effect in this experiment because the web Pods do not define a nodeSelector or node affinity. nodeTaintsPolicy: Honor, however, matters because our nodes have taints.

nodeTaintsPolicy

It controls whether node taints are considered when the scheduler calculates the topology spread.

In our Kind cluster, the control-plane node has a NoSchedule taint that the web Pods do not tolerate.

With nodeTaintsPolicy: Honor, the scheduler respects that taint and excludes the control-plane node from the spread calculation. It therefore calculates the distribution using only worker and worker2.

Without Honor

Without Honor, the default policy is Ignore. The control-plane node could then be counted as an empty topology domain even though the web Pods cannot actually run there.

A distribution such as 1 / 1 / 0 could make the next Pod violate maxSkew: 1 and remain Pending.

Honor does not add a taint or evict existing Pods. It only controls which nodes participate in the topology-spread calculation.

whenUnsatisfiable: DoNotSchedule

If scheduling a Pod would violate maxSkew, the scheduler leaves it Pending.

In other words, DoNotSchedule prioritizes satisfying the spread constraint over scheduling the Pod immediately.

labelSelector

labelSelector:
  matchLabels:
    app: web

The label selector determines which Pods are counted when calculating the distribution.

We are adding the constraint to a Deployment, so shouldn’t Kubernetes automatically count all Pods from that Deployment? No. The scheduler counts Pods whose labels match this selector. The web Pod template must therefore carry the matching app: web label.

I know it feels a little weird. You can inspect the result visually with Headlamp.

Let’s see what we did:

With four replicas, the scheduler places two Pods on each worker.

Now we cordon worker2 and recreate the web Pods.

kubectl cordon marks the node unschedulable;
kubectl drain would first cordon it and then evict eligible Pods.

In our cluster, the cordoned node receives the node.kubernetes.io/unschedulable:NoSchedule taint.

With nodeTaintsPolicy: Honor, the scheduler excludes that node from the spread calculation.

Because we have not set minDomains, worker becomes the only eligible topology domain. If it has enough capacity, all replacement Pods can be scheduled there.

But in the second case, did we hide the issue under the carpet?

Yes and no.

With maxSkew: 1, we wanted to say:

If one worker disappears, do not keep stacking replicas on the other one.

But that is not what the manifest currently says. It only limits skew across the topology domains that participate in the calculation.

To require at least two eligible domains, we add minDomains: 2:

topologySpreadConstraints:
  - maxSkew: 1
    minDomains: 2 # Require at least two eligible domains (workers)
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    nodeAffinityPolicy: Honor
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app: web

When fewer than two eligible domains remain, Kubernetes treats the global minimum Pod count as zero when calculating skew.

Below, we scale the Deployment to five replicas and compare two scenarios.

First, worker2 is cordoned. With only one eligible domain and minDomains: 2, Kubernetes schedules one Pod on worker; the other four remain Pending because placing another Pod would violate maxSkew: 1.

After I uncordon worker2, both domains become eligible again and all five Pods run with a balanced 3/2 distribution.

This is the difference between maxSkew and minDomains:

maxSkew controls balance across eligible domains, while minDomains specifies how many eligible domains must exist before Kubernetes permits further placement. It preserves the intended topology, not application availability by itself.

What comes next?

We now know how to control where new Pods may be scheduled, but maintenance introduces another question: how many running Pods may Kubernetes voluntarily evict at the same time?

Eviction? Maintance? Why do we care?

Read about PodDisruptionBudgets (PDBs) and lets found out together :D