Three RabbitMQ Pitfalls on AKS That Can Also Happen on EKS

Recently, I had an opportunity to perform minor version upgrades across a group of AKS clusters.

As a prerequisite, I added the following settings to a RabbitMQ cluster running with the RabbitMQ Cluster Operator.

• podAntiAffinity
• PodDisruptionBudget (PDB)

The goal was to distribute the three RabbitMQ Pods across different nodes and maintain RabbitMQ cluster availability while the nodes were being upgraded.

It should have been a simple matter of configuring anti-affinity and a PDB, but I managed to run into one trap after another.

1. The PV zone constraints conflicted with anti-affinity, leaving a Pod permanently stuck in Pending
2. After recreating a PVC, the RabbitMQ nodes disagreed about cluster membership
3. Everything looked healthy in Kubernetes, but a RabbitMQ consumer had silently disappeared

I still got stuck even after discussing the problem with Claude Code, so I decided to write this down for future reference.
I am also an AWS Community Builder, so I have included another perspective: how the incidents I actually experienced on AKS can be understood when translated into EKS and EBS terms.
The AKS sections are based on an actual incident response. The EKS sections are observations based on the official AWS documentation; I did not reproduce the same failures on EKS.

I have been working with Kubernetes for about three months. Please keep in mind that this article is written from a beginner's perspective.

Test environment

Item	Configuration
RabbitMQ	4.x
Metadata store	Mnesia, confirmed in later error logs
Deployment method	RabbitMQ Cluster Operator
Kubernetes	AKS
Storage	Azure Disk using managed-premium
RabbitMQ topology	3 replicas
Queues	Quorum queues

Not every RabbitMQ 4.x environment uses Mnesia.
For new deployments starting with RabbitMQ 4.2.0, Khepri is the default metadata store.
In this article, I concluded that this environment was using Mnesia because Mnesia: appeared in the later logs from the actual incident.

The anti-affinity and PDB configuration

I added the following anti-affinity configuration to the RabbitMQ Cluster Operator Custom Resource.

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 3
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: rabbitmq
          topologyKey: kubernetes.io/hostname

This forces Kubernetes not to place two or more Pods with app.kubernetes.io/name: rabbitmq on the same node.
With the RabbitMQ Cluster Operator, app.kubernetes.io/name is set to the name of the RabbitmqCluster resource.
Because the Custom Resource in this environment was named rabbitmq, I used the selector shown above.

I configured the PDB as follows.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-rabbitmq
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: rabbitmq

This limits the number of RabbitMQ Pods that can be unavailable at the same time because of voluntary disruptions such as drain to one.

Trap 1: PV zone imbalance caused an anti-affinity deadlock

Strictly speaking, this was not a deadlock in the formal sense. In this article, I use the term "deadlock" for convenience to describe a state in which multiple placement constraints cannot all be satisfied, making the Pod unschedulable.

What I expected

I expected the Operator to restart the RabbitMQ Pods one at a time after anti-affinity was applied, then place each Pod on a different node.

rabbitmq-server-0 → node-0
rabbitmq-server-1 → node-1
rabbitmq-server-2 → node-2

What actually happened

One of the three Pods remained in Pending, and the rolling update stopped.
When I checked the Events section with kubectl describe pod, I found the following message.

0/3 nodes are available:
1 node(s) didn't match pod anti-affinity rules,
2 node(s) had volume node affinity conflict.

The PV for server-2 was fixed to Zone A, but the only available node in Zone A already had another RabbitMQ Pod on it. The PV could not be attached in another zone, so there was nowhere to schedule the Pod.

Root cause

The PVs used for persistent storage had nodeAffinity rules that restricted the zones in which their disks could be attached.
The following is the key I found in this environment.

topology.disk.csi.azure.com/zone

Two of the three Azure Disks had been created in the same zone.
However, the relevant node pool had only one node in that zone.

From the scheduler's point of view, every node fell into one of the following categories:

• It violated pod anti-affinity
• It violated volume node affinity

Because none of the conditions changed on retry, the Pod remained permanently in Pending. There was no valid placement left.
At this point, I assumed that recreating the PVC would fix the problem quickly. That assumption set up the next trap.

How to identify and avoid it

Before applying anti-affinity, check which PV each PVC uses and what topology constraints are attached to those PVs.

NAMESPACE=<namespace>

for i in 0 1 2; do
  PVC="persistence-rabbitmq-server-$i"

  PV=$(kubectl -n "$NAMESPACE" get pvc "$PVC" \
    -o jsonpath='{.spec.volumeName}')

  AFFINITY=$(kubectl get pv "$PV" \
    -o jsonpath='{range .spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*]}{.key}={.values}{"\n"}{end}')

  echo "server-$i: PVC=$PVC PV=$PV"
  echo "${AFFINITY:-no nodeAffinity}"
  echo
done

kubectl get nodes -L topology.kubernetes.io/zone

The most dangerous case is when multiple PVs are concentrated in the same zone and only one eligible node exists in that zone. In that situation, hard anti-affinity makes the Pods unschedulable unless another eligible node is available.

There are two possible approaches. If keeping the Pods running is more important than strict node-level separation, change requiredDuringScheduling to preferredDuringScheduling.

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/name: rabbitmq
            topologyKey: kubernetes.io/hostname

Kubernetes will place the Pods on different nodes whenever possible, but it can place them on the same node when no alternative exists. The trade-off is that a single node failure could take down multiple Pods at once.

If hard anti-affinity must be preserved, you need to add eligible nodes in the zone where the PVs are concentrated.

How the same issue maps to EKS and EBS

The underlying structure is the same on EKS.

Perspective	AKS in this incident	EKS
Block storage	Azure Disk	Amazon EBS
Failure domain	Availability Zone	Availability Zone
PV placement constraint	Check topology.disk.csi.azure.com/zone	Check topology.kubernetes.io/zone or topology.ebs.csi.aws.com/zone
Possibility of the same failure	Actually occurred	Can occur when the same conditions are present

An EBS volume and the EC2 instance to which it is attached must be in the same Availability Zone.
For example, suppose the EBS volumes for rabbitmq-server-0 and rabbitmq-server-2 are both in ap-northeast-1a, and only one eligible node exists in that zone. Hard anti-affinity would prevent the second RabbitMQ Pod from being scheduled. This is the same problem I encountered on AKS.

For the EBS CSI Driver, WaitForFirstConsumer is recommended for dynamic provisioning.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-wait-for-first-consumer
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

With WaitForFirstConsumer, the EBS volume is created in an appropriate Availability Zone after the scheduler has considered the Pod's placement requirements.

This applies only to the initial dynamic provisioning process.
It does not automatically move an existing EBS volume that is already created and bound to a PVC into another Availability Zone after anti-affinity is added.
Before adding anti-affinity on EKS, I would at least check the following.

kubectl get pv
kubectl get nodes -L topology.kubernetes.io/zone
kubectl get storageclass -o yaml

The Azure Disk topology key was specific to AKS, but the structure of the problem is common to both AKS and EKS.

Trap 2: RabbitMQ nodes disagreed about cluster membership after the PVC was recreated

To resolve the scheduling deadlock from Trap 1, I deleted the PVC for the blocked server-2 Pod and allowed it to be recreated in another zone.

kubectl delete pvc persistence-rabbitmq-server-2

I did not expect this single action to trigger a chain of crashes across the entire cluster.
As a result, server-2 started with an empty data directory and none of its previous local data.

Reviewing the actual logs in chronological order

A careful review of the logs showed that the information confirmed during the first crash was different from what was confirmed during the later crashes.

1. server-2 was recreated with an empty data directory

After I deleted the PVC for server-2, a new PVC and disk were created.
server-2 then started without its previous local data.

2. server-1 crashed during a normal rolling restart

The Operator then restarted server-1 as the next target in the rolling update.
The logs showed that the startup process for the Mnesia application was killed, but they did not include enough information to determine the exact cause.

3. inconsistent_cluster appeared later on server-0

When server-0 crashed afterward, it produced the following error.

exception exit: {{inconsistent_cluster,
  "Mnesia: node 'rabbit@rabbitmq-server-0' thinks it's clustered with node
   'rabbit@rabbitmq-server-2', but 'rabbit@rabbitmq-server-2' disagrees"},
  {rabbit,start,[normal,[]]}}

This was the first point at which the Mnesia cluster membership inconsistency was confirmed.

Why the later inconsistent_cluster error occurred

RabbitMQ nodes store cluster identity and membership information in their local data directories.
The later failure appears to have been caused by the following state.

server-0 / server-1
  └─ Still consider server-2 to be an existing cluster member

Recreated server-2
  └─ Started from an empty data directory
  └─ Does not have the previous cluster information

Result
  └─ The two sides disagree about the same node name

The official RabbitMQ documentation describes the same kind of error between an empty or reset node and another node that still considers it to be a previous cluster member.

Node '[e-mail]' thinks it's clustered with node
'[e-mail]', but '[e-mail]' disagrees

The later log from server-0 matched this pattern.

What I should have checked first

Before deleting the PVC and recreating the node, I should have checked the state of the cluster and the quorum queues.

kubectl exec -n <namespace> rabbitmq-server-0 -- \
  rabbitmq-diagnostics cluster_status

kubectl exec -n <namespace> rabbitmq-server-2 -- \
  rabbitmq-queues check_if_node_is_quorum_critical

Having three Pods with two of them in Running does not necessarily mean it is safe to remove the third. It is better to inspect the actual member placement of each quorum queue.

If the target node is being removed permanently, remove it formally from the cluster using a healthy node.

NODE_TO_REMOVE='[e-mail]-nodes.<namespace>'

kubectl exec -n <namespace> rabbitmq-server-0 -- \
  rabbitmqctl forget_cluster_node "$NODE_TO_REMOVE"

However, forget_cluster_node is not a universal solution. Removing a node also removes the quorum queue members that were hosted on that node, so it must be executed only when the affected queues can still maintain quorum.

This issue is not specific to Azure Disk. The same problem can occur on EKS with EBS if a RabbitMQ node is recreated from an empty data directory.

Trap 3: A consumer silently disappeared

After applying anti-affinity and the PDB, I proceeded with the AKS minor version upgrade.
In this environment, the following operations were performed on each node.

cordon
  ↓
drain
  ↓
reimage

What I noticed several days later

The upgrade itself completed without errors, and I was honestly relieved.

• All nodes were Ready on the new version
• The RabbitMQ Pods were 3/3 Running
• The application Pods were also Running
• The HTTP health checks returned 200 OK

Several days later, however, someone reported that data integration between two services had stopped.
When I checked the RabbitMQ queues, I found the following state.

$ rabbitmqctl list_queues name messages consumers

some.busy.queue      617224    0
some.other.queue     0         182

Messages kept accumulating in some.busy.queue, but its consumer count was 0.
The Pod for the affected application was still Running, and its HTTP health check was healthy.
In other words, everything looked normal from the outside, while only the functionality that depended on RabbitMQ had stopped.

Why I did not notice it

When a RabbitMQ node restarts, every client Connection, Channel, and Consumer connected to that node is lost.

Some official RabbitMQ client libraries, including the Java and .NET clients, provide automatic recovery from reconnecting the Connection through re-registering Consumers. Whether recovery actually succeeds, however, depends on the client library and version, whether automatic recovery is enabled, and how the application implements its own connection management.

In this incident, only some applications failed either to reconnect to RabbitMQ or to re-register their consumers.
The failure was also invisible to the Kubernetes probes. The Pod's liveness and readiness probes checked only whether an HTTP endpoint was reachable. They did not monitor the RabbitMQ connection or the consumer registration state. As a result, the application kept returning HTTP 200 even though no consumer was registered. Kubernetes considered the Pod healthy, while the actual business processing had stopped.

The application logs contained RabbitMQ connection errors, but Kubernetes did not restart the Pod because the Pod itself had not stopped.
When I checked the timestamps of the queued messages, they had started accumulating immediately after the RabbitMQ restart.
Based on that timing, I concluded that the cause was a failed reconnection or consumer re-registration after RabbitMQ restarted.

Detection and recovery

After this incident, I changed the operational procedure so that the consumer counts for all queues are checked after any RabbitMQ restart.
The following is a simple check that excludes DLX queues and detects queues that have pending messages but no consumers.

rabbitmqctl list_queues name messages consumers \
  | awk -F'\t' \
    '$1 !~ /dlx/ && $2+0>0 && $3+0==0 {
      print "no consumers with pending messages: "$1"  pending="$2
    }'

A consumer count of 0 is not always an error. Some queues are consumed only during batch jobs and normally have no active consumers. In practice, I started recording a baseline of consumer counts before maintenance, then comparing it after every RabbitMQ Pod restart to identify queues whose consumer counts had decreased.

When a consumer had not recovered, rolling the affected application's Deployment usually restored it.

kubectl rollout restart deployment/<affected-app>

This is a recovery action, not a root-cause fix. Looking back, I should have made the RabbitMQ connection state, consumer registration state, and last processing time externally observable, separately from the HTTP liveness check.

How the same issue maps to EKS node updates

During a default EKS Managed Node Group update, the process is roughly as follows.

1. Create new nodes
2. Select an old node to update
3. Drain the Pods from the old node
4. Cordon the node
5. Terminate the old node
6. Repeat for all target nodes

If a PDB does not allow a Pod eviction, the update can fail with PodEvictionFailure.
However, even when the PDB works correctly and moves the RabbitMQ Pods safely one at a time, the client Connections attached to those Pods are still disconnected. A PDB controls how many Pods can be disrupted at once; it does not guarantee that consumers will recover.
For that reason, after an EKS Managed Node Group update, it is safer to check not only whether the RabbitMQ Pods are Running, but also the consumer counts and message backlog.

Conclusion

Even having the Kubestronaut title does not help much without hands-on experience.
Kubernetes is still difficult, even with AI support.