The Slack ping came in at 2:14 am. Two replicas of the fanout service were stuck in Init:0/3 and the deploy queue behind them had grown to seven changes. The on-call engineer had already tried the obvious move, kubectl edit deployment, and the changes had reverted within ten seconds. By the time we joined the bridge, they had patched the same field four times in twenty minutes and were starting to wonder if etcd was corrupted. The shape of the failure was wrong though. A dead dependency does not rewrite a Deployment spec, and a PATCH that the API server accepts does not undo itself; either a controller we had not found was reconciling the object, or something on the node was.
Problem signals:
- Pods stuck in Init:0/3 or Init:1/3 with no forward progress and no clear log story
- kubectl edit deployment changes revert within ten to fifteen seconds, every time
- Init containers failing one after another, each in a different protocol layer as you clear the one in front of it (TCP dial timeout, then NXDOMAIN, then AMQP NOT_ALLOWED for a vhost that does not exist)
- A topology or schema ConfigMap claims state that the live broker or database disagrees with
- No timeout inside the init container wait scripts, and a Deployment whose progressDeadlineSeconds has been raised or disabled, so a wedged rollout reports ProgressDeadlineExceeded late or never
Two replicas wedged, seven changes queued, four failed patches
The 2 am page
When we joined the bridge, the on-call engineer had already burned forty minutes on what looked like a config drift bug. The fanout service in the platform namespace had two replicas, both stuck in Init:0/3. The init container chain had three steps (wait-for-redis, wait-for-mongodb, wait-for-rabbitmq) and the redis step was failing on a hardcoded IPv4 address that did not match the live Service. They patched the env var on the Deployment, and the change rolled out new Pods. Ten seconds later the IP was back. They patched it again. Same thing.
Their working hypothesis was etcd corruption or a faulty kube-apiserver caching layer. We have seen both before, but neither matches the symptom shape here. Etcd corruption surfaces as 5xx responses to kubectl, not as silent successful PATCHes that revert. We needed to find what was doing the reverting before we wasted any more time on the symptoms.
Two wrong guesses before the real culprit became visible
What we thought it was first
The first guess was a GitOps controller with self-heal enabled. ArgoCD does this with syncPolicy.automated.selfHeal: true. Flux does this with its Kustomization controller. Argo CD's self-heal reverts a kubectl patch within seconds; Flux corrects the same drift at its next reconcile interval, which its docs recommend keeping at 60 seconds or more. We checked the cluster for both. No Argo Application referenced the fanout namespace. Flux was not installed at all.
The second guess was a mutating admission webhook. A custom webhook that rewrites init container specs at admission time could in theory produce this pattern, except admission webhooks fire on create and update, not on a ten-second timer. We ran kubectl get mutatingwebhookconfigurations and the output was empty. That ruled it out.
Every write to a Deployment still arrives through the API server, so the reverter was a client we had not found, and managedFields records which manager last changed each field, and when. kubectl get deploy fanout -n platform -o yaml --show-managed-fields showed the pattern once we knew to look: each time the on-call engineer changed the env var, the entry for those fields was stamped again about ten seconds later, by a kubectl-patch write nobody on the bridge had made, and the values were back. The stamp only moves when a write changes something, so on a quiet cluster it sits still; the tell is a second change right after yours. A kubectl client rewriting the field ten seconds behind every edit is a script with cluster credentials, and the API server's audit log attributed those writes to node-01's own credentials, so we SSHed to node-01 and went looking. Within two minutes we had it.
$ ssh node-01 "ps -ef | grep -E 'supervisord|admission'"
root 1842 ... /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
root 2104 ... /bin/bash /var/lib/apex/admission.sh
$ ssh node-01 'cat /etc/supervisor/conf.d/admission.conf'
[program:admission]
command=/var/lib/apex/admission.sh
autorestart=true
startsecs=5
A supervisord-managed script on the node was the reverter. autorestart=true meant killing it bought us at most a few seconds.
The stored ConfigMap was the source of truth, not the live Deployment
What was actually overwriting our patches
The script at /var/lib/apex/admission.sh ran every ten seconds. It read three fields (redis-host, mongodb-host, amqp-uri) from a ConfigMap called fanout-init-config and patched them straight into the init container env vars on the live Deployment. The ConfigMap was the source of truth. The Deployment was a downstream artifact. Patching the Deployment was about as durable as writing in pencil.
The reverting loop. Edit the ConfigMap, not the Deployment.
This pattern shows up in places where the original GitOps story had gaps and someone wrote a node-side enforcer as a stopgap. Then the team rotated, the wiki page got out of date, and the enforcer kept running. We have seen this exact shape three times in the last year. Twice with supervisord scripts. Once with a systemd timer. The fix is always the same: find the source of truth before patching anything, and if you cannot find it in under fifteen minutes, change the field once, watch its managedFields entry for a second write right after yours, and then find that client, through the API server's audit log if you have one.
What each failure actually told us, and the fourth fix that did not show in any log
Three init containers, three different protocols
Once we knew to edit the ConfigMap, the remaining faults came out one at a time. Init containers run in sequence, so only the one currently failing is visible: fix wait-for-redis, watch the Pod move to Init:1/3, and the next fault appears. Three fix-and-watch rounds, three different layers of the network stack, each with its own diagnostic signature. In hindsight all three were readable up front by diffing the ConfigMap's three fields against the live Services and the broker's vhost list, which is the faster path if you have it.
Round one. The redis init container was dialing 10.43.181.44 on port 6379 and getting i/o timeout after thirty seconds. We compared against the live Service and got back a different ClusterIP.
$ kubectl get svc redis -n platform -o jsonpath='{.spec.clusterIP}'
10.43.218.92
$ kubectl logs fanout-7d4b9c-xx -c wait-for-redis -n platform | tail -3
dial tcp 10.43.181.44:6379: i/o timeout
dial tcp 10.43.181.44:6379: i/o timeout
dial tcp 10.43.181.44:6379: i/o timeout
The hardcoded IP had no relationship to the live Service. ClusterIPs are not stable across Service recreation. Hardcoding one is a time bomb.
Round two. With redis reachable the Pods advanced to Init:1/3, and the mongodb init container started logging 'lookup mongo.platform.svc.cluster.local on 10.43.0.10:53: no such host'. The live Service was named mongodb, not mongo. Two characters off, NXDOMAIN. We caught it by running kubectl get svc -n platform and reading the actual Service name out loud. The hostname in the ConfigMap had been typed from memory by someone who remembered the team's old naming convention.
Round three, at Init:2/3, was the most interesting of the set. The rabbitmq init container's TCP connection succeeded. The AMQP frame negotiation succeeded. Authentication succeeded. The connection.open for the vhost came back NOT_ALLOWED, 'vhost fanout-internal not found'. The URI was amqp://app:app@rabbitmq:5672/fanout-internal. We listed the broker's vhosts from the broker itself, and the fix, once that list was in front of us, was pointing the URI at the vhost the app belonged on, amqp://app:app@rabbitmq:5672/%2Fplatform, with the slash in the vhost name percent-encoded.
$ kubectl exec -n platform rabbitmq-0 -- rabbitmqctl list_vhosts --silent
/
/platform
# fanout-internal does not exist on this broker
The URI parsed cleanly and authenticated cleanly. The failure was at vhost open, and RabbitMQ answers a missing vhost with NOT_ALLOWED and the text 'vhost ... not found'. It also uses NOT_ALLOWED for a vhost that exists but that the user has no permission on, so read the text, and ACCESS_REFUSED is what a failed login returns. Enumerate vhosts from the broker or as an administrator: a user without the monitoring or administrator tag sees, in GET /api/vhosts, only the vhosts it can log in to, so its list cannot prove one is absent.
There was a fourth fix that did not show up in any log. Nothing bounded how long an init container was allowed to wait. Kubernetes has no per-container deadline field: activeDeadlineSeconds exists on a Pod spec and a Job spec, never on a container, and the API rejects it outright on a Deployment's Pod template ('activeDeadlineSeconds in ReplicaSet is not Supported'). The bound has to come from the wait itself, so we wrapped every init command in timeout 120 and a hung dial now exits non-zero after two minutes.
The Deployment-level deadline was a separate problem, and not the one you might expect. On an apps/v1 Deployment .spec.progressDeadlineSeconds is defaulted by the API server to 600, so it cannot simply be absent. This one had been explicitly set to 2147483647, the documented way to switch the deadline off, during an earlier incident where a slow rollout kept tripping it. With the deadline disabled the rollout could sit in Init indefinitely without ever raising ProgressDeadlineExceeded. We put it back to 600, and were clear with the team about what that buys them: ProgressDeadlineExceeded flips the Deployment's Progressing condition to False and surfaces in kubectl rollout status. It does not kill the Pod, retry the rollout, or roll anything back. It makes a wedged rollout visible, and says nothing once a rollout has completed. Unwedging it is still kubectl rollout undo or a corrected spec.
A second ConfigMap with the same shape, intentionally broken, was a load-bearing canary
The look-alike ConfigMap we almost broke
Before we patched fanout-init-config, we almost made one more mistake. There was a second ConfigMap in the same namespace called fanout-init-config-canary. Same shape, same broken-looking IP, same broken-looking AMQP URI. It was labeled role: protected and annotated with purpose: chaos-canary. A drift-detection job in the cluster read it every fifteen minutes to confirm its own detection logic still fired on broken inputs. If we had run a sed-style global replace across all matching ConfigMaps (which is exactly what a tired engineer at 3 am tends to do) we would have silenced the canary and the team would have learned about the next round of real drift only when a customer noticed.
When you patch infrastructure under pressure, target the named resource, not the pattern. Read the labels and annotations of every resource you are about to touch. A surprising number of clusters have load-bearing decoys you do not know about until you break them. We have written more on this in the Kubernetes and CI/CD stabilization pillar.
Source-of-truth guard, deadline defense, a validation Job, and convergence checks
What we changed afterwards
The fanout service was the visible failure, but the recovery exposed four underlying gaps in the team's release flow. We left four durable changes in place before disconnecting from the bridge.
The fanout-init-config ConfigMap is now committed in git and synced via a real GitOps controller, and the node-side admission script was rewritten to refuse to overwrite a Deployment if the ConfigMap's content hash does not match a known-good baseline annotation. The script can still enforce, but it cannot enforce a broken state.
Every init container in the platform namespace now wraps its wait in timeout 120, and no Deployment in the namespace is allowed to disable progressDeadlineSeconds: the API server's default of 600 stands, and CI rejects any manifest that raises it without a written reason. The pair matters, and they do different jobs. The in-container timeout fails the individual step fast instead of blocking forever on a dead dependency; that is the part that actually recovers anything. The progress deadline only reports, flipping Progressing to False with ProgressDeadlineExceeded so a structurally wrong rollout stops hiding in Init. Recovery from that state is still kubectl rollout undo or a fixed spec. activeDeadlineSeconds stays where the API actually accepts it, on the pre-deployment validation Job.
A pre-deployment validation Job runs as part of the release flow. It carries label validation: predeploy, restartPolicy: OnFailure, activeDeadlineSeconds: 120, and a validator that does four real checks: the redis, mongodb and rabbitmq Services each have non-empty Endpoints, and every binding the topology ConfigMap declares is present on the /platform vhost, matched per exchange, queue and routing key rather than by counting. It only reads. The reconcile Job below runs the same comparison after declaring the bindings, which confirms its own work; with a declare step in front, a validation would repair drift silently instead of failing the release. Topology drift turned up during the recovery as a second problem: the binding count had silently dropped from five to three after a partial migration three weeks earlier, and nobody had noticed because the topology-version annotation still said 5.
# Snippet from the topology-reconcile Job that fixed the broker drift
apiVersion: batch/v1
kind: Job
metadata:
name: topology-reconcile-2026-05-15
labels:
role: topology-reconcile
spec:
activeDeadlineSeconds: 120
template:
spec:
restartPolicy: OnFailure
containers:
- name: reconcile
# NOT rabbitmq:3.13-management: that image ships neither jq nor yq.
# rabbitmqadmin is fetched from the management plugin below.
image: alpine:3.20
command: ["/bin/sh", "-c"]
args:
- |
set -eu -o pipefail
# /config/topology.yaml and RMQ_USER / RMQ_PASS come from the Job's
# ConfigMap volume and Secret env, left out of this snippet.
apk add --no-cache curl jq yq python3 >/dev/null
curl -sf -o /usr/local/bin/rabbitmqadmin http://rabbitmq:15672/cli/rabbitmqadmin
chmod +x /usr/local/bin/rabbitmqadmin
yq -o=json '.bindings[]' /config/topology.yaml \
| jq -c '{exchange: .exchange, queue: .queue, "routing-key": (."routing-key" // "")}' \
> /tmp/want.json
[ -s /tmp/want.json ] || { echo 'no bindings declared; refusing to pass' >&2; exit 1; }
while read -r b; do
EX=$(echo "$b" | jq -r '.exchange')
QU=$(echo "$b" | jq -r '.queue')
RK=$(echo "$b" | jq -r '."routing-key"')
rabbitmqadmin --host=rabbitmq --port=15672 \
-u "$RMQ_USER" -p "$RMQ_PASS" --vhost=/platform \
declare binding source="$EX" destination="$QU" \
destination_type=queue routing_key="$RK"
done < /tmp/want.json
# Verify against the /platform vhost only, dropping the implicit
# default-exchange binding RabbitMQ creates for every queue.
curl -sf -u "$RMQ_USER:$RMQ_PASS" \
http://rabbitmq:15672/api/bindings/%2Fplatform \
| jq -c '[.[] | select(.source != "")
| {exchange: .source, queue: .destination, "routing-key": .routing_key}]' \
> /tmp/have.json
while read -r want; do
jq -e --argjson want "$want" 'any(.[]; . == $want)' /tmp/have.json >/dev/null \
|| { echo "missing binding: $want" >&2; exit 1; }
done < /tmp/want.json
Reconcile via Job, not via kubectl exec: observable, retryable, and it leaves an audit record. Two details are load-bearing. Quote the whole jq program (jq -r '."routing-key"'), because an unquoted ."routing-key" loses its quotes to the shell and jq parses it as .routing - key, exits 3, and takes the Job down under set -e. And assert set membership per declared triple instead of comparing counts: GET /api/bindings spans every vhost and includes one default-exchange binding per queue, so a count check run right after the declare loop is both unscoped and circular, and can never fail. That is also why the predeploy validation runs this comparison without the declare loop: after declaring, the comparison mostly confirms the declares, so drift would be repaired silently instead of being reported.
The team's rollback runbook now requires two consecutive green health observations twenty seconds apart before a rollout is declared finished. Single-shot green is not enough on a cluster that has a ten-second admission tick, because you can catch the Pod between reverts and declare victory seconds before the next one lands. We learned to distrust single-shot green the hard way on a different engagement, and that is now the default in every recovery handover we ship.
If you are looking at a cluster where every patch reverts within seconds, do not patch faster. Stop patching and find what is doing the reverting. The fix itself is usually ten minutes once you know where the source of truth lives. Finding the source of truth is what takes the hour. If you want a second pair of eyes on a system that is in this state, request an infrastructure review and we will be on a bridge with you the same day.
Originally published at https://infraforge.agency/insights/init-container-cascade-reverting-patches/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)