The Slack ping came in at 2:14 am. Two replicas of the fanout service were stuck in Init:0/3 and the deploy queue behind them had grown to seven changes. The on-call engineer had already tried the obvious move, kubectl edit deployment, and the changes had reverted within ten seconds. By the time we joined the bridge, they had patched the same field four times in twenty minutes and were starting to wonder if etcd was corrupted. The shape of the failure was wrong though. A dead dependency does not rewrite a Deployment spec, and a PATCH that the API server accepts does not undo itself; either a controller we had not found was reconciling the object, or something on the node was.
Problem signals:
- Pods stuck in Init:0/3 or Init:1/3 with no forward progress and no clear log story
- kubectl edit deployment changes revert within ten to fifteen seconds, every time
- Init containers failing one after another, each in a different protocol layer as you clear the one in front of it (TCP dial timeout, then NXDOMAIN, then AMQP ACCESS_REFUSED)
- A topology or schema ConfigMap claims state that the live broker or database disagrees with
- No timeout inside the init container wait scripts, and a Deployment whose progressDeadlineSeconds has been raised or disabled, so a wedged rollout never even reports ProgressDeadlineExceeded
Two replicas wedged, seven changes queued, four failed patches
The 2 am page
When we joined the bridge, the on-call engineer had already burned forty minutes on what looked like a config drift bug. The fanout service in the platform namespace had two replicas, both stuck in Init:0/3. The init container chain had three steps (wait-for-redis, wait-for-mongodb, wait-for-rabbitmq) and the redis step was failing on a hardcoded IPv4 address that did not match the live Service. They patched the env var on the Deployment. The init container restarted. Ten seconds later the IP was back. They patched it again. Same thing.
Their working hypothesis was etcd corruption or a faulty kube-apiserver caching layer. We have seen both before, but neither matches the symptom shape here. Etcd corruption surfaces as 5xx responses to kubectl, not as silent successful PATCHes that revert. We needed to find what was doing the reverting before we wasted any more time on the symptoms.
Two wrong guesses before the real culprit became visible
What we thought it was first
The first guess was a GitOps controller with self-heal enabled. ArgoCD does this with syncPolicy.automated.selfHeal: true. Flux does this with its Kustomization controller. Both will revert a kubectl patch within seconds if the live spec drifts from the source of truth in git. We checked the cluster for both. No Argo Application referenced the fanout namespace. Flux was not installed at all.
The second guess was a mutating admission webhook. A custom webhook that rewrites init container specs at admission time could in theory produce this pattern, except admission webhooks fire on create and update, not on a ten-second timer. We ran kubectl get mutatingwebhookconfigurations and the output was empty. That ruled it out.
The reverting was not coming from inside the cluster. It had to be coming from the node itself. We SSHed to the node where one of the fanout pods was scheduled and went looking. Within two minutes we had it.
$ ssh node-01 'ps -ef | grep admission'
root 1842 ... /usr/bin/supervisord -c /etc/supervisor/conf.d/admission.conf
root 2104 ... /bin/bash /var/lib/apex/admission.sh
$ ssh node-01 'cat /etc/supervisor/conf.d/admission.conf'
[program:admission]
command=/var/lib/apex/admission.sh
autorestart=true
startsecs=5
A supervisord-managed script on the node was the reverter. autorestart=true meant killing it bought us at most a few seconds.
The stored ConfigMap was the source of truth, not the live Deployment
What was actually overwriting our patches
The script at /var/lib/apex/admission.sh ran every ten seconds. It read three fields (redis-host, mongodb-host, amqp-uri) from a ConfigMap called fanout-init-config and patched them straight into the init container env vars on the live Deployment. The ConfigMap was the source of truth. The Deployment was a downstream artifact. Patching the Deployment was about as durable as writing in pencil.
The reverting loop. Edit the ConfigMap, not the Deployment.
This pattern shows up in places where the original GitOps story had gaps and someone wrote a node-side enforcer as a stopgap. Then the team rotated, the wiki page got out of date, and the enforcer kept running. We have seen this exact shape three times in the last year. Twice with supervisord scripts. Once with a systemd timer. The fix is always the same: find the source of truth before patching anything, and if you cannot find it in under fifteen minutes, stop and look on the nodes.
What each failure actually told us, and the fourth fix that did not show in any log
Three init containers, three different protocols
Once we knew to edit the ConfigMap, the remaining faults came out one at a time. Init containers run in sequence, so only the one currently failing is visible: fix wait-for-redis, watch the Pod move to Init:1/3, and the next fault appears. Three fix-and-watch rounds, three different layers of the network stack, each with its own diagnostic signature. In hindsight all three were readable up front by diffing the ConfigMap's three fields against the live Services and the broker's vhost list, which is the faster path if you have it.
Round one. The redis init container was dialing 10.43.181.44 on port 6379 and getting i/o timeout after thirty seconds. We compared against the live Service and got back a different ClusterIP.
$ kubectl get svc redis -n platform -o jsonpath='{.spec.clusterIP}'
10.43.218.92
$ kubectl logs fanout-7d4b9c-xx -c wait-for-redis -n platform | tail -3
dial tcp 10.43.181.44:6379: i/o timeout
dial tcp 10.43.181.44:6379: i/o timeout
dial tcp 10.43.181.44:6379: i/o timeout
The hardcoded IP had no relationship to the live Service. ClusterIPs are not stable across Service recreation. Hardcoding one is a time bomb.
Round two. With redis reachable the Pods advanced to Init:1/3, and the mongodb init container started logging 'lookup mongo.platform.svc.cluster.local on 10.43.0.10:53: no such host'. The live Service was named mongodb, not mongo. One character off, NXDOMAIN. We caught it by running kubectl get svc -n platform and reading the actual Service name out loud. The hostname in the ConfigMap had been typed from memory by someone who remembered the team's old naming convention.
Round three, at Init:2/3, was the most interesting of the set. The rabbitmq init container's TCP connection succeeded. The AMQP frame negotiation succeeded. Authentication succeeded. The vhost open returned ACCESS_REFUSED. The URI was amqp://app:app@rabbitmq:5672/fanout-internal. We port-forwarded to the management API and listed valid vhosts.
$ kubectl port-forward -n platform svc/rabbitmq 15672:15672 &
$ curl -s -u app:app http://localhost:15672/api/vhosts | jq -r '.[].name'
/
/platform
# fanout-internal does not exist on this broker
The URI parsed cleanly and authenticated cleanly. The failure was at vhost open. Always enumerate vhosts before assuming auth or credentials.
There was a fourth fix that did not show up in any log. Nothing bounded how long an init container was allowed to wait. Kubernetes has no per-container deadline field: activeDeadlineSeconds exists on a Pod spec and a Job spec, never on a container, and putting it on a Deployment's Pod template is worse than useless because it kills healthy pods once they have been active that long. The bound has to come from the wait itself, so we wrapped every init command in timeout 120 and a hung dial now exits non-zero after two minutes.
The Deployment-level deadline was a separate problem, and not the one you might expect. On an apps/v1 Deployment .spec.progressDeadlineSeconds is defaulted by the API server to 600, so it cannot simply be absent; only the long-dead extensions/v1beta1 left it unset. This one had been explicitly set to 2147483647, the documented way to switch the deadline off, during an earlier incident where a slow rollout kept tripping it. With the deadline disabled the rollout could sit in Init indefinitely without ever raising a condition. We put it back to 600, and were clear with the team about what that buys them: ProgressDeadlineExceeded flips the Deployment's Progressing condition to False and surfaces in kubectl rollout status. It does not kill the Pod, retry the rollout, or roll anything back. It makes the wedge visible. Unwedging it is still kubectl rollout undo or a corrected spec.
A second ConfigMap with the same shape, intentionally broken, was a load-bearing canary
The look-alike ConfigMap we almost broke
Before we patched fanout-init-config, we almost made one more mistake. There was a second ConfigMap in the same namespace called fanout-init-config-canary. Same shape, same broken-looking IP, same broken-looking AMQP URI. It was labeled role: protected and annotated with purpose: chaos-canary. A drift-detection job in the cluster read it every fifteen minutes to confirm its own detection logic still fired on broken inputs. If we had run a sed-style global replace across all matching ConfigMaps (which is exactly what a tired engineer at 3 am tends to do) we would have silenced the canary and the team would have learned about the next round of real drift only when a customer noticed.
When you patch infrastructure under pressure, target the named resource, not the pattern. Read the labels and annotations of every resource you are about to touch. A surprising number of clusters have load-bearing decoys you do not know about until you break them. We have written more on this in the Kubernetes and CI/CD stabilization pillar.
Source-of-truth guard, deadline defense, a validation Job, and convergence checks
What we changed afterwards
The fanout service was the visible failure, but the recovery exposed four underlying gaps in the team's release flow. We left four durable changes in place before disconnecting from the bridge.
The fanout-init-config ConfigMap is now committed in git and synced via a real GitOps controller, and the node-side admission script was rewritten to refuse to overwrite a Deployment if the ConfigMap's content hash does not match a known-good baseline annotation. The script can still enforce, but it cannot enforce a broken state.
Every init container in the platform namespace now wraps its wait in timeout 120, and no Deployment in the namespace is allowed to disable progressDeadlineSeconds: the API server's default of 600 stands, and CI rejects any manifest that raises it without a written reason. The pair matters, and they do different jobs. The in-container timeout fails the individual step fast instead of blocking forever on a dead dependency; that is the part that actually recovers anything. The progress deadline only reports, flipping Progressing to False with ProgressDeadlineExceeded so a structurally wrong rollout stops hiding in Init. Recovery from that state is still kubectl rollout undo or a fixed spec. activeDeadlineSeconds stays where the API actually accepts it, on the pre-deployment validation Job.
A pre-deployment validation Job runs as part of the release flow. It carries label validation: predeploy, restartPolicy: OnFailure, activeDeadlineSeconds: 120, and a validator that does three real checks: redis, mongodb, and rabbitmq Services each have non-empty Endpoints, AND every binding the topology ConfigMap declares is present on the /platform vhost, matched per exchange, queue and routing key rather than by counting. Topology drift was the other half of this incident; the binding count had silently dropped from five to three after a partial migration three weeks earlier, and nobody had noticed because the topology-version annotation still said 5.
# Snippet from the topology-reconcile Job that fixed the broker drift
apiVersion: batch/v1
kind: Job
metadata:
name: topology-reconcile-2026-05-15
labels:
validation: predeploy
spec:
activeDeadlineSeconds: 120
template:
spec:
restartPolicy: OnFailure
containers:
- name: reconcile
# NOT rabbitmq:3.13-management: that image ships neither jq nor yq, and
# rabbitmqadmin is a Python script served by the management plugin itself.
image: alpine:3.20
command: ["/bin/sh", "-c"]
args:
- |
set -eu
apk add --no-cache curl jq yq python3 >/dev/null
curl -sf -o /usr/local/bin/rabbitmqadmin http://rabbitmq:15672/cli/rabbitmqadmin
chmod +x /usr/local/bin/rabbitmqadmin
yq -o=json '.bindings[]' /config/topology.yaml \
| jq -c '{exchange: .exchange, queue: .queue, "routing-key": ."routing-key"}' \
> /tmp/want.json
while read -r b; do
EX=$(echo "$b" | jq -r '.exchange')
QU=$(echo "$b" | jq -r '."routing-key"' >/dev/null; echo "$b" | jq -r '.queue')
RK=$(echo "$b" | jq -r '."routing-key"')
rabbitmqadmin --host=rabbitmq --port=15672 \
-u "$RMQ_USER" -p "$RMQ_PASS" --vhost=/platform \
declare binding source="$EX" destination="$QU" \
destination_type=queue routing_key="$RK"
done < /tmp/want.json
# Verify against the /platform vhost only, dropping the implicit
# default-exchange binding RabbitMQ creates for every queue.
curl -sf -u "$RMQ_USER:$RMQ_PASS" \
http://rabbitmq:15672/api/bindings/%2Fplatform \
| jq -c '[.[] | select(.source != "")
| {exchange: .source, queue: .destination, "routing-key": .routing_key}]' \
> /tmp/have.json
while read -r want; do
jq -e --argjson want "$want" 'any(.[]; . == $want)' /tmp/have.json >/dev/null \
|| { echo "missing binding: $want" >&2; exit 1; }
done < /tmp/want.json
Reconcile via Job, not via kubectl exec: observable, retryable, and it leaves an audit record. Two details are load-bearing. Quote the whole jq program (jq -r '."routing-key"'), because an unquoted ."routing-key" loses its quotes to the shell and jq parses it as .routing - key, exits 3, and takes the Job down under set -e. And assert set membership per declared triple instead of comparing counts: GET /api/bindings spans every vhost and includes one default-exchange binding per queue, so a count check run right after the declare loop is both unscoped and circular, and can never fail.
The team's rollback runbook now requires two consecutive green health observations twenty seconds apart before a rollout is declared finished. Single-shot green is not enough on a cluster that has a ten-second admission tick, because you can catch the Pod between reverts and declare victory ninety seconds before the next one lands. We learned to distrust single-shot green the hard way on a different engagement, and that is now the default in every recovery handover we ship.
If you are looking at a cluster where every patch reverts within seconds, do not patch faster. Stop patching and find what is doing the reverting. The fix itself is usually ten minutes once you know where the source of truth lives. Finding the source of truth is what takes the hour. If you want a second pair of eyes on a system that is in this state, request an infrastructure review and we will be on a bridge with you the same day.
Originally published at https://infraforge.agency/insights/init-container-cascade-reverting-patches/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)