The on-call team had been chasing a 30% 401 rate on profile-service for two hours when we got pulled in. As far as their dashboards showed, it was only profile-service, only some pods, only authenticated requests. Two of those three were wrong. The shape of that number is what had thrown them off: a 30% failure rate on a 3-pod deployment looks exactly like one pod out of three running a different config, so that is where they had been digging. It was not a pod problem. All three profile-service pods were identical, and 30% was simply the share of traffic that carried a token at all. It was not confined to profile-service either: like-service had been rejecting every token-carrying request for the same week, and nobody was looking at it because the 401-rate alert had been wired up on profile-service and not on like-service. What was underneath both was a week-old JWT key rotation hotfix that had landed in the live cluster, never made it to Git, and ArgoCD auto-sync had been disabled across three applications and quietly left off. By the time we opened a terminal there were four versions of the same ConfigMap floating around: one in Git, three in three namespaces, none of them in agreement.
Problem signals:
- A service is returning 401s on a stable fraction of requests, and the fraction tracks the share of traffic that carries a token rather than any pod ratio
- ArgoCD shows applications as OutOfSync but auto-sync is disabled and nobody remembers turning it off
- kubectl diff against the rendered Helm or Kustomize output shows changes nobody can attribute to a recent PR
- Multiple namespaces have a propagated copy of the same ConfigMap and the copies disagree
- A recent incident postmortem mentions a manual kubectl edit or kubectl patch that was never followed by a Git commit
The first 20 minutes: mapping how far the drift had spread
Four ConfigMaps, four different values
The initial theory from the on-call lead was that a pod had missed the last restart and was still holding the pre-rotation JWT public key. Reasonable theory. It was wrong, but only because it was incomplete.
We ran the obvious diff first. Pull the ConfigMap from each of the three namespaces, pull the manifest from the Git repo at HEAD, compare. What we expected to find was two values: a correct one in the cluster and a stale one in Git, or the reverse. What we actually found was four.
# auth-service namespace
$ kubectl -n auth get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM} {.data.JWT_PUBLIC_KEY_ID}'
RS256 key-2024-11-rot
# like-service namespace (propagated copy)
$ kubectl -n like get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM} {.data.JWT_PUBLIC_KEY_ID}'
RS256 key-2024-09
# profile-service namespace (propagated copy)
$ kubectl -n profile get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM} {.data.JWT_PUBLIC_KEY_ID}'
HS256 key-2024-09
# Git, main branch
$ grep -E 'JWT_(ALGORITHM|PUBLIC_KEY_ID)' deploy/*/auth-config.yaml
deploy/auth/auth-config.yaml: JWT_ALGORITHM: HS256
deploy/auth/auth-config.yaml: JWT_PUBLIC_KEY_ID: key-2024-09
# (and the same stale pair in like and profile manifests)
What the diff actually showed. Four states of the same ConfigMap.
The story behind the four states reconstructed quickly from the previous week's incident channel. During the rotation, an SRE had patched auth-service's ConfigMap directly with the new RS256 key. They then walked the change into the like-service namespace and got the algorithm right but typo'd the key ID, leaving the old one. They ran out of focus before reaching profile-service, intended to come back to it, and did not. ArgoCD auto-sync had been disabled across all three applications during the incident as a guardrail and never re-enabled. These applications run automated sync with selfHeal: true, so that toggle is the only reason the cluster state survived a week without self-heal reverting it back to the stale Git values. With auto-sync on and selfHeal off (the default), the manual patch would have survived too, and the applications would simply have sat OutOfSync.
So the 30% 401 rate had a clean explanation, and it had nothing to do with pods. Every profile-service pod was reading the same never-patched ConfigMap, so all three were validating tokens as HS256 against the old key ID while auth-service had moved to issuing RS256-signed tokens. Every request that carried a token failed. The requests that survived were the health checks, the static reads and the unauthenticated endpoints that never touch token validation, and on this service those are roughly seven requests in ten.
The same explanation covered like-service, which is the part nobody had joined up yet. Getting the algorithm right and the key ID wrong fails just as hard as getting both wrong: like-service was selecting the pre-rotation public key and rejecting every RS256 signature auth-service produced. It had been returning 401s on 100% of its token-carrying traffic for the same week. The blast radius was two services, not one. It read as one because only profile-service had a 401-rate alert pointed at it.
The decision that almost broke production a second time
Why Git was the wrong source of truth
The instinct, when you find drift between Git and a cluster, is to trust Git. That is the whole point of GitOps. The pull request is the source of truth and the cluster is downstream. Run an ArgoCD sync, let it overwrite the live state, move on.
That instinct would have broken auth-service, the last service still working. Not in 30 seconds, which is the trap: syncing a ConfigMap changes the object and nothing else, so ArgoCD would have reported Synced and Healthy immediately while the running auth-service pods carried on validating tokens correctly from the values they had loaded at boot. Git held the pre-rotation HS256 values. The new private key that auth-service was signing tokens with did not match the public key Git was about to push into the ConfigMap. The damage would have sat dormant in the cluster until something rolled those pods, a node drain, a routine deploy, an HPA scale-up, and at that moment auth-service would have come back unable to validate anything it had ever issued, invalidating every token in flight across all three services rather than just the two that were already failing. A time bomb with a fuse we did not control is worse than a visible outage.
We had to invert the model. For this one incident, the auth-service namespace's live ConfigMap was the canonical truth, and Git was stale. The recovery had to flow live-to-Git first, then Git-to-cluster for the other two namespaces, then a rollout to make the other two namespaces actually pick the values up, and only then could auto-sync be turned back on. The order mattered.
Recovery flow. Live state was canonical for one application, Git was canonical after the merge for the other two, and neither of those two recovered until its pods were rolled.
How we got the canonical values into Git and synced the stragglers
Committing a live hotfix back to Git without breaking auth
The commit itself was unremarkable once we had a clear model. We pulled the auth-service ConfigMap, extracted the two fields, and updated all three manifests in the deploy repo in a single PR with a postmortem link in the description. The PR title was 'Hotfix reconcile: commit post-rotation JWT values from live state (incident #INC-441)' because future-us was going to want to know why these values arrived without an upstream change.
The one step that is not optional is merging that PR before running any sync. ArgoCD syncs the revision the Application tracks, which for us is main, not the branch the fix is sitting on. Run the sync loop while the PR is still open and auth-service gets reconciled against a main that still says JWT_ALGORITHM: HS256 and JWT_PUBLIC_KEY_ID: key-2024-09, which is precisely the second outage the previous section is about. Merge first, confirm the tracked revision, then sync.
# 1. Export canonical values from auth-service namespace
KID=$(kubectl -n auth get cm auth-config -o jsonpath='{.data.JWT_PUBLIC_KEY_ID}')
ALG=$(kubectl -n auth get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM}')
# 2. Patch the three manifests on a branch, commit, push
git checkout -b hotfix/inc-441-reconcile-jwt
for d in deploy/auth deploy/like deploy/profile; do
yq -i ".data.JWT_PUBLIC_KEY_ID = \"$KID\" | .data.JWT_ALGORITHM = \"$ALG\"" "$d/auth-config.yaml"
done
git add deploy/auth deploy/like deploy/profile
git commit -m 'Reconcile JWT config from live auth-service (post-rotation hotfix, INC-441)'
git push -u origin hotfix/inc-441-reconcile-jwt
# 3. Merge the PR. ArgoCD tracks main; an unmerged branch syncs the STALE values.
gh pr create --fill
gh pr merge --squash
git fetch origin main
MERGED=$(git rev-parse origin/main)
# 4. Sync per application, in order, and assert each landed on the merged revision
for app in auth-service like-service profile-service; do
argocd app sync $app --prune=false --revision "$MERGED"
argocd app wait $app --health --timeout 180
test "$(argocd app get $app -o json | jq -r .status.sync.revision)" = "$MERGED" || exit 1
done
# 5. The sync updates the ConfigMap object only. These pods read JWT_ALGORITHM and
# JWT_PUBLIC_KEY_ID as env vars at boot, so nothing changes until they are rolled.
for ns in like profile; do
kubectl -n $ns rollout restart deploy/${ns}-service
kubectl -n $ns rollout status deploy/${ns}-service --timeout=180s
done
The merge, the sync sequence and the rollout that actually applies it. auth-service syncs first as a no-op safety check, which is only true because the commit is already on the tracked revision.
We synced auth-service first deliberately. It was already correct, so the sync should be a no-op. If it had shown a diff we did not expect, that was our signal to stop and re-audit before touching like-service or profile-service. It came back clean, which told us our commit matched the live state exactly. Then like-service and profile-service synced and both reported Synced and Healthy within a couple of seconds, which is exactly what a ConfigMap-only change does and exactly why it proves nothing: the running pods were still holding the pre-rotation values they had loaded at startup, and the 401s carried on at the same rate. The recovery is the rollout, not the sync. Within 40 seconds of the profile-service rollout completing, the 401 rate in Prometheus went from 30% to 0, and like-service's went to zero with it.
If you cannot add a rollout step to your recovery, the checksum/config annotation on the pod template is the usual alternative, but check its precondition before you rely on it. checksum/config is not a Kubernetes- or ArgoCD-recognized annotation and has no built-in behavior: nothing in the cluster or in ArgoCD watches a ConfigMap and updates it for you. It only forces a rollout when something recomputes its value. That means Helm, where the template renders it as a sha256sum of the ConfigMap on every render (checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}), or Kustomize's configMapGenerator, which appends a content hash to the ConfigMap name and so changes the envFrom / configMapKeyRef reference in the pod spec and forces a new ReplicaSet. Our deploy repo is hand-written static YAML patched in place with yq against a fixed auth-config name, and in that setup a literal checksum/config annotation is a permanent no-op: the ConfigMap data changes, the annotation does not, the pod template stays byte-identical, ArgoCD sees no diff on the Deployment, nothing rolls, and the reader believes they are covered while their pods still hold pre-rotation values behind a green Synced and Healthy. With raw manifests you have two working options and no third: keep the explicit rollout, or make the same yq pass that edits .data also write the new checksum into the pod template annotation in the same commit. What is not fine is assuming a green Synced and Healthy means the process is reading the new values, because for a ConfigMap it almost never does.
Auto-sync we left off until the 401 rate had been at zero for ten minutes and we had eyes on the Jaeger traces showing fresh successful auth flows end to end. Only then did we re-enable auto-sync on all three applications, in the same order as the sync. We have written more about the order-of-operations on multi-app reconciles in the ArgoCD and GitOps recovery playbook.
Two cheap controls that prevent the next split-state week
What we changed about hotfix discipline after this one
The technical recovery was straightforward once the model was right. The interesting part of this incident was how a one-hour rotation hotfix turned into a week of latent drift. Two things had to go wrong together: a manual change that did not get committed, and an auto-sync toggle that did not get turned back on. The tempting reading is that self-heal was the safety net we disabled, and that is backwards. Had auto-sync been left on with selfHeal: true, the self-heal loop would have reverted auth-config in the auth namespace back to the stale HS256 and key-2024-09 within a reconciliation cycle, and because syncing a ConfigMap changes the object and nothing else, the running auth-service pods would have carried on validating tokens correctly from the values they had loaded at boot. Nothing would have failed, nothing would have alerted, and the Application would have flipped back to Synced. Self-heal would have erased the hotfix and hidden the drift behind a green Synced, planting exactly the dormant time bomb described above rather than catching anything, which is why an uncommitted manual change plus selfHeal: true is more dangerous, not less. The other branch is no better: had the change been committed and auto-sync left off, Git and live agree, the application sits Synced, and self-heal has nothing to act on. The control that actually catches this is the auto-sync watchdog plus a rollout-aware check, not self-heal.
We made two changes to the platform after this. The first was a scheduled job that lists ArgoCD applications with auto-sync disabled and posts to a channel if any of them have been in that state for more than four hours. The listing half is easy; the duration half is where the obvious implementation is wrong. An Application carries no timestamp for a syncPolicy edit, so there is no field that says when auto-sync went off. Reaching for .status.operationState.finishedAt measures the last sync instead, which fires on every long-lived manually-synced app and stays silent on an app disabled five hours ago that someone hand-synced an hour into the incident, which is the exact shape of INC-441. So the watchdog keeps its own state file, records the first run at which it saw each app with auto-sync off, and alerts on its own elapsed counter. If your Applications are themselves GitOps-managed, the commit that removed syncPolicy.automated works as the timestamp too. It is a couple of dozen lines of bash around argocd app list -o json plus that file. It has caught the same pattern twice in the last quarter, both times within the same incident as the original change instead of a week later.
# Posted to platform-alerts when auto-sync has been off for >4h on any app.
# .status.operationState.finishedAt is the last SYNC time, not the time the
# syncPolicy was edited, and the Application carries no timestamp for that edit,
# so the watchdog records its own first-seen time per app and counts from there.
STATE=/var/lib/argocd-watchdog/autosync-off.tsv
NOW=$(date -u +%s)
touch "$STATE"
argocd app list -o json \
| jq -r '.[] | select(.spec.syncPolicy.automated == null) | .metadata.name' \
| sort > /tmp/autosync-off.now
# keep the original first-seen stamp for apps still disabled, stamp new ones now,
# and drop the apps whose auto-sync came back on
while read -r app; do
seen=$(awk -F'\t' -v a="$app" '$1 == a { print $2; exit }' "$STATE")
printf '%s\t%s\n' "$app" "${seen:-$NOW}"
done < /tmp/autosync-off.now > "$STATE.next"
mv "$STATE.next" "$STATE"
awk -F'\t' -v now="$NOW" 'now - $2 > 14400 {
printf "%s: auto-sync off for %dh\n", $1, (now - $2) / 3600 }' "$STATE"
The auto-sync watchdog. It counts from its own first-seen record, because the Application object has no timestamp for a syncPolicy edit. The cheapest control with the highest ROI we shipped this year.
The second change was a rule we now apply to every incident we run: if a hotfix lands in the cluster via kubectl, the same incident does not close until the change is in a merged PR. Not the next day. Not 'we'll get to it'. The incident commander treats the Git commit as a recovery step, not a follow-up. That sounds like a process rule, and it is, but it has a sharp version: the on-call's runbook for manual ConfigMap patches now includes the export-and-PR commands at the bottom of the same page. The friction to do it right is now lower than the friction to defer it.
When the cluster and Git disagree and you cannot just sync your way out
If your GitOps is in a split state right now
The hard part of this kind of incident is not the kubectl or the argocd CLI. The hard part is figuring out which system is the source of truth for which field right now, when the answer is not 'Git, always'. Get that wrong and an ArgoCD sync will take production down a second time on top of whatever is already broken. We have seen the same shape of failure four times this year: a rotation, a migration, an emergency schema change, and a CRD upgrade, each of which left some subset of clusters carrying values that Git did not yet know about.
InfraForge runs these reconciles every week. We know the order to commit, the order to merge, the order to sync, which workloads have to be rolled before a config change means anything, the checks that catch a propagated copy you forgot about, and the questions to ask before you trust Git over the live state. If your auto-sync has been off for a week and you are not sure what would happen when you turn it back on, book an infrastructure review with our team and we will be on a bridge with you the same day to walk the drift before you touch anything.
Originally published at https://infraforge.agency/insights/argocd-drift-three-namespaces-jwt-configmap-hotfix/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)