The on-call team had spent two hours today chasing a 30% 401 rate on profile-service, an alert that had fired and been snoozed for most of the week, when we got pulled in. As far as their dashboards showed, it was only profile-service, only some pods, only authenticated requests. Two of those three were wrong. The shape of that number is what had thrown them off: a 30% failure rate on a 3-pod deployment looks exactly like one pod out of three running a different config, so that is where they had been digging. It was not a pod problem. All three profile-service pods were identical, and 30% was simply the share of traffic that carried a token at all. It was not confined to profile-service either: like-service had been rejecting every token-carrying request for the same week, and nobody was looking at it because the 401-rate alert had been wired up on profile-service and not on like-service. What was underneath both was a week-old JWT key rotation hotfix that had landed in the live cluster, never made it to Git, and ArgoCD auto-sync had been disabled across three applications and quietly left off. By the time we opened a terminal the same ConfigMap was in four places, Git and three namespaces, holding three different values between them.
Problem signals:
- A service is returning 401s on a stable fraction of requests, and the fraction tracks the share of traffic that carries a token rather than any pod ratio
- ArgoCD shows applications as OutOfSync but auto-sync is disabled and nobody remembers turning it off
- kubectl diff against the rendered Helm or Kustomize output shows changes nobody can attribute to a recent PR
- Multiple namespaces have a propagated copy of the same ConfigMap and the copies disagree
- A recent incident postmortem mentions a manual kubectl edit or kubectl patch that was never followed by a Git commit
The first 20 minutes: mapping how far the drift had spread
Four places, three different values
The initial theory from the on-call lead was that a pod had missed the last restart and was still holding the pre-rotation JWT key. Reasonable theory. It was wrong, but only because it was incomplete.
We ran the obvious diff first. Pull the ConfigMap from each of the three namespaces, pull the manifest from the Git repo at HEAD, compare. What we expected to find was two values: a correct one in the cluster and a stale one in Git, or the reverse. What we actually found was three values across four places: profile-service's never-patched copy still matched Git.
# auth-service namespace
$ kubectl -n auth get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM} {.data.JWT_PUBLIC_KEY_ID}'
RS256 key-2024-11-rot
# like-service namespace (propagated copy)
$ kubectl -n like get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM} {.data.JWT_PUBLIC_KEY_ID}'
RS256 key-2024-09
# profile-service namespace (propagated copy)
$ kubectl -n profile get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM} {.data.JWT_PUBLIC_KEY_ID}'
HS256 key-2024-09
# Git, main branch
$ grep -E 'JWT_(ALGORITHM|PUBLIC_KEY_ID)' deploy/*/auth-config.yaml
deploy/auth/auth-config.yaml: JWT_ALGORITHM: HS256
deploy/auth/auth-config.yaml: JWT_PUBLIC_KEY_ID: key-2024-09
# (and the same stale pair in like and profile manifests)
What the diff actually showed: four places, three values.
The story behind the three states reconstructed quickly from the previous week's incident channel. During the rotation, an SRE had patched auth-service's ConfigMap directly with the new RS256 key. They then walked the change into the like-service namespace and got the algorithm right but missed the key ID, leaving the old one. They ran out of focus before reaching profile-service, intended to come back to it, and did not. ArgoCD auto-sync had been disabled across all three applications during the incident as a guardrail and never re-enabled. These applications run automated sync with selfHeal: true, so that toggle is the only reason the cluster state survived a week without self-heal reverting it back to the stale Git values. With auto-sync on and selfHeal off (the default), the manual patch would have survived too, but only until the next commit to the tracked branch: automated sync runs once per new commit, and that sync would have written the stale Git values straight back.
So the 30% 401 rate had a clean explanation, and it had nothing to do with pods. Every profile-service pod was reading the same never-patched ConfigMap, so all three were validating tokens as HS256 against the old key ID while auth-service had moved to issuing RS256-signed tokens. Every request that carried a token failed. The requests that survived were the health checks, the static reads and the unauthenticated endpoints that never touch token validation, and on this service those are roughly seven requests in ten.
The same explanation covered like-service, which is the part nobody had joined up yet. Getting the algorithm right and the key ID wrong fails just as hard as getting both wrong: like-service was selecting the pre-rotation key and rejecting every RS256 signature auth-service produced. It had been returning 401s on 100% of its token-carrying traffic for the same week. The blast radius was two services, not one. It read as one because only profile-service had a 401-rate alert pointed at it.
The decision that almost broke production a second time
Why Git was the wrong source of truth
The instinct, when you find drift between Git and a cluster, is to trust Git. That is the whole point of GitOps. The pull request is the source of truth and the cluster is downstream. Run an ArgoCD sync, let it overwrite the live state, move on.
That instinct would have broken auth-service, the last service still working. Not in 30 seconds, which is the trap: syncing a ConfigMap changes the object and nothing else, so ArgoCD would have reported Synced and Healthy immediately while the running auth-service pods carried on validating tokens correctly from the values they had loaded at boot. Git held the pre-rotation HS256 values. The new private key that auth-service was signing tokens with did not match the algorithm and key ID Git was about to push into the ConfigMap. The damage would have sat dormant in the cluster until something started new auth-service pods, whether a node drain, a routine deploy or an HPA scale-up, and every one of those pods would have come up unable to validate anything auth-service had issued. A time bomb with a fuse we did not control is worse than a visible outage.
We had to invert the model. For this one incident, the auth-service namespace's live ConfigMap was the canonical truth, and Git was stale. The recovery had to flow live-to-Git first, then Git-to-cluster for the other two namespaces, then a rollout to make the other two namespaces actually pick the values up, and only then could auto-sync be turned back on. The order mattered.
Recovery flow. Live state was canonical for one application, Git was canonical after the merge for the other two, and neither of those two recovered until its pods were rolled.
How we got the canonical values into Git and synced the stragglers
Committing a live hotfix back to Git without breaking auth
The commit itself was unremarkable once we had a clear model. We pulled the auth-service ConfigMap, extracted the two fields, and updated all three manifests in the deploy repo in a single PR with a postmortem link in the description. The PR title was 'Hotfix reconcile: commit post-rotation JWT values from live state (incident #INC-441)' because future-us was going to want to know why these values arrived without an upstream change.
The one step that is not optional is merging that PR before running any sync. ArgoCD syncs the revision the Application tracks, which for us is main, not the branch the fix is sitting on. Run the sync loop while the PR is still open and auth-service gets reconciled against a main that still says JWT_ALGORITHM: HS256 and JWT_PUBLIC_KEY_ID: key-2024-09, which is precisely the second outage the previous section is about. Merge first, prove main carries the new values, then sync.
# 1. Export canonical values from auth-service namespace
KID=$(kubectl -n auth get cm auth-config -o jsonpath='{.data.JWT_PUBLIC_KEY_ID}')
ALG=$(kubectl -n auth get cm auth-config -o jsonpath='{.data.JWT_ALGORITHM}')
[ -n "$KID" ] && [ -n "$ALG" ] || { echo "empty export; stop"; exit 1; }
# 2. Patch the three manifests on a branch, commit, push
git checkout -b hotfix/inc-441-reconcile-jwt
for d in deploy/auth deploy/like deploy/profile; do
yq -i ".data.JWT_PUBLIC_KEY_ID = \"$KID\" | .data.JWT_ALGORITHM = \"$ALG\"" "$d/auth-config.yaml"
done
git add deploy/auth deploy/like deploy/profile
git commit -m 'Reconcile JWT config from live auth-service (post-rotation hotfix, INC-441)'
git push -u origin hotfix/inc-441-reconcile-jwt
# 3. Merge the PR. ArgoCD tracks main; an unmerged branch syncs the STALE values.
# On a branch with a merge queue, gh returns once the PR is queued or set to
# auto-merge, before anything merges, so prove main carries the new values
# first. If it is still queued, wait for the merge, re-run step 1 so KID and
# ALG are set again, and resume from git fetch.
gh pr create --title "Hotfix reconcile: commit post-rotation JWT values from live state (incident #INC-441)" \
--body "Postmortem: incidents/INC-441.md"
gh pr merge --squash || exit 1
git fetch origin main
MERGED=$(git rev-parse origin/main)
git show "$MERGED:deploy/auth/auth-config.yaml" | grep -qF "$KID" \
&& git show "$MERGED:deploy/auth/auth-config.yaml" | grep -qF "$ALG" \
|| { echo "main does not carry $ALG / $KID yet"; exit 1; }
# 4. auth-service is already correct, so its diff against the merged revision must
# be empty before anything is synced. argocd app diff exits 1 on any diff and
# 2 on an error. It ignores Secrets, so a Secret the same hotfix touched needs
# its own check.
argocd app diff auth-service --revision "$MERGED" \
|| { echo "diff found, or diff failed, on auth-service; stop and re-audit"; exit 1; }
# 5. Sync per application, in order, stopping on the first failure
for app in auth-service like-service profile-service; do
argocd app sync $app --prune=false --revision "$MERGED" || exit 1
argocd app wait $app --health --timeout 180 || exit 1
done
# 6. The sync updates the ConfigMap object only. These pods read JWT_ALGORITHM and
# JWT_PUBLIC_KEY_ID as env vars at boot, so nothing changes until they are rolled.
for ns in like profile; do
kubectl -n $ns rollout restart deploy/${ns}-service
kubectl -n $ns rollout status deploy/${ns}-service --timeout=180s || exit 1
done
The merge, the checks that prove it landed, the sync sequence and the rollout that actually applies it. auth-service is diffed before anything syncs: it is already correct, so any diff there means something in auth-service is not what we think it is, the commit or other drift, and the script stops before anything is synced; a stop leaves the commit on main, which is safe while auto-sync is off.
We diffed auth-service first, before syncing anything. It was already correct, so the diff had to be empty, and any diff was our signal to stop and re-audit before anything was synced. It came back empty, which told us our commit matched the live state exactly. Then the three applications synced, and like-service and profile-service both reported Synced and Healthy within a couple of seconds, which is exactly what a ConfigMap-only change does and exactly why it proves nothing: the running pods were still holding the pre-rotation values they had loaded at startup, and the 401s carried on at the same rate. The recovery is the rollout, not the sync. Within 40 seconds of the profile-service rollout completing, the 401 rate in Prometheus went from 30% to 0; like-service, rolled first, had already dropped to zero a minute earlier.
If you cannot add a rollout step to your recovery, the checksum/config annotation on the pod template is the usual alternative, but check its precondition before you rely on it. checksum/config is not a Kubernetes- or ArgoCD-recognized annotation and has no built-in behavior: nothing in the cluster or in ArgoCD watches a ConfigMap and updates it for you. It only forces a rollout when something recomputes its value. That means Helm, where the template renders it as a sha256sum of the ConfigMap on every render (checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}), or Kustomize's configMapGenerator, which appends a content hash to the ConfigMap name and so changes the envFrom / configMapKeyRef reference in the pod spec and forces a new ReplicaSet. Our deploy repo is hand-written static YAML patched in place with yq against a fixed auth-config name, and in that setup a literal checksum/config annotation is a permanent no-op: the ConfigMap data changes, the annotation does not, the pod template stays byte-identical, ArgoCD sees no diff on the Deployment, nothing rolls, and the reader believes they are covered while their pods still hold pre-rotation values behind a green Synced and Healthy. With raw manifests and nothing new installed, you have two working options: keep the explicit rollout, or make the same yq pass that edits .data also write the new checksum into the pod template annotation in the same commit. The third is a controller that rolls workloads when a referenced ConfigMap changes, such as Stakater Reloader with its reloader.stakater.com/auto annotation, if you are willing to run one. Under Argo CD, run it with --reload-strategy=annotations: its default strategy adds an environment variable to the pod template, which reads as drift, and its README prefers annotations in GitOps environments for exactly that reason. What is not fine is assuming a green Synced and Healthy means the process is reading the new values, because for a ConfigMap it almost never does.
Auto-sync we left off until the 401 rate had been at zero for ten minutes and we had eyes on the Jaeger traces showing fresh successful auth flows end to end. Only then did we re-enable auto-sync on all three applications, in the same order as the sync. We have written more about the order-of-operations on multi-app reconciles in the ArgoCD and GitOps recovery playbook.
Two cheap controls that prevent the next split-state week
What we changed about hotfix discipline after this one
The technical recovery was straightforward once the model was right. The interesting part of this incident was how a one-hour rotation hotfix turned into a week of latent drift. Two things had to go wrong together: a manual change that did not get committed, and an auto-sync toggle that did not get turned back on. A third let it run for a week: the profile-service alert fired and kept getting snoozed. The tempting reading is that self-heal was the safety net we disabled, and that is backwards. Had auto-sync been left on with selfHeal: true, the self-heal loop would have reverted auth-config in the auth namespace back to the stale HS256 and key-2024-09 within a reconciliation cycle, and because syncing a ConfigMap changes the object and nothing else, the running auth-service pods would have carried on validating tokens correctly from the values they had loaded at boot. Nothing would have failed, nothing would have alerted, and the Application would have flipped back to Synced. Self-heal would have erased the hotfix and hidden the drift behind a green Synced, planting exactly the dormant time bomb described above rather than catching anything, which is why an uncommitted manual change plus selfHeal: true is more dangerous, not less. The other branch is no better: had the change been committed and auto-sync left off, Git and live agree, the application sits Synced, and self-heal has nothing to act on. The control that actually catches this is the auto-sync watchdog plus a rollout-aware check, not self-heal.
We made two changes to the platform after this. The first was a scheduled job that lists ArgoCD applications with auto-sync disabled and posts to a channel if any of them have been in that state for more than four hours. The listing half is easy; the duration half is where the obvious implementation is wrong. An Application carries no timestamp for a syncPolicy edit, so there is no field that says when auto-sync went off. Reaching for .status.operationState.finishedAt measures the last sync instead, which fires on every long-lived manually-synced app and stays silent on an app disabled five hours ago that someone hand-synced an hour into the incident, which is the exact shape of INC-441. So the watchdog keeps its own state file, records the first run at which it saw each app with auto-sync off, and alerts on its own elapsed counter. If your Applications are themselves GitOps-managed, the commit that removed syncPolicy.automated, or set its enabled field to false, works as the timestamp too. It is a couple of dozen lines of bash around argocd app list -o json plus that file. It has caught the same pattern twice in the last quarter, both times within the same incident as the original change instead of a week later.
#!/usr/bin/env bash
# Run from cron; its output goes to platform-alerts when auto-sync has been off
# for >4h on any app.
# .status.operationState.finishedAt is the last SYNC time, not the time the
# syncPolicy was edited, and the Application carries no timestamp for that edit,
# so the watchdog records its own first-seen time per app and counts from there.
set -o pipefail
STATE=/var/lib/argocd-watchdog/autosync-off.tsv
NOW=$(date -u +%s)
touch "$STATE"
# Auto-sync is off when the automated block is gone or, on current Argo CD, when it
# is present with enabled: false. A failed or empty list must not read as "nothing
# is off", because the loop below would then wipe every first-seen stamp.
APPS=$(argocd app list -o json) \
|| { echo "argocd app list failed; state left untouched" >&2; exit 1; }
[ "$(printf '%s' "$APPS" | jq 'length')" -gt 0 ] \
|| { echo "argocd app list returned no applications; state left untouched" >&2; exit 1; }
printf '%s' "$APPS" \
| jq -r '.[] | select(.spec.syncPolicy.automated == null or .spec.syncPolicy.automated.enabled == false) | .metadata.name' \
| sort > /tmp/autosync-off.now || exit 1
# keep the original first-seen stamp for apps still disabled, stamp new ones now,
# and drop the apps whose auto-sync came back on
while read -r app; do
seen=$(awk -F'\t' -v a="$app" '$1 == a { print $2; exit }' "$STATE")
printf '%s\t%s\n' "$app" "${seen:-$NOW}"
done < /tmp/autosync-off.now > "$STATE.next"
mv "$STATE.next" "$STATE"
awk -F'\t' -v now="$NOW" 'now - $2 > 14400 {
printf "%s: auto-sync off for %dh\n", $1, (now - $2) / 3600 }' "$STATE"
The auto-sync watchdog. It counts from its own first-seen record, because the Application object has no timestamp for a syncPolicy edit. The cheapest control with the highest ROI we shipped this year.
The second change was a rule we now apply to every incident we run: if a hotfix lands in the cluster via kubectl, the same incident does not close until the change is in a merged PR. Not the next day. Not 'we'll get to it'. The incident commander treats the Git commit as a recovery step, not a follow-up. That sounds like a process rule, and it is, but it has a sharp version: the on-call's runbook for manual ConfigMap patches now includes the export-and-PR commands at the bottom of the same page. The friction to do it right is now lower than the friction to defer it.
When the cluster and Git disagree and you cannot just sync your way out
If your GitOps is in a split state right now
The hard part of this kind of incident is not the kubectl or the argocd CLI. The hard part is figuring out which system is the source of truth for which field right now, when the answer is not 'Git, always'. Get that wrong and an ArgoCD sync will take production down a second time on top of whatever is already broken. We have seen the same shape of failure four times this year: a rotation, a migration, an emergency schema change, and a CRD upgrade, each of which left some subset of clusters carrying values that Git did not yet know about.
InfraForge runs these reconciles every week. We know the order to commit, the order to merge, the order to sync, which workloads have to be rolled before a config change means anything, the checks that catch a propagated copy you forgot about, and the questions to ask before you trust Git over the live state. If your auto-sync has been off for a week and you are not sure what would happen when you turn it back on, book an infrastructure review with our team and we will be on a bridge with you the same day to walk the drift before you touch anything.
Originally published at https://infraforge.agency/insights/argocd-drift-three-namespaces-jwt-configmap-hotfix/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)