Kubernetes controllers built on client-go have a silent failure mode that's easy to miss. When a projected ServiceAccount token becomes permanently invalid, your controller keeps running, passes health checks, shows Ready, and does absolutely nothing.
The Problem
For in-cluster auth, client-go reads the projected ServiceAccount token from file via CachedFileTokenSource and re-reads it on a fixed interval. But it does not check the response status. If the API server returns 401, the same token gets sent again. The error is logged, but the process never exits. The pod stays Running/Ready.
// client-go's default: log and retry. Forever.
func DefaultWatchErrorHandler(ctx context.Context, r *Reflector, err error) {
// logs the error
// returns (triggering a retry with backoff)
}
Where I Found It
During a Helm upgrade via ArgoCD on KAI-Scheduler (CNCF Sandbox GPU scheduler), the scheduler's ServiceAccount got deleted and recreated. Running pods still held tokens bound to the old SA UID. The scheduler kept retrying 401s indefinitely while showing Running/Ready. ArgoCD reported the application as Synced/Healthy the entire time.
Full incident: #1751. The 401 handling gap was tracked separately in #1817.
The Fix
Submitted a PR that adds a transport wrapper on rest.Config that exits on 401:
type unauthorizedRoundTripper struct {
rt http.RoundTripper
}
func (t *unauthorizedRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
resp, err := t.rt.RoundTrip(req)
if err == nil && resp.StatusCode == http.StatusUnauthorized {
log.InfraLogger.Errorf("API server returned 401 Unauthorized, exiting to trigger pod restart")
os.Exit(1)
}
return resp, err
}
One call in the startup path:
config := clientconfig.GetConfigOrDie()
config.Wrap(wrapExitOnUnauthorized)
Every client created from this config goes through the wrapper. On 401, the process exits, kubelet restarts it, fresh token is mounted.
Why os.Exit and not graceful shutdown? If your token is invalid, you can't release your leader election lease either (that's an API call too). The lease expires on its own in 15 seconds. os.Exit matches what kube-scheduler upstream does for unrecoverable errors.
If you build controllers on client-go, worth checking whether yours handles 401. Without an exit or a failed health probe, the pod stays Running and nothing self-heals.
Top comments (2)
Exiting on the first 401 turns an invisible failure into a loud one, and that's the real win here. A crash loop shows up on every dashboard, while a pod that says Ready but does nothing shows up on none. The edge I'd poke at is a transient 401 though. A token rotation race or a brief auth layer hiccup could return one bad response to an otherwise healthy pod, and across several replicas that's a synchronized restart. Did you consider requiring two or three consecutive 401s before exiting, or was the read that a real 401 from the API server is never transient in practice?
Good question!
For projected ServiceAccount tokens, a 401 from the API server is effectively permanent. The token is bound to the pod's UID, so if it's rejected, every subsequent request with that same token will also be rejected. client-go re-reads the token file on a ~50 second interval, but if the file itself contains the same invalid token, re-reading doesn't help.