Rotating API keys is routine maintenance — until it causes a production incident. The window between revoking an old key and deploying a new one is where things break: services return 401s, alerts fire, and someone spends 20 minutes tracing an outage back to a key swap they thought was safe. This article covers two patterns that make rotation genuinely zero-downtime, with working code for both.
Why Naive Rotation Breaks Things
The standard advice: generate a new key, update your environment variable, redeploy. Simple. But this creates a race condition that's easy to miss in testing and painful in production:
- T=0: Old key revoked
- T=+N (seconds to minutes): New key propagated to all running instances
- During T=0 to T=+N: Every API call returns 401
Even "zero-downtime" deployment strategies don't help here if the key is injected into the container environment at startup. Rolling updates replace pods gradually — the old pods carry the old key, the new pods get the new key, but if the old key was revoked before the rollout completes, the old pods start failing.
The fix isn't to deploy faster — it's to accept both keys simultaneously during a transition window.
Pattern 1: Dual-Key Rotation
The principle: your validation logic always accepts two keys — the current and the previous. When you rotate, the old current becomes the new previous (with an expiry timestamp), and a fresh key becomes current. During the overlap window, both keys work.
import time
import secrets
import hashlib
from dataclasses import dataclass
from typing import Optional
@dataclass
class StoredKey:
key_hash: str
created_at: float
expires_at: Optional[float] = None
class DualKeyStore:
def __init__(self):
self.current: Optional[StoredKey] = None
self.previous: Optional[StoredKey] = None
@staticmethod
def _hash(raw: str) -> str:
return hashlib.sha256(raw.encode()).hexdigest()
def rotate(self, overlap_seconds: int = 300) -> str:
# Returns the new raw key. Store it securely — shown only once.
raw = secrets.token_urlsafe(32)
new_key = StoredKey(key_hash=self._hash(raw), created_at=time.time())
if self.current:
self.current.expires_at = time.time() + overlap_seconds
self.previous = self.current
self.current = new_key
return raw
def is_valid(self, raw: str) -> bool:
h = self._hash(raw)
now = time.time()
if self.current and self.current.key_hash == h:
return True
if self.previous and self.previous.key_hash == h:
if self.previous.expires_at is None or self.previous.expires_at > now:
return True
return False
A few things worth noting: raw keys are never stored — only SHA-256 hashes. The raw key is returned exactly once, at generation. The overlap_seconds parameter controls how long old clients have to migrate before their key stops working; 300 seconds is a reasonable default for most deployment pipelines.
Pattern 2: Versioned Key Tokens
A cleaner approach, similar to how Stripe structures its API keys: embed a version identifier in the key itself. The validator extracts the version, loads the corresponding signing secret, and verifies. Rotation means adding a new version and setting an expiry on the old one.
package apikey
import (
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"fmt"
"strings"
"time"
)
type KeyVersion struct {
Secret []byte
ExpiresAt time.Time
}
type VersionedStore struct {
versions map[int]*KeyVersion
latest int
}
func NewVersionedStore() *VersionedStore {
return &VersionedStore{versions: make(map[int]*KeyVersion)}
}
func (s *VersionedStore) AddVersion(v int, secret []byte, expiresAt time.Time) {
s.versions[v] = &KeyVersion{Secret: secret, ExpiresAt: expiresAt}
if v > s.latest {
s.latest = v
}
}
// Issue returns a signed token embedding the version and client ID.
func (s *VersionedStore) Issue(clientID string) string {
kv := s.versions[s.latest]
mac := hmac.New(sha256.New, kv.Secret)
mac.Write([]byte(clientID))
sig := base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
return fmt.Sprintf("v%d_%s_%s", s.latest, clientID, sig)
}
// Validate returns (clientID, true) on success.
func (s *VersionedStore) Validate(token string) (string, bool) {
parts := strings.SplitN(token, "_", 3)
if len(parts) != 3 {
return "", false
}
var ver int
if _, err := fmt.Sscanf(parts[0], "v%d", &ver); err != nil {
return "", false
}
clientID, sig := parts[1], parts[2]
kv, ok := s.versions[ver]
if !ok || (!kv.ExpiresAt.IsZero() && time.Now().After(kv.ExpiresAt)) {
return "", false
}
mac := hmac.New(sha256.New, kv.Secret)
mac.Write([]byte(clientID))
expected := base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
if !hmac.Equal([]byte(sig), []byte(expected)) {
return "", false
}
return clientID, true
}
To rotate: call AddVersion with a new version number and a fresh secret. Set ExpiresAt on the old version to time.Now().Add(10 * time.Minute). Both versions validate during the overlap window; expired versions automatically return ("", false) afterward.
The version prefix in the token (v2_, v3_) is not a security feature — don't rely on it alone. The HMAC signature is what provides authenticity.
Propagating the New Key
The validation logic is the easy part. The harder problem is making sure every service instance receives the new key before the old one expires.
Three strategies in increasing order of robustness:
Rolling restart with secret manager: Store the current key in your secret manager (Vault, AWS Secrets Manager, Doppler). When rotating, update the secret, then trigger a rolling restart. Each new pod reads the current key at startup. This works if your overlap window exceeds your deployment time — which isn't always true for large clusters.
Hot reload: The application polls the secret manager every 30–60 seconds and updates its in-memory key store without restarting. This decouples rotation from deployment entirely. The downside is added complexity: you need to handle the poll gracefully — don't crash if the secret manager is temporarily unavailable.
Internal key endpoint: Consumers call an internal /internal/keys/active endpoint to retrieve the current key. Your key service handles rotation server-side; clients pick up changes on the next poll or at startup. This is how most high-availability systems are wired — it makes rotation a pure backend operation with no client coordination required.
For teams looking to integrate these patterns into a broader hardening strategy, the security hardening checklists on our site cover key lifecycle management alongside secrets scanning, CI/CD controls, and runtime monitoring.
Edge Cases to Handle
Clock skew: Expiry timestamps compared across multiple nodes can fail unexpectedly if clocks differ by more than a few seconds. Add a 30-second skew buffer to your expiry window, or synchronize with NTP.
Concurrent rotation: If two operators trigger a rotation simultaneously, you may lose track of which key is "previous." Wrap the rotate operation in a distributed lock — a Redis SET NX with a TTL or a database advisory lock is sufficient.
Emergency revocation: You need a break-glass path to immediately invalidate a key, bypassing the overlap window. In the Python example, set expires_at = time.time() - 1 on both current and previous. In the Go version, set ExpiresAt = time.Now().Add(-time.Second) on the affected version. Test this path before you need it.
Audit logs: Log every key validation failure with timestamp, source IP, and which key version was attempted. A sudden spike in failures from an unexpected IP is often the first signal that a compromised key is being tested elsewhere.
The Takeaway
The rule is simple: never let the system reach a state where no valid key exists. Both patterns above enforce this — dual-key rotation through an explicit overlap window, versioned keys through version-scoped expiry.
Your overlap window length is the only real tuning parameter. Make it longer than your worst-case deployment time, but short enough that a leaked key doesn't stay valid indefinitely. Five to ten minutes covers most pipelines without being reckless.
Pick the pattern that matches your infrastructure: dual-key is simpler to implement and reason about; versioned tokens are cleaner at scale and give you an audit trail per key version baked into the token format itself.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)