DEV Community

Indra Gusti Prasetya
Indra Gusti Prasetya

Posted on • Originally published at indragustiprasetya.com

Rotate API Keys Without Downtime: Dual-Key Pattern

#tips

Almost every team knows it should rotate credentials. The reason it doesn't is fear. The last time someone swapped a key in production, a service lost its auth mid-request and paged the on-call at 2 a.m., so the key now sits there for two years. That stale key becomes the one credential nobody touches, which is exactly the one an attacker keeps.

The fix is the dual-key grace-period pattern: for a short window, both the old and the new credential are valid at once, so consumers roll over on their own schedule and nothing breaks. Simple idea. The trap is that the ordering flips depending on whether you are rotating a bearer credential (an API key, an AWS access key, a webhook secret) or a signing key (a JWT/JWKS private key), and getting that ordering backwards is the single most common way a "safe" rotation still takes down auth. If you are already moving CI off long-lived secrets, pair this with killing static AWS keys in favor of OIDC; rotation is the discipline for the keys you cannot yet delete. And if you own a fleet of service accounts and machine tokens, the same pattern is the backbone of non-human identity governance. These tips are for the people who own the credentials but do not want to own the incident.

The tips

  1. Rotate additively, never in place. Deploy the new key before you touch the old one. The whole pattern rests on one rule: a working credential is never removed before its replacement is proven under real traffic. The sequence is always create, deploy, verify, revoke, never revoke then create. AWS's own IAM rotation guidance follows exactly this five-step order: create the second key, update the app, disable (not delete) the old key, validate, then delete. That "disable first" step is your undo button. A deactivated key reactivates in seconds; a deleted one is gone.

  2. On AWS, treat the two-active-keys limit as the feature it is. IAM caps you at two access keys per user, and that cap is not a nuisance. It is precisely enough to hold an old and a new key side by side during rollover. Run the rotation as CLI so it is scriptable and shows up in CloudTrail:

   aws iam create-access-key --user-name svc-deploy
   # deploy the new key to the workload, confirm traffic, THEN:
   aws iam update-access-key --access-key-id AKIAOLD --status Inactive --user-name svc-deploy
   # wait out the grace period watching for AccessDenied, THEN:
   aws iam delete-access-key --access-key-id AKIAOLD --user-name svc-deploy
Enter fullscreen mode Exit fullscreen mode

If create-access-key fails because two keys already exist, that is a stuck prior rotation. Clean it up before you start a new one.

  1. Before you rotate, find out who is actually using the key. The scary part of rotation is not knowing which consumer will break. AWS answers this directly. aws iam get-access-key-last-used --access-key-id AKIAOLD returns the last-used date, the region, and the service name that called it last. Pull an IAM credential report and read the access_key_x_last_used_date column across every user first. A key whose last-used date is N/A or months old is safe to disable on the spot. You are not rotating it, you are decommissioning a credential nobody uses, which is the cheapest win in the whole exercise.

  2. Size the grace period to your slowest consumer, then make it as short as that allows. The overlap window has to outlast the longest interval any client waits before it picks up the new key: a cron job that runs nightly, a mobile app that only refreshes on launch, a cached config that redeploys weekly. Most provider docs land somewhere between 24 hours and 7 days for bearer keys. Longer is not safer. Every extra hour is an hour a possibly-leaked old key still works. Pick the smallest window that clears your slowest known consumer, then write that number into the runbook so the next person does not guess.

  3. For JWT signing keys, invert the order. Publish the new public key before you sign with it. This is the gotcha that catches teams who assume signing keys behave like API keys. They do not. A verifier fetches your JWKS and caches it. If you start signing tokens with a new kid before that key is published and caches have refreshed, verifiers hit "key not found" and reject perfectly valid tokens. The correct sequence: add the new key to the JWKS endpoint, wait at least one full JWKS cache TTL so every verifier has it, then flip signing to the new key, and keep the old public key published until the last token it signed has expired.

   grace period (JWKS) = JWKS cache TTL  +  max token lifetime  +  safety buffer
Enter fullscreen mode Exit fullscreen mode

  1. Always sign with a kid, and never pull a public key while live tokens still reference it. The kid (key ID) header is what lets a verifier pick the right key out of a JWKS that legitimately holds two keys mid-rotation. Seeing two keys at your JWKS endpoint during overlap is normal and expected, not a bug to "fix." Retire the old public key only once a full max-token-lifetime has passed since you stopped signing with it. Pull it earlier and you invalidate tokens that are still inside their valid window. This is the same care agent identity built on SPIFFE and OAuth depends on when workloads verify each other's tokens.

  2. Let webhook providers hold both secrets for you, and verify against an array. Webhook signing secrets rotate the same way, and good providers automate the overlap for you. Stripe's "Roll secret" keeps the current secret valid for up to 24 hours while the new one is live, signing each event with both during the window. Your endpoint's job is to verify against a list, not a single value, which is what the official SDKs support:

   const secrets = [process.env.STRIPE_WHSEC_NEW, process.env.STRIPE_WHSEC_OLD];
   // try each; accept the event if any signature validates
Enter fullscreen mode Exit fullscreen mode

Deploy with both set, confirm deliveries still succeed, then drop the old one from the environment.

  1. Verify with real traffic, not a health check, before you revoke. "The new key works" has to mean production requests are succeeding on it, not that a synthetic probe returned 200. Watch the actual signal for your credential type: AccessDenied counts in CloudTrail for IAM keys, 401/403 rates at the API gateway for API keys, signature-verification failures in your webhook logs, "key not found" rejections for signing keys. Cut over only when that error line stays flat under load. A green probe against an empty code path has fooled more rotations than any provider bug.

  2. Automate the schedule, but keep a manual break-glass path. Rotation that depends on a human remembering is the rotation that quietly stops happening. Use AWS Config's access-keys-rotated rule (or your secrets manager's built-in rotation) to enforce a max key age of 90 days, and 30 to 60 for sensitive workloads. Keep the manual create/disable/delete runbook current and tested anyway, because the day you actually need to rotate fast is the day a key leaks, and that is the worst possible moment to be debugging your own automation.

  3. Be honest about what rotation does and does not do. Rotation shrinks the window a leaked credential stays useful. It is containment, not detection. It does nothing to tell you a key leaked, and nothing to stop exfiltration while the key is still valid. So pair every rotation policy with a leak alert: secret scanning on your repos, anomalous-use detection on the credential. Treat an unexpected region or service in get-access-key-last-used as a signal to rotate now, off-schedule, not at the next 90-day tick. Rotation limits the blast radius. Something else has to spot the blast.

Wrap-up

If you take one habit from all of this, make it the ordering rule: the new credential is proven in production before the old one is revoked, and for signing keys it is published before it is used. Every zero-downtime rotation is just that rule applied carefully, with a grace period sized to your slowest consumer. Script it into a runbook once, and the credential nobody rotates stops being a category on your infrastructure.

Sources

Originally published at indragustiprasetya.com

Top comments (0)