Rotating signing keys without breaking live tokens

A signing-key rotation can be simple on paper: generate a new key, change the signer, and deploy. In a running service, that sequence can reject valid tokens. A token signed just before the change may remain valid for another fifteen minutes. At the same time, a receiving API may still have a JWKS snapshot without the new key.

The failure is confusing because each component may look healthy. The auth server signs a token. The resource service is serving requests. The JWKS endpoint returns JSON. Yet a call fails because they disagree about which public keys are available at that moment.

I built Orionis so the auth server can hold multiple signers, publish all their public keys in JWKS, and select one active_kid for new tokens. That gives a planned rotation an overlap period.

This article is about a routine, planned rotation of a service-token signing key. The goal is to change the key used for new tokens while letting already issued tokens reach their normal expiry. That goal determines the order of every deployment step.

The four states

T0  JWKS: [old]       signing: old
T1  JWKS: [old, new]  signing: old
T2  JWKS: [old, new]  signing: new
T3  JWKS: [new]       signing: new

T0 — current state. Existing tokens carry the old kid. Resource services can validate them with the old public key.

T1 — publish first. Add the new signer while leaving the old signer active. Roll this state to every auth-server instance before moving on. Every JWKS endpoint must be able to serve both public keys before any instance starts issuing tokens with the new kid.

T2 — switch signing. Change active_kid to the new key, keeping both keys in JWKS. During this rollout, some auth instances may still sign with the old key while others sign with the new one. Both kinds of token remain verifiable because every instance publishes both public keys.

T3 — retire old. Count from the last token signed by the old key. After the maximum token lifetime and allowed clock leeway have elapsed, remove the old signer and its public JWK. Do not count only from the moment the T2 rollout began.

The local Orionis example uses a 15-minute access-token lifetime, but production TTL is configurable. The waiting period must follow the longest lifetime actually issued during the overlap.

Notice the asymmetry between T1 and T3. Publishing a new public key early only prepares verifiers for signatures they may soon see. Removing the old public key early has an immediate cost: a token that is still within its valid time window becomes unverifiable. Publication goes ahead of signing; removal trails it.

The same pattern works when resource services have different JWKS cache ages. A receiver with an older set fetches when it sees the new kid. A receiver that has already fetched both keys can validate either token locally. During T2, both must be valid states. Do not assume every receiver refreshes at the same instant.

A useful clock calculation starts at the final old-signed token, not at deployment start. Suppose the last old token is signed at 12:10, the longest issued lifetime is 15 minutes, and every receiver uses Orionis's default 30-second verifier leeway. The earliest planned retirement is after 12:25:30. If any old signer remains during the T2 rollout, that time moves forward. A receiving service with a different leeway or an issuer with a longer TTL needs a later cutoff.

I would write that cutoff down before the rollout. “Wait fifteen minutes” is too easy to apply from the wrong starting point. Record when the final auth instance stopped using the old signer, add the longest relevant token lifetime and the receiver's accepted clock leeway, then consider T3. If the last old-signed token time is uncertain, keep the overlap longer.

Switching active_kid does not edit tokens already in circulation. Even after every auth instance reports the new active key, consumers can legitimately receive old-signed tokens until those tokens expire or their callers replace them.

Why the order matters with several instances

Imagine two auth-server tasks behind one load balancer. Task A already signs with new, but task B is still at T0 and publishes only old. Billing receives a new token, sees an unknown kid, and fetches JWKS. If that fetch reaches task B, the new public key is absent and the request fails. Making T1 a completed rollout before T2 removes that mismatch.

The reverse mismatch happens at retirement. If one auth instance at T2 still signs with old while another instance at T3 no longer publishes old, a fresh token can be invalid as soon as it is issued, depending on which instance serves the JWKS fetch. T3 must wait until all signers have switched, not merely until a deployment controller has started replacing them.

This is a distributed-systems problem hiding inside a configuration change. The useful unit of progress is “every serving instance has reached the required state, and the next state remains valid while instances overlap.” A single green deployment event does not establish that invariant by itself.

This is also why I would verify the JWKS view through every serving instance or through an equivalent deployment check, rather than relying on one load-balanced response. A single successful fetch proves only that one response contained the key.

Treat kid as part of the key's identity

Each signer needs a distinct kid. Orionis rejects duplicate signer IDs when constructing the server. Its JWKS provider also rejects duplicate IDs in a fetched set. Those checks are useful, but they do not make reusing an old ID for different key material a good rotation strategy.

A receiver with a fresh cached key for kid=old will try that public key when it sees a token header naming old. If the auth server has silently changed the private key behind the same ID, the receiver has no reason to fetch yet: the key is present and its cache is fresh. Signature verification fails until it refreshes. A new key deserves a new kid, while the old key keeps its ID during the overlap.

The token header's kid is only a selector. It does not prove that a token came from Orionis, and it is not a URL to fetch. The receiving API uses its configured JWKS endpoint, obtains the public key for that ID, checks the algorithm, and verifies the signature and claims. The JWKS validation notes describe that path.

What changes in the config

The standalone server accepts a keys list and an active_kid. Here is the relevant part of a T1 configuration; the rest of the server configuration is omitted:

{
  "active_kid": "old",
  "keys": [
    {
      "kid": "old",
      "private_key_pem_env": "ORIONIS_SIGNING_KEY_PEM_OLD"
    },
    {
      "kid": "new",
      "private_key_pem_env": "ORIONIS_SIGNING_KEY_PEM"
    }
  ]
}

For T2, change only active_kid to new. Use distinct kid values; reusing one for different public keys would make cache behavior ambiguous.

I prefer to make T1 and T2 separate reviewable diffs. The T1 diff adds the new key reference while leaving active_kid unchanged. The T2 diff changes one value: active_kid. This gives a clear place to stop if key injection or JWKS publication is wrong.

The standalone configuration validates that active_kid names a loaded signer. A missing PEM environment variable fails key loading; conflicting private_key_path and private_key_pem_env settings are rejected. Catching these at startup is much safer than discovering a partially configured key set through failed API calls.

The private Ed25519 PEM values belong in a secret manager. The production ECS template injects them as environment variables from AWS Secrets Manager. AppConfig contains only the variable names and the non-secret configuration, such as key IDs and client secret hashes.

Private-key access and public-key publication have opposite requirements. Only the auth-server process should read the private PEM. The JWKS endpoint deliberately exposes the public counterpart so resource services can verify tokens. At T1, check secret injection and public JWKS output separately. A public key appearing in JWKS is expected; a private PEM appearing in logs or configuration history is not.

A rollout I would put in a runbook

I would make each transition observable and pause after it:

1. Prepare new. Generate and store a new Ed25519 PKCS8 private key through the normal secret-management path. Choose a distinct kid; keep the old key available.
2. Deploy T1. Add both key references, keep active_kid=old, and replace every serving auth task. Confirm every JWKS response can advertise both IDs while newly issued tokens still name old.
3. Deploy T2. Change active_kid to new and replace all serving auth tasks. Confirm new tokens name new, old-signed tokens still work, and a resource service validates new-signed tokens.
4. Wait for the old acceptance window. Start the clock after the last old-signing instance is gone. Use the longest token TTL and receiving leeway, including custom verifier configurations.
5. Deploy T3. Remove the old signer and its public key, replace all auth tasks, and confirm JWKS contains only new. Verify a fresh authorized request again.

The checks need to reach the same topology that serves real traffic. If a load balancer hides individual task addresses, use deployment state plus a per-instance check or equivalent telemetry. Repeating curl through the load balancer does not guarantee that you sampled every instance.

For the T2 check, I would keep a known-good old token from before the switch and obtain a new-signed token after it. Both should validate at a resource service while still within their valid time windows. Keep those bearer values private; record the result and the relevant kid, not the raw tokens, in the rollout log.

A rollback has its own overlap

If the new signer causes a problem at T2, I can move active_kid back to old while continuing to publish both public keys. Rolling tasks back to old signing stops new tokens from using the problematic key, but it does not erase tokens already signed by new. Removing the new public key immediately would break those still-valid tokens.

The same waiting rule applies in reverse: keep the key that verified already-issued tokens until their acceptance window closes, unless an incident requires deliberate early invalidation. A rollback plan should include the JWKS contents, not only the active signer field.

Rollback has a deployment-order requirement too. If T2 already issued tokens under new, rolling back to a task that serves only the original T0 JWKS set is unsafe. The useful rollback target is T1: publish both keys and sign new tokens with old. That preserves verification for new-signed tokens already circulating while stopping further issuance under new.

If the problem is that some receivers have not learned the new key, restoring T1 buys time to repair their JWKS path. If the new private key may be compromised, the decision changes: continuing to publish its public key allows tokens signed under it to validate, including potentially malicious ones. That is an incident with an explicit availability-versus-revocation decision, not a routine rollback.

Remember how configuration reaches the process

In the ECS setup described in the deployment guide, Orionis fetches AppConfig once at startup, with a 30-second deadline. Activating a new AppConfig version alone does not update running auth tasks. Each transition from T0 through T3 requires a rolling task replacement, followed by a check that the active signer and JWKS contents match the intended state.

The startup model matters when reading deployment dashboards. An AppConfig deployment can be COMPLETE while old ECS tasks still run with the configuration they fetched earlier. A new task can also fail startup if it cannot read AppConfig or the injected key environment variable. Treat configuration activation and task replacement as separate events, and verify live signer state after both complete.

For T1, verify that every serving auth instance advertises both keys. For T2, request a new token and inspect its kid, then make an authorized request through a resource service. Keep checking that a token signed just before the switch still works until it expires.

For a simple smoke check, inspect the set of public key IDs:

curl -s https://auth.example.internal/.well-known/jwks.json |
  jq -r '.keys[].kid'

At T1 and T2, expect both IDs. At T3, expect only the new one. The server startup log also reports active_kid and signer count. Inspect a freshly issued token's header locally to confirm which kid signed it; avoid pasting a real token into an external decoder.

If AppConfig is unavailable at startup, the ECS task does not silently fall back to a previous config. In the deployment described here, signing-key PEMs are injected by ECS from Secrets Manager, while the application reads AppConfig at startup. That makes task health, startup logs, and an authorized end-to-end request useful checks after each rollout.

Diagnose a failed switch

The first useful question is: which token kid failed, and which keys did the relevant JWKS response contain? Those facts narrow the search without exposing the token itself.

Symptom	Place to inspect first
New token names new, but some JWKS responses contain only old	Incomplete T1 rollout or an old auth task still serving JWKS
JWKS contains new, but one receiver rejects only new tokens	Its JWKS URL, cache refresh, algorithm metadata, or signature verification
Old-signed tokens fail immediately after T3	Old public key retired before their acceptance window ended
New auth tasks never become healthy	AppConfig startup fetch, secret injection, active_kid, or key-loading error

This is a starting point, not a diagnosis from an HTTP status alone. The Gin middleware returns 401 for token verification errors, including when an unknown kid cannot be fetched. The caller's response does not say whether the signature was bad or JWKS was unreachable. Correlate it with server-side errors and deployment state without logging bearer tokens.

One controlled pre-production exercise makes the invariant tangible: intentionally leave one auth instance at T0 and try a token signed under new. The failed case shows why T1 must finish everywhere first. Then put every instance at T1 and repeat; both old and new tokens should have a verifiable public key. The exercise tests the rollout order before a real key change.

The smoke test that is easy to miss

A single successful request after T2 proves less than it seems. The caller may still have an old token in its cache, so the request exercises only the old key. Or the request may hit a receiver whose JWKS cache already contains both keys, leaving another cold receiver untested.

I would test four paths, each for a different reason:

Path	What it tells me
Old token through a warm receiver	The old key remains published and accepted during overlap
Newly issued token through a warm receiver	A receiver with cached JWKS can discover or already use new
Newly issued token through a fresh receiver	Cold JWKS fetch returns both keys and the token validates
Newly issued token after T3	The retired key is gone without breaking the active one

For the second path, make sure the token was actually minted after the switch and inspect its header locally. The orders client in the runnable example caches access tokens, so repeatedly running a long-lived caller is not a reliable way to force a new token at an exact second. Request one directly from /oauth/token or use a fresh client instance for the smoke check. Do not treat a green invoice call with an old token as proof that the new signer works.

For the third path, a fresh resource-service instance can expose configuration mistakes that a warm instance masks. It must fetch JWKS from the configured URL before validating its first protected request. That path also tells you whether the auth server is reachable from the receiver's network environment, not just from the operator's shell.

These are tests of the transition, not a demand to make every production consumer restart. The point is to cover both key IDs and both cache states while the overlap exists. Once T3 completes, the old-token test should end by natural expiry; it should not become a permanent fixture expecting an old key to remain published.

This sequence is for routine rotation. A suspected private-key compromise is a different incident: retaining the old public key keeps tokens signed with that key verifiable during the overlap. Respond according to the compromise and revocation requirements of your system.

The central rule is publish the new public key before using the new private key, and retain the old public key until old tokens are gone. The architecture notes and production security guide contain the supporting model