A zero-downtime key rotation changes the incident-response choice: keep both credentials valid long enough to avoid refused checkout traffic, but treat every extra minute of overlap as unbounded exposure until evidence narrows it. Short answer: revoke the leaked key as soon as the replacement is confirmed healthy, preserve immutable request and control-plane records, and reconstruct impact by intersecting credential identity, time, principal, resource, and observed side effects. A usage total alone cannot establish what the key touched.
I have been paged for missed jobs and duplicate deliveries. That history makes me distrust a clean aggregate graph during a credential incident: retries can inflate calls, delayed queue work can appear after the initiating request, and a successful HTTP status does not prove that an order or refund changed. For an e-commerce API key used by checkout workers, the operational question is narrower than "was there traffic?" It is: which authenticated actions occurred during the exposure window, against which stores and orders, and which durable changes followed?
What did the API key actually touch after the leak?
Start a timeline with four independently recorded moments: the earliest plausible disclosure, the replacement key's creation, the last accepted request using the old key, and revocation. Do not define the first moment from the first suspicious log entry. Exposure may precede observation. Preserve the original timestamps, clock source, retention policy, and export hash; later normalization should create a derived copy rather than rewriting evidence.
Then join records on stable identifiers. The useful request record carries a non-secret credential identifier or fingerprint, authenticated principal, request ID, route or operation class, target account or resource ID, result, source network metadata, and event time. Never log the key itself. Control-plane audit events should show creation, scope changes, and revocation. Application events or database audit records establish whether a request produced a durable side effect. Queue metadata connects an accepted request to work that ran later.
This is the invariant: blast radius is the set of evidenced resources and effects, plus a clearly labeled unknown set created by missing telemetry. It is not the count under a spike.
Counts mislead.
A practical evidence matrix keeps claims honest:
| Question | Primary evidence | Corroboration | What it cannot prove alone |
|---|---|---|---|
| Was the old key accepted? | Authentication or gateway request record | Usage series by credential ID | That state changed |
| Which store or order was addressed? | Authorized principal and resource ID | Application trace or structured event | The final outcome |
| Did a mutation persist? | Database audit or domain event | Queue completion and request ID | That no later rollback occurred |
| Did activity continue after revocation? | Authentication decisions after the revocation event | Denied-request series | Earlier access was harmless |
The overlap window is an availability budget
Zero-downtime rotation usually requires a bounded dual-key interval: deploy consumers with the replacement, verify that they are using it, revoke the old credential, then watch denied requests for stale consumers. In a checkout path, immediate revocation reduces exposure but can refuse legitimate traffic from a worker that has not reloaded configuration. A long overlap protects availability while preserving an attacker's access. Neither risk disappears; the incident commander chooses which one to bound.
Use an explicit ceiling rather than "wait until things look stable." The ceiling can be expressed as a maximum overlap duration and a maximum accepted-request count for the old identifier. The exact values belong to the service's threat model and error budget, not to a universal recipe. High-impact write scopes, such as refunds or address changes, justify a shorter overlap and stronger verification than read-only catalog access. If the leaked credential permits privilege changes or evidence deletion, isolate or revoke first; the normal availability trade-off does not apply.
A five-minute usage bucket is useful for locating a burst, but it is a lossy projection. It can hide two actors behind one count, merge retries with new work, and cross the revocation boundary. Keep raw event times for the forensic path even if dashboards aggregate them.
Make rotation state explicit in the code path
The preventative path should emit a stable key identifier and request ID at authentication, without retaining secret material. It should also distinguish an old key accepted during overlap from a revoked key. This small Go example shows the shape; storage and cryptographic verification are deliberately interfaces because those choices depend on the deployment.
package authn
import (
"context"
"errors"
"time"
)
type KeyState string
const (
Active KeyState = "active"
Overlap KeyState = "overlap"
Revoked KeyState = "revoked"
)
type KeyRecord struct {
ID string
Principal string
State KeyState
ExpiresAt time.Time
}
type Store interface {
Verify(ctx context.Context, presented string) (KeyRecord, error)
}
type AuditSink interface {
Authentication(ctx context.Context, keyID, principal, requestID string,
state KeyState, accepted bool, observedAt time.Time) error
}
func Authenticate(ctx context.Context, store Store, audit AuditSink,
presented, requestID string, now time.Time) (KeyRecord, error) {
rec, err := store.Verify(ctx, presented)
accepted := err == nil && rec.State != Revoked && now.Before(rec.ExpiresAt)
// Record the non-secret identifier even for overlap traffic.
_ = audit.Authentication(ctx, rec.ID, rec.Principal, requestID, rec.State, accepted, now)
if !accepted {
return KeyRecord{}, errors.New("credential rejected")
}
return rec, nil
}
The audit sink should fail independently from the business request only if that behavior has been chosen in advance. Failing checkout because the evidence pipeline is briefly unavailable can create a larger incident; silently losing all security records creates a different one. A common design is durable local buffering with backpressure and an alert before capacity is exhausted. The important property is observable loss: responders must know which interval is incomplete.
There is another trap in the example. If verification fails before a key ID can be derived safely, do not invent one or record the presented value. Emit a rejection with the request ID and a reason class, then correlate it through other metadata. Secrets-management guidance recommends auditing who requested and used secrets while ensuring logged data does not contain the secret itself.
Reconstruct impact without double-counting retries
Build the incident dataset from raw records, not screenshots. Normalize clocks, retain both source and ingestion timestamps, and group retries by an idempotency key or stable operation identifier when one exists. For queued work, follow the request ID into enqueue, lease, completion, and domain-event records. A missing completion is not automatically a failed action; it is an unresolved branch.
I use three result classes because a binary affected/not-affected label overstates weak evidence:
- Confirmed: the old credential authenticated and a resource read or durable mutation is evidenced.
- Attempted: a request is evidenced, but authorization, completion, or persistence failed.
- Indeterminate: retention gaps, sampling, clock uncertainty, or missing correlation prevent a defensible result.
Run the join twice. First, enumerate by credential ID inside the broad exposure interval. Second, enumerate durable effects by principal and resource, including delayed jobs whose completion falls outside that interval. Compare the sets. This catches work accepted before revocation but executed afterward. It also exposes unrelated activity by the same service principal that would be wrongly attributed if credential identity were absent.
Idempotency changes the consequence, not the evidence. Two delivery attempts with one idempotency key may produce one refund, but both attempts matter when reconstructing actor behavior. Preserve attempt count and final effect separately.
Keep both views.
When this method is insufficient
Credential-level reconstruction works only if identities survive aggregation. If all keys for a service share one label, access logs are sampled, resource identifiers are discarded, or audit retention is shorter than detection time, the evidence cannot support a precise boundary. State that limit. Expand the working blast radius to every resource authorized by the credential during the unresolved interval, then use downstream domain records to reduce it where possible.
The method also needs adjustment for stolen session tokens minted by the key, delegated credentials, caches that continue serving authorized responses, and systems where authentication and authorization happen in different trust domains. In those cases, trace the derived identity chain and use the latest possible expiry as the outer time bound. Do not stop at the parent key's revocation event.
Finish recovery with two watches: rejected use of the old identifier, which reveals stale workloads or continued hostile attempts, and successful use of the replacement, segmented by workload. Keep alerts on both. The spend ceiling still matters because verbose forensic storage and high-cardinality metrics have real operational cost, but dropping identity or resource dimensions to control that cost converts a known storage bill into an unknown incident boundary. Prefer tiered retention: queryable recent events, immutable archived evidence, and aggregate series for long-term trends.
The closure test is plain: every authorized operation is mapped to confirmed, attempted, or indeterminate; delayed work is reconciled; the old key is rejected; the replacement is healthy; and evidence gaps are recorded as risk rather than erased by an attractive chart.
Top comments (0)