Rotate a production API key with an overlap period, and audit auto-recharge or post-paid billing separately from credential access. For a developer tools service, the deciding constraint is whether finance and the incident responder can reconstruct access and spending across that transition without interrupting legitimate traffic.
TL;DR: With a prepaid balance and auto-recharge, investigate the chain from funding attempt to usable balance; with post-paid billing, investigate the chain from usage attribution to the account's payment obligations and access policy. Neither funding model proves that a replacement key works, and a successful key rotation proves nothing about payment settlement. Keep those checks separate.
This runbook treats auditability as the selection criterion. The proposed controls below are design choices, not promises about an unnamed provider's billing behavior; access suspension rules, reporting delays, and support for overlapping keys must be confirmed for the service you actually use.
What page fired during the rotation?
Start the postmortem with the signal that reached the responder. Was it an authentication failure after deployment, a failed funding attempt, or a balance warning? Those are different investigations, even if they appeared during the same change window and the dashboard gives them one red status.
Distrust the dashboard.
A request authenticated with the new credential is evidence about that request. It does not establish that every worker has received the replacement, that the funding instrument is usable, or that the displayed spending total includes every relevant request. Conversely, a payment alert does not establish that the new secret is wrong. Replacing credentials again before distinguishing those cases would add another change to reconstruct while leaving the original question unanswered.
Use a deliberately small example: a developer tools service has 2 worker groups, an old credential identified as key-old, and a replacement identified as key-next. These are illustrative labels, never secret values. Require the deployment record to identify which group changed and when, so an error can be attributed to a specific stage rather than vaguely to “the rotation.”
The OWASP Secrets Management Cheat Sheet addresses secret rotation, lifecycle management, and auditing. Apply those concerns here by recording the actor, operation, time, outcome, and nonsecret credential identifier for each transition. Keep the secret itself out of logs. Finance needs a traceable account relationship, not a copy of the token.
Where does API auto-recharge or post-paid billing failure show up for finance?
Auto-recharge is a funding mechanism for a prepaid balance: the investigation follows whether an attempted replenishment produced available funds. Post-paid billing moves the investigation toward usage already incurred and its subsequent billing and payment handling. The exact enforcement behavior belongs in the account's documented terms. Do not invent a universal suspension deadline or assume either model always stops requests at a particular point.
The distinction matters during rotation because a credential and its billing account are separate identifiers in this runbook's evidence model. Record their relationship explicitly. If both credentials use one account, revoking the old credential does not establish that the account's funding issue has been resolved; if the replacement is attached to a different account, a successful request alone does not establish that finance approved that destination.
| Funding arrangement | Evidence finance should request | Separate access question |
|---|---|---|
| Prepaid balance with auto-recharge | Funding attempt outcome, resulting balance evidence, account identifier, and timestamps | Did the replacement credential authenticate against the intended account? |
| Post-paid billing | Usage attribution, applicable billing period, invoice or payment status when available, and account identifier | Which documented account restrictions could affect continued access? |
These are proposed evidence requirements.
A provider may expose them at different levels of detail. If the available records cannot distinguish a funding attempt from completed replenishment, that gap limits what the responder can conclude, even when the dashboard looks reassuring.
Record the gap.
The trade-off is operational work. Keeping access records and financial evidence independently traceable requires more reconciliation than accepting a single status light, but it gives finance a way to investigate an unexpected charge without asking an engineer to paste secrets into a ticket. Compare funding modes on that basis before discussing price.
Stage the replacement behind an evidence gate
Before changing traffic, confirm that the credential system permits the intended overlap and that revocation can be performed independently. If it requires an immediate replacement with no overlap, this runbook's continuity strategy does not apply; a different deployment mechanism must be established before promising an uninterrupted rotation.
For a system that supports overlap, create the replacement with the permissions the workload requires, record its approved billing-account relationship, and deliver it through the service's secret distribution mechanism. Probe it using a permitted operation whose side effects are understood. A probe that creates billable work needs an explicit test budget and attribution, even though price is not the decision axis here. Then move one worker group. Retain the old credential while observing the replacement, and move the remaining group only after the first group produces attributable success evidence. This ordering buys a rollback path at the cost of temporarily retaining another usable credential. Keep that interval bounded by a documented observation plan; “we will revoke it later” is not a completion condition. In the illustrative deployment, suppose the first group authenticates with key-next while a scheduled task in the second group still references key-old: the first group's success cannot authorize revocation. The evidence record must identify the remaining consumer and its deployment state before the change proceeds, rather than treating one successful probe as coverage of the entire service.
The following Go example implements only the retirement gate. It checks evidence gathered by deployment, request observation, and finance reconciliation; it does not fetch secrets, rotate credentials, or assert that a payment has settled.
package rotation
import "errors"
type Evidence struct {
ChangeID string
OldKeyID string
NewKeyID string
ExpectedAccountID string
ObservedAccountID string
ExpectedGroups int
VerifiedGroups int
OldKeyUseStillObserved bool
AccessEvidenceRef string
FinanceEvidenceRef string
AuditRecordCommitted bool
}
func CanRetireOldKey(e Evidence) error {
if e.ChangeID == "" || e.OldKeyID == "" ||
e.NewKeyID == "" || e.OldKeyID == e.NewKeyID {
return errors.New("distinct credential identifiers required")
}
if e.ExpectedAccountID == "" ||
e.ObservedAccountID != e.ExpectedAccountID {
return errors.New("billing account attribution unverified")
}
if e.ExpectedGroups < 1 || e.VerifiedGroups != e.ExpectedGroups ||
e.OldKeyUseStillObserved {
return errors.New("deployment evidence incomplete")
}
if e.AccessEvidenceRef == "" || e.FinanceEvidenceRef == "" ||
!e.AuditRecordCommitted {
return errors.New("durable evidence references required")
}
return nil
}
For the illustrative deployment, ExpectedGroups is 2. A verified group must have both deployment confirmation and observed requests using the replacement; merely counting healthy processes would make this gate misleading. The caller must validate distinct group identities, evidence freshness, and the records behind the references.
A nonempty string isn't proof.
Define the observation interval before executing the change, accounting for the workload's normal request cadence and background jobs. A quiet interval cannot prove that an infrequent job has migrated. The gate deliberately refuses retirement while old-key use is still observed, but the observer's coverage determines how much confidence that condition deserves.
Verify continuity before closing the change
Check the request path and the financial path independently. On the request path, verify replacement-key traffic for every expected worker group, inspect authentication failures, and exercise any scheduled work that would otherwise escape the observation interval. On the financial path, confirm the approved account attribution and attach the relevant funding or billing evidence, including its timestamp and any known reporting delay.
Do not demand an invoice that does not yet exist.
For a post-paid account, the change record can identify the usage evidence available now and assign later reconciliation to an owner; it must not label that evidence as a final settled amount. For auto-recharge, distinguish a recorded replenishment attempt from evidence that the balance became available. These distinctions prevent a rotation ticket from making financial claims its records cannot support.
Revoke the old credential after the retirement conditions pass. Then verify that legitimate traffic continues with the replacement and, using a controlled request if appropriate, confirm the old credential is rejected. Record the revocation outcome. Until that step is evidenced, the access lifecycle remains open.
Route actionable alerts to the people who can respond: the service owner for credential rejection or incomplete rollout, and the designated finance owner for unresolved funding evidence. Agree on escalation when either threatens continuity. An alert should identify the account and change record without revealing a secret.
Roll back without losing the audit trail
Before revocation, an unsuccessful rollout can return traffic to the old credential if that credential remains trusted and usable. Preserve the failed stage's evidence and investigate its actual signal. A rollback of secret distribution does not reverse billable requests or repair a funding problem.
After revocation, do not assume the old credential can be reactivated. Follow the credential system's documented recovery procedure, which may require issuing another replacement. If compromise prompted the rotation, returning to the suspect credential is not an acceptable continuity plan; prepare another trusted credential path before moving traffic.
Close the change when access has migrated, revocation is evidenced, and finance has either reconciled the available records or accepted a named follow-up for evidence that arrives later. The useful choice between auto-recharge and post-paid billing is the one whose funding and access boundaries the team can explain from records during an incident. A green dashboard cannot supply that explanation.
References
- OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
Top comments (0)