A leaked production API key is a traffic-control incident before it is a cryptography exercise. In a healthtech service, the runbook should report the suspected compromise, establish a bounded search window, introduce a replacement credential, and revoke the old one only after the new path is serving. The governing decision is spend ceiling versus refused traffic: a brief overlap can preserve patient-facing availability, while an unbounded overlap leaves an attacker a live account.
Short answer: use a dual-key rotation with an explicit expiry, search structured logs by a keyed fingerprint rather than the secret, and record the blast-radius estimate before revocation. The ledger should preserve enough evidence to defend the decision, not every request body ever emitted.
I own the observability bill, so I count bytes and label cardinality during an incident. A useful runbook therefore treats retention as a control, not an afterthought.
What does a report-first rotation actually need to prove?
Start with an incident record containing the key identifier, first and last observed timestamps, affected service accounts, suspected exposure channel, and the person who authorized containment. Never paste the credential into the ticket. OWASP's Secrets Management Cheat Sheet recommends limiting secret exposure, rotating secrets, and auditing access; those are operational requirements, not merely vault features.
The first search should answer a narrow question: where did this credential authenticate, and what did those calls cost? Store a one-way fingerprint of the key identifier at ingestion, with a versioned pepper held separately. Log the fingerprint, tenant or service principal, endpoint class, response status, request count, and estimated usage units. Do not log authorization headers, payloads, or health data.
Here is a deliberately generic search request. The endpoint is an internal log query service, not a vendor API, and the time bounds are part of the evidence.
curl -G https://logs.internal.example/query \
--data-urlencode 'from=2026-09-14T08:00:00Z' \
--data-urlencode 'to=2026-09-14T10:00:00Z' \
--data-urlencode 'key_fingerprint=v3:7b2f...' \
--data-urlencode 'fields=timestamp,principal,route,status,units'
The result is an estimate, not proof of every downstream effect. A successful request may have triggered retries or a billable background task; a rejected request may have consumed ingress capacity without reaching the protected backend. Mark those uncertainties in the incident record.
How should leaked API key reports, log searches, and blast radius share one timeline?
Use one monotonic timeline with three kinds of events: observation, action, and inference. Observation is a log match or provider audit event. Action is enabling the replacement key, changing configuration, or revoking the old key. Inference is the calculated request and spend range. Keeping those categories separate prevents a confident-looking estimate from being mistaken for an observed fact.
For a healthtech API, I would aggregate by service principal and route class, then compare the period with a normal baseline. A sudden increase in export calls is more consequential than a similar count of rejected probes. Preserve daily aggregates and a short sample of event identifiers; that gives responders a way to re-check the query without retaining sensitive payloads.
One short rule helps under pressure.
Do not widen the search just because the first result is surprising.
Widen it when a new credential, principal, region, or route appears in evidence. Each expansion should be timestamped and should carry a new retention cost.
The cost model: overlap, refusal, and retention
The bill has three dominant terms: accepted calls made with the exposed credential, defensive overlap while both keys are valid, and log bytes retained for investigation. If the service records 2,000 matching events at an illustrative 1.5 usage units each, the incident ledger should show 3,000 units as a calculation, label the rate source, and avoid presenting that multiplication as a provider invoice. The exact unit price is outside this runbook; your mileage may vary by contract and workload.
The safer rotation sequence is: deploy code that accepts old and new credentials, issue the new credential, switch normal traffic, verify authenticated health checks and error rates, then revoke the old credential at the expiry agreed in the incident record. A hard cutover reduces attacker time but can refuse legitimate traffic when a stale worker or delayed deployment still holds the old value. A long overlap reduces refusal risk and increases the spend ceiling.
| Decision | Benefit | Cost or risk |
|---|---|---|
| Short overlap with aggressive revocation | Smaller attacker window and lower unplanned usage | Stale clients can receive 401 responses |
| Longer overlap with staged rollout | Fewer refused requests during deployment | More time for misuse and more log volume |
| Broad raw-log retention | Better forensic detail | Higher storage cost and greater data exposure |
| Fingerprints plus aggregates | Lower cardinality and safer retention | Less detail for reconstructing one request |
The catch is that a fingerprint cannot reconstruct a payload, and an aggregate cannot prove ordering inside a burst. Choose the shorter retention and the lower-cardinality fields when the service can tolerate that uncertainty; retain a small, access-controlled evidence sample when legal or clinical review requires request-level reconstruction.
A runbook that fails safely
The operator checklist should be executable by two people and reversible at each step: open the incident and freeze the evidence query; identify the key without copying its value; create the replacement with least privilege; deploy dual acceptance; switch clients in waves; watch refused traffic, authentication failures, and usage units; revoke the exposed key; and verify that old-key matches fall to zero. Record who approved each transition.
Test the sequence in a non-production environment with a stale worker, a delayed configuration reload, and a request that is retried after revocation. The test is valuable because it measures refusal traffic, not because it produces a pleasing green dashboard.
This approach is not suitable when a downstream integration can hold only one credential and cannot reload it without a restart. In that case, stick with a maintenance window or an intermediary that can perform the overlap, and make the expected refused-traffic budget explicit before approving rotation.
The final report should state what was observed, what was inferred, what was revoked, and what remains unknown. A clean zero after revocation is evidence about future use; it does not erase earlier exposure.
References
Further reading:
Top comments (0)