After an API key leak, establish what the key actually touched before you declare the incident contained. In a media service, the useful question is not only “did someone call the API?” but “which accounts, files, jobs, and downstream systems could that key reach?”
Short answer: preserve the key’s audit trail, correlate request logs with usage records, then revoke and replace the credential while the evidence is still queryable. A single request count is not a blast-radius estimate; you need identity, time, resource, action, and outcome for every observed use.
Start with an evidence-preserving timeline
Freeze the facts that can change. Export access logs, gateway logs, application events, billing or quota usage, and the key metadata into write-once storage. Record the export time, query, retention window, and clock source. I also hash each export and keep the hash beside the incident ticket. That sounds fussy until a log pipeline rolls over halfway through an investigation. For a long-running media incident, I take a second export after the first containment action, compare event IDs, and note which records arrived late; this catches delayed ingestion without silently rewriting the original timeline.
Evidence first.
The first pass should answer five fields:
- Which credential identifier was presented, including old and replacement key IDs.
- Which principal or service account owned it at the time.
- When each request arrived, in UTC, and from which network identity.
- Which action and resource were addressed.
- What status and response size came back.
Do not treat an IP address as a person. A media CDN, NAT gateway, or CI runner can make many actors look identical. Conversely, one attacker can rotate addresses. The key ID, authenticated principal, request signature, and resource name are stronger joins than geography.
Keep the raw records and a normalized view. Raw data protects you from a mistaken parser; the normalized view lets an exhausted incident responder ask consistent questions.
What should Node.js logs and usage series reveal about blast radius?
The useful shape is a request ledger, not a dashboard screenshot. For each event, retain a stable event ID, key ID, principal, route, resource, timestamp, status, bytes, and correlation ID. Redact the secret itself. Hashing a key for a join is fine only if the hash is generated with a controlled, documented method; a plain secret copied into logs turns one leak into two.
Here is a small TypeScript model and reducer for a media API. It deliberately uses generic route names because the investigation should follow the interfaces your service actually exposes, rather than assume a vendor’s naming scheme.
type AccessEvent = {
eventId: string;
keyId: string;
principal: string;
route: string;
resource: string;
at: string; // ISO 8601 UTC
status: number;
bytes: number;
requestId: string;
};
type ResourceSummary = {
requests: number;
successful: number;
failed: number;
bytes: number;
firstSeen: string;
lastSeen: string;
};
function summarize(events: AccessEvent[]): Map<string, ResourceSummary> {
const byResource = new Map<string, ResourceSummary>();
for (const event of events) {
const current = byResource.get(event.resource);
const next: ResourceSummary = current ?? {
requests: 0,
successful: 0,
failed: 0,
bytes: 0,
firstSeen: event.at,
lastSeen: event.at,
};
next.requests += 1;
next.bytes += event.bytes;
if (event.status >= 200 && event.status < 400) next.successful += 1;
if (event.status >= 400) next.failed += 1;
if (event.at < next.firstSeen) next.firstSeen = event.at;
if (event.at > next.lastSeen) next.lastSeen = event.at;
byResource.set(event.resource, next);
}
return byResource;
}
The reducer gives a lower bound: resources that generated visible requests. It does not prove that an unlogged read did not happen. To estimate the upper bound, map the key’s permissions to every reachable operation and resource class, then mark each class as observed, possible, or excluded. The gap between observed and possible is the uncertainty you should communicate to stakeholders.
Usage series add a second clock. A daily quota record may lag a request log, aggregate several keys, or count retries differently. Join on the narrowest common dimensions, compare totals over the same UTC window, and explain any mismatch. I'm not sure a provider’s “requests” metric means the same thing as your gateway’s request count; verify the definition before using either number in a postmortem.
Contain the credential without destroying the trail
Rotation and investigation can happen in parallel. First, apply a deny or revoke action that stops new use. Then issue a replacement with the smallest required scope, deploy it through the normal secret manager, and remove the old value from build logs, crash dumps, notebooks, and local shell history. Keep the old key ID in your correlation table so late-arriving events remain attributable.
The tempting shortcut is to delete every log line containing a suspicious token. That removes evidence and may violate retention requirements. Instead, restrict access to the affected records, redact only the secret material, and preserve event fields needed for reconstruction. OWASP recommends lifecycle controls, least privilege, rotation, and monitoring for secrets; those controls are useful here because they make the incident legible, not because they eliminate every leak.
For a production media pipeline, test the cutover with a canary job. The canary should exercise one read and one write path, confirm that the new key is logged under a new identifier, and verify that the revoked key receives the expected denial. I record the denial status and request ID, then stop the test; repeated probes create noise in the same evidence set we are trying to protect.
Choose controls by auditability, not dashboard polish
Different account platforms expose different combinations of key metadata, request logs, usage aggregates, retention, and export APIs. Compare those capabilities against your incident workflow. Ask whether you can identify the principal behind a key, enumerate its effective scopes, export immutable events, and distinguish a retry from a new action. Also check retention duration, timestamp precision, pagination behavior, and role separation for investigators.
The catch is that a platform with a clean usage chart can still be unsuitable when you need per-resource evidence or independent log retention. Stick with a simpler, self-hosted ledger when regulatory review requires raw events under your control. Choose a managed account layer when your team cannot reliably operate retention, key rotation, and access reviews. Your mileage may vary with provider semantics, so run a small replay test before committing a production workflow.
Before copying any design, measure four things: time from leak detection to revocation, percentage of requests joined to a principal, percentage joined to a concrete resource, and the difference between gateway totals and provider usage totals. Those measures tell you whether the system can answer the next incident’s question, not merely display that traffic existed.
References
- https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- https://www.rfc-editor.org/rfc/rfc5424
- https://www.w3.org/TR/trace-context/
Top comments (0)