Short answer: freeze the credential, preserve an evidence window, rotate it, then search every authenticated event to bound the key's blast radius.
I run a one-person SaaS, so an incident runbook has to be executable while I am also answering support mail. The constraint that changes the design is blast radius: one leaked key must not unlock every environment, queue, and billing action. A short, boring sequence beats a clever dashboard when the clock is running.
Build the evidence window before changing state
Start a timeline in UTC. Record when the key was discovered, where it appeared, who had access to that location, and the last known legitimate request. Export the relevant log partitions before deletion or rotation changes the evidence. Keep the original export read-only and hash it. I've found that this boring ledger prevents a later argument about which clock was authoritative, especially when a queue worker and the API gateway write different timestamps.
The first pass should answer three questions: which principal used the key, which resources did it touch, and what is the earliest suspicious timestamp? A request ID, source network, user agent, action, and response status are more useful than a pile of raw text. Redact the secret value itself; a fingerprint or key ID is enough to correlate events.
Contain first.
This is the small TypeScript shape I use for an evidence record. It deliberately accepts an unknown event because log schemas drift between services.
type AuthEvent = {
at: string;
keyId?: string;
requestId?: string;
action: string;
resource?: string;
sourceIp?: string;
status: number;
};
export function inWindow(event: AuthEvent, start: Date, end: Date): boolean {
const time = Date.parse(event.at);
return Number.isFinite(time) && time >= start.getTime() && time <= end.getTime();
}
Do not “clean up” the logs first. That makes later claims hard to defend.
How should a leaked API key runbook report, rotate, and search logs?
The order is intentional. Report the compromise to the person or team that owns the account, then disable or restrict the exposed key, and only after the evidence window is captured create its replacement. Rotation without reporting loses context; reporting without containment leaves the door open.
For a small service, the report can be a ticket with a fixed schema: discovery time, key ID fingerprint, suspected source, containment action, rotation time, affected environments, and an owner. Attach the immutable log export and its hash. If a third party issued the credential, use its abuse or security channel as well. Your mileage may vary on retention, so write down the policy that actually applies instead of assuming seven or thirty days.
After rotation, search by key ID, principal, and request ID rather than by the secret string. Search the pre-rotation interval and a small post-rotation interval to catch delayed jobs. Count unique resources and actions, then inspect outliers manually. A single read of a public health endpoint is different from a permission-changing call, even if both returned 200.
type Finding = { resource: string; action: string; firstSeen: string; lastSeen: string };
export function summarize(events: AuthEvent[], keyId: string): Finding[] {
const grouped = new Map<string, Finding>();
for (const event of events) {
if (event.keyId !== keyId) continue;
const resource = event.resource ?? "unknown";
const bucket = `${resource}:${event.action}`;
const prior = grouped.get(bucket);
if (!prior) {
grouped.set(bucket, { resource, action: event.action, firstSeen: event.at, lastSeen: event.at });
} else {
prior.firstSeen = event.at < prior.firstSeen ? event.at : prior.firstSeen;
prior.lastSeen = event.at > prior.lastSeen ? event.at : prior.lastSeen;
}
}
return [...grouped.values()];
}
The output is a lead list, not proof of damage. Verify state changes in the source system, and preserve both positive and negative results.
Keep one credential from becoming a platform-wide incident
The design work happens before the drill. Give each environment and automation job a separate key. Scope permissions to the smallest action set, put expiry on short-lived credentials where possible, and require a second control for destructive operations. A leaked read-only staging key should not reach production data; that boundary is worth more than another alert.
I also add a canary action to the runbook: a harmless, uniquely named request that should appear in the audit stream. If the canary is absent, the search is incomplete. If it appears under an unexpected principal or network, the investigation expands. This catches the common failure where an analyst searches an application log that never recorded authentication events.
The catch is operational overhead. More keys mean more renewal paths, and strict scopes can block a deploy at the worst moment. That is a real trade-off. Keep a documented break-glass path with separate approval and logging; do not turn the everyday key into the break-glass key for convenience.
At higher volume, stream audit events into a write-once store, normalize timestamps and action names at ingestion, and attach a stable credential ID to every request. Add a scheduled drill that plants a test credential, detects its use, and measures time to containment. Alerting should page on high-impact actions, while low-risk reads remain searchable without waking someone at 03:00.
I would keep the decision rule simple: choose the architecture that makes one credential's maximum reachable set obvious in a single query. If a system cannot show that set, it is not ready for a leaked-key drill. For teams that cannot operate separate scopes or durable audit logs, a managed identity system may be a better fit; for highly regulated workloads, retain an auditable self-hosted control plane even if it costs more operator time.
Top comments (0)