A leaked API key creates two clocks: stop its authority now, and preserve enough evidence to calculate which media events still deserve billing attribution. Short answer: record the compromise, freeze the relevant identity and retention state, revoke the credential, issue a replacement with narrower scope, and search immutable request and business-event records from the earliest plausible exposure through revocation. Do not delay revocation while investigating. Do not infer the blast radius from application errors alone.
For a media backend, the expensive part of this runbook is usually evidence retention, not key generation. If the service processes 120 million play, impression, and conversion events per day, retaining a 600-byte security envelope for 90 days represents about 6.48 TB before replication and indexing; keeping the full 8 KB payload for the same period would represent about 86.4 TB. Those are illustrative arithmetic inputs, not benchmark results, but they expose the dominant term: retained bytes multiplied by retention duration, copies, and index amplification. The useful change is to retain a small, queryable attribution envelope for longer while placing full payloads behind shorter, policy-driven retention.
What evidence will settle the bill?
The incident question is not merely whether the stolen key was used. It is which accepted requests changed billable state, which duplicates were suppressed, which events were later reversed, and which downstream statements incorporated those events. An access log can establish that a request reached an edge. It cannot, by itself, establish that a ledger mutation committed.
The durable envelope should carry a credential identifier that is safe to log, never the secret itself; a request or event identifier; tenant and media-property identifiers; authenticated principal and granted scope; server receipt time; response class; idempotency key or deterministic event key; and the committed business outcome. For billing, retain the attribution window, content or campaign identifier, amount and currency where applicable, ledger transaction identifier, and reversal reference. This is an audit trail, not a payload archive.
A secret fingerprint deserves care. A plain hash of a low-entropy credential can become an offline lookup target, while logging even a prefix can expose useful material. Prefer an internal, random credential ID assigned at issuance and written into authentication audit records. If correlation must be derived, use a keyed construction under a separate audit key, restrict access to the result, and document its rotation semantics. OWASP's secrets guidance calls for auditing who requested and used a secret, identifying expired or unused secrets, and never logging the secret value.
One boundary matters. Billing evidence and security telemetry have different access patterns, but the join between them must be stable. The security side identifies which credential acted during the exposure window. The billing side identifies which of those authenticated event IDs committed, deduplicated, or reversed. A shared opaque event ID makes that join possible without copying sensitive payloads into every log system.
How should a leaked API key compromise report guide the log search?
Reporting first does not mean opening a ticket and waiting for approval. It means creating an incident record with an immutable timestamp, the credential ID, reporter, suspected exposure channel, earliest possible disclosure time, affected tenant, and evidence-retention hold. That record can be created in seconds and gives responders a common case ID before authentication state changes.
Then revoke. Immediately.
Revocation should invalidate the credential at every verification point, including caches, gateways, background consumers, and long-lived connections that would otherwise continue under previously accepted authentication. Replacement is separate: issue a new credential to the legitimate workload with least privilege, distribute it through the normal secrets mechanism, verify healthy authentication using its new credential ID, and retire any overlap deliberately. Never reactivate the compromised value to recover availability.
The ordering is an exactly-once problem in miniature. The report operation needs an idempotency key so repeated pages or automation retries converge on one case. Revocation also needs an idempotent state transition: active -> revoked succeeds once, while subsequent requests return the already-recorded revocation time. A transactional outbox can publish cache invalidations after the authoritative credential row commits, preventing the state in which responders see a revocation while a verifier never receives the change.
type RevokeCommand struct {
IncidentID string
CredentialID string
RequestedAt time.Time
}
func Revoke(ctx context.Context, db *sql.DB, cmd RevokeCommand) (time.Time, error) {
tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
if err != nil {
return time.Time{}, err
}
defer tx.Rollback()
var revokedAt time.Time
err = tx.QueryRowContext(ctx, `
UPDATE api_credentials
SET status = 'revoked', revoked_at = COALESCE(revoked_at, $1),
incident_id = COALESCE(incident_id, $2)
WHERE credential_id = $3
RETURNING revoked_at`,
cmd.RequestedAt, cmd.IncidentID, cmd.CredentialID,
).Scan(&revokedAt)
if err != nil {
return time.Time{}, err
}
_, err = tx.ExecContext(ctx, `
INSERT INTO auth_outbox (event_id, event_type, credential_id, occurred_at)
VALUES ($1, 'credential.revoked', $2, $3)
ON CONFLICT (event_id) DO NOTHING`,
cmd.IncidentID+":revoke", cmd.CredentialID, revokedAt,
)
if err != nil {
return time.Time{}, err
}
return revokedAt, tx.Commit()
}
The example assumes a unique constraint on the outbox event ID and an existing credential row; production code must distinguish a missing ID from a database failure. It also deliberately carries one incident timestamp rather than inventing different times in each subsystem. The audit record should capture request time and commit time when that distinction matters.
Search outward from authenticated requests
Set the initial window from the earliest credible exposure, not the first suspicious request. End it only after revocation has propagated to all verifiers and in-flight work has completed or been rejected. Record both bounds and their evidence. If disclosure time is unknown, widen the window to the last known-safe handling event, such as issuance or a verified secret scan; uncertainty is a reason to search more data, not to manufacture a precise start time.
Search in stages:
- Enumerate authentication decisions for the compromised credential ID, including denied attempts after revocation.
- Join accepted request IDs to ingestion acknowledgments and durable media events.
- Trace committed event IDs into deduplication records, attribution decisions, ledger entries, invoices, exports, and reversals.
- Store the result under the incident ID, together with query version, parameters, execution time, and dataset watermark.
Counts must balance.
For every stage, reconcile counts. Suppose the authentication audit shows 48 accepted requests, ingestion records show 47 durable events and one rejected schema, deduplication reduces those to 41 unique events, and the billing ledger contains 39 postings plus two nonbillable classifications. This is a worked example, not a claimed incident. Its value is the conservation rule: every accepted request reaches exactly one explained terminal category. A missing row is not zero impact. It is an unresolved discrepancy.
Late-arriving media events complicate the endpoint. A request accepted before revocation may enter attribution after the first query, while a replay may arrive with a previously seen idempotency key. Take a dataset watermark, rerun after the maximum documented processing delay, and compare by stable event ID. The final report should separate attempted use, authenticated use, accepted ingestion, unique business events, and posted financial effects instead of presenting one ambiguous affected-request number.
Clock skew is another trap. Use server receipt and commit timestamps to define enforcement and financial ordering; preserve client timestamps as claims, not authority. Correlate records with IDs whenever possible. Wall-clock proximity is a weak fallback.
Retention is a costed control
Keeping everything forever sounds defensible until access, deletion, replication, and discovery obligations arrive. A compact envelope changes the equation because its value lies in identifiers and outcomes, not in repeated media metadata or request bodies. Partition it by event date and tenant boundary, encrypt it, restrict queries through incident roles, and audit access to the audit store.
A practical calculation starts with daily event volume V, retained bytes per event B, days D, replication factor R, and an empirically measured index factor I: V x B x D x R x (1 + I). Measure B and I from the actual schema and storage engine. Compression, sparse indexes, and partitioning change physical cost, so arithmetic from logical rows is a capacity estimate rather than an invoice forecast. Price is not the decision rule; provable attribution coverage is.
Retention also has a compliance ceiling. PCI DSS 4.0.1 requires at least 12 months of audit-log history, with at least the most recent three months immediately available for analysis, for systems in its scope. That does not automatically make every media event PCI data or require one retention period for all records. Scope the control with compliance counsel and the qualified assessor, document the mapping, and apply stricter legal or contractual requirements where they exist. Data minimization still applies.
Scope first.
Keep the long-lived envelope, not the secret and not the full request body. Full payloads may contain personal data, editorial metadata, tokens, or content identifiers that add risk without improving credential-to-ledger proof. A short payload tier can support debugging and replay under tightly controlled access; the longer tier should preserve only fields required for authentication, idempotency, attribution, reconciliation, and audit.
This creates a deliberate failure cost. After full payload expiry, responders may prove that an event committed and affected a charge but may be unable to reconstruct every original field or replay a parser defect byte for byte. State that limitation in the retention decision. Preserve schema versions, canonical event hashes, validation outcomes, and deterministic attribution inputs to narrow the loss without retaining the entire body.
Prove the runbook before an exposure
A runbook that has never met delayed queues, cache invalidation, and partial telemetry is prose, not a control. Exercise it with a synthetic credential and synthetic media events outside production, then test production mechanisms without inserting fake billable activity. The drill should verify that reporting is idempotent, revocation reaches every verifier, replacement does not inherit excess scope, and an investigator can reconcile authentication decisions to final billing outcomes.
Define pass criteria before the exercise: a bounded propagation interval established by measurement; zero accepted requests after that bound; every pre-bound accepted request mapped to one terminal category; no secret values in collected evidence; and a second operator able to reproduce query results from the incident package. Avoid promising instant revocation unless the architecture measures and enforces it. Cache topology and disconnected workers make that word expensive.
Observe the machinery continuously. Alert on authentication with revoked credential IDs, outbox age, verifier-consumption lag, audit-write failures, unexplained reconciliation deltas, and retention jobs that remove data under an active legal or incident hold. Treat audit-write failure according to transaction risk: a high-value financial mutation may need to fail closed, while low-risk telemetry can enter a durable quarantine path. Either choice must be explicit and tested during dependency failure.
The final report should contain the timeline, scope, evidence queries and watermarks, count reconciliation, financial adjustments, credential replacement proof, containment verification, data-retention decisions, and follow-up owners. Separate observed facts from assumptions. For media billing, close the incident only when disputed events can be corrected through append-only adjustments or reversals; editing old ledger rows destroys the history the response is meant to preserve.
The end state is modest: the stolen authority is gone, legitimate processing uses a narrower replacement, and each potentially affected media event has an auditable disposition. Confidence comes from reconciliation, not from an empty error dashboard.
References
- OWASP, Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- NIST SP 800-61 Rev. 2, Computer Security Incident Handling Guide: https://csrc.nist.gov/pubs/sp/800/61/r2/final
- NIST SP 800-92, Guide to Computer Security Log Management: https://csrc.nist.gov/pubs/sp/800/92/final
- PCI Security Standards Council, PCI DSS v4.0.1: https://www.pcisecuritystandards.org/document_library/?category=pcidss&document=pci_dss
- RFC 2104, HMAC: Keyed-Hashing for Message Authentication: https://www.rfc-editor.org/rfc/rfc2104
Top comments (0)