DEV Community

Trkfpn392751
Trkfpn392751

Posted on

Leaked API Key Compromise Runbook Explained — Protecting Tenant Billing Attribution

#security #api #incidentresponse

TL;DR: A leaked API key runbook should report the compromise, freeze the evidence window, rotate the credential, and preserve an immutable key ID for the log search. For a marketplace, the response is incomplete until security impact and billing impact are both bounded. Never reuse the compromised key's identifier or overwrite its audit record; doing either turns a contained credential incident into an accounting dispute.

The ordering is deliberate. Reporting opens the incident and fixes a common clock. Capturing the key metadata gives responders a stable search handle. Revocation stops new use. Replacement restores the tenant's service without erasing which credential made earlier calls. Log search then establishes the earliest suspicious use, the latest accepted use, affected operations, and billable units. If the key can issue or revoke tenant-scoped credentials, treat those administrative actions as part of the blast radius too.

What should a leaked API key compromise report preserve?

A leaked API key is both an authentication secret and an attribution label. Rotation addresses the first role. It can damage the second if the platform updates one database row in place, redacts the old identifier from searchable records, or starts reporting replacement traffic under the same logical identity.

Do not log the secret itself. OWASP recommends keeping secrets out of logs and treating rotation as an auditable lifecycle operation. Store a non-secret, immutable key_id alongside each authorization decision instead. The incident record should capture that ID, the tenant, declared scopes, issuance time, revocation time, reporter, and a UTC evidence-window start before any destructive credential operation occurs.

The hard boundary is the authorization decision, not the time an application log happens to flush. A request admitted before revocation may finish afterward. A queued marketplace operation may execute much later. Record both the request acceptance time and the downstream job or ledger event ID, so the team can distinguish unauthorized admission from delayed processing of already accepted work.

This matters most when usage drives billing. Tenant ID alone is too coarse: several keys may belong to one tenant, and only one may be exposed. IP address is weak evidence because legitimate clients can share egress addresses and an attacker can change addresses. The stable join is key_id -> authorization event -> operation or job -> usage ledger entry.

Preserve identity. Revoke authority.

The incident sequence

Open one incident timeline in UTC and assign an incident commander. Record how the report arrived, but do not paste the leaked value into a ticket or chat. If the report includes the secret, move it to the approved restricted evidence channel and remove avoidable copies according to the organization's retention policy.

Then execute this order:

  1. Mark the credential as suspected and record its immutable key ID, tenant, scopes, and known exposure time. Set the search start earlier than the claimed exposure when repository history, build artifacts, or message retention could predate discovery.
  2. Snapshot or preserve the relevant audit, authorization, queue, and billing records under the incident's evidence-retention rules. Record query parameters and clock assumptions.
  3. Revoke the old key at the authorization source. Verify denial with the old credential through a controlled request that cannot create marketplace side effects.
  4. Issue a new key with the minimum scopes the tenant currently needs. Deliver it through the normal secret channel, never through the incident ticket.
  5. Search by old key_id, then correlate request IDs into job and billing records. Expand the search to credentials created, revoked, or permissioned by the old key.
  6. Reconcile accepted operations with ledger entries. Quarantine uncertain charges rather than silently deleting them, because deletion destroys the trail needed for correction and review.

Revocation must be idempotent. Repeating it should leave the credential disabled and should not create a second billing or audit event that looks like another compromise. Replacement issuance needs its own idempotency key as well; an impatient retry during an incident must not mint several active tenant credentials.

A minimal evidence query in Go

The following example uses generic SQL interfaces and assumes the platform already writes immutable authorization events. It searches by non-secret key ID and a bounded UTC interval, returns the fields needed for correlation, and leaves interpretation to the incident process. Parameterized queries matter here because incident input is still input.

package incident

import (
    "context"
    "database/sql"
    "fmt"
    "time"
)

type AuthEvent struct {
    OccurredAt time.Time
    RequestID  string
    TenantID   string
    Operation  string
    Decision   string
    Units      int64
}

func FindAuthEvents(ctx context.Context, db *sql.DB, keyID string, from, to time.Time) ([]AuthEvent, error) {
    if keyID == "" || !from.Before(to) {
        return nil, fmt.Errorf("invalid evidence window")
    }

    rows, err := db.QueryContext(ctx, `
        SELECT occurred_at, request_id, tenant_id, operation, decision, billable_units
        FROM authorization_events
        WHERE key_id = ? AND occurred_at >= ? AND occurred_at < ?
        ORDER BY occurred_at, request_id`, keyID, from.UTC(), to.UTC())
    if err != nil {
        return nil, fmt.Errorf("query authorization events: %w", err)
    }
    defer rows.Close()

    var events []AuthEvent
    for rows.Next() {
        var event AuthEvent
        if err := rows.Scan(&event.OccurredAt, &event.RequestID, &event.TenantID,
            &event.Operation, &event.Decision, &event.Units); err != nil {
            return nil, fmt.Errorf("scan authorization event: %w", err)
        }
        events = append(events, event)
    }
    if err := rows.Err(); err != nil {
        return nil, fmt.Errorf("iterate authorization events: %w", err)
    }
    return events, nil
}
Enter fullscreen mode Exit fullscreen mode

The placeholder syntax varies by SQL driver, but the contract should not: the end boundary is exclusive, timestamps are normalized to UTC, and results have a deterministic order. Keep rejected requests in the result. They prove that revocation propagated and may reveal continued attempts after containment.

This query also has a clear limitation: it cannot recover events that were never durably recorded, and it cannot prove which human controlled a stolen credential. The operational trade-off is to retain enough keyed authorization metadata for incident analysis without retaining the secret or unrelated request content. RFC 3339 timestamps make the exported window portable, but synchronized clocks and documented ingestion delay are still required before responders can treat its edges as precise.

Do not infer chargeability from a 200 response alone. The ledger should define which event creates a billable unit, and reconciliation should join on immutable IDs. Retries are the usual trap: three accepted delivery attempts may represent one idempotent marketplace action, three distinct actions, or one action plus two rejected duplicates. The billing rule, not the access log count, decides.

How do you bound the blast radius without guessing?

Start with four explicit bounds: time, tenant, capability, and money. Time runs from the earliest plausible exposure to confirmed revocation propagation. Tenant comes from the credential binding, then expands only if the key had cross-tenant administrative authority. Capability is the set of scopes actually authorized, checked against observed operations. Money is the reconciled usage ledger, separated into confirmed legitimate, confirmed unauthorized, and unresolved units.

Use evidence states instead of one dramatic number. A defensible incident note might say that all authorization events for one key_id were searched for a recorded interval, every accepted request ID was joined to the job table, and all resulting ledger entries were classified. It should also name gaps: a retention cutoff, a queue lacking request IDs, or clock uncertainty. Absence of a log row is not proof of absence when the logging pipeline itself has not been verified.

There are two searches, not one. The first finds direct use of the leaked key. The second follows side effects: newly issued credentials, scope changes, exports, marketplace payouts, queued jobs, and ledger mutations attributable to those requests. If administrative credential creation was permitted, each child credential becomes another search root even after the parent is revoked.

Keep the scope controlled. Searching every tenant "just in case" increases handling of unrelated customer data and makes the result harder to audit. Expand only when a concrete permission or correlated event crosses the original boundary.

No fishing expedition.

Verification, rollback, and closure

Containment is verified when the old secret is denied at every authorization path, caches have converged within their documented behavior, and attempts using its key ID appear as rejected events without producing jobs or ledger entries. Availability is verified separately with the replacement key: one read operation and one idempotent write appropriate to its scopes are enough. Do not use a real payout or irreversible seller action as the test.

Rollback does not mean reactivating the exposed key. If the replacement causes trouble, revoke that replacement and issue another credential through the same idempotent workflow. Restoring a known-compromised secret trades an availability incident for an uncontrolled security incident and corrupts the containment timestamp.

Before closure, reconcile counts across authorization events, queue submissions, completed jobs, and billing records. They need not be equal, but every difference needs a reason such as denial, deduplication, cancellation, or failure before the billable event. Attach the saved queries, result hashes or snapshot identifiers, time-zone convention, and reviewer sign-off to the incident record. Preserve data under the established retention and legal policies rather than inventing a new retention period during the page.

Finally, turn the outcome into controls: alert on use after revocation, test key-rotation idempotency, exercise credential lineage in a drill, and verify that each billable unit can be traced back to one authorization decision without storing the secret. The practical success criterion is crisp: the tenant is operating on a least-privilege replacement, the old authority is dead, every suspicious side effect has an owner, and disputed usage can be explained from durable evidence.

References

Top comments (0)