Short answer: use the key inventory to decide what must lose access, and use application audit logs to determine what the credential did before and during containment. A leaked-key drill needs both records joined by a stable, non-secret key ID. When the B2B SaaS account platform reaches its preset spend ceiling, revoke or disable the credential even if that refuses traffic; the log cannot substitute for that control, and an inventory row cannot reconstruct past requests.
This split keeps the drill honest. Inventory drives the containment decision. Events establish impact and verify that later requests were refused. For AI-backed endpoints, I would also record usage units in the event stream so the same drill can enforce a token-cost ceiling without placing prompts or raw secrets in the inventory.
Can API key inventory and application audit logs answer the same question?
They cannot.
An inventory is current-state evidence. It should let the reviewer identify a credential without storing its plaintext value, find its owner, see its intended scope and status, and locate rotation or expiration metadata. OWASP's secrets-management guidance recommends lifecycle handling that includes creation, rotation, revocation, and expiration. That is the useful mental model: an API key is an asset with a lifecycle, not a string buried in configuration.
An application audit log is event evidence. It tells the reviewer that a subject attempted an action at a time, against a resource, and what the application decided. For a leak drill, the interesting sequence is use, detection, revocation, and a denied retry. Keep timestamps and a correlation ID, but minimize sensitive payloads. OWASP explicitly warns that secrets must never be logged in plaintext and recommends centralizing and protecting audit logs.
The difference matters under pressure. Asking “Which active keys belong to this tenant?” is an inventory query. Asking “Did this key export customer data after the suspected leak time?” is a log query. One row may be enough for the first answer; the second requires an ordered event history. Imagine the reviewer finds key_7f3a marked revoked at 08:02. That snapshot supports a present-tense claim about status, but it does not show whether an invoice read at 08:01 was allowed, whether the request belonged to the test tenant, or whether another attempt at 08:03 was denied. Only the ordered application events can supply that sequence. Reverse the evidence and the gap changes: three events for key_7f3a do not reveal whether another active key exists for the same tenant, who owns it, or whether its intended scope includes invoice reads. The review needs both halves, and each half must stay within its job.
| Evidence | Best answer | Dangerous assumption |
|---|---|---|
| Key inventory snapshot | Which credential should be active, owned, scoped, rotated, or revoked? | A revoked row proves the key was never used |
| Application audit events | Which actions were attempted, allowed, or denied? | No event proves no credential exists |
| Joined drill view | Was the right key contained, and were later attempts refused? | Matching on a key prefix is a reliable join |
Build the smallest useful drill
The data flow is plain: the control plane emits a sanitized inventory snapshot, the application emits append-only decision events, and a drill runner joins both on key_id. It accumulates chargeable usage after the declared leak time. Once usage meets the exercise ceiling, the runner marks the key revoked; any subsequent simulated request must be denied. In production, revocation belongs in the authorization path, not in the reporting job.
The example below models that contract without a vendor SDK. It uses integer usage units rather than currency because prices and billing dimensions can change, while the safety rule remains stable. The values are fixture data for the drill, not a benchmark.
from dataclasses import dataclass
from datetime import datetime, timezone
@dataclass
class KeyRecord:
key_id: str
tenant_id: str
owner: str
status: str
usage_ceiling: int
@dataclass(frozen=True)
class AuditEvent:
occurred_at: datetime
key_id: str
action: str
usage_units: int
decision: str
correlation_id: str
def run_leak_drill(
key: KeyRecord,
events: list[AuditEvent],
leaked_at: datetime,
) -> dict[str, object]:
relevant = sorted(
(event for event in events
if event.key_id == key.key_id and event.occurred_at >= leaked_at),
key=lambda event: event.occurred_at,
)
consumed = 0
evidence: list[dict[str, object]] = []
for event in relevant:
expected = "deny" if key.status == "revoked" else "allow"
if event.decision != expected:
raise AssertionError(
f"{event.correlation_id}: expected {expected}, got {event.decision}"
)
if event.decision == "allow":
consumed += event.usage_units
if consumed >= key.usage_ceiling:
key.status = "revoked"
evidence.append({
"correlation_id": event.correlation_id,
"decision": event.decision,
"status_after": key.status,
"usage_after": consumed,
})
return {
"key_id": key.key_id,
"tenant_id": key.tenant_id,
"final_status": key.status,
"usage_units": consumed,
"evidence": evidence,
}
leaked_at = datetime(2026, 9, 18, 8, 0, tzinfo=timezone.utc)
key = KeyRecord("key_7f3a", "tenant_204", "billing-api", "active", 100)
events = [
AuditEvent(leaked_at, "key_7f3a", "invoice.read", 40, "allow", "req_801"),
AuditEvent(leaked_at.replace(minute=2), "key_7f3a", "invoice.read", 60, "allow", "req_802"),
AuditEvent(leaked_at.replace(minute=3), "key_7f3a", "invoice.read", 0, "deny", "req_803"),
]
result = run_leak_drill(key, events, leaked_at)
assert result["final_status"] == "revoked"
assert result["usage_units"] == 100
assert result["evidence"][-1]["decision"] == "deny"
The assertions are the point.
A dashboard that turns red is weaker evidence than an executable rule stating that the first request after revocation is denied. This fixture also makes the business trade-off visible: after 100 usage units, availability loses to containment. Pick that ceiling before the exercise, because negotiating it during an incident quietly converts a control into a suggestion.
Where do otherwise convincing reviews fail?
The first failure is joining on the secret itself. That spreads the credential into analytics, test fixtures, and log storage. Generate an opaque identifier at issuance, store the secret through an appropriate secrets-management system, and propagate only the identifier into authorization decisions and audit events. A short display prefix may help an operator recognize a key, but it is a poor relational key because collisions and formatting changes are avoidable sources of ambiguity. The second failure is treating successful revocation as proof of limited impact. Revocation changes future authorization. It says nothing about actions already allowed, so the review still needs events from the suspected exposure window and enough context to connect each decision to a tenant, action, resource category, and correlation ID. Then there is the quiet failure: the application records requests but omits the authorization decision. After revocation, a request can reach the service and still be correctly refused. Without an explicit allow or deny, reviewers may confuse attempted traffic with completed work. Record the decision close to enforcement, and test the denied path. Logs have their own exposure risk too. Do not record the raw key, request authorization header, prompts containing customer material, or full response bodies merely because storage is available. Preserve the minimum evidence needed for reconstruction, restrict access to that evidence, and define retention around the review obligation. More data can mean more liability.
Denied is a result.
How should the spend ceiling affect refused traffic?
For this drill, make the ceiling an authorization input rather than an alert threshold. An alert says someone should act. A deny decision limits further consumption even when nobody has opened the page. The cost is deliberate false refusal if accounting data arrives late or the ceiling is too low, so the design needs a documented policy for stale counters and emergency access.
Use conservative units that the application can count deterministically. An AI feature might count accepted model-input and model-output units; another endpoint might count jobs or records. The drill does not need a mutable price table in its hot path. Finance can translate usage into money elsewhere, while authorization compares a stable counter with the approved ceiling.
There is a sharp edge here. If several workers authorize concurrently against an eventually consistent total, each can observe room below the ceiling and collectively overshoot it. The safety requirement determines the counter design: a hard ceiling needs an atomic reservation or a deliberately reserved buffer, while an approximate ceiling can accept bounded overshoot. Write that tolerance into the eval before choosing storage.
Decide before deploying.
Turn the exercise into an access-review artifact
Run the exercise with a synthetic credential scoped to a test tenant and a fixed exposure time. Confirm that the inventory names an accountable owner and intended scope, then produce allowed events until the ceiling is reached. Attempt one more action. The resulting artifact should show the same key_id across the inventory snapshot, the allowed sequence, the revocation transition, and the denied retry, with correlation IDs that let a reviewer inspect individual decisions.
Also test absence. A credential present in logs but missing from inventory indicates an issuance or synchronization gap; a credential present in inventory but absent from logs may be unused, or logging coverage may be incomplete. Those cases are not equivalent, and the review should not auto-resolve either one.
Before calling the drill complete, verify clock handling, duplicate-event behavior, access to the evidence store, and redaction. Re-run the Python fixture in CI whenever the authorization or metering contract changes. Keep a small set of adversarial cases too: an event exactly at the leak time, two events racing near the ceiling, an unknown key ID, and a denied event with zero usage. This is where an eval-driven workflow earns its keep. It turns a policy sentence into a regression test.
The final review decision can stay compact: inventory establishes what should be trusted now; audit events establish what happened; the joined evidence proves containment. If the ceiling and availability conflict during this leak drill, the preapproved ceiling wins, and the refused request becomes required evidence rather than an operational surprise.
Top comments (0)