A tenant key should be revocable from an already-running admin path, and that revocation must continue working while the event-ingestion path is unhealthy. The design I use is a small, separately authorized control operation that writes a deny record to durable storage, then makes every request check that record before accepting platform events. The important test is not request speed; it is how much of the system one leaked credential can reach during an outage.
I started with a process-local set of revoked keys. It passed a notebook demo and failed the first realistic thought experiment: two API instances disagreed, and a restart erased the decision. In healthtech, that is an unacceptable source of ambiguity.
The short version: shared state wins, but it adds a dependency you must monitor.
It fails closed.
What should the admin endpoint actually revoke?
Revoke a key identifier, not a tenant row and not an entire account. Store a hash of the presented secret, a status, an actor, a reason, and timestamps. The admin request should be idempotent: repeating it returns the same disabled state, while an audit event records who requested it. Keep the control route independent from the event payload schema, so a malformed tenant event cannot block the emergency action.
The endpoint below is intentionally plain Node.js. It assumes a repository with an atomic upsert and a request verifier that can read the same repository. The example uses a generic path and avoids coupling the decision to a particular database product.
async function revokeKey(req, res) {
const actor = req.auth?.subject;
if (!actor || !req.auth?.permissions?.includes("keys:revoke")) {
return res.status(403).json({ error: "forbidden" });
}
const { keyId, reason } = req.body ?? {};
if (typeof keyId !== "string" || keyId.length < 8) {
return res.status(400).json({ error: "invalid keyId" });
}
const record = await keyStore.disable({
keyId,
reason: typeof reason === "string" ? reason.slice(0, 240) : "unspecified",
actor,
disabledAt: new Date().toISOString()
});
await auditLog.append({ type: "tenant_key_disabled", keyId, actor });
return res.status(200).json({ keyId: record.keyId, status: "disabled" });
}
The event handler must fail closed when it cannot determine key status. A temporary read error is an outage signal, not permission to accept traffic. Cache positive revocation results briefly, but never cache an enabled result longer than the risk window you have agreed to in your threat model.
How do you keep one credential from widening the outage?
Separate credentials by capability and blast radius. The ingestion worker needs to submit events; it should not be able to call the disable operation or read unrelated tenant records. The admin service needs a narrow control credential and a network path that remains available when ingestion queues are saturated. Put rate limits and a bounded body size on both paths.
For a healthtech incident, I measure three boundaries before shipping: time from an authenticated revoke request to rejection at every instance, behavior when the revocation store is unavailable, and the maximum event volume accepted after a key is disabled. Those are eval cases, not dashboard wishes. I run them from a small Python harness against staging because the notebook-to-prod gap is where optimistic assumptions hide.
import time
import requests
def wait_until_rejected(event_url, token, timeout=20):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
response = requests.post(
event_url,
headers={"Authorization": f"Bearer {token}"},
json={"type": "test", "payload": {"synthetic": True}},
timeout=2,
)
if response.status_code in (401, 403):
return time.monotonic()
time.sleep(0.25)
raise TimeoutError("revocation was not observed")
Do not put raw secrets in logs, traces, queue messages, or evaluation fixtures. Hash identifiers before correlation, and give the audit record enough context to investigate without reproducing protected health information. OWASP's secrets guidance is useful here because rotation, access policy, and observability are one operating discipline, not three independent tickets.
What does a safe failure look like?
During a dependency outage, the control route should return a clear failure and leave the previous state unchanged; a caller must not mistake a timeout for successful revocation. The data plane should reject requests when it cannot verify a key, with a bounded retry policy that cannot create a traffic storm. Once storage recovers, replay the revoke operation safely because the operation is idempotent.
A deploy-free control is still a production feature. Protect it with separate authentication, approval policy appropriate to the tenant risk, and an alert when revocation latency or denied-event counts move outside the baseline. Test concurrent revoke and ingest requests, process restarts, clock skew, and a partially unavailable store. The point is containment: one credential should map to one tenant capability, and one administrative decision should propagate predictably.
This pattern has a real trade-off: it is a poor fit for a tiny single-process service that has no durable tenant state; the extra store and audit path may cost more operational attention than the risk warrants. It is also weaker than a network-level kill switch when a credential can reach systems outside the event API. In those cases, use the control that owns the larger boundary, and keep this endpoint as a tenant-scoped backstop.
Measure those properties before copying the pattern into another service.
Top comments (0)