Revoke each tenant-scoped key before deleting the tenant's operational rows, but retain a minimal, immutable decommissioning record outside that deletion boundary. The deciding constraint is auditability: after the workspace is gone, an investigator still needs evidence that access ended, which actor approved it, and which cleanup steps completed.
TL;DR: Treat offboarding as a resumable state machine, not one database transaction. Freeze new key issuance, enumerate key identifiers, request revocation, verify that every key is unusable, write an append-only evidence record, and only then remove tenant-owned data. If revocation cannot be confirmed, stop before deletion and retry. This order avoids losing the identifiers needed to disable credentials, while the retained tombstone prevents deletion from erasing the proof.
Should tenant offboarding revoke keys before delete to avoid orphan rows?
A database cascade can remove a tenant row and its local key metadata atomically. It cannot prove that a credential checked by another service, cached verifier, secrets store, or gateway has stopped working. Delete-first therefore creates an awkward blind spot: the control-plane row that named the credential disappears before the external invalidation has been verified. A successful SQL commit is evidence of deletion, not evidence of revoked access.
The opposite naive design also has a failure mode. Revoke a key, crash, and leave the tenant active, and the customer sees a partially disabled workspace. That outcome is recoverable if the workflow records state and retries idempotently. Losing the key identifier after delete-first is harder: the worker may no longer know what must be revoked.
This distinction matters for developer tools because one workspace can have several automation keys used by CI, agents, or package publishing. The job is scoped key issuance and revocation per tenant, so the unit of evidence is each credential identifier, not a vague tenant-level disabled = true flag. Never place the secret value in an audit event. OWASP recommends lifecycle controls for secrets, including revocation, expiration, and audit logging, while also warning that logs must not expose secret material.
The durable rule is revoke, verify, record, then delete.
Ordering is the control.
Model offboarding as evidence-producing states
I would make the workflow explicit: ACTIVE -> FROZEN -> REVOKING -> VERIFIED -> DELETED. FROZEN rejects key creation and rotation, closing the race where an issuance request lands after enumeration. REVOKING may be entered many times. VERIFIED means every key captured at the freeze boundary has a confirmed terminal status, not merely that revocation requests were sent.
Keep two kinds of records. Operational key rows belong to the tenant and can be deleted under the normal retention policy. A separate tombstone contains only non-secret evidence: a workflow ID, tenant pseudonymous identifier, credential fingerprints or opaque IDs, approval actor, timestamps, outcome codes, and an integrity field. Its access policy and retention period should be explicit because even minimized audit data is still data. NIST SP 800-57 Part 1 describes key-management lifecycle phases that include deactivation and destruction; the local state machine turns that lifecycle idea into an observable application workflow.
One sharp edge is concurrent issuance. Freezing in the application while another writer bypasses that check is insufficient. Imagine the worker reads three key rows, then an automation process creates a fourth key before the tenant row is deleted. The original three are revoked and the audit record looks complete, yet the newest credential survives in an external verifier with no corresponding local row. Enforce the transition where all writers observe it: for example, lock the tenant row while moving it to FROZEN, and require issuance to lock or conditionally update the same state. Recheck the frozen state in the same transaction that commits a new key reference. The exact database primitive can differ, but the invariant must be testable: no key can be committed after the workflow's enumeration boundary. A concurrency test should hold one transaction open at that boundary and prove that the other transaction cannot produce a usable credential.
A focused FastAPI worker
The core logic does not need to know a particular credential provider. A narrow protocol makes the revocation and verification semantics visible, and dependency injection keeps the notebook-friendly test double usable in the production path.
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Protocol
class KeyControl(Protocol):
def revoke(self, key_id: str, request_id: str) -> None: ...
def is_active(self, key_id: str) -> bool: ...
@dataclass(frozen=True)
class KeyRef:
key_id: str
@dataclass(frozen=True)
class OffboardingJob:
job_id: str
tenant_id: str
keys: tuple[KeyRef, ...]
def revoke_and_verify(job: OffboardingJob, control: KeyControl) -> dict:
outcomes: list[dict] = []
for key in job.keys:
request_id = f"{job.job_id}:{key.key_id}"
control.revoke(key.key_id, request_id=request_id)
if control.is_active(key.key_id):
raise RuntimeError(f"revocation not verified for {key.key_id}")
outcomes.append({
"key_id": key.key_id,
"status": "revoked",
"verified_at": datetime.now(timezone.utc).isoformat(),
})
return {
"job_id": job.job_id,
"tenant_id": job.tenant_id,
"outcomes": outcomes,
}
There is a deliberate stop before deletion. The returned document should be committed to the audit store first; only a successful evidence write allows the deletion stage to begin. In a real worker, the provider call needs bounded retries and a terminal operator-review state for ambiguous results. The deletion handler must also be idempotent because a worker can crash after committing deletion but before acknowledging its queue message.
Do not mistake the sample's is_active call for universal instant consistency. Verification semantics depend on the actual enforcement path. If authorization uses a local cache, a control-plane lookup alone may say revoked while a cached decision still grants access. Verification should exercise, or conservatively account for, every path that can authorize the credential. RFC 7009 makes this issue concrete for OAuth tokens: revocation can involve propagation delay, and the authorization server should invalidate related tokens when supported. A scoped API key system needs its own documented equivalent.
Test the order, not merely the endpoints
The most useful evaluation harness injects failure after every durable boundary. I would start with a fake KeyControl, a temporary database, and a table-driven set of crash points. No model call belongs in this loop; deterministic security invariants should stay fast and cheap enough to run on every change.
The harness should establish four properties:
- Deletion is never invoked while any enumerated key remains active or unknown.
- Retrying the same job does not create a second logical revocation or contradictory audit outcome.
- Issuance cannot succeed after the tenant reaches
FROZEN. - After tenant data is deleted, the tombstone can still answer who approved the action, which credential IDs were checked, and when verification completed.
Then add the ugly cases: a timeout whose revocation may have succeeded, a verifier cache that expires later than the control-plane update, an audit-store write failure, and a crash immediately after deletion commits. Property-based tests can vary key counts and failure positions. Integration tests should use the real authorization path in a non-production environment, because a mock cannot reveal an undocumented cache.
Keep the evaluation output compact. Counts of unresolved credentials, oldest workflow age, retry count, and time spent in each state are more actionable than logging every polling attempt. Alert on a workflow stalled before VERIFIED; a stalled post-deletion acknowledgment is operational noise if idempotent replay proves the tombstone and deletion agree.
What should you measure before adopting this order?
Measure revocation propagation time at the actual authorization boundary, not just API response latency. Record the median and tail separately, then choose a verification deadline that reflects the risk of lingering access. Also measure how long freezes block legitimate operations, how often retries encounter an already-revoked credential, and whether every deletion can be joined to exactly one completed tombstone.
Audit usefulness deserves an evaluation too. Give a reviewer only the tombstone and ask them to reconstruct the decision: who authorized offboarding, what was disabled, what verification meant, and whether deletion followed it. Missing answers expose schema gaps before an incident or compliance review does.
There is a trade-off. Retaining more fields makes investigations easier, but expands the privacy and security footprint of the audit store. Start with opaque identifiers and outcome metadata, prohibit raw keys and request payloads, apply a documented retention schedule, and test access to the evidence separately from access to live tenant data.
The sequence is ready to copy only after those measurements match the system's threat model. Revoke-first preserves the handles required to terminate access; verification turns a requested action into evidence; a minimal tombstone keeps that evidence available after deletion. The state machine is the mechanism that makes partial failure boring and recoverable.
Top comments (0)