A customer-support tenant can keep sending billable messages while its prepaid balance is being closed. Short answer: fence new sends first, revoke credentials next, and delete tenant data only after dependent work has drained and deletion checks pass. Deleting the tenant row first risks leaving child rows and queued jobs without an owner; revoking a token alone does not stop work already admitted by a worker. The design decision is how long to tolerate refused support traffic while protecting the spend ceiling, not which delete call runs fastest.
What has to stop before the tenant disappears?
Consider a support organization with an outbound message queue, a prepaid balance, sender credentials, and message records. At offboarding, a queued password-reset message and a support reply may still be waiting for dispatch. A row delete cannot retract either one from a worker that already loaded it. A credential revocation cannot undo a message that has already crossed the delivery boundary. Treat both as separate controls, and identify exactly where billable work becomes irrevocable.
The first transition should be a durable tenant state such as closing, checked at admission and again immediately before an outbound send or balance reservation. New traffic gets an explicit refusal. Existing work needs a policy: cancel it, or let a bounded set finish under a reserved ceiling. For a strict prepaid ceiling, cancel uncommitted work and release its reservations; never assume a refund for an already accepted delivery. Do not send an OTP merely because it was queued before the state changed. Late OTP delivery can confuse a user and invalidate the intent of the shutdown.
There is a race here. A worker may read active while the offboarding transaction changes it to closing. Serialize the final balance reservation and the state check in one database transaction, using an appropriate row lock or conditional update. Then a send must have a recorded reservation from before the fence, or it must fail closed. Suppose two workers take different support messages from the same queue: one reserves balance just before the closing transaction takes the lock, while the other waits and reads the committed closing state. The first reservation needs an explicit finish-or-cancel rule; the second worker must refuse the send. A queue acknowledgment by itself does not distinguish them. The state fence is a local design rule, not a claim that credential revocation propagates instantly across every service.
No new reservations.
Which order leaves fewer orphan rows?
Fence, revoke, drain, delete is the safer default when credentials can outlive a database row. Record an idempotent offboarding request and transition the tenant to closing; stop new reservations and dispatch; revoke each credential at its authority; reconcile queued and in-flight jobs; only then remove rows according to dependency and retention rules. An independent cleanup worker can retry a failed revocation without restoring traffic.
Delete-first can be defensible only when an earlier access fence is already durable, dependent data is either removed transactionally or intentionally retained with a valid owner, and all asynchronous consumers enforce the fence. Otherwise the parent vanishes while live credentials or jobs still reference its ID. A foreign key with ON DELETE CASCADE handles declared database relationships, but it cannot delete an external credential, an already-enqueued task, or a record stored outside that database. A restrictive foreign key is often useful during rollout: a failed parent delete exposes an unaccounted dependency instead of hiding it. For example, a support-message delivery receipt may arrive after its parent conversation has been erased. Decide before deployment whether that late receipt is rejected, attached to a retained audit identity, or discarded under the retention policy; an unowned receipt row is none of those. This is why the deletion inventory must include callbacks and scheduled retries as well as tables.
Keep the identity needed for retries.
| Sequence after the admission fence | Immediate advantage | Failure to contain |
|---|---|---|
| Revoke, reconcile, delete | Access is disabled before destructive cleanup | Revocation can stall; retain a fenced, inspectable tenant until retry succeeds |
| Delete, then revoke | Local row cleanup may finish sooner | Revocation retry needs a separate durable identity and credential inventory; lost references can strand access |
Neither order should mean deleting evidence that policy requires you to retain. Separate operational rows from a minimal audit record with a documented retention period and access controls. Privacy deletion requests and statutory retention obligations need a policy decision, not an unconditional cascade. OWASP's secrets guidance also treats revocation and rotation as lifecycle operations; removing an application row is not equivalent to invalidating its secrets.
How do you prove the shutdown held?
Make the offboarding operation resumable. Persist an operation ID, tenant ID, requested time, current phase, credential identifiers, and per-step outcomes before starting external work. Avoid storing raw secrets in that ledger. A repeated request for the same tenant should resume the same operation, while a concurrent request cannot reopen the tenant. Keep a separate tombstone or retained audit identifier if deletion removes the primary tenant row; otherwise a retry may have no key with which to revoke a remaining credential.
At the dispatch boundary, log an operation ID, the admission decision, a reservation ID if one exists, and a terminal send outcome. Do not log OTP values or full message bodies. Measure the count of accepted sends after the closing timestamp, unresolved reservations, still-active credentials, queued jobs, and retained child rows whose owner is absent. An accepted send after the fence is an incident to investigate, not a metric to average away. For delivery gaps, separate provider acceptance from recipient delivery in the event model; acceptance alone does not prove that a code arrived.
Test the awkward interleavings: close between queue admission and dispatch, lose the revocation response after the remote side succeeds, crash after a balance reservation, then retry deletion. Also test a foreign-key failure with one remaining dependent row. The expected result is a fenced tenant with a visible incomplete operation and no new billable admission. A timeout from a revocation endpoint is ambiguous; query its state or retry idempotently instead of marking the credential revoked on faith. Simulate a delayed delivery callback after cleanup, too: it should resolve against an authorized retained identifier or be deliberately rejected, never recreate the deleted account merely to satisfy a callback.
Retries aren't proof of success.
Rollout without widening the spend window
First deploy the closing checks to every producer and worker, and confirm old workers have drained before using the new offboarding path. Next add the durable operation record and revocation reconciliation. Finally enable dependent-row deletion in small batches, with a dry-run count of rows by table and a stop condition when active credentials or unresolved sends remain. Check retention and audit requirements before scheduling physical erasure.
The trade-off is visible: strict fencing may refuse legitimate support replies while the balance is being settled. That refusal should be explicit to callers, with a route for an authorized operator to resolve the balance or finish the shutdown. Quietly accepting traffic during cleanup makes the spend ceiling impossible to reason about.
References
- OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- PostgreSQL documentation, foreign keys and referential actions: https://www.postgresql.org/docs/current/ddl-constraints.html
- RFC 7009, OAuth 2.0 Token Revocation: https://www.rfc-editor.org/rfc/rfc7009
Top comments (0)