DEV Community

ThomasMoore157
ThomasMoore157

Posted on

Tenant Offboarding Sequence Explained — Revoke Credentials Before Deleting Data

#saas #security #offboarding

When a healthtech tenant leaves, revoke the tenant's API credentials before making its data unreachable, then complete deletion from a separately authorized worker. That ordering limits the blast radius if an old integration keeps retrying while the offboarding job is still running; it also gives the audit trail a clean security boundary.

Short answer: disable and revoke every credential first, record the event, quarantine the tenant, and only then run the deletion workflow. Keep the deletion token and the runtime token separate, and make both operations idempotent.

The page that wakes the on-call

The alert usually arrives late. A queue-depth alarm fires because an integration is still sending events for a tenant that should have been retired. The on-call sees repeated authentication failures, a growing dead-letter queue, and a compliance ticket asking whether any protected health information was processed after the contract end.

The first useful question is not “did deletion finish?” It is “can any credential that belonged to this tenant still create work?” A status flag in the tenant table is not enough if workers validate a cached token, and deleting rows first can make it harder to prove which credential was active at each step.

Freeze first.

I keep the sequence explicit: freeze new work, revoke credentials at the issuer, write an immutable offboarding event, and place the tenant in a quarantine state. The deletion worker consumes that event only after it confirms the credential state. A retry receives the same tenant identifier and produces the same end state; it does not mint a second cleanup job with broader access. In a real incident review, this is the paragraph I would expand into a timeline: at 09:00 the contract boundary is recorded, at 09:01 the gateway rejects new events, at 09:02 the issuer marks the key unusable, and at 09:03 the broker drains messages already accepted. The exact minutes are illustrative, not a service promise; what matters is that each transition has a durable timestamp and a responsible actor. If the issuer is eventually consistent, the gateway-side quarantine closes that gap, while a worker that has already loaded a token must check tenant state again before writing anything. Without those two checks, a “successful” deletion can coexist with a late side effect and an audit trail that cannot explain it.

That is a small distinction with a large operational consequence. In a healthtech system, one shared key can turn a single tenant departure into a cross-tenant incident, so credential scope and ownership matter more than shaving a few minutes from the job.

How should a tenant offboarding sequence revoke an API key before deletion?

The sequence should be a state machine, not a script that calls “delete” and hopes the network agrees. A useful set of states is active, frozen, credentials_revoked, quarantined, deletion_running, and deleted. Each transition records actor, timestamp, request id, and the credential version it observed.

The revoke step must cover machine credentials, webhook signing secrets, refresh tokens, and any service-account grants issued for that tenant. OWASP recommends a managed secrets lifecycle with rotation, revocation, least privilege, and auditable access; those controls are relevant here because offboarding is a secrets-lifecycle event, not just a row cleanup.

The deletion step should use a different narrowly scoped identity. It reads the tenant's deletion manifest, removes data from primary stores and indexes, and schedules provider-specific erasure where immediate removal is not possible. Backups need a documented retention and restore policy: a database delete does not magically rewrite an immutable backup, and pretending otherwise creates a misleading compliance record.

Here is the shape I want reviewers to see in code. The concrete storage and identity providers can change; the ordering cannot.

package offboarding

import (
    "context"
    "fmt"
)

type Credentials interface {
    Revoke(ctx context.Context, tenantID string) error
    IsRevoked(ctx context.Context, tenantID string) (bool, error)
}

type Tenants interface {
    Freeze(ctx context.Context, tenantID string) error
    Quarantine(ctx context.Context, tenantID string) error
}

type Deleter interface {
    DeleteManifest(ctx context.Context, tenantID string) error
}

func Offboard(ctx context.Context, tenantID string, creds Credentials, tenants Tenants, deleter Deleter) error {
    if err := tenants.Freeze(ctx, tenantID); err != nil {
        return fmt.Errorf("freeze tenant: %w", err)
    }
    if err := creds.Revoke(ctx, tenantID); err != nil {
        return fmt.Errorf("revoke credentials: %w", err)
    }
    revoked, err := creds.IsRevoked(ctx, tenantID)
    if err != nil {
        return fmt.Errorf("verify revocation: %w", err)
    }
    if !revoked {
        return fmt.Errorf("credentials are not revoked")
    }
    if err := tenants.Quarantine(ctx, tenantID); err != nil {
        return fmt.Errorf("quarantine tenant: %w", err)
    }
    if err := deleter.DeleteManifest(ctx, tenantID); err != nil {
        return fmt.Errorf("delete tenant data: %w", err)
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

The verification call is deliberate. A successful revoke request is not the same thing as a verified security state, especially when token caches and asynchronous issuers are involved. I am not sure every identity provider exposes the same consistency guarantee, so the contract should state the maximum revocation delay and the compensating control, such as rejecting the tenant at the gateway while propagation completes.

Where does the failure boundary belong?

Treat the gateway, event broker, workers, and data stores as separate failure domains. The gateway should reject a frozen or quarantined tenant before publishing new events. Consumers should check tenant state again before performing side effects, because an event may have been accepted just before the freeze. That second check is cheap insurance against an in-flight message.

Observability needs to follow the state machine. Emit counters for revoke requests, verified revocations, rejected tenant events, deletion retries, and records held by legal policy. Alert on time spent between frozen and credentials_revoked, and on any post-revocation event that reaches a side-effecting consumer. A low error rate can still hide a dangerous delay if the metric only counts completed deletions.

Thresholds have a cost. Set them too low and routine propagation lag pages the on-call; set them too high and the system quietly accepts work after the contract boundary. Start with a measured service-level objective for revocation propagation, then tune the alert against that SLO rather than against an arbitrary minute count.

Buy, build, or keep the boundary small?

The decision is less about a fashionable platform and more about who owns the irreversible step.

Approach Useful when Trade-off
Identity-provider revocation plus an in-house state machine You need a precise audit trail and tenant-specific policy Your team owns retries, provider semantics, and deletion evidence
Managed lifecycle workflow The provider offers documented revocation and retention controls Policy may be less expressive, and portability can suffer
Self-hosted identity and storage controls You need control over data residency and network boundaries On-call load, patching, backup erasure, and recovery testing stay with you

The catch is that a managed workflow is not suitable when it cannot prove credential scope or expose a revocation timestamp. Stick with a simpler in-house boundary when the workflow must coordinate several stores, a legal hold, and a broker you already operate. Conversely, self-hosting is a poor fit for a small team that cannot exercise restore and key-rotation drills; the operational risk becomes part of the product.

A review checklist that survives an outage

Before shipping, test the ugly interleavings: revoke succeeds and quarantine times out; deletion retries after a process restart; a stale event arrives after quarantine; a legal hold blocks one dataset; and an operator repeats the command with the same request id. Verify that no path logs raw secrets, that deletion credentials cannot read application data, and that an audit export can connect each action to an actor.

Run the exercise during a controlled maintenance window, then inspect the evidence as if you were an incident responder. The useful artifact is not a green dashboard. It is a timeline showing when new work stopped, when credentials became unusable, what data classes were removed, and what remains under retention policy.

References

Further reading

Top comments (0)