DEV Community

XerxesCross2735
XerxesCross2735

Posted on

Deleted Tenant Data Still Appearing: Isolate Live Credentials and Stale Catalog Paths

#tenant #security #python

Short answer: if a deleted e-commerce tenant's records still appear when an old API key is used, treat the response as an authorization incident until you know which layer served it. Revoking a key, deleting source rows, removing search documents, and invalidating cached responses are separate operations. Test each boundary independently with the same tenant-scoped credential; a disappearing database row alone proves very little.

The evaluation constraint matters: the probe must distinguish a refused request from a successful request returning zero results. Both can look like an empty storefront in a dashboard, but only the former establishes that the retired credential cannot read anything. Start with a response captured before offboarding, including its status, tenant identifier, key identifier (never the secret), request time, and whether it came through a CDN, application cache, search index, or origin. Avoid logging the bearer token itself; OWASP's secrets guidance covers exposure in logs and secret lifecycle handling.

Why is data still appearing for a deleted tenant after key revocation?

A tempting first check is to query the primary database for the deleted tenant. That checks retention, not authorization. Instead, send a request using the retired key to an endpoint that normally returns that tenant's catalog, then send the same request with a known-invalid key. If both yield a successful response containing tenant data, inspect the authentication path and any cached response served before it. If the retired key is refused but a separate active credential still returns old listings, the remaining problem is likely data lifecycle or tenant isolation, not key revocation. These are diagnostic branches, not proof of any particular implementation bug. Compare the exact product identifier returned on each path: a catalog page with no matches tells a different story from a response that contains a retired listing under the wrong tenant. Preserve the response headers and the nonsecret request ID as well, so an operator can correlate the observation with cache and origin logs without putting credentials in a ticket.

One empty response isn't enough.

Record where the key is checked. A locally verified signed token can remain accepted until its expiration unless the verifier checks a revocation mechanism; a server-side lookup can apply a changed key state on the next lookup, subject to its own cache. RFC 7009 describes revocation for OAuth tokens but does not automatically revoke arbitrary application API keys. Document the actual contract for your keys and test against it. Do not assume deleting a tenant row changes the result of every verifier already running.

Revocation needs its own assertion.

A focused offboarding probe

For a storefront with tenant A and tenant B, seed one unmistakable product identifier in each tenant's catalog. Issue a scoped key for A, verify it cannot retrieve B's product, revoke A's key, and repeat the exact A and B reads. Then run the same checks through the public edge and directly against the origin in a controlled test environment. A private origin test helps isolate edge caching; it is not a substitute for testing the public path users actually reach.

The assertion should be about authorization, not an empty list. This small Python sketch assumes a test client and a fixture that supplies an already-revoked key; it deliberately avoids prescribing a vendor's endpoints:

from urllib.parse import quote


def assert_revoked_key_is_refused(client, revoked_key, tenant_id):
    path = f"/tenants/{quote(tenant_id, safe='')}/catalog"
    response = client.get(path, headers={"Authorization": f"Bearer {revoked_key}"})
    assert response.status_code in (401, 403)
    assert b"product_" not in response.content
Enter fullscreen mode Exit fullscreen mode

The status assertion reflects the test contract, not a universal status-code rule: decide whether an unrecognized token and a recognized but forbidden principal should be distinguishable, then keep that behavior consistent. RFC 6750 documents bearer-token error responses for OAuth-protected resources. Keep raw credentials out of test reports. Store only a stable, nonsecret key identifier beside each result.

Why can a correct revocation still leave visible records?

A response cache might key on URL alone, even though authorization changes which tenant's catalog is visible. A search index might still contain documents after the source rows are removed. A worker might retry an old indexing event after deletion. These are hypotheses to verify with a request trace and the index's document state, not reasons to silently turn off security checks. HTTP caching rules in RFC 9111 explain how shared caches handle authenticated responses; application-level caches also need keys and invalidation rules that preserve tenant and authorization boundaries.

The ordering trade-off is sharp. Refusing the retired key first limits further reads through that credential while asynchronous deletion catches up. Blocking every tenant-facing operation until all replicas and indexes converge may reduce stale visibility, but it also increases refused traffic during offboarding. Define the desired boundary explicitly: a revoked key must be refused even when old documents remain, while permitted retention or deletion workflows may have separate completion criteria. For an AI shopping assistant, repeat the probe against its retrieval path too; a vector or search result that bypasses tenant filtering can surface an old catalog entry even if the transactional query is clean.

Measure before adopting the sequence

Run this as an eval, not a notebook-only demonstration. Track time from revocation request to the last successful authenticated read, count refused reads for still-authorized tenants during offboarding, and measure the time until deleted product identifiers disappear from search and cached responses. Record status codes and cache provenance without recording secrets or customer content. Repeated probes at the edge and origin reveal different failure modes; one passing request does not establish a maximum propagation delay.

Those measurements frame the spend-ceiling versus refused-traffic decision. Aggressive global invalidation can consume operational capacity and reject unrelated tenant traffic; narrow, tenant-aware invalidation requires careful cache keys and coverage across every reader. Choose based on observed exposure and load in your own system, then make the key-revocation assertion a release gate. The most useful final artifact is a timeline showing when authentication stopped accepting the key and when every data-serving path stopped returning the retired tenant's products.

References

Further reading

Top comments (0)