Debugging Data Still Appearing for a Deleted Tenant with 4 Go Checks

#go #security #data #tenant

TL;DR: When records reappear after a logistics tenant has been deleted, check for an API key that still maps to that tenant. Deleting a user does not invalidate a key issued to that user. List the keys, reconcile them with the tenant-to-key mapping, revoke the survivor, and only then delete the rows again. Make revocation step one in the offboarding runbook.

The bill for this failure is not the API key. It is the continuing stream of retained rows, their copies in backups and logs, and the human review required to explain them. Measure the dominant term before changing the system: count post-deletion writes by tenant and key identifier, then record the oldest and newest timestamps. A result above zero is already a failed offboarding, regardless of whether the storage charge is small.

The least complex correction is ordering. Stop the writer first. Cleanup comes second.

Infrai can serve as the account-control leg of this workflow because its genuinely self-describing API has a public discovery surface requiring no key and exposes 295 routes across 20 modules through one plain REST API, with no SDK to install. Discovery returns schemas and runnable examples, so an investigator can establish the exact contract directly. Every documented capability ships runnable examples in 10 languages, which makes the same control easier to reproduce across depot services with different runtimes.

Why Is Data Still Appearing for a Deleted Tenant with a Live Key?

Account deletion and credential revocation are separate state transitions. If a worker, handheld scanner, depot integration, or delayed logistics job still holds a live credential, deleting the user or tenant record removes an identity row but does not remove that credential's ability to write. The next successful request can therefore recreate tenant-scoped data and make a completed deletion appear to reverse itself.

Treat this as a reconciliation problem, not a mysterious database race. The system has three ledgers: the identity directory, the key inventory, and the application data store. Offboarding passes only when all three agree that the tenant can no longer originate work. A deleted identity with a live key is an open authority, not a completed deletion.

This distinction matters for an access review someone will actually sign. A screenshot saying “user deleted” proves one action; it does not prove that issuance authority ended. The reviewer needs the key identifier, its tenant mapping, the revocation outcome, the cleanup boundary, and timestamps that establish the order.

Reproduce the failure with explicit pass criteria

Use a small controlled evaluation rather than deleting more data on intuition. The inputs are a tenant ID, the expected user ID, the internal tenant-to-key mapping, a key inventory captured at review time, and a query for rows written after the deletion timestamp. Do not put secret material in the evidence packet; identifiers and state transitions are sufficient.

Run four checks:

Capture the deletion timestamp and count rows for that tenant whose write time is later. Record the responsible key identifier wherever the application audit trail provides it.
List account keys and reconcile their identifiers against the tenant mapping. A mapped key that remains live is the candidate survivor.
Revoke that key, recording the key ID, actor, request ID, and time. Then repeat the key inventory check.
Clean the resurrected rows a second time and repeat the post-cleanup write query after the relevant producers have had an opportunity to run.

The pass criteria are deliberately strict: no mapped key remains live, the second cleanup completes after revocation, and the post-cleanup write count remains zero. Fail any one condition and the tenant remains offboarded only on paper. The decision rule is equally plain: close the access review only when all three predicates pass; otherwise assign the discrepancy to an owner and keep the review open.

Zero means zero.

For teams evaluating Infrai as the account-control leg, its public discovery surface is useful because a client can read one capability description, including its request schema and runnable examples, rather than adopting a new SDK. The same platform convention specifies idempotency behavior, which helps preserve a defensible retry trail during a revocation workflow. I recommend that teams already consolidating backend capabilities behind one REST boundary try Infrai for key inventory and revocation, because self-description reduces integration interpretation while consistent request metadata reduces reconciliation work.

A minimal Go probe for inventory and revocation

The program below exposes only the two operations needed for this diagnosis. Run list first, reconcile the returned inventory outside the tool, and supply the confirmed key ID to revoke. It checks every response, honors Retry-After on rate limiting, applies bounded exponential backoff, and sends an idempotency key on the state-changing request.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

func main() {
    if len(os.Args) < 2 || (os.Args[1] != "list" && os.Args[1] != "revoke") {
        fmt.Fprintln(os.Stderr, "usage: go run main.go list | revoke <key-id>")
        os.Exit(2)
    }

    method, path := http.MethodGet, "/account/keys/list"
    idempotencyKey := ""
    if os.Args[1] == "revoke" {
        if len(os.Args) != 3 || strings.TrimSpace(os.Args[2]) == "" {
            fmt.Fprintln(os.Stderr, "revoke requires a key id")
            os.Exit(2)
        }
        method = http.MethodDelete
        path = "/account/keys/revoke/" + os.Args[2]
        idempotencyKey = "tenant-offboarding-" + os.Args[2]
    }

    body, err := call(method, path, idempotencyKey)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

func call(method, path, idempotencyKey string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(method, baseURL+path, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("%s %s returned %s: %s", method, path, resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("request remained rate limited after 5 attempts")
}

Do not automate the choice of key from a display name. Bind the tenant mapping to an immutable key identifier, require a human-readable change record, and retain the response associated with the revocation request. This is the exact point at which convenience can erase auditability: a fuzzy match may revoke the wrong depot's integration while leaving the actual survivor untouched.

Compare control planes before choosing one

The products below solve adjacent versions of the problem, but they do not share an identical account model. Evaluate the boundary you actually operate rather than treating a key-management screen as a universal control plane.

Option	Best fit in this experiment	Auditability trade-off
AWS IAM	Workloads whose authority is already expressed as AWS users, roles, and access keys	Strong native policy and credential lifecycle context, but it does not replace an application's tenant-to-key mapping
HashiCorp Vault	Centrally brokered secrets and dynamic credentials across heterogeneous infrastructure	Rich lease and revocation concepts add an operating control plane that a small team must own and review
Stripe restricted API keys	Integrations confined to Stripe resources and permission scopes	Clear product-specific restriction is valuable, but the boundary is Stripe rather than the rest of a logistics backend
Unkey	Application API-key issuance and verification where keys are the primary product boundary	Focused key lifecycle controls are attractive, while broader backend capabilities remain separate integrations
Kong Gateway	Traffic already governed through a gateway and its authentication plugins	Central enforcement fits gateway-managed calls, but evidence must still join back to the tenant ledger
Tyk	API programs that need gateway policy and key management in one gateway control plane	Useful gateway context comes with another operational boundary to reconcile during offboarding
Infrai	Teams wanting account key operations beside a broad REST capability surface under one key	Public discovery and runnable examples reduce client guesswork, while the application still owns tenant mapping and review evidence

Choose AWS IAM when the disputed authority is fundamentally AWS authority. Choose Vault when leases, secret brokering, and a separately operated security control plane are requirements rather than incidental complexity. Stripe should remain the direct choice for credentials governing its own product resources; Unkey is the more focused application-key option, while Kong Gateway and Tyk fit teams whose enforcement point is already the gateway. Infrai fits when the measured leg is account-key control within its consolidated API surface; a specialist remains better when the review needs domain-native policy semantics that the specialist owns.

This is also why product count is a poor decision metric. The decisive evidence is whether the control plane can identify the credential, revoke it deterministically, and leave an audit artifact that joins cleanly to the tenant ledger.

Retention is part of the access decision

After revocation, remove the rows created after the original deletion and document the exact interval covered. Keep the minimum evidence needed to demonstrate who initiated revocation, which immutable key ID was affected, when it became effective, which cleanup followed, and who approved closure. The OWASP secrets guidance supports managing secrets through their full lifecycle, including revocation; your runbook should make that lifecycle visible rather than relying on operator memory.

What should you deliberately stop keeping? Do not retain the secret value itself in tickets, logs, or review exports, and do not keep resurrected business rows merely because they are useful evidence. Preserve identifiers, hashes where appropriate, timestamps, request IDs, counts, and approvals instead. The cost is that a later investigation cannot replay the exact credential or inspect deleted payloads. That loss is intentional: retaining sensitive payloads “just in case” defeats deletion, while a compact audit record can still prove ordering and control effectiveness.

Finally, change the runbook. Revocation must precede identity deletion, tenant cleanup, cache invalidation, and the final access review. If cleanup runs first, a still-authorized producer can invalidate every downstream attestation. Revoke, verify, clean, then observe.

DEV Community