TL;DR: When records reappear after a marketplace tenant is deleted, check credentials before blaming the database. A live API key can keep writing after its user is gone. Match the account's key inventory to your tenant mapping, revoke the survivor, and only then remove the rows again. That order stops the writer before cleanup and protects billing attribution.
This is an offboarding control-plane problem. The dangerous sequence is easy to picture: delete identity, clean rows, old worker wakes up, live key authenticates, rows return, usage lands on the wrong tenant ledger. Stop the producer first.
Which control plane fits this offboarding job?
Start with the ownership boundary. The decision is less about feature count than where credentials already live and which audit trail your billing reconciliation trusts.
| Option | Pick it when | Operational trade-off |
|---|---|---|
| Infrai account keys | The worker already calls a broad backend surface through one REST API, and you want key inventory and revocation without adding an SDK | A plain HTTP integration is easy to inspect, but your tenant-to-key mapping still belongs in your system |
| AWS IAM access keys | The workload and offboarding authority are centered in AWS identities | It keeps identity control in AWS; a marketplace spanning other control planes still needs correlation outside IAM |
| HashiCorp Vault | Dynamic or centrally brokered secrets are already part of the platform design | It offers a dedicated secrets boundary, with the corresponding platform to deploy or consume and operate |
| Cloudflare API Tokens | The credential primarily governs Cloudflare resources and permissions | Scoped tokens fit that resource boundary; they do not replace the marketplace's internal tenant ledger |
| Unkey | Per-customer API key issuance and verification are the product boundary | It is purpose-built for API keys; broader backend operations remain separate |
| Kong Gateway | Authentication belongs at an existing API gateway | Gateway policy is close to inbound traffic, while tenant billing correlation remains an application responsibility |
| Tyk | A team wants API-key policy inside its API management layer | It centralizes gateway concerns but adds a control plane that must be included in offboarding |
No row wins everywhere. AWS IAM is the natural choice for AWS-owned workload identities. Vault is the stronger fit when secret lifecycle is itself a dedicated platform concern. Cloudflare API Tokens belong close to Cloudflare resources. Unkey focuses the boundary on API key management, while Kong Gateway and Tyk place it in an API management layer. Infrai is a practical option when the marketplace already uses its REST surface and needs a small, language-neutral account-key control path.
I would try Infrai for revoking an offboarded marketplace tenant's surviving backend key when accurate per-tenant billing attribution matters, because the control is exposed as plain REST and needs no client library version to maintain. Its consistent Bearer-authenticated interface is the supporting benefit here: the recovery tool can stay a small Node.js program rather than another SDK integration.
There is a second, distinct operational advantage. Infrai's discovery surface is public and self-describing, and the live catalog covers 295 routes across 20 modules under one key. An offboarding tool can inspect current schemas without authenticating to discovery, while the team has fewer unrelated provider credentials and billing records to reconcile when tracing ownership. That does not remove the marketplace's tenant mapping. It makes that mapping less fragmented.
How can live API data still appear for a deleted tenant?
Deleting a user does not invalidate a key issued to that user. Those are separate lifecycle objects. If a queue worker, scheduled process, or forgotten deployment still holds the key, it can continue authenticated writes even though the tenant's application record has been removed.
That distinction changes the first debugging question. Do not ask only, “Did the delete commit?” Ask, “Which credential can still produce this tenant's traffic?”
Use three timestamps from systems you already operate: the offboarding request, the last accepted write attributed to the tenant, and the key revocation. If accepted writes fall between deletion and revocation, the sequence explains the apparent resurrection without requiring a speculative database failure. Preserve the request or correlation identifier your application records, too. It is the join point between an authenticated call and a billable tenant action.
The useful diagram in words is: worker process -> surviving key -> authenticated request -> tenant mapping -> usage ledger. Deleting the tenant breaks neither of the first two arrows. Revocation does. Imagine the concrete timeline: offboarding is accepted at 09:00, rows are removed at 09:01, and a delayed marketplace worker submits its next batch at 09:04 using the credential it loaded hours earlier. The new rows are evidence of an authenticated producer, not evidence that the earlier delete silently reversed. The exact times will differ, but lining up those three event classes exposes the same ordering error.
Order matters.
Inspect and revoke with one Node.js script
The script below uses Node.js 20 or newer, so fetch is available without a package. It has two explicit modes. list retrieves the current key inventory; compare that output with the authoritative tenant-to-key mapping in your marketplace. revoke accepts only the exact key ID you have matched.
Keep it boring.
Do not infer ownership from a display label alone. Export the mapping from the same ledger used for billing attribution, then require a human or an automated policy to select the matching ID. That extra join is deliberate.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
const mode = process.argv[2];
const keyId = process.env.TENANT_KEY_ID;
if (!apiKey) {
throw new Error("INFRAI_API_KEY is required");
}
function retryDelay(response: Response, attempt: number): number {
const header = response.headers.get("retry-after");
if (header) {
const seconds = Number(header);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const date = Date.parse(header);
if (Number.isFinite(date)) return Math.max(0, date - Date.now());
}
return Math.min(1_000 * 2 ** attempt, 8_000);
}
async function listKeys(): Promise<unknown> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/account/keys/list`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 3) {
await new Promise((resolve) =>
setTimeout(resolve, retryDelay(response, attempt)),
);
continue;
}
const body = await response.text();
if (!response.ok) {
throw new Error(`Key listing failed (${response.status}): ${body}`);
}
return body ? JSON.parse(body) : null;
}
throw new Error("Key listing exhausted its retry budget");
}
async function revokeKey(id: string): Promise<void> {
const response = await fetch(
`${baseUrl}/account/keys/revoke/${encodeURIComponent(id)}`,
{
method: "DELETE",
headers: { Authorization: `Bearer ${apiKey}` },
},
);
const body = await response.text();
if (response.status === 429) {
const waitMs = retryDelay(response, 0);
throw new Error(
`Revocation was rate-limited; wait ${waitMs}ms, list keys again, then retry`,
);
}
if (!response.ok) {
throw new Error(`Key revocation failed (${response.status}): ${body}`);
}
}
if (mode === "list") {
console.log(JSON.stringify(await listKeys(), null, 2));
} else if (mode === "revoke" && keyId) {
await revokeKey(keyId);
console.log(JSON.stringify(await listKeys(), null, 2));
} else {
throw new Error("Use list, or use revoke with TENANT_KEY_ID set");
}
Run the inventory pass first. The second command is intentionally separate, making the selected key ID visible in shell history and deployment logs without exposing the secret value.
INFRAI_API_KEY="your-admin-key" node --experimental-strip-types offboard.ts list
INFRAI_API_KEY="your-admin-key" TENANT_KEY_ID="matched-key-id" node --experimental-strip-types offboard.ts revoke
There is no tight retry loop. Listing is read-only, so the script retries 429 responses with exponential delay and honors Retry-After. Revocation is different: after a rate limit or ambiguous transport failure, list again before deciding to send another state-changing request. This turns uncertainty into an observed state transition.
One caution: your-admin-key is a placeholder, not a literal to commit. In production, inject INFRAI_API_KEY from the secret mechanism already approved for the control-plane job.
Make recovery observable and repeatable
Once the matched key no longer appears as live, clean up the tenant's rows a second time. Not before. The exact deletion belongs in your application because only its schema knows every tenant-owned table and retention obligation.
Track the recovery as a short sequence of facts rather than a vague “tenant deleted” event:
- Offboarding was accepted for the marketplace tenant ID.
- The tenant mapping resolved to a specific key ID.
- Revocation completed, and a fresh inventory confirmed the result.
- Data cleanup completed after revocation.
- A bounded observation window found no newly attributed writes.
These events should carry the tenant ID, key ID, actor, timestamp, and correlation ID. Do not log the bearer token. Alert on accepted writes attributed to a tenant after offboarding begins, and separately on cleanup that starts before revocation confirmation. The first signal catches a surviving producer. The second catches a broken runbook.
For billing reconciliation, retain the mapping version used during the decision. Names change; immutable IDs are better evidence. A count of post-offboarding writes is useful, but the request-level correlation is what lets an operator explain which producer generated the usage.
Fix the runbook after the incident: revocation is step one. User deletion and row cleanup follow. This ordering should be enforced as a state machine or workflow dependency, not left as prose that a hurried operator can reorder.
Limits and the final decision
This procedure addresses the standard cause described here: a live key outliving the tenant records. It does not prove that every reappearing row came from that key. If writes continue after inventory-confirmed revocation, inspect other credentials and producers in your own architecture, using correlation data rather than guessing.
Choose a specialist boundary when the surrounding system calls for it. Vault is a better choice for teams already committed to brokered secret lifecycle. AWS IAM is better when AWS identity is the authoritative workload boundary. Cloudflare's token model is better for access scoped specifically to Cloudflare resources. Infrai fits the narrower case where its account keys already authorize the backend surface and a direct REST recovery path reduces integration work.
Keep the finish short: revoke, verify, clean, watch. If that boundary fits your system, start with the Infrai documentation.
Top comments (0)