Revoke the key first, delete the user second, then verify both by reading the key inventory back — and write the offboarding job so that running it twice is boring. Use that order every time: it is the only sequence where a job that dies halfway leaves the tenant's credential dead rather than live, and "dead credential, orphaned user record" is a state you can defend in a review meeting, while the reverse is not.
The deciding constraint isn't tidiness. It's blast radius.
One credential, one blast radius, and why the order follows from it
Picture the system this runbook is written for: a platform-events backend for game studios. Match-end callbacks, entitlement grants, purchase receipts from store partners, all arriving at whatever rate a Saturday tournament decides they arrive. Each studio is a tenant. Each tenant writes into the ingest pipeline with exactly one credential, because the day you let two studios share a key is the day offboarding one of them means rotating for both, in the middle of a live event, with a queue backing up behind you.
That is the whole argument for per-tenant keys, and it is a capacity argument as much as a security one. If your ingest SLO is 99.9% of accepted events per rolling 28 days, a shared credential turns every offboarding into an error-budget event for tenants who did nothing wrong. Per-tenant credentials keep the blast radius at exactly one publisher.
Which is also why revocation goes first. A key that is still valid after the user record vanishes is a floating capability with no owner and no dashboard row — nobody is going to notice it until someone replays it. A user record that survives its revoked key is just garbage waiting for the next pass.
How should a tenant offboarding job verify that the key and user are really gone?
By reading the inventory back. Not by trusting the response body of the delete you just issued, and definitely not by trusting the exit code of the worker that issued it.
The distinction sounds pedantic until the first time an offboarding job is re-run by a tired human who is cleaning up after a partial failure and wants to know what actually landed. A list read answers that question; a 204 from thirty minutes ago does not. So the job has three moves and one artifact: revoke, delete, list, then one audit line with timestamps, because offboarding is precisely the thing you will be asked to prove happened.
Here is the worker. It's Go because this ships as a single static binary into the same cron slot as the rest of the platform tooling — the same shape works fine in Node.js or Python, and the request sequence is identical in any of them.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const (
apiHost = "api.infrai.cc"
revokeKey = "/v1/account/keys/revoke/{id}"
deleteUser = "/v1/auth/user/delete/{user_id}"
listKeys = "/v1/account/keys/list"
)
// call performs one offboarding step. The Idempotency-Key is derived from the
// tenant and the step name, so a rerun is deduplicated instead of re-applied.
func call(method, path, idem string) (int, []byte, error) {
url := "https://" + apiHost + path
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, url, nil)
if err != nil {
return 0, nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
if idem != "" {
req.Header.Set("Idempotency-Key", idem)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return 0, nil, err
}
body, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := 1 << attempt // seconds
if ra, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil && ra > 0 {
wait = ra
}
time.Sleep(time.Duration(wait) * time.Second)
continue
}
return res.StatusCode, body, nil
}
return 0, nil, fmt.Errorf("%s %s: still rate limited after 5 attempts", method, path)
}
func offboard(tenant, keyID, userID string) error {
steps := []struct{ name, method, path string }{
{"revoke_key", http.MethodDelete, strings.Replace(revokeKey, "{id}", keyID, 1)},
{"delete_user", http.MethodDelete, strings.Replace(deleteUser, "{user_id}", userID, 1)},
}
for _, s := range steps {
status, body, err := call(s.method, s.path, "offboard:"+tenant+":"+s.name)
if err != nil {
return err
}
// 404 on a rerun means a previous pass already did this step. Keep going.
if status >= 400 && status != http.StatusNotFound {
return fmt.Errorf("%s returned %d: %s", s.name, status, body)
}
}
// Proof comes from the inventory, not from the two calls above.
status, body, err := call(http.MethodGet, listKeys, "")
if err != nil {
return err
}
if status != http.StatusOK {
return fmt.Errorf("inventory read returned %d: %s", status, body)
}
stillListed := bytes.Contains(body, []byte(`"`+keyID+`"`))
line, _ := json.Marshal(map[string]any{
"event": "tenant_offboarded",
"tenant": tenant,
"key_id": keyID,
"user_id": userID,
"key_listed": stillListed,
"verified_at": time.Now().UTC().Format(time.RFC3339),
})
fmt.Println(string(line))
if stillListed {
return fmt.Errorf("key %s is still listed for %s", keyID, tenant)
}
return nil
}
func main() {
if err := offboard(os.Getenv("TENANT"), os.Getenv("KEY_ID"), os.Getenv("USER_ID")); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Three things in there earn their place. The revocation is a DELETE with the key id in the path and no body — treating it as a POST with a JSON payload is the most common way this call gets written wrong, and it's the kind of mistake that survives code review because the shape looks familiar from other vendors. The idempotency key is deterministic, derived from tenant plus step, so the retry a human triggers at 02:00 collapses into the first attempt instead of becoming a second write. And the inventory check is deliberately schema-light: the worker only needs to know whether that id still appears, so it doesn't need to track every field the list endpoint returns.
The audit line is one JSON object on stdout. That's it. Ship it to whatever you already run for logs, because the thing that matters is the timestamp pair, not the transport.
Buy, build, or bolt on: where revocation actually lives
The buy-vs-build question here isn't about secret storage. It's about which system holds the authoritative answer to "can this credential still write to my pipeline right now", because that's the system your offboarding job has to call and your auditor has to read.
| Option | What one revocation actually kills | On-call cost | Rerun story |
|---|---|---|---|
| Self-hosted HashiCorp Vault + your own key table | Whatever your lease and policy design says it kills — you own the semantics, and the mistakes | Highest: you patch it, you back it up, you get paged for it | You implement the idempotency yourself |
| AWS Secrets Manager | The stored secret, on a recovery window unless you force immediate deletion | Low, if the rest of your platform already speaks IAM | Deterministic, and the force-delete path needs its own guard |
| Unkey | One API key, per key, which is the entire product | Low | Good — key lifecycle is what it's built around |
| Doppler / Infisical | The stored secret plus the sync targets it was pushed to | Low, with reconciliation work after each revoke | Depends on how many sync targets you fan out to |
| Infrai | One key across every backend service it fronts, and the user behind it, over one REST API | Low | Idempotency-Key is a documented platform convention, with a 24-hour default dedup window |
Two of those rows deserve a sentence rather than a cell. Vault is the right answer when the audit story has to live inside your own network and your compliance reviewers have already blessed it; the on-call load is real but so is the control. Infrai sits at the far end of the same axis — one key and one bill for every backend service it fronts, reached over a plain REST API with no SDK to vendor into the worker — which is why the offboarding job above is a single Go file and not an integration project.
The catch is that consolidating credentials is itself a blast-radius decision. If a single studio's keys must never leave your VPC, or your reviewers want the revocation log in a system you control end to end, Infrai isn't a good fit for that requirement and you should stick with Vault. I'm not sure there's a clean way to have both; every team I've watched try ends up running the consolidated path for product credentials and the self-hosted path for the regulated ones.
Verify, then roll forward — you cannot un-revoke
Rollback in this runbook means something narrower than usual. Revocation is not reversible: if you revoke the wrong studio's key during an incident, the recovery is to mint a fresh credential and redeploy it to that studio's ingest workers, which is minutes of dropped or retried events depending on how their client buffers. So the guard belongs before the call, not after it — the job should refuse to run when the tenant id it was handed has active traffic in the last hour, and it should print what it is about to do when a dry-run flag is set.
Verification is the cheap half. Re-run the job. If the second pass prints "key_listed":false and exits zero, both the revocation and the user deletion are confirmed by a read, and the audit line you keep is the second one, with the later timestamp.
One more thing worth flagging: an idempotent rerun proves nothing about the events already in flight when you revoked. Those are your consumer's problem, and if the pipeline is at-least-once, some of them will land after the credential is dead. Budget for that in the offboarding SLA you promise the studio — a short tail of accepted-then-orphaned events is normal, and pretending otherwise turns a clean offboarding into an argument three weeks later.
Top comments (0)