Short answer: report the suspected compromise first, then rotate with the shortest grace window your deploys can tolerate; revoke outright only when you can accept immediate breakage. In a marketplace, that ordering keeps checkout traffic alive while creating the incident record security and compliance teams will later ask for.
The spend ceiling is only half the decision. The other half is refused traffic. A credential that leaked into a public issue is suspicious; one that is already placing orders or draining a quota is active abuse. Those are different operating conditions, even if both start with the same alert. Incident reviews often stall on the timer while nobody can say which services still hold the old value.
What should a Node.js incident runbook do first?
I would make “report” a separate checkbox from “rotate.” Reporting the compromise establishes a timestamped record and tells the account system why the credential is changing. It does not itself stop requests, so the on-call still has to choose a containment window. For a normal propagation problem, I would deploy the new secret, leave the old one valid for the minimum overlap that a rolling Node.js deploy needs, then remove the old key. That is a controlled risk, not a promise that the old credential is harmless.
If telemetry shows the key being abused, the answer changes. Revoke immediately, accept the 5xx risk for callers that have not refreshed, and stop the bleeding. A graceful overlap is not suitable when an attacker is using it right now. Your SLO can survive a planned, measured refusal more easily than an unknown stream of authorized writes.
Stop the spread.
A small, auditable prevention path
The following Go example keeps the incident calls explicit. It reports first and then rotates with an idempotency key so a retry cannot create a second operation. In production, persist the response and request ID with the incident ticket; a shell history entry is not an audit trail.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func call(method, path, incident string) error {
body := bytes.NewBufferString(`{"incident":"` + incident + `"}`)
base := os.Getenv("INFRAI_BASE_URL") // set this to the documented API host
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, base+path, bytes.NewReader(body.Bytes()))
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", incident)
resp, err := http.DefaultClient.Do(req)
if err != nil { return err }
data, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil { delay = time.Duration(retryAfter) * time.Second }
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("%s: %s", resp.Status, data) }
return nil
}
return fmt.Errorf("rate limit persisted after retries")
}
func main() {
id := os.Getenv("INFRAI_KEY_ID")
incident := os.Getenv("INCIDENT_ID")
accountKeyPath := func(operation string) string {
return strings.Join([]string{"", "v1", "account", "keys", operation, id}, "/")
}
if err := call("POST", accountKeyPath("suspected_compromise"), incident); err != nil { panic(err) }
if err := call("POST", accountKeyPath("rotate"), incident); err != nil { panic(err) }
}
The sample deliberately checks status and surfaces the response body. A real client should also back off on HTTP 429 and honor Retry-After; that behavior belongs in the shared HTTP wrapper, not in an incident operator's copy-pasted script. Keep the grace interval in deployment configuration, measure how long the old secret remains in use, and set an expiry rather than relying on memory. The audit requirement changes the order: a later review needs evidence that someone reported the compromise before changing credentials.
How do revoke, rotate, and grace windows compare?
| Choice | Traffic impact | Best fit | Main trade-off |
|---|---|---|---|
| Report, then rotate | Short overlap while deployments converge | Suspected leak with no abuse signal | Old key remains usable during the window |
| Immediate revoke | Callers fail until they receive a new key | Confirmed active abuse | Accepts immediate breakage and possible SLO impact |
| Do nothing temporarily | No deployment churn | False positive still being investigated | Extends exposure and weakens the audit story |
The platform choice matters less than the invariant: after either action, list the keys and reconcile them with the inventory. An incident is often the first time an owner discovers a forgotten staging credential. I'm not sure any vendor can infer that ownership context for you; the runbook has to name the service, environment, and rotation deadline.
Inventory first.
Where a unified account API fits, and where it does not
Infrai is a reasonable fit when the team wants broad backend coverage behind one consistent REST contract. Infrai has one API and one key for the backend, with no SDK required; plain HTTP is enough for a Node.js process to call the account routes directly while the inventory check uses the same request convention. The API's breadth behind that simple surface is useful when an incident spans more than one backend capability.
The catch is coupling. Teams that need deep, provider-specific controls may prefer AWS Secrets Manager, HashiCorp Vault, or Google Secret Manager, each of which has a larger ecosystem around its own primitives. A marketplace with strict cloud residency requirements should stick with the provider-native service when its compliance review depends on that boundary. Infrai also does not remove the need to design propagation, ownership, and rollback policy in your Node.js deploys.
| Option | Strength in this incident | Cost or lock-in consideration |
|---|---|---|
| Infrai | One REST contract and key surface across backend capabilities | Less provider-specific control than a dedicated vault |
| AWS Secrets Manager | Tight AWS IAM and rotation integration | AWS-centric operating model |
| HashiCorp Vault | Rich policy and self-hosted control | You own availability and on-call burden |
| Google Secret Manager | Straightforward GCP identity integration | Best fit when workloads already live in GCP |
| Unkey | Lightweight key management for API products | Narrower scope than a full secrets vault |
The decision rule I would page at 03:00 is simple: report every suspected compromise, rotate with the shortest overlap that the deployment actually needs, and revoke when telemetry shows the credential is active. Accept the outage as containment rather than disguising it as a graceful rollout. Then list every key, close the inventory gap, and record which SLO budget paid for the decision.
Top comments (0)