A readiness check that calls an account API on every request creates the wrong dependency: an otherwise healthy e-commerce node can leave the load balancer because a control-plane lookup slowed down. For a 2-node Node.js service, choose a cached readiness snapshot fed by a bounded background probe; mark the node degraded when credential or tier evidence goes stale, and reserve not-ready for states that make serving unsafe.
Short answer: Validate the API credential and billing tier out of band, cache the last verified result with its age and attribution fields, and let readiness make a local, deterministic decision.
This distinction matters during a leaked-key drill. The drill must prove that the old credential was revoked, the replacement maps to the intended account and tier, and each node adopted it before traffic returns. A green process check proves none of those things. A live upstream check proves more, but it also imports upstream latency and rate limits into every readiness decision.
What should a Node.js readiness health endpoint cache for a degraded API credential and tier?
Cache evidence, not a bare Boolean. The snapshot needs the credential generation or non-secret fingerprint, the observed account identifier, the observed tier, the time of the last successful verification, and a reason code suitable for metrics. Never store the credential itself in the health payload. OWASP recommends limiting secret exposure, rotating secrets, and logging secret-management events without logging the secret; a one-way fingerprint of the key identifier supports the drill while keeping the key out of logs and responses.
The endpoint should expose only what an orchestrator and an operator need. A public body might contain status, credential_generation, tier, checked_at, and reason. Keep account identifiers behind authenticated diagnostics if they are sensitive. The cache is per node because the operational question is per node: did this process load generation 18, or is one replica still serving with generation 17?
Use three states. ready means the latest verified credential generation and tier match the deployment's expected attribution. degraded means the cached evidence is older than the warning budget but still within the serving budget, so traffic continues while an alert fires. not_ready means the evidence has crossed the hard freshness limit, the credential was rejected, or the account or tier does not match. This policy needs two timers rather than one because a single threshold turns harmless probe jitter into synchronized eviction.
I'm not sure what freshness window fits every account API; nobody can determine that from HTTP semantics alone. Set it from the leaked-key revocation objective, the provider's documented rate limits, and the number of replicas. If the security objective says every node must adopt a replacement within 10 minutes, a 15-minute cache is indefensible.
The incident invariant: attribution must survive rotation
Treat the leaked-key exercise as a state transition with evidence. Before rotation, record the expected account and billing tier through a protected control-plane check. Issue the replacement through the approved secret manager, deploy a new generation, then revoke the old credential and verify every node independently. In a 2-node rehearsal, hold the rollout after the first replacement: node A should report generation 18 with the expected tier while node B still reports generation 17, so the deployment gate remains closed even though both processes answer ordinary health checks. Resume only after node B publishes a verified generation 18 snapshot. Then revoke generation 17 and confirm that neither node can return to ready with it. This is test data and a procedure, not a claim about a production incident. The invariant is precise: no ready node may serve with an unverified generation, and no verified generation may resolve to the wrong billing account or tier.
That last clause catches an expensive class of mistakes. A replacement key can authenticate successfully yet belong to a staging account, a different merchant, or a tier with different quotas. Authentication answers "is this credential accepted?" Attribution answers "who will own the usage?" The drill is incomplete until both answers are checked.
One node is easy.
With two replicas, a rolling deployment can leave one process on the previous secret because of a stale mount or missed reload; with more replicas, the tail gets harder to see, so the deployment gate should compare the count of ready nodes at the new generation with the desired replica count. Use a gauge grouped by generation and reason, never by raw key fingerprint, to avoid turning secrets or high-cardinality identifiers into metric labels.
How do live probes and cached snapshots make different failure promises?
| Design | Attribution evidence | Failure coupling | Best fit | Catch |
|---|---|---|---|---|
| Live account lookup inside readiness | Fresh on every probe | Readiness inherits account API latency, quotas, and network reachability | Low-frequency administrative checks | A control-plane slowdown can evict healthy serving nodes |
| Background lookup plus local snapshot | Bounded by the configured age limit | Request path is isolated from probe latency | Multi-node services with an explicit rotation SLO | Evidence can be stale until the hard limit |
| Startup-only validation | Accurate only at process start | No recurring account API dependency | Immutable short-lived jobs | It cannot detect later revocation or tier drift |
The capacity math should be done before choosing an interval. If N replicas probe every T seconds, steady-state load is approximately N/T requests per second, before retries. Add jitter so a deployment doesn't align every node on the same second, cap each attempt with a timeout shorter than the interval, and allow only one in-flight probe per node. Retries need a budget too; unconstrained exponential retry can turn an upstream impairment into a self-inflicted burst.
Don't hide the distinction between degraded and not_ready. A degraded response can remain HTTP 200 if the load balancer only understands binary health, while the body and metric carry the warning. Return HTTP 503 only when local policy says this node must stop receiving traffic. That policy should be identical across replicas and covered by tests around both time boundaries.
A preventative Go sidecar path
The implementation below can run beside the Node.js service. It keeps the network verifier behind an interface, publishes snapshots atomically, and makes the HTTP handler local. The verifier's concrete account endpoint is deliberately absent because route and response shapes are provider-specific; binding invented paths into readiness code is worse than leaving the adapter explicit.
package readiness
import (
"context"
"encoding/json"
"net/http"
"sync/atomic"
"time"
)
type Evidence struct {
Status string `json:"status"`
CredentialGeneration int `json:"credential_generation"`
Tier string `json:"tier"`
CheckedAt time.Time `json:"checked_at"`
Reason string `json:"reason"`
}
type Observation struct {
CredentialGeneration int
AccountID string
Tier string
}
type Verifier interface {
Verify(ctx context.Context) (Observation, error)
}
type Checker struct {
latest atomic.Pointer[Evidence]
expectedGeneration int
expectedAccount string
expectedTier string
degradeAfter time.Duration
failAfter time.Duration
}
func (c *Checker) Publish(obs Observation, checkedAt time.Time) {
status, reason := "ready", "verified"
if obs.CredentialGeneration != c.expectedGeneration {
status, reason = "not_ready", "credential_generation_mismatch"
} else if obs.AccountID != c.expectedAccount || obs.Tier != c.expectedTier {
status, reason = "not_ready", "billing_attribution_mismatch"
}
c.latest.Store(&Evidence{status, obs.CredentialGeneration, obs.Tier, checkedAt, reason})
}
func (c *Checker) Handler(now func() time.Time) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
evidence := c.latest.Load()
if evidence == nil {
writeEvidence(w, http.StatusServiceUnavailable, Evidence{Status: "not_ready", Reason: "never_verified"})
return
}
result := *evidence
age := now().Sub(result.CheckedAt)
code := http.StatusOK
if age > c.failAfter {
result.Status, result.Reason = "not_ready", "evidence_expired"
code = http.StatusServiceUnavailable
} else if age > c.degradeAfter && result.Status == "ready" {
result.Status, result.Reason = "degraded", "evidence_stale"
} else if result.Status == "not_ready" {
code = http.StatusServiceUnavailable
}
writeEvidence(w, code, result)
})
}
func writeEvidence(w http.ResponseWriter, code int, evidence Evidence) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(evidence)
}
The background loop should call Verifier.Verify on a jittered schedule and invoke Publish only with a complete observation. On transport errors, retain the last successful snapshot and let its age move through degraded to not-ready; separately increment a low-cardinality failure counter. This preserves the difference between "the last verified attribution is aging" and "the credential belongs to the wrong account," which leads to very different operator action.
Test the handler with a fake clock at degradeAfter - 1ns, exactly at the boundary, and failAfter + 1ns. Test generation, account, and tier mismatches independently. Then run the drill with both replicas visible: rotate, observe generation 18 on each node, revoke generation 17, and confirm that an intentionally stale test replica leaves readiness only after the hard budget.
No guesswork.
When should you choose the live check instead?
Stick with a live check when the endpoint is an authenticated, low-frequency administrative gate and fresh control-plane truth is more important than serving continuity. It is also reasonable for a deployment preflight that runs once, has a strict timeout, and fails closed before any traffic reaches the process.
Cached readiness is not suitable when revocation must take effect faster than the shortest safe polling interval, or when policy forbids serving on previously verified evidence. In that case, use request-time authorization or a locally pushed revocation signal; readiness polling cannot promise instantaneous revocation. Startup-only validation remains a good fit for short-lived, immutable workers that cannot outlive the credential's intended window.
The trade is explicit: caching reduces failure coupling but spends a bounded amount of freshness. The right design is the one whose hard age limit fits the revocation SLO and whose attribution fields prove that accepted traffic reaches the intended bill.
Top comments (0)