A readiness health endpoint for an e-commerce API should check more than process uptime: during a leaked-key drill, it must stop every node from admitting traffic under stale credential and account-tier state while billing events remain attributable to the replacement key.
Short answer: make the readiness endpoint evaluate credential verification and tier-cache age as separate dependencies, return 503 for an invalid credential or a cache older than its bounded grace period, and expose a degraded-but-ready state only while a previously verified tier value is still fresh enough to preserve correct billing attribution.
This is a narrow contract. Liveness should still answer whether the process can make progress; readiness answers whether this node should receive a new checkout request. Mixing those questions invites a restart loop during a dependency incident and tells the scheduler nothing useful about whether routing one more paid request is safe.
What should a readiness health endpoint check for cached API credential and tier data?
The endpoint should check four things: whether the configured credential has been verified, which credential fingerprint was verified, when the account tier was last refreshed, and whether the cached tier remains inside an explicit grace window. Never return the secret, even in a digestible diagnostic payload. OWASP recommends centralized lifecycle management, rotation, revocation, least privilege, and auditability for secrets; a readiness response is an operational signal, not a secret-discovery interface.
For the leaked-key drill, the important invariant is stronger than “the API answers.” A node may accept a checkout only if its verified credential generation matches the active generation and its tier snapshot is recent enough that usage can be posted to the intended account. A stale tier can silently damage attribution even when authentication succeeds. That is why I would model these as independent facts instead of one green boolean — it keeps the drill honest when rotation succeeds but cache refresh lags.
Use three externally meaningful states:
| State | Credential | Tier cache | Routing decision |
|---|---|---|---|
| ready | verified active generation | fresh | admit new traffic |
| degraded | verified active generation | stale, but inside grace | admit traffic and alert |
| not ready | invalid, wrong generation, missing, or unverifiable | expired or unknown | drain new traffic |
The grace window is a risk budget, not a convenience timeout. Set it below the shortest period in which a wrong tier could breach the billing-attribution SLO, then subtract detection and drain time. If the business requires attribution to change immediately after rotation, the grace budget is zero and “degraded” cannot be a ready state.
No ambiguity there.
Treat the leaked-key drill as a state transition
A useful drill starts before revocation. Record the active credential generation, the tier snapshot version, cache age, ready-node count, and the correlation identifier used by billing events. Rotate to a new secret, force each node to re-verify it through the normal code path, revoke the old secret, and watch old-generation nodes leave readiness. Finally, send a bounded synthetic checkout and confirm that its usage record carries the new credential fingerprint and expected account tier. Do not put the raw key in logs or labels.
The ugly failure mode is split state: eight nodes have verified generation 42, two still believe generation 41 is active, and both groups emit superficially valid health responses. A load balancer cannot repair that. The readiness decision has to compare local verified state with an authoritative generation marker, while the rollout policy must retain enough ready capacity for the checkout SLO. With ten nodes and a required minimum of eight ready nodes, for example, rotating more than two at once consumes the entire readiness budget; that arithmetic belongs in the deployment controller before anyone starts the drill. These are example planning numbers, not universal thresholds.
Keep billing verification outside the health request itself. A probe that writes a charge or usage event on every poll creates cost, noise, and a new failure surface. Instead, the readiness path reads already-maintained verification state, while one bounded synthetic transaction at the end of the drill proves attribution end to end.
Fast probe, deep drill.
Implement a bounded readiness state machine in Go
The handler below has no network call in the request path. A separate reconciler owns credential verification and tier refresh, then publishes an immutable snapshot. That separation gives the probe a predictable latency ceiling and prevents a burst of health checks from amplifying trouble upstream.
package readiness
import (
"encoding/json"
"net/http"
"sync/atomic"
"time"
)
type Snapshot struct {
CredentialVerified bool
CredentialGeneration string
ActiveGeneration string
TierKnown bool
TierUpdatedAt time.Time
}
type Store struct {
value atomic.Value
}
func (s *Store) Publish(snapshot Snapshot) {
s.value.Store(snapshot)
}
func (s *Store) Load() (Snapshot, bool) {
v := s.value.Load()
if v == nil {
return Snapshot{}, false
}
return v.(Snapshot), true
}
type Handler struct {
Store *Store
Now func() time.Time
FreshFor time.Duration
GraceFor time.Duration
}
type response struct {
Status string `json:"status"`
Reason string `json:"reason,omitempty"`
TierAgeSeconds int64 `json:"tier_age_seconds,omitempty"`
}
func (h Handler) ServeHTTP(w http.ResponseWriter, _ *http.Request) {
snapshot, ok := h.Store.Load()
if !ok {
h.write(w, http.StatusServiceUnavailable, response{Status: "not_ready", Reason: "state_unknown"})
return
}
credentialReady := snapshot.CredentialVerified &&
snapshot.CredentialGeneration != "" &&
snapshot.CredentialGeneration == snapshot.ActiveGeneration
if !credentialReady {
h.write(w, http.StatusServiceUnavailable, response{Status: "not_ready", Reason: "credential_unverified"})
return
}
if !snapshot.TierKnown {
h.write(w, http.StatusServiceUnavailable, response{Status: "not_ready", Reason: "tier_unknown"})
return
}
age := h.Now().Sub(snapshot.TierUpdatedAt)
if age < 0 {
age = 0
}
body := response{Status: "ready", TierAgeSeconds: int64(age.Seconds())}
if age > h.FreshFor+h.GraceFor {
body.Status = "not_ready"
body.Reason = "tier_cache_expired"
h.write(w, http.StatusServiceUnavailable, body)
return
}
if age > h.FreshFor {
body.Status = "degraded"
body.Reason = "tier_cache_stale"
}
h.write(w, http.StatusOK, body)
}
func (h Handler) write(w http.ResponseWriter, status int, body response) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Cache-Control", "no-store")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(body)
}
Register it on a dedicated internal listener or protect it with network policy; public diagnostics give an attacker timing information about rotation. Also validate FreshFor and GraceFor as positive deployment configuration. The standard library deliberately ignores the encoder error here because headers have already been sent, but production code should increment an internal encode-error counter without changing the response contract.
I wouldn't include account IDs, tier names, key prefixes, upstream error text, or timestamps precise enough to fingerprint rotation in this payload. The stable reason codes are sufficient for automation. Rich detail belongs in access-controlled metrics and structured logs, joined by node and deployment identifiers.
Prove degradation without turning readiness into monitoring
Test the state machine with a fake clock: unknown snapshot, wrong credential generation, unknown tier, fresh tier, stale-within-grace tier, and expired tier. Assert both status code and reason code. Then run a deployment test that advances cache age across both boundaries while requests are flowing; the expected sequence is ready, degraded, not_ready, with no return to ready until a newly verified snapshot is published.
Readiness is still the wrong place to establish the full SLO. Export counters for each transition, a gauge for tier-cache age, and the count of ready nodes by credential generation. Alert on sustained degraded state and on mixed generations, but page only when the user-facing or attribution error budget is threatened. I'm not sure a universal degraded-duration threshold exists; traffic shape, drain latency, and the financial impact of misattribution determine it, and a load test plus one observed rotation is what resolves that uncertainty.
The catch is that cached degradation is not suitable when every request needs a strongly current entitlement, tier downgrade, fraud hold, or spending limit. Fail closed and perform an authoritative check in those paths, accepting the latency and dependency cost. Conversely, a self-hosted verifier may be justified when offline operation and control outweigh on-call load; a managed control plane may fit a small platform team that can accept its availability and lock-in boundaries.
| Decision | Buy or managed control plane | Build or self-host |
|---|---|---|
| Rotation workflow | lower implementation load; external dependency | full policy control; team owns automation |
| Attribution evidence | verify export and audit semantics | define event schema and retention directly |
| Probe integration | adapt vendor state to local contract | local contract is native, but maintenance is yours |
| On-call burden | dependency escalation and contract review | capacity, upgrades, security patches, recovery |
Neither column removes the need for a local readiness contract. The decision rule is whether the team can name an owner, an SLO, a capacity budget, and a recovery test for the chosen side. If it cannot, the architecture isn't ready for a leaked-key drill.
Top comments (0)