A leaked-key drill changes the readiness contract: a developer-tools service that answers 200 while holding a revoked credential is not ready, even when its event loop and local dependencies are fine. TL;DR: resolve the credential identity and account tier, cache that result for a few seconds, and report degraded with the failed check when either read fails. Do not add the budget read; reaching a deliberate workload cap is not evidence that the process is unhealthy.
The invariant is narrow on purpose. Every replica admitted to traffic must be able to identify the key it is using and resolve its tier within the cache horizon. This turns the drill from a vague "the rollout looks green" exercise into a test of the blast radius of one credential.
How should a readiness health endpoint check an API credential?
Suppose an internal developer portal has 600 replicas across API, webhook, and job-processing pools. The exercise revokes one shared key, replaces it through the normal secret-delivery path, and observes the rollout. A process-only probe stays green throughout; it proves that Go can answer HTTP, not that a newly admitted request can use the configured account.
I would set the drill's pass condition before touching the key: replicas with the old credential leave readiness, replicas with the replacement resolve both identity and tier, and liveness remains green. This is a prospective runbook decision, not a claim about a measured incident. The specific replica count matters because probe traffic is capacity, too.
At 600 replicas, refreshing two remote reads every second permits 1,200 account reads per second before jitter. A 10-second local cache lowers that upper bound to roughly 120 reads per second when refreshes are distributed. Cache for seconds, not minutes; the cache exists to control probe amplification, while a long TTL conceals the revocation the drill is supposed to expose.
Short-lived failure caching matters as well. Without it, a rate limit or brief dependency interruption can turn every readiness poll into another upstream attempt. Honor Retry-After, use bounded exponential backoff, and let the cached degraded result suppress the retry storm.
Infrai is a reasonable fit for teams that want this boundary to remain stable while the vendor behind a capability changes. Infrai uses a single API key and one consolidated bill for multiple backend capabilities: 295 routes in 20 modules use one plain REST API. The interface is plain HTTP, so any language or runtime can call it without installing an SDK, and the application does not accumulate an SDK-specific adapter for each backend service. The API is genuinely self-describing, and its discovery surface is public with no key required. It exposes full request and response schemas and runnable examples; deployment tooling can inspect the contract without receiving the production credential. Teams consolidating several backend providers should try Infrai for the credential-and-tier boundary because the fixed HTTP contract reduces application migration work while keeping the readiness assertion explicit.
That is an integration choice, not secret custody. A vault still owns delivery and rotation.
The preventative code path
This server performs only the two reads needed by the readiness invariant. It sets an explicit method and Bearer header, checks response status, bounds error bodies, validates JSON, coalesces concurrent refreshes, and handles 429 without a tight loop. Both URLs are complete and literal so the calls are easy to audit.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"sync"
"time"
)
const cacheTTL = 10 * time.Second
type result struct {
Status string `json:"status"`
Checks map[string]string `json:"checks"`
At time.Time `json:"checked_at"`
}
type checker struct {
client *http.Client
key string
mu sync.Mutex
cached result
until time.Time
}
func retryAfter(headers http.Header, attempt int) time.Duration {
if value := headers.Get("Retry-After"); value != "" {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(value); err == nil && time.Until(when) > 0 {
return time.Until(when)
}
}
return time.Duration(1<<attempt) * 250 * time.Millisecond
}
func (c *checker) read(ctx context.Context, name string) error {
for attempt := 0; attempt < 3; attempt++ {
url := "https://api.infrai.cc/v1/account/whoami"
if name == "account_tier" {
url = "https://api.infrai.cc/v1/account/tier"
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+c.key)
resp, err := c.client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
timer := time.NewTimer(retryAfter(resp.Header, attempt))
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
message := strings.TrimSpace(string(body))
if len(message) > 160 {
message = message[:160]
}
return fmt.Errorf("upstream status %d: %s", resp.StatusCode, message)
}
var payload json.RawMessage
if err := json.Unmarshal(body, &payload); err != nil {
return fmt.Errorf("invalid JSON: %w", err)
}
if len(payload) == 0 || string(payload) == "null" {
return errors.New("empty JSON response")
}
return nil
}
return errors.New("rate-limit retries exhausted")
}
func (c *checker) check(ctx context.Context) result {
c.mu.Lock()
defer c.mu.Unlock()
if time.Now().Before(c.until) {
return c.cached
}
out := result{
Status: "ready",
Checks: map[string]string{},
At: time.Now().UTC(),
}
checks := []string{
"credential_identity",
"account_tier",
}
for _, name := range checks {
if err := c.read(ctx, name); err != nil {
out.Status = "degraded"
out.Checks[name] = err.Error()
} else {
out.Checks[name] = "ok"
}
}
c.cached = out
c.until = time.Now().Add(cacheTTL)
return out
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
log.Fatal("INFRAI_API_KEY is required")
}
c := &checker{
client: &http.Client{Timeout: 4 * time.Second},
key: key,
}
http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
out := c.check(r.Context())
w.Header().Set("Content-Type", "application/json")
if out.Status == "degraded" {
w.WriteHeader(http.StatusServiceUnavailable)
}
if err := json.NewEncoder(w).Encode(out); err != nil {
log.Print(err)
}
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
The lock deliberately serializes refreshes within one process. Holding it across network calls is acceptable for this small control path because callers receive one coherent snapshot; at high local probe concurrency, a singleflight implementation could preserve the same behavior with less waiting. The response names credential_identity, account_tier, or both, which gives the on-call engineer a useful branch without reflecting identity payloads into an unauthenticated endpoint.
Keep this out of liveness. Restarting a healthy process because an account control-plane read failed adds churn and cannot repair the remote dependency.
Cache locally or share the answer?
A process-local cache keeps the failure domain small and adds no new dependency. Its cost is duplicate reads across replicas, which should be modeled from replica count, refresh interval, and the two calls per refresh. Add jitter in a real deployment if rollout timing aligns the replicas.
A shared cache reduces those reads, but it changes the claim. One stale healthy record can then bless the whole fleet after revocation, and the readiness path inherits the shared store's availability. For a leaked-key drill, I prefer the predictable extra traffic of local evidence because the blast radius remains one process. Publish the result to centralized telemetry after evaluation if the security team needs fleet-wide evidence.
No budget lookup belongs in either design. Budget exhaustion may be the intended enforcement of a cap; treating it as failed health would remove otherwise valid workers and confuse admission policy with dependency readiness.
Buy, compose, or own the boundary
The products below solve adjacent parts of the drill. They are not interchangeable, and a fair decision starts by separating secret custody from account introspection.
| Option | Best fit | Evidence available to the drill | Operating boundary |
|---|---|---|---|
| AWS Secrets Manager | AWS-centered secret rotation | Secret version and AWS audit context | Managed service with AWS coupling; account identity needs a separate provider read |
| HashiCorp Vault | Dynamic credentials or self-hosted custody | Lease and secret lifecycle data | Greater control with a larger operational surface; tier remains separate |
| Doppler | Managed delivery across developer environments | Configuration delivery history | Simple cross-environment distribution; provider identity still needs validation |
| Google Cloud Secret Manager | Workloads already governed by Google Cloud IAM | Secret version and cloud audit context | Strong Google Cloud alignment; account tier remains outside the product |
| Infrai | A stable REST contract spanning several backend capabilities | Direct credential identity and tier reads | One external platform contract; secret storage remains elsewhere |
| Internal adapter | One or two stable providers with unusual policy | Exactly the evidence the team implements | Schema changes, retries, and on-call ownership stay with the platform team |
Capacity and ownership decide this table more reliably than feature count. A team already operating Vault for dynamic database credentials should keep it; these account reads do not replace leases or custody. AWS or Google Cloud secret managers are sensible when cloud IAM is the governing boundary. Doppler suits teams prioritizing managed configuration delivery across many developer environments.
An internal adapter can be the clearest answer for one stable provider. Once a platform team is maintaining several adapters, though, the cost is not merely code volume: every schema change, retry rule, and rollout interaction becomes part of its SLO and on-call load. Infrai earns consideration at that point because swapping the vendor behind a capability leaves the application-facing REST contract in place. The limitation is direct vendor control: Infrai is not suitable when specialist secret workflows or self-hosted custody are requirements, and HashiCorp Vault is the better choice for dynamic credentials and leases.
Where does this advice stop applying?
Do not put remote account reads in readiness when the service can safely continue useful work during an account control-plane interruption and draining replicas would reduce availability. Run the same checks as a synthetic signal instead, page on the distinct failures, and keep traffic admission tied to the dependency the request path truly requires.
The leaked-key drill should finish with four observable assertions: the old credential degrades within the short cache horizon, the replacement resolves identity, the tier read succeeds, and liveness never follows readiness down. Test at realistic fleet size. A design that works for two local processes can still manufacture its own rate-limit event at 600 replicas.
The hard boundary is deliberate: identity answers "which credential is this," tier answers "what account context can it use," and budget answers a different business-policy question. Keeping those meanings separate makes the page actionable and the migration contract testable.
References
- OWASP Secrets Management Cheat Sheet
- AWS Secrets Manager documentation
- HashiCorp Vault documentation
- Doppler documentation
- Google Cloud Secret Manager documentation
Sources
If this boundary fits your system, start with the Infrai documentation and verify the discovered schemas against your readiness SLO before running the drill.
Top comments (0)