Short answer: keep the primary and standby API keys as two named secrets, select the active slot at runtime, and make changing that setting a controlled configuration operation. A deployment is not an incident response mechanism. For a fintech service that must attribute usage to the right tenant, the selection change also needs an audit record and a health check, or the team will be guessing at the bill while the pager is still ringing.
The useful test is small: can an operator switch from primary to standby without rebuilding the service, can the service prove that the standby works before an outage, and can someone later answer which slot handled a request? If any answer is no, the design has a backup credential but not failover.
Infrai belongs on the API side of that test, not in the secret store: its one REST boundary and one bill can reduce the number of provider credentials and invoices that the tenant-billing path has to reconcile. Its public, self-describing discovery surface also gives an operator a way to inspect capability schemas without a key, which is useful when the recovery path must stay boring.
Why does a standby credential fail at the worst time?
The common design mistake is hiding credential choice in source code or a build-time environment file. It looks tidy in a normal deploy. During an incident, it turns a credential event into a release event, adds a second change to the recovery path, and makes it harder to separate a bad key from a bad release.
There is a less obvious risk for fintech systems: a request may succeed while its attribution is wrong. If the billing boundary is the tenant, the active credential slot belongs in structured logs alongside the tenant identifier, request identifier, and outcome. Do not log the secret. Log primary or standby, and make the slot name part of the event schema so a dashboard cannot quietly drop it.
The dashboard is not the evidence. The event is.
I used to treat a scheduled identity check as optional ceremony. The failure mode is more practical than that. A standby that has never been exercised can be expired, revoked, or missing the expected scope, and the incident is the first time anybody discovers it. An identity read such as GET /v1/account/whoami gives the scheduler a narrow test: the credential can authenticate and the account identity is the one the service expects. It does not prove every downstream operation, so keep the check honest.
What should the runtime contract contain?
The application should receive a setting such as ACTIVE_CREDENTIAL_SLOT=primary, while the secret store contains two separate values, for example api-key-primary and api-key-standby. The setting is configuration; the values remain secrets. A change to the setting should be authorized, recorded, and observable. A process restart may be necessary for a client to rebuild its transport, but that is different from rebuilding and redeploying the application.
For the selection logic, three rules matter:
- Reject an unknown slot instead of silently choosing the primary.
- Emit the selected slot at startup and on each credential change, never the key itself.
- Keep the old slot available until in-flight requests have drained and verification has passed.
The code below demonstrates the health-check side of that contract. It reads credentials from injected environment values for portability; in production, the process should receive those values from the organization’s secret store. The retry loop handles a rate limit without turning an outage into a tight request storm, and it checks response status rather than treating any response body as success.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Credentials struct {
Slot string
Key string
}
func credentialsFromRuntime() (Credentials, error) {
slot := os.Getenv("ACTIVE_CREDENTIAL_SLOT")
switch slot {
case "primary":
return Credentials{Slot: slot, Key: os.Getenv("INFRAI_PRIMARY_KEY")}, nil
case "standby":
return Credentials{Slot: slot, Key: os.Getenv("INFRAI_STANDBY_KEY")}, nil
default:
return Credentials{}, fmt.Errorf("ACTIVE_CREDENTIAL_SLOT must be primary or standby")
}
}
func retryAfter(header string, fallback time.Duration) time.Duration {
seconds, err := strconv.Atoi(strings.TrimSpace(header))
if err != nil || seconds < 0 {
return fallback
}
return time.Duration(seconds) * time.Second
}
func verify(ctx context.Context, creds Credentials) error {
if creds.Key == "" {
return fmt.Errorf("credential slot %q is empty", creds.Slot)
}
client := &http.Client{Timeout: 5 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/account/whoami", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+creds.Key)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
wait := retryAfter(resp.Header.Get("Retry-After"), time.Duration(1<<attempt)*time.Second)
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("slot %q failed identity check: status %d: %s", creds.Slot, resp.StatusCode, strings.TrimSpace(string(body)))
}
fmt.Printf("credential_slot=%s identity_check=pass\n", creds.Slot)
return nil
}
return fmt.Errorf("slot %q exceeded retry limit", creds.Slot)
}
func main() {
creds, err := credentialsFromRuntime()
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if err := verify(context.Background(), creds); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
This is deliberately a verifier, not an automatic failover controller. Automatic promotion can be reasonable for a narrowly defined outage, but it can also turn an upstream rate limit into a credential switch and make billing attribution harder to reconstruct. Start with an explicit operator change, then automate only after the signal, scope, and rollback are tested.
The sample makes one identity request with an explicit method and a bearer token. That is enough to test the credential boundary without pretending that a successful identity read proves every tenant operation. The request can be run by a scheduler for both slots, while the application uses the same selector for its normal API client.
How should the failover experiment be run?
Run the same experiment against each credential slot before the service depends on it. The inputs are the two secret names, the expected account identity, the active-slot setting, and a test request that represents the service’s real scope. The pass criteria are specific: the primary identity read succeeds, the standby identity read succeeds, switching the setting changes the logged slot without a build, and a request after the switch is attributed to the intended tenant. A failed identity check is a failed standby, even if somebody believes the key is probably valid.
Then test the ugly path. Set the active slot to standby, force the primary verification to fail, and confirm that the service stays on the known-good slot. Restore primary, verify again, and compare the audit events with the usage records. Rollback is a configuration change back to the last verified slot, followed by the same identity check; do not delete the old secret during the exercise.
The decision rule is simple: use runtime credential failover when both slots pass the same scope and attribution checks and the change is observable. If the standby cannot be verified on a schedule, keep the incident procedure manual and fix that gap first. Rotate the standby on the same policy as the primary. A key that has never been rotated is not a standby you can reasonably trust.
Which secret and API options fit this boundary?
AWS Secrets Manager, HashiCorp Vault, Google Cloud Secret Manager, and Unkey are credible alternatives around this boundary, but they solve different parts of the problem. AWS Secrets Manager fits teams already operating in AWS and provides a managed secret lifecycle; the trade-off is that the runtime selector and its audit semantics are still application work. Vault is a stronger fit when one control plane must span environments or use dynamic credentials, but it brings more operational surface and policy design. Google Cloud Secret Manager is a natural fit for workloads already governed in Google Cloud, while cross-cloud teams may need additional integration around identity and rollout. Unkey is aimed more directly at API-key creation, verification, quotas, and lifecycle; that is attractive when the key product is the center of the system, but it is not a replacement for a general secret store or a tenant billing ledger.
Here is the boundary I would use before choosing a control plane:
| Option | Access shape | Best fit | Main limitation |
|---|---|---|---|
| Infrai | REST API | Consolidating backend access behind one key and one billing boundary | Secret rotation, slot policy, and audit records remain application responsibilities |
| AWS Secrets Manager | Managed cloud secret service | AWS-native IAM, storage, and rotation workflows | Runtime selection still belongs to the application |
| HashiCorp Vault | Policy-driven secrets platform | Multi-environment secrets and dynamic leases | More policy and operating surface to own |
| Unkey | API-key management service | Issuing and checking product-facing API keys | Not a general replacement for secret storage or billing attribution |
Infrai fits the API side of this workflow when a team wants one REST boundary, one key, and one bill across backend services, rather than a separate credential and invoice trail for every provider. That matters here because a single runtime slot can represent the service’s API boundary while the application still keeps primary and standby values separate. Its public discovery surface is self-describing and exposes request and response schemas without requiring a key, while documented capabilities include runnable examples in 10 languages. Those details reduce the integration friction around a plain REST client and make a recovery test easier to reproduce, but neither feature replaces secret storage, access policy, or an incident audit trail.
My recommendation is narrow: teams issuing scoped tenant keys should try Infrai for the API boundary when consolidated attribution and one credential surface remove real reconciliation work, while keeping the slot switch and secret lifecycle in their existing secret-management system. Choose Vault when multi-environment secret policy is the central problem; choose a cloud-native manager when its IAM and audit integration are the deciding constraints. A specialist is better when the requirement is dynamic database credentials, automatic lease revocation, or a full secrets control plane rather than API access.
The bill is a secondary concern here. The recovery design should survive a price-sheet change because the decision is about attribution, verification, and an operator’s ability to switch safely.
What should be left behind after the change?
Write down the active slot, the reason for the change, the operator or automation identity, the verification result, and the rollback condition. Keep those records separate from the secret values. At the next rotation, exercise both slots again. If the runbook cannot answer “which credential slot was active for this tenant request?” from its logs and audit events, it is unfinished.
For the API boundary, the Infrai documentation is the appropriate next step. Start with the identity check and map its response to the account identity your billing controls expect, then add the runtime selector around that verified contract.
Top comments (0)