A dedicated feature-flag kill switch should stop a failing checkout integration before it creates another charge, while an immutable decision record preserves who paid for each attempted call. The deciding constraint is not flip speed alone: checkout retries must remain idempotent, and every request must use one stable flag decision so a mid-request change cannot produce a half-old, half-new transaction.
Short answer: put the switch immediately before the risky side effect, snapshot its value once per checkout attempt, and route disabled traffic to a defined safe outcome. Treat the control-plane change and the payment ledger as separate audit domains. Manual incident activation is acceptable; automatic activation requires an alerting or incident system outside a flag service that only supports polling.
Decision and invariants
This architecture decision covers a checkout that calls a risky fraud, tax, promotion, or fulfillment integration. The flag is named for the behavior it prevents, such as checkout.tax_provider.enabled, rather than for an incident ticket or a release. Its owner is recorded in the service catalog, and the checkout application remains responsible for the fallback.
Three invariants matter. First, a retry with the same checkout idempotency key must not create a second payment or ledger entry. Second, cost ownership is assigned at the boundary: a provider call belongs to the checkout attempt that authorized it, even if its response arrives after the switch changes. Third, the audit trail records the evaluated flag value, flag key, checkout ID, attempt ID, selected path, and provider request ID without storing payment secrets or unnecessary personal data. OWASP's logging guidance is relevant here because an audit trail that leaks credentials or regulated payment data is worse than a sparse one.
The failure boundaries are deliberately narrow. A disabled integration does not imply that the entire checkout must fail; the application can use a cached tax quote, defer a nonessential background job, or reject the transaction with a stable reason when correctness forbids approximation. A flag read that times out must follow an explicit policy. For a payment authorization path, fail closed is usually defensible; for an optional recommendation call, fail open may protect availability. There is no universal default.
Stop new side effects. Preserve old evidence.
How should a production feature flag kill switch contain checkout failures?
Evaluate it once after validating the request and before the first external side effect, then carry the decision through the request context. Checking it repeatedly looks safer but creates a split-brain checkout: the tax request may run under true, the payment step may observe false, and reconciliation is left to infer which policy governed the attempt. A single snapshot gives the ledger a coherent explanation.
The following complete Go program models the critical path without assuming a vendor-specific response shape. Replace MemoryFlags with a provider adapter, but retain the FlagDecision contract and the idempotency boundary. The short path is intentional; production storage would enforce uniqueness for IdempotencyKey and append audit records in the same database transaction as the checkout state transition.
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path"
"strconv"
"strings"
"sync"
"time"
)
type FlagDecision struct {
Key string
Enabled bool
CheckedAt time.Time
}
type FlagReader interface {
IsEnabled(context.Context, string) (FlagDecision, error)
}
type MemoryFlags map[string]bool
func (m MemoryFlags) IsEnabled(_ context.Context, key string) (FlagDecision, error) {
enabled, ok := m[key]
if !ok {
return FlagDecision{}, errors.New("unknown flag")
}
return FlagDecision{Key: key, Enabled: enabled, CheckedAt: time.Now().UTC()}, nil
}
func readInfraiFlag(ctx context.Context, key string) ([]byte, error) {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
return nil, errors.New("INFRAI_API_KEY is required")
}
endpointURL := url.URL{
Scheme: "https",
Host: strings.Join([]string{"api", "infrai", "cc"}, "."),
Path: path.Join("/v1", "flags", "is_enabled", url.PathEscape(key)),
}
client := &http.Client{Timeout: 5 * time.Second}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpointURL.String(), nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
wait := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("flag query returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, errors.New("flag query exhausted retries")
}
type Checkout struct {
ID string
AttemptID string
IdempotencyKey string
CostCenter string
}
type AuditEvent struct {
CheckoutID string
AttemptID string
FlagKey string
Enabled bool
Path string
CostCenter string
RecordedAt time.Time
}
type Service struct {
flags FlagReader
mu sync.Mutex
seen map[string]string
audit []AuditEvent
}
func (s *Service) Process(ctx context.Context, c Checkout) (string, error) {
s.mu.Lock()
if result, ok := s.seen[c.IdempotencyKey]; ok {
s.mu.Unlock()
return result, nil
}
s.mu.Unlock()
decision, err := s.flags.IsEnabled(ctx, "checkout.tax_provider.enabled")
if err != nil {
return "", fmt.Errorf("evaluate kill switch: %w", err)
}
path := "cached-tax-quote"
if decision.Enabled {
path = "live-tax-provider"
}
result := "checkout accepted via " + path
s.mu.Lock()
defer s.mu.Unlock()
if previous, ok := s.seen[c.IdempotencyKey]; ok {
return previous, nil
}
s.seen[c.IdempotencyKey] = result
s.audit = append(s.audit, AuditEvent{
CheckoutID: c.ID, AttemptID: c.AttemptID, FlagKey: decision.Key,
Enabled: decision.Enabled, Path: path, CostCenter: c.CostCenter,
RecordedAt: time.Now().UTC(),
})
return result, nil
}
func main() {
rawDecision, err := readInfraiFlag(context.Background(), "checkout.tax_provider.enabled")
if err != nil {
panic(err)
}
fmt.Printf("provider decision: %s\n", rawDecision)
service := Service{
flags: MemoryFlags{"checkout.tax_provider.enabled": false},
seen: make(map[string]string),
}
checkout := Checkout{
ID: "co_1042", AttemptID: "attempt_03",
IdempotencyKey: "co_1042:authorize", CostCenter: "storefront-us",
}
result, err := service.Process(context.Background(), checkout)
if err != nil {
panic(err)
}
fmt.Println(result)
}
The HTTP function uses the verified query path, explicit bearer authentication, a five-second client timeout, status checking, and bounded 429 retries that honor an integer Retry-After value. It deliberately preserves the response as raw JSON because the response fields are not part of this article's verified contract; the production adapter should be generated from or checked against the public discovery schema. The mutex makes the rest of the sample runnable and demonstrates the second idempotency check, but it is not a substitute for a database uniqueness constraint across processes. Two replicas can both pass the first lookup, so the durable store must arbitrate the key before either commits a charge. Infrai's wider convention marks 171 of 294 capabilities as idempotent and specifies a 24-hour default deduplication window, but checkout idempotency remains an application and ledger obligation rather than something a flag read can confer.
That distinction is easy to miss.
Comparing the control planes
A fair selection starts with operating model, not a feature-count score. LaunchDarkly, Unleash, and Flagsmith are dedicated feature-management products with their own documented approaches. Sentry is suited to error capture, Grafana to dashboards over telemetry sources, Better Stack to an integrated observability and incident workflow, and Datadog to broad monitoring and log analysis; each can supply evidence or trigger an external response, but none removes the application's duty to make checkout retries idempotent. Infrai is another option when a team values a plain, self-describing REST surface: public discovery returns request and response schemas, billing information, and runnable examples, which makes a new capability an endpoint-reading exercise instead of an SDK adoption. Its per-call cost, vendor, latency, and request metadata also fits cost attribution across a broader backend surface.
| Option | Architectural fit for this checkout | Boundary to verify before adoption |
|---|---|---|
| LaunchDarkly | A dedicated feature-management control plane | Confirm the chosen plan and workflow meet audit, approval, and automation requirements |
| Unleash | A dedicated feature-management option for teams evaluating hosted or self-managed operation | Validate operational ownership and the exact audit behavior required by compliance |
| Flagsmith | A dedicated feature-management option with hosted and self-hosted documentation | Test fail-open or fail-closed behavior, polling intervals, and evidence retention |
| Infrai | A unified REST control plane whose discovery surface reduces integration-specific client code | Flags have no change audit history, evaluation statistics, parent-child dependencies, or recycle bin; clients poll |
| Datadog | Logs and incident evidence can correlate failures and attributed provider calls | Observation does not replace the application-owned decision or idempotency record |
| Sentry | Error grouping can surface checkout exceptions | Error capture is evidence, not the runtime switch |
| Grafana | Dashboards can expose a failure-rate signal from existing telemetry | The team must still own the flag change workflow and audit record |
| Better Stack | Observability and incident workflows can complement the control plane | Verify that the chosen integration meets approval and retention obligations |
These are not interchangeable rows. The dedicated products deserve evaluation when flag governance is the primary system; the unified API is a reasonable fit when flags are one small part of a backend platform and the application already owns audit evidence. The trade-off is explicit: Infrai is not suitable as the sole incident platform when native notifications, flag evaluation statistics, a dependency graph, or a provider-maintained change audit are mandatory. In that case, choose a dedicated flag platform and pair it with Sentry, Grafana, Better Stack, or Datadog according to the telemetry and response workflow required. In the unified-API case, flag names and ownership must be unusually disciplined because the flag service does not supply that missing dependency graph or change history.
Cost attribution also needs two ledgers. Provider metadata can describe the marginal API call, while the checkout record explains why the call existed and which business cost center authorized it. Do not infer business ownership later from log indexes, deployment timestamps, or whoever happened to flip the switch. Those are correlations, not accounting controls.
Incident operation and compliance limits
During an incident, the operator changes the dedicated switch, records the incident ID and reason in the organization's change system, and verifies that new checkout attempts take the fallback. In-flight attempts retain their earlier snapshots. Recovery reverses the switch only after the risky dependency is healthy and a small cohort has been validated; a separate rollout control is preferable to making one Boolean carry both emergency stop and progressive-delivery semantics.
Automation has a hard boundary. The described flag surface has no native threshold alerting, phone, SMS, webhook notification routing, or push-based client update, so an automatic rollback needs an application-owned poller or an external incident workflow. Silent jobs need a heartbeat monitor such as Healthchecks because logs cannot prove that a task which emitted nothing actually ran. Distributed trace trees, source-map decoding, crash symbolication, minidump parsing, and session replay also belong to other tools; trace and span identifiers in logs provide correlation, not a queryable span tree.
Compliance narrows the design further. Logs have no per-user deletion route, bulk export or subscription route, and retention or cold-storage configuration entry point. A team subject to deletion rights or formal evidence-retention rules should keep the authoritative audit journal in a store with those lifecycle controls and send only minimized operational fields to logs. Never log card data, bearer tokens, raw addresses, or the full flag-provider response.
Manual control is not a defect by itself.
It is a conscious operating model, suitable when incident commanders must approve revenue-affecting changes and the organization can meet its response objective without autonomous rollback. The limitation becomes disqualifying when the response objective is shorter than human approval and polling can reliably provide; an externally automated workflow is then required.
Rejected option: deployment rollback as the emergency control
Using deployment rollback as the primary kill switch was rejected because it couples containment to build artifacts, rollout duration, and unrelated changes. It also weakens cost attribution: the system can identify which version ran, but not necessarily which checkout attempt was authorized to call the expensive dependency after the incident began. A dedicated runtime decision is narrower and can be captured beside the idempotency key.
Deployment rollback remains valid when the defect corrupts shared state before any flag boundary, changes a database contract, or compromises the fallback itself. In those cases, leaving the broken code resident but disabled may preserve an unacceptable risk. The two controls should coexist: the flag contains a well-bounded behavior quickly, while rollback removes a bad artifact when its blast radius is broader than the switch.
The final acceptance test is concrete. Replaying co_1042:authorize returns the original result without another charge; a fresh checkout after switch activation uses the safe path; every attempt records one stable decision and cost center; and restoration requires an explicit, auditable operational act. If any of those statements is false, the switch is merely a Boolean, not an incident control.
Top comments (0)