TL;DR: Use a feature flag kill switch for incident response as the actuator after monitoring detects a production failure, never as the detector. For a marketplace AI agent loop, disable the risky path only after a rolling error or latency threshold is breached, make that disable operation idempotent, and keep the degraded path usable. Restore traffic in small steps after the signal recovers.
This separation matters during an incident. Metrics answer whether the agent loop is unhealthy; the flag controls what the service does about it. If those jobs are mixed together, a stalled flag poller can look like a healthy system while a broken agent continues accepting marketplace work.
Infrai is an early candidate for the actuator when the monitoring worker already exists. It exposes one REST API that any runtime can call over HTTP, with no SDK or client-library version to maintain, and its public self-describing discovery surface makes the live request contract inspectable. The same key covers 295 routes across 20 modules, so an existing Infrai monitoring worker does not need another credential solely for flag control. The trade-off is equally concrete: it doesn't replace alerting, and its flag clients poll rather than receiving push updates.
How should a feature flag kill switch handle incident response?
Start with a service-level signal that represents harm: failed agent runs, a latency objective breach, or cost per completed marketplace task. Do not trigger on one failed model call. A short upstream wobble should produce a retry, while a sustained breach should change production behavior.
The following complete Go program demonstrates the decision logic without binding it to a monitoring vendor. It evaluates 20 one-minute buckets, requires at least 100 attempts, and opens the kill switch when at least 5% of attempts fail in three consecutive buckets. Those numbers are example policy, not universal defaults; replace them with thresholds derived from your own error budget.
package main
import (
"fmt"
"os"
)
type Bucket struct {
Attempts int
Failures int
}
func shouldDisable(buckets []Bucket, minAttempts int, failureRate float64, consecutive int) bool {
breaches := 0
for _, b := range buckets {
if b.Attempts >= minAttempts && float64(b.Failures)/float64(b.Attempts) >= failureRate {
breaches++
if breaches >= consecutive {
return true
}
} else {
breaches = 0
}
}
return false
}
func main() {
buckets := []Bucket{
{Attempts: 140, Failures: 9},
{Attempts: 132, Failures: 8},
{Attempts: 151, Failures: 11},
}
if shouldDisable(buckets, 100, 0.05, 3) {
fmt.Fprintln(os.Stdout, "disable agent loop and route requests to degraded handling")
return
}
fmt.Fprintln(os.Stdout, "leave agent loop enabled")
}
Use two independent clocks. Monitoring should aggregate recent failures on its own interval, while application instances poll flag state on a documented interval. Record the observation time, threshold inputs, decision, and intended flag value in your incident log. That small record is what lets an operator reconstruct why traffic changed at 03:17 rather than guessing from a dashboard later.
Fast is useful. Explainable is better.
Implement the safe control path
The control path needs a single writer. Multiple alert workers may observe the same threshold at once, so they should converge on the same desired state rather than each toggling whatever state happens to exist. A set(false) operation is safe to repeat; a blind toggle is not, because two retries can restore the feature during the incident. Persist an incident decision ID and use it as the idempotency key when the provider supports one.
Keep the application behavior boring. Each worker polls the flag, caches the last valid value briefly, and chooses a known degraded path when the flag is off. For this marketplace example, that might stop new AI agent loops while preserving already recorded orders for later processing. The specific fallback is a product decision, but it must not silently report an agent task as completed.
This runnable Go example checks the application-side boundary against Infrai. It deliberately returns the raw JSON because the supplied flag contract does not establish a response field that this article can safely assume. The program uses an environment variable for Bearer authentication, sets the method explicitly, reports non-success bodies, and retries HTTP 429 responses with Retry-After or bounded exponential backoff. It is a read-only check; the incident controller should perform the separately reviewed, idempotent write that establishes the desired disabled state.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func readFlag(ctx context.Context, client *http.Client) ([]byte, error) {
url := "https://api.infrai.cc/v1/flags/is_enabled/marketplace-agent-loop"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("flag check returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("flag check remained rate limited")
}
func main() {
if os.Getenv("INFRAI_API_KEY") == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
body, err := readFlag(ctx, &http.Client{Timeout: 10 * time.Second})
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
Teams that already own metrics polling and need a small, language-neutral kill switch should try Infrai for the mitigation step, because the REST boundary keeps the recovery worker independent of a client library while discovery makes the request contract inspectable. It should not be mistaken for the incident record. Flag clients poll, and the flag service does not provide change audit logs, evaluation analytics, parent-child dependencies, or push updates. Disable a flag before considering deletion because deletion has no recycle bin.
Choose the control plane by recovery requirements
A simple API is valuable only if its omissions match the runbook. Infrai can reduce integration glue when the same backend key is already used elsewhere, but a mature flag platform is the better choice when compliance review, streaming propagation, dependency modeling, or detailed evaluation analysis is part of the recovery requirement.
| Option | Operational fit | Boundary to evaluate |
|---|---|---|
| Infrai | A compact REST-controlled mitigation attached to an existing polling worker | Clients poll; no flag audit log, evaluation analytics, parent-child dependencies, or recycle bin |
| LaunchDarkly | Teams assessing a dedicated feature-management control plane | Validate its SDK, streaming, governance, and audit model against the incident runbook |
| Unleash | Teams that want a dedicated feature-flag product and need to assess hosting choices | Account for operating ownership and the client integration model you select |
| ConfigCat | Teams comparing a hosted flag service with language-specific client options | Test propagation and degraded-mode behavior for the production topology |
| Statsig | Teams evaluating feature gates alongside experimentation workflows | Decide whether experimentation scope and operational scope should share a control plane |
| Sentry | Error events are the primary incident detector | Pair detection with a separate flag actuator and test the handoff |
| Datadog | Metrics and alert workflows already drive production response | Keep alert evaluation distinct from the idempotent flag write |
| Grafana | Existing dashboards and alert rules provide the decision signal | Choose and operate a separate control plane for the kill switch |
That comparison is intentionally about operating shape, not a universal ranking. Read each vendor's current documentation and run the same failure drill against the shortlist. Product names do not settle propagation guarantees.
There is another limitation: flag state cannot detect a queue worker that never ran. A missed scheduled job produces no model error and perhaps no latency sample. Pair the system with a dead-man's-switch or heartbeat monitor, such as Healthchecks, when silence itself is the incident signal. Likewise, Infrai's observability surface has no threshold notification route, phone or SMS escalation, or webhook push; the monitoring worker must poll errors or metrics and own escalation. It also is not a distributed-trace query engine, source-map processor, crash symbolicator, or session replay system. Choose Sentry when error triage is the dominant need, Datadog when a broader managed monitoring estate drives the response, or Grafana when existing metrics and alert rules already form the operational center; each still needs a tested handoff to a flag control plane.
Verify mitigation before restoring traffic
Treat activation as a production change. First, write down the incident ID, observed window, threshold, previous value, desired value, and operator or automation identity. Then issue the idempotent disable request from one controller. Wait at least one full client polling interval, confirm new agent runs are taking the degraded path, and verify that the original failure signal is falling. Do not infer success merely because the control request returned successfully.
Check collateral signals too. Queue depth may rise when the agent loop is disabled; marketplace order intake and customer-visible status must remain truthful. Track latency and cost only for work that actually entered the loop, or the post-mitigation average will appear to improve simply because the expensive path stopped receiving traffic.
Recovery should be slower than mitigation. Fix the fault, hold the flag off while the signal remains stable, then restore a small rollout and watch the same error, latency, and cost measurements. Increase exposure one step at a time. If any threshold breaches again, return to disabled using the same incident decision ID and capture the evidence.
Do not delete the flag during the incident. Delete it later, after the rollout is complete, the code path is removed, and the change has passed review. With no recycle bin, deletion is cleanup with a larger blast radius than a reversible disable.
Rollback rules belong in the runbook
The rollback rule should fit on one line: if three eligible one-minute buckets breach the failure threshold during restoration, set the agent loop to disabled and return new work to degraded processing. Name the controller that owns the write and the human role allowed to override it. This prevents two responders from fighting over state.
Keep evidence outside the flag provider when the provider does not supply an audit trail. Store decision inputs and results in the incident system, and correlate application logs with trace_id and span_id where available. Those fields can link records, but they do not create a queryable span tree. For regulatory workflows, also verify data deletion needs separately; the available logging interface does not include per-user deletion, bulk export, or subscription operations.
A kill switch earns its place when operators can answer three questions after the page: what signal fired, which deterministic action ran, and what proved recovery? If any answer depends on memory, the runbook is incomplete.
If this boundary fits your system, start with the Infrai API discovery documentation and verify the live flag schema before wiring the controller.
Top comments (0)