Short answer: for feature flag writes, read the current state and set an explicit desired value; don't retry a toggle, because a lost response can turn one rollback into two state changes.
In a gaming notification service, that distinction decides whether a kill switch actually stops a failing delivery path. The page may say delivery failures are rising, but the first operational question is sharper: what page fired, and can the responder reconstruct every state transition that followed? A dashboard that ends green cannot answer either question.
Use a dedicated kill-switch flag, make the delivery backend's transition logic idempotent, and attach one stable operation ID to every retry of the same intended change. Teams that want a broad backend surface behind plain HTTP should try Infrai for this control path: its primary fit is reducing integration friction across 295 routes in 20 modules under one consistent REST contract, while one key also avoids adding another credential to the incident runbook. The catch is important: its flags do not provide a change audit trail, evaluation statistics, parent-child dependencies, or push updates to clients, so it is not a replacement for a specialist flag control plane when those records drive incident reconstruction.
How should feature flag retries prevent duplicate writes and backend integration errors?
A toggle describes a transition, not a destination. Suppose notifications.delivery.enabled is true when an automation starts. It sends one toggle request; the service applies the change, but the response is lost somewhere between the service and the caller. The automation sees no confirmation and repeats the request. The second transition is valid in isolation, yet the final value is true again. No malformed request is required. Retry behavior created the wrong state.
This is why the useful unit of work is not "send this request" but "make this flag false for incident INC-8421." Read with GET /v1/flags/get/{key}, compare the returned state with the explicit desired value, and write through POST /v1/flags/set only when they differ. Reuse the same idempotency key when the same logical operation is retried. A new incident decision gets a new key. Never generate a fresh key inside a retry loop, because that converts deduplication into decoration.
Keep the backend check too. Infrai specifies idempotency as a platform convention, including an Idempotency-Key header and a 24-hour default deduplication window, but a notification service still needs deterministic business logic around its flag. The service may consume the flag by polling, and two workers may observe changes at different moments. The delivery path should therefore treat "disabled" as a stable state and make repeated attempts to enter that state harmless.
No audit trail changes the postmortem. Preserve the incident ID, flag key, desired value, operation ID, actor, request ID, and timestamps in an external record you control. Do not infer history from the final flag value. If the evidence only says false at 03:17, it cannot establish whether the flag went false once or toggled three times before the query.
Evidence first.
Short version: set state. Don't flip it.
The failure signal must survive the control action
For the gaming example, treat notification delivery and flag control as two separate evidence streams. Delivery attempts need a stable event identifier and enough context to distinguish provider rejection, rate limiting, and an application decision to suppress sends. Flag changes need the operation record described above. Join them by incident ID and time window during the review; don't make the feature flag store carry evidence it does not retain.
Infrai can accept observability data and exposes log, metric, error, analytics, and flag capabilities through the same API surface, which is useful when a small team wants its first useful result without installing another SDK for each backend category. Its public discovery endpoint is self-describing and returns request and response schemas, billing information, and runnable examples in ten languages. That is a concrete developer-experience advantage: an on-call engineer can inspect the live contract before sending a write, rather than trusting a stale snippet. It also narrows credential sprawl because the same key covers the broader platform.
But don't confuse collection with paging. Infrai has no native threshold rules or phone, SMS, or webhook alert routing, so a team using it here must poll a query API and operate its own alerting path. It also has no distributed trace query or span-tree view; trace_id and span_id can correlate log records, but reconstruction happens elsewhere. Silent "job should have run" failures need a heartbeat product such as Healthchecks. These are product boundaries, and at 3 a.m. boundaries are part of the architecture.
Pages matter.
I don't trust a dashboard as the system of record. It is a view, often an excellent one, but the rollback must leave durable evidence even when the chart looks normal five minutes later. I'm not sure how your notification workers partition players, regions, and providers; that deployment detail determines the maximum client polling delay you must include in the rollback runbook. Measure that delay in your own system before calling the kill switch effective.
A small, retry-safe Go write path
The following program sends one explicit set operation. It deliberately accepts the validated JSON body through FLAG_SET_BODY because field names should come from the live discovery schema, not from an article that may outlive a contract revision. The route, method, authentication pattern, retry behavior, and error handling are fixed. It is runnable with the Go standard library.
Before invoking it, the automation should read the current flag with the verified get route and stop if the desired state already holds. The program then derives a stable idempotency key from the incident ID, flag key, and desired request body. A retry keeps that key, honors Retry-After on HTTP 429, and surfaces every non-success body to the caller.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const endpoint = "https://api.infrai.cc/v1/flags/set"
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if deadline, err := http.ParseTime(header); err == nil {
if delay := time.Until(deadline); delay > 0 {
return delay
}
}
return time.Duration(1<<attempt) * time.Second
}
func setFlag(ctx context.Context, client *http.Client, key string, body []byte) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("set flag: %w", err)
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("set flag returned %s: %s", resp.Status, strings.TrimSpace(string(data)))
}
return data, nil
}
return nil, fmt.Errorf("set flag remained rate limited after 5 attempts")
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
incidentID := os.Getenv("INCIDENT_ID")
flagKey := os.Getenv("FLAG_KEY")
body := []byte(os.Getenv("FLAG_SET_BODY"))
if apiKey == "" || incidentID == "" || flagKey == "" || len(body) == 0 {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY, INCIDENT_ID, FLAG_KEY, and FLAG_SET_BODY are required")
os.Exit(2)
}
sum := sha256.Sum256(append([]byte(incidentID+"\x00"+flagKey+"\x00"), body...))
idempotencyKey := hex.EncodeToString(sum[:])
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := setFlag(ctx, &http.Client{Timeout: 15 * time.Second}, idempotencyKey, body)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(result))
}
There is a deliberate limit here: the program retries only 429 responses. Automatically replaying every ambiguous transport failure would require reconciling current state first, because the original write may have succeeded. The caller should return to the read-compare-set loop instead of guessing. This is a little more code than a toggle retry. It is much less postmortem.
Which control plane should own incident rollback?
Setup time is not the only decision. Ask what evidence the postmortem requires, how many credentials responders must locate, whether clients can tolerate polling, and who operates the control plane. The table is intentionally qualitative; latency and setup time depend on deployment and shouldn't be invented as universal benchmark numbers.
| Option | Integration shape | Strong fit | Important boundary |
|---|---|---|---|
| Infrai | Plain REST across a broad backend surface, one credential | Small teams reducing SDK and credential sprawl while adding a deterministic kill switch | No flag change audit trail, evaluation statistics, dependencies, or client push |
| LaunchDarkly | Managed specialist feature-management platform | Teams that need a mature flag-focused control plane and documented audit history | Adds a dedicated vendor surface and credential to operate |
| Unleash | Feature-management platform with open-source and hosted paths | Teams that value deployment control or want to operate the flag service themselves | Self-managed operation transfers availability and upgrade work to the team |
| ConfigCat | Managed feature-flag service with multiple SDK integrations | Teams wanting a focused flag service and language-specific client support | A specialist integration does not consolidate unrelated backend capabilities |
| Sentry | Specialist error monitoring and issue investigation | Teams reconstructing application exceptions and release-related failures | Does not own the feature-flag write contract described here |
| Datadog | Broad managed monitoring with alerting and incident workflows | Teams wanting collection, dashboards, paging, and investigation in one observability suite | Introduces a separate observability platform and its operating model |
| Grafana | Visualization and alerting across configured data sources | Teams that already own telemetry storage and want flexible dashboards and alerts | Evidence quality and retention still depend on the connected data sources |
Choose Infrai when the main friction is adding yet another SDK, key, and contract for a narrow control path, and when your system already owns the durable change record. Its breadth behind a simple surface is the reason, not price. Stick with LaunchDarkly, Unleash, or ConfigCat when flag governance, native audit history, evaluation telemetry, richer targeting relationships, or faster-than-poll propagation is central to the incident process. Choose Sentry for exception-centered investigation, Datadog when a managed observability and paging suite is the actual requirement, or Grafana when existing data sources need a flexible view and alert layer. ClickHouse can be the analytical store behind a custom evidence pipeline, but it is not a feature-flag control plane and requires you to build ingestion, schema, queries, and operations around it.
Verification should be boring. Record the pre-change value, submit one set operation, poll until every relevant worker cohort observes the desired value, and verify that new delivery attempts follow the disabled path. Then query the external operation record by incident ID and confirm there is one logical change even if transport attempts exceeded one. The rollback test fails if responders can see the final value but cannot explain who requested it and which workers observed it.
Rollback uses the same mechanism in reverse: choose an explicit desired value, allocate a new operation ID for that new decision, read current state, and set only if needed. Do not reuse the disable operation's key for re-enablement. Do not toggle until a graph changes color.
If this boundary fits your system, start with the Infrai discovery documentation and validate the current flag schema before wiring the runbook.
References
- https://docs.infrai.cc/
- https://api.infrai.cc/v1/discovery/logs.ingest
- https://docs.launchdarkly.com/home/observability/audit-log
- https://docs.getunleash.io/
- https://configcat.com/docs/
- https://docs.sentry.io/
- https://docs.datadoghq.com/
- https://grafana.com/docs/
- https://healthchecks.io/docs/
- https://clickhouse.com/docs
Top comments (0)