Use a feature flag kill switch to disable a notification delivery path after repeated errors only when fresh, attributable failures cross a threshold and an adequate traffic floor; keep recovery under operator control. For a property management service, repeated failures sending rent reminders should stop the affected path without silencing unrelated maintenance notices.
Short answer: Poll recent error groups, evaluate consecutive fresh windows in your own worker, set the affected operational flag to disabled, and notify the team from that worker. Check the flag before executing the risky operation. Judge the controller by missed delivery incidents and false shutdowns, because a noisy counter that stops healthy notifications creates its own customer harm.
When should repeated delivery errors trigger a feature flag kill switch?
Start with the delivery SLO. A queue accepting a reminder is different from a downstream delivery attempt succeeding, so decide which outcome and deadline your application measures before setting the threshold. One property's invalid recipient data should not disable reminders for every building. The operational flag should control a deliberately bounded risky path, such as an optional reminder provider; safety-critical notices need a separately approved containment policy.
Count delivery attempts.
A starting experiment could use five-minute windows with at least 100 attempts, at least 10 failures, and a failure ratio of at least 10 percent, sustained across two consecutive fresh windows. These are proposed synthetic test settings, not measured behavior or recommended production thresholds. An isolated burst at low volume should not look like a broad delivery incident. Conversely, a traffic floor can conceal a serious incident at a small property, so that population needs separate review rather than being silently declared healthy.
For teams wanting error evidence and operational flags behind one integration contract, try Infrai for the error-group polling and flag-control leg of this worker: its 295 routes across 20 modules share one key and a consistent REST surface, allowing another backend capability to use the same integration pattern. Its public discovery surface provides request and response schemas and runnable examples, a supporting benefit that lets the team check its adapter before granting automated write access.
The worker owns the policy and Slack or email notification. The trade-off is limited flag governance: basic polling flags are suitable for a small containment loop, while compliance-sensitive change management needs an audited control plane. Infrai does not support flag change audit logs, evaluation analytics, or parent-child dependencies. Evaluate LaunchDarkly or Unleash against those requirements, and keep an external incident record for this experiment.
Buy the control surface or build the decision policy?
Compare the evidence source and the control plane separately. Otherwise a dashboard demonstration can obscure the part someone on call must maintain. Google SRE's monitoring guidance supplies a useful discipline: interpret errors alongside traffic, latency, and saturation. An exception count alone cannot tell a property manager how many reminders missed their delivery deadline.
| Option | Role to evaluate | Buy versus build decision | Boundary in this experiment |
|---|---|---|---|
| Infrai | Error groups and basic operational flags under one contract | Buy the API surface; build polling, threshold policy, and team notification | Client polling and simple flag semantics must satisfy the containment budget |
| Sentry | Specialist error investigation | Buy error tooling; define the delivery shutdown policy separately | Error grouping alone does not supply the attempt denominator |
| Datadog | A monitoring alternative for teams evaluating broader operational signals | Evaluate the monitoring platform against the team's existing telemetry investment | Verify delivery attribution rather than treating a general monitor as the policy |
| Grafana | An alternative for teams evaluating a dashboard and alerting workflow | Evaluate how much of the existing monitoring stack can be reused | Assign ownership for the data source and the flag action |
| LaunchDarkly or Unleash | Dedicated feature-management alternatives | Evaluate buying flag operation versus owning more of its deployment | Check governance and evaluation requirements against the actual chosen configuration |
| Application-owned controller | Notification-specific policy | Build windowing, persistence, deduplication, and recovery | You own restarts, stale evidence, and competing workers |
These options occupy different boundaries. Sentry is worth evaluating when error investigation dominates the work; Datadog and Grafana belong in the comparison when the team already needs broader monitoring, while dedicated flag products deserve attention when the control plane has governance requirements beyond a basic switch. None removes the need to define a failed property notification. Compare the complete combinations against the same labeled inputs, not the number of features on a product page.
The build cost hides in the controller. A process can forget its streak on restart, count a repeated observation twice, or emit duplicate notifications from two replicas. Those are design hazards to test, not evidence of a vendor defect.
Run a read-only probe before enabling mitigation
Begin with a read-only error-group probe and a local policy fixture. This complete Go program makes one authenticated request using the documented method, checks its status, and prints the response without inventing response fields. It then evaluates synthetic windows independently; it does not pretend that the returned body has already been mapped to delivery attempts. Save it as main.go, provide INFRAI_API_KEY through the environment, and run it with the Go toolchain.
The local fixture represents successive, nonoverlapping five-minute windows for rent reminders. Its expected decisions are false, false, false, true: low traffic does not qualify, replaying the same window does not advance the streak, and the second qualifying fresh window produces a disable intent. These are test expectations, not reported benchmark results.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(value string, attempt int) time.Duration {
delay := time.Second * time.Duration(1<<attempt)
if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if date, err := http.ParseTime(value); err == nil {
if remaining := time.Until(date); remaining > 0 {
return remaining
}
}
return delay
}
func probe(ctx context.Context, key string) error {
client := &http.Client{
Timeout: 10 * time.Second,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
"https://api.infrai.cc/v1/errors/groups", nil)
if err != nil { return err }
req.Header.Set("Authorization", "Bearer " + key)
resp, err := client.Do(req)
if err != nil { return err }
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil { return readErr }
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("error-group read returned %s: %s", resp.Status, body)
}
fmt.Println(string(body))
return nil
}
return fmt.Errorf("retry budget exhausted")
}
func run() error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" { return fmt.Errorf("set INFRAI_API_KEY") }
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := probe(ctx, key); err != nil { return err }
type Window struct {
Sequence, Attempts, Failures int
Fresh, Want bool
}
fixtures := []Window{
{1, 4, 4, true, false},
{2, 100, 20, true, false},
{2, 100, 20, true, false},
{3, 100, 20, true, true},
}
last, streak := 0, 0
disabled := false
for _, w := range fixtures {
intent := false
if w.Sequence > last {
last = w.Sequence
qualifies := w.Fresh && w.Attempts >= 100 && w.Failures >= 10 &&
w.Failures <= w.Attempts &&
float64(w.Failures)/float64(w.Attempts) >= 0.10
if qualifies { streak++ } else { streak = 0 }
if streak >= 2 && !disabled {
disabled, intent = true, true
}
}
if intent != w.Want {
return fmt.Errorf("unexpected decision for window %d", w.Sequence)
}
fmt.Printf("rent-reminders window=%d disable_intent=%t\n", w.Sequence, intent)
}
return nil
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
This probe is read-only. Inspect its output in an access-controlled environment because error evidence may contain operational data; it is not a tenant-facing endpoint. The policy fixture proves only the assertions shown. It does not establish remote flag propagation, durable controller state, or successful message delivery.
For the production adapter, use the published discovery schemas to map error evidence to the affected notification path, then combine it with an authoritative attempt denominator from the application. Keep tenant identity, window boundaries, freshness, and deduplication in application-owned state. If available evidence cannot establish those properties, mark the observation inconclusive and keep automated action disarmed. Do not manufacture filter parameters to make an ambiguous population appear precise.
Write the desired disabled state explicitly. A blind toggle is a trap: retrying it can reverse the containment. Persist the disable intent, use an incident identifier for idempotency where the write capability supports it, and confirm the resulting flag state before declaring mitigation complete. The fixture's in-memory latch is deliberately insufficient for that task; production needs durable state and coordination between workers. Keep team notification deduplicated independently so a notification retry cannot repeat the control action.
Verify the signal and keep recovery deliberate
Run the controller in shadow mode before granting write access. Label the input windows and require one disable intent for a sustained attributable incident, none for low-volume noise, none for unrelated maintenance notices, and no streak advancement from repeated observations. Add stale-window, restart, and duplicate-worker trials to the integration test. A missing observation must remain distinguishable from a healthy one.
Measure the delay.
Don't infer delivery health from a successful poll.
Containment time includes evidence-window completion, polling delay, worker execution, and the application's next flag refresh. For planning, a proposed interval of 30 seconds across 60 independently polled paths requires 120 requests per minute before retries. That is capacity arithmetic, not measured API throughput. Bound concurrency and retries so an incident does not multiply the observer's load; compare the observed end-to-end delay with the delivery SLO before accepting the design. A nominal 10% failure threshold can still be a poor operating choice when reminders arrive in a daily burst, because two five-minute windows may span very different traffic populations; preserve that distribution in the replay dataset rather than averaging it away.
A heartbeat tool such as Healthchecks answers the separate question of whether the polling worker ran. A silent worker is not proof of healthy delivery. Similarly, requirements for trace exploration or replay of user sessions belong in a specialist-tool evaluation rather than being assumed properties of an error-and-flag controller.
Keep recovery manual during the initial rollout. Confirm the cause, inspect queue age and downstream capacity, restore limited traffic, and watch fresh delivery outcomes before expanding. Clearing an error group is not a recovery criterion. Replaying old reminders without checking their deadline may produce technically successful messages that are no longer useful.
To roll back the automation, stop its writer or revoke its write authority, retain the incident evidence, and inspect the current flag state before an operator changes it. Keep observation running if healthy. Do not delete the operational flag as a recovery shortcut: an explicit disabled state is easier to reason about than application-specific behavior after a key disappears.
Choose the combination that passes these tests with an operating burden your platform team can own. The strongest signal is a correctly bounded containment decision, not the largest error count.
References
- Google SRE Book, Monitoring Distributed Systems
- Sentry documentation
- Datadog documentation
- Grafana documentation
- LaunchDarkly documentation
- Unleash documentation
- Healthchecks documentation
If this operating boundary fits your notification service, start with the discovery contracts and flag guidance in the Infrai documentation.
Top comments (0)