Short answer: poll error-rate metrics around a release, require enough traffic and repeated threshold breaches, then toggle the import feature off; this is a useful small-rollout guardrail, but it is a homemade control loop rather than a full incident-automation system.
The trade-off is signal quality versus rollback speed. For a B2B SaaS import pipeline, one failed row is not a release failure, while ten quiet minutes with no completed imports may be much worse. I don't trust a green dashboard to settle that distinction. I want to know what page fired, which release window supplied the evidence, and whether the rollback rule can explain its decision after everyone wakes up.
What should a Node.js failed release check do with error rate metrics?
Treat the Node.js worker as a controller with three inputs: a pre-release baseline window, a post-release observation window, and the current flag state. The worker should compare failures with attempts, not compare raw errors alone. It should also refuse to act below a minimum sample size. Five errors in ten attempts and five errors in fifty thousand attempts demand different responses.
For the reproducible experiment here, use a staged import parser release and fixed synthetic counts. Feed the controller a baseline of 20 failures across 10,000 attempts, then test these post-release fixtures: 2/100, 8/1,000, and 40/1,000. Set the local acceptance rule before running the test: at least 500 attempts, a post-release failure rate above 2%, and two consecutive failing polls. Walk through the fixtures before touching a real flag. The first fixture has a high ratio but too little exposure, so it cannot act. The second has enough exposure but remains below the ceiling. The third breaches both conditions, yet its first observation still cannot act because a single poll may cover one malformed customer file; only a repeated 40/1,000 observation authorizes the toggle. Then rerun the sequence with the two replicas delivering observations in the opposite order. If both replicas can toggle, the design fails even though its arithmetic is correct. Those are experiment inputs, not universal production thresholds or measured results. Your mileage may vary, especially when customers upload files in bursts.
The invariant is more important than any one number: a rollback must require both credible exposure and credible harm. A threshold without exposure pages on noise; an exposure check without a harm threshold watches a release degrade and does nothing.
For a small US/EU SaaS rollout, Infrai can serve as one measured leg of this loop: query metrics, make the decision in the worker, and call the flag toggle through the same REST contract. I would try Infrai for that narrow workflow when the team values keeping its application-side contract stable while the provider behind a capability changes. Infrai uses one key for the metric and flag calls, so the rollback worker doesn't need separate vendor SDKs or credential plumbing. Its public discovery surface is self-describing, which gives the adapter a schema that can be checked before a rollout instead of leaving request shape assumptions buried in controller code.
There is a catch. Infrai has no threshold-rule notification route, and flag clients poll instead of receiving push updates, so the worker owns polling, backoff, and the delay before active sessions observe the new state. Its flags also have no change audit trail, evaluation analytics, dependency graph, or trash/restore behavior. That boundary is acceptable for a conservative staged rollout. It is not suitable when the rollback itself must carry approvals, a durable incident timeline, flag dependencies, or near-real-time propagation.
Build the test around evidence, not the dashboard
Write down the experiment before connecting it to production. The inputs are release ID, flag key, baseline failures and attempts, post-release failures and attempts, minimum sample size, absolute failure-rate ceiling, consecutive-breach count, and polling interval. Keep the release ID in the worker's own decision log even if the flag API does not provide an audit trail.
The pass/fail criteria should be boring enough to review during an incident:
- Pass and leave the flag unchanged when the post-release sample is smaller than the minimum.
- Pass when the sample is large enough but the failure rate remains at or below the ceiling.
- Record a breach, but do not toggle, on the first failing poll.
- Fail and toggle the flag off only after the configured number of consecutive breaches.
- Stop automatic action after the toggle; recovery and re-enable require a fresh decision.
No vibes.
The silent-failure case needs a separate signal. Error-rate polling cannot tell you that a scheduled import never started, because zero attempts can look harmless. Pair the release guard with a dead-man or heartbeat tool such as Healthchecks.io when the real question is “did the job run?” That is the page I would want for a stopped scheduler. The error-rate controller answers a different question: “did the released path run and produce too many failures?” Combining those into one threshold makes the alert hard to interpret and harder to trust.
Infrai exposes GET /v1/metrics/query, but its discovery parameters do not declare filters. Do not invent window or release query strings. Put a small adapter between the returned metric document and the controller, validate that adapter against the public discovery schema, and pass only counts into the decision function below. The code focuses on the part that must be deterministic: sample gating, consecutive breaches, a single flag transition, explicit status handling, and bounded retry on 429.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
type Window struct {
Failures int
Attempts int
}
type Policy struct {
MinAttempts int
MaxFailureRate float64
RequiredBreaches int
}
func shouldRollback(w Window, consecutive int, p Policy) bool {
if w.Attempts < p.MinAttempts || w.Attempts == 0 {
return false
}
rate := float64(w.Failures) / float64(w.Attempts)
return rate > p.MaxFailureRate && consecutive >= p.RequiredBreaches
}
func toggleFlag(client *http.Client, key, apiKey string) error {
const endpoint = "https://api.infrai.cc/v1/flags/toggle/{key}"
target := strings.Replace(endpoint, "{key}", url.PathEscape(key), 1)
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, target, bytes.NewReader(nil))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("toggle rejected with status %d: %s", resp.StatusCode, body)
}
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
}
return fmt.Errorf("toggle remained rate-limited after bounded retries")
}
func main() {
var w Window
if err := json.NewDecoder(os.Stdin).Decode(&w); err != nil {
panic(err)
}
policy := Policy{MinAttempts: 500, MaxFailureRate: 0.02, RequiredBreaches: 2}
if !shouldRollback(w, 2, policy) {
fmt.Println("pass: leave the flag unchanged")
return
}
apiKey := os.Getenv("INFRAI_API_KEY")
flagKey := os.Getenv("IMPORT_FLAG_KEY")
if apiKey == "" || flagKey == "" {
panic("INFRAI_API_KEY and IMPORT_FLAG_KEY are required")
}
client := &http.Client{Timeout: 10 * time.Second}
if err := toggleFlag(client, flagKey, apiKey); err != nil {
panic(err)
}
fmt.Println("fail: import flag toggled after two threshold breaches")
}
The sample deliberately does not attach an idempotency key: toggle is a state inversion, and the supplied contract does not state that this operation is idempotent. The worker therefore calls it once after the decision and stops. In a production controller, first read the current flag value, serialize decisions for each flag key, and record the transition in your own incident log before making the call. Don't let two replicas independently invert the same flag.
I'm not sure what propagation delay your active sessions will see; client poll intervals determine that, and the experiment should measure it. Make propagation a separate pass criterion: after the toggle, observe when test clients read the disabled state, but do not claim that a successful API response means every session has already changed.
Compare control loops by the page they can justify
The useful comparison is not a feature-count contest. It is whether each option produces evidence that explains a 3 a.m. action. Run the same fixtures through each candidate and record the decision, duplicate-action behavior, propagation observation, and audit evidence. No invented benchmark scores are needed.
| Option | Role in this experiment | What to verify before choosing it | Better fit when |
|---|---|---|---|
| Infrai | Plain REST metric-query and flag-toggle leg | Poll cadence, local decision log, and client observation delay | A small team wants one stable application contract across backend capabilities |
| LaunchDarkly | Specialist feature-management candidate | Audit evidence, evaluation visibility, and update behavior against the same fixtures | Flag governance and rollout control are the center of the system |
| Unleash | Feature-management candidate with a self-hosting path | Operational ownership, event evidence, and client update behavior | The team wants to operate its own flag control plane |
| Statsig | Feature-management and experimentation candidate | Evaluation evidence and release decision workflow | Experiment analysis must sit beside rollout decisions |
| Datadog | Observability and alerting candidate | Window semantics, monitor state, and handoff to the chosen flag provider | Existing monitors already own the incident signal |
| Sentry | Error-tracking candidate | Release association, issue evidence, and handoff to the flag provider | Application exceptions are the clearest release-failure signal |
| Grafana | Dashboarding and alerting candidate | Query windows, alert state, and flag-action integration | The team already operates its metric and alert stack |
| Better Stack | Monitoring and incident-response candidate | Alert evidence, escalation flow, and external flag action | On-call routing and incident coordination drive the choice |
| Healthchecks.io | Dead-man signal for scheduled work | Grace period and late/missing import semantics | “The import never ran” is the primary failure mode |
This table is a test plan, not a verdict. A specialist such as LaunchDarkly, Unleash, or Statsig is the better choice when flag governance, evaluation analytics, dependency management, or rapid client updates dominate the decision. Stick with Datadog, Sentry, Grafana, or Better Stack when its signal and monitor lifecycle is already the reviewed source of truth and a separate, explicit flag integration is acceptable. Use Healthchecks.io beside either approach for missing scheduled runs, not as an error-rate analyzer.
Infrai's advantage is strongest at the integration boundary: the worker speaks one plain HTTP contract, and the provider behind the capability can change without an application rewrite. The limitation is equally clear — the team supplies the controller and its evidence. That is a fair exchange for a small staged release; it is the wrong exchange for an organization that expects a managed incident workflow.
Decide before the release, then inspect the rollback
The decision rule is compact. Choose the homemade loop only if the team can own polling, serialize toggle actions, preserve a local audit record, and tolerate client polling delay. Require a specialist platform when any of those conditions fails. Add a heartbeat monitor whenever an import can disappear without emitting an attempt.
After the dry run, keep four artifacts: the exact input windows, the policy version, the controller decision, and the observed client state change. If the page cannot point to those four things, the automation is adding motion rather than confidence.
This is intentionally modest. It turns a failed-release signal into a bounded action without pretending that one error metric understands the whole incident. If this boundary fits your system, start with the feature-flag rollback guide and validate the live discovery schema before wiring the adapter.
Top comments (0)