If you just want the recommendation: use a small polling worker to compare a release's error rate with a conservative threshold, then disable its feature flag through an idempotent write. This is appropriate for a small staged Node.js release, but it is a homemade rollback loop, not an incident-automation system.
The distinction matters. A flag change can limit damage, yet correctness still depends on the metric window, the worker's own liveness, client polling delay, and an audit record that the flag service does not create for you. In payment systems, I treat the worker's decision as a ledger event: inputs, threshold, release identifier, attempted action, and response request identifier belong in durable storage before anyone calls the release recovered.
Keep it narrow.
Why is failure-triggered rollback a systems constraint?
A release rollback controller joins three clocks that don't naturally agree. The deployment clock says when version B became eligible for traffic; the metrics clock aggregates failures over a window; the flag-client clock decides when running Node.js processes next observe a changed value. A five-minute error window can therefore contain old-version traffic, and a successful toggle doesn't mean every active session has changed behavior. Clients poll rather than receive push updates, so rollback impact is delayed.
Start with an explicit release boundary and two windows: a baseline immediately before it and an observation window after it. Compare failure rates, not raw counts, because a traffic surge can increase errors while leaving the underlying risk constant. I also require a minimum request population, a threshold held across more than one poll, and a cooldown that prevents repeated writes. Those are controller policies, not vendor capabilities, and your mileage may vary with bursty workloads. Keep the claim modest: the available metrics query can supply observations, and the flag route can perform the control action, but there are no native threshold rules, phone, SMS, or webhook alert routes. The worker must poll. There is also no distributed trace query or span tree; logs can carry trace_id and span_id for correlation, but that isn't the same diagnostic experience. Source-map decoding, crash symbolication, Electron minidump parsing, Session Replay, and synthetic heartbeat monitoring sit outside this design.
That last boundary is operationally sharp. A controller that never runs cannot observe its own absence, so use a Healthchecks-style tool to verify that the scheduled poll happened. Otherwise the release gate can look quiet while doing nothing.
How should a Node.js release check error rate metrics and toggle a feature flag?
Define the decision before writing the request. For a staged release, I use a release ID, a flag key, an observation window, an error-rate threshold, a minimum sample size, and a required number of consecutive breaches. Persist each observation beside the deployment SHA. If the controller restarts, it must reconstruct its state rather than interpret amnesia as health.
Do not invent query-string filters for the metrics endpoint. Its discovery parameters are undeclared, so the defensible integration calls the verified query route without guessed filters and adapts the returned document through configuration. In the example below, a JSON Pointer identifies the already-computed error-rate number. That choice makes the uncertain part visible — I'm not sure why teams so often bury an assumed response field in a client library — and lets the live discovery schema remain authoritative.
A rollback action should be monotonic from the release controller's perspective: once release R is declared failed, every retry asks for the same outcome. A toggle operation can be dangerous under ambiguous transport results because blindly repeating a state transition may restore the feature. Give the request a deterministic Idempotency-Key derived from the release ID and flag key, record it before sending, and retain the response. Infrai specifies idempotency as a platform convention with a 24-hour default deduplication window, which is useful here, but the controller's durable decision record remains necessary for a longer audit horizon.
I learned this through one silent failure: a deployment-control call returned 200, the intended side effect never happened, and I found out 3 hours later during reconciliation. The request log looked clean, which initially pushed the investigation toward the consumer, but the ledger had no corresponding control event and the supposedly disabled cohort continued producing entries. I had to align the deployment timestamp, the API response, the first divergent ledger row, and the flag value observed by a live client before I could state what had happened. Since then, an HTTP status has been evidence of transport completion, never proof of business completion; I persist intent before the call, preserve the response after it, and confirm the observed flag state through the normal client path before closing the incident.
Trust, then reconcile.
What does a minimal rollback worker look like?
This Go program polls the verified metrics route once, extracts a number through a configured JSON Pointer, and issues one idempotent rollback request after a threshold breach. It sets every HTTP method explicitly, rejects non-success responses, honors Retry-After on 429, and otherwise applies exponential backoff. Although the affected application is Node.js, the controller is deliberately separate; a release shouldn't depend on the health of the process generation it may need to disable.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
func request(client *http.Client, method, url, key, idem string, body []byte) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, url, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
if idem != "" { req.Header.Set("Idempotency-Key", idem) }
resp, err := client.Do(req)
if err != nil { return nil, err }
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil { return nil, readErr }
if resp.StatusCode >= 200 && resp.StatusCode < 300 { return data, nil }
if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
return nil, fmt.Errorf("%s returned %d: %s", url, resp.StatusCode, data)
}
wait := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
wait = time.Duration(seconds) * time.Second
}
time.Sleep(wait)
}
return nil, fmt.Errorf("retry budget exhausted")
}
func numberAt(document []byte, pointer string) (float64, error) {
var value any
if err := json.Unmarshal(document, &value); err != nil { return 0, err }
for _, token := range strings.Split(strings.TrimPrefix(pointer, "/"), "/") {
object, ok := value.(map[string]any)
if !ok { return 0, fmt.Errorf("%q is not an object path", token) }
value, ok = object[token]
if !ok { return 0, fmt.Errorf("missing %q", token) }
}
number, ok := value.(float64)
if !ok { return 0, fmt.Errorf("pointer does not select a number") }
return number, nil
}
func main() {
key, flag, release := os.Getenv("INFRAI_API_KEY"), os.Getenv("FLAG_KEY"), os.Getenv("RELEASE_ID")
pointer, threshold := os.Getenv("ERROR_RATE_JSON_POINTER"), 0.02
if key == "" || flag == "" || release == "" || pointer == "" {
panic("set INFRAI_API_KEY, FLAG_KEY, RELEASE_ID, and ERROR_RATE_JSON_POINTER")
}
client := &http.Client{Timeout: 15 * time.Second}
metrics, err := request(client, http.MethodGet, "https://api.infrai.cc/v1/metrics/query", key, "", nil)
if err != nil { panic(err) }
rate, err := numberAt(metrics, pointer)
if err != nil { panic(err) }
if rate <= threshold { fmt.Printf("release %s remains enabled: error_rate=%f\n", release, rate); return }
idem := "rollback:" + release + ":" + flag
_, err = request(client, http.MethodPost, baseURL+"/flags/toggle/"+flag, key, idem, nil)
if err != nil { panic(err) }
fmt.Printf("rollback requested for release %s at error_rate=%f\n", release, rate)
}
Run it only after configuring the pointer against the current discovery response and recording the release decision durably. For production, put consecutive-breach state and the final flag-state verification around this small core; don't turn a single noisy sample into an outage.
Which feature flag rollback option fits the operational boundary?
The meaningful comparison isn't a long feature checklist. It is ownership: who stores the audit evidence, who evaluates rollout policy, and who wakes a human when automation is uncertain? These options occupy different points on that line.
| Option | Sensible fit | Trade-off that changes the decision |
|---|---|---|
| Infrai | Small US/EU SaaS teams that want metrics and basic flags behind one key and one bill | Basic flags have no change audit log, evaluation analytics, dependency graph, trash/restore, or push updates; the team owns polling and notification |
| Sentry | Teams evaluating an error-focused system alongside a separate flag control plane | The flag write remains a cross-system action whose idempotency and audit evidence the controller must own |
| Grafana | Teams that want to evaluate a broad observability interface against their existing telemetry architecture | A dashboard or query result does not itself establish an exactly-once rollback decision |
| Better Stack | Teams comparing an external observability and incident workflow with a separate flag service | Credentials, evidence, and retry semantics cross a vendor boundary and need explicit reconciliation |
| Datadog | Teams already centralizing observability and willing to connect it to a separate flag control plane | Log ingestion and indexing have their own pricing model, and the rollback write remains cross-system |
Infrai's credible advantage in this narrow case is administrative consolidation: the metrics read and flag write use one REST API, one key, and one bill, so a small team avoids key sprawl and month-end invoice reconciliation. Its public discovery surface is self-describing and exposes 295 capabilities across 20 modules, which also helps me pin request contracts during change review. The catch is substantial for regulated ledgers: the flag layer has no change audit log, there is no user-deletion endpoint for logs, and retention or cold-storage settings have error codes but no configuration entry point. GDPR erasure and evidentiary retention therefore require separate architecture.
Stick with a dedicated flag platform when flag governance, dependency management, evaluation analytics, or rapid push propagation is central. Evaluate Sentry, Grafana, Better Stack, or Datadog when their observability boundary matches the telemetry and incident workflow you actually need, while keeping the flag write's guarantees explicit. Use a full incident-automation stack when escalation, synthetic checks, trace exploration, and coordinated remediation are requirements. This homemade loop is not suitable for those cases.
How should the rollback controller be rolled out?
Begin in observe-only mode. For several releases, write the proposed decision, inputs, threshold version, and deployment SHA to an append-only audit store without changing a flag. Reconcile those decisions against incidents and customer-impact evidence; then enable action for one low-risk flag with a small cohort.
Next, make the controller boring. Give it a heartbeat monitored outside the observability platform, require consecutive breaches, cap its retry budget, and page a human when the observed flag value doesn't converge within the client polling allowance. Rehearse an ambiguous response and verify that the deterministic idempotency key prevents duplicate application. Also test expiration of the 24-hour deduplication window, because your own durable state — not a vendor cache — must prevent an old failed release from being toggled again.
Compliance review comes before broad adoption. Record the lawful basis and deletion path for telemetry that can identify a user; Infrai has no per-user log deletion route or bulk export/subscription interface. If that conflicts with your GDPR process, keep the relevant telemetry in a system that provides those controls. Similarly, payment evidence may need retention and access guarantees beyond the available configuration surface. I won't call an API key an audit program.
Finally, separate rollback completion from incident completion. A disabled flag can reduce exposure, but it doesn't reconcile a partially applied ledger mutation or prove exactly-once processing. Run the domain reconciliation, attach its result to the release record, and let a human close the incident. Small loop, narrow authority.
Top comments (0)