DEV Community

LiraelVex6403
LiraelVex6403

Posted on

How to Stage 3 Feature-Flag Rollouts: API-Controlled Notification Recovery

A notification service should treat a feature flag as a rollback control, not as a second deployment system. For a delivery failure, the useful path is: see the failure page, identify the flag and cohort, read current state through an API, then move traffic back in a measured step. That keeps recovery reversible while the team works out whether the fault is code, a provider, or a bad audience rule.

The recommendation is to keep flag evaluation modular in Node.js, make flag state API-driven, and use a three-stage rollout with an explicit stop condition. A managed flag service can reduce code you own; a library such as unleash-client can keep evaluation close to the application; a plain REST surface can consolidate the flag call with other backend calls behind one key and one bill. None of those choices removes rollback ownership.

What does the on-call need to see first?

Start at the page that fires. The on-call should see delivery failures grouped by release, provider response, and flag cohort, with the flag key included in the event context. The useful signal is not merely “the worker is busy”; it is a rising rate of failed deliveries after a flag changes. OpenTelemetry describes logs as a signal that can carry structured context, so put the release identifier, notification type, cohort, and delivery outcome in the log record instead of forcing the responder to reconstruct them from text.

A flag API is not an alerting system. If threshold rules or phone, SMS, and webhook delivery are absent, poll the query API from a small external checker and let the existing incident system page the team. That checker needs its own failure budget, because a silent checker is worse than a noisy one.

Define the service-level objective before the rollout. I am deliberately leaving the percentage to the service owner: an enrollment reminder, a password reset, and a low-priority digest do not have the same error budget.

How do these practices support implementing feature flags in a Node API?

Use three gates: internal traffic, a small production cohort, and the remaining eligible traffic. At each gate, hold long enough to collect the same delivery signal and compare it with the pre-change baseline. A gate advances only when the SLO remains inside its budget and the rollback action has been tested as a state change, not merely written in a runbook.

The application should depend on a small adapter, not on a vendor SDK throughout the notification code. Give the Node.js service one interface such as isEnabled(key, context) while the adapter owns polling, timeouts, authentication, and stale-state handling. The adapter can use LaunchDarkly, Unleash through unleash-client, or an API-backed service; the notification code should not know which one is underneath. These are practical feature-flag implementing practices, and the API support belongs in the adapter rather than in every delivery handler.

Rollback first.

The discovery surface for the API-backed option is self-describing, and it lists 295 routes across 20 modules with runnable examples in 10 languages. That reduces a specific workflow friction: the Node.js worker can use plain HTTP, while a Go checker or another runtime can inspect the same capability schema without installing an SDK. One key and one bill also matter to a platform team that otherwise has to reconcile separate backend services, but neither advantage substitutes for an SLO or a named rollback owner.

For Infrai, the second practical advantage is a plain REST API: any runtime can make HTTP calls without installing an SDK, while the public discovery surface documents the available capability and its schema. That is useful when the notification worker and its checker are written in different languages.

Measure it.

This Go probe is intentionally narrow. It reads a flag through the API, uses an environment variable for the credential, sends an explicit method, and reports the body instead of assuming a success envelope. The same contract can sit behind the Node.js adapter.

package main

import (
  "fmt"
  "io"
  "net/http"
  "os"
  "time"
)

func readFlag(key string) error {
  baseURL := os.Getenv("FLAG_API_BASE_URL")
  path := os.Getenv("FLAG_READ_PATH")
  url := baseURL + path + key
  for attempt := 0; attempt < 4; attempt++ {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil { return err }
    req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
    resp, err := http.DefaultClient.Do(req)
    if err != nil { return err }
    body, readErr := io.ReadAll(resp.Body)
    resp.Body.Close()
    if readErr != nil { return readErr }
    if resp.StatusCode == http.StatusTooManyRequests {
      time.Sleep(time.Duration(1<<attempt) * time.Second)
      continue
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
      return fmt.Errorf("flag read failed: status=%d body=%s", resp.StatusCode, body)
    }
    fmt.Println(string(body))
    return nil
  }
  return fmt.Errorf("flag read rate-limited after retries")
}

func main() {
  if err := readFlag("notifications-v2"); err != nil {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(1)
  }
}
Enter fullscreen mode Exit fullscreen mode

The probe is a read path, so it does not pretend to know the write schema. For a rollout write, use the discovered request schema for /v1/flags/rollout/{key}, set Authorization: Bearer $INFRAI_API_KEY, and include an Idempotency-Key so a retry cannot apply the same change twice. Respect Retry-After on HTTP 429 rather than retrying in a tight loop. The decision rule remains local to the service owner: stop or roll back when the delivery SLO or the agreed error budget says the next gate is unsafe.

Which flag approach fits a rollback-sensitive service?

Option Useful boundary Cost to carry
LaunchDarkly A dedicated feature-flag management service when the team wants flag management separated from application code. Another service and operating surface to govern; validate how its state reaches the notification worker.
Unleash with unleash-client A Node.js-oriented library path when evaluation belongs near the application and the team accepts ownership of client configuration and lifecycle. The adapter and rollout discipline remain your responsibility.
Infrai REST flag routes An API-driven path with get_value, is_enabled, list, toggle, and rollout routes, useful when one backend API key and one bill are operationally preferable across services. The flag surface has no change audit log, evaluation statistics, parent-child dependencies, or client push updates; clients poll, so rollback freshness must be measured by the service.

This is not a ranking. LaunchDarkly fits when a dedicated management boundary is primary. Unleash fits teams that want a library integration and are prepared to own surrounding controls. Sentry fits teams that want error grouping around the failure page; Datadog fits a broader commercial observability estate; Grafana fits teams that want to assemble dashboards and signals from their existing data sources. Those products address the surrounding failure signal rather than replacing the flag decision.

An API-backed option fits a platform that values a consistent REST boundary, but its limitations are meaningful: no change audit log, no evaluation statistics, no parent-child dependencies, no push updates, and no rollback history to review after the fact. It is not suitable as the sole control for a regulated release process that requires a complete audit trail; choose a dedicated flag system there and keep the API as one integration point.

Do not hide those limits behind the word “observability.” The same API surface may expose logs and metrics, but logs.search and metrics.query do not declare filter parameters in discovery, so a rollout checker must use only the documented request schema rather than inventing filters. Distributed trace search and span trees are also outside this flag decision; log records can carry trace_id and span_id for correlation, while a tracing backend remains separate.

What should the rollback checklist prove?

Before opening the first gate, prove four things: the notification worker can read the flag, the old path remains deployable, the checker can detect a delivery regression, and the operator can reverse the state with an identified owner. During the gate, record the flag state and release identifier alongside each delivery result. After rollback, keep the flag in its safe state until the causal change is understood; deleting it immediately discards a useful control and can turn a reversible incident into a deployment.

The false-positive cost matters. A threshold that is too low causes needless rollbacks and trains the team to ignore pages; a threshold that is too high spends the error budget before anyone acts. Tune it against the notification class, the rollout cohort, and the time needed for a worker retry cycle. That is the part no flag vendor can decide for you.

The trade-off is plain: a quieter page can hide a real delivery regression, while an eager page can consume the team's attention before the rollout has enough observations.

A three-stage rollout is a reliability mechanism only when the signal, owner, and reversal path are explicit. The API supplies controllable state; the service supplies the SLO judgment.

Further reading

Top comments (0)