IronspireDraven77

Posted on Sep 10

Node.js Notification Rollbacks — Feature Flag Fetch Timeout Diagnosis

#observability #node #sre

Short answer: treat a feature flag fetch timeout in a Node.js edge function as a rollback-safety problem, not merely a networking problem: give every fetch a deadline with AbortController, retain a last-known-good flag snapshot, and record enough evidence to distinguish an upstream delay from a poller, runtime, or rollout failure.

The important trade-off is freshness versus a predictable failure mode. A notification service that blocks delivery while it waits for a control-plane answer can turn a cautious feature rollout into a delivery incident. A service that silently invents a default can be worse, because the page may say “provider errors” while the actual trigger was a flag client that discarded its last usable state. At 3 a.m., the dashboard is not the evidence. The first questions are: what page fired, which flag revision did the affected request evaluate, and could the new release be rolled back without changing that revision?

How should Node.js edge functions troubleshoot feature flag fetch timeouts?

Start at the boundary where the decision affects notification delivery. Give the polling request its own deadline, abort it when that budget expires, and classify the result separately from HTTP responses. fetch() rejects when its signal is aborted; an HTTP error response, by contrast, still produces a response and should be classified by status. Mixing those paths into one generic “flag unavailable” counter destroys the distinction the incident responder needs.

For each poll attempt, capture a small, stable event: a hashed or allow-listed flag identifier, deployment revision, edge region, attempt start time, configured deadline, elapsed duration, outcome, response status when one exists, and the revision or age of the snapshot ultimately used. Don't put flag payloads, targeting attributes, email addresses, or tenant secrets in logs. The diagnostic join is between the poll attempt, the evaluated snapshot, the notification attempt, and the deployment — not between a person and every attribute the flag system happened to inspect.

The polling loop also needs one owner. If every incoming edge request starts a background poll, concurrency can multiply during traffic spikes and the runtime may terminate work after the response has returned. Prefer the platform's supported scheduled or lifecycle mechanism, or fetch within the request only when the deadline fits inside the request budget. I'm not sure any generic interval recommendation survives across edge runtimes; execution lifetime, timer behavior, and background-work guarantees are platform contracts, so verify those before choosing a polling cadence.

This is the evidence sequence I want to see for one affected delivery:

The deployment revision and request trace identify the code that evaluated the flag.
The evaluation event names the exact cached flag revision and its age.
The preceding poll event says success, deadline_exceeded, network_error, or http_error; those are separate outcomes.
The notification event records the resulting route, such as primary provider, alternate provider, or suppressed send, without copying message content.
The alert points to a customer-facing delivery objective, while the poller signal supplies diagnosis.

One page. Several clues.

The failure is usually outside the timeout line

AbortController can stop waiting, but it cannot choose a safe business result. That choice belongs in a policy tied to each flag. For a flag that selects a new notification provider, the safest timeout behavior may be to continue using the last-known-good snapshot. For an emergency kill switch, stale state may be unacceptable; a deliberately conservative local value can be safer. Document that policy next to the flag definition and test it before rollout, because a universal “false on error” rule can reverse the intended safety property.

The postmortem frame helps. If the feature release was rolled back but delivery failures continued, code rollback and flag rollback were probably independent control planes. A process restart may have erased an in-memory snapshot. A new deployment may have changed the cache key or environment name. Two regions may have evaluated different revisions. None of those hypotheses is proven by a latency chart; the snapshot revision and deployment correlation either support them or they don't.

Abort early.

There is another sharp edge: deadline budgets nest. Suppose an edge request has time remaining for authentication, flag evaluation, provider selection, and the outbound delivery call. The flag fetch cannot consume the entire request budget and still leave a viable delivery path. Set its deadline from the remaining budget, reserve time for the action that actually serves the customer, and make the fallback decision immediately after the abort. A timer that fires after the outer request has already been canceled provides clean-looking telemetry and no operational protection. Now carry that reasoning through a regional rollout: the old deployment may read snapshot revision A, the new deployment may read revision B, and a rollback can restore the old code without restoring A. The evaluation event therefore needs both revisions on the same record. If delivery recovers only after the flag changes back, the flag revision was part of the recovery; if it recovers as the deployment converges while B remains active, the code revision is the stronger lead. This is a bounded diagnostic comparison, not proof of causation, but it gives the responder a falsifiable next question instead of another dashboard to stare at.

Avoid retrying blindly inside that same request. A second attempt can be reasonable when the remaining budget permits it and the failure is plausibly transient, but synchronized retries from many edge instances create more control-plane load exactly when it is responding slowly. Add bounded jitter to scheduled polls, cap retry attempts, and keep the last-known-good snapshot outside the retry loop. Your mileage may vary on exact timings; the invariant is measurable budget, bounded work, and an explicit stale-state policy.

Build the preventative path around state, not exceptions

The following Go example is a compact model for a synthetic poll probe and cache updater. It uses a context deadline, validates the response before publishing it, and preserves the previous snapshot on failure. The same state transitions should surround Node.js fetch and its abort signal; the point is the rollback-safe commit boundary, not the client syntax.

package flags

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "sync"
    "time"
)

type Snapshot struct {
    Revision string          `json:"revision"`
    Flags    map[string]bool `json:"flags"`
    LoadedAt time.Time       `json:"-"`
}

type Store struct {
    mu      sync.RWMutex
    current Snapshot
}

func (s *Store) Refresh(parent context.Context, client *http.Client, endpoint string, budget time.Duration) error {
    ctx, cancel := context.WithTimeout(parent, budget)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
    if err != nil {
        return fmt.Errorf("build flag request: %w", err)
    }

    started := time.Now()
    resp, err := client.Do(req)
    if err != nil {
        // Record the context cause and elapsed time as poll evidence.
        return fmt.Errorf("poll flags after %s: %w", time.Since(started), err)
    }
    defer resp.Body.Close()

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("poll flags: status %d", resp.StatusCode)
    }

    var next Snapshot
    if err := json.NewDecoder(resp.Body).Decode(&next); err != nil {
        return fmt.Errorf("decode flag snapshot: %w", err)
    }
    if next.Revision == "" || next.Flags == nil {
        return fmt.Errorf("validate flag snapshot: missing revision or flags")
    }

    next.LoadedAt = time.Now()
    s.mu.Lock()
    s.current = next
    s.mu.Unlock()
    return nil
}

Notice what the code does not do: it does not clear current before the request, publish partially decoded data, or combine transport failure with a non-success status. In a Node.js implementation, create one controller per attempt, arrange for the deadline to abort that controller, pass its signal to fetch, and always clear the timer after settlement. If the runtime supports AbortSignal.timeout(), it can express the deadline directly, but the surrounding state machine remains the same.

Test the ugly path. A local test server should delay headers beyond the deadline, return a non-success response, close the connection early, and return invalid or incomplete JSON. Assert that each case has a distinct outcome and that none replaces the last-known-good snapshot. Then test a deployment rollback while the cached revision is newer than the code revision; the compatibility rule must be known, not discovered under a page.

Which signals make rollback safe?

Rollback safety requires low-cardinality operational metrics and high-context events, with traces linking the two where sampling permits. The metric should answer “is notification delivery outside its objective?” Labels such as region, deployment revision, delivery channel, and coarse outcome can help. Raw tenant IDs, request IDs, flag keys, and exception text usually create dangerous cardinality on metrics; keep diagnostic identifiers in structured events or trace attributes under explicit limits.

Signal	Question it answers	Rollback use
Delivery success and latency	Are users receiving notifications?	Confirms impact and recovery
Poll outcome and duration	Did the control-plane request meet its budget?	Separates deadline, network, and HTTP paths
Snapshot revision and age	What state made the decision?	Detects stale or divergent evaluation
Deployment revision by region	Which code was running?	Checks whether rollback actually converged
Fallback decision count	How often did policy replace fresh evaluation?	Exposes hidden degradation

Alert on the delivery symptom first. A rising count of aborted polls deserves investigation, but it should page only when it threatens a user-facing objective or consumes the stale-state budget; otherwise it becomes the kind of alert that teaches the on-call engineer to distrust the pager. The poll metric is supporting evidence. The delivery objective is the fire.

Trace context can connect the edge request, flag evaluation, and notification attempt, but sampling means a trace is never the sole audit record. Emit a compact evaluation event even when a full trace is absent, and make its timestamp, deployment revision, and snapshot revision sufficient for correlation. Keep clock skew in mind when comparing regions. If sequence matters, a revision or monotonic attempt counter is stronger evidence than wall-clock order alone.

Where this design should not be used

The catch is that last-known-good state is not automatically safe. Don't use this design for a flag whose stale value can violate an authorization boundary, regulatory hold, or immediate shutdown requirement. In those cases, keep the decision in an authoritative request path with a documented conservative response, or remove the remote flag from the safety boundary entirely. Stick with request-time evaluation when decisions must reflect current user attributes and the platform can provide a deadline that still leaves room for the notification operation.

Polling is also a poor fit when the control plane already offers a runtime-supported streaming or push contract and the edge environment can hold that connection reliably. Conversely, a push channel without durable local state still needs a startup and reconnection policy. The transport changes; the rollback questions do not.

Cost belongs in the design review, though it should not decide the incident policy. Poll frequency multiplied by edge regions, processes, and retries determines request volume, while high-cardinality telemetry determines storage and query cost. Reduce duplicate pollers, batch state into a versioned snapshot, and set retention by diagnostic value. Never trade away the revision evidence merely to make a graph cheaper; without it, the next rollback is guesswork.

The final readiness test is blunt: kill or delay the flag endpoint in a controlled environment, roll the notification release backward, and verify that delivery follows the documented fallback while the evidence still names both code and flag revisions. If that test cannot explain what page would fire, the system isn't ready for an incident.

DEV Community