DorianReed2186

Posted on Oct 7

Diagnosing Feature Flag Fetch Timeouts at the Edge for Rollback-Safe Pricing

#node #observability #typescript

A remote feature flag timeout must never leave a pricing request guessing which rule to use. The safe default is to put a deadline around each configuration fetch, keep the last validated snapshot outside the request path, and record enough evidence to distinguish caller cancellation from an upstream timeout. For a new B2B SaaS pricing rule, the rollback unit is the validated snapshot, not the individual HTTP attempt. Serve a known snapshot or the documented baseline; never turn uncertainty into a new price.

TL;DR: use one AbortController per fetch, classify aborts separately from transport failures, validate before publishing a snapshot, and expose snapshot age plus rule revision in traces and logs. Poll in a long-lived Node.js process. In an edge function, assume instances are disposable and make correctness independent of any timer surviving after the response.

How should a feature flag fetch timeout affect rollback?

A message such as fetch failed is not a diagnosis. Four states matter: the caller ended the work, the local deadline expired, the remote service rejected the request, or a network error prevented an HTTP response. They have different owners and different rollback implications.

AbortController supplies an AbortSignal to fetch; aborting the controller can terminate the fetch operation. Node.js exposes the browser-compatible controller and fetch globals. Those primitives solve cancellation, but they do not choose a fallback or produce useful operational evidence on their own.

For pricing, keep the evidence small and high-cardinality fields out of metric labels. A trace or structured log can carry a deployment revision, flag revision, outcome, elapsed time, snapshot age, and execution environment. Aggregate counters should use bounded dimensions such as outcome and runtime, while request IDs belong in logs or trace attributes.

One specific trap is reusing a controller across polling iterations. Once aborted, its signal remains aborted. Every later request then fails immediately, which resembles a persistent upstream outage even after the original slow request has gone away. Create the controller inside each attempt. Troubleshooting gets much easier when deadline, caller_abort, and network_error are separate outcomes rather than three paths to the same generic exception: an operator can act on sustained deadline pressure, an application owner can inspect caller cancellation, and the networking path can own connection failures. The distinction costs a few extra branches, but it prevents a retry policy from treating deliberate cancellation as an outage.

That is the trade-off.

Put the fallback contract in code

The runnable example uses only runtime APIs and a generic endpoint. It separates fetching, validation, snapshot publication, and evaluation. The pricing baseline is deliberately boring: when no validated snapshot exists, the new rule stays off.

type PricingConfig = {
  revision: string;
  newPricingRule: boolean;
};

type Snapshot = {
  config: PricingConfig;
  fetchedAt: number;
};

type FetchOutcome =
  | { kind: "fresh"; snapshot: Snapshot; elapsedMs: number }
  | {
      kind: "deadline" | "caller_abort" | "http_error" | "invalid" | "network_error";
      elapsedMs: number;
      status?: number;
    };

const BASELINE: PricingConfig = {
  revision: "baseline",
  newPricingRule: false,
};

let lastValidated: Snapshot | undefined;

function isPricingConfig(value: unknown): value is PricingConfig {
  if (typeof value !== "object" || value === null) return false;
  const candidate = value as Record<string, unknown>;
  return (
    typeof candidate.revision === "string" &&
    candidate.revision.length > 0 &&
    typeof candidate.newPricingRule === "boolean"
  );
}

async function fetchConfig(
  endpoint: string,
  timeoutMs: number,
  callerSignal?: AbortSignal,
): Promise<FetchOutcome> {
  const startedAt = performance.now();
  const controller = new AbortController();
  let deadlineExpired = false;

  const timer = setTimeout(() => {
    deadlineExpired = true;
    controller.abort(new Error("configuration deadline exceeded"));
  }, timeoutMs);

  const forwardCallerAbort = () => controller.abort(callerSignal?.reason);
  callerSignal?.addEventListener("abort", forwardCallerAbort, { once: true });

  try {
    const response = await fetch(endpoint, {
      signal: controller.signal,
      headers: { accept: "application/json" },
    });
    const elapsedMs = Math.round(performance.now() - startedAt);

    if (!response.ok) {
      return { kind: "http_error", status: response.status, elapsedMs };
    }

    const candidate: unknown = await response.json();
    if (!isPricingConfig(candidate)) return { kind: "invalid", elapsedMs };

    const snapshot = { config: candidate, fetchedAt: Date.now() };
    lastValidated = snapshot;
    return { kind: "fresh", snapshot, elapsedMs };
  } catch (error: unknown) {
    const elapsedMs = Math.round(performance.now() - startedAt);
    if (controller.signal.aborted) {
      return {
        kind: deadlineExpired ? "deadline" : "caller_abort",
        elapsedMs,
      };
    }
    return { kind: "network_error", elapsedMs };
  } finally {
    clearTimeout(timer);
    callerSignal?.removeEventListener("abort", forwardCallerAbort);
  }
}

function currentPricingConfig(maxStaleMs: number): PricingConfig {
  if (!lastValidated) return BASELINE;
  const ageMs = Date.now() - lastValidated.fetchedAt;
  return ageMs <= maxStaleMs ? lastValidated.config : BASELINE;
}

Parsing and validation finish before lastValidated changes. Readers never observe a half-updated object.

There are two clocks for different jobs. performance.now() measures elapsed request time without depending on wall-clock adjustments. Date.now() records snapshot age across evaluations. Neither timestamp should be treated as a globally ordered event sequence across machines.

Make polling fit the runtime

A conventional Node.js service can start one polling loop during process initialization. Add jitter so replicas do not refresh in lockstep, prevent overlapping attempts, and stop the loop during graceful shutdown. The request handler should only read the current immutable snapshot; it should not wait for a control-plane call before calculating a quote. This keeps configuration latency out of the customer-facing path.

Edge functions change the lifecycle assumption. A returned response may be followed by instance suspension or disposal, so an in-memory interval is not a dependable scheduler. Fetch during a request only when the platform lifecycle and latency budget allow it, or refresh through external scheduled execution and deliver a versioned snapshot through storage available to the function. Local memory can be an opportunistic cache, but it cannot be the sole correctness mechanism.

Keep one rule across both environments: only a complete, validated revision becomes active. If revision pricing-42 is the rollout candidate and operators disable it, the next validated snapshot must restore the previous behavior without a deploy. A timeout during that refresh does not prove either revision is correct. It proves only that fresh control-plane state is unavailable.

Short polling intervals reduce detection delay but create more requests and more correlated load during trouble. Long intervals reduce background work but extend the time before a rollback is observed. Choose the interval from the maximum acceptable rollback propagation time, then reserve room for retries and jitter. For an illustrative 60-second rollback objective, a team might reserve 15 seconds for detection, two 5-second attempts, and the remaining 35 seconds for scheduling jitter and propagation; those are planning inputs, not universal defaults. Do not derive the interval from a library default.

This pattern has limits. A stale snapshot is unsuitable when policy forbids using old pricing state at all, and falling back to a baseline is unsuitable when that baseline could violate an existing contract. In either case, fail the quote explicitly and send it to a review path rather than manufacture a price. Polling is also a poor fit when rollback must propagate faster than the shortest sustainable polling interval. A pushed, versioned snapshot can reduce detection delay, though it adds connection lifecycle and replay handling. The choice is between bounded staleness and more delivery machinery, not between a good and bad implementation.

No hidden magic.

Observe decisions, not just requests

A fetch span explains the control-plane attempt. A separate event at evaluation time explains what the application did. That second record matters while checking disputed invoices: rule revision, boolean decision, snapshot age bucket, and whether the source was fresh, stale, or baseline. Do not attach account IDs or raw flag payloads to metric labels.

W3C Trace Context defines the traceparent and tracestate fields used to propagate trace context across services. Propagate context when the configuration service participates in tracing, but do not let missing tracing headers affect flag correctness. Observability is evidence, not a dependency of the pricing decision.

A compact event interface keeps instrumentation replaceable:

type RefreshEvent = {
  outcome: FetchOutcome["kind"];
  elapsedMs: number;
  runtime: "node" | "edge";
  activeRevision: string;
  snapshotAgeMs: number | null;
};

type RecordRefresh = (event: RefreshEvent) => void;

function reportRefresh(
  record: RecordRefresh,
  outcome: FetchOutcome,
  runtime: RefreshEvent["runtime"],
): void {
  record({
    outcome: outcome.kind,
    elapsedMs: outcome.elapsedMs,
    runtime,
    activeRevision: lastValidated?.config.revision ?? BASELINE.revision,
    snapshotAgeMs: lastValidated ? Date.now() - lastValidated.fetchedAt : null,
  });
}

The operational view needs both rate and age. A low timeout rate can hide a stuck poller if there are no attempts. Alerting only on failures misses silence. Track time since the last successful validated refresh, and compare it with the rollback objective.

An HTTP success is not a configuration success until the body is validated and published. This distinction prevents a health view from reporting green while every application instance rejects the same malformed revision.

Test the rollback path before rollout

Unit tests should use a local stub or injected fetch function, not a live configuration service. Cover a response before the deadline, a response after it, a caller abort, a non-success status, invalid JSON, a valid but wrong-shaped object, and recovery on the next poll. Use fake timers carefully: advance both the deadline and mocked response, then assert that the timer and abort listener are cleaned up.

The higher-value test runs two revisions through the evaluator. Start with the baseline, publish a validated enabled revision, make refreshes time out, and verify the stale policy. Then publish a validated disabled revision and verify rollback. This catches an implementation where telemetry says the refresh succeeded but the request path still reads an older object.

Run one load test with many replicas and jitter enabled. The goal is not an invented universal throughput number. It is to see whether restarts or a remote slowdown synchronize polling in your deployment, and whether retry traffic competes with quote traffic for sockets, CPU, or logging capacity. Measure locally.

The release checklist is a rollback argument

Before enabling the new pricing rule, write down the baseline behavior, maximum snapshot age, fetch deadline, polling interval, and maximum rollback propagation time. Confirm that the deadline is shorter than the execution budget and that a failed refresh cannot overwrite a valid snapshot. Confirm that logs distinguish deadline, caller cancellation, HTTP status, invalid content, and network failure without storing secrets or full pricing payloads.

Then rehearse the bad path. Delay the endpoint beyond the deadline, return an invalid document, terminate the caller, and disable the rule in a later valid revision. Watch the application continue with its stated fallback and eventually adopt the rollback. Check that the trace identifies the slow attempt while the evaluation record identifies the revision actually used.

Ship only when those observations agree. A graph of fast fetches is comforting, but the real release criterion is narrower: under control-plane uncertainty, every quote uses an explicit, explainable rule revision.

DEV Community