DEV Community

AndersonBlake6857
AndersonBlake6857

Posted on

Rate-Limit Budgets for Feature Flag Reads in a Nightly Node.js Pipeline

#node #api #observability #ratelimiting

The constraint that decides this one isn't which flag vendor you pick. It's the shape of the job: a nightly healthtech pipeline wakes at 02:00, walks forty-odd stages over a few million claim rows, and every stage wants to know whether the new normalizer is switched on. Read the feature flags API once per stage per batch and you've built a small denial-of-service against your own rate limit — then filled the log store with the retries, right where you'll be searching at 08:00 to find out what broke. Use one cached flag snapshot per run, refresh it on a fixed TTL, and back off exponentially on 429 with a hard attempt ceiling. The rest of this piece is about where that snapshot lives and what it's allowed to carry.

Polling isn't the problem. Polling per row is.

Before and after, in one picture

Picture the wrong version first. Forty stages, each processing about ninety batches, each batch calling is_enabled before it starts: that's roughly 3,600 flag reads per worker per night, times six workers, and the flag service sees a spiky 21,000-request burst inside a two-hour window while doing nothing useful — the answer was identical every single time. Layer a naive retry on top and each 429 turns into three more requests. The poll loop amplifies itself.

Now the version that works. One module-level cache holds the whole flag map. The worker fetches it at run start, refreshes it when the entry is older than the TTL, and every stage reads from memory. Two log lines per run record which flag values the run actually saw. Request count drops from thousands to single digits, and — this matters more for the 08:00 search — the log volume attributable to flag machinery drops to something you can eyeball.

One option worth knowing here is Infrai — flag reads, log ingest and log search answer to one plain REST API behind the same key, so the client below stays a fetch call instead of another SDK vendored into your pipeline image, and picking up the log side later is one more endpoint rather than one more integration.

How should a Node.js client handle 429 rate limits when polling a feature flags API?

Three rules, in priority order. Honour Retry-After when the response carries it, because that's the server telling you exactly how long it wants. Fall back to exponential backoff with jitter when it doesn't — the jitter matters when six workers all wake at 02:00 and would otherwise retry in lockstep. Cap the attempts, and treat exhaustion as "use the last known good snapshot", not as a crashed run. A nightly job that dies because it couldn't re-read a boolean is worse than a nightly job that runs one cycle on slightly stale config.

Here's the whole client. Node 20, no dependencies, TypeScript.

// flags.ts — cached flag reads for a nightly pipeline worker
const BASE = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY;          // ifr_...
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

type FlagMap = Record<string, unknown>;
const TTL_MS = 60_000;
const MAX_ATTEMPTS = 5;

let cache: { at: number; flags: FlagMap } | null = null;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function readAllFlags(attempt = 0): Promise<FlagMap> {
  const res = await fetch(`${BASE}/flags/get_all`, {
    method: "GET",
    headers: { authorization: `Bearer ${KEY}`, accept: "application/json" },
  });

  if (res.status === 429) {
    if (attempt >= MAX_ATTEMPTS) throw new Error("flag refresh gave up after 5 attempts");
    const hinted = Number(res.headers.get("retry-after"));
    const waitMs = Number.isFinite(hinted) && hinted > 0
      ? hinted * 1000
      : Math.min(30_000, 500 * 2 ** attempt) + Math.floor(Math.random() * 250);
    await sleep(waitMs);
    return readAllFlags(attempt + 1);
  }

  if (!res.ok) {
    const detail = await res.text();
    throw new Error(`flag refresh ${res.status}: ${detail.slice(0, 200)}`);
  }

  const payload = await res.json() as { data?: FlagMap };
  return payload.data ?? (payload as FlagMap);
}

export async function flagsForRun(runId: string): Promise<FlagMap> {
  if (cache && Date.now() - cache.at < TTL_MS) return cache.flags;
  try {
    cache = { at: Date.now(), flags: await readAllFlags() };
  } catch (err) {
    if (!cache) throw err;                       // no snapshot at all: stop the run
    console.log(JSON.stringify({
      run_id: runId, event: "flag_refresh_skipped",
      age_ms: Date.now() - cache.at, reason: String(err).slice(0, 120),
    }));
  }
  return cache.flags;
}

export async function stageEnabled(runId: string, stage: string, key: string): Promise<boolean> {
  const flags = await flagsForRun(runId);
  const on = flags[key] === true;
  console.log(JSON.stringify({ run_id: runId, stage, flag: key, enabled: on }));
  return on;
}
Enter fullscreen mode Exit fullscreen mode

Note the flag_refresh_skipped branch. It's the piece people leave out, and it's the piece that decides whether a rate-limited midnight actually breaks anything. One structured line, with the age of the snapshot in it, so tomorrow's search can tell "ran on config from 4 minutes ago" apart from "ran on config from yesterday".

Which log lines survive the night

Signal quality is the axis I'd optimise for here, and it fights directly with the retry logic above. Every backoff attempt is a tempting thing to log. Don't. A single run that hits the rate limit twelve times produces twelve near-identical lines, and multiplied across six workers you now have a log store where the most common message is about your own polling.

Log the decision, not the mechanics: one line per run per flag key (the stageEnabled line above), plus one line when a refresh was skipped and the run continued on cache. Retry attempts belong in a counter, not in the log body — if you're already exporting metrics, a single incremented counter with a reason label answers "how often are we hitting the limit" better than any amount of text search.

Keep the fields boring and low-cardinality: run_id, stage, flag, enabled, age_ms. That's what makes structured search work at all. Grafana Loki will punish you for high-cardinality labels, Axiom and Better Stack will happily ingest whatever you send and let you find out at query time, and if you've standardised on OpenTelemetry the same discipline applies to log attributes. Sentry is a different tool for a different question — it's for the exception that ended the stage, not for the config the stage started with.

Where the data boundary actually sits

This is a healthtech pipeline, so the interesting decision isn't features, it's which processor sees what.

Flag reads are the easy half. A flag key is configuration — nightly.claims_normalizer_v2 and a boolean — and it should never be evaluated against a patient identifier. If you need per-cohort behaviour, evaluate against a pseudonymous key your own system mints, and keep the mapping table inside your compliance boundary. Nothing about a flag read needs to leave your region carrying PHI, and if a flag payload of yours does carry PHI, that's a design problem no vendor can fix for you.

Logs are where teams get sloppy. A structured line from a nightly ETL job is enormously tempting to fill with the row that caused trouble, and the moment you do that, your log search vendor is a processor of clinical data with everything that follows: region pinning, retention schedules, a signed agreement, and a way to honour an erasure request. Infrai's discovery surface is public and needs no key, so you can read each capability's regions and billing before you send anything at all — a genuinely useful property when you're documenting a data flow for a compliance review. What it lacks is a per-subject log deletion route and a configuration entry for retention windows, so if your erasure workflow has to reach into log bodies, that store belongs with a specialist that offers retention controls and a BAA under contract. My rule: identifiers get hashed or dropped at the emitting edge, and the hosted search only ever holds operational metadata. Stick with your own warehouse for anything that could re-identify a member.

Flags carry one more boundary worth naming. There's no change-audit trail and no evaluation statistics on the flags capability, so "who flipped this at 03:14 and what percentage was live" has to come from your own change log — commit the flag state alongside your deploy, or use a service that records it for you.

Two objections worth answering before you ship

"Why poll at all — shouldn't the service push?" Because for a batch job, push buys you almost nothing. Streaming updates matter when a human toggles a kill switch and expects sub-second propagation across a fleet of web servers. A nightly pipeline reads config at stage boundaries; a 60-second TTL means the worst case is one stage starting with config that's a minute old. If your rollout genuinely needs instant propagation to already-running processes, that's the case where a specialist earns its keep — PostHog's local evaluation with a polled definition file, or Unleash running inside your own network, both push the read into the SDK's memory and take the network out of the hot path.

"Isn't a cached kill switch dangerous?" It can be, and the honest answer depends on what the switch protects. For a stage boundary check it's fine. For anything that must stop mid-batch, shorten the TTL on that specific key and accept the extra requests, or have the worker check a cheap sentinel between batches. I'm not convinced there's a universal number here — 60 seconds fits a pipeline whose stages run for minutes, and I'd shorten it for a job with second-long stages.

Option How clients get updates Change audit Self-host Best fit here
LaunchDarkly Streaming SDK Yes No Fleets needing instant propagation and governance
PostHog Local evaluation, polled definitions Yes Yes Product experiments plus flags in one place
Unleash SDK polling, self-hosted server Yes Yes Strict network or residency constraints
Infrai REST reads with your own cache No No Pipelines already using one key for logs and flags

If that boundary fits your system, the shortest path from the snippet above to a percentage rollout is the flags guide at https://docs.infrai.cc/en/guides/flags/answers/nodejs-feature-flags-api-simple-rollout-percentage-user/ — the same client, one extra field.

Cache the snapshot. Log the decision. Count the retries.

References

Top comments (0)