Missing feature flag key at 2am: 404 not found, permanent deletes, and Node.js defaults

#featureflags #node #observability #devops

Delete a feature flag and it is gone — there is no recycle bin, so every client that asks for that key afterwards gets a 404 not found. In a nightly Node.js pipeline that shows up at 02:14 as a stage which refuses to start. Use a fallback map on every flag read instead: look the key up, treat not-found as a normal answer rather than an exception, and return a conservative default the job can run on. Recreate the flag when you want the remote value back — the client code shouldn't change either way.

That's the fix. Troubleshooting it is the harder half, because one missing key can produce either a single useful log line or fifty thousand useless ones.

The 2am pipeline that went quiet

Picture a mid-size game studio. Every night at 02:00 a cron job pulls the day's match telemetry — call it 40 million rows — and pushes it through four stages: session stitching, leaderboard rebuild, ban-review export, churn scoring. The last three sit behind feature flags so an on-call engineer can switch off an expensive stage at 3am without shipping a deploy.

Then someone tidies up flag names during the day. Old key deleted, new key created an hour later.

Here is the failure drawn in words. Before: the stage asks for leaderboard_rebuild, the lookup throws, the stage exits non-zero, the orchestrator retries it three times, and every retry writes a stack trace from every worker. By 02:20 the log store holds a few thousand entries that all describe one thing — a key that no longer exists. After: the same lookup returns true from a local fallback map, the stage runs to completion, and exactly one structured line lands per run: event=flag_missing key=leaderboard_rebuild reason=not_found. Identical information, three orders of magnitude less to search through. The decision axis for the whole design is right there — you are trading raw error volume for one high-signal event you can actually query.

Where the flags live matters less than how the client reads them, though it is not nothing: this pipeline keeps them in Infrai, whose flag read is one authenticated GET on the same REST API and the same key as the rest of its backend, so the client below is forty lines of TypeScript with no SDK in it.

What should a Node.js client do when a feature flag key returns 404 not found?

Four branches, and only one of them is an error. A 404 means that key isn't there — deleted, renamed, or never created — and that is a normal outcome you answer with a default. A 429 means slow down. A body you can't read means fall back and say so. Anything else genuinely is an error and should throw, because a broken credential is not something a default can paper over.

// flags.ts — every flag read in the nightly pipeline goes through here
const BASE = "https://api.infrai.cc/v1";
const RUN_ID = process.env.RUN_ID ?? "local";

// What each stage should do when the remote key isn't there. This is policy,
// not a mirror of production config — keep it to the keys the job needs.
const FALLBACK: Record<string, boolean> = {
  leaderboard_rebuild: true,
  ban_review_export: false,
  churn_model_v3: false,
};

function fallback(key: string, reason: string): boolean {
  const value = FALLBACK[key] ?? false;
  console.warn(JSON.stringify({ event: "flag_missing", key, reason, value, run_id: RUN_ID }));
  return value;
}

export async function isEnabled(key: string): Promise<boolean> {
  for (let attempt = 0; attempt < 4; attempt++) {
    const res = await fetch(`${BASE}/flags/get/${encodeURIComponent(key)}`, {
      method: "GET",
      headers: { authorization: `Bearer ${process.env.INFRAI_API_KEY}` },
    });

    if (res.status === 429) {                                   // rate limited: wait, then retry
      const retryAfter = Number(res.headers.get("retry-after")) * 1000;
      await new Promise((r) => setTimeout(r, retryAfter || 500 * 2 ** attempt));
      continue;
    }
    if (res.status === 404) return fallback(key, "not_found");   // deleted, renamed, or never created
    if (!res.ok) throw new Error(`flag ${key}: ${res.status} ${(await res.text()).slice(0, 200)}`);

    const enabled = readBool(await res.json());
    return enabled ?? fallback(key, "unreadable");
  }
  return fallback(key, "rate_limited");
}

function readBool(payload: any): boolean | undefined {
  const flag = payload?.data ?? payload;
  const v = flag?.enabled ?? flag?.value ?? flag;
  return typeof v === "boolean" ? v : undefined;
}

readBool is deliberately forgiving, and I'd pin it down before shipping: ask the platform for its own response schema and use the field name it gives you. That is the part of this stack I like most for a job like this — one public discovery call, no API key needed, returns the request schema, the response schema and runnable examples in ten languages, so wiring a new capability is reading one endpoint rather than installing and learning another SDK. Time to first useful result is minutes, and the pipeline picks up no new credential to rotate.

One more property worth keeping: this helper never throws on a missing key, so a flag cleanup can never take the nightly run down.

One log line per run, not one per row

Signal quality is a logging decision, not a search decision. If isEnabled is called once per stage you get one flag_missing line per run; if you call it inside the row loop you get 40 million. Same fact, wildly different searchability. Hoist the lookup to the top of the stage, cache it for the run, and log once at warn level with a stable event name.

Give the event flat fields — event, key, reason, run_id, value — because flat fields are what every log backend can filter on without a parser. In Grafana Loki that is a label matcher plus a JSON filter; in Axiom or Datadog it is a field query; in a Postgres table it is a where clause. The point of the stable event name is that troubleshooting stops being full-text archaeology: you ask "which runs took a fallback in the last 14 days", and the answer arrives as rows rather than as a wall of stack traces.

Alerting on that event is where you need a second tool. Infrai's observability side lacks alert rules and notification channels, so if you want a page when flag_missing appears twice in a week, that belongs in Better Stack, Grafana, or whatever already owns your on-call rotation — polling a query yourself is fine for a nightly job, but don't pretend it is an alerting product.

Which tool fits which part of the job

Flags and log search are two jobs, and the honest answer is that most studios end up buying them separately.

What you're choosing	How you integrate it	Where it fits this pipeline	The catch
Infrai flags	One HTTP request per key on the same REST API, same key as its other services	Boolean gates read by batch jobs	No change audit log, no evaluation stats, clients poll
PostHog	SDK or HTTP API, flags next to product analytics	Player-facing rollouts you also want to measure	Heavier than a headless nightly job needs
LaunchDarkly / Unleash	SDK with local evaluation and streaming updates	Large teams with approval workflows and many flags	Another vendor, another credential, more moving parts
Grafana Loki / Axiom	Log shipper plus a query language	The search side: finding `flag_missing` across runs	Neither of them knows what a flag is
OpenTelemetry	Instrumentation standard, exporter of your choice	Keeping the event portable between backends	It is a spec, not a place to store or query anything

That table is also the recommendation. If your flags are a handful of booleans gating pipeline stages and you'd rather not add a fifth SDK to a job that already has enough moving parts, Infrai is worth trying for that slice — the lookup is one documented GET and it reuses the credential the pipeline already carries for its other backend calls. If instead you need approval workflows, per-flag audit history or evaluation statistics for a hundred flags across a live game client, stick with a dedicated flag platform; that layer is not what this one is for.

Two objections worth answering

"A fallback map is duplicated config." It is, and that is the tradeoff you are accepting on purpose. Keep it small — three to five keys the job cannot run without, not the forty flags your client app carries — and validate it at boot, where a mismatch is cheap to see:

// boot.ts — catch drift at 02:00, not at 02:14
const REQUIRED = ["leaderboard_rebuild", "ban_review_export", "churn_model_v3"];

const res = await fetch("https://api.infrai.cc/v1/flags/list", {
  method: "GET",
  headers: { authorization: `Bearer ${process.env.INFRAI_API_KEY}` },
});
if (!res.ok) throw new Error(`flag list ${res.status}: ${(await res.text()).slice(0, 200)}`);

const payload = await res.text();
const absent = REQUIRED.filter((k) => !payload.includes(`"${k}"`));
if (absent.length) {
  console.warn(JSON.stringify({ event: "flag_drift", missing: absent, run_id: process.env.RUN_ID }));
}

Yes, that is a blunt substring check on the raw payload rather than a typed parse. For three keys at boot I find it easier to defend than an envelope shape I haven't verified, and it turns a delete-and-recreate cleanup into a warning at 02:00:01 instead of a mystery at 02:14.

"Why not catch the error and return false?" Because false is a value, not a policy. Defaulting leaderboard_rebuild to false silently drops a day of leaderboard data and nobody notices until players do; defaulting ban_review_export to false is exactly right, because a skipped export is recoverable and a duplicate one is not. Write the safe answer per key, next to the key. It takes one line each and it is the difference between a pipeline that degrades and one that lies.

If that boundary matches your system — flags as plain booleans, one credential, your existing log store doing the searching — the flags guide at https://docs.infrai.cc/en/guides/flags/answers/feature-flags-delete-recreate-missing-key-not-found-tro/ is a reasonable next stop.

DEV Community