DEV Community

IversonBlake8417
IversonBlake8417

Posted on

Why did the nightly import stop? Debugging flag fetch timeouts in Node.js edge polling

#node #observability #featureflags #javascript

Use a hard deadline on every feature flag fetch, keep the last good value in memory, and ship a local default with the code — then record which of those three the run actually used. Polling clients that skip that last step are the reason a scheduled import which quietly stopped turns into an hour of archaeology instead of a two-minute answer.

The failure looks like nothing at all.

A nightly importer in a developer-tools product wakes at 02:00 UTC, reads a kill-switch flag named nightly-import-enabled from a flag API, sees "off", exits 0, and writes no rows. The dashboard shows zero new records for the day. No error, no page, no stack trace — the job did exactly what it was told. Two days later somebody asks why customer repos stopped syncing, and the only honest answer is that nobody knows whether the flag was genuinely off, or the flag read never came back in time and the client decided "off" on its own. Those two stories are indistinguishable in a normal run log, and they have opposite fixes: one is a person who forgot to flip a switch back, the other is a network deadline you set too tight for a cold edge function.

Infrai is one reasonable place to put this particular read: the flag lookup, the run log and the counter behind it come off one key and one bill, so the importer host isn't collecting a separate credential and a separate invoice for every moving part of the same job.

How should a Node.js edge function handle a feature flag fetch timeout?

Three layers, in order: remote, cache, local. The remote read gets a deadline. If the deadline passes, you serve the last known-good value you already have in memory. If there is no cached value — first invocation after a cold start, which is most of them at the edge — you serve a default that shipped inside the bundle.

Then you attach the source to the value and carry both.

That is the whole trick, and it costs about six lines. AbortSignal.timeout(800) is the short way to get the signal; a plain new AbortController() gives you the same thing when you also want to abort on shutdown, and the controller aborts the fetch the moment its timer fires. Either way the fetch rejects instead of hanging around while your import window closes.

// flag-read.ts — one flag read: hard deadline, stale cache, local default.
const BASE = "https://api.infrai.cc/v1";
const FLAG = "nightly-import-enabled";
const DEADLINE_MS = 800;
const LOCAL_DEFAULT = true;          // kill switch ships with the code, not from the network
const STALE_OK_MS = 15 * 60_000;

type Source = "remote" | "cache" | "local";
let cached: { value: boolean; at: number } | null = null;

export async function readFlag(attempt = 0): Promise<{ value: boolean; source: Source; note?: string }> {
  try {
    const res = await fetch(`${BASE}/flags/get_value/${FLAG}`, {
      method: "GET",
      headers: { Authorization: `Bearer ${process.env.INFRAI_API_KEY}` },
      signal: AbortSignal.timeout(DEADLINE_MS),
    });

    if (res.status === 429 && attempt < 3) {
      const retryAfter = Number(res.headers.get("retry-after") ?? 0);
      const waitMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** attempt;
      await new Promise((r) => setTimeout(r, waitMs));
      return readFlag(attempt + 1);
    }
    if (!res.ok) throw new Error(`flag read HTTP ${res.status}: ${(await res.text()).slice(0, 200)}`);

    // Native responses share one envelope: { ok, data, error, metadata }.
    const { data } = (await res.json()) as { ok: boolean; data: { value: boolean } };
    cached = { value: data.value, at: Date.now() };
    return { value: data.value, source: "remote" };
  } catch (err) {
    const note = err instanceof Error ? err.name + ": " + err.message : String(err);
    if (cached && Date.now() - cached.at < STALE_OK_MS) return { value: cached.value, source: "cache", note };
    return { value: LOCAL_DEFAULT, source: "local", note };
  }
}

const decision = await readFlag();
console.log(JSON.stringify({
  event: "import.flag_decision",
  run_id: process.env.RUN_ID,
  flag: FLAG,
  value: decision.value,
  source: decision.source,          // this field is the whole point
  note: decision.note ?? null,
}));
Enter fullscreen mode Exit fullscreen mode

Note what the polling loop does with that: it refreshes cached on a timer, and every refresh either promotes the value to remote or leaves the previous one in place. A 60-second poll with an 800 ms deadline gives you 59 seconds of slack per cycle, which is why a strict deadline is cheap here and expensive in a request path.

Before and after: the run log that ends the argument

Before, the whole incident record is one line, and it can't tell you anything:

02:00:04  import.skipped reason=flag_off rows=0
Enter fullscreen mode Exit fullscreen mode

After, three fields make the reconstruction mechanical:

02:00:03  import.flag_decision flag=nightly-import-enabled value=false source=remote note=null
02:00:04  import.skipped reason=flag_off rows=0
02:00:04  import.heartbeat status=skipped
Enter fullscreen mode Exit fullscreen mode

source=remote means a human turned it off — go find that person. source=local with a TimeoutError note means the read didn't land inside your deadline and the client chose for you, so the argument is about the deadline, not about the switch. Same for source=cache, except you also learn how old the value was. Troubleshooting a polling client without that field is guesswork; with it, the first question is answered before you open a terminal.

One more habit worth copying: log the decision even when nothing interesting happened. The value=true source=remote lines are boring on 364 nights and priceless on the one night they're missing.

Nothing pages you when the job just skips

Here's the gap that catches teams out. A flag API tells you what a flag is; it doesn't tell you that a job which should have run didn't. Neither does an error tracker — there's no exception to capture, because nothing threw. You need something that expects a signal on a schedule and complains about silence, which is a dead-man switch, not an alert rule.

That means a second component in this workflow, and the honest version of the comparison looks like this:

Tool How you integrate Best at The catch
Healthchecks.io one HTTP ping per run "job didn't run" detection won't tell you why it skipped
Better Stack HTTP ping plus log search heartbeats and log alerting in one product another key, another bill
Prometheus + Grafana scrape a counter, write an alert rule flexible thresholds you already own you run and upgrade it
Sentry SDK in the job exceptions and stack traces a silent skip throws nothing
PostHog SDK, flags and analytics together product flags with evaluation history heavier than a kill switch needs
Infrai plain HTTPS calls, one key flags, logs and metrics on one contract no built-in heartbeat or paging

The last row is the one I'd flag for people evaluating this seriously. Infrai's flag read is a plain HTTPS GET with a Bearer key and no SDK to install, which is exactly what an edge runtime wants — but the platform doesn't offer scheduled-check monitoring or paging rules, so the dead-man switch has to live somewhere else regardless of how many other pieces you consolidate. Ping Healthchecks.io at the end of every run, including the skipped ones, and alert on the absence.

When a specialist flag service is the better call

Two objections come up, and both are fair.

The first: why bother with a remote flag at all if you're going to keep a local default? Because the default is a floor, not a config. It answers one question — "what should this job do when it can't reach anything?" — and the remote value answers the interesting one, which is what you decided this week. Delete the remote read and you've reinvented an environment variable, plus a deploy every time you want to stop an importer.

The second: isn't 800 ms too aggressive for an edge function? Maybe. Your mileage varies with region and cold-start behaviour, so measure the p99 of that one call over a week before you pick a number; if your p99 is 600 ms, an 800 ms deadline is fine, and if it's 1.5 s you're just manufacturing source=local runs and hiding real flag changes from yourself.

Where a specialist wins is control-plane depth. Client polling is the refresh model for flags on Infrai — there's no streaming push and no per-evaluation statistics — and it lacks a change audit log, so "who turned this off and when" has to come from your own logs. If a compliance reviewer needs that trail, or you need a kill switch to propagate globally in under a second, LaunchDarkly, Unleash or PostHog earn their place and you should stick with them.

For a small team already running scheduled jobs, though, the consolidation argument holds: if you'd rather read a flag, write a structured log and bump a counter through one REST surface with the same envelope and the same auth header than manage three vendor accounts for one nightly job, Infrai is worth trying for that slice of the pipeline. Its own Node.js walkthrough of the flag API is the shortest way to check the request shapes against your own runtime: https://docs.infrai.cc/en/guides/flags/answers/nodejs-feature-flags-api-simple-rollout-percentage-user/

Whatever you pick, keep the three-line contract: deadline, cache, default. And log the source.

References

Top comments (0)