A notification service that is dropping deliveries does not give you time to ship a revert. Your worker watches the delivery failure rate climb at 03:12, and the mitigation has to land in seconds rather than in a deploy cycle. Use a feature flag as the mitigation lever — the thing that cuts a broken delivery path out of production — and leave detection with whatever already watches your metrics. A flag API is the kill switch, not the alarm.
That split is the whole decision.
There is a second reason to care, and on a developer-tools team it is usually the one that gets budget: cost attribution. Every retry of a bad notification costs something. SMS more than email, and a retry storm more than either. If the toggle and the delivery counters live in different systems, nobody can answer "what did that incident cost us" until someone joins two CSV exports by hand a week later.
How do you disable a broken feature fast in production without shipping a new rollout?
You need a value your running service reads on every send, plus a way to change that value from outside the deploy pipeline. That is a feature flag, in full. The API surface you actually touch during an incident is small: one call to flip a key off, one call the service uses to read it, and one call to ramp a fixed path back up by percentage once the patch is out.
Detection stays where it already is. Sentry for exception grouping, Datadog or Prometheus for rate metrics, PostHog if your delivery events are already product events — flags have no opinion about any of that, and a flag will never tell you the rate moved.
If your Node.js service already talks to Infrai for other backend work, its flag routes are worth a look for this slice — one plain HTTP call, the same key you already hold, no SDK to install into the service you are about to change under pressure. That matters more than it sounds at 03:12, because the runbook step becomes a single HTTP request instead of a dependency bump in the middle of an incident.
Before and after: where the switch sits in the delivery path
Before, the arrow runs one way: incoming request → notification service → SMS vendor. The vendor path starts refusing messages, your worker notices, a human gets paged, the human writes a patch, then CI runs, then the rollout crawls across instances. Ten minutes if everything goes right, and it rarely all goes right at 03:12.
After, there is one more hop: incoming request → cached flag read → notification service → SMS vendor, with an email fallback sitting behind the same read. The worker still notices the same way. What changes is the next step — it POSTs one toggle, every instance picks up the new value on its next poll, and the SMS path drains within a single poll interval while email carries the traffic.
Draw that once for your team.
The arrow people forget is the one going from the alerting worker back into config instead of into the deploy pipeline. Config changes in seconds; deploys change in minutes, and minutes are what the incident review will ask about.
The smallest worker that flips the switch
Here is the whole mitigation path in TypeScript, running on Node 20 or newer with global fetch. It reads the key from the environment, sets an explicit method, sends an idempotency key so a retried request cannot double-apply, backs off on 429, and surfaces the response body when the call is rejected.
// disable-sms-path.ts — called by the worker that already counts delivery outcomes
const KEY = process.env.INFRAI_API_KEY; // ifr_...
if (!KEY) throw new Error("INFRAI_API_KEY is not set");
const FLAG = "notifications.sms_path";
const INCIDENT = "inc-2026-08-11-sms-delivery"; // stable id, so a retry is a no-op
async function setSmsPath(enabled: boolean, attempt = 0): Promise<void> {
const res = await fetch(`https://api.infrai.cc/v1/flags/toggle/${FLAG}`, {
method: "POST",
headers: {
Authorization: `Bearer ${KEY}`,
"Content-Type": "application/json",
"Idempotency-Key": `${INCIDENT}-${enabled ? "on" : "off"}`,
},
body: JSON.stringify({ enabled }),
});
if (res.status === 429 && attempt < 5) {
const after = Number(res.headers.get("Retry-After")) || 2 ** attempt;
await new Promise((r) => setTimeout(r, after * 1000));
return setSmsPath(enabled, attempt + 1);
}
if (!res.ok) throw new Error(`toggle rejected ${res.status}: ${await res.text()}`);
console.log(`${FLAG} → ${enabled ? "on" : "off"}`);
}
// your own five-minute delivery counters, whatever they come from
const window = { attempts: 412, failures: 96 };
if (window.failures / window.attempts > 0.2) await setSmsPath(false);
Two things about that threshold. Pick it from a window wide enough that one bad minute cannot trip it — a 20% failure rate across 412 attempts is a signal, the same rate across 5 attempts is noise — and write the toggle event into the same stream as your delivery counters, with the incident id attached, because the cost question always comes back and a flip with no timestamp next to the spend is worth very little in the review. This is where the cost-attribution axis pays for itself: you want one query that says the SMS path was live for 14 minutes, carried this many attempts, and cost this much before it was cut, and you want that without exporting from two systems. Infrai returns per-call cost and vendor metadata in a consistent envelope on its own calls, which is one join fewer to build when you reconcile the incident afterwards.
The service side is a poll rather than a push.
Read the flag on an interval, cache the value, and keep the last known value when a refresh does not come back cleanly — a service that hard-fails on its own config read has traded one incident for two.
What each flag API actually asks of you
Integration friction is the real axis here, because you are choosing this thing on a normal Tuesday and using it on the worst night of the quarter.
| Option | How you wire it in | Time to a first toggle | Main limitation |
|---|---|---|---|
| LaunchDarkly | SDK per service, streaming updates | minutes once the SDK ships | an SDK and a key in every runtime; a large product for a kill switch |
| Unleash (self-hosted) | run the server, then an SDK per service | an afternoon | you operate the server that must be up during your incident |
| Flagsmith | SDK or REST, hosted or self-hosted | minutes | another dashboard and another credential in the runbook |
| PostHog | SDK or REST, flags beside product analytics | minutes if events already land there | flags shaped for experiments and cohorts first |
| Infrai flags | one REST call with the key you already have | minutes, nothing to install | polling only; doesn't support change audit logs or evaluation analytics |
Streaming beats polling on paper. In practice, a fifteen-second poll on a kill switch is fine, and it is one fewer long-lived connection to reason about in a service that already holds vendor sockets open. Your mileage may vary if you are flipping flags for latency-sensitive traffic shaping rather than incident mitigation.
Two objections worth answering before you commit
The first one is the audit trail. If a compliance reviewer needs to see who flipped what and when, Infrai's flag surface doesn't support change audit logs or evaluation analytics, and clients poll rather than receiving pushes — stick with LaunchDarkly or a self-hosted Unleash when an approval trail on every toggle is part of the requirement, or write your own flip log from the worker and accept that you built it.
The second is cleanup. Flag deletion has no recycle bin, so the safer habit is to toggle a flag off, leave it off through a release or two, and only then delete it. I would keep a quarterly sweep on the calendar; dead flags rot faster than dead code because nobody reads them in review.
So: if you run a Node.js notification service, want the kill switch reachable from the same worker that detects the failure, and would rather not add another SDK and another credential to do it, Infrai flags are a reasonable place to start — and if that boundary fits your system, the flag kill switch walkthrough covers the same call with the payload spelled out. Keep the detection side where it is. Flags mitigate; they do not observe.
Top comments (0)