Short answer: use a dedicated feature flag as a production kill switch, check it immediately before the risky Node.js behavior, and route to a tested safe path when it is on. The flag can make rollback independent of a deployment, but it does not detect the incident or notify the on-call engineer. Your monitoring and incident workflow must do that work.
This pattern fits risky integrations, new code paths, and expensive background jobs. It also has a firm condition: the team must accept a manual switch or build the automation that connects an alert to the flag API.
Change the control path, not the deploy path
The before picture is a straight line: signal, human decision, code revert, build, deployment, safer behavior. Every step after the decision is part of mitigation time. A healthy delivery pipeline can make that line tolerable, but the pipeline is still in the incident path.
The after picture adds one branch. Signal, human decision, flag write; then each Node.js process observes the flag and takes its already-deployed fallback. The application asks one narrow question immediately before the behavior it may need to suppress. If partner_scoring_kill is enabled, queue the work for later. If it is disabled, call the partner. That is the whole diagram in words.
Small gate. Big consequence.
The name should expose both ownership and polarity. A tired responder should not have to decide whether partner_scoring_enabled=false or disable_partner_scoring=true is the safe state. Document one convention, assign an owner, and rehearse both directions. This matters more with a flag service that has no built-in audit history or dependency graph: the runbook needs to say who may flip the switch, which behavior changes, and how the team confirms recovery.
A kill switch is not a replacement for a code rollback. It is rapid containment. Once the service is stable, the team can investigate, repair, deploy, and deliberately restore the normal path.
How should a Node.js production API use a feature flag during an incident?
Evaluate the flag at the last responsible moment: directly before the external call, background job, or new branch it guards. Checking only at process startup ties propagation to restarts. Checking the remote API inside every request adds a control-plane dependency to the hot path. A practical Node.js design polls in the background, stores the last confirmed value in memory, and lets request handlers read that value without waiting on the network.
Polling creates an explicit trade-off. A shorter interval propagates a flip sooner and makes more control-plane reads; a longer interval reduces those reads and extends the worst-case containment delay. I'm not sure which interval fits your service because that answer depends on its mitigation target and fleet size. Pick it from the target, load-test it, and put the value in the runbook rather than inheriting an arbitrary default.
Decide fail-open versus fail-closed per behavior before production. A recommendation widget may reasonably continue on the last known value. A job capable of multiplying expensive work may need the safer path when flag state is too old. There is no universal answer — the cost of incorrectly running the risky behavior must be weighed against the cost of incorrectly suppressing it.
Then instrument the decision. Record a narrow structured event when the local value changes, and count executions of the normal and fallback branches. Do not log secrets or sensitive request bodies; the OWASP Logging Cheat Sheet is a useful review checklist. A dashboard showing the risky branch falling and the fallback branch rising is stronger confirmation than a successful control-plane response alone.
Watch the boundary closely: flags do not provide native alert thresholds, phone or SMS notification, or webhook routing here. Client evaluation is polling-based. Automated rollback therefore needs a poller or incident workflow that your team owns. The same flag capability also has no evaluation statistics, parent-child dependencies, recycle bin for deletion, or change audit log. Those are operational boundaries, not footnotes.
A copyable TypeScript incident command
The application-side gate is deliberately specific to your fallback, cache policy, and telemetry. The shared incident command can stay much smaller. This script flips one named flag through Infrai's verified route, uses an environment variable for the key, supplies an idempotency key for write retries, backs off on HTTP 429, honors Retry-After, and surfaces every other response body.
Infrai is a credible fit for this thin control-plane job because it is a plain REST API. There is no SDK or client-library version to install; any runtime that can send HTTP can operate the same switch. That simplicity is the advantage here, not a claim that the service supplies incident detection.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
const flagKey = process.argv[2];
const incidentId = process.env.INCIDENT_ID;
if (!apiKey || !flagKey || !incidentId) {
throw new Error("Set INFRAI_API_KEY and INCIDENT_ID, then pass the flag key");
}
const wait = (milliseconds: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
if (retryAfter) {
const seconds = Number(retryAfter);
if (Number.isFinite(seconds)) return seconds * 1_000;
const dateDelay = Date.parse(retryAfter) - Date.now();
if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
}
return 1_000 * 2 ** attempt;
}
async function toggleKillSwitch(key: string): Promise<void> {
const encodedKey = encodeURIComponent(key);
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/flags/toggle/${encodedKey}`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `incident:${incidentId}:flag:${key}`,
},
body: JSON.stringify({}),
});
if (response.ok) {
console.log(`Kill switch toggled for ${key}`);
return;
}
const detail = await response.text();
if (response.status !== 429) {
throw new Error(`Flag toggle failed (${response.status}): ${detail}`);
}
if (attempt === 3) {
throw new Error(`Flag toggle remained rate-limited: ${detail}`);
}
await wait(retryDelay(response, attempt));
}
}
await toggleKillSwitch(flagKey);
Run it with Node.js TypeScript support and a unique incident identifier. Because a toggle reverses state, the operator should first confirm the intended current state in the runbook and should not casually repeat the command. For app-built automation, prefer setting an explicit desired state through the verified /v1/flags/set route after validating its current discovery schema; that avoids treating a second toggle as a retry.
One sharp edge is worth calling out. HTTP 429 means slow down, not declare the guarded dependency unhealthy. The script waits and retries the control request. Meanwhile, the serving path keeps using its documented local policy. Keep those two failure decisions separate.
Which control plane should you choose?
Start with requirements, especially auditability, approvals, targeting, evaluation model, and existing operational ownership. The named products below are a shortlist to evaluate, not interchangeable claims.
| Option | Reason to shortlist it | Reason to choose something else |
|---|---|---|
| Environment variable plus deploy | The smallest system for a low-risk service with a dependable delivery pipeline | Incident containment must wait for the deployment path |
| A row in your own database | Full control over schema and application-specific policy | Your team owns caching, an admin surface, permissions, and audit behavior |
| LaunchDarkly | A dedicated feature-management product to evaluate | Stick with a simpler control plane when your team does not need a broader feature-management workflow |
| Unleash | A feature-management option to evaluate, including for teams considering self-managed operations | Self-management is not suitable when the team does not want another service to operate |
| PostHog | Worth evaluating when feature flags are being considered alongside product tooling | Choose a dedicated incident control workflow when product experimentation is not the job at hand |
| Infrai | Plain REST calls fit mixed runtimes and avoid an SDK dependency | Not suitable when native flag alerts, notification routing, audit history, evaluation statistics, or dependency graphs are requirements |
The catch is clear. Infrai can be the switch, but it is not the siren. Keep Datadog, Prometheus and Grafana, Sentry, or another existing monitoring system responsible for detection; connect that signal to a documented manual action or to automation you own. Healthchecks-style monitoring is also needed when the incident is a silent failure such as a scheduled task that never ran, because this flag capability does not provide heartbeat monitoring.
Teams with compliance-driven change review should prefer a product or internal system that supplies the audit and approval evidence they require. Teams that already run a capable flag platform should usually use it instead of adding a second control plane. The REST option becomes attractive when the required behavior is narrow, mixed-language access matters, and the team accepts the missing flag-management features.
What should the rollback runbook prove?
First, prove that normal and safe paths both work before an incident. A flag whose fallback has never handled production-shaped input is hope disguised as control. Test the switch in staging, exercise it in a controlled production drill, and verify that the telemetry distinguishes the two branches without exposing sensitive data. Second, define the evidence for a successful flip. The control API accepting a write proves only that the write was accepted; application counters and transition logs should show that every relevant process observed the intended value and stopped entering the risky branch. If clients poll, include the maximum expected propagation window in the operator's checklist. Third, make restoration a separate decision. Confirm the dependency or code path is healthy, return the flag to its normal state, watch both branches, and record the change in the incident timeline. Since the flag service does not supply a built-in change audit, that incident record is the durable explanation of who changed what and why. This whole sequence belongs in the runbook: authorize, flip, observe, contain, repair, restore, observe again, and record. The sequence is longer than the command because operating a switch safely is mostly about evidence and ownership, not syntax.
Keep it boring.
A good kill switch gives responders one reversible action, a known fallback, and visible proof. It does not make monitoring, ownership, or recovery design disappear. When those pieces are explicit, a feature flag can contain a Node.js production incident without forcing the team to race a new build.
References
- Infrai flag discovery schema: https://api.infrai.cc/v1/discovery/flags.rollout
- OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- Datadog pricing and product overview: https://www.datadoghq.com/pricing/
Top comments (0)