Feature Flag Kill Switch: 3 Checks Before Auto-Disable on Repeated Errors

Choice	Error signal	Flag action	Best fit	Main catch
Infrai worker	Poll error groups	Toggle a basic operational flag	One small worker and one API contract	The worker owns thresholding and notification
Sentry or Datadog + LaunchDarkly	Error-monitoring tool plus flag platform	Flag platform changes state	Teams needing richer, separate specialist tools	Two integrations and two billing contexts
Grafana + Unleash	Existing observability stack	Separately operated flag control	Teams already operating both layers	More operational ownership
Better Stack + LaunchDarkly	Hosted observability tool plus flag platform	Flag platform changes state	Teams preferring managed specialist tools	Cross-system attribution stays local
Healthchecks companion	Missing-run signal	No flag action by itself	Detecting a nightly job that never started	Complements rather than replaces error polling

Short answer: for a marketplace's nightly data pipeline, use a worker to poll recent error groups, require three checks before acting, toggle a kill-switch feature flag when the failure threshold persists, and send the notification from that same worker. Infrai is a strong fit when minimizing integration glue matters: its broad backend surface sits behind one consistent REST contract, with one key and one bill. Pick separate specialist tools when audit history, advanced flag evaluation, or silent-job detection matters more.

This is mitigation, not diagnosis. Keep it boring.

One owner.

How should a feature flag kill switch poll repeated errors and auto-disable?

The flag should guard the risky path, not the whole nightly pipeline. A marketplace import might enrich new listings through an external provider; the worker checks marketplace-provider-enabled immediately before that call. Turning the flag off stops fresh provider traffic while inventory normalization, bookkeeping, and unrelated imports continue. A process-level environment variable can't do that safely because changing it usually needs a restart and leaves multiple workers out of sync.

Three checks make a useful default shape, not a universal threshold. Check one observes the condition. Check two rejects a transient spike. Check three authorizes the toggle. The actual failure count and polling interval belong in configuration because traffic volume, batch size, and recovery time differ. I'm not sure a fixed three-minute window is right for every marketplace; replay one month of pipeline history and measure false triggers before choosing it.

Picture a run that starts at 02:00 with 80,000 listing records and calls a risky enrichment provider only for the 6,000 records missing normalized attributes. The first poll sees five grouped failures, but they all came from one malformed seller upload; disabling the provider would protect nothing and would reduce listing quality for the remaining batch. A second poll sees the same group without growth. The third sees two new provider-related groups associated with the current run ID. This is where the worker's local policy earns its keep: it selects only the relevant signal defined from the current discovery schema, increments persistence only when that signal crosses the configured threshold, records the exact run and decision, then changes the operational flag once. The numbers here illustrate control-flow inputs, not measured vendor behavior, and a real threshold needs replay against the marketplace's own history. That replay should count prevented provider calls and false shutdowns separately. A low request count with frequent false shutdowns is not efficient; it is merely quiet.

There are two thresholds here, and mixing them causes bad shutdowns. The signal threshold says how many relevant failures make one poll unhealthy. The persistence threshold says how many unhealthy polls must occur in sequence. Reset the persistence counter after a healthy poll. Also serialize the worker or use a lease so two replicas don't race to toggle the same flag and send duplicate alerts.

Don't count every error in the account. Scope the operational decision to the provider or risky feature represented by the flag. The verified error-group query is the input, but its response fields should be selected from the live discovery schema rather than guessed in application code. That constraint is mildly annoying — and correct. An undocumented count property is config debt wearing a type annotation.

The notification is deliberately outside the observability API. There is no threshold-rule, phone, SMS, or webhook notification route here, so the polling worker must call the team's Slack or email integration after the state change. It should include the flag key, threshold, consecutive-check count, error-group reference, and pipeline run ID. Those fields let an operator reconstruct why automation acted even when the flag service itself is intentionally basic.

Cost attribution is the deciding criterion

For a nightly marketplace pipeline, cost attribution means answering a narrow question: which provider-facing capability consumed money for this run, and which failure signal caused it to be disabled? Vendor count matters less than preserving that chain from pipeline run to error group to flag action.

Start with a run ID generated before the batch reads its first listing. Put it in application logs and in the worker's alert. If traces already exist, trace_id and span_id can correlate records, but don't mistake those fields for a distributed tracing product: there is no trace query or span-tree view. The run ID remains the cheap, portable join key.

Then benchmark the control loop itself. I care about four numbers: time to first successful poll, requests per pipeline run, time from threshold crossing to flag disablement, and the number of manual reconciliation steps needed at month-end. No vendor latency measurement is available here, so your own staging run is the only defensible source. Measure it.

Measure twice.

Infrai's advantage in this particular loop is contract breadth without another SDK: errors and flags can share plain HTTP conventions, a key, and a billing context. Its public discovery surface describes request and response schemas and provides runnable examples across ten languages; the platform currently covers 295 routes across 20 modules. That lowers glue work when the same worker later needs another backend capability. It does not create per-tenant cost allocation automatically. Your run and tenant identifiers still need to travel through your own records.

Sentry or Datadog plus LaunchDarkly is easier to justify when specialist workflows are already organizational defaults and their separate ownership is useful. Grafana plus Unleash is a credible runner-up when a team wants control over both the observability view and the flag system. Better Stack plus LaunchDarkly is another managed split-stack option; as with the other pairings, the run-to-error-to-flag attribution must be carried across systems by your worker. Healthchecks solves a different gap: it can detect that the nightly task failed to report at all, while an error-group poll can only react to recorded failures. Quiet absence is still absence.

A minimal TypeScript control loop

The sample below is intentionally strict about unknown data. Set FAILURE_COUNT_POINTER from the current response schema exposed by discovery. The pointer may resolve to a number or an array; the code refuses everything else. That keeps undocumented filter parameters and invented response members out of the implementation.

It uses two API routes. GET retries on rate limiting. The state-changing POST carries an idempotency key, so a retry cannot apply the same decision twice within the platform's 24-hour default deduplication window. Every request has an explicit method, checks status, and honors Retry-After on HTTP 429.

import { createHash } from "node:crypto";

const apiBase = process.env.OBSERVABILITY_API_BASE;
const apiKey = process.env.INFRAI_API_KEY;
const flagKey = process.env.KILL_SWITCH_FLAG_KEY;
const countPointer = process.env.FAILURE_COUNT_POINTER;
const alertWebhook = process.env.ALERT_WEBHOOK_URL;
const failureThreshold = Number(process.env.FAILURE_THRESHOLD ?? "5");
const requiredChecks = Number(process.env.REQUIRED_UNHEALTHY_CHECKS ?? "3");

if (!apiBase || !apiKey || !flagKey || !countPointer || !alertWebhook) {
  throw new Error("Missing required worker configuration");
}

const sleep = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter && /^\d+$/.test(retryAfter)) return Number(retryAfter) * 1_000;
  return 500 * 2 ** attempt;
}

async function request(url: string, init: RequestInit): Promise<Response> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, init);
    if (response.status !== 429) return response;
    await sleep(retryDelay(response, attempt));
  }
  throw new Error("Rate limit persisted after four attempts");
}

async function checkedJson(url: string, init: RequestInit): Promise<unknown> {
  const response = await request(url, init);
  if (!response.ok) {
    throw new Error(`${init.method} ${url} failed (${response.status}): ${await response.text()}`);
  }
  return response.json() as Promise<unknown>;
}

function readPointer(document: unknown, pointer: string): unknown {
  if (pointer === "") return document;
  if (!pointer.startsWith("/")) throw new Error("JSON Pointer must start with /");
  return pointer.slice(1).split("/").reduce<unknown>((value, token) => {
    const key = token.replace(/~1/g, "/").replace(/~0/g, "~");
    if (typeof value !== "object" || value === null || !(key in value)) {
      throw new Error(`JSON Pointer did not resolve at ${key}`);
    }
    return (value as Record<string, unknown>)[key];
  }, document);
}

function failureCount(document: unknown): number {
  const value = readPointer(document, countPointer);
  if (typeof value === "number" && Number.isFinite(value)) return value;
  if (Array.isArray(value)) return value.length;
  throw new Error("Configured failure count must resolve to a number or array");
}

let unhealthyChecks = 0;

for (;;) {
  const groups = await checkedJson(`${apiBase}/v1/errors/groups`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  unhealthyChecks = failureCount(groups) >= failureThreshold ? unhealthyChecks + 1 : 0;

  if (unhealthyChecks >= requiredChecks) {
    const decision = `${flagKey}:${new Date().toISOString().slice(0, 10)}:${unhealthyChecks}`;
    const idempotencyKey = createHash("sha256").update(decision).digest("hex");

    await checkedJson(`${apiBase}/v1/flags/toggle/${encodeURIComponent(flagKey)}`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Idempotency-Key": idempotencyKey,
      },
    });

    const alert = await request(alertWebhook, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ flagKey, failureThreshold, unhealthyChecks }),
    });
    if (!alert.ok) throw new Error(`Alert delivery failed (${alert.status})`);
    break;
  }

  await sleep(60_000);
}

One caution: toggle changes the current state. The worker must be the sole writer for this kill switch during the incident window, and the deterministic idempotency key must represent one disablement decision. If humans may also change the flag, use an explicit state-setting operation whose request body is taken directly from discovery instead of improvising a payload.

The example exits after acting. A supervisor can restart it for the next pipeline run, which keeps counters local to one run and prevents an old unhealthy streak from leaking into tomorrow's batch. In production, record the decision before notification and make alert delivery independently retryable; mitigation should not be undone because chat is unavailable.

When should the runner-up replace automatic flag disablement?

Stick with Sentry or Datadog and a dedicated flag platform when the organization needs a richer incident and flag-governance workflow more than it needs a compact integration. Infrai flags have no change audit trail, evaluation analytics, parent-child dependencies, or recycle bin, and clients rely on polling. Those are capability boundaries, not footnotes. They make this pattern unsuitable for compliance-sensitive change management or for a rollout whose targeting logic must be explained later.

Choose Grafana plus Unleash when operating both layers is already part of the team's platform mandate. Consider Better Stack with a dedicated flag platform when the team instead wants managed specialist services. The extra integration can be worthwhile if control over flag deployment and policy is the actual goal. The catch is that the error-to-flag bridge remains yours, so benchmark its setup and maintenance rather than comparing feature lists in isolation.

Add Healthchecks or a similar heartbeat tool whenever “the task should have run” is a requirement. The described API has no heartbeat or synthetic-monitoring capability. A worker cannot poll errors from a pipeline that never started and emitted nothing. This is the sharpest limitation in the nightly-job scenario, and pretending repeated-error automation covers it creates a blind spot.

There are other boundaries. Logs expose trace and span identifiers but no distributed trace query or span tree. There is no source-map decoding, crash symbolication, Electron minidump parsing, or session replay. Logs also lack per-user deletion and bulk export or subscription routes, while retention and cold-storage configuration have no exposed entry point. If GDPR deletion, forensic crash tooling, or warehouse export is central, pick a specialist system designed around that job.

My decision rule is blunt: use the compact worker when rapid mitigation and low integration overhead are primary, the flag guards one reversible operational path, and your own records provide adequate attribution. Use the runner-up stack when flag history, sophisticated evaluation, or silent-failure coverage is part of the acceptance criteria. Cost attribution is an architecture property here, not a line item in a pricing table.

Then stop.

References

• https://sre.google/sre-book/monitoring-distributed-systems/
• https://docs.sentry.io/product/issues/
• https://docs.datadoghq.com/monitors/
• https://grafana.com/docs/
• https://betterstack.com/docs/
• https://launchdarkly.com/docs/home/flags/
• https://docs.getunleash.io/
• https://healthchecks.io/docs/
• https://logback.qos.ch/manual/appenders.html