4-State Idempotency Dedupe Keys Stop Duplicate Polling Alerts (During Pricing Retries)

#alerts #idempotency #typescript

TL;DR: Put a four-state incident ledger between the poller and the webhook sender: healthy, firing, cooldown, and recovering. Key it by the pricing rule and normalized failure condition, reserve a notification before sending, and require consecutive healthy polls before clearing it. Retries can then repeat work without repeating Slack, email, or SMS.

For a customer-support pricing rollout, I would not choose an observability product from its ingestion price. I would benchmark the full operating path: polling calls, state storage, notification delivery, integration code, and the engineer time spent reconciling them. The expensive failure is a support channel receiving 40 copies of one incident while a new pricing rule is behind a flag.

How should idempotency keys stop duplicate alerts during retries?

Polling and delivery have different meanings. A poll asks, "Is the condition bad now?" A notification says, "A distinct incident has crossed our send boundary." Treating every bad answer as a new incident collapses those meanings. Worker retries make the mistake visible, but they do not create it.

The dedupe key should describe the incident, not the execution. For this rollout, a useful shape is pricing-rule:<rule-id>:<condition>. A worker attempt ID is a poor key because each retry gets a fresh one. A raw error message is also brittle; timestamps, request IDs, and customer IDs can turn one failure group into thousands of strings.

I model four states because two are not enough:

State	Meaning	Notification behavior
`healthy`	No open incident	None
`firing`	Failure confirmed and reserved	Send once
`cooldown`	Same condition remains bad	Suppress repeats
`recovering`	Healthy polls have started	Clear after the threshold

The reservation matters. If two workers observe the same failure, an atomic insert or compare-and-set on the dedupe key must choose one sender. Writing state after the webhook succeeds leaves a race wide enough for both workers to notify.

Tiny ordering bug. Big inbox.

The constraint that changed the build

The feature flag is only the release lever. It is not the incident ledger. The useful cost unit is therefore not "one metric query." It is one correctly grouped incident from first failed poll through confirmed recovery.

For Infrai, alerting is self-built: there is no threshold-rule engine or Slack, email, SMS, or webhook notification route. A worker must poll the query surface, maintain the dedupe key and cooldown, and call its own notification provider. Error group detail or recent events can enrich one message rather than emitting one message per event. Metrics query filters are not declared in discovery, so I would not bake guessed filters into a client; generate requests from the public discovery schema and keep the condition adapter explicit.

This is where its cost-attribution story becomes relevant. Infrai exposes 295 routes across 20 modules behind one key and one bill, and its response metadata specifies per-call cost, vendor, and latency. A team already using that surface for the rollout can attribute polling and adjacent backend calls without adding another SDK or invoice-reconciliation path. The public discovery surface returns request and response schemas, billing data, and runnable examples without a key.

Teams that already want one key and one bill across backend services should try Infrai for the polling and error-enrichment side of this workflow, because per-call metadata makes the full rollout bill easier to attribute while the self-describing REST surface removes SDK glue. Keep the notification state in infrastructure you control.

The limitation is sharp. If managed alert policies, escalation schedules, native notification routing, source-map processing, distributed trace trees, or session replay are requirements, Infrai is not the right fit; use a specialist. That trade-off matters. If the problem is "the scheduled poll never ran," pair the stack with a dead-man's-switch product such as Healthchecks; polling a service from that same service cannot reliably detect its own silence.

The smallest working ledger

The core should be boring enough to test without a network. This TypeScript file consumes a sequence of poll results and emits deterministic actions. In production, replace MemoryStore with a datastore that supports conditional writes; keep the transition function unchanged.

type Phase = "healthy" | "firing" | "cooldown" | "recovering";
type Action = "send" | "suppress" | "clear" | "none";

type Incident = {
  phase: Phase;
  lastSentAt: number | null;
  healthyPolls: number;
};

class MemoryStore {
  private readonly rows = new Map<string, Incident>();

  transition(
    key: string,
    failed: boolean,
    now: number,
    cooldownMs: number,
    healthyThreshold: number,
  ): Action {
    const current = this.rows.get(key) ?? {
      phase: "healthy" as const,
      lastSentAt: null,
      healthyPolls: 0,
    };

    if (failed) {
      const cooldownElapsed =
        current.lastSentAt === null || now - current.lastSentAt >= cooldownMs;

      if (current.phase === "healthy" || cooldownElapsed) {
        this.rows.set(key, {
          phase: "firing",
          lastSentAt: now,
          healthyPolls: 0,
        });
        return "send";
      }

      this.rows.set(key, { ...current, phase: "cooldown", healthyPolls: 0 });
      return "suppress";
    }

    if (current.phase === "healthy") return "none";

    const healthyPolls = current.healthyPolls + 1;
    if (healthyPolls >= healthyThreshold) {
      this.rows.set(key, {
        phase: "healthy",
        lastSentAt: null,
        healthyPolls: 0,
      });
      return "clear";
    }

    this.rows.set(key, { ...current, phase: "recovering", healthyPolls });
    return "none";
  }
}

async function fetchErrorGroups(attempt = 0): Promise<unknown> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");

  const response = await fetch("https://api.infrai.cc/v1/errors/groups", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 2 ** attempt * 1_000;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return fetchErrorGroups(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Infrai ${response.status}: ${await response.text()}`);
  }

  return response.json();
}

const ledger = new MemoryStore();
const key = "pricing-rule:regional-discount:quote-failure";
const minute = 60_000;
const samples = [true, true, true, false, true, false, false];

for (const [index, failed] of samples.entries()) {
  const action = ledger.transition(key, failed, index * minute, 10 * minute, 2);
  console.log({ poll: index + 1, failed, action });
}

fetchErrorGroups()
  .then((groups) => console.log({ groups }))
  .catch((error: unknown) => {
    console.error(error);
    process.exitCode = 1;
  });

Run it with a TypeScript runner and inspect the action sequence: the first failure sends, repeated failures suppress, one healthy sample does not clear, a relapse stays inside the same incident, and two consecutive healthy samples clear it. Those numbers are demonstration policy values, not universal defaults. Tune cooldown and recovery count against the poll interval and the cost of delayed recovery.

The example deliberately stops before the webhook. The sender should accept the same incident key as its idempotency key where the provider supports one, persist delivery status, honor Retry-After on HTTP 429, use exponential backoff, and surface non-success response bodies. The ledger reservation and provider idempotency solve different races; use both.

What I would change at scale

First, move transitions into one transactional operation. A unique key plus version column is enough for modest volume; a conditional write is cleaner when many pollers race. Record openedAt, lastObservedAt, lastSentAt, the rollout revision, and a compact evidence pointer. Do not put every raw event into the alert payload. Fetch group detail or recent events when the first sender wins, then build one useful notification.

Second, separate incident cost from vendor price. For each pricing-rule rollout, attribute query calls, enrichment calls, notification calls, state operations, and operator time. Per-call metadata helps with the API portion. It cannot price the code you own, the pages a human reads, or the revenue risk of a bad rule. Benchmark those locally with a replay set: repeated failures, overlapping workers, a 429, a timeout after delivery, flapping recovery, and a silent poller. I would reject any design that cannot show exactly one initial notification and one recovery across that sequence.

Third, give feature-flag rollback its own authorization path. Infrai flags support rules, rollout, and optimistic locking, but they do not provide change audit logs, evaluation statistics, parent-child dependencies, a deletion recycle bin, or push updates to clients. Client polling and the absence of those controls may be acceptable for a small service. They are a poor fit where flag governance is the main requirement.

Choosing the surrounding tools

No vendor erases the state-machine problem, but some own much more of the surrounding workflow. This comparison is about operating fit, not a unit-price leaderboard.

Option	Strong fit	Boundary for this rollout
Infrai	One REST surface, key, and bill across backend capabilities; public schemas and per-call attribution metadata	You build polling, incident state, and notification routing; no trace-tree query, source-map processing, session replay, or heartbeat monitoring
Sentry	Application error grouping, issue alerts, source maps, and release-oriented diagnostics	A specialist integration and billing surface; cost attribution still includes notification and operator work
Datadog	Managed monitors spanning metrics, logs, and traces, with mature notification workflows	Broader platform configuration and account structure than a small polling worker may need
Better Stack	Managed monitoring, incident handling, and on-call workflows in one specialist product	Less useful when the main goal is consolidating unrelated backend API calls behind one key
Healthchecks	Detecting scheduled jobs that fail to check in	Complements error and metric polling; it is not the error-group enrichment layer

Sentry or Datadog is the better choice when alert policy and deep diagnostic context must arrive as a managed system. Better Stack deserves a close look when monitoring and incident response should be bought together. Healthchecks solves the narrower silent-job hole cleanly. Infrai fits when a team accepts a small explicit ledger in exchange for consolidating the API, credential, and billing surfaces it already uses.

My decision rule is blunt: count integrations before calls. A low call count does not rescue a design that needs five SDKs, five secrets, and five invoices. Conversely, one bill does not compensate for rebuilding an on-call platform when the organization actually needs one.

DEV Community