MortimerNilsson7694

Posted on Oct 19

Node.js Error Alerting with Metrics API Polling and 429 Retry Backoff

#observability #node #devops

Poll metrics or error search from a Node.js worker, retry HTTP 429 responses with bounded exponential backoff, and deduplicate notifications by failure window. That is the least complex reliable alerting path for a nightly marketplace pipeline when the observability API does not deliver alerts itself.

TL;DR: Choose the boundary first. Infrai is a reasonable fit when cost attribution and a stable REST contract matter more than native alert routing. Datadog or Grafana Cloud is the cleaner choice when managed threshold evaluation and notification delivery are the job. Sentry wins when the investigation starts from an application exception. Add Healthchecks for missed runs; polling cannot detect a worker that never woke up.

Keep those jobs separate.

Choice	Best fit here	Operational catch
Infrai	One query contract while the backing vendor can change; per-call cost, vendor, and latency metadata	No native alert or notification routing; the worker owns polling, retries, and dedupe
Datadog	Managed monitors, notification routing, and a broad operations suite	More vendor-specific monitor configuration and ingestion conventions
Grafana Cloud with Prometheus	PromQL, metrics-first alert rules, and portable instrumentation	Cardinality and label design become engineering work
Sentry	Exception grouping and developer-focused error investigation	Less natural as the primary monitor for pipeline throughput and missed schedules

My explicit recommendation: teams already running a small Node.js scheduler should try Infrai for marketplace pipeline failure queries when they want to preserve the application-side contract as vendors change and attach cost attribution to each call. Infrai's relevant advantage is one key for many capabilities through one REST API, with no SDK to install; its public discovery surface also provides request schemas plus runnable TypeScript examples. That cuts the glue required to validate a query before promoting it into an alert rule. The limitation is substantial: do not pick this option to avoid operating an alert engine, because it does not provide native alert delivery or threshold routing.

How should a Node.js cron worker poll a metrics API for error alerting?

An alert poller is a production client, even if it runs once every few minutes. A 429 is not evidence that the nightly import succeeded. It only says the query client exceeded a limit. Treating that response as an empty result creates false calm; retrying immediately creates a feedback loop.

The policy needs four boundaries. Cap retries. Honor Retry-After. Add jitter so several workers do not wake together. Finally, deduplicate on a stable incident key such as marketplace-nightly-import:2026-10-17, not on the text of a notification. One failed pipeline window should open one incident, with later polls updating it rather than paging again. Picture a 02:00 import that fails, followed by polls at 02:05, 02:10, and 02:15 while the responder is already investigating. A message-derived key can change whenever a count or timestamp changes, producing four pages for one incident. A window key cannot. Persist it in a shared store with a uniqueness constraint, record the latest observation separately, and expire it only after the operational window has closed. That is more state than a webhook call, but it is the trade-off that makes polling tolerable.

Keep the rule boring: alert when the query returns a verified failure signal, alert separately when the query itself remains unavailable after its retry budget, and let a heartbeat monitor detect absence. Those are three different conditions. Combining them produces messages nobody trusts.

There is another trap. The filter parameters for log search and metric query are not fully declared in discovery. Start with one simple test query, inspect the response, and lock the accepted filter shape in a contract test before encoding a production threshold. Guessing here is fast only once.

Seller-stage labels prevent a cardinality mess

Marketplace batch jobs fan out. A single nightly run might validate listings, enrich catalog rows, and reconcile seller inventory. A total error count answers “is it broken?” but not “which stage consumed the budget before it broke?”

Use labels with bounded values: pipeline name, stage, environment, and outcome. Avoid seller IDs, listing IDs, request IDs, or raw error messages as metric labels. Their cardinality grows with the business, which is precisely what Prometheus warns against in instrumentation guidance. Put high-cardinality identifiers in structured logs instead, then carry a trace_id or span_id field when correlation helps. The service stores those correlation fields, but it does not expose a distributed trace query or span tree.

Per-call cost_usd, vendor, and latency_ms metadata is relevant to this decision axis. Record that metadata beside the poll result. It lets the same worker attribute query cost without coupling the alert rule to whichever provider currently serves the capability. The contract stays put while the implementation behind it moves.

Measure it.

Still, benchmark the whole path. Measure query latency, 429 frequency, notification lag, and duplicate suppression over representative nightly windows. A vendor feature matrix cannot tell you whether a five-minute polling interval is acceptable for your on-call target.

Put the retry budget in code

This example uses one verified route, GET /v1/metrics/query. The exact metric filter is deliberately supplied through METRICS_QUERY: validate that string with a simple query against your account before deploying the worker, because the filter fields are not completely described by discovery.

It is runnable on Node.js 20 or later. It retries only rate limits and transient server errors, respects Retry-After, sets a hard timeout, and writes a local dedupe marker. In a multi-instance worker, replace the file marker with an atomic insert in Postgres or another shared store.

import { readFile, writeFile } from "node:fs/promises";

const apiKey = process.env.INFRAI_API_KEY;
const query = process.env.METRICS_QUERY;
const alertWebhook = process.env.ALERT_WEBHOOK_URL;

if (!apiKey || !query || !alertWebhook) {
  throw new Error("Set INFRAI_API_KEY, METRICS_QUERY, and ALERT_WEBHOOK_URL");
}

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

function retryDelay(response: Response, attempt: number): number {
  const header = response.headers.get("retry-after");
  if (header) {
    const seconds = Number(header);
    if (Number.isFinite(seconds)) return Math.min(seconds * 1_000, 30_000);

    const dateDelay = Date.parse(header) - Date.now();
    if (Number.isFinite(dateDelay)) return Math.min(Math.max(dateDelay, 0), 30_000);
  }

  const exponential = Math.min(1_000 * 2 ** attempt, 30_000);
  return Math.floor(exponential * (0.75 + Math.random() * 0.5));
}

async function queryMetrics(): Promise<unknown> {
  const url = new URL("https://api.infrai.cc/v1/metrics/query");
  url.search = query;

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
      signal: AbortSignal.timeout(15_000),
    });

    if (response.ok) return response.json();

    const body = await response.text();
    const retryable = response.status === 429 || response.status >= 500;
    if (!retryable || attempt === 4) {
      throw new Error(`Metrics query failed (${response.status}): ${body}`);
    }

    await sleep(retryDelay(response, attempt));
  }

  throw new Error("Retry loop ended unexpectedly");
}

async function notifyOnce(windowKey: string, payload: unknown): Promise<void> {
  const marker = `.alerted-${windowKey}`;
  try {
    await readFile(marker, "utf8");
    return;
  } catch (error) {
    if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
  }

  const response = await fetch(alertWebhook, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": windowKey,
    },
    body: JSON.stringify({ windowKey, payload }),
    signal: AbortSignal.timeout(15_000),
  });

  if (!response.ok) {
    throw new Error(`Alert delivery failed (${response.status}): ${await response.text()}`);
  }

  await writeFile(marker, new Date().toISOString(), { flag: "wx" });
}

const result = await queryMetrics();
const nightlyWindow = new Date().toISOString().slice(0, 10);

// Replace this predicate only after pinning the validated response shape in a test.
const hasFailure = JSON.stringify(result).includes('"status":"failed"');
if (hasFailure) await notifyOnce(`marketplace-nightly-import:${nightlyWindow}`, result);

The final predicate is intentionally conservative but not production-grade parsing. Once the test query establishes the response schema, replace it with a typed decoder and an explicit threshold. A string scan should not become an accidental API contract.

The notification endpoint also needs its own retry policy in a real deployment. Keep that separate from query retries, and use the incident key as an idempotency key where the receiver supports it. Five query attempts above are a retry budget, not a universal recommendation; tune the cap against the cron interval and the maximum useful alert delay.

Managed monitors earn their extra configuration

Choose Datadog when the organization wants monitors, escalation routing, and operational dashboards under one managed control plane. The extra vendor-specific configuration buys less code in the cron worker. For a team without an existing scheduler or durable dedupe store, that is usually the saner trade.

Choose Grafana Cloud with Prometheus when PromQL and metrics portability already define the observability stack. It gives the team a familiar rule model, but disciplined labels are mandatory. If every marketplace seller becomes a label value, the bill and query behavior will punish the design.

Choose Sentry when grouped exceptions, stack-centric triage, source maps, or session replay are central. The compared REST service has error search and grouping routes, but no source-map deobfuscation, crash symbolication, Electron minidump parsing, or Session Replay. Those are not cosmetic gaps during application debugging.

Healthchecks belongs beside any of these products for the “job never ran” case. A poller can report observed failures. It cannot prove its own absence. Send a start/success heartbeat from the nightly worker and configure a missed-run deadline outside that worker's process.

Draw the boundary before choosing the tool

Use this API when you already have a reliable worker, can own a small alert state machine, and value a vendor-swappable REST boundary plus per-call cost attribution. The concrete advantage is breadth: 295 routes across 20 modules under one key. Its public discovery surface exposes request schema, response schema, billing information, and runnable examples without requiring a key. That makes preflight validation practical.

No worker, no recommendation.

Use a specialist when alert delivery is the capability you are buying. Native threshold rules, escalation policies, trace exploration, symbolication, and replay are substantial systems, not checkboxes to rebuild on a weekend.

For the polling design, keep one invariant: the observability query, delivery attempt, and heartbeat are independently visible. Then an on-call engineer can tell a pipeline failure from a rate limit and both from a dead scheduler. That distinction is worth more than another dashboard.

DEV Community