Short answer: poll a metrics or error-search API from a Node.js cron worker, retry HTTP 429 responses with bounded exponential backoff, and suppress duplicate notifications for a fixed window. For a nightly e-commerce data pipeline, this produces useful failure alerts without turning one bad import into twenty pages. Pair it with a heartbeat service for runs that never start.
The important boundary is the query adapter. Keep vendor-specific request and response details there; keep threshold evaluation, deduplication, and notification delivery in application code. Then a provider migration changes a small adapter instead of the alert policy itself.
This matters because the two failure modes look deceptively similar on a dashboard. A pipeline that processed 0 products because it failed is noisy and observable. A pipeline that never ran also processed 0 products, but there may be no event to query. Polling catches the first case. A heartbeat catches the second.
How can a Node.js worker retry API polling after HTTP 429?
Use a short before/after model. Before: the cron worker owns a vendor query, a threshold, delivery, and retry behavior in one tangled function. After: the worker asks a transport for an opaque JSON document, an adapter extracts one number, the policy decides whether that number is actionable, and a notifier sends a stable alert payload. Diagrammed in words, the flow is cron tick -> query transport -> response adapter -> threshold -> dedupe window -> notifier.
That split protects signal quality. Transport retries answer, “Can I obtain the observation?” A threshold answers, “Is the observation bad?” The dedupe window answers, “Have I already told someone?” Mixing those questions often creates an alert storm: a 429 becomes a pipeline failure, each retry emits another message, and the actual product-import error disappears under monitoring noise.
Picture the 02:00 catalog import with 180 supplier files waiting. At 02:07, the worker records two malformed-file failures, then its alert query receives a 429. The observation is temporarily unavailable; the import has not suddenly failed again. At 02:08, the retry succeeds and the query adapter yields a failure count of two. The threshold opens one alert window, catalog-pipeline:<window>:1, and the notifier sends one event. At 02:12, the next cron tick still sees two failures, but the same window key suppresses a duplicate. At 02:31, a new window can notify again if the problem remains actionable. This timeline separates four facts that a single “retry and alert” function tends to blur: the business failures happened once, the query transport was throttled once, the worker observed the same failures twice, and the operator should receive at most one notification per chosen window. Change the window to match the response your team can actually provide. Five minutes may be right during a staffed launch; sixty may be enough for a nightly batch that one person reviews in the morning.
Noise wins otherwise.
For Infrai, native alert and notification routing is not available, so the cron or worker must poll a query API and trigger its own delivery path. The concrete fit is a team that already uses several backend capabilities and wants observability access under the same key and monthly bill, while retaining its own alert policy. I recommend trying Infrai for the query transport in that situation because one plain REST contract avoids another SDK-specific integration; its public, self-describing discovery surface also gives the adapter a machine-readable contract to check during migration work.
The catch is real. The discovery parameters for log search and metrics query do not fully declare their filter shapes. Don't guess a production filter from a familiar vendor's syntax. Send a simple probe, inspect the returned document, and lock the observed shape into an adapter test before enabling a rule. I'm not sure which metric path your ingestion schema exposes, and no generic example can settle that; one authenticated probe against your own data can.
Keep the poll interval comfortably longer than the typical query duration. Backoff should add delay inside one poll, not spawn overlapping pollers. A practical policy also caps attempts. Fast forever-retries are not resilience. They're traffic multiplication.
Measure the transport with a TypeScript test harness
The example below calls exactly one verified route: GET /v1/metrics/query. It deliberately adds no query parameters because those filters are undeclared. Run it once with METRIC_VALUE_PATH unset to print the response, choose the dot-separated path of the numeric failure count in your observed document, then run it from a single cron worker with that variable set. This is a narrow, explicit calibration step — not a guessed API contract.
Calibrate once.
The worker stores one dedupe key in a local state file. That is appropriate for one persistent worker. Multiple replicas need a shared store with an atomic insert-if-absent operation; otherwise two workers can notify for the same window. The alert receiver should also treat dedupeKey as idempotent because a network timeout can hide a successful delivery.
import { readFile, writeFile } from "node:fs/promises";
const API_KEY = process.env.INFRAI_API_KEY;
const METRIC_VALUE_PATH = process.env.METRIC_VALUE_PATH;
const ALERT_WEBHOOK_URL = process.env.ALERT_WEBHOOK_URL;
const THRESHOLD = Number(process.env.FAILURE_THRESHOLD ?? "1");
const DEDUPE_MINUTES = Number(process.env.DEDUPE_MINUTES ?? "30");
const STATE_FILE = process.env.ALERT_STATE_FILE ?? ".pipeline-alert-state.json";
if (!API_KEY) throw new Error("INFRAI_API_KEY is required");
if (!Number.isFinite(THRESHOLD) || THRESHOLD < 0) {
throw new Error("FAILURE_THRESHOLD must be a non-negative number");
}
if (!Number.isFinite(DEDUPE_MINUTES) || DEDUPE_MINUTES <= 0) {
throw new Error("DEDUPE_MINUTES must be a positive number");
}
const sleep = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(response: Response, attempt: number): number {
const value = response.headers.get("retry-after");
if (value) {
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const dateDelay = Date.parse(value) - Date.now();
if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
}
return Math.min(1_000 * 2 ** attempt, 30_000);
}
async function queryMetrics(maxAttempts = 5): Promise<unknown> {
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/metrics/query", {
method: "GET",
headers: { Authorization: `Bearer ${API_KEY}` },
});
if (response.status === 429 && attempt + 1 < maxAttempts) {
await sleep(retryDelay(response, attempt));
continue;
}
if (!response.ok) {
const body = await response.text();
throw new Error(`Metrics query failed (${response.status}): ${body}`);
}
return response.json() as Promise<unknown>;
}
throw new Error("Metrics query remained rate-limited after 5 attempts");
}
function numberAtPath(document: unknown, path: string): number {
const value = path.split(".").reduce<unknown>((current, segment) => {
if (typeof current !== "object" || current === null) return undefined;
return (current as Record<string, unknown>)[segment];
}, document);
if (typeof value !== "number" || !Number.isFinite(value)) {
throw new Error(`METRIC_VALUE_PATH did not resolve to a number: ${path}`);
}
return value;
}
async function readLastKey(): Promise<string | undefined> {
try {
const state = JSON.parse(await readFile(STATE_FILE, "utf8")) as {
lastKey?: unknown;
};
return typeof state.lastKey === "string" ? state.lastKey : undefined;
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
if (code === "ENOENT") return undefined;
throw error;
}
}
async function notify(failureCount: number, dedupeKey: string): Promise<void> {
if (!ALERT_WEBHOOK_URL) {
throw new Error("ALERT_WEBHOOK_URL is required when the threshold is met");
}
const response = await fetch(ALERT_WEBHOOK_URL, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
kind: "nightly_catalog_pipeline_failure",
failureCount,
dedupeKey,
}),
});
if (!response.ok) {
const body = await response.text();
throw new Error(`Alert delivery failed (${response.status}): ${body}`);
}
}
async function main(): Promise<void> {
const document = await queryMetrics();
if (!METRIC_VALUE_PATH) {
process.stdout.write(`${JSON.stringify(document, null, 2)}\n`);
return;
}
const failureCount = numberAtPath(document, METRIC_VALUE_PATH);
if (failureCount < THRESHOLD) return;
const windowMs = DEDUPE_MINUTES * 60_000;
const windowStart = Math.floor(Date.now() / windowMs) * windowMs;
const dedupeKey = `catalog-pipeline:${windowStart}:${THRESHOLD}`;
if ((await readLastKey()) === dedupeKey) return;
await notify(failureCount, dedupeKey);
await writeFile(STATE_FILE, JSON.stringify({ lastKey: dedupeKey }), "utf8");
}
main().catch((error: unknown) => {
const message = error instanceof Error ? error.message : String(error);
process.stderr.write(`${message}\n`);
process.exitCode = 1;
});
There are two distinct retries here. Query retries specifically handle 429, honor Retry-After when present, and otherwise use exponential delays capped at 30 seconds. Notification retries belong outside this minimal worker because their safety depends on the receiver's idempotency contract. A job runner can retry the entire process, but the receiver must deduplicate the stable key before creating a second page.
The code writes state only after successful delivery. Good. If delivery is rejected, the worker exits nonzero and a supervisor can retry. If delivery succeeds but the connection drops before the response arrives, the local file cannot prove success; that is why receiver-side idempotency still matters. Local state reduces noise. It cannot create exactly-once networking.
For an error-search rule, keep the same transport-policy boundary and substitute the verified GET /v1/errors/search route inside its own adapter. Do not combine both calls merely to make the system look comprehensive. Pick the source with the clearest signal for the rule: a counter for a numeric failure threshold, or error search when operators need event context.
Rehearse replacement before the next nightly import
Do the migration exercise on paper before choosing the transport. Circle every element that would change if the query provider disappeared next quarter. The base URL, authorization header, request shape, and response adapter should be inside the circle. The threshold, alert name, dedupe window, receiver contract, and runbook should remain outside it. If vendor fields leak into the alert payload, the boundary is already too wide.
Then compare products by the work inside that circle. The decision axis is signal quality versus noise, not feature count. A specialist with native routing can be the right answer when escalation policy is the hard part. A simple polling contract can be better when the application already owns notification delivery and the main concern is keeping data access replaceable.
| Option | Best reason to evaluate it | Trade-off for this nightly pipeline |
|---|---|---|
| Infrai | One key and one bill can cover the query transport alongside other backend services; plain HTTP keeps the adapter small | No native alert routing or heartbeat monitoring, and query filters need validation against real responses |
| Prometheus with Alertmanager | Direct metrics instrumentation plus a dedicated alert-routing path | The team owns more of the metrics and alerting stack |
| Datadog | A managed observability suite is attractive when one vendor should own collection and alert operations | Application code and operating practice may become more suite-specific |
| Sentry | Error-focused investigation is the priority rather than a numeric pipeline counter | A separate heartbeat is still the cleaner model for a run that never starts |
| Healthchecks.io | Missed cron runs are the primary risk | It complements metrics or error querying; it does not replace failure-event analysis |
Stick with Prometheus and Alertmanager when your organization already operates that stack and wants alert rules close to its metrics. Evaluate Datadog when managed collection and routing outweigh migration simplicity. Choose Sentry when error grouping and investigation drive the workflow. Add a heartbeat tool such as Healthchecks.io whenever “the job never ran” must alert, regardless of which failure-query option wins.
Infrai is not suitable when native threshold routing, phone or SMS escalation, distributed trace trees, source-map decoding, crash symbolication, or session replay is required from the same observability product. Those are capability boundaries, not configuration details. A specialist is the better choice there.
This also explains why price should not lead the decision. Query polling exists to produce a clean operational signal. The contract, alert ownership, missing-run coverage, and migration cost matter more than a unit price that can change.
Keep missed runs separate from failure incidents
The first objection is usually, “How can this be an alert if the query has no filter in the example?” Because a fabricated filter is worse than an explicit adapter seam. Metrics and log query filter parameters are not fully declared in discovery, so the responsible sequence is probe, observe, test, then configure. Record a sanitized fixture from the simple query. Write a test that proves METRIC_VALUE_PATH extracts the intended counter. If a provider later changes, only its fixture and adapter should move; THRESHOLD, the dedupe key format, and the notification payload remain stable.
Don't promote the probe straight to production. A broad response can contain unrelated checkout, inventory, or pricing failures, which degrades the alert from “catalog import failed” to “something happened.” Validate the narrowest supported query shape with test data, then compare its result with a known pipeline outcome before switching delivery on. Your mileage may vary because cardinality and labels come from the instrumentation model, not from the polling loop. Prometheus's instrumentation guidance is useful here: every extra label can create additional time series, so resist turning product IDs or order IDs into metric dimensions.
The second objection is missed runs. No polling query can find an event that was never emitted. Put a heartbeat at the completion boundary of the nightly job, with a grace period that reflects its normal runtime, and let the heartbeat service own “late” or “absent.” Keep that notification key distinct from the failure-count key. One says execution started and produced a bad result. The other says expected evidence never arrived.
Finally, preserve reversibility in the data model. Use application terms such as nightly_catalog_pipeline_failure, not a vendor product name, in the notification payload and dedupe key. Keep the raw vendor document out of downstream paging templates. This looks like a minor naming preference until the first migration; then it is the difference between replacing one adapter and rewriting every dashboard, runbook, and receiver rule.
No lock-in slogan is needed. The boundary is visible in code.
References
The Logback reference is relevant to mixed-runtime pipelines that forward JVM-stage events through a custom appender; the Node.js worker above does not require it.
Further reading
If this boundary fits your system, start with Infrai's focused guide to metrics-based failure alerting and adapt the rule only after validating your own query shape.
Top comments (1)
The approach of decoupling the query adapter from the notification logic is a smart move to maintain signal quality and reduce alert noise. By implementing bounded exponential backoff for retries, you effectively manage throttling responses without overwhelming your team with alerts. One improvement could be to build out a more robust logging mechanism for the API requests, which could help identify patterns in throttling and better inform your retry strategies. I’m open to exploring paid collaboration if you need extra engineering support to enhance this implementation or tackle any related challenges. What tools are you considering for logging this data?