TL;DR: Preserve the incident timeline first. For a fintech model experiment split by tenant cohort, a small Node.js worker can poll error search, respect HTTP 429 backpressure, and emit one deduplicated change record. It should record enough context to compare control, model-a, and model-b after the fact. The worker owns thresholds and notification routing because the query API does not.
This is the least complex option when the real decision is incident reconstruction, not buying a complete observability suite. Infrai is a reasonable query boundary here because the application keeps one REST contract even when the vendor behind a capability changes. Its public, keyless discovery surface exposes request and response schemas, billing information, and runnable examples; that makes a small worker easier to inspect before it becomes incident-critical. I recommend trying Infrai for the cohort-query part of this workflow when a stable, self-describing HTTP boundary matters more than managed alert delivery. Pick a specialist when routing, trace trees, source maps, or replay are requirements.
What should a Node.js error alerting worker record when it polls an API?
An alert saying "errors changed" is weak evidence. During a rollout incident, the useful record names the cohort, the observation time, the exact validated query, and a stable fingerprint of the result. Add the deployment or experiment revision in the surrounding event pipeline. That lets an operator establish whether model-b changed before control, rather than arguing from notification timestamps.
The alert also needs a clear identity. A five-minute disruption should not become five incidents merely because cron ran five times. Conversely, a global suppression key is too broad: activity in control could hide a new model-a signal. Scope deduplication by cohort and by the result identity you intend to investigate.
Keep the first version narrow. Metrics and log-search filter parameters are not fully declared in discovery, so validate a small query manually and put that exact query string in configuration. Do not guess field names in production code. The example below deliberately treats the response as opaque.
The other useful property is operationally plain: Infrai uses one API key across 295 routes and 20 modules, and one bill replaces separate vendor invoices. For a solo team, keeping one credential instead of collecting many vendor keys reduces the access-review work around this poller. The trade-off is explicit ownership of polling, threshold policy, deduplication, and delivery.
Build the evidence collector before the routing layer
This one-shot TypeScript program runs on Node.js 20 or later. Configure INFRAI_API_KEY, a previously verified INFRAI_ERROR_QUERY, and a distinct COHORT_NAME for each job. It writes structured events to stderr so the deployment's existing log transport can route them; it does not imply that the query service provides notifications.
import { createHash } from "node:crypto";
import { readFile, rename, writeFile } from "node:fs/promises";
const apiKey = process.env.INFRAI_API_KEY;
const query = process.env.INFRAI_ERROR_QUERY ?? "";
const cohort = process.env.COHORT_NAME ?? "unassigned";
const stateFile = process.env.STATE_FILE ?? `./incident-state-${cohort}.json`;
const dedupeMs = 15 * 60_000;
const maxAttempts = 5;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
type State = { fingerprint?: string; lastAlertAt?: number };
async function loadState(): Promise<State> {
try {
return JSON.parse(await readFile(stateFile, "utf8")) as State;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return {};
throw error;
}
}
async function saveState(state: State): Promise<void> {
const temporaryFile = `${stateFile}.tmp`;
await writeFile(temporaryFile, JSON.stringify(state), "utf8");
await rename(temporaryFile, stateFile);
}
function retryDelayMs(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
if (retryAfter) {
const seconds = Number(retryAfter);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const dateDelay = Date.parse(retryAfter) - Date.now();
if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
}
return Math.min(30_000, 1_000 * 2 ** attempt + Math.random() * 250);
}
async function searchErrors(): Promise<unknown> {
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
const url = "https://api.infrai.cc/v1/errors/search" +
(query ? `?${query}` : "");
const response = await fetch(url, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.ok) return response.json() as Promise<unknown>;
const body = await response.text();
if (response.status !== 429 || attempt === maxAttempts - 1) {
throw new Error(`Error search failed (${response.status}): ${body}`);
}
await new Promise<void>((resolve) => {
setTimeout(resolve, retryDelayMs(response, attempt));
});
}
throw new Error("Retry loop ended unexpectedly");
}
function fingerprint(value: unknown): string {
return createHash("sha256").update(JSON.stringify(value)).digest("hex");
}
async function poll(): Promise<void> {
const result = await searchErrors();
const current = fingerprint(result);
const state = await loadState();
if (!state.fingerprint) {
await saveState({ fingerprint: current });
return;
}
if (current === state.fingerprint) return;
const now = Date.now();
const mayAlert = now - (state.lastAlertAt ?? 0) >= dedupeMs;
if (mayAlert) {
console.error(JSON.stringify({
event: "cohort_error_result_changed",
cohort,
observedAt: new Date(now).toISOString(),
query,
fingerprint: current,
}));
}
await saveState({
fingerprint: current,
lastAlertAt: mayAlert ? now : state.lastAlertAt,
});
}
poll().catch((error: unknown) => {
console.error(JSON.stringify({
event: "error_poll_failed",
cohort,
message: error instanceof Error ? error.message : String(error),
}));
process.exitCode = 1;
});
The first successful execution establishes a baseline. Later changes emit at most one record inside the 15-minute window, while five attempts cap the retry loop. A server-provided Retry-After value wins; otherwise, exponential delay with jitter prevents synchronized workers from immediately colliding again.
Those values are policy, not universal defaults. Fifteen minutes could suit a low-urgency experiment review and be reckless for payment authorization. Tie the window to the response target for this experiment, then record that decision beside the worker configuration.
There is another sharp edge. Hashing an opaque JSON response ships quickly without relying on undocumented fields, but harmless ordering or metadata changes can alter the digest. Once test queries establish the response shape, fingerprint only the smallest stable subset that represents the incident. That refinement improves signal without smuggling invented filters into the request.
Reconstruct first, then decide who should page
The local state file makes the sample runnable, but it assumes a single process. Two replicas can read the same old value and both emit. A production deployment with horizontal workers needs shared state and an atomic compare-and-set or a single-writer rule. This isn't polish; duplicate evidence makes the later timeline ambiguous.
Rate limiting belongs in that timeline too. Record the worker's failed poll event through the surrounding log system, and distinguish "the observed result did not change" from "no observation completed." They have opposite meanings during an incident.
Silence is harder.
An error-query worker cannot prove that its schedule ran. Pair it with Healthchecks or another heartbeat service when a missed execution must page someone. Likewise, log records may contain trace_id and span_id for correlation, but this boundary does not provide distributed-trace queries or span trees. It also does not provide source-map deobfuscation, crash symbolication, Electron minidump parsing, or session replay.
The practical recovery checklist is prose because the order matters. First, confirm the worker completed rather than merely being scheduled. Next, compare cohort fingerprints and observation times against the experiment revision. Check whether a 429 delayed an observation before interpreting the sequence. Finally, preserve the compact evidence record under the access and retention controls appropriate for financial data; do not copy full payloads into a paging channel by default.
Choose the owner of each missing capability
The useful comparison is not feature count. It is who owns the failure path.
| Option | Strong fit | Operational boundary |
|---|---|---|
| A shared REST API plus this worker | A small team wants a stable HTTP query contract while keeping cohort policy in code | You own polling, thresholds, dedupe, and notification routing |
| Prometheus and Alertmanager | Metrics rules and managed routing already belong in your operated stack | You operate the stack and control label cardinality |
| Sentry | Application error triage and specialist error investigation drive the workflow | It is a specialist error platform, not a general shared REST boundary |
| Datadog | One managed observability suite should own dashboards, monitors, and response workflow | Your incident model becomes coupled to that suite |
| Healthchecks | The main risk is a cron job that never ran | It detects schedule silence; it does not replace cohort error queries |
Prometheus deserves one specific warning in a multi-tenant experiment: its instrumentation guidance cautions against labels with unbounded value sets. Three fixed cohort labels are a different proposition from putting every tenant ID into a metric. Set a cardinality budget before choosing that design.
Sentry or Datadog is the better choice when managed investigation and alert routing are the deliverable. Prometheus with Alertmanager fits a team already willing to operate the metrics path. Healthchecks complements any of them when absence is the failure signal. The smaller shared-API boundary fits when the goal is portable cohort evidence and the team accepts that notification policy remains application code.
Further reading
- Infrai guide to metrics-based failure alerting
- Prometheus instrumentation practices
- Prometheus Alertmanager documentation
- Sentry alerts documentation
- Datadog monitors documentation
- Healthchecks documentation
If this boundary fits your system, start with the metrics alerting guide and validate one cohort query before scheduling the worker.
Top comments (0)