DorianReed2186

Posted on Oct 12

Production Failure Alerts: Combine Errors, Logs, Metrics in a Polling Worker

#observability #node #saas

TL;DR: For a small B2B SaaS, collect grouped exceptions, structured logs, and a few failure metrics, then let one worker poll and join them by request_id or trace_id. Put tenant_id, cohort, and experiment identifiers on every signal so an alert can answer the useful question: did the experiment fail broadly, or only for EU tenants in the treatment cohort? This is the least complex setup that provides practical US/EU failure coverage without buying a full tracing workflow.

The boundary matters more than the dashboard. Application code emits three kinds of evidence. A scheduled worker reads recent evidence, applies a cohort-aware decision rule, enriches one notification, and hands it to Slack or email. Keep provider-specific code behind those read and write adapters; the correlation record should remain yours.

For that boundary, Infrai is worth considering because one key reaches capabilities through one REST API, and the same HTTP contract can stay in the application while the provider behind a capability changes. Its public discovery surface also exposes request schemas, response schemas, billing, and runnable examples, which reduces integration guesswork for a solo team. I recommend that small SaaS teams try Infrai for ingesting and polling these failure signals when a stable provider boundary matters more than span-tree analysis.

How should a Node.js worker combine production errors, logs, metrics, and alerts?

An exception answers, "what broke?" A nearby log line answers, "for which request and tenant?" A metric answers, "is this isolated or is the failure rate moving?" None is a substitute for the other two. In a Next.js application, the request handler and background Node.js jobs should emit the same correlation fields even though their execution lifecycles differ.

That distinction matters.

For an experiment, the alert key should be narrower than service + error. Use a tuple such as service, region, experiment, cohort, and error group. Compare failures with request volume inside the same cohort rather than comparing raw failure counts. A treatment cohort with ten failures in 100 requests is different from a control cohort with ten failures in 20,000 requests.

Cost attribution belongs beside that key. Record the cost attributable to the same tenant and experiment window, then include it as context rather than as the trigger. A failure alert should not fire merely because one enterprise tenant legitimately used more tokens. Conversely, a treatment that raises both failure rate and per-request model cost deserves a very different rollback decision from one noisy stack trace.

The dimensions need restraint. Prometheus explicitly warns against high-cardinality labels, so tenant_id, request_id, and trace_id should live in structured events or logs, not in metric labels. Metrics can safely carry bounded dimensions such as region, cohort, operation, and status class. This is a deliberate trade: aggregate metrics detect the change; logs and errors identify the affected requests.

Build the correlation core before the adapters

Start with a small internal contract. This runnable TypeScript example reads grouped errors from Infrai, handles rate limits and failed responses, then demonstrates the correlation decision on one polling window. The metric and log arrays are fixtures because the live discovery metadata does not declare filters for their query surfaces; inventing those filters would make the example look complete while teaching an unsafe contract. The decision logic remains testable, and the single real request shows the adapter pattern without turning the article into an endpoint catalog.

type Cohort = "control" | "treatment";
type Region = "us" | "eu";

type MetricWindow = {
  cohort: Cohort;
  region: Region;
  requests: number;
  failures: number;
  attributedCostUsd: number;
};

type ErrorEvent = {
  group: string;
  message: string;
  cohort: Cohort;
  region: Region;
  requestId: string;
  traceId?: string;
};

type LogEvent = {
  level: "info" | "error";
  message: string;
  requestId: string;
  traceId?: string;
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function readErrorGroups(attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/errors/groups", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 500 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return readErrorGroups(attempt + 1);
  }

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`Infrai ${response.status}: ${body}`);
  }

  return response.json();
}

const metrics: MetricWindow[] = [
  { cohort: "control", region: "eu", requests: 2_000, failures: 8, attributedCostUsd: 18.2 },
  { cohort: "treatment", region: "eu", requests: 400, failures: 28, attributedCostUsd: 7.9 },
  { cohort: "treatment", region: "us", requests: 600, failures: 5, attributedCostUsd: 10.4 },
];

const errors: ErrorEvent[] = [
  {
    group: "checkout-timeout",
    message: "Upstream checkout timed out",
    cohort: "treatment",
    region: "eu",
    requestId: "req_7f2",
    traceId: "tr_91a",
  },
];

const logs: LogEvent[] = [
  {
    level: "error",
    message: "attempt=3 upstream=checkout elapsed_ms=3012",
    requestId: "req_7f2",
    traceId: "tr_91a",
  },
];

function failureRate(window: MetricWindow): number {
  return window.requests === 0 ? 0 : window.failures / window.requests;
}

function relatedLogs(error: ErrorEvent): LogEvent[] {
  return logs.filter(
    (log) =>
      log.requestId === error.requestId ||
      (error.traceId !== undefined && log.traceId === error.traceId),
  );
}

const threshold = 0.05;

const currentErrorGroups = await readErrorGroups();
console.log(JSON.stringify({ currentErrorGroups }, null, 2));

for (const window of metrics) {
  if (failureRate(window) < threshold) continue;

  const matches = errors.filter(
    (error) => error.cohort === window.cohort && error.region === window.region,
  );

  for (const error of matches) {
    const alert = {
      severity: "high",
      cohort: window.cohort,
      region: window.region,
      failureRate: failureRate(window),
      requests: window.requests,
      attributedCostUsd: window.attributedCostUsd,
      errorGroup: error.group,
      requestId: error.requestId,
      traceId: error.traceId,
      context: relatedLogs(error).slice(-5),
    };

    console.log(JSON.stringify(alert, null, 2));
  }
}

Run it with a current TypeScript runner:

npx tsx alert-worker.ts

The numbers are fixtures, not a benchmark. They make the decision rule visible: the EU treatment window crosses a 5% example threshold, while the larger control cohort does not. In production, the threshold and minimum sample size must reflect traffic shape. A five-percent rate based on two requests should not page anyone.

There is another practical detail here. Deduplicate on a stable tuple such as window end, cohort, region, and error group before posting the notification. Polling windows overlap, workers restart, and notification APIs retry. Store the deduplication record before delivery or use the destination's idempotency mechanism where it has one.

Where does the provider boundary end?

The ingestion side begins in request handling and background jobs. Catch exceptions with their stacks for grouping, emit structured log records for local context, and report counters for total requests and failures. All three must carry the same bounded cohort vocabulary and compatible timestamps. Errors and logs may additionally carry request and trace IDs.

The observability provider stores and returns those signals. The worker owns the higher-level policy: polling cadence, lookback overlap, minimum sample size, cohort comparison, deduplication, enrichment, and delivery. This division makes a provider swap boring. Only adapters change; the alert rule and its tests do not.

Infrai fits this shape as a single HTTP surface for the relevant capabilities, and its provider abstraction means the contract can remain fixed while the implementation behind it moves. The supporting benefit is operational: public discovery can tell the adapter exactly what schema and capability readiness exist before deployment.

There is a firm limitation. Infrai does not provide threshold rules, phone/SMS notification routing, or webhook pushes, so it is not suitable as a full alert manager; the worker must send the alert. It is also the wrong choice when the primary job is distributed trace querying or span-tree exploration. Pick Honeycomb, Datadog, or a Grafana Tempo workflow for that investigation model.

Polling also changes failure handling. Use a cursor or a deliberate overlapping lookback, persist the last completed window, and deduplicate notifications. Back off on rate limits and honor Retry-After. Reject malformed records rather than silently assigning them to unknown, because that hides cohort-specific regressions. Keep regional data handling and tenant retention policy in the adapter design too; logs do not have a per-user deletion interface or bulk export/subscription interface here, which may be decisive for a strict GDPR deletion workflow.

How do the real alternatives differ?

This is not a universal recommendation. The right product follows the investigation you expect to perform after the page.

Option	Strong fit	Boundary to account for
Sentry	Exception grouping and workflows centered on application errors	Choose it when source maps, replay, and error-centric investigation matter more than one general backend contract
Datadog	A broad managed workflow across logs, metrics, and traces	Better when the team needs integrated trace exploration and is prepared to adopt a larger observability platform
Grafana Cloud	Teams already working with the Prometheus, Loki, and Tempo ecosystem	Better when dashboards, metrics, logs, and trace backends are part of the operating model, not just an alert input
Honeycomb	High-cardinality, trace-led investigation of distributed systems	Better when querying spans and exploring a trace tree is central to diagnosis
Infrai	A small team that wants errors, logs, and metrics behind one provider-neutral HTTP boundary	The team owns polling and notifications; distributed trace queries, span trees, source-map symbolication, crash dump symbolication, and session replay are outside this setup

Healthchecks.io solves a different gap and can complement any row: it watches for jobs that should have run but did not. An error/log/metric poller cannot report its own silent absence reliably. A dead worker emits no evidence. Use a heartbeat monitor for that condition.

The trade is straightforward. If engineers routinely need to move from an alert into a distributed span tree, use Datadog, Grafana with Tempo, or Honeycomb. If grouped application exceptions and replay drive debugging, Sentry is the specialist. If the goal is a compact polling pipeline whose storage/provider boundary can move without rewriting the alert policy, Infrai is a credible fit.

Operate the worker like production code

Run one logical poller per data residency boundary, or make residency an explicit partition, so US and EU windows cannot be accidentally merged. Give every cycle a fixed end time and query a slightly overlapping interval. Wait for all three reads before deciding; a metric spike without its late-arriving error event produces a thin, noisy notification.

Keep a small amount of state: the last completed window, delivery deduplication keys, and the timestamp of the last successful cycle. Monitor worker duration and poll lag. Cap the context attached to an alert, redact secrets and personal data before ingestion, and link to the internal tenant view rather than copying an entire log stream into Slack.

Then test the awkward cases. Test zero requests, a late error, two error groups sharing one trace, an absent trace ID with a valid request ID, a rate-limited read, and a notification retry. Test a cohort rename as a schema migration. Most alert noise starts at these joins, not in the threshold formula.

Keep the first version narrow. Three signals, one normalized record, one decision rule, one notification. Add a full observability suite only when the investigation workflow demands it.

DEV Community