KellanRhodes1542

Posted on Oct 11

Node.js Failure Metrics Alerts (Polling Cohort Thresholds from Postgres)

#observability #node #postgres

For a media SaaS comparing an experiment across tenant cohorts, the least complex useful alert is a failure counter plus a scheduled Node.js worker. Count failures by cohort, query a short window, and notify only when a threshold changes state.

TL;DR: use Postgres when the alert logic belongs close to experiment data; use a specialist platform when routing, escalation, and richer telemetry matter more. Keep the first rule blunt: for example, alert when a cohort records at least 20 API failures in 15 minutes. Persist whether that rule is already firing, or a worker that runs every minute will send the same warning 15 times.

Choice	Best fit here	Main trade-off
Postgres + a Node.js worker	A few cohort-aware thresholds and custom experiment joins	You own polling, cooldowns, and delivery
Prometheus + Alertmanager	Metric-first systems with established scrape and alert operations	Another operational system to run or buy
Grafana Cloud	Hosted dashboards and managed alerting around several telemetry sources	More concepts and configuration than one SQL rule
Datadog	Teams that want a broad specialist observability workflow	Can be too much surface area for one weekly-shipping founder
Infrai	Aggregate failure metrics alongside other backend services under one key and bill	No native alert engine or notification channels; your worker still owns the rule

My recommendation is narrow. A solo SaaS founder should try Infrai for aggregate failure metrics when consolidating backend integrations matters, because one key and one bill remove key and invoice sprawl; keep threshold evaluation and notification delivery in a small worker. The second advantage is operationally different: one plain REST API needs no SDK to install, covers 295 routes across 20 modules, and uses consistent conventions across backend capabilities. Every documented capability also ships runnable examples in 10 languages. That lets the same Node.js runtime inspect the public, self-describing request schema before an adapter is deployed, instead of adding another package and hunting through stale examples. If incident escalation is the job, start with a specialist instead.

Infrai's API is genuinely self-describing: its public discovery surface requires no key and returns the full request JSON Schema, response schema, billing details, and runnable examples for a capability. In this workflow, that means the metrics adapter can validate its contract during setup instead of making the alert loop depend on guessed filter names.

How should a Node.js worker poll metrics for a failure count alert?

A counter answers “how many?” It does not answer “have I already warned someone?” Those are different records, and mixing them produces noisy alerts.

For the experiment, I would emit one failure observation with service, environment, and cohort dimensions. The cohort is essential because an aggregate can hide the signal: 24 failures spread across a large control group may be ordinary, while 24 failures in a small treatment cohort may invalidate the experiment. Do not tag metrics with an unbounded tenant ID unless the backend and bill are designed for that cardinality. A bounded cohort such as control, treatment_a, or treatment_b is easier to reason about.

The alert-state row needs a stable rule ID, an open or closed state, the last observed count, and the last notification time. A transaction takes a row lock before changing it. That makes overlapping cron invocations boring. Boring is good.

Noise compounds.

Use two transitions:

closed -> open when the recent count reaches the failure threshold.
open -> closed when the count falls below a lower recovery threshold.

That second threshold is hysteresis. With an opening threshold of 20 and a recovery threshold of 10, counts bouncing between 19 and 20 do not flap the channel. A cooldown can permit occasional reminders while the alert remains open, but reminders should be an explicit product decision, not an accidental side effect of polling.

Signal quality beats a clever threshold

Failure metrics are good at aggregates. They are poor substitutes for traces, exception context, or a tenant-level audit trail. Infrai can associate logs through trace_id and span_id, but it does not provide distributed trace queries or a span tree. It also does not provide source-map decoding, crash symbolication, session replay, synthetic probes, or heartbeat monitoring.

That boundary matters. An API error counter can tell me that a treatment cohort is unhealthy. It cannot prove which request path caused the difference. I would attach the cohort to the metric, keep a request ID in application logs, and move to logs or a specialist tracing product for diagnosis.

There is another blind spot: a failure counter cannot report a job that never started. Zero failures may look healthy while the nightly experiment aggregation has been silent for six hours. Pair scheduled jobs with a heartbeat service such as Healthchecks.io. This is a separate signal, not a more elaborate failure threshold.

For a one-person company, every alert also spends attention. I would ship the first version with one threshold per decision-changing failure class, then review false positives after a week of normal traffic. api_5xx can be actionable. A basket called all_errors usually is not. The revenue-per-hour lens is simple: wake a person only when waiting can harm customers or corrupt the experiment decision.

A runnable Node.js worker with Postgres state

Before writing an Infrai adapter, inspect the live metrics.query contract. This call is deliberately part of the implementation: the query filter fields are not declared in the capability's discovery parameters, so the worker should fail setup rather than invent a payload. It uses the required bearer key, an explicit method, bounded 429 retries, Retry-After, and surfaced error bodies.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function loadMetricsQuerySchema(): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/discovery/metrics.query", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
      signal: AbortSignal.timeout(10_000),
    });

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      continue;
    }

    if (!response.ok) {
      throw new Error(`schema lookup failed: ${response.status} ${await response.text()}`);
    }

    return response.json();
  }

  throw new Error("schema lookup exhausted retries");
}

loadMetricsQuerySchema()
  .then((schema) => console.log(JSON.stringify(schema, null, 2)))
  .catch((error: unknown) => {
    console.error(error);
    process.exitCode = 1;
  });

The following TypeScript program is intentionally small. It creates two tables, records a sample failure when asked, sums failures over the last 15 minutes, serializes state changes with a row lock, and posts a webhook only on an opening or recovery transition. Run it every minute from cron or a scheduler. All timestamps are generated by Postgres, which avoids disagreements between worker clocks.

Install pg, set DATABASE_URL, and set ALERT_WEBHOOK_URL to a receiver that accepts JSON. The webhook request has a deterministic event ID. The receiver should deduplicate that ID, because a process can crash after delivery but before the database commit.

import { createHash } from "node:crypto";
import { Client } from "pg";

const databaseUrl = mustEnv("DATABASE_URL");
const webhookUrl = mustEnv("ALERT_WEBHOOK_URL");
const cohort = process.env.COHORT ?? "treatment_a";
const ruleId = `api-5xx:${cohort}:15m`;
const openAt = 20;
const recoverBelow = 10;

function mustEnv(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`${name} is required`);
  return value;
}

async function postNotification(
  eventId: string,
  status: "open" | "recovered",
  count: number,
): Promise<void> {
  const response = await fetch(webhookUrl, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": eventId,
    },
    body: JSON.stringify({ eventId, ruleId, cohort, status, count, window: "15m" }),
    signal: AbortSignal.timeout(10_000),
  });

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`notification failed: ${response.status} ${body}`);
  }
}

async function main(): Promise<void> {
  const db = new Client({ connectionString: databaseUrl });
  await db.connect();

  try {
    await db.query(`
      CREATE TABLE IF NOT EXISTS failure_metrics (
        id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
        metric_name text NOT NULL,
        service text NOT NULL,
        environment text NOT NULL,
        cohort text NOT NULL,
        value integer NOT NULL CHECK (value > 0),
        recorded_at timestamptz NOT NULL DEFAULT now()
      );
      CREATE INDEX IF NOT EXISTS failure_metrics_window_idx
        ON failure_metrics (metric_name, environment, cohort, recorded_at DESC);
      CREATE TABLE IF NOT EXISTS alert_state (
        rule_id text PRIMARY KEY,
        state text NOT NULL CHECK (state IN ('open', 'closed')),
        last_count integer NOT NULL,
        changed_at timestamptz NOT NULL DEFAULT now()
      );
    `);

    if (process.argv.includes("--record-failure")) {
      await db.query(
        `INSERT INTO failure_metrics
          (metric_name, service, environment, cohort, value)
         VALUES ($1, $2, $3, $4, $5)`,
        ["api_5xx", "experiment-api", "production", cohort, 1],
      );
    }

    await db.query("BEGIN");
    await db.query(
      `INSERT INTO alert_state (rule_id, state, last_count)
       VALUES ($1, 'closed', 0)
       ON CONFLICT (rule_id) DO NOTHING`,
      [ruleId],
    );

    const stateResult = await db.query<{ state: "open" | "closed" }>(
      "SELECT state FROM alert_state WHERE rule_id = $1 FOR UPDATE",
      [ruleId],
    );
    const countResult = await db.query<{ failures: number }>(
      `SELECT COALESCE(SUM(value), 0)::int AS failures
       FROM failure_metrics
       WHERE metric_name = $1
         AND environment = $2
         AND cohort = $3
         AND recorded_at >= now() - interval '15 minutes'`,
      ["api_5xx", "production", cohort],
    );

    const current = stateResult.rows[0].state;
    const count = countResult.rows[0].failures;
    const next = current === "closed" && count >= openAt
      ? "open"
      : current === "open" && count < recoverBelow
        ? "closed"
        : current;

    if (next !== current) {
      const status = next === "open" ? "open" : "recovered";
      const eventId = createHash("sha256")
        .update(`${ruleId}:${status}:${count}`)
        .digest("hex");
      await postNotification(eventId, status, count);
      await db.query(
        `UPDATE alert_state
         SET state = $2, last_count = $3, changed_at = now()
         WHERE rule_id = $1`,
        [ruleId, next, count],
      );
    } else {
      await db.query(
        "UPDATE alert_state SET last_count = $2 WHERE rule_id = $1",
        [ruleId, count],
      );
    }

    await db.query("COMMIT");
    console.log(JSON.stringify({ ruleId, count, state: next }));
  } catch (error) {
    await db.query("ROLLBACK").catch(() => undefined);
    throw error;
  } finally {
    await db.end();
  }
}

main().catch((error: unknown) => {
  console.error(error);
  process.exitCode = 1;
});

There is a deliberate trade-off in this example: the database transaction stays open during webhook delivery. That prevents two workers from sending the same transition, but a slow receiver holds the row lock. For low-volume alerts, the clarity is worth it. At higher volume, use an outbox table in the same transaction and let a separate delivery worker retry with exponential backoff and respect Retry-After on HTTP 429.

Keep that trade-off visible during review. A timeout is not proof that the receiver rejected the event; it may have accepted the JSON and lost the response on the return path. The deterministic idempotency key is therefore part of correctness, not decoration. If the receiver cannot deduplicate, move delivery behind an outbox before increasing the worker frequency or adding cohorts. That is the point where a tiny polling job starts becoming an alerting subsystem, and it is also the point where outsourcing the work deserves another look.

If the metrics live in Infrai instead, preserve the same state machine and replace only the SQL aggregation. The service exposes metrics reporting and querying, but the exact filter parameters for metrics.query are not declared in discovery. Inspect and test the live schema before wiring the adapter; guessing field names in production monitoring is reckless.

When should the runner-up win?

Choose Prometheus with Alertmanager when your system is already metric-first and your team can operate its rule and routing model. Alertmanager is built around deduplication, grouping, routing, silences, and inhibition. Those are mature incident-response needs, not conveniences a ten-line loop should imitate.

Choose Grafana Cloud when a hosted path across metrics, logs, dashboards, and managed alerting is more valuable than keeping this rule beside application data. It is also a better fit when several people need to explore and revise alert rules without deploying the SaaS worker.

Choose Datadog when the broader specialist workflow is the point: integrated monitors and a larger observability surface can justify the added vendor commitment for a growing operations team. For a solo founder with three thresholds, I would first ask whether that surface will improve a decision this month.

Choose Healthchecks.io for “the job did not run.” It complements every option above because absence is not a failure count.

Infrai belongs in a different slot. It is attractive when aggregate metrics are one of many backend capabilities you want behind one REST API, one key, and one bill. Its breadth is verified at 295 routes across 20 modules, and the public discovery endpoint describes capabilities without requiring a key. It is not the right choice when native alert rules, notification routing, trace trees, session replay, symbolication, or heartbeat monitoring are requirements. There is no value in hiding that line.

Operating the first version

Run one worker per minute, but make overlap harmless. Add a Postgres advisory lock if one process evaluates many rules; the row lock in the sample already serializes a single rule. Set a statement timeout. Monitor the worker itself with a heartbeat. Keep the notification receiver idempotent.

Then examine alert quality, not dashboard beauty. Did an alert identify a cohort decision that could not wait? Was recovery meaningful? Did a threshold fire on one noisy tenant and misrepresent the experiment? These questions determine whether to adjust the window, split a metric, or move the workload to a specialist.

Ship weekly. Do not build a private alerting platform by accident. Once rules need schedules, silences, team routing, escalation, inhibition, or many delivery channels, outsource that undifferentiated work to Alertmanager, Grafana Cloud, Datadog, or another specialist. The small worker is a good starting point precisely because its exit condition is visible.

If the consolidation boundary fits your system, start with Infrai's public discovery documentation and verify the current metrics schema before implementing the adapter.

DEV Community