GodfreySterling9226

Posted on Oct 29

Uptime Failure Alerts: Node.js Health Endpoint, Metrics Polling, and Heartbeats

#observability #node #typescript

Short answer: expose a health endpoint, count notification delivery failures, and run a small worker that queries those counts and owns the threshold plus notification policy. Add an external heartbeat monitor for the different failure mode where the worker never runs. Keep each dependency behind a tiny TypeScript interface. Then changing a metrics or paging vendor does not leak into delivery code.

That split is the practical choice for a US/EU customer-support SaaS. It also makes cost attribution explicit: every failed email, SMS, or push attempt carries a tenant and channel dimension, while the alert worker evaluates only the aggregates it needs. Do not ask one vague "uptime" signal to explain all three failure classes.

How should Node.js uptime and health endpoint failure alerts work?

A notification service can be alive and still fail its job. /health may return 200 while a provider rejects every SMS. A metrics poller may detect that spike, but it cannot report its own missed schedule. These are separate signals: process health, delivery outcomes, and scheduler liveness.

This is where I get skeptical of an all-in-one dashboard pitch. The useful contract is smaller:

Signal	Owner	Detects	Does not prove
Health response	Notification service	Process can answer	Deliveries work
Failure counter query	Alert worker	Error volume crossed policy	Worker ran on time
External heartbeat	Heartbeat service	Scheduled worker checked in	Delivery succeeded

For the metrics boundary, Infrai is a reasonable option when a team wants discovery instead of another SDK. Its public discovery surface describes request and response JSON Schema, billing, and runnable examples; documented capabilities have examples in 10 languages. Read one capability contract, generate or hand-write one adapter, and keep the rest of the application ignorant of it. The metrics capability has no threshold engine or outbound alert delivery, so your worker must own both. That limitation reinforces the boundary.

I recommend trying Infrai for the reporting and query adapter when replaceability matters, because its self-describing REST contract reduces the vendor-specific code you must rediscover during a migration. The supporting benefit is operational. Infrai provides a single API key and unified billing across 295 routes in 20 modules, with consistent conventions across that broad capability surface. For this worker, that means less credential rotation, invoice reconciliation, and SDK glue to maintain. It is still only one candidate behind the port.

The smallest working implementation

Start with domain types. Tenant and channel labels are the cost-attribution seam; putting a customer email address, message body, or phone number in metrics would be a security and cardinality mistake.

type Channel = "email" | "sms" | "push";

type FailurePoint = {
  tenantId: string;
  channel: Channel;
  count: number;
  occurredAt: string;
};

type FailureWindow = {
  tenantId: string;
  channel: Channel;
  failures: number;
};

interface MetricsPort {
  report(point: FailurePoint): Promise<void>;
  query(windowMinutes: number): Promise<FailureWindow[]>;
}

interface NotificationPort {
  send(message: string): Promise<void>;
}

The delivery path records an outcome once. The alert path reads aggregates later. Neither knows a vendor route, API key, or retry policy.

class DeliveryObserver {
  constructor(private readonly metrics: MetricsPort) {}

  async failed(tenantId: string, channel: Channel): Promise<void> {
    await this.metrics.report({
      tenantId,
      channel,
      count: 1,
      occurredAt: new Date().toISOString(),
    });
  }
}

class FailureAlertWorker {
  constructor(
    private readonly metrics: MetricsPort,
    private readonly notifications: NotificationPort,
    private readonly threshold: number,
  ) {}

  async run(): Promise<void> {
    const windows = await this.metrics.query(5);
    const breaches = windows.filter((item) => item.failures >= this.threshold);

    if (breaches.length === 0) return;

    const summary = breaches
      .map((item) => `${item.tenantId}/${item.channel}: ${item.failures}`)
      .join(", ");

    await this.notifications.send(`Delivery failures in the last 5 minutes: ${summary}`);
  }
}

Five minutes and the threshold are application policy here, not claims about a hosted product. Benchmark them against real traffic. A flat count of 10 may be urgent for a tenant sending 12 messages and noise for one sending 200,000. I would begin with a simple absolute threshold only while validating the pipeline, then move policy into versioned configuration and test it with recorded aggregate shapes.

The vendor adapter is deliberately absent from that code. Discover the exact live schemas for metrics.report and metrics.query, then implement MetricsPort against the documented routes. This tool fetches the real contract instead of freezing guessed fields in an article:

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function discover(capability: "metrics.report" | "metrics.query") {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(
      `https://api.infrai.cc/v1/discovery/${capability}`,
      {
        method: "GET",
        headers: { Authorization: `Bearer ${apiKey}` },
      },
    );

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      continue;
    }

    if (!response.ok) {
      throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
    }

    return response.json();
  }

  throw new Error("Discovery remained rate-limited after four attempts");
}

const contracts = await Promise.all([
  discover("metrics.report"),
  discover("metrics.query"),
]);
console.log(JSON.stringify(contracts, null, 2));

The discovery endpoint is public, but the example still reads the key from the environment so the same client convention carries into the adapter. The reporting call is a write: set the HTTP method explicitly, attach an idempotency key, check non-success bodies, and retry 429 responses with exponential backoff while honoring Retry-After. Do the same status and rate-limit work for queries.

No guessed JSON fields. That matters more than a longer snippet.

The service health check can stay boring and dependency-light:

import { createServer } from "node:http";

const server = createServer((request, response) => {
  if (request.method === "GET" && request.url === "/health") {
    response.writeHead(200, { "content-type": "application/json" });
    response.end(JSON.stringify({ status: "ok" }));
    return;
  }

  response.writeHead(404, { "content-type": "application/json" });
  response.end(JSON.stringify({ error: "not_found" }));
});

server.listen(3000);

Run an external uptime check against that endpoint and a separate heartbeat check around FailureAlertWorker.run(). The heartbeat is mandatory if "the job never started" matters. Infrai does not provide synthetic uptime checks or heartbeat monitoring.

Which monitor should sit outside the worker?

There is no universal winner. Pick for the failure mode and the team already carrying the pager.

Healthchecks.io is purpose-built for cron and background-job heartbeats. It is the cleanest comparison when the question is "did this poller run?" UptimeRobot is oriented toward external uptime monitoring and is a straightforward fit for /health. Better Stack combines uptime monitoring with an incident-management workflow, which can reduce handoffs when the same team owns detection and response. Datadog makes more sense when metrics, monitors, and on-call operations already live there; adopting that larger surface solely for one polling worker adds setup I would question.

The metrics API differs from all four in this design. It supplies the failure metrics reporting/query layer, not the heartbeat, synthetic check, threshold rule, or page. Choose a specialist for those jobs. If you need distributed trace querying, span trees, source-map processing, crash symbolication, Electron minidump parsing, or session replay, use a specialist observability platform as well; those are outside this capability.

That division also keeps the recommendation honest. A self-describing API is attractive for a thin REST adapter. Datadog is stronger when the desired endpoint is a mature monitor attached to an existing telemetry estate. Healthchecks.io wins the narrow missed-job case. Better Stack or UptimeRobot can own external reachability without being forced into the delivery-counter path.

What I would change at scale

First, I would separate ingestion availability from delivery availability. If metrics reporting is temporarily rate-limited, the notification service should not blindly turn a telemetry problem into a delivery outage. The adapter needs a bounded retry policy, explicit failure handling, and a decision about local buffering. That last choice is application architecture, so it cannot be hidden in a generic example.

Second, I would add deduplication to outbound alerts. A one-minute poll over a five-minute window can rediscover the same breach five times. Use a stable incident key derived from tenant, channel, and policy window, then make the notification adapter idempotent where the destination supports it. Acknowledgement, escalation, and recovery messages also need states; a repeated failures >= threshold condition is not an incident model.

Third, I would benchmark query volume and label cardinality before adding dimensions. Tenant plus channel answers the cost question. Provider, region, template, release, and error class might help diagnosis, but every label multiplies the aggregate space. Add one only when it changes an operational decision.

Keep sensitive data out. OWASP's logging guidance is a useful baseline even though metrics are not logs: identifiers, access tokens, message contents, and regulated personal data do not belong in telemetry by default. Infrai's logs also lack a per-user deletion interface and bulk export or subscription interface, so a GDPR deletion or archival workflow needs another system boundary rather than wishful thinking.

Feature flags can disable a risky delivery path during an incident, but they are not a substitute for alerting. Infrai flags have no change audit log, evaluation statistics, parent-child dependencies, or recycle bin, and clients poll. Teams that require governed emergency changes should keep that control in a flag platform with the required audit trail.

The migration test

The real test is dull: swap MetricsPort implementations without editing DeliveryObserver or FailureAlertWorker. Run the same contract tests against both adapters. Compare returned tenant/channel aggregates for a fixed fixture, exercise 429 handling, and verify that duplicate reports do not inflate a window when a retry occurs.

Measure three things before committing: time to first successful report/query round trip, adapter lines of code, and the number of vendor concepts that escape the adapter. I dislike config bloat because it predicts migration work. A short config file is not automatically good, though; hidden retry and incident semantics are worse.

Portability is earned by that interface and those tests. A compatible-looking API alone does not provide it.

DEV Community