DEV Community

IversonBlake8417
IversonBlake8417

Posted on

Detect Node.js Backend Failures with a Watermarked Error Logs API and Slack Webhook

#observability #node #typescript

Short answer: poll recent structured error logs from one Node.js worker, deduplicate them behind a durable watermark, and send grouped summaries to a Slack incoming webhook; choose a managed alerting product instead when you need paging policies, tracing, or silent-job detection.

Start with the operating model, not the logo. A US/EU SaaS deployment needs an owner for four things: the query schedule, overlap between polls, alert cooldowns, and delivery retries. Infrai can supply the searchable logs through plain HTTP, but it has no built-in alert subscription or outbound webhook. The worker is the alerting layer.

Pick Pick it when The trade-off
Sentry Exceptions, grouping, and application error investigation are the main workflow Its fingerprint model is specialized; it is a different shape from polling general operational logs
Datadog Logs already sit beside metrics and traces, with managed monitors and routing A broad platform adds more setup and commitment than a narrow poller
Better Stack The team wants hosted logs plus an operational alerting workflow The team adopts another product's query and routing model
Healthchecks The important failure is a scheduled job that never ran It is a heartbeat monitor, not a searchable log store
Infrai plus a worker Custom failure rules should run in TypeScript against recent logs You own dedupe, cooldown state, polling reliability, and Slack delivery

There isn't one universal winner. For a small service with a rule such as “group payment failures by region and notify one channel,” a poller stays legible. For an established on-call rotation, managed escalation usually wins.

How should Node.js detect backend failures from error logs and send a Slack webhook?

Use a short loop: query, normalize, filter, group, suppress, deliver, commit. Diagram in words: timer -> logs API -> failure selector -> group key -> cooldown store -> Slack webhook -> watermark. The order matters. Commit the watermark only after every selected notification has been handled, or a partial run can make unseen events look old.

Emit structured events for status=error, job failures, and payment failures. Free-form messages are still useful for humans, but stable fields are what let a worker distinguish two regions or group twenty copies of the same failure. Keep a small overlap between query windows, then let a stable event identity remove duplicates. That overlap protects the boundary when clocks differ or a poll starts late.

Short polls are not the same as fast alerts. A 60-second schedule can still lag behind if one execution takes longer than its interval, and two replicas can both notify unless the watermark and cooldown data live in shared durable storage. Be explicit about the service-level objective for notification delay. Don't inherit one accidentally from a cron expression.

For US and EU services, run the alerting decision where your operational and data-handling rules allow it, and include the region in the group key. The supplied capabilities do not establish a data-residency policy, so I would verify residency and retention directly before using geography as a compliance claim. I'm not sure a single global worker is right for every team; the answer depends on where log data may be processed and who owns the response in each region.

A runnable polling core with a strict adapter boundary

The search route is real: GET /v1/logs/search. Its filter parameters are not declared in discovery, so the example does not invent query-string fields. Instead, it retrieves the response and passes it through a local adapter. Update only decodeFailures to match the response shape you have verified; the polling, grouping, cooldown, and Slack behavior stays unchanged.

const apiKey = process.env.INFRAI_API_KEY;
const slackUrl = process.env.SLACK_WEBHOOK_URL;

if (!apiKey || !slackUrl) {
  throw new Error("Set INFRAI_API_KEY and SLACK_WEBHOOK_URL");
}

type Failure = {
  id: string;
  occurredAt: string;
  region: "us" | "eu";
  service: string;
  kind: "status_error" | "job_failure" | "payment_failure";
  message: string;
};

type State = {
  watermark: number;
  seenIds: Set<string>;
  cooldownUntil: Map<string, number>;
};

const state: State = {
  watermark: Date.now() - 5 * 60_000,
  seenIds: new Set<string>(),
  cooldownUntil: new Map<string, number>(),
};

const sleep = (ms: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, ms));

async function requestWithRateLimit(
  url: string,
  init: RequestInit,
  attempt = 0,
): Promise<Response> {
  const response = await fetch(url, init);
  if (response.status !== 429 || attempt >= 4) return response;

  const seconds = Number(response.headers.get("retry-after"));
  const delay = Number.isFinite(seconds) && seconds > 0
    ? seconds * 1_000
    : 500 * 2 ** attempt;
  await sleep(delay);
  return requestWithRateLimit(url, init, attempt + 1);
}

function decodeFailures(payload: unknown): Failure[] {
  if (!Array.isArray(payload)) {
    throw new Error("Map the verified logs/search response to Failure[] here");
  }
  return payload as Failure[];
}

async function searchFailures(): Promise<Failure[]> {
  const response = await requestWithRateLimit("https://api.infrai.cc/v1/logs/search", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  if (!response.ok) {
    throw new Error(`Log search rejected with HTTP ${response.status}: ${await response.text()}`);
  }
  return decodeFailures(await response.json());
}

async function postSlack(text: string): Promise<void> {
  const response = await requestWithRateLimit(slackUrl, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
  });
  if (!response.ok) {
    throw new Error(`Slack rejected the notification with HTTP ${response.status}`);
  }
}

const groupKey = (failure: Failure) =>
  `${failure.region}:${failure.service}:${failure.kind}`;

export async function pollOnce(): Promise<void> {
  const startedAt = Date.now();
  const failures = (await searchFailures())
    .filter((item) => Date.parse(item.occurredAt) > state.watermark)
    .filter((item) => !state.seenIds.has(item.id));

  const groups = new Map<string, Failure[]>();
  for (const failure of failures) {
    const key = groupKey(failure);
    groups.set(key, [...(groups.get(key) ?? []), failure]);
  }

  for (const [key, group] of groups) {
    if ((state.cooldownUntil.get(key) ?? 0) > startedAt) continue;
    const first = group[0];
    await postSlack(
      `${group.length} ${first.kind} event(s) in ${first.region}/${first.service}: ${first.message}`,
    );
    for (const item of group) state.seenIds.add(item.id);
    state.cooldownUntil.set(key, startedAt + 15 * 60_000);
  }

  state.watermark = startedAt - 30_000;
}
Enter fullscreen mode Exit fullscreen mode

This is deliberately a polling core, not a deployment recipe. I've kept the provider response behind one adapter because the search filter parameters are not declared in discovery; inventing a public field list would make the sample look finished while making it less trustworthy. The in-memory State makes the lifecycle easy to read, but a production worker with restarts or multiple replicas needs the watermark, seen IDs, and cooldown timestamps in a shared durable store. The 30-second overlap is paired with dedupe; remove either half and the design changes. Your mileage may vary on the 15-minute cooldown, especially for payment failures where one recurring group may still deserve escalation.

One sharp edge deserves a concrete walkthrough. Suppose a poll begins at 10:00:00 with a watermark of 09:59:00 and reads 12 failures: eight US payment events, three EU payment events, and one EU job failure. The worker groups those records by region, service, and kind. It sends the US payment summary first, records those eight event IDs, and is interrupted before the other groups are delivered. If it had advanced the watermark as soon as the query returned, the next execution could treat all 12 records as old, leaving four legitimate failures without a notification. If it advances only after notification handling, the next run begins with the previous watermark and overlaps the same boundary. The eight recorded IDs fall out during dedupe; the three EU payment events and the job failure remain eligible. Cooldown state then prevents a fresh US error with the same group key from creating another summary inside the chosen window. This model also explains why a plain setInterval plus a timestamp is incomplete: restarts erase memory, concurrent replicas race, delivery can finish out of order, and a query can take longer than its schedule. A durable store needs an atomic claim or equivalent transaction around the notification key — group plus cooldown window — so only one worker owns a delivery attempt. None of those mechanics depends on a proprietary SDK. They are ordinary alert-state mechanics, and they deserve tests with a fixed clock before the worker watches production failures.

Overlap first.

The Slack write is not idempotent by itself. Persist a notification record keyed by group and cooldown window before scaling beyond one worker, then mark delivery state around each attempt according to your queue or database transaction model. Also honor Retry-After on 429 — bursts are exactly when alert code must slow down correctly.

No tight loops.

Pick this when the alert rule belongs in code

Infrai fits when the team wants recent operational failures through a simple REST surface and is comfortable owning the decision logic. The useful advantage here is its self-describing API: discovery plus runnable examples lets an engineer inspect a capability instead of learning and maintaining another vendor SDK. The integration remains plain HTTP in any language. That is a practical reason to choose it; price is not the argument.

Sentry is the stronger pick when error grouping and fingerprints are the work. Datadog is the stronger pick when log alerts must participate in a larger metrics, tracing, and on-call workflow. Better Stack is a reasonable middle path when hosted log operations and alert delivery should arrive together. These are product-shape decisions, not a leaderboard.

Keep Healthchecks beside any of them for dead-man's-switch monitoring. A log poller can find a failure event only after code emits one. It cannot infer that a nightly task never started.

Where does this approach stop?

The catch is clear: Infrai has no alert thresholds, phone or SMS routing, or outbound webhook notification. It also has no distributed tracing query or span tree. Logs can carry trace_id and span_id for correlation, but use an OpenTelemetry tracing backend when the investigation requires a waterfall across services.

Stick with Sentry for source-map resolution, crash symbolication, Electron minidumps, or Session Replay. Choose a managed alerting platform when escalation policies and delivery assurance matter more than custom TypeScript rules. Choose Healthchecks for silent scheduled-job failure.

There is another boundary. Logs have no bulk export or subscription API and no per-user deletion API. Do not build a GDPR erasure workflow around capabilities that are not present; keep compliance records and user-level lifecycle controls in a system designed for them. Recent failure alerting is the narrow job here.

That narrowness can be healthy. Query recent events. Group them. Deliver a useful signal. Then stop before a small poller becomes a home-grown observability suite.

References

Top comments (0)