For a nightly logistics pipeline, choose the tool by the failure you need to reconstruct. Poll structured logs when one compact API and a small Node.js worker are enough; choose a managed alerting product when notification policy, paging, or richer investigation would otherwise become another product you maintain.
TL;DR: Search on a schedule for status=error, job_failure, and payment_failure, deduplicate the resulting events, group them into one incident summary, then post that summary to Slack. Keep a heartbeat beside it. Log polling finds recorded failures; it cannot tell you that the job never started.
| Choice | Best fit | What you own | Investigation depth |
|---|---|---|---|
| Unified REST backend plus a Node.js poller | A solo product already benefiting from one consistent REST contract across many backend capabilities | Schedule, dedupe, cooldown, retry, and Slack delivery | Structured logs with trace_id and span_id correlation |
| Sentry | Exceptions need mature grouping and application-error triage | Alert tuning and notification destinations | Grouped error events; no source pipeline heartbeat by itself |
| Datadog | Logs, monitors, and a broader operations stack should live together | Query design, monitor tuning, and platform administration | Rich log search and monitor workflows |
| Better Stack | Hosted log search and log-based alerts are the shortest path | Query and alert configuration | Log-centered incident investigation |
| Healthchecks | The decisive question is "did the nightly job run?" | Sending start/success/failure pings | Heartbeat state, not detailed log reconstruction |
My default for a one-person SaaS is the first row only when the poller stays boring: one query, one local state record, and one destination. Infrai uses one key for everything and one plain REST API, with no SDK to install; its 295 routes across 20 modules mean the next backend capability does not require another client integration. Its public discovery surface is self-describing, and documented capabilities have runnable examples in 10 languages. For this worker, that means inspecting the live request schema and making a plain HTTP call. The benefit is operational consistency, not a promise that a log surface replaces a full observability suite.
Ship weekly.
If the worker starts acquiring escalation calendars, source-map processing, or a custom query language, outsource that undifferentiated work to the product built for it.
How should Node.js detect backend failures in error logs?
"Backend failure" is too vague to page on. For this pipeline, the useful unit is a failed operational step: a carrier manifest import, a route-normalization job, or a payment capture associated with a shipment. Emit structured fields at the point of failure, including a stable job identifier, the failure kind, the stage, and the available trace_id or span_id.
The alert should answer three questions without opening a laptop: which run failed, how many related failure records appeared, and which correlation IDs lead back to the surrounding logs. Imagine one manifest import containing 400 bad rows. Posting 400 messages hides the useful fact: one run failed at one stage and probably needs one corrective action. Group those records by the stable run identifier and failure kind, show a few representative messages, and retain the correlation IDs that lead to neighboring records. An operator can then decide whether to rerun the whole manifest, quarantine one shipment, or wait for a carrier correction. That is incident reconstruction rather than keyword forwarding.
There is a firm limitation: correlation fields help join log records, but they do not create a distributed trace or a span tree. A team that needs causal service maps, source-map deobfuscation, crash symbolication, Electron minidumps, or session replay should use tooling that explicitly supplies those investigation modes. Sentry is the natural candidate when exception grouping is central; its documented fingerprint rules also give engineers direct control over which events belong together.
For a nightly job, add a separate heartbeat. Healthchecks is designed around periodic-job pings and grace times, which covers the silent case where a scheduler, queue producer, or machine fails before any error log exists. This distinction matters.
No log query can retrieve an event that was never emitted.
Two criteria that decide the architecture
The first criterion is incident reconstruction cost. A useful polling result is not a raw count. It is a compact bundle keyed by run and failure type, with representative messages and correlation IDs. If creating that bundle requires joining many services or retaining a large local copy of the logs, the small worker has stopped being small. Datadog or Better Stack will usually be the cleaner choice because log queries and alert evaluation already live in their managed workflows. This is the central trade-off: a tiny poller removes an integration, while a growing poller quietly turns alerting into software you own.
The second criterion is ownership of notification state. A polling design must remember what it has sent, suppress repeats during a cooldown, and retry temporary failures without double-posting. It must also handle rate limits. These are ordinary mechanics, but they consume the same engineering hours as customer features. The revenue-per-hour test is blunt: if this worker cannot be understood during one coffee and changed in one release, buy the alerting layer.
I would keep compliance workflows elsewhere. This log path has no bulk export, subscription, or per-user deletion API, so it is a poor foundation for data-subject erasure or archival pipelines. GDPR Article 17 makes deletion a separate system requirement, not an alert checkbox. Keep the operational payload narrow, set an appropriate retention policy in the system of record, and avoid copying customer data into Slack.
A small polling worker
The implementation below makes no assumptions about undocumented server-side filters. It fetches the log-search response, walks the returned JSON, selects structured failure objects client-side, and derives a deterministic fingerprint. The state file gives the worker a 24-hour dedupe window. A real deployment should place that state on durable private storage if jobs can move between machines.
The code has one intentional constraint: Slack receives one aggregate message per polling run. Fewer notifications make each one worth reading.
import { createHash } from "node:crypto";
import { readFile, writeFile } from "node:fs/promises";
type Json = null | boolean | number | string | Json[] | { [key: string]: Json };
type Failure = {
fingerprint: string;
kind: string;
jobId: string;
message: string;
traceId?: string;
spanId?: string;
};
const INFRAI_API_KEY = process.env.INFRAI_API_KEY;
const INFRAI_BASE_URL = ["https://api", "infrai", "cc/v1"].join(".");
const SLACK_WEBHOOK_URL = process.env.SLACK_WEBHOOK_URL;
const STATE_FILE = process.env.ALERT_STATE_FILE ?? "/tmp/pipeline-alert-state.json";
const DEDUPE_MS = 24 * 60 * 60 * 1_000;
if (!INFRAI_API_KEY || !SLACK_WEBHOOK_URL) {
throw new Error("INFRAI_API_KEY and SLACK_WEBHOOK_URL are required");
}
async function requestWithBackoff(url: string, init: RequestInit): Promise<Response> {
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(url, init);
if (response.status !== 429) return response;
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: Math.min(1_000 * 2 ** attempt, 30_000);
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
throw new Error("Rate limit persisted after five attempts");
}
function objectsIn(value: Json): Array<Record<string, Json>> {
if (Array.isArray(value)) return value.flatMap(objectsIn);
if (value === null || typeof value !== "object") return [];
return [value, ...Object.values(value).flatMap(objectsIn)];
}
function asText(value: Json | undefined): string | undefined {
return typeof value === "string" && value.length > 0 ? value : undefined;
}
function toFailure(event: Record<string, Json>): Failure | undefined {
const status = asText(event.status);
const eventName = asText(event.event);
const kind = eventName ?? status;
const isFailure = status === "error" || eventName === "job_failure" || eventName === "payment_failure";
if (!isFailure || !kind) return undefined;
const jobId = asText(event.job_id) ?? "unknown-job";
const message = asText(event.message) ?? "No message supplied";
const traceId = asText(event.trace_id);
const spanId = asText(event.span_id);
const fingerprint = createHash("sha256")
.update(JSON.stringify({ kind, jobId, message, traceId, spanId }))
.digest("hex");
return { fingerprint, kind, jobId, message, traceId, spanId };
}
async function loadState(): Promise<Record<string, number>> {
try {
return JSON.parse(await readFile(STATE_FILE, "utf8")) as Record<string, number>;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return {};
throw error;
}
}
async function main(): Promise<void> {
const logsResponse = await requestWithBackoff(`${INFRAI_BASE_URL}/logs/search`, {
method: "GET",
headers: { Authorization: `Bearer ${INFRAI_API_KEY}`, Accept: "application/json" },
});
if (!logsResponse.ok) {
throw new Error(`Log search failed (${logsResponse.status}): ${await logsResponse.text()}`);
}
const payload = (await logsResponse.json()) as Json;
const failures = objectsIn(payload).map(toFailure).filter((item): item is Failure => item !== undefined);
const now = Date.now();
const state = await loadState();
const recentState = Object.fromEntries(
Object.entries(state).filter(([, sentAt]) => now - sentAt < DEDUPE_MS),
);
const fresh = failures.filter((failure) => recentState[failure.fingerprint] === undefined);
if (fresh.length === 0) return;
const lines = fresh.slice(0, 20).map((failure) => {
const correlation = [failure.traceId, failure.spanId].filter(Boolean).join(" / ");
return `- ${failure.kind} | ${failure.jobId} | ${failure.message}${correlation ? ` | ${correlation}` : ""}`;
});
const slackResponse = await requestWithBackoff(SLACK_WEBHOOK_URL, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ text: `Nightly pipeline: ${fresh.length} new failure(s)\n${lines.join("\n")}` }),
});
if (!slackResponse.ok) {
throw new Error(`Slack delivery failed (${slackResponse.status}): ${await slackResponse.text()}`);
}
for (const failure of fresh) recentState[failure.fingerprint] = now;
await writeFile(STATE_FILE, JSON.stringify(recentState), { mode: 0o600 });
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
Run this worker after the pipeline or every few minutes from an external scheduler. The ordering is important: write dedupe state only after Slack accepts the message. If delivery fails, the next scheduled run retries the same aggregate rather than silently marking it sent. Slack's incoming-webhook URL is the credential here, so keep it in an environment variable and never attach the backend API authorization header to it.
This example deliberately does not pretend that trace_id is a trace viewer. It is a search handle.
That is the limit.
When the runner-up is better
Choose Sentry when application exceptions dominate and automatic event grouping is more valuable than keeping all backend calls behind one contract. Its grouping and fingerprint model is documented, inspectable, and purpose-built. It also becomes the stronger fit when source maps or release-oriented error triage are required.
Choose Datadog when the log alert must sit beside metrics, dashboards, and an established on-call operation. Its log monitors can evaluate search queries and notify from the managed platform. This transfers more machinery out of the repository, though the team still has to tune queries and thresholds.
Better Stack is compelling when the requirement is narrower: ingest logs, create a saved SQL-style query, and alert from it. It removes the custom polling loop while preserving a log-first workflow. Compare its supported ingestion and retention model against the fields your logistics events actually carry.
Healthchecks is not a runner-up for log investigation; it is the companion for absence detection. Use it when missing the nightly dispatch import is worse than receiving its error five minutes late. One success or failure ping can settle the "did it run?" question while the log system handles the "why did it fail?" question.
The decision rule remains small. Build the poller when correlation IDs plus structured failure records reconstruct the incident and the state machine fits in one file. The polling approach is not suitable when alert routing, distributed traces, source maps, session replay, bulk export, or per-user log deletion is required. Buy managed alerting when people, escalation, compliance, or investigation depth are the hard part. That keeps observability proportional to the business while leaving the next weekly release room to ship.
Top comments (0)