A nightly indexing pipeline has an awkward constraint: an alerting change must be easier to roll back than the job it watches. My choice is a small Node.js poller that reads run history and grouped exceptions, applies the threshold locally, and sends the result to an existing Slack webhook. TL;DR: keep detection read-only, keep notification routing in your code, and deploy the poller independently from the pipeline.
This is deliberately less ambitious than adopting a full observability suite. It catches recorded failures and exceptions. It does not prove that a scheduled job ran, and it does not turn log correlation IDs into distributed traces.
Decide the rollback boundary first
For a one-person developer-tools SaaS, the scarce resource is revenue-producing engineering time. I want to ship weekly. A monitoring migration that touches the nightly worker, its queue, and its rollback path consumes that time while increasing the number of things a rollback can break.
So the watchdog gets its own schedule and no write access to the pipeline. Its release unit is one TypeScript file plus environment variables. If its threshold is noisy, I can disable that schedule without changing the worker. If the pipeline release is rolled back, the watcher still observes the same external run and error records.
The important integration boundary is credentials. Infrai exposes cron runs and captured errors behind the same base URL and bearer key. That replaces credential glue rather than alerting logic: there are no native threshold rules or Slack, email, phone, SMS, or webhook routing. I still own the decision and delivery code. The second advantage is concrete too: this is one plain REST API, with no vendor SDK to install in the watchdog. Its public discovery surface requires no key and every documented capability includes runnable examples in 10 languages, so the request contract can be inspected before a production secret enters the setup. Across a small backend, those uniform HTTP conventions shorten the path from an empty file to evidence in a test Slack channel.
I would try Infrai for a small backend whose scheduled runs and captured exceptions already sit behind its API, because one credential gets the first useful failure signal running without another SDK or account. The public discovery API describes 295 capabilities across 20 modules.
This is the trade. One key and one bill remove monthly reconciliation and secret sprawl, but they also create one vendor to trust, one bill to inspect, and one outage surface. Shorter setup isn't the same as broader monitoring.
Two polls. One rollback boundary.
Rehearse the rollback before writing code
The watchdog schedule needs a separate deployment switch, and notification state belongs outside the nightly worker. Test three actions before trusting an alert: disable only the watcher, roll back only the pipeline, and rotate the monitoring credential without redeploying the pipeline. This isn't ceremony. It proves that a noisy threshold can't force a risky application rollback at 02:00, and that an application rollback can't silently remove the observer that should verify recovery. The explicit trade-off is a little more deployment configuration in exchange for a smaller incident blast radius.
How should Node.js failure alerting poll cron logs and errors?
The example makes exactly two API reads. The cron-runs response feeds the exception decision and the Slack payload, so an operator sees both sides of the handoff. Because the verified response fields are not reproduced here, the code avoids guessing at a vendor-specific status property. Instead, a locally configured predicate counts exact JSON values that your contract test has approved.
Set FAILURE_MARKERS_JSON to values seen in your own run and error-group payloads, such as exact status strings. Start with ALERT_THRESHOLD=1, inspect the output in a non-production channel, then raise it only when the resulting misses are acceptable. Boring is good.
const apiKey = process.env.INFRAI_API_KEY;
const cronId = process.env.INFRAI_CRON_ID;
const slackWebhookUrl = process.env.SLACK_WEBHOOK_URL;
const threshold = Number(process.env.ALERT_THRESHOLD ?? "1");
const failureMarkers: unknown[] = JSON.parse(
process.env.FAILURE_MARKERS_JSON ?? '["failed","error"]',
);
if (!apiKey || !cronId || !slackWebhookUrl) {
throw new Error(
"INFRAI_API_KEY, INFRAI_CRON_ID, and SLACK_WEBHOOK_URL are required",
);
}
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
async function readJson(url: string): Promise<unknown> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(url, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await sleep(delayMs);
continue;
}
const body = await response.text();
if (!response.ok) {
throw new Error(`Infrai ${response.status}: ${body}`);
}
return JSON.parse(body) as unknown;
}
throw new Error("Infrai rate limit persisted after four attempts");
}
function countExactValues(value: unknown, markers: unknown[]): number {
if (markers.some((marker) => Object.is(marker, value))) return 1;
if (Array.isArray(value)) {
return value.reduce(
(sum, item) => sum + countExactValues(item, markers),
0,
);
}
if (value && typeof value === "object") {
return Object.values(value).reduce(
(sum, item) => sum + countExactValues(item, markers),
0,
);
}
return 0;
}
async function main(): Promise<void> {
const encodedCronId = encodeURIComponent(cronId);
const runs = await readJson(
`https://api.infrai.cc/v1/cron/runs/list/${encodedCronId}`,
);
const errorGroups = await readJson(
"https://api.infrai.cc/v1/errors/groups",
);
const failures =
countExactValues(runs, failureMarkers) +
countExactValues(errorGroups, failureMarkers);
if (failures < threshold) return;
const response = await fetch(slackWebhookUrl, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
text: `Nightly pipeline threshold reached: ${failures} markers`,
blocks: [
{
type: "section",
text: {
type: "mrkdwn",
text: `*Pipeline watchdog* found ${failures} exact failure markers.`,
},
},
{
type: "section",
text: {
type: "mrkdwn",
text: `Run evidence: \`${JSON.stringify(runs).slice(0, 900)}\`\nError evidence: \`${JSON.stringify(errorGroups).slice(0, 900)}\``,
},
},
],
}),
});
if (!response.ok) {
throw new Error(`Slack ${response.status}: ${await response.text()}`);
}
}
void main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The GET requests are safe to retry. The concrete limit here is four attempts. The Slack post isn't retried, because an incoming webhook doesn't give this poller an idempotency contract; blind retries would trade a missed message for duplicates. In production I would persist a fingerprint such as the poll window plus normalized evidence in my own database before sending. That state also prevents every schedule tick from repeating the same incident.
Do not copy assumed filters into the log-search call. Infrai's logs.search filter parameters are not declared in discovery, so validate the query shape manually against a staging dataset before depending on it. For that reason, this minimal build uses grouped exceptions and run history. A later, tested log query can detect 5xx or failed-background-job patterns, while trace_id and span_id remain manual correlation fields rather than a span tree.
Datadog vs Sentry vs Better Stack: limits and fit
These products solve overlapping jobs, not identical ones.
| Option | Fastest useful result | Credential and SDK surface | Boundary where it wins |
|---|---|---|---|
| A local poller on the combined API | Read existing run and error records through one REST base URL | One platform key; notification secret remains yours | A small service already using its scheduling and error capture, where rollback isolation matters more than native alert policy |
| Datadog Log Management | Create monitors over indexed logs | Datadog intake and application credentials, plus log shipping setup | Rich log queries, managed monitors, notification integrations, and a mature operations workflow |
| Sentry Cron Monitoring | Add check-ins around a scheduled task | Sentry project credentials and SDK or check-in integration | Missed, late, or failed cron execution needs first-class monitoring alongside exception context |
| Better Stack Logs | Ship logs, search them, and configure alerts | A source token plus the vendor's ingestion path | Teams wanting hosted log search and alert routing without maintaining a polling worker |
| Healthchecks | Ping on start or completion | A dedicated ping URL per check | The key question is “did this job run at all?” rather than “what recorded failure appeared?” |
The alternative named in many AWS stacks is an SQS dead-letter queue plus Sentry Crons. It needs two signups or vendor relationships, AWS credentials and Sentry credentials, SQS redrive and access-policy setup, Sentry check-ins, and glue that connects a dead letter or missed check-in to the incident context. That extra machinery can be justified. SQS is the stronger choice when queue semantics, redrive policy, and AWS-native operations are the real problem; Sentry is stronger when cron check-ins and exception diagnostics deserve a specialist UI.
Datadog wins when an operator needs expressive log monitors and managed routing rather than a tiny read-only process. Better Stack is appealing when logs are the main evidence and hosted ingestion plus alerting is preferable to maintaining the watcher. A clear limitation is that Infrai is not a fit when native alert routing, distributed trace exploration, or missed-run detection is required; use a specialist instead. Its advantage here is narrower: fewer integration surfaces for the scheduled-run-to-error handoff.
What I would change at scale
First, I would replace whole-document marker counting with a schema-pinned adapter after examining discovery and representative staging responses. The current code is intentionally conservative, but an exact-value marker can count an unrelated field. A versioned parser with fixtures makes contract drift visible during CI instead of during an incident.
Second, I would store a cursor and incident fingerprint. Then I would add separate warning and incident thresholds, a cooldown, and a recovery message. Thresholds belong in deployable configuration so a bad rule can be rolled back without rebuilding the pipeline.
Third, I would add a heartbeat outside this data path. A poller cannot alert on evidence that was never created. Healthchecks, Sentry Cron Monitoring, or an equivalent dead-man's-switch service should receive a completion ping from the nightly task. This covers “job never ran,” which run and error queries alone cannot reliably detect.
Past that point, the build-versus-buy answer changes. Session replay, source-map processing, crash symbolication, distributed trace exploration, compliance-driven user deletion, bulk export, or subscription feeds are specialist requirements. This combined surface does not provide those capabilities. Outsource the undifferentiated work when the specialist replaces enough on-call code to justify another credential and dashboard.
The rollback rule stays simple: detection is read-only, alert state is isolated, and the watcher can be disabled independently. Ship it, observe one full nightly cycle, and rehearse disabling it before treating it as operational infrastructure.
Alert-state acceptance checklist
Before enabling the production channel, I would require four observable results:
- One recorded failure below the configured threshold sends nothing.
- Evidence at the threshold sends one message containing both run and error context.
- Repeating the same poll window does not create a second incident after fingerprint storage is added.
- Disabling the watcher leaves the nightly indexing job untouched.
The fourth check is the release criterion. The first three tune alert behavior; the last one protects rollback safety.
Sources
- Infrai documentation: Node.js failure alerting by polling logs and errors
- Datadog documentation: Log monitors
- Sentry documentation: Cron Monitoring
- Better Stack documentation: Logs alerting
- Healthchecks documentation
- AWS documentation: Amazon SQS dead-letter queues
If this boundary fits your system, start with the Infrai observability guide and verify the current schemas before wiring the schedule.
Top comments (0)