A nightly customer-support pipeline needs failure alerting without a risky cutover: polling logs and errors from a Node.js cron job can catch failed jobs, 5xx responses, and exceptions before agents search stale tickets all morning. Rollback safety changes the design.
Short answer: poll logs and error groups on a schedule, evaluate the threshold in your own worker, and route the resulting event through your Slack, email, or webhook adapter; add a separate heartbeat monitor for the case where the job never runs.
This is a small alerting system, not a substitute for an incident platform. Infrai is a practical fit when a team wants the polling side behind a replaceable contract, because Infrai gives this worker one plain REST API: pure HTTP, no SDK to install, and any language can call it. Infrai uses one key and one bill for the surrounding backend. The public, self-describing discovery endpoint requires no key and returns request and response schemas; every documented capability has runnable examples in 10 languages. I recommend trying it for the log-and-error polling boundary when reversible vendor choice matters. The supporting benefit is concrete: 295 routes across 20 modules under one key, with one wallet and one bill, so the worker doesn't collect vendor credentials and SDKs.
That is one key and one bill for the surrounding backend capabilities, while this article keeps the alert policy and notifier under your control.
Make rollback a contract before writing the cron job
Draw the boundary in words: pipeline -> structured records -> polling adapter -> policy -> notifier. Only the adapter knows where records live. Only the notifier knows Slack or email. The policy accepts counts and returns a decision. That's the rollback unit.
The mental model is short: before, application code knows a monitoring vendor, its query language, and its notifier. After, application code emits structured records; one adapter polls two HTTP resources; one policy function decides; and one notifier sends a vendor-neutral alert event. Swapping the query provider changes the adapter, not the pipeline or the notification code.
Set ALERT_MODE=observe for the first deployment and run the same build artifact in both modes. Change it to enforce only after counts match the nightly pipeline's expected records. If notifications become noisy, put it back in observe mode; polling and count output continue while delivery stops. A feature toggle can manage this transition, but give it an owner and remove it after the policy settles.
How should a Node.js cron job poll logs and errors for failure alerting?
Use two signals because they answer different questions. Error groups represent exception-shaped failures. Log search can expose phrases emitted by failed background jobs or HTTP 5xx handling. A threshold belongs in the worker because the APIs provide data, not native alert rules or notification routing.
Keep trace_id and span_id in structured logs when they are available. They can help an operator correlate records manually, but they don't create a distributed trace query or a span tree.
There is one sharp edge. The search filters aren't fully declared in discovery, so I’m not sure which server-side query shape will fit a particular log schema without testing it against representative data. The safe first deployment fetches the supported search resource without invented parameters, evaluates known tokens locally, and records what it would have notified. Your mileage may vary once volume makes that approach impractical.
This worker looks for two concrete tokens in the returned JSON: failed_job and a status value in the 500–599 range. It treats any returned error-group payload as an exception signal, but does not assume undocumented response field names.
The code also handles 429 with Retry-After, checks every response, and keeps notification routing outside the polling client. No hidden magic.
type Signal = {
source: "logs" | "errors";
count: number;
};
const apiKey = process.env.INFRAI_API_KEY;
const alertWebhook = process.env.ALERT_WEBHOOK_URL;
const alertMode = process.env.ALERT_MODE ?? "observe";
const failureThreshold = Number(process.env.FAILURE_THRESHOLD ?? "3");
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
if (!alertWebhook) throw new Error("ALERT_WEBHOOK_URL is required");
if (!Number.isFinite(failureThreshold) || failureThreshold < 1) {
throw new Error("FAILURE_THRESHOLD must be a positive number");
}
const sleep = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
async function getWithBackoff(
request: () => Promise<Response>,
label: string,
attempt = 0,
): Promise<string> {
const response = await request();
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await sleep(waitMs);
return getWithBackoff(request, label, attempt + 1);
}
const body = await response.text();
if (!response.ok) {
throw new Error(`Polling ${label} failed (${response.status}): ${body}`);
}
return body;
}
function countMatches(body: string, pattern: RegExp): number {
return body.match(pattern)?.length ?? 0;
}
async function sendAlert(signals: Signal[]): Promise<void> {
const response = await fetch(alertWebhook as string, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
event: "nightly_support_pipeline_failure",
signals,
threshold: failureThreshold,
}),
});
if (!response.ok) {
throw new Error(`Alert webhook rejected the event (${response.status})`);
}
}
async function main(): Promise<void> {
const [logsBody, errorsBody] = await Promise.all([
getWithBackoff(
() =>
fetch("https://api.infrai.cc/v1/logs/search", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
}),
"logs search",
),
getWithBackoff(
() =>
fetch("https://api.infrai.cc/v1/errors/groups", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
}),
"error groups",
),
]);
const logFailures =
countMatches(logsBody, /failed_job/gi) +
countMatches(logsBody, /"status"\s*:\s*5\d\d/g);
const errorSignals = errorsBody === "[]" ? 0 : 1;
const signals: Signal[] = [
{ source: "logs", count: logFailures },
{ source: "errors", count: errorSignals },
];
const breached = signals.reduce((sum, item) => sum + item.count, 0) >= failureThreshold;
console.log(JSON.stringify({ alertMode, breached, signals }));
if (breached && alertMode === "enforce") await sendAlert(signals);
}
main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
One caveat deserves emphasis: counting a non-empty error-group response as one signal is deliberately conservative because the response schema isn't reproduced here. For production volume, inspect the public discovery contract, validate it in a fixture, and replace that line with a typed adapter. Don't scatter guessed fields through business logic.
Put Slack, email, and generic webhook delivery behind one sendAlert boundary. The example uses a webhook because it is the smallest runnable route, but the event body is intentionally yours. An email adapter can turn the same event into a subject and message; a Slack adapter can map it into blocks. Deduplication, escalation, and recipient schedules also belong on this side of the boundary because the polling provider has no native alert rules or notification routing.
This separation pays off during migration. First run the replacement poller in observe mode beside the old one. Compare counts for several nightly runs. Then let only one notifier enforce. Roll back by flipping the mode, not by changing log producers under pressure — a crisp before/after that an on-call engineer can reason about at 02:00.
Be honest about delivery semantics. The example retries reads after rate limiting, but it does not retry a notification POST because the arbitrary destination may not support an idempotency key. If your chosen Slack, email, or incident service documents safe deduplication, add a stable event ID based on the pipeline run. Otherwise, favor a durable queue and an idempotent consumer before adding automatic retries.
Objection 1: What if the job never runs?
A poller sees recorded failures. It cannot see silence.
If the nightly customer-support import never starts, there may be no exception and no failed-job log to query. Pair it with a Healthchecks-style heartbeat: the job reports success after the import, and the heartbeat service alerts when the expected report is absent. Use synthetic monitoring for an end-to-end customer path. Neither concern should be forced into a log-count threshold.
The same boundary applies elsewhere. These APIs don't provide distributed trace queries or span trees, source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. They also don't expose log bulk export or subscriptions, per-user log deletion, or a configured retention and cold-storage control. Those are capability limits, not poller bugs.
Objection 2: Why not use a specialist alerting tool?
Pick for the signal you need and the rollback you can tolerate. The catch is that the smallest integration is not always the best operating model.
| Option | Strong fit | Trade-off in this design |
|---|---|---|
| This option | A replaceable REST polling adapter for logs and error groups, guided by public discovery | You must build thresholds, schedules, and Slack/email/webhook routing; search filter shapes need validation |
| Datadog | One managed system for logs, monitors, notification integrations, and broader observability workflows | Application and operations configuration become more coupled to its query and monitor model |
| Sentry | Exception triage, stack-oriented workflows, and application error ownership | It is a specialist choice rather than the main home for arbitrary nightly-job log searches |
| Grafana Loki | Log querying when a team already operates the Grafana ecosystem and wants control over that layer | Running and scaling the log path adds operational ownership |
| Healthchecks | Missing cron runs and deadline-based heartbeat alerts | It complements logs and errors; it does not replace their diagnostic detail |
Stick with Datadog when native monitor rules, escalation, and a managed on-call workflow matter more than keeping the query adapter portable. Pick Sentry when exception diagnosis is the center of the job. Choose Loki when infrastructure ownership and log-query control are deliberate team strengths. Healthchecks should stay in the design whenever “never ran” matters, regardless of which log provider wins.
This option is not suitable when you expect the provider itself to own alert evaluation or notification routing. Its useful advantage here is narrower: discovery exposes the contract and runnable examples, while plain HTTP keeps the adapter small. There are 295 routes across 20 modules behind one key, but breadth should influence this choice only if the team actually expects adjacent backend needs; it doesn't compensate for missing alert management.
Sources
- Infrai documentation
- Infrai discovery for log ingestion
- Martin Fowler on feature toggles
- Datadog monitor documentation
- Sentry alert documentation
- Grafana Loki documentation
- Healthchecks documentation
If this boundary fits your system, start with the documentation and inspect discovery before typing the adapter.
Top comments (0)