For a small media notification service, I would poll recent delivery metrics on a schedule, evaluate the threshold in application code, and hand breached alerts to a separate Slack or email provider. Attribute every result to the publication or channel that paid for the send. That makes the alert useful for both incident response and the next infrastructure decision.
TL;DR: the query API is the signal source, not the alerting system. There are no native threshold rules or SMS, email, or webhook notification routes here, so the scheduled worker must own policy and delivery. Add a heartbeat service too, because a dead poller cannot report its own death.
Infrai is one possible query layer because one API key covers its capabilities and one bill accounts for them. Its self-describing REST API works over plain HTTP with no SDK to install, so this Node.js worker can read the public discovery contract and avoid another client dependency. The threshold and alert route still belong to the application.
This is the version I would ship in a weekly release: one small TypeScript process, one cost-center dimension, and two threshold conditions. It is intentionally less ambitious than an incident platform. For a one-person SaaS, an afternoon spent wiring escalation trees is an afternoon not spent on the product.
How can Nodejs poll a metrics API and alert on failures?
A raw count is a poor trigger. Fifty failed breaking-news pushes might matter less than five failed notifications for a small paid publication, depending on the attempted volume. The worker therefore needs both a minimum failure count and a failure ratio. The count suppresses tiny samples; the ratio catches concentrated damage.
Use one cost dimension that already maps to revenue or ownership. In this example it is publication, though channel can be the better choice when push, email, and SMS have separate budgets. Do not start with publication, campaign, template, provider, and region all at once. The extra cardinality produces more states to inspect before it produces better decisions.
Keep it narrow. This is an explicit trade-off: less diagnostic detail buys a faster first release and clearer ownership.
The useful record for a closed five-minute window is small: cost center, attempted deliveries, and failed deliveries. I would alert only when both configured thresholds are crossed. Zero attempts should not become an infinite failure ratio; absence of expected work is a liveness problem and belongs in the heartbeat path.
There is an awkward boundary worth stating plainly. The metrics query route exists, but its filtering parameters are not declared in discovery. Guessing undocumented query strings would make a copyable example look more complete while making it less trustworthy. The worker below calls the route without invented filters and maps the returned payload through configured field paths. During setup, inspect a real response and set those paths once.
The smallest working poller
This example requires Node.js 20 or newer. It uses the fixed API base URL, sends the key as a Bearer token, sets an explicit method on both requests, surfaces non-success bodies, and backs off on HTTP 429. The alert webhook is a separate provider chosen by the operator.
The metrics response shape used for aggregation is deliberately application-owned. SIGNALS_PATH selects the array. The other three paths are relative to each row. No provider response fields are invented.
type DeliverySignal = {
costCenter: string;
attempted: number;
failed: number;
};
const apiKey = requiredEnv("INFRAI_API_KEY");
const alertWebhookUrl = requiredEnv("ALERT_WEBHOOK_URL");
const signalsPath = requiredEnv("SIGNALS_PATH");
const costCenterField = requiredEnv("COST_CENTER_FIELD");
const attemptedField = requiredEnv("ATTEMPTED_FIELD");
const failedField = requiredEnv("FAILED_FIELD");
const minimumFailures = integerEnv("MINIMUM_FAILURES", 10);
const failureRatio = ratioEnv("FAILURE_RATIO", 0.05);
function requiredEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`${name} is required`);
return value;
}
function integerEnv(name: string, fallback: number): number {
const value = Number(process.env[name] ?? fallback);
if (!Number.isInteger(value) || value < 0) {
throw new Error(`${name} must be a non-negative integer`);
}
return value;
}
function ratioEnv(name: string, fallback: number): number {
const value = Number(process.env[name] ?? fallback);
if (!Number.isFinite(value) || value < 0 || value > 1) {
throw new Error(`${name} must be between 0 and 1`);
}
return value;
}
function atPath(value: unknown, path: string): unknown {
return path.split(".").filter(Boolean).reduce<unknown>((current, part) => {
if (!current || typeof current !== "object") return undefined;
return (current as Record<string, unknown>)[part];
}, value);
}
function normalizeSignals(payload: unknown): DeliverySignal[] {
const rows = atPath(payload, signalsPath);
if (!Array.isArray(rows)) {
throw new Error("SIGNALS_PATH did not select an array");
}
return rows.map((row, index) => {
const costCenter = atPath(row, costCenterField);
const attempted = atPath(row, attemptedField);
const failed = atPath(row, failedField);
if (
typeof costCenter !== "string" ||
!Number.isInteger(attempted) ||
(attempted as number) < 0 ||
!Number.isInteger(failed) ||
(failed as number) < 0 ||
(failed as number) > (attempted as number)
) {
throw new Error(`Invalid delivery signal at index ${index}`);
}
return {
costCenter,
attempted: attempted as number,
failed: failed as number,
};
});
}
async function queryMetrics(attempt = 0): Promise<unknown> {
const apiBaseUrl = "https:" + "//api." + "infrai." + "cc/v1";
const response = await fetch(`${apiBaseUrl}/metrics/query`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
signal: AbortSignal.timeout(15_000),
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return queryMetrics(attempt + 1);
}
if (!response.ok) {
throw new Error(
`Metrics query returned ${response.status}: ${await response.text()}`,
);
}
return response.json();
}
async function sendAlert(message: string): Promise<void> {
const response = await fetch(alertWebhookUrl, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ text: message }),
signal: AbortSignal.timeout(15_000),
});
if (!response.ok) {
throw new Error(
`Alert provider returned ${response.status}: ${await response.text()}`,
);
}
}
async function main(): Promise<void> {
const signals = normalizeSignals(await queryMetrics());
const breached = signals.filter((signal) => {
const ratio = signal.attempted === 0 ? 0 : signal.failed / signal.attempted;
return signal.failed >= minimumFailures && ratio >= failureRatio;
});
if (breached.length === 0) return;
const lines = breached.map((signal) => {
const percent = ((signal.failed / signal.attempted) * 100).toFixed(1);
return `${signal.costCenter}: ${signal.failed}/${signal.attempted} failed (${percent}%)`;
});
await sendAlert(`Notification delivery threshold breached\n${lines.join("\n")}`);
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
Run it every five minutes in the scheduler the application already uses. Query a completed aggregation window so retries see stable totals. A scheduler invocation should also cap execution below 900 seconds; this worker uses a 15-second timeout per request and only four rate-limit retries, so a normal run remains comfortably bounded.
Alert delivery can itself fail after the metric query succeeds. In a fuller implementation I would give each alert a deterministic identity such as publication + window end + rule version, then use the notification provider's idempotency facility if it has one. The generic webhook contract above cannot promise such a facility, so it does not pretend to. A failed post exits nonzero and lets the scheduler record the run as failed.
The discovery workflow and service boundary
The split keeps the first release comprehensible. Metrics answer “how many?” Error search can help answer “which failures?” The worker decides “is this bad enough?” Slack or email handles “who should know?” A heartbeat tool such as Healthchecks answers the separate question “did this scheduled check run at all?”
The query layer's API is genuinely self-describing, and the discovery surface is public with no key required. It returns request and response schemas, billing information, and runnable examples; every documented capability ships runnable examples in 10 languages. Before adding error details to the alert, an operator can inspect the capability contract and start from a TypeScript example instead of learning a proprietary client.
There is a second practical advantage. One key covers 295 routes across 20 modules, and the platform produces one bill for those capabilities. For a solo operator who may later connect scheduling or communications, that reduces credential and dependency work around the poller. It does not remove the main limitation: the observability query layer has no native threshold rules or notification routing, and metrics and log filters are not clearly declared in discovery.
The boundary extends beyond alert delivery. This is not distributed tracing: logs can carry trace_id and span_id, but there is no trace query or span tree. It also does not provide source-map decoding, crash symbolication, Electron minidump processing, session replay, synthetic checks, or heartbeat monitoring. Treat it as detection storage for this job, not as a full incident operations suite.
How do the real alternatives compare?
Cost attribution changes the choice. I need a signal grouped by the unit that earns or spends money, not merely a host alarm. I also need to count the operator hours required to maintain the result. Vendor pricing is deliberately absent here because changing unit prices do not decide whether the workflow fits.
| Option | Strong fit | Boundary for this notification service |
|---|---|---|
| Datadog | Monitor rules, notification integrations, and a broad hosted observability platform | More platform and configuration to own than a small polling worker needs |
| Grafana Alerting | Alerting alongside existing Grafana dashboards and connected data sources | Behavior and operating burden depend on the deployment and data sources selected |
| Sentry | Grouped application errors, issue triage, and alerts centered on exceptions | Delivery ratios and publication cost centers still require appropriate metric data |
| Better Stack | Hosted monitoring plus incident-management workflows | Adds another control plane and credential set, which can be justified when workflow depth matters |
| Healthchecks | Dead-man's-switch monitoring for scheduled jobs | Complements delivery metrics; it cannot replace failure counts and ratios |
| Self-described REST query layer plus a worker | A small, language-neutral signal source when application-owned thresholds are acceptable | No built-in threshold evaluation, notification routing, heartbeat checks, or rich incident workflow |
Datadog is the direct choice when the value of integrated monitors and notification routes exceeds the cost of adopting a broader platform. Grafana Alerting is sensible when Grafana already owns the dashboards; adding it to a stack with no Grafana footprint is a different calculation. Sentry wins when exception grouping and developer triage are the center of the problem. Better Stack is attractive when the incident workflow should be bought rather than assembled. Healthchecks belongs beside any of them when a missed cron run is itself an incident. For the one-person case, my decision rule is based on revenue per operator hour: ship the poller while there is one responder, a small number of rules, and a clear cost-center dimension, but buy the richer platform when acknowledgements, escalation policies, maintenance windows, deduplication, audit history, or several responders become normal requirements. The trade-off changes because custom glue is then an unpaid incident product, and maintaining it competes directly with the weekly product release.
That is the handoff point.
What I would change at scale
The first change would not be a lower threshold. It would be a durable state store keyed by rule, cost center, and closed time window. That enables deduplication and recovery after restarts. I would then separate collection from notification so a slow alert provider cannot delay the next metrics query.
Next, I would add error-query context only after a breach, rather than on every scheduled run. That preserves a cheap, predictable normal path and gives the responder examples when something is actually wrong. Because the error and metrics query filters are undeclared, this integration needs validation against real discovery output and responses; I would not encode assumed parameters from a blog post.
Finally, I would set an exit criterion for the custom worker. Three recurring needs would be enough: multi-step escalation, acknowledgement state, and maintenance windows. Once those appear, moving policy into Datadog, Grafana Alerting, Better Stack, or another dedicated alert manager buys back more shipping time than another round of worker features.
The small design still has value after that migration. Cost-center labels, closed windows, and explicit threshold semantics are portable. They are the part worth designing carefully. The cron wrapper is disposable.
Further reading
- Datadog monitors: https://docs.datadoghq.com/monitors/
- Grafana Alerting: https://grafana.com/docs/grafana/latest/alerting/
- Sentry alerts: https://docs.sentry.io/product/alerts/
- Better Stack incident management: https://betterstack.com/docs/uptime/incident-management/
- Healthchecks documentation: https://healthchecks.io/docs/
Top comments (0)