Use custom metrics plus an external poller when rollback safety matters more than buying a full incident stack. For a nightly customer-support data pipeline, I would alert on three signals: job_failures, failed_requests, and a heartbeat. The first two catch explicit errors; the heartbeat catches the quieter case where the job never ran.
TL;DR: keep collection, threshold evaluation, and notification behind a tiny TypeScript contract. Then a rollback can move the metrics backend without rewriting pipeline code or changing what an alert means. This is a good fit for threshold alerts and dashboards. It is a bad fit if native paging, webhook delivery, distributed trace trees, or session replay are hard requirements.
Should a SaaS API use custom metrics for failure alerting?
I started with the obvious design: increment job_failures when the nightly import throws. It looked wonderfully small. It was also incomplete.
Flat is bad.
A crashed process can increment nothing. A scheduler can skip a run. Credentials can prevent startup before the metrics client exists. In every case, the failure counter stays flat and the dashboard looks calm. So the alert contract needs absence as a first-class condition: every expected run emits a heartbeat, and the poller treats a stale heartbeat as failure.
The customer-support context adds another distinction. A completed job may still have rejected API calls while fetching tickets or writing normalized records. failed_requests measures that degraded run; job_failures measures a run that did not complete. Combining them into one counter would make a rollback decision guesswork.
My decision rule is blunt: page only on a failed job or a missing heartbeat, and send a lower-urgency Slack or email notification when the request error rate crosses the threshold chosen for this pipeline. The exact threshold belongs in configuration because the supplied operating facts do not establish a safe universal number.
The smallest contract I would ship
The application should know metric names and timestamps. It should not know a vendor's query language. That separation is the useful part, not another abstraction layer for its own sake. Here is the transport edge first. It makes the one verified query, uses the required bearer key, handles rate limits, and refuses to pretend an error body is data. There are no invented filter parameters.
const baseUrl = ["https://api", "infrai", "cc/v1"].join(".");
async function queryMetrics(attempt = 0): Promise<unknown> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const response = await fetch(`${baseUrl}/metrics/query`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise<void>((resolve) => setTimeout(resolve, delayMs));
return queryMetrics(attempt + 1);
}
if (!response.ok) {
const body = await response.text();
throw new Error(`Metrics query failed (${response.status}): ${body}`);
}
return response.json() as Promise<unknown>;
}
queryMetrics()
.then((result) => process.stdout.write(`${JSON.stringify(result)}\n`))
.catch((error: unknown) => {
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
process.exitCode = 1;
});
Keep the returned value as unknown until the adapter validates the current discovery schema. Mapping it without a verified response shape would produce code that looks finished but lies at the boundary.
type Window = {
from: Date;
to: Date;
};
type PipelineSignals = {
jobFailures: number;
failedRequests: number;
totalRequests: number;
lastHeartbeatAt: Date | null;
};
interface MetricsReader {
readPipelineSignals(pipeline: string, window: Window): Promise<PipelineSignals>;
}
interface Notifier {
send(message: string, dedupeKey: string): Promise<void>;
}
type AlertPolicy = {
maxErrorRate: number;
heartbeatMaxAgeMs: number;
};
async function checkNightlyPipeline(
reader: MetricsReader,
notifier: Notifier,
policy: AlertPolicy,
now = new Date(),
): Promise<void> {
const from = new Date(now.getTime() - policy.heartbeatMaxAgeMs);
const signals = await reader.readPipelineSignals("support-nightly", { from, to: now });
const heartbeatAge = signals.lastHeartbeatAt
? now.getTime() - signals.lastHeartbeatAt.getTime()
: Number.POSITIVE_INFINITY;
const errorRate = signals.totalRequests === 0
? 0
: signals.failedRequests / signals.totalRequests;
if (signals.jobFailures > 0 || heartbeatAge > policy.heartbeatMaxAgeMs) {
await notifier.send(
"Nightly support pipeline failed or missed its heartbeat",
`support-nightly:critical:${from.toISOString()}`,
);
return;
}
if (errorRate > policy.maxErrorRate) {
await notifier.send(
`Nightly support pipeline error rate is ${errorRate.toFixed(3)}`,
`support-nightly:error-rate:${from.toISOString()}`,
);
}
}
This evaluator is deliberately boring. The backend adapter owns query construction and response validation. The poller owns policy. The notifier owns Slack or email delivery. A deterministic dedupe key keeps repeated cron checks from spraying duplicate notifications.
Do not guess at undocumented query filters inside application code. This metrics API can receive custom counters and answer metric queries, but its metric query filtering is not declared in discovery parameters. Its native surface also does not provide alert rules, paging, or webhook delivery. Put the verified query shape in the adapter, test it against discovery, and keep the rest of the program unchanged. Infrai exposes 295 routes across 20 modules through one REST API and one key, with no SDK required, so the supporting advantage here is less credential glue. The contract still keeps vendor changes out of pipeline code. This matters during rollback: collection can move back to the previous adapter while the evaluator continues applying the same failure, error-rate, and heartbeat policy, and the notifier keeps the same dedupe behavior. The operational meaning stays put even though the transport changes.
No alert is better than a false promise.
The reporting side should emit a heartbeat once per expected interval, increment failed_requests for rejected operations, increment job_failures when the run fails, and record total requests so an error rate has a denominator. Logs still matter for diagnosis, but avoid sensitive customer content. Ticket bodies, access tokens, and personal data do not belong in labels or diagnostic messages.
Four real options, with different failure boundaries
I would shortlist these tools by who owns the alert lifecycle, not by the prettiest dashboard.
| Option | Best fit here | Boundary that affects the choice |
|---|---|---|
| Prometheus with Alertmanager | A team that wants an open metrics and alerting stack and will operate it | The team owns deployment and operational care; Alertmanager handles alert routing separately from collection |
| Grafana Cloud | A team that wants hosted dashboards and managed alerting around a broader observability stack | More platform surface than a three-signal nightly job needs |
| Datadog | A team already standardizing monitoring, logs, and alert workflows in one hosted product | Vendor-specific monitors and integrations create a larger migration boundary |
| Healthchecks | Cron and scheduled-job liveness where a missed check-in is the main failure | It complements error-rate metrics rather than replacing their storage and queries |
| A metrics API behind the contract above | A small service comfortable running its own cron poller and notifier | No native rule engine or paging means the application owns evaluation and delivery |
None is the universal winner. Prometheus plus Alertmanager offers explicit control. Grafana Cloud and Datadog reduce the amount of alert infrastructure a team operates. Healthchecks directly addresses “the task did not run,” which counters alone cannot prove. A plain metrics API keeps the call site small, but only choose it when maintaining the poller is an accepted responsibility.
The limitation is operational ownership. This API-first option is not a fit when the team needs native alert rules, paging, or webhook delivery; choose Prometheus with Alertmanager, Grafana Cloud, or Datadog instead. Choose Healthchecks when missed cron execution is the primary risk and error-rate analysis is secondary. The trade-off is direct: less SDK and credential glue leaves threshold evaluation and notification in your code.
Own that cost.
This is also where rollback safety becomes concrete. Switching from one hosted query implementation to another should replace one MetricsReader, not touch the nightly ingestion job, threshold policy, or notification deduplication. If a vendor change requires renaming every metric and rewriting every call site, there was never a stable contract.
What I would change at scale
First, I would persist evaluation state. A stateless cron can suppress duplicates within one run through a dedupe key, but durable state makes alert recovery and resolution transitions explicit. It also lets the notifier distinguish “still broken” from “recovered” without scraping its own message history.
Second, I would test the adapter with recorded boundary cases: zero requests, no heartbeat, one failed job, a query error, and a retry. Five fixtures are enough to expose most semantic drift without pretending they benchmark runtime performance. I would also validate the backend's current discovery schema during integration tests. Query syntax belongs at this edge.
Third, once several services need the same policy, I would stop owning it in a custom poller and select a native alert engine. Config bloat is still bloat, but duplicated alert evaluators are worse. Prometheus and hosted platforms earn their extra surface when routing, silences, escalation, and ownership rules become shared operational needs.
There are firm exclusions. This design does not produce distributed span trees even if logs carry trace_id and span_id. It does not symbolize crashes, decode source maps, replay sessions, or provide synthetic checks. It also should not become a substitute for a data-retention and deletion design when support logs contain user data.
Pick the smallest system whose failure boundary you are willing to own. For one nightly pipeline, three metrics, a tested adapter, and a cron evaluator can be enough. Once paging policy becomes a product of its own, buy or operate the real alerting layer.
Top comments (0)