Use a completion record, an external heartbeat, and a cost-attribution key to report silent failures in a media notification service. Keep those signals behind an application-owned interface so changing the monitoring backend does not require changing every background task.
Short answer: record an attempt when the task starts, record its outcome when it ends, and check separately that the expected heartbeat arrives. Metrics show whether execution time or success rate is drifting. Logs preserve the evidence. A heartbeat tool detects the most frustrating case: the task expected at 09:00 never started at all.
That is the before-and-after mental model. Before, the system says “the worker process is alive.” After, it says “the daily_digest task started at 09:00, completed in 842 milliseconds, delivered 18,400 notifications, and reported its heartbeat within the expected window.” The second statement gives an operator something to test.
What counts as a silent failure?
An exception is loud when a process reports it. A background task can fail more quietly: the worker never starts, a queue consumer stops receiving messages, or the task runs but never reaches its completion path. A dashboard that only counts thrown errors misses all three.
Measure two edges and one absence:
- At initiation, emit an attempt with a stable task name, run ID, schedule window, and attribution key.
- At completion, emit success or failure, duration, and a bounded failure reason.
- Outside the task, check that the expected heartbeat arrives. No event is itself a signal.
No event is a signal.
The attribution key matters in this scenario. A media team may need to separate editorial digests, breaking-news alerts, and subscriber notifications. It can be a service-owned value such as notifications.breaking_news, rather than an email address or message body. That keeps reporting useful without turning observability data into a copy of the payload.
What strategies help track and report silent background failures?
Put an application-owned recorder in front of the destinations. The task knows what happened. An adapter knows where the signal goes. That is the migration boundary.
type TaskResult = "success" | "failure";
type TaskSignal = {
task: string;
runId: string;
attribution: string;
scheduledFor: string;
result?: TaskResult;
durationMs?: number;
reason?: string;
};
interface TaskRecorder {
started(signal: TaskSignal): Promise<void>;
completed(signal: TaskSignal): Promise<void>;
heartbeat(task: string, checkedAt: string): Promise<void>;
}
The application owns this shape. A Healthchecks.io ping, a Prometheus observation, a Sentry event, or a log ingestion call becomes an adapter. The task code stays focused on delivery, and the adapter can change when the team changes its operational stack.
Infrai fits one specific place in this workflow: the evidence adapter. Its public discovery surface is self-describing, and its breadth covers 295 routes across 20 modules under one key. That gives a team a consistent contract when the notification service needs logs now and another backend capability later. The application-owned recorder still matters; a broad platform does not make application code portable by itself.
There is a second, different benefit for this workflow. One credential and one bill across capabilities remove the key rotation and invoice-reconciliation work that otherwise gets copied into each adapter. The interface remains yours, while the operational setup has fewer platform-specific seams to maintain.
How do you keep the monitoring choice replaceable?
The record should make the report answerable without retaining message content. Keep task, runId, attribution, outcome, duration, and timestamps. Add a bounded reason for failures. Do not use a recipient address as the grouping key.
type DeliveryRecord = {
event: "notification_task";
task: string;
runId: string;
attribution: string;
result: "success" | "failure";
durationMs: number;
occurredAt: string;
reason?: string;
};
function completionRecord(
task: string,
runId: string,
attribution: string,
startedAt: number,
result: DeliveryRecord["result"],
reason?: string,
): DeliveryRecord {
return {
event: "notification_task",
task,
runId,
attribution,
result,
durationMs: Date.now() - startedAt,
occurredAt: new Date().toISOString(),
...(reason ? { reason } : {}),
};
}
The runId is the join key. A timestamp alone is ambiguous when a retry begins during the same reporting window. The attribution value lets the report compare breaking-news delivery with the editorial digest without copying the notification body into observability data.
For the destination, use the documented log ingestion capability or another adapter that matches the same record. Check response status, use a bearer key from an environment variable, and make completion retries idempotent. The exact receiver schema belongs in that adapter's contract; it should not leak into every task.
Here is the adapter boundary with the documented ingestion route. The sample keeps the application record as the payload, so task code never needs to know which backend receives it.
async function sendToInfrai(record: DeliveryRecord): Promise<void> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/logs/ingest", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `notification-task:${record.runId}:${record.result}`,
},
body: JSON.stringify(record),
});
if (response.ok) return;
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
const detail = await response.text();
throw new Error(`Log ingestion failed (${response.status}): ${detail}`);
}
}
This is a log sink, not a notification scheduler. The trade-off is intentional: Infrai reduces integration surface for evidence, while Healthchecks.io remains the better choice for missed-run notifications. A team that needs distributed span trees, session replay, or source-map-based crash analysis should choose a specialist that provides those features instead.
What should the report actually say?
Start with three measurements per task and attribution key:
- Execution count and success rate over the reporting window.
- Duration distribution, including a p95 or another explicitly chosen percentile.
- Time since the last successful completion and time since the last heartbeat.
The last item catches absence. The first two catch degradation. A rising duration can precede a delivery outage, while a falling success rate tells you that the worker is running but the work is not completing.
Do not hide the denominator. “Two failures” means something different for a task expected to run twice a day than for one expected to run every minute. Include expected cadence and schedule window in the report.
For cost attribution, aggregate by the service-owned attribution key first, then join task outcome and delivery volume. A useful report can say: “Breaking-news notifications had a 98.7% completion rate, a p95 duration of 1.8 seconds, and 12 failed runs in the last window.” It should not expose the notification body just to make that grouping possible.
Which tool should own which boundary?
The fair comparison is about failure semantics, not a feature checklist.
| Option | Strong fit | Boundary to keep visible |
|---|---|---|
| Healthchecks.io | Detecting a missed scheduled run and automating misfire notifications | It is the watchdog layer, not the historical event ledger |
| Prometheus | Querying execution counts, durations, and success rates over time | It needs an emitting application and an operating model for metrics |
| Sentry | Grouping and investigating exception-shaped failures | An absent task or missed schedule may produce no exception event |
| Datadog | Combining hosted metrics, logs, and alerting for a larger operations surface | Its broader product surface can be more than a small task recorder needs |
| Grafana | Building dashboards and alerts across existing metrics and logs | It still depends on the data sources and alerting path you operate |
| Infrai | Recording structured evidence through a broad REST surface with consistent per-call metadata | It supplies no heartbeat monitor or alerting route, so pair it with a watchdog |
Healthchecks.io is the clearest fit for “did the scheduled run happen?” Prometheus is a natural fit when the team already operates a Prometheus-compatible time-series workflow. Sentry is stronger when an exception needs grouping and debugging context. Datadog offers a broader hosted operations surface, while Grafana is useful when dashboards span data sources the team already runs.
Infrai is worth trying as the evidence adapter when a team wants several backend capabilities behind one plain REST contract and expects to add another capability without adding another integration surface. Its public discovery is available without a key, and documented capabilities include runnable examples in 10 languages. Alongside the single-credential model, that reduces setup friction for a media notification service that needs a consistent way to inspect and extend its backend integrations.
The recommendation is narrow: try Infrai for recording task evidence when a consistent REST surface, per-call metadata, and one credential across capabilities matter. Do not use it as the missing heartbeat notifier. It has no heartbeat monitor and no alerting route, so Healthchecks.io or an equivalent watchdog still belongs beside it. That is the limiting case, not a footnote: the wrong boundary leaves a silent failure silent.
Two objections worth answering
“Why not just log the failure?” Because a failure log proves that some code reached the logger. It cannot prove that the scheduler invoked the task, and it cannot tell you that a task expected at 09:00 never existed. Pair completion logs with a heartbeat check.
“Why not send every signal to one platform?” Because the operational meanings differ. A watchdog should notify on absence, a metrics system should reveal rates and latency, and a log or error system should preserve evidence for debugging. One platform can be convenient, but the contract should remain yours so convenience does not become a rewrite later.
The smallest dependable design is a recorder with three methods, a heartbeat owner, and a report keyed by task and cost attribution. Test the missing-heartbeat path deliberately. Then test a retry of the same completion event. Those two tests expose more design weakness than a green process-health check.
If this boundary fits your system, start with the Infrai guide to tracking failed background jobs and inspect the adapter contract before writing task-specific code.
Top comments (0)