TL;DR: Treat notification delivery as a chain of scheduled intent, worker execution, provider outcome, and business ownership. Emit one small, typed event at each boundary. Join those events by delivery ID, then alert on missing transitions as well as thrown exceptions. That catches the quiet cases: a cron job that never starts, a worker that acknowledges too early, and an API that returns success before useful work happens.
Exception capture alone cannot prove that scheduled work ran. A healthcheck alone cannot explain which team or tenant owns the failed spend. The useful unit is a delivery attempt with an owner, a channel, a terminal outcome, and enough timing data to distinguish waiting from working.
How should backend error tracking connect cron jobs and workers?
Some failures throw. Those are easy. Others are absences: no scheduler event, no worker start, or no terminal delivery result. There is no exception to capture because the expected transition never occurred.
Silence wins.
A notification API can also accept work successfully while the later delivery fails. Treating the HTTP response as the final outcome collapses two different facts into one green check. That makes search fast and diagnosis weak.
The monitoring model should cover latency, traffic, errors, and saturation. Those four signals are a useful baseline from the Google SRE monitoring chapter. For this service, map them to queue age, attempts by channel, terminal failures, and worker capacity. Then add one domain signal: overdue state transitions. It exposes silence without pretending every silence is an exception.
Cost attribution changes the design. Counting errors by endpoint is not enough. Every attempt needs a stable owner key before it enters the queue, because reconstructing ownership later requires glue, and glue tends to become permanent config.
Build the smallest useful event trail
Start with one schema. Keep it boring. A discriminated union prevents impossible combinations while leaving the storage backend open.
type DeliveryStage =
| "scheduled"
| "worker_started"
| "provider_accepted"
| "delivered"
| "failed";
type DeliveryEvent = {
deliveryId: string;
ownerId: string;
channel: "email" | "sms" | "push";
stage: DeliveryStage;
occurredAt: string;
durationMs?: number;
errorClass?: string;
};
interface EventSink {
write(event: DeliveryEvent): Promise<void>;
}
ownerId is mandatory. Do not substitute a display name or an email address. The field groups operational evidence and attributed usage under one stable boundary; the mapping to human-readable metadata belongs elsewhere.
Instrument the worker around the real side effect. Record acceptance separately from delivery because they answer different questions. This interface avoids coupling the evidence model to a vendor SDK.
type SendResult = { accepted: boolean; providerCode?: string };
interface DeliveryProvider {
send(deliveryId: string): Promise<SendResult>;
}
async function runDelivery(
sink: EventSink,
provider: DeliveryProvider,
base: Omit<DeliveryEvent, "stage" | "occurredAt">,
): Promise<void> {
const started = Date.now();
await sink.write({ ...base, stage: "worker_started", occurredAt: new Date().toISOString() });
try {
const result = await provider.send(base.deliveryId);
await sink.write({
...base,
stage: result.accepted ? "provider_accepted" : "failed",
occurredAt: new Date().toISOString(),
durationMs: Date.now() - started,
errorClass: result.accepted ? undefined : result.providerCode ?? "rejected",
});
} catch (error: unknown) {
await sink.write({
...base,
stage: "failed",
occurredAt: new Date().toISOString(),
durationMs: Date.now() - started,
errorClass: error instanceof Error ? error.name : "unknown",
});
throw error;
}
}
This gives exception search a typed failure, gives health logic two timestamps to compare, and makes owner-level grouping possible. It does not claim that acceptance means delivery. A later callback or reconciliation pass must emit delivered or failed.
Do not acknowledge the queue item before the evidence write required by the reliability contract. Otherwise the job can disappear between the side effect and the record. Choose the order explicitly. If duplicate sends are more damaging than missing telemetry, persist an idempotency key and delivery state together before calling the provider. If telemetry loss is worse, make the sink part of the completion path. I would choose the former for user-visible notifications because a duplicate SMS or email escapes the developer-tools boundary and reaches a person; a missing diagnostic event stays inside the operational system. That choice still needs an exact counter outside sampled detail, plus a reconciliation job that finds delivery state with no matching evidence. The opposite choice can be valid for an internal notification with strict audit requirements. There is no universal order, and pretending otherwise hides the real trade-off.
Detect silence with deadlines, not optimism
A scheduler heartbeat says the scheduler ran. It does not say every intended notification reached a worker. Model both facts. For each scheduled delivery, define a deadline for worker_started; for each accepted attempt, define a separate deadline for a terminal result.
No transition by the deadline becomes a synthetic failure event. Label it schedule_overdue or result_overdue, keep the original delivery ID and owner ID, and route it through the same search path as explicit failures. One query can now answer which owners have failed attempts, where the chain stopped, and how long each attempt waited.
Two clocks. Two meanings.
Benchmark this path with operator questions. Use a small acceptance list: can the system find one delivery ID, group failures by owner and channel, distinguish thrown errors from overdue transitions, and show queue age without joining five dashboards? Measure ingestion delay and query latency with a representative event count. Do not borrow a target from a blog post. Set it from the alert deadline and incident workflow.
A custom logging appender can forward events, and the Logback manual documents the appender extension point. The architectural warning is broader than a logging library: logs are transport, not the state model. If parsing prose is required to recover ownerId or deliveryId, the schema has already leaked.
Make attribution useful without turning it into billing
Cost attribution needs counts at defensible boundaries. Track scheduled attempts, provider-accepted attempts, terminal outcomes, and retries by owner and channel. Keep money out of the event schema unless the amount is fixed at event time and comes from an authoritative ledger. Rates change. Operational events should not silently become invoices.
The trade-off is cardinality. Owner and delivery identifiers are essential for search, but putting every delivery ID into metric labels can overload a metrics path. Store high-cardinality identifiers in events or traces. Export bounded metric dimensions such as channel, stage, and error class, then calculate owner summaries in an event query or batch aggregation.
This split keeps alerts readable. Alert on a ratio or overdue count grouped by a bounded operational dimension. Attach representative delivery IDs for investigation. Use owner-level reports for attribution and prioritization, not as the only page trigger.
Before deployment, test four cases: an exception during send, an explicit rejection, a scheduled event with no worker start, and an accepted event with no terminal result. Also replay the same delivery ID. Duplicate evidence should not create a second billable attempt.
What I would change at scale
At higher volume, separate the append-only evidence stream from derived views. One consumer builds low-cardinality service metrics. Another builds searchable delivery timelines and owner summaries. Retention can differ because aggregate health and per-delivery investigation have different value over time.
Version the event schema, enforce allowed transitions, and sample successful detail only after verifying that owner totals remain exact. Failures and overdue transitions should retain enough context to diagnose the boundary. Success sampling without a separate exact counter corrupts attribution.
The decision rule stays plain: choose an error-tracking architecture only if it can ingest structured application events, search stable correlation keys, represent missing transitions, preserve retry semantics, and export data for owner-level aggregation. Evaluate those behaviors with the four failure cases above. Product feature lists are weaker evidence.
Top comments (0)