TL;DR: For a B2B notification service, choose a backend exception API by the evidence it preserves for a rollback: stable error groups, deployment identity, job and tenant context, retry state, and a searchable event trail. Do not ask exception capture to prove that a cron job ran. A process that never started throws nothing, so record scheduled executions separately and make the rollback decision from both streams.
This is a narrow decision with expensive consequences. A worker can fail after an email provider accepts a request, before the local delivery row is committed. Blindly rolling back may replay the message; refusing to roll back may leave every new attempt broken. The useful system is the one that distinguishes those cases quickly without binding application code to its storage backend.
What should a backend exception tracking API show for cron jobs?
Start with the decision, not a feature list. For each exception, preserve a release identifier, durable job identifier, notification channel, attempt number, operation phase, original stack, and causal chain. Tenant context can scope impact, but use an opaque internal ID rather than an email address or message body.
The operation phase matters most. before_send, provider_accepted, and delivery_committed describe different recovery choices. An exception before the send can enter the normal retry policy. An exception after acceptance needs reconciliation against the provider-facing idempotency key before another attempt. An exception after the local commit is evidence of a reporting problem, not proof that delivery failed.
No single field proves the outcome.
A searchable group should collect events that share a corrective action. Grouping every TimeoutError together is too coarse: timeouts while acquiring work and timeouts after a provider request have different replay risks. Grouping on the full message is too fine because request IDs fragment one defect into thousands of groups. A practical fingerprint uses exception type, normalized stack location, worker operation, and phase. Keep volatile values as searchable attributes.
RFC 5424 defines severity values, with lower numerical values representing greater severity. Map them deliberately at the boundary; do not assume an application label has universal meaning. An exhausted delivery attempt can be an error, while a recovered transient attempt may remain a warning. Write down and test that mapping because paging behavior often depends on it.
Instrument the boundary once
The application should emit one small, vendor-neutral event shape. An adapter can forward it to an exception API, log index, or self-hosted collector. This keeps a storage migration out of the delivery loop and makes rollback metadata mandatory.
The runnable TypeScript below separates low-cardinality grouping inputs from high-cardinality search attributes.
type Phase = "before_send" | "provider_accepted" | "delivery_committed";
type FailureEvent = {
fingerprint: readonly string[];
message: string;
stack?: string;
attributes: {
release: string;
worker: "notification-delivery";
jobId: string;
tenantId: string;
channel: "email" | "sms" | "webhook";
phase: Phase;
attempt: number;
};
};
interface ExceptionSink {
capture(event: FailureEvent): Promise<void>;
}
function topFrame(error: Error): string {
return error.stack?.split("\n")[1]?.trim() ?? "unknown-frame";
}
async function reportFailure(
sink: ExceptionSink,
error: Error,
context: FailureEvent["attributes"],
): Promise<void> {
await sink.capture({
fingerprint: [error.name, topFrame(error), context.worker, context.phase],
message: error.message,
stack: error.stack,
attributes: context,
});
}
const stdoutSink: ExceptionSink = {
async capture(event) {
process.stdout.write(`${JSON.stringify(event)}\n`);
},
};
await reportFailure(stdoutSink, new Error("provider response was not committed"), {
release: "2026-10-10.3",
worker: "notification-delivery",
jobId: "job_01J9Z8",
tenantId: "tenant_7F2",
channel: "email",
phase: "provider_accepted",
attempt: 2,
});
The fingerprint has only four components by design. Adding tenant or job IDs would make an event easy to isolate by group, but it would destroy aggregation and raise indexing overhead. Those IDs belong in attributes, where an operator can filter after opening the group.
I would accept that extra search step. The alternative creates one group per job, which looks precise until an operator has to decide whether a release caused one defect or 8,000 unrelated defects. This is a real limitation of the compact fingerprint: two failures at the same frame and phase can share a group even when their payload conditions differ. Raw events therefore have to remain accessible, and the attributes need enough context to split the group during investigation. If policy or storage limits prevent that retention, this grouping scheme is not suitable; use a coarser operational metric for detection and keep detailed failure records in the application data store.
Capture must not turn a delivery failure into a stuck worker. Put a short bound on the adapter, buffer only within an explicit memory or disk budget, and define what happens when the sink is unavailable. The business retry policy stays authoritative. Telemetry reports the failure; it must not silently retry the notification.
How do you catch a job that throws nothing?
You cannot infer non-execution from exceptions. If the scheduler never launches the worker, the runtime has no exception to capture. Excellent grouping and search do not alter that boundary.
Silence wins.
For a system that does not want heartbeat monitoring, keep an execution ledger in the application database. Create a row when a scheduled run is claimed, then close it with succeeded, failed, or partial, timestamps, and the release ID. A periodic query can flag expected windows with no claimed row and runs that never reached a terminal state. This is domain state rather than a pulse sent to an observability service.
The ledger has costs and limits. It adds writes to the primary data path, needs cleanup or partitioning as history grows, and cannot detect a scheduler outage unless some independent process checks expected windows. For work without a durable schedule, a queue-age metric may be a better expectation signal. Exception tracking has the opposite boundary: it gives rich failure detail after code runs, but it is not a replacement for liveness or demand monitoring.
It answers a harder rollback question too. Suppose exception volume rises after release 2026-10-10.3. The ledger shows whether that release processed fewer jobs, while grouped exceptions show where attempts failed. Compare both with the prior release over equivalent scheduling windows. The Google SRE monitoring model gives useful framing: errors and latency matter, but traffic and saturation supply context. For a worker, claimed jobs approximate traffic; queue age and concurrency pressure can expose saturation.
Do not invent a universal numeric rollback threshold. Batch size, tenant mix, retry delay, and traffic change the baseline. Define a policy from normal windows, then require enough completed or failed attempts to avoid reacting to one early event. The trigger should identify release and phase, while the operator checks replay safety for provider_accepted failures.
Evaluate the API with a replay drill
Feature matrices hide the failure mode that matters. Feed each candidate the same synthetic sequence: two errors before send, three after provider acceptance, one after commit, a retried job, and a run that was never claimed. Use two release IDs and repeat one exception with changing request and tenant IDs. Ask an engineer who did not build the fixture to reconstruct events.
The exception system passes if it creates stable groups across volatile IDs, preserves raw events, filters by release and phase, retains causal stacks, and exports data through a documented interface. The missing run should appear only in the execution-ledger query. If an exception dashboard finds that absence, inspect where its independent expectation signal is configured.
Search responsiveness matters, but benchmark it with the projected event shape and retention rather than a demo dataset. So does cardinality. Indexing jobId is useful during an incident; grouping by it is harmful. Test oversized attributes, cause chains, duplicate submissions, ingestion timeouts, and clock skew. Record observed behavior as acceptance evidence.
Cost is a constraint, not the conclusion. Estimate event volume from attempts and failure rates, then include indexed attributes, retention, egress, and operational labor. Sampling can control repetitive failures, but preserve the first event for a new release and enough later events to see phase concentration. Never sample the execution ledger: it detects missing runs.
Operate the rollback path before production
Before release, verify that the build injects one immutable release ID into the worker, exception events, and execution ledger. Trigger a known failure at each delivery phase outside production. Confirm that secrets and message content are absent, groups remain stable when job IDs change, and search isolates the new release. Then stop the scheduler for one expected window and confirm that the ledger query, rather than the exception stream, detects the gap.
Practice the response. Pause new claims, inspect the failure phase, reconcile provider-accepted jobs by idempotency key, choose rollback or forward repair, and resume from a known cursor. Afterward, verify both the exception trend and ledger completion states. A rollback is safe only when the evidence distinguishes failed work from work whose outcome is uncertain.
That rule is more durable than a product checklist. Backend exception tracking compresses many events into a searchable corrective action. The execution ledger covers silence. Together they let a small team ship quickly without treating every spike as permission to replay customer notifications.
Sources
- Google SRE Book, "Monitoring Distributed Systems": https://sre.google/sre-book/monitoring-distributed-systems/
- RFC 5424, "The Syslog Protocol": https://datatracker.ietf.org/doc/html/rfc5424
Top comments (0)