Short answer: capture every terminal checkout failure as a structured error event, keep retry attempts as low-cardinality counters, and join both to the originating order with stable identifiers rather than putting order IDs into metric labels.
The expensive part of background job error tracking is rarely the exception itself. It is the repeated payload: stack traces copied across retries, request context copied into every log, and labels whose cardinality grows with orders, customers, or job IDs. A useful design starts by pricing those bytes and labels before choosing a Node.js worker library or a storage backend.
For a PostgreSQL-backed cron worker, BullMQ, or Agenda, the boundary is the same: one logical checkout job may have several attempts but only one final outcome. Record the attempts cheaply; preserve a richer event when the retry policy is exhausted. This separates alerting from investigation and keeps cost attribution attached to the checkout workflow.
What the observability bill is actually made of
A retention estimate does not need a vendor calculator. Start with four inputs: jobs per day, average attempts per job, bytes per attempt event, and retained days. If a shop runs 400,000 checkout jobs per day, averages 1.08 attempts, emits 900 bytes for each attempt, and retains those events for 30 days, the raw event volume is about 11.7 GB before indexing, replicas, compression, or protocol overhead. These are illustrative inputs, not a benchmark; substitute measurements from a representative production hour.
The multiplication matters because retry behavior affects two terms at once. A downstream timeout can increase both the attempt count and the size of the associated stack-trace logs. Keeping a 12 KB exception, serialized job payload, and request context for every attempt in the same example would raise the raw retained volume to roughly 156 GB. Compression may change the stored total, but relying on an unknown compression ratio is a weak budget. Measure it after ingestion and treat it as a separate factor.
Retry noise compounds.
Count cardinality next. Labels such as service=checkout-worker, queue=payment-capture, environment=production, and outcome=failed have bounded value sets and support useful aggregation. order_id, customer_id, job_id, full error messages, and stack traces do not. They belong in an indexed event field, a sampled log body, or a trace attribute whose storage policy is deliberately constrained. An order ID in a metric label can create one time series per order; the same ID in a terminal error event creates one searchable record per failed job.
Indexes aren't free.
This is the first deliberate omission: don't retain a full exception for every successful attempt or routine retry. Keep a counter by workflow, reason class, and attempt number, then retain the complete failure envelope only at terminal failure and on a small, declared sample of recovered retries. The lost detail has a cost: when a retry succeeds, a rare intermediate stack difference may no longer be available. If transient failures are the investigation target, increase the recovered-retry sample temporarily rather than turning permanent full capture back on.
How should a Node.js cron worker capture background job retry failures?
Treat capture as a state transition, not as a catch block that emits whatever happens to be in memory. The worker should construct a stable job identity when work is scheduled, increment an explicit attempt number, classify the failure, and decide whether another attempt remains. A PostgreSQL cron worker may persist this state in a job table; BullMQ or Agenda may manage parts of the lifecycle. The error-tracking contract should remain independent of that choice.
There are three useful signals. First, increment an attempt counter with bounded dimensions. Second, emit a terminal failure event after the last permitted attempt. Third, mark the job record failed in the same state machine that prevents another worker from treating it as successful. If recording the event and updating job state cannot share a transaction, use an outbox row in the job-state transaction and deliver it asynchronously. That avoids making the checkout result depend on the availability of an observability destination.
Keep the order of operations explicit. Suppose attempt 3 receives an illustrative 409 with reason code CHECKOUT_LOCK_TIMEOUT, and the policy permits three attempts. The worker classifies the reason, records the terminal state plus an outbox event, commits, and acknowledges the job only after that commit. If the process exits after the commit but before acknowledgment, redelivery should find the terminal state and avoid charging the customer twice. Error capture is evidence of the transition; it is not the authority for payment state.
That distinction is small. It prevents expensive ambiguity.
The following example shows the shape of a terminal event sent by an outbox dispatcher to a pseudonymous internal collector. It is deliberately a shell example because the transport contract should work from any worker runtime; the endpoint and values are placeholders for an internal interface, not a public vendor API.
curl --fail-with-body --request POST \
--url "https://telemetry.example.invalid/v1/errors/capture" \
--header "Authorization: Bearer ${OBSERVABILITY_TOKEN}" \
--header "Content-Type: application/json" \
--data '{
"schema_version": 1,
"occurred_at": "2026-08-17T09:42:31Z",
"service": "checkout-worker",
"workflow": "payment-capture",
"job_id": "job_8f3c2a",
"order_ref": "ord_72c9e1",
"attempt": 3,
"max_attempts": 3,
"terminal": true,
"reason_code": "CHECKOUT_LOCK_TIMEOUT",
"status_code": 409,
"duration_ms": 1840,
"exception_type": "CheckoutLockError"
}'
Do not include the checkout payload by default. OWASP's logging guidance warns against recording data such as access tokens, authentication passwords, payment card data, and sensitive personal data directly in logs. An allowlist is easier to audit than a growing denylist: schema version, timestamps, service and workflow names, opaque internal references, attempt counts, a controlled reason code, duration, and exception type. Put stack traces in a separately governed field only when they are required for diagnosis, and scrub them before ingestion.
A failure envelope that supports cost attribution
Cost attribution works when every stored field answers a specific query. workflow assigns volume to checkout. service assigns operational ownership. terminal distinguishes customer-visible exhaustion from recovered retries. reason_code groups failures without storing arbitrary messages. attempt reveals retry amplification. Those five dimensions should have controlled vocabularies; the per-job identifiers stay searchable event fields, never metric labels.
| Question | Signal | Retention choice |
|---|---|---|
| Is checkout failing now? | Terminal-failure counter by workflow and reason class | Keep aggregated series long enough for trends |
| Are retries amplifying volume? | Attempt counter by attempt number and outcome | Keep aggregated series; no stack trace required |
| Which order needs investigation? | Terminal event with opaque order and job references | Keep for the support and incident window |
| What changed inside the exception? | Scrubbed stack trace on terminal events and sampled recovered retries | Use the shortest useful diagnostic window |
The event also needs an idempotency key, even if the compact example omits one to keep the payload readable. Derive it from stable values such as job ID, attempt number, and event kind, then enforce uniqueness at the collector or outbox consumer. Delivery can be at least once without charging the storage budget for duplicate terminal events. Avoid putting that key into a metric label.
Use controlled reason codes rather than exception messages. Messages change with library versions, include variable values, and fragment dashboards. A small taxonomy might separate dependency timeout, concurrency conflict, invalid job input, and exhausted rate limit. Review the count before adding a code; a reason value that embeds an endpoint, order, or customer has quietly become another high-cardinality identifier.
I'm not sure which exact fields deserve indexing in your system because that depends on incident queries and the storage engine's indexing model. The way to resolve it is concrete: take the last few investigations, list the predicates responders actually used, and index only stable fields that materially reduced search time. Your mileage may vary — especially if support searches by order reference while engineers search by reason code — but every index should have a named query owner.
Retention, sampling, and the detail you give up
Set retention by signal class rather than by one global number. Aggregated counters are compact and useful for long trends. Terminal events need to cover the period in which support and engineering investigate a checkout failure. Full stacks have the highest diagnostic density and the greatest risk of accidental sensitive content, so they deserve a shorter window and narrower access. GDPR Article 17 also establishes a right to erasure under specified conditions; keeping opaque references and documented deletion paths makes retention policy enforceable rather than aspirational.
Sampling must preserve the numerator used for reliability decisions. Count every terminal outcome, then sample the heavy diagnostic attachment. Randomly sampling entire failures can make a low-volume reason disappear. A better policy retains all terminal envelopes, all occurrences of a newly introduced bounded reason code for a short evaluation window, and a fixed fraction of scrubbed stacks for common recovered retries. Store the sampling decision and policy version with the event so investigators know what absence means.
Do the math again after the policy change. For each signal class, multiply daily events by retained bytes and days, then add indexing and replication factors measured from the chosen backend. Attribute the result to workflow=payment-capture, not to whichever shared worker happened to emit it. This turns a general observability bill into a checkout decision: perhaps terminal envelopes justify their storage while routine successful-attempt logs do not.
The catch is that terminal-only detail is not suitable when the engineering question concerns retry timing, lock contention across attempts, or a failure that corrupts state before eventual success. In that case, retain structured attempt events for the affected workflow during a bounded diagnostic window, or use traces with controlled attributes. Stick with full per-attempt capture when regulation or a defined audit requirement demands it, but separate that evidence store from general debug logs and apply its own access and deletion controls.
No policy recovers discarded bytes later.
Top comments (0)