Short answer: For a Postgres-backed cron worker, background job error tracking should page when a due import produces no durable result after its retry window, while treating individual exceptions as supporting evidence.
For a B2B SaaS import pipeline, track three separate facts: the schedule was due, an attempt ran, and a result was committed. Page only when the first fact exists and the last one does not.
I've carried a pager through alerts that meant nothing and through the much worse case where the signal that mattered never arrived. That changes the first question in a postmortem. I don't ask which dashboard looked red. I ask: what page fired, and did it describe lost customer work?
Consider a bounded, synthetic incident: tenant 8421 has a Postgres-backed import due every 15 minutes. The cron worker claims the job at 02:00, the parser rejects one upstream record, and the retry policy schedules three more attempts. A generic error counter rises four times. A queue-depth alert may never move because each attempt is claimed promptly. Meanwhile, the tenant sees no new rows. Paging on each exception creates four pages for one customer-visible failure; paging on queue depth creates none. The invariant is narrower: after the due time plus the allowed completion window, there must be exactly one terminal outcome tied to that scheduled run.
Quiet is a symptom.
Retries can hide it.
What should error tracking capture for a cron worker retry failure?
Capture one immutable identity for the scheduled run, then attach every attempt and terminal outcome to it. In a Node.js worker using BullMQ or Agenda, the library's job identifier can help correlate attempts, but the business run still needs its own stable key because queue mechanics and customer outcomes answer different questions. A practical key is (tenant_id, import_id, scheduled_for). Store it in Postgres under a uniqueness constraint so a retry cannot invent a second logical run.
The event record needs enough context to answer the page without exposing the imported payload: run_id, tenant and import identifiers, scheduled time, attempt number, worker release, outcome class, duration, and an error category. Keep credentials, access tokens, raw files, and sensitive field values out of logs. OWASP's logging guidance treats security-relevant logging, data to exclude, sanitization, and log access as design concerns; this is why an opaque source-record fingerprint is safer than copying the failed record into an exception message.
Use terminal states with operational meaning. succeeded means the expected result was durably committed. permanent_failure means retry policy is exhausted or the error is classified as non-retryable. cancelled means an authorized operator or policy ended the run. An attempt-level error is not terminal while another retry remains. This distinction stops four failed attempts from looking like four broken imports.
A result can also be succeeded_zero_rows, but only if zero rows is a legitimate, observable business outcome. Don't silently merge it with success when a normal feed always contains records. I'm not sure a universal zero-row threshold exists; the import owner must resolve that using the source contract and historical expectations, not a generic monitoring default.
| Signal | What it proves | Page on it? |
|---|---|---|
| Schedule due | Work should exist | No; use it to start the clock |
| Attempt started | A worker claimed work | No; useful for diagnosis |
| Attempt failed | One execution failed | Usually no while retries remain |
| Result committed | Customer-visible work completed | Clear the pending condition |
| Terminal failure | No retry will produce this run's result | Yes |
| Missing terminal outcome | The completion window expired without a result | Yes |
How should a Node.js cron worker capture Postgres retry failures?
Treat scheduling, execution, and result accounting as separate state transitions, even if one process performs all three. The same shape applies to a Node.js cron worker, BullMQ processor, or Agenda job: first insert the logical run idempotently; then lease and execute an attempt; finally commit a terminal outcome in the same database transaction as the imported result when that is feasible. The example is in Go because the state machine matters more than the queue client.
The preventative path below records a result only after applyImport commits its domain change. A retryable error records the attempt and returns it to the caller, which can schedule the next attempt. Exhaustion writes one terminal failure. The unique run key prevents overlapping schedulers from creating two logical imports.
package imports
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
)
type Run struct {
ID string
TenantID int64
ImportID int64
ScheduledFor time.Time
Attempt int
MaxAttempts int
}
type Store struct {
DB *sql.DB
}
func (s *Store) Execute(ctx context.Context, run Run) error {
tx, err := s.DB.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("begin import transaction: %w", err)
}
defer tx.Rollback()
if err := recordAttempt(ctx, tx, run); err != nil {
return err
}
rows, err := applyImport(ctx, tx, run)
if err != nil {
category := classify(err)
terminal := run.Attempt >= run.MaxAttempts || !retryable(category)
if logErr := recordFailure(ctx, tx, run, category, terminal); logErr != nil {
return errors.Join(err, logErr)
}
if commitErr := tx.Commit(); commitErr != nil {
return errors.Join(err, commitErr)
}
return err
}
if err := recordSuccess(ctx, tx, run, rows); err != nil {
return err
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit import result: %w", err)
}
return nil
}
The abbreviated helpers must use parameterized SQL and constrained categories rather than embedding raw exception text. recordSuccess belongs in the import transaction, so the customer result and the observability result agree. recordFailure can persist the category and attempt metadata without preserving the source row. If a process exits before it can record either outcome, the separate missing-outcome detector still catches the run after its deadline. That last path matters: exception capture can observe code that reports an error, but it cannot prove that every due job ever started.
For retry timing, store next_attempt_at and max_attempts with the run rather than reconstructing policy from log timestamps. Give each attempt a lease expiration, and let a reconciler recover expired leases idempotently. A deploy can then interrupt an attempt without converting a transient process boundary into duplicate customer data. Test the transitions directly: two schedulers racing must create one run; two workers racing must obtain one active lease; a successful commit followed by redelivery must not apply the import twice; and an exhausted attempt must create one terminal outcome.
Why do worker error alerts miss a silent scheduled import?
Error trackers answer, "which reported exceptions resemble each other?" The pager question is, "which expected customer result is absent?" Those sets overlap, but neither contains the other. A malformed source record may generate a loud exception while a retry later succeeds inside the completion window. Conversely, a disabled schedule, a clock or lease mistake, or a worker that never claims the run can leave no application exception at all. The detector therefore has to compare expected runs with terminal outcomes, not merely count failures.
Run that detector independently of the worker. At each check, select scheduled runs whose deadline has passed and which have neither success nor terminal failure, then emit one deduplicated alert keyed by run_id. The alert should say which tenant import is late, when it was due, when its deadline expired, the last observed attempt, and the owning service. It should not repeat on every polling cycle. Update the same incident until the run reaches a terminal state.
This is where signal quality beats volume. A warning can record an individual retry. A ticket can collect repeated permanent failures for a noncritical integration. A page is reserved for a breached result deadline or a terminal failure on an import whose service objective requires immediate response. Those routes should be tested with synthetic run records during deployment, including one due run with no attempt, one retry that later succeeds, and one exhausted run. Verify the notification identity as well as the query; a correct detector feeding a deduplication key that changes every minute is still noisy at 3am.
Dashboards remain useful for trends such as completion latency, retry rate, and outcomes by category. They are evidence during diagnosis. They are not the contract. The contract is the expected result and its deadline.
One run. One outcome.
There is also a deletion problem. Long-lived operational records can become a shadow copy of customer data if error messages contain imported fields. Define retention separately for attempt metadata, diagnostic logs, and customer payloads; keep the join possible through opaque identifiers; and make erasure workflows able to locate personal data where it is actually stored. GDPR Article 17 describes the right to erasure and its conditions and exceptions. It does not turn indiscriminate logging into a sound design. Data minimization at capture time makes later handling tractable.
Where does this incident rule stop working?
The catch is that a missing-result page is not suitable for work with no declared schedule, no meaningful deadline, or intentionally best-effort delivery. For ad hoc exports, page on a user-visible request deadline instead. For continuous streams, use progress or freshness against an agreed watermark rather than manufacturing cron runs. For low-criticality batch work, open a ticket after the deadline instead of waking someone.
Stick with attempt-level paging when one failed attempt is itself a security or safety event and a later retry cannot undo the exposure. Keep queue-depth paging when backlog threatens a capacity limit across many jobs, but don't confuse that with proof that a particular tenant import stopped producing results. These signals can coexist because they protect different invariants.
No single retry count or grace period is correct for every import. Start from the customer promise, subtract enough time for diagnosis or recovery, and make the remaining window explicit in stored policy. Then rehearse the failure modes. If the team cannot explain why a specific page fires, which result is missing, and what action is available before the promise is breached, the page isn't ready.
Top comments (0)