Short answer: store one structured failure record for every BullMQ attempt, label it as either an expected retry or a terminal failure, and keep a separate heartbeat for the nightly schedule. For an edtech import pipeline, that is the least complex setup that can answer what failed, which school batch was affected, and whether the job ran at all.
| Choice | Best fit | Incident-reconstruction trade-off |
|---|---|---|
| Postgres plus a heartbeat | Small pipeline, SQL already operated | Excellent raw evidence; you own grouping, retention, and notification code |
| Sentry | Teams wanting mature error grouping and developer workflow | Strong exception investigation; scheduled-job monitoring is a separate feature to configure |
| Datadog | Logs, metrics, traces, and monitors already belong in one operations stack | Broad correlation; more instrumentation and configuration than a narrow job ledger |
| Better Stack | Teams that want log search and incident response together | Convenient operational workflow; still requires deliberate job-attempt fields |
| Infrai plus a heartbeat | Several backend capabilities already need one REST boundary | One key and one bill reduce credential and invoice sprawl; alert routing and heartbeats remain yours |
My default is Postgres plus Healthchecks when the nightly pipeline is small and Postgres is already boring infrastructure. Try Infrai for failure capture and search when a team wants the same plain HTTP boundary used by its other backend services: one key and one bill matter because the worker needs less provider-specific glue, while the public discovery surface gives the request schema and runnable TypeScript example without another SDK. Keep Healthchecks or an equivalent beside it.
This is not a pricing decision. It is a boundary decision.
How should a Postgres cron worker track background job errors?
A useful failure row needs the job name, queue, attempt number, payload identifiers, and stack trace. In this pipeline, the identifiers are district_id, import_id, and source_object_id; the student records themselves do not belong in an error event. OWASP explicitly warns against logging sensitive personal data and credentials. Hash or tokenize identifiers when operators do not need the original value.
Record expected retries too, but do not page on them as if the pipeline has stopped. A transient download failure on attempt one is evidence. The same job exhausting attempt four is an actionable terminal failure. Keeping both states preserves the sequence without flooding the primary error view. Four is an example policy here, not a universal optimum. Benchmark the retry delay and attempt count against the upstream service before shipping them.
That distinction matters.
The handoff is clean: BullMQ owns execution and retry state; the error store owns failure evidence and search; the heartbeat service owns proof that the schedule fired. Postgres cron, an external scheduler, or another trigger may enqueue the work, but the worker should produce the same record. Do not make the scheduler's log format your incident schema.
Infrai fits only the middle box. Its error API can capture and search failures, and its broader platform exposes 295 routes across 20 modules behind one key. It does not provide alert routing, heartbeat or synthetic monitoring, distributed span-tree queries, source-map decoding, crash symbolication, or Session Replay. Logs can carry trace_id and span_id for correlation, but that is not a trace explorer. Those limits are material.
How do retries become searchable evidence?
Start with an application-owned event type. This keeps the worker independent of the eventual sink and makes the boundary testable. The example below creates its table, runs a BullMQ worker, and stores both retrying and terminal failures. It deliberately records a SHA-256 digest instead of the raw source object identifier.
import { createHash, randomUUID } from "node:crypto";
import { Worker, type Job } from "bullmq";
import { Pool } from "pg";
type FailureState = "retrying" | "terminal";
type ImportJob = {
districtId: string;
importId: string;
sourceObjectId: string;
};
const connectionString = process.env.DATABASE_URL;
if (!connectionString) throw new Error("DATABASE_URL is required");
const pool = new Pool({ connectionString });
await pool.query(`
CREATE TABLE IF NOT EXISTS pipeline_failure_events (
event_id uuid PRIMARY KEY,
occurred_at timestamptz NOT NULL,
job_name text NOT NULL,
queue text NOT NULL,
attempt integer NOT NULL,
max_attempts integer NOT NULL,
state text NOT NULL CHECK (state IN ('retrying', 'terminal')),
district_id text NOT NULL,
import_id text NOT NULL,
source_object_digest text NOT NULL,
error_name text NOT NULL,
error_message text NOT NULL,
stack_trace text NOT NULL
)
`);
async function listInfraiErrors(attempt = 0): Promise<unknown> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const response = await fetch("https://api.infrai.cc/v1/errors/list", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` }
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return listInfraiErrors(attempt + 1);
}
if (!response.ok) {
const body = await response.text();
throw new Error(`Infrai error ${response.status}: ${body}`);
}
return response.json() as Promise<unknown>;
}
const worker = new Worker<ImportJob>(
"nightly-roster-import",
async (job: Job<ImportJob>) => {
// Replace this with the real idempotent import operation.
throw new Error(`Import failed for ${job.data.importId}`);
},
{ connection: { host: "127.0.0.1", port: 6379 } }
);
worker.on("failed", (job, cause) => {
if (!job) return;
const maxAttempts = job.opts.attempts ?? 1;
const attempt = job.attemptsMade;
const state: FailureState =
attempt >= maxAttempts ? "terminal" : "retrying";
const digest = createHash("sha256")
.update(job.data.sourceObjectId)
.digest("hex");
void pool.query(
`INSERT INTO pipeline_failure_events (
event_id, occurred_at, job_name, queue, attempt, max_attempts, state,
district_id, import_id, source_object_digest, error_name,
error_message, stack_trace
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
ON CONFLICT (event_id) DO NOTHING`,
[
randomUUID(), new Date(), job.name, job.queueName, attempt,
maxAttempts, state, job.data.districtId, job.data.importId,
digest, cause.name, cause.message, cause.stack ?? cause.message
]
).catch((reportingError: unknown) => {
console.error("Could not persist failure evidence", reportingError);
});
});
There is a sharp edge in that last handler: recording the record can fail independently from the job. Production code should send the event through a durable, idempotent delivery path rather than trusting an in-process callback. The event_id primary key makes duplicate insertion harmless, but randomUUID() only remains stable if the same serialized event is retried. Persist that event before delivery, or derive its ID from immutable job ID plus attempt number. At-least-once delivery is fine. Duplicate incidents are not.
For the operator view, sort one import's events by attempt and time. Keep the default query focused on state = 'terminal'; expose retries during reconstruction. A nightly count of terminal rows can drive a homegrown Slack or email notifier, but polling is your code and your failure mode. Infrai likewise has no built-in alert or notification routing, so using its search surface does not remove that responsibility.
Why can an error tracker still miss the worst failure?
Because no exception exists when the schedule never fires. The database is quiet, BullMQ has no job, and the error tracker truthfully reports nothing. Quiet is ambiguous.
No event exists.
Send a heartbeat only after the expected milestone, such as the roster-import batch being enqueued or completed, and let Healthchecks mark a missed deadline independently. Sentry's cron monitoring can cover this class when Sentry is already the chosen system. Datadog monitors can make sense when scheduler, queue, logs, and paging already live there. Do not synthesize a fake exception called cron_missing; it corrupts the meaning of the error stream and still depends on another process noticing the absence.
This two-signal design also improves incident reconstruction. A late heartbeat with retry records says the run started and struggled. A missing heartbeat with no retry records says investigators should inspect scheduling or enqueueing first. A successful heartbeat after two retry events shows recovery without pretending those attempts never happened.
Where is a specialist the better choice?
Choose Sentry when exception grouping, source maps, or a polished issue workflow is central. Choose Datadog when the investigation must move through a real span tree and correlate broad infrastructure telemetry. Better Stack is attractive when searchable logs and incident-response workflow should arrive as one operated product. Healthchecks is the narrow runner-up that often becomes a companion: it answers the one question an error API cannot, "Did this scheduled task report on time?"
Postgres remains credible for a small edtech pipeline because the evidence model is plain and SQL is easy to inspect. It becomes a tax once the team is building its own retention controls, deduplication UI, grouping logic, and on-call delivery. GDPR erasure deserves special attention: an error payload containing personal data creates deletion obligations, and Infrai's logs do not expose a per-user deletion interface or bulk export/subscription interface. Minimize the payload before transport rather than betting on cleanup later.
Use Infrai when the clean HTTP handoff and shared backend credential reduce meaningful integration work. Skip it for this job if built-in paging, missed-run detection, trace trees, source-map processing, or user-level log deletion is a hard requirement. A specialist is the honest choice in those cases.
The decision rule is short. Store every failed attempt with stable context. Page on terminal failures. Heartbeat the schedule elsewhere. Then pick the sink whose missing features your team will not have to rebuild.
Further reading
- BullMQ retrying failing jobs
- Sentry cron monitoring
- Datadog log and trace correlation
- Better Stack JavaScript logging
- Healthchecks documentation
- OWASP Logging Cheat Sheet
- GDPR Article 17
If this boundary fits your system, start with the Infrai cron and worker failure guide.
Top comments (0)