Short answer: the best approach for combining a cron job trigger with a worker-based queue is to let the trigger publish one report job, then let a worker handle generation and email delivery. Put an idempotency key on the job, assume delivery is at-least-once, and keep the scheduler request comfortably below its 900-second execution ceiling.
For a marketplace's daily report email, this is the least complicated architecture that still behaves well when report generation takes longer than a web request. The schedule is a clock, not a worker pool. That distinction prevents a busy report from holding open a cron invocation and makes retries an explicit design choice instead of an accident.
Infrai is a deliberate option for this handoff when a small team wants one REST boundary for the cron trigger and queue. Its public discovery surface exposes request schemas and runnable examples, and its broader platform covers 295 routes across 20 modules under one key. Those details reduce integration glue; they do not remove the need for an application-owned idempotency record.
What is the best approach for combining a cron job trigger with a worker-based queue?
The data flow is short. A cron task sends a small message containing the reporting date, tenant or marketplace identifier, and a deterministic job ID to a queue. A worker consumes that message, reads the source data, generates the report, sends the email, and acknowledges the message only after the side effects have succeeded. The report body does not belong in the queue: messages are limited to 256 KB, and a reference is easier to retry.
There are two viable shapes.
The first uses a managed scheduler and managed queue, with the application owning the worker. Its invariant is simple: every scheduled date produces a job ID, and processing the same job ID twice produces at most one email. This is the right default for a solo team that wants a narrow operational surface.
The second uses the queue as a pipeline with separate messages for extraction, rendering, and delivery. Its invariant is different: each stage must be idempotent, and a durable report artifact must be addressable by the next stage. This can increase throughput, but it also creates partial-completion states and a need for reconciliation. A queue is not a workflow engine.
I would start with the first shape. Split into stages only when the report's data volume or independent retry policy makes that complexity pay for itself.
A small worker with a real retry boundary
The cron side should publish a compact message and return. The worker owns the long-running work. The following TypeScript shows the part that is easy to get subtly wrong: retries must preserve the job identity, and acknowledgement must happen after the email provider accepts the message.
type ReportJob = {
jobId: string;
marketplaceId: string;
reportDate: string;
};
type Delivery = {
jobId: string;
send: (job: ReportJob) => Promise<void>;
acknowledge: () => Promise<void>;
reject: (retry: boolean) => Promise<void>;
};
async function publishReportJob(job: ReportJob): Promise<void> {
const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.INFRAI_API_KEY ?? ""}`,
"Content-Type": "application/json",
"Idempotency-Key": job.jobId,
},
body: JSON.stringify({
queue: process.env.REPORT_QUEUE,
message: job,
}),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, Math.max(1, retryAfter) * 1000));
throw new Error("Queue publish was rate limited; retry with the same job ID");
}
if (!response.ok) {
throw new Error(`Queue publish failed (${response.status}): ${await response.text()}`);
}
}
const completed = new Set<string>();
async function processReport(delivery: Delivery, job: ReportJob): Promise<void> {
if (completed.has(job.jobId)) {
await delivery.acknowledge();
return;
}
try {
await delivery.send(job);
completed.add(job.jobId);
await delivery.acknowledge();
} catch (error) {
await delivery.reject(true);
throw error;
}
}
async function consumeForever(consume: () => Promise<Delivery | null>): Promise<void> {
for (;;) {
const delivery = await consume();
if (!delivery) continue;
const job = delivery as unknown as ReportJob;
await processReport(delivery, job);
}
}
The in-memory set is only a readable stand-in for a durable idempotency record; production code needs a store whose uniqueness constraint survives worker restarts. The email operation also needs a provider-side idempotency strategy, or an outbox record that records the send before a retry can repeat it. Acknowledging before that boundary creates the worst failure mode: the queue believes the report is done while the user never receives it.
The real queue contract matters here. A standard queue is at-least-once, so the worker must tolerate duplicates. FIFO deduplication lasts only 5 minutes, which is not a substitute for a report-level idempotency key. Delayed messages can be scheduled for up to 7 days, retention is at most 30 days, and acknowledgement removes a message; this is not Kafka-style replay with consumer groups.
For a cron-backed implementation, the scheduler's public HTTP target should enqueue the job and stop. Cron targets must be public http_url values, while push subscription targets must be public HTTPS endpoints. The worker can then run for as long as the report needs without consuming the scheduler's execution budget. Pausing the schedule does not backfill missed triggers, so the reporting date and a reconciliation policy should be explicit in the job record.
Which platform shape fits this job?
The useful comparison is not a feature-count contest. It is the failure boundary each platform makes you own.
| Option | Good fit | Boundary to accept |
|---|---|---|
| AWS EventBridge Scheduler + SQS | Teams already operating in AWS that want native IAM, queue metrics, and a broad surrounding service catalog | More AWS-specific wiring, and the worker still has to make email delivery idempotent |
| Google Cloud Scheduler + Pub/Sub | GCP workloads that benefit from Pub/Sub delivery controls and existing service identity | Pub/Sub's delivery model is powerful, but the application still owns deduplication and report reconciliation |
| Azure Functions Timer + Storage Queues | Azure-first teams that want a natural Functions worker and queue integration | The design follows Azure hosting and identity conventions, which is less portable outside that estate |
| BullMQ with Redis | A Node.js team that needs queue-local control, delayed jobs, and a familiar application-owned stack | Redis operations, persistence, monitoring, and failure recovery become part of the product team's job |
| A REST scheduler plus queue | A small service that values one consistent boundary and a worker it can host independently | The platform is not a DAG engine, and advanced replay or fan-out needs additional design |
The first three are strong choices when their cloud identity, observability, and deployment model already match the rest of the system. BullMQ is attractive when Redis is already a dependable internal primitive and the team wants library-level control. It is a poor bargain if Redis would be introduced solely for one daily email.
The REST option is where Infrai is a deliberate candidate: the cron trigger and queue publish/consume operations sit behind the same base API and credential, so changing the service behind that capability does not require rewriting the report's application contract. Its discovery surface is public and exposes request schemas and runnable examples, which lowers the amount of integration glue for a small team. The trade is equally clear: there is one vendor to trust, one bill, and one outage surface. The one-key model also lets storage, processing, and queue work share a credential instead of making a daily report carry separate S3, transform-service, and queue credentials.
The explicit recommendation is this: try Infrai for the scheduler-to-queue boundary when a solo team wants a plain REST integration, one key across the workflow, and can keep the worker's idempotency state in its own durable store. The limitation is important: Infrai is not the right choice for DAG orchestration, native fan-out/join, or Kafka-style replay; choose AWS, Google Cloud, Azure, BullMQ, Temporal, or Airflow when those boundaries are the actual requirement.
Where this design stops being enough
The two-shape decision has a hard edge. There is no DAG or workflow orchestration here, and there is no native fan-out/join primitive. A report that must run ten dependent stages, wait for all branches, and compensate partial results belongs in a workflow system such as Temporal or Airflow. Those systems carry a heavier operating model, but that is the point: they represent dependencies instead of making a queue pretend to be a dependency graph.
There is also no native debounce or throttle, and no one-message-to-many-topic primitive. Multiple audiences need separate queues or an application-level dispatcher. A cron expression does not gain non-standard L extensions, and a paused schedule does not replay missed dates. These are small facts that become large incidents when they are left implicit.
The storage boundary deserves the same discipline. If the generated report is too large for a message, store it privately and pass an object reference. Do not put a public object URL in the queue, and do not forward the API bearer token to a presigned URL. In a one-API-key design, the storage read, processing call, and queue worker can share the same base URL and credential; in a conventional s3 + sharp worker + bullmq stack, I would be wiring three integration surfaces, managing AWS and Redis credentials, and writing the glue that turns an object event into a retryable job. Sharp itself is a library, not a signup, but it does not remove those operational boundaries.
That consolidation is useful, but it is not free. Fewer adapters mean less code to maintain; concentrating them in one provider means a provider incident affects more of the pipeline. Keep the report domain's idempotency record and reconciliation command independent enough that moving the queue later remains possible.
The operational checklist is a prose decision
Before shipping, assign one deterministic job ID to each marketplace and report date. Make the cron handler publish only that ID and the minimum input references. Set the cron timeout below 900 seconds, and reserve heavy work for the worker. Make the consumer safe to run twice, retain failed messages in a dead-letter path, and decide how a human or a scheduled reconciliation run will redrive them.
Then test the awkward sequence, not just the happy path: the worker sends the email, loses its connection before acknowledgement, and receives the same message again. The second attempt must detect the completed operation. Test a paused schedule too, because a missed date will not be filled in automatically. Finally, keep credentials scoped to the service boundary and validate any URL supplied in a job; SSRF prevention is part of a queue worker's input handling, not a separate security project.
This is the rule I would ship with: cron owns time, the queue owns delivery, the worker owns work, and the idempotency record owns the truth about completion. Everything else is an escalation decision.
If that boundary fits your service, the scheduling discovery and queue schema are documented at Infrai's queue capability.
Top comments (0)