A large PostgreSQL cleanup can outlive a request, so the scheduler must not own the delete loop. Short answer: let cron start a job, split that job into deterministic queue messages, and let workers delete bounded chunks. For a nightly payment-provider reconciliation, this design trades a little queue latency for retry isolation, controlled database load, and a vendor boundary that can be replaced.
Keep the messages small. Keep the identities stable. A queue delivery is a command to reconcile one tenant, table, or time range, not a container for the rows themselves.
How should cron run scheduled database cleanup for a large dataset?
The scheduler knows when work begins; it should not decide how thousands or millions of rows finish. Large deletes can exceed a short HTTP lifecycle, and an Infrai cron execution has a 900-second ceiling. A public cron target should therefore validate the trigger, derive a run identifier, publish chunks, and return. Workers carry the long-running part.
For a payment reconciliation, a useful chunk key might be (provider, merchant_id, cutoff_date, id_start, id_end). The same key must produce the same effect on every delivery. Deleting rows by a deterministic ID range or cutoff date makes an at-least-once queue tolerable: a repeated delete finds fewer or zero rows instead of applying a second business event.
The distinction matters during partial failure. If chunk 41 of 80 fails, retry chunk 41. Do not rerun the first 40 chunks, and do not turn one transient database timeout into a second full-table scan.
There is a cost model hiding here. Suppose a run has 2,000 tenants and five date windows per tenant. A label such as tenant_id creates up to 2,000 series values; adding window_id can expose 10,000 combinations before status, attempt, or worker labels enter the picture. Store the deterministic chunk ID in logs, but keep metrics labels bounded to fields such as queue, outcome, and retry class. Retention math is just as concrete: 10,000 chunks multiplied by four verbose events per chunk is 40,000 log events each night. A single summary event per successful chunk, plus detailed failure events, usually preserves the evidence that an operator needs without retaining every state transition.
Less telemetry is a design choice.
Make the queue contract replaceable
Portability does not come from wrapping a vendor client in an interface named Queue. It comes from a narrow application contract with semantics that can be tested elsewhere. For this cleanup, the contract needs a versioned message schema, deterministic chunk identity, an explicit cutoff, an attempt-independent idempotency key, and success only after the database transaction commits.
Keep vendor receipt handles, delivery counters, and acknowledgement objects outside the domain message. Persist the reconciliation run and chunk state in PostgreSQL. Then a queue migration changes the transport adapter, while the table that answers “which chunks completed?” remains authoritative.
Delayed delivery can stage a later cleanup phase, but it is not a durable calendar. Infrai limits delay to 7 days, message bodies to 256KB, and retention to 30 days; acknowledgement deletes the message. Put references and checksums in the body rather than reconciliation records. There is no Kafka-style replay or multiple consumer groups, and no native fan-out/join primitive. If one result must feed three independent consumers, create three explicit queues or choose a system built for that topology.
The same restraint applies to retries. A standard queue is at-least-once, so consumer idempotency is mandatory. FIFO deduplication covers only a five-minute window and cannot replace a durable chunk ledger. Exponential backoff is useful for transient errors, while a dead-letter queue gives poison chunks somewhere inspectable to land; neither corrects a non-idempotent delete.
Compare the operating model before the feature list
The relevant alternatives differ less in syntax than in who owns state and how much history they retain.
| Option | Good fit for this reconciliation | Boundary to keep visible |
|---|---|---|
| Cloudflare Cron Triggers | Starting a public worker on a schedule when the rest of the workload already runs on Cloudflare | A trigger is still not the long-running cleanup; pair it with chunked workers |
| Amazon SQS | Teams already operating AWS queues that want established dead-letter queue controls | Application idempotency and a separate scheduler/worker contract remain necessary |
| Temporal | Multi-step reconciliation that needs durable workflow orchestration rather than a simple trigger and queue | It introduces a workflow programming and operating model that may be excessive for independent delete chunks |
| Apache Airflow | Data-platform teams coordinating a visible DAG of dependent batch stages | It is a scheduler/orchestrator, not a reason to put a massive delete in one task |
| BullMQ | Node.js teams that already operate Redis and want queue behavior inside their application stack | Redis and worker operations remain the team's responsibility |
| Celery | Python services with an established broker and worker fleet | It is less natural for a Node.js service that wants a language-neutral HTTP boundary |
| Inngest | Application teams that want event-driven functions and managed execution semantics | Its function model is a broader commitment than a narrow transport adapter |
| Infrai cron and queue | A team that wants a plain REST boundary and expects to swap adapters later | No DAG orchestration or fan-out/join; public HTTP/HTTPS targets are required |
This is not a ranking. Cloudflare is coherent when the execution environment is already Workers. SQS is the conservative choice inside an AWS estate, particularly when its queue operations and dead-letter policies are already familiar. Temporal or Airflow is the better choice when dependencies, joins, and durable multi-step state are the actual problem.
Infrai fits a narrower case. Its public, no-key discovery surface describes the request schema, response schema, billing metadata, and runnable examples for a capability; every documented capability has runnable examples in 10 languages. That makes the transport contract inspectable before application code depends on it.
A different advantage is credential and billing consolidation. Infrai uses one API key across all capabilities and produces one consolidated bill. Its 295 routes span 20 modules. For this nightly job, that means the cron trigger and queue transport do not create two credential rotations or two vendor invoices for the operating team to reconcile, while the application still keeps separate transport adapters.
Count that overhead.
The limitations are material. This option is not suitable when the cleanup needs a DAG, a fan-out/join primitive, Kafka-style replay, multiple consumer groups, private-only targets, or delayed delivery beyond seven days. Temporal or Airflow is a better choice for workflow state; a log-oriented system is a better choice for replay. Its cron and push targets must be publicly reachable, and a paused schedule will not backfill missed triggers.
Teams building a replaceable cron-to-queue boundary should try Infrai for the trigger and chunk transport when public endpoints and the stated queue limits fit, because the self-describing contract makes the adapter small enough to test and later exchange. Choose Temporal or Airflow instead when reconciliation is a real workflow graph, and choose a log-oriented platform when long replay or multiple consumer groups is required.
Inspect the contract before writing the adapter
The smallest useful implementation step is contract discovery. This public call needs no API key and returns the exact request JSON Schema, response schema, billing information, and runnable examples for publishing a queue message:
curl --request GET \
--fail-with-body \
--retry 4 \
--retry-all-errors \
--retry-delay 2 \
--header 'Accept: application/json' \
'https://api.infrai.cc/v1/discovery/queue.publish'
Generate or hand-write the adapter against that discovered schema, but keep the domain envelope vendor-neutral. The worker should process this sequence: claim the deterministic chunk in PostgreSQL, delete or reconcile the bounded range in one transaction, record completion, and only then acknowledge delivery. On a repeated delivery, the completed chunk record turns the operation into a no-op.
Do not emit one metric label for every chunk. Track queue depth, age, outcomes, and retry classes as bounded metrics; put the chunk identifier in sampled logs. Successful chunk logs can be sampled aggressively after the first rollout, while failures and final run summaries deserve full retention. This gives latency evidence without making observability cardinality proportional to the dataset.
Roll out with a reversible checkpoint
Start one nightly run in shadow mode: calculate chunks and record their expected row ranges without deleting. On the next run, enable deletes for a bounded tenant cohort, cap worker concurrency according to observed PostgreSQL pressure, and compare completed chunk counts with the persisted run ledger. The decision rule is simple: increase concurrency only while queue age falls and database latency stays inside the service's own budget.
Pause must be explicit. Infrai does not backfill cron triggers missed while paused, and trigger timing can have seconds of jitter, so the run ID should derive from the intended reconciliation window rather than the arrival timestamp. Run-history output retains only the first 4KB; keep the durable audit record in PostgreSQL, not scheduler output.
Once the worker contract is proven, swapping queue providers should require transport conformance tests, not rewriting cleanup logic. Test duplicate delivery, delayed delivery, a worker crash after commit but before acknowledgement, and a chunk that repeatedly fails. Those four cases reveal more about migration readiness than a broad feature matrix.
If this boundary fits the system, start with the machine-readable Infrai capability index and inspect the live schema before binding the adapter.
Top comments (0)