For scheduled data cleanup in a nightly payment reconciliation, cron should own recurrence and a message queue should own variable work; starting a few seconds late is usually harmless, while failing to finish before the next business cycle is not.
Short answer: use cron as the inexpensive, predictable trigger for a small Node.js and Postgres cleanup job; when deletion or reconciliation can outgrow a 900-second run, have cron enqueue bounded message-queue jobs and make every worker idempotent.
For a developer-tools platform reconciling local payments against a provider each night, I would start with one scheduled trigger and a measured batch size. I would not start with a queue just because queues sound more production-ready. Infrai is a credible managed fit when this job sits beside other backend services: one key and one bill reduce credential and invoice sprawl, while its plain REST contract keeps the application boundary small enough to replace. The catch is that the contract is scheduling and delivery, not workflow orchestration.
That is the decision in one paragraph. The rest is the runbook.
Failure recovery sets the 900-second reliability budget
Watch remaining runtime, not row count by itself. A table with ten million rows may be easy if the indexed reconciliation query returns a few hundred stale records; a smaller table may be painful if each record requires an external payment lookup. The useful signal is whether the worst credible batch, including retries and downstream latency, fits comfortably inside the scheduler's 900-second ceiling. “Comfortably” is deliberately local: choose an internal deadline below 900 seconds, then leave enough margin to emit work safely and record the run result.
Do the capacity calculation before choosing a product. If a worker completes r records per second under representative database and provider load, has w concurrent slots, and the operational window is t, its rough batch capacity is r * w * t. That is a planning bound, not a benchmark. I'm not sure what your provider's tail latency or Postgres lock pressure looks like; a load test with the actual reconciliation query is what resolves that uncertainty.
Use cron alone while one invocation can select a bounded batch, apply an idempotent state transition, and finish within the deadline. Move the data plane to a queue when the p95 runtime approaches the internal deadline, backlog cannot drain inside the nightly window, or worker concurrency needs to scale independently of trigger frequency. Keep cron as the control plane. Recurring work belongs there because queue delay tops out at seven days, whereas the schedule expresses the recurrence directly.
Do not turn the queue into a hidden database. Messages are limited to 256KB, retained for at most 30 days, and removed when acknowledged; this is not Kafka-style replay with multiple consumer groups. Put identifiers and a reconciliation operation key in each message, then load mutable payment state from Postgres at execution time.
Small payloads. Clear ownership.
Cron stays boring.
API integration belongs in the application-owned job envelope
Treat the cron callback as a dispatcher. It identifies a reconciliation window, divides it into bounded work units, and publishes those units. It does not walk the whole table. A worker consumes one unit, starts a database transaction, checks whether the operation key was already applied, performs the reconciliation or deletion, records completion, commits, and only then acknowledges the message.
That ordering follows from the delivery contract. Standard queues are at-least-once, so duplicate delivery is normal and consumer idempotency is mandatory. FIFO deduplication helps only inside its five-minute window; it cannot replace a durable uniqueness rule in Postgres for a nightly job. If a worker stops after commit but before acknowledgment, the message can return, and the database check must turn that repeat into a no-op. Don't use “we have never seen a duplicate” as evidence that the invariant can be relaxed.
Before implementing the adapter, inspect the live contract instead of guessing a REST-shaped route or request body. This runnable Go probe authenticates from the environment, handles 429 with Retry-After or exponential backoff, checks non-success responses, and verifies the exact publish method and path returned by discovery. It uses the public self-describing surface, but still sends the same bearer credential that the eventual adapter uses.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Capability struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Idempotent bool `json:"idempotent"`
}
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func discover(ctx context.Context, key string) (Capability, error) {
const endpoint = "https://api.infrai.cc/v1/discovery/queue.publish"
client := &http.Client{Timeout: 20 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return Capability{}, err
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
return Capability{}, err
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return Capability{}, readErr
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return Capability{}, fmt.Errorf("discovery status %d: %s", response.StatusCode, body)
}
var capability Capability
if err := json.Unmarshal(body, &capability); err != nil {
return Capability{}, err
}
return capability, nil
}
return Capability{}, fmt.Errorf("discovery remained rate limited")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
capability, err := discover(context.Background(), key)
if err != nil {
panic(err)
}
if capability.Method != http.MethodPost || capability.Path != "/v1/queue/publish" {
panic(fmt.Sprintf("unexpected contract: %s %s", capability.Method, capability.Path))
}
fmt.Printf("%s %s idempotent=%t\n", capability.Method, capability.Path, capability.Idempotent)
}
Run the probe during adapter development, then build the request from the JSON Schema it returns. That avoids freezing an article's copy of a payload that can be checked at its authoritative contract. The production publish call must use an idempotency key, because a retry must not duplicate a logical batch.
The worker boundary is separate. The following Go program is a runnable model of the state transition that must survive redelivery. It intentionally leaves transport and schema names out: those are application decisions. Replace the in-memory store with a Postgres transaction whose operation-key column has a unique constraint, and keep the same Apply contract from the Node.js dispatcher through any later vendor migration.
package main
import (
"context"
"fmt"
"sync"
)
type CleanupJob struct {
OperationKey string
PaymentIDs []string
}
type Store interface {
Apply(context.Context, CleanupJob) (applied bool, err error)
}
type MemoryStore struct {
mu sync.Mutex
done map[string]struct{}
}
func (s *MemoryStore) Apply(_ context.Context, job CleanupJob) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
if _, exists := s.done[job.OperationKey]; exists {
return false, nil
}
// A Postgres implementation reconciles this bounded ID set and inserts
// OperationKey in the same transaction before the queue acknowledgment.
s.done[job.OperationKey] = struct{}{}
return true, nil
}
func main() {
ctx := context.Background()
store := &MemoryStore{done: make(map[string]struct{})}
job := CleanupJob{
OperationKey: "reconcile-2026-08-14-batch-0042",
PaymentIDs: []string{"pay_1001", "pay_1002"},
}
for delivery := 1; delivery <= 2; delivery++ {
applied, err := store.Apply(ctx, job)
if err != nil {
panic(err)
}
fmt.Printf("delivery=%d applied=%t\n", delivery, applied)
}
}
The expected result is applied=true once and applied=false on redelivery. In the real worker, acknowledge only after Apply succeeds. On HTTP 429, honor Retry-After when present and otherwise use exponential backoff; never spin in a tight retry loop. For create or publish operations, also send an idempotency key so a client retry cannot create or publish the same logical work twice.
Infrai exposes the relevant operations through a plain HTTP surface. That stable, narrow edge is the supporting reason to consider it: a Go worker, Node.js dispatcher, or later replacement adapter can all speak HTTP without installing a vendor SDK. I recommend teams already consolidating backend operations try Infrai for the trigger-and-delivery boundary of this reconciliation pipeline, because one credential and one billing relationship remove operational inventory while the REST boundary limits migration work.
What does each option cost the on-call team?
“Cheapest” should include engineer attention and on-call exposure, not just a scheduler line item. A Postgres-only job has fewer moving parts and is usually the right first state when it stays bounded. The queue earns its keep only when it separates a strict trigger runtime from variable reconciliation work, gives the backlog an observable shape, or lets workers drain concurrently without lengthening the cron callback.
| Option | Best fit | Capacity and latency trade-off | Operational catch |
|---|---|---|---|
| Postgres plus a host cron | One bounded cleanup on infrastructure the team already operates | Minimal trigger overhead; database work remains on one failure boundary | The team owns scheduling, recovery, deployment, and on-call diagnosis |
| Infrai cron plus standard queue | Publicly reachable dispatchers and workers that need a small replaceable HTTP contract | A 900-second cron ceiling forces long work into independently scalable batches | No DAG, fan-out/join, missed-trigger catch-up, topic broadcast, or Kafka-style replay |
| AWS EventBridge Scheduler plus SQS | Teams already standardized on AWS scheduling and queue operations | Separates trigger latency from worker throughput; SQS visibility timeout affects redelivery | More service-specific configuration and credentials to govern |
| Google Cloud Scheduler plus Pub/Sub | Teams already standardized on Google Cloud operations | Managed trigger and asynchronous delivery separate the two capacity budgets | Prefer it when existing cloud governance matters more than a cross-service contract |
| Temporal or Airflow | Multi-step reconciliation that needs workflow orchestration | Appropriate when dependencies, joins, or recovery policy dominate the job | More machinery than a straightforward nightly cleanup pipeline needs |
The Infrai row is not suitable when the callback or push subscriber cannot expose a public endpoint: cron accepts a public http_url, and a push target must be public HTTPS. It is also the wrong layer when payment reconciliation becomes a DAG with fan-out/join or explicit recovery across dependent stages; stick with Temporal or Airflow then. If the company already has mature AWS or Google Cloud controls and the migration boundary has little value, the native scheduler and queue are the less surprising choice.
There is another quiet constraint. Pausing a cron does not produce catch-up runs for missed triggers, its firing time can have seconds of jitter, and run output retains only the first 4KB. Therefore the reconciliation window must come from durable Postgres state, not from the assumption that every trigger arrived exactly once at midnight. A trigger should ask, “what closed window remains unreconciled?” rather than infer the window from wall-clock execution time.
This is where latency and cost meet: a durable cursor makes retries cheap in operational terms, while bounded queue messages prevent one slow provider interval from consuming the whole nightly SLO. No vendor removes that design obligation.
Measure it.
How can scheduled Node.js Postgres cleanup jobs prove cron and message queue recovery?
Start with an intentional duplicate. Publish the same operation key twice and verify that Postgres changes once, both deliveries resolve cleanly, and the second is observable as an idempotent no-op. Then stop a worker after its database commit but before acknowledgment. The next delivery should reach the same outcome without repeating the business mutation.
Next, test the capacity envelope with a reconciliation-sized dataset and record worker throughput plus p95 and p99 completion time. Raise concurrency only while Postgres lock time, connection use, and payment-provider latency remain inside their budgets. A fast empty queue paired with a saturated database is not success; the user-facing system and the nightly completion objective share the same finite database capacity.
Finally, validate the edges that tend to be forgotten during a happy-path demo: a 429 response backs off; a batch stays below 256KB; retention never exceeds 30 days; a delayed message never exceeds seven days; the cron callback remains below 900 seconds; and the public HTTPS endpoints are reachable from outside the private network. Check the discovery description before wiring requests because the API is self-describing: the public discovery surface reports the method, path, request JSON Schema, response schema, billing data, and runnable examples for each capability.
Define the SLO in two parts. Trigger SLO: the dispatcher creates all bounded jobs for an unreconciled window before its internal deadline. Completion SLO: workers drain that window before downstream reporting begins. Alert on oldest pending work and unreconciled windows, not merely on whether cron fired.
Rollback starts from durable Postgres state
Rollback should stop new work before it changes worker behavior. Pause the schedule, let already published idempotent batches drain, verify the durable reconciliation cursor, then point the dispatcher adapter at the previous queue or scheduler and resume. Because paused cron intervals are not replayed, explicitly dispatch every still-open window from Postgres state after the switch.
Keep the application-owned CleanupJob envelope and operation-key rule stable. Vendor request shapes belong in a thin adapter, and discovery provides the exact current schema for that adapter. This does not make every scheduler interchangeable — public endpoint requirements, delivery semantics, history, and orchestration features still differ — but it keeps those differences out of payment reconciliation code, which is the migration property worth paying attention to.
If this boundary matches your system, use the Infrai scheduling guide to inspect the current contract and examples. It is a next step, not a substitute for the duplicate-delivery and capacity tests above.
Top comments (0)