DEV Community

SilasFletcher5857
SilasFletcher5857

Posted on

Node.js Scheduled Data Cleanup: Compare Cron Scheduler Retries and Idempotency

Short answer: For scheduled data cleanup, use the cron scheduler your team already operates, then put retries behind a durable worker and make every reconciliation effect idempotent; trigger price matters less than recovery behavior.

I've been paged by missed jobs and duplicate deliveries. Those incidents leave the same runbook note: a schedule is a hint that work should begin, not proof that it ran once. In a property management system, a duplicate reconciliation can notify a tenant twice or apply the same correction twice, while a missed run can leave yesterday's payment-provider records unmatched. The useful invariant is narrower: for a given property, provider account, and settlement date, every intended mutation has one stable idempotency key.

Retries are normal. Repeated side effects are a design error.

How should teams compare cron schedulers for scheduled data cleanup?

It shouldn't own the whole recovery policy. A scheduler should create a run request with a deterministic identity, record when that request became due, and hand it to an execution path that can be retried. The worker should claim the run, enumerate reconciliation items, and make each side effect conditional on a unique key. If the trigger fires twice, both attempts point at the same logical run. If the worker stops after item 417, the next attempt resumes against recorded item state instead of guessing what happened.

For nightly payment reconciliation, I would form the run ID from the settlement date and provider account, not the process start timestamp. Timestamps make duplicates look unique. A key such as provider_account + settlement_date + operation survives redelivery and makes the database the final authority. Keep the scheduler's delivery identifier as diagnostic metadata, but don't use it as the business key.

A queue can improve isolation, yet it doesn't create exactly-once effects. RabbitMQ's acknowledgement model separates delivery acknowledgement from the business transaction, so an acknowledged message and a committed database write still require deliberate ordering. BullMQ gives a worker-oriented queue abstraction, but the job body must remain safe to run again. The useful promise is at-least-once execution with idempotent effects. Anything stronger needs proof across every external boundary.

This is the long pole.

The failure budget sets the real cost boundary.

The trigger charge is a poor proxy for cost. Include worker execution, queue and storage operations, logs, egress, incident response, and the engineering cost of a second control plane. A repository-native trigger may reduce setup for a low-risk maintenance script; an infrastructure scheduler may fit a team that already operates that cloud; a worker-native trigger may avoid another runtime. Your mileage may vary because request volume and existing operational ownership dominate the total.

Picture a run due at 02:00. It fetches a provider page, compares payments with local lease ledgers, and writes a result for each property. A transient provider response such as 429 may delay page three. The attempt should return a retryable error without erasing completed item records. A malformed record belongs in a terminal review state with its source identifier and reason; retrying the entire night forever only hides a data-quality problem. The expensive failure is an operator spending the morning reconstructing which effects landed.

The run state should distinguish due, running, retryable, complete, and needs_review. Those names aren't magic. Their value is that the operator can tell a delayed attempt from a finished business run. Store scheduled_for, attempt_count, a lease expiry, and the last classified error. Then alert on age: a run still incomplete beyond the reconciliation deadline is actionable, while one short retry usually isn't.

Write down who owns each transition. The platform team may own trigger delivery, the payments team may own provider classification, and property operations may own records in needs_review. Shared ownership without a named handoff is how a retry sits unnoticed until the morning report. An audit record should show the logical run ID, attempt owner, transition time, and reason without storing payment secrets.

There is a subtle race here. Suppose attempt A loses its lease while blocked on the payment provider. Attempt B acquires the run and finishes an item. A wakes up and tries the same write. A process mutex won't help because the attempts may live on different hosts. The item table needs a unique constraint on the business idempotency key, and any outbound correction needs the same key at the receiving boundary when that API supports it. Without that constraint, the design is relying on timing. Timing eventually loses.

Keep it boring.

The execution contract in Go

The scheduling platform can call a generic internal entry point, but it should not receive permission to perform reconciliation logic itself. The handler below computes a stable run identity, inserts it once, and enqueues only when this call created the row. The worker claims an expiring lease and records each item result under a unique key. The interfaces make the transaction boundaries visible; production implementations should perform CreateRun atomically and enforce uniqueness in storage.

package reconcile

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "time"
)

type Store interface {
    CreateRun(ctx context.Context, runID, account string, day time.Time) (created bool, err error)
    ClaimRun(ctx context.Context, runID, owner string, leaseUntil time.Time) (bool, error)
    ListPending(ctx context.Context, runID string) ([]Item, error)
    RecordResult(ctx context.Context, key string, result Result) (created bool, err error)
    CompleteRun(ctx context.Context, runID string) error
}

type Queue interface {
    Enqueue(ctx context.Context, runID string) error
}

type Provider interface {
    Compare(ctx context.Context, item Item) (Result, error)
}

type Item struct {
    PropertyID string
    AccountID  string
    Day        time.Time
}

type Result struct {
    Status string
}

func stableID(parts ...string) string {
    h := sha256.New()
    for _, part := range parts {
        h.Write([]byte{0})
        h.Write([]byte(part))
    }
    return hex.EncodeToString(h.Sum(nil))
}

func Schedule(ctx context.Context, s Store, q Queue, account string, day time.Time) error {
    runID := stableID(account, day.UTC().Format("2006-01-02"))
    created, err := s.CreateRun(ctx, runID, account, day.UTC())
    if err != nil || !created {
        return err
    }
    return q.Enqueue(ctx, runID)
}

func Execute(ctx context.Context, s Store, p Provider, runID, owner string, now time.Time) error {
    claimed, err := s.ClaimRun(ctx, runID, owner, now.Add(10*time.Minute))
    if err != nil {
        return err
    }
    if !claimed {
        return nil
    }

    items, err := s.ListPending(ctx, runID)
    if err != nil {
        return err
    }
    for _, item := range items {
        result, err := p.Compare(ctx, item)
        if err != nil {
            return err
        }
        key := stableID(item.AccountID, item.PropertyID, item.Day.UTC().Format("2006-01-02"), "reconcile")
        if _, err := s.RecordResult(ctx, key, result); err != nil {
            return err
        }
    }
    return s.CompleteRun(ctx, runID)
}
Enter fullscreen mode Exit fullscreen mode

The shortcoming is intentional: this sample stops on any provider error. In a real service, classify timeouts and rate limits as retryable, send invalid records to review, and cap attempts by the business deadline. I'm not sure what that deadline should be for your payment provider; settlement rules and the morning operations handoff determine it. Put the value in configuration, test it with a fake clock, and write the escalation path before enabling the schedule.

Tests should call Schedule twice for the same date, execute the same run concurrently, expire a lease mid-page, and replay an item after its result exists. The assertion is always about stored effects, not function-call counts. During deployment, keep the old worker able to read the new run schema until every queued job using the old shape has drained. Otherwise a safe scheduler choice can still produce an unsafe rollout.

Platform recovery evidence comes before feature counts.

GitHub Actions scheduled workflows, Cloudflare Workers Cron Triggers, and Amazon EventBridge Scheduler can each sit on the trigger side of this design. They have different execution and ownership boundaries, so a universal ranking would be misleading. Compare the boundary your team will operate, then run the same failure drills against the shortlisted options.

Candidate Boundary to examine Evidence to collect before adoption
GitHub Actions scheduled workflow Repository workflow and runner Delayed-run behavior, concurrency behavior, secret ownership, and run-history retention
Cloudflare Workers Cron Trigger Worker runtime Execution limits, retry semantics, overlap behavior, and observability export
Amazon EventBridge Scheduler AWS schedule and target Target retry policy, dead-letter handling, identity policy, and delivery metrics

Don't score the table from marketing pages. Create a disposable schedule, prevent its worker from running through one due window, restore it, and observe whether the platform skips, delays, or redelivers. Start two attempts for the same run ID and confirm the unique constraint admits one result. Revoke target credentials and verify that the failure becomes visible to the on-call channel. These tests answer the recovery question that a feature checklist misses.

The catch is that this trigger-plus-worker pattern is not suitable for sub-second dispatch, hard real-time deadlines, or workflows that must commit atomically across a database and a payment provider. Use a transactional outbox when a local database mutation must reliably publish follow-up work. Stick with a simple in-process timer when one continuously running instance owns a low-impact task and duplicate execution has no material effect. For multi-step human approvals, use a durable workflow engine rather than stretching cron into one.

The runbook is the developer experience that matters during an incident.

A dashboard should answer four questions without a log search: which settlement dates are incomplete, which accounts are blocked, how old the oldest unfinished run is, and whether attempts are making progress. Track due-to-start delay separately from execution duration. The first points toward trigger or queue trouble; the second points toward provider latency or worker capacity. One combined latency chart muddies the response.

Page on business lateness, not every retry. The runbook should include the query for an incomplete run, the controlled action that requests a replay using the same run ID, the rule for moving poison data to review, and the evidence required before declaring the night complete. Never advise an operator to mint a new ID just to make a retry move; that defeats the duplicate guard exactly when pressure is highest.

Recovery evidence before migration

Introduce the run table and item uniqueness constraint before changing the trigger. Shadow-create deterministic run IDs while the existing job remains authoritative, then compare candidate counts without applying a second set of effects. Once the counts and classifications agree, direct one account through the worker path and retain the ability to stop new claims. Rollback should disable claims; it should not delete run history or regenerate identifiers.

During deployment, keep the old worker able to read the new run schema until every queued job using the old shape has drained. Schema compatibility is part of the retry contract. If a release changes how an item key is derived, treat it as a data migration with collision checks, not a harmless refactor.

For data cleanup that deletes records rather than reconciling payments, add a tombstone or candidate phase. First mark rows eligible under a documented retention rule, record the candidate set, and delete from that bounded set on a later step. The same retry design applies, but restoration and legal retention now matter more than provider reconciliation. This is where “cheapest cron” stops being a useful question: the trigger is tiny compared with the blast radius of an unreviewable delete.

Ship only after a staging drill covers a missed due window, duplicate delivery, worker termination, expired credentials, and a partial provider page. A scheduler that passes those tests and fits the team's existing ownership model is a defensible choice. The product name is secondary.

References

Further reading

The primary documentation URLs in References define the current delivery semantics, execution limits, acknowledgement behavior, and retry controls to verify before running the failure drills.

Top comments (0)