KnutBerg8412

Posted on Oct 17

Go Scheduled Payment Data Cleanup — Compare 4 Cron Scheduler Latency and Cost Tests

#go #cron #sre

Short answer: choose a cron scheduler for nightly payment reconciliation by testing admission latency, duplicate delivery, missed-run recovery, and execution isolation; the cheapest trigger is a poor bargain if it can silently widen the cleanup backlog.

For a customer-support system, the job is deceptively plain: reconcile yesterday's payment-provider records, repair local status drift, and make corrected state available before agents begin answering billing tickets. The scheduling decision should follow the error budget for that outcome. If agents need corrected data by 06:00 UTC, a trigger at 05:59 with a one-minute runtime has no capacity margin. A trigger at 02:00, feeding bounded workers that can replay partitions, converts a brittle deadline into an observable batch SLO.

Don't optimize the clock first.

How should cost govern a cron scheduler comparison for scheduled data cleanup?

I would run four tests against the whole path, not just the timer: trigger admission latency, duplicate invocation, an intentionally unavailable worker, and a cleanup partition that exceeds its normal runtime. This is a capacity-planning exercise — arrival rate, service rate, backlog age, and deadline are the useful variables. A scheduler's dashboard can show a green invocation while the payment cursor remains twelve hours behind.

The bounded production scenario is one nightly reconciliation with 24 hourly partitions. Each partition reads a stable provider time range, compares external payment IDs with local records, and writes an idempotent result. That number is an example of partition shape, not a benchmark: use enough partitions to replay a narrow failure domain, but not so many that scheduling and database overhead dominate useful work. I'm not sure there is a universal best partition duration; provider rate limits, record skew, and the local write path settle it, and a load test with production-shaped distributions is what resolves the uncertainty.

A useful acceptance sheet is short:

Test	Evidence to retain	Failure decision
Admission latency	Intended time and observed start time	Does the remaining window preserve the batch SLO?
Duplicate trigger	Two runs with the same logical date	Do idempotency keys prevent repeated mutation?
Worker unavailable	Trigger accepted while workers are stopped	Is work durably recoverable without guessing?
Long partition	One partition crosses the next tick	Does overlap stay bounded by a lease or concurrency rule?

These tests separate a trigger from a job system. A trigger decides when work becomes eligible. A job system owns durable state, acknowledgements, retry policy, concurrency, and backlog. Combining those responsibilities can be convenient for a small workload, but treating them as the same thing makes incident diagnosis muddy: an absent trigger, a rejected enqueue, and a slow consumer create the same visible symptom — stale support data — while requiring different repairs.

Govern the payment data window before touching the clock

Consider a production incident as a tabletop rather than a claimed war story. The 02:00 trigger fires, the payment provider answers with HTTP 429 for several partitions, and the next run begins before the first has drained. If the job means “delete or update everything older than midnight,” the second run can race the first, retry already-applied writes, or move a shared cursor past records that were never reconciled. The invariant revealed by this scenario is more valuable than any scheduler feature list: each unit of cleanup needs a stable identity, a bounded input interval, and a result that can be applied more than once without changing the final state.

The obvious first model is cron -> cleanup(). It is also where SLOs disappear. The safer model is cron -> create logical run -> enqueue partitions -> workers -> reconcile run; its extra durable state costs storage and operational attention, but it gives on-call engineers a ledger of what should have happened, what did happen, and what remains eligible for retry. This doesn't create exactly-once execution. It makes at-least-once execution tolerable through idempotent effects. RabbitMQ's acknowledgement documentation is useful here because it distinguishes consumer acknowledgements from publisher confirms: the broker accepting a message and a worker completing its effect are separate events.

Here is the preventative boundary in Go. The scheduler calls Start, while the repository enforces uniqueness for the logical date and the queue carries partition identities. Implementations can use a database uniqueness constraint and a durable queue; the interface keeps vendor choice outside the reconciliation rule.

package reconcile

import (
    "context"
    "errors"
    "fmt"
    "time"
)

type Partition struct {
    RunDate string
    Start   time.Time
    End     time.Time
}

type RunRepository interface {
    CreateOnce(ctx context.Context, runDate string) (created bool, err error)
}

type Queue interface {
    Enqueue(ctx context.Context, key string, p Partition) error
}

type Starter struct {
    Runs  RunRepository
    Queue Queue
}

func (s Starter) Start(ctx context.Context, runDate string, dayStart time.Time) error {
    created, err := s.Runs.CreateOnce(ctx, runDate)
    if err != nil {
        return fmt.Errorf("create reconciliation run: %w", err)
    }
    if !created {
        return nil // A repeated trigger refers to the existing logical run.
    }

    var enqueueErrs []error
    for hour := 0; hour < 24; hour++ {
        start := dayStart.Add(time.Duration(hour) * time.Hour)
        p := Partition{RunDate: runDate, Start: start, End: start.Add(time.Hour)}
        key := fmt.Sprintf("payment-reconcile/%s/%02d", runDate, hour)
        if err := s.Queue.Enqueue(ctx, key, p); err != nil {
            enqueueErrs = append(enqueueErrs, fmt.Errorf("enqueue %s: %w", key, err))
        }
    }
    return errors.Join(enqueueErrs...)
}

One detail matters: after CreateOnce succeeds, failed partition enqueues must remain discoverable in the run ledger so a reconciler can enqueue the missing work. Returning an error is necessary for observability, but it isn't the recovery mechanism. A periodic sweeper can compare expected partition identities with accepted ones and fill gaps; because identities are stable, that repair is safe to repeat.

Slow down here.

Test each trigger boundary with the same harness

GitHub Actions scheduled workflows are attractive when cleanup already lives beside repository automation. Its documentation says scheduled workflows run from the latest commit on the default branch, can be delayed during high load, and may be dropped under sufficiently high load; the shortest interval is five minutes. Those properties can fit a non-urgent maintenance task, but they make the workflow history an inadequate system of record for a deadline-bound reconciliation unless the job also records durable run intent elsewhere. Repository access and workflow permissions also expand the blast radius beyond “wake a worker.”

Cloudflare Workers Cron Triggers invoke a Worker's scheduled handler on UTC time. Cloudflare documents that trigger changes can take several minutes, up to 15 minutes, to propagate. This model keeps the trigger close to lightweight edge code, while the execution limits and the need for durable reconciliation state still determine whether the work belongs in one invocation or should be handed to another worker path. It is a boundary choice, not a guarantee that a large batch finishes because the clock fired.

Amazon EventBridge Scheduler exposes one-time and recurring schedules, flexible time windows, retry policy, and dead-letter queue configuration. Those controls make delivery policy explicit, with the trade-off of cloud IAM, target configuration, and another provider-specific control plane. For teams already operating that environment, those dependencies may be normal; for a portable or self-hosted stack, they are lock-in and on-call surface that must earn their place.

BullMQ represents a different boundary. It is a Redis-backed Node.js queue with job scheduling and worker semantics, so choosing it means owning or buying the Redis layer and operating consumers rather than selecting a remote trigger alone. RabbitMQ can supply durable messaging and acknowledgement semantics, but cron eligibility remains a separate concern. These aren't rankings. They are different allocations of responsibility between source control automation, edge execution, a cloud scheduler, and an application-owned queue.

Option shape	Best evidence from the four tests	Operational cost to count
Repository-scheduled workflow	Workflow start history plus external run ledger	Repository permissions, variable start time, runner use
Edge cron handler	Trigger event plus durable handoff record	Execution limits, deployment propagation, state service
Cloud scheduler targeting a worker	Delivery attempts, retry policy, dead-letter records	IAM, target policy, provider control plane
Application queue with repeatable jobs	Queue state, acknowledgements, backlog age	Data-store operations, consumers, upgrades, paging

Price should be measured from the current official calculators against expected invocations, execution duration, storage, logs, egress, and on-call ownership; a trigger's sticker price alone excludes most of the system. Published rates and free allowances change, so hard-coding a winner into an architecture review ages badly. The latency side is equally contextual: nightly work rarely needs millisecond admission, but it does need enough recovery time before the support-data deadline.

A Go implementation for idempotent partition admission

The worker needs a lease or equivalent concurrency guard, an idempotency key at the mutation boundary, retry classification, and a checkpoint that advances only after the partition's effects are durable. A 429 response is retryable under the provider's policy; malformed input or an impossible state transition usually needs quarantine and review. Don't turn every error into an immediate retry. Correlated retries can consume the remaining batch window faster than ordinary processing.

For observability, retain logical run date, partition identity, attempt count, intended trigger time, accepted time, worker start time, completion time, and a low-cardinality outcome. From those fields, derive admission delay, queue delay, processing duration, oldest incomplete partition age, success ratio, and projected finish time. Alert on user impact or a threatened deadline rather than on one failed attempt; the retry can be healthy while the backlog trend is not. Logs help explain a partition, metrics show the fleet, and the run ledger answers the awkward audit question: which payment intervals were actually reconciled?

Capacity planning closes the loop. Let W be the available window in seconds, R the sustained records processed per second after rate limiting, and N the expected record count. Require N/R to sit comfortably below W, then test the margin with skew and retries. The exact headroom is an SLO policy decision, not a universal constant. If a provider permits only modest concurrency, earlier scheduling or incremental reconciliation can buy more safety than a larger worker fleet.

Deployment deserves the same skepticism. Pin the cleanup binary or image version, make schema changes backward compatible with in-flight workers, test duplicate and out-of-order partitions, and rehearse replay from the durable ledger. A scheduler configuration change should pass review and emit an auditable event. For a nightly job, a staging clock that runs every few minutes can validate orchestration, but test data must preserve the production invariants: stable external IDs, overlapping updates, pagination, and rate-limit responses.

No mystery metrics.

When should the deployment stay small?

Stick with a simple repository or host cron when the cleanup is disposable, quick, idempotent as a whole, easy to rerun, and has no hard completion deadline. A small cache purge whose next run safely catches up does not automatically justify a queue, run ledger, leases, and another pager. The catch is that “small” must be supported by observed duration and backlog, not by the current row count.

A managed scheduler is not suitable when provider-specific IAM and targets violate portability requirements, while a self-hosted queue is a poor fit when the team cannot own its data store, upgrades, and consumer capacity. An edge handler is the wrong execution home for work that exceeds its documented resource model, though it may still be a clean trigger. Repository automation should remain repository automation when production credentials or unpredictable start delay would breach the service boundary.

The decision rule is deliberately unbranded: define the reconciliation deadline, make run intent durable, test the four failure conditions, and choose the least operational machinery that still preserves the invariant under replay. Cost matters. So do the pages created by the architecture.

DEV Community