DEV Community

RaffertyBarrett4726
RaffertyBarrett4726

Posted on

Scheduled Data Cleanup: Choosing a Cheap, Easy Queue for a Small SaaS

Short answer

Short answer: the right choice for scheduled data cleanup is the design that preserves a recovery contract across a missed tick, a lost acknowledgement, and a duplicate delivery. A database-backed scheduler, a hosted queue, BullMQ, and RabbitMQ can all carry work; none of them makes the cleanup idempotent for you. Cost and ease follow from the amount of delivery state your team must operate.

The shipment example matters here. A cleanup job may delete expired rows, archive records, or fan out a shipment update to many subscribers. In all three cases, a timeout can leave the producer unsure whether the consumer finished. Treat every delivery as at-least-once unless the system explicitly proves otherwise.

The decision starts with the recovery boundary

The first decision is where recovery state lives. Separate four concerns: deciding that work is due, claiming a unit of work, performing it, and recording the result. A cron expression answers only the first question. Cron has long been a Unix scheduling convention, but the convention does not give an application a durable job state, a retry policy, or a deduplication key.

For scheduled cleanup, the database is often the most useful source of truth. Store a row for each cleanup window or tenant, with a stable key such as tenant_id + cleanup_date. A scheduler finds due rows and enqueues work. The worker claims the row, processes a bounded batch, and records completion. If the enqueue call or worker process is interrupted, the next attempt can use the same key.

A hosted queue reduces the amount of queue infrastructure the team operates. That can be the easiest choice when the team does not want to own broker upgrades, persistence settings, dashboards, and alerting. It is not automatically the cheapest: the real comparison includes the queue charge, the scheduler, observability, network paths, and the engineering time spent proving recovery behavior.

The catch is operational ownership. If the hosted service cannot meet your data residency, retention, private-network, or regional delivery requirements, use a component you can run inside the required boundary. If a hosted queue does not expose the retry, visibility, or delivery semantics your cleanup needs, it is the wrong fit even if its first invoice is small.

How should a small SaaS worker claim scheduled data cleanup safely?

A missed tick is visible. Duplicate work is quieter and can be worse: two workers can both decide that a shipment update is ready, or two cleanup attempts can delete and then recreate the same derived record. I have been paged by missed jobs and duplicate deliveries, so I put the retry contract ahead of the scheduler brand.

Use a stable idempotency key at the business boundary. For a shipment update, that might be shipment_id + event_version; for cleanup, it might be tenant_id + cleanup_window. Persist the key where the side effect is committed. A retry then becomes a lookup and a no-op, rather than a second irreversible action.

A queue's retry count is not an idempotency system. Exponential backoff reduces pressure during a dependency failure, but it does not tell you if the first attempt committed just before its response was lost. The worker needs a durable record and a transaction boundary that the team can explain during a postmortem.

Three words: assume duplicates happen.

For database-backed claiming, PostgreSQL documents FOR UPDATE SKIP LOCKED as useful for avoiding lock contention with multiple consumers accessing a queue-like table. The claim should be short; do not hold a row lock while calling a slow external subscriber. Claim a batch, commit the claim, perform the work, and update the result with a guarded transition.

type CleanupJob struct {
\tID        int64
\tTenantID  int64
\tWindowKey string
\tStatus    string
\tAttempts  int
}

func claimSQL(limit int) string {
\treturn `
WITH due AS (
  SELECT id
  FROM cleanup_jobs
  WHERE status = 'pending'
    AND run_after <= now()
  ORDER BY run_after, id
  FOR UPDATE SKIP LOCKED
  LIMIT $1
)
UPDATE cleanup_jobs AS j
SET status = 'running',
    attempts = attempts + 1,
    claimed_at = now()
FROM due
WHERE j.id = due.id
RETURNING j.id, j.tenant_id, j.window_key, j.status, j.attempts`
}
Enter fullscreen mode Exit fullscreen mode

The SQL is a claiming pattern, not a complete queue implementation. Add a lease timeout so a worker process that disappears does not hold work forever. Keep the cleanup batch bounded, and make the completion update conditional on the job still belonging to that attempt. Your mileage may vary with transaction isolation and workload; verify those choices against the PostgreSQL behavior in your deployment.

Consider one tenant whose cleanup window contains 18,000 records and whose shipment subscribers are split across email, webhook, and an internal analytics consumer. The scheduler should publish references to bounded slices, not one heroic message. Suppose the worker claims slice 12, commits that claim, sends the webhook, and loses its connection before recording completion. A lease expiry makes slice 12 visible again. The second worker sees the same operation key, checks the durable side-effect record, and records a duplicate attempt without sending a second webhook. If the first worker had not committed the side effect, the second worker performs it and writes the result. Now add a deployment during that window: old and new workers can overlap because both respect the claim transition and the lease owner. Finally, a policy change marks the tenant exempt after the scheduler selected the slice; the worker re-checks eligibility before deletion and records an exemption rather than treating the stale selection as authority. This is why the job row, the idempotency key, and the audit result belong in the design before the queue is selected. A queue can transport the reference, but it cannot infer whether a legal hold, shipment cancellation, or prior commit should win.

What does each operating model leave your team owning?

The comparison should start with the failure you need to control, not feature count. BullMQ is a queue library built around Redis, so it is a natural fit when the application already has a Redis operational path and a Node.js worker. For a Go service, that choice adds a runtime and ownership boundary unless another service already owns the queue. That is a team-shape trade-off, not a quality verdict.

A hosted queue is attractive when the team wants a managed delivery layer and a small operational surface. Ask for concrete answers about retry delays, dead-letter handling, message visibility, ordering, maximum payloads, regional placement, and metrics. “It retries” is not enough. You need to know what happens after a worker receives a message and then loses its connection.

RabbitMQ is a broker option when routing, exchanges, acknowledgements, consumer coordination, and explicit delivery controls matter. Those controls also create operational work: persistence, capacity, upgrades, topology, and alerts become part of the service you run. A small SaaS may accept that work when message routing is central to the product; it may be a poor fit for one nightly cleanup table.

Option Good fit Main cost or risk Questions to answer
Database-backed scheduler Small cleanup volume, strong transactional ownership Database contention and custom worker logic How are leases, retries, and retention implemented?
BullMQ Existing Redis and Node.js operations Redis and runtime coupling Which team owns Redis durability and the worker?
Hosted queue Small team, low broker-operations appetite Provider semantics and boundary constraints Are retry, visibility, region, and dead-letter behavior sufficient?
RabbitMQ Rich routing and acknowledgement controls Broker operations and topology Who drills recovery and manages capacity?

No option wins in the abstract. For this Go-shaped cleanup worker, the relevant boundary is the one the team can operate, observe, and recover. A database-backed scheduler has a small starting surface; a hosted queue moves delivery operations outside the application; a broker adds explicit routing and acknowledgement machinery. Those are different ownership contracts, not a leaderboard.

Rehearse failure, then make rollback boring

Test the ambiguous edges before production. Kill the worker after it claims a job. Delay the database response after the side effect commits. Deliver the same shipment event twice. Run two schedulers at once. Advance the clock across a daylight-saving transition for each region you serve. The expected result is either one durable side effect or a recorded, inspectable failure that can be retried safely.

Instrument the state transitions, not just queue depth. Track due jobs, claimed jobs, age of the oldest lease, attempts by reason, dead-letter volume, cleanup rows affected, and idempotency-key collisions. Alert on work age and repeated attempts. A queue with zero visible messages can still have a stuck running set.

Rollback needs a business rule. If a cleanup version is wrong, stop claiming new rows, preserve the job records, and make the operation reversible where possible. Do not “fix” the queue by deleting messages before you know whether their side effects committed. For shipment fan-out, replay from the durable event or outbox record; for destructive cleanup, prefer a staged mark-and-delete window when the retention policy allows it.

The recommendation is not suitable when the database cannot absorb the claim traffic, when external delivery requires broker features you do not have, or when strict cross-region guarantees exceed the chosen component. Stick with a more capable broker or hosted service when those constraints are real. I'm not sure any cost estimate stays stable long enough to be a selection criterion by itself, so use a measured workload and a recovery exercise to settle the decision.

References

Top comments (0)