Short answer: use nightly cron only to call a small Node.js HTTP endpoint; let that endpoint enqueue a cleanup sweep, then let idempotent queue workers delete old renewal records in bounded batches. Keep a short cleanup inline only when its worst-case duration fits comfortably inside the request deadline.
For an edtech renewal system, “old” is a business rule, not created_at < now(). A reminder may have been created weeks ago and still need to wait for a district's renewal deadline. The safe flow is cron to HTTP dispatch, dispatch to queue, and queue worker to database, with the deadline and retention cutoff frozen in the message. That split spends one queue message per batch, but it keeps a growing table from turning a nightly maintenance task into a long-running request.
Retention invariants for scheduled cleanup
Cron should schedule intent, not perform an unbounded delete. At the chosen nightly time, it sends an authenticated request to a narrow endpoint. The endpoint validates the request, chooses a stable cutoff, publishes a small sweep message, and returns 202 Accepted. A worker consumes that message and removes at most one batch of eligible records. If a full batch was found, it publishes the next cursor and stops.
That last stop matters. A loop that says “keep deleting until the table is clean” merely moves the long-running job from HTTP into one oversized queue delivery. One message per bounded unit gives the system places to retry, measure progress, and yield capacity to live renewal traffic. It also makes the cost visible: reducing the batch size lowers lock time and usually increases queue operations; increasing it does the reverse.
The business predicate should be written down before the cron expression. In this example, a delivery record can be removed only after its reminder is terminal, its business deadline has passed, and its retention window has expired. A pending reminder is never cleanup input, even if it is old by creation time. This distinction protects schools whose purchasing calendars don't match a simple age threshold. Consider a reminder created on May 1 for a district whose contract decision is due September 30: creation age says “old” during the summer, but the business clock says “still pending.” If the district moves that decision to October 15 while a sweep is waiting, the worker must re-read the current row and leave it alone. After the reminder is sent or cancelled, a separate retention interval starts; only then can the same conditional delete remove its delivery metadata. That sequence is why the queue carries a cutoff and cursor rather than a verdict about every record. Don't put student details, email bodies, or a large list of record IDs in the message. The worker can query by the frozen cutoff and cursor, so the message stays small, contains no unnecessary learner data, and can be inspected without turning an operations console into a second data store.
A copyable endpoint and worker
The TypeScript below keeps the transport adapters generic. db can be backed by a PostgreSQL client, while queue can be backed by a managed or self-hosted broker. The important contract is local: publishing accepts an idempotency key, and the delete statement repeats every eligibility condition instead of trusting an earlier read.
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { timingSafeEqual } from "node:crypto";
type Sweep = {
sweepId: string;
cutoffIso: string;
afterId: string | null;
};
type Candidate = { id: string };
interface Database {
findCleanupCandidates(input: {
cutoffIso: string;
afterId: string | null;
limit: number;
}): Promise<Candidate[]>;
deleteIfEligible(input: { id: string; cutoffIso: string }): Promise<boolean>;
}
interface Queue {
publish(message: Sweep, options: { idempotencyKey: string }): Promise<void>;
}
const BATCH_SIZE = 200;
function authorized(req: IncomingMessage): boolean {
const supplied = Buffer.from(req.headers.authorization ?? "");
const expected = Buffer.from(`Bearer ${process.env.CRON_TOKEN ?? ""}`);
return supplied.length === expected.length && timingSafeEqual(supplied, expected);
}
export function startDispatcher(queue: Queue): void {
createServer(async (req: IncomingMessage, res: ServerResponse) => {
if (req.method !== "POST" || req.url !== "/internal/renewal-cleanup") {
res.writeHead(404).end();
return;
}
if (!authorized(req)) {
res.writeHead(401).end();
return;
}
const now = new Date();
const cutoff = new Date(now.getTime() - 30 * 24 * 60 * 60 * 1000);
const day = now.toISOString().slice(0, 10);
const message: Sweep = {
sweepId: `renewal-cleanup:${day}`,
cutoffIso: cutoff.toISOString(),
afterId: null,
};
await queue.publish(message, { idempotencyKey: message.sweepId });
res.writeHead(202).end();
}).listen(8080);
}
export async function processSweep(
message: Sweep,
db: Database,
queue: Queue,
): Promise<void> {
const rows = await db.findCleanupCandidates({
cutoffIso: message.cutoffIso,
afterId: message.afterId,
limit: BATCH_SIZE,
});
for (const row of rows) {
await db.deleteIfEligible({ id: row.id, cutoffIso: message.cutoffIso });
}
if (rows.length === BATCH_SIZE) {
const afterId = rows.at(-1)!.id;
await queue.publish(
{ ...message, afterId },
{ idempotencyKey: `${message.sweepId}:${afterId}` },
);
}
}
findCleanupCandidates must order by an immutable, indexed key and use id > afterId; offset pagination can skip rows as earlier rows disappear. deleteIfEligible should be a conditional delete along the lines of status IN ('sent', 'cancelled') AND deadline_at < cutoff. If another process changes the status between selection and deletion, the delete affects zero rows. That is a successful no-op, not a reason to guess what happened.
The example freezes a 30-day retention cutoff and deletes 200 candidates per delivery. Those are configuration choices, not universal recommendations. I'm not sure what the right batch size is for a particular database without its query plan, row width, index shape, and production latency budget. Start with a deliberately bounded batch, inspect lock time and worker duration, then adjust one variable at a time.
There is one subtle gap to close in a production adapter: publishing the next cursor and acknowledging the current delivery must follow the broker's delivery contract. At-least-once delivery means the current message may return. The repeated eligibility check makes that harmless, while the deterministic idempotency key prevents needless duplicate work when the broker supports deduplication. If it doesn't, correctness still comes from the database predicate.
Small boundary. Big payoff.
How can Node.js cron schedule nightly data cleanup for long-running jobs?
The cleanup code is easy compared with defining eligibility. A renewal reminder has at least three clocks: the business deadline supplied by the school or contract, the moment the reminder reached a terminal state, and the retention deadline required by policy. Mixing them into one olderThanDays parameter creates an attractive but unsafe abstraction. Model the transition explicitly. A pending reminder waits until deadline_at; a sent or cancelled reminder becomes eligible only after the retention interval; an active renewal remains outside the cleanup query. If a customer changes a deadline, the current source-of-truth row wins. The queued sweep carries a cutoff for repeatability, but it doesn't carry a stale assertion that a specific reminder is safe to delete.
Use UTC instants inside the service and keep the school's time zone alongside the business schedule. “Midnight” without a zone is not a deadline. Daylight-saving transitions make local times repeat or disappear, so convert the agreed local deadline to an instant when the business event is created, then test that conversion with the zones the application actually supports.
An archive requirement changes the transaction. Deleting after an asynchronous archive upload can lose the only copy if the steps are treated as one vague job. Record an explicit archive state, make the archive operation repeatable, and permit deletion only after that state is durable. If regulations require immutable retention or legal holds, this simple delete worker is not suitable; keep the records in the governed store and let the worker expire only disposable delivery metadata.
Capacity planning after correctness
The least expensive architecture is often the inline endpoint: one cron invocation, one indexed delete, no queue. Use it when the number of eligible rows has a hard upper bound, the query finishes well inside the HTTP time budget, and a retry can repeat the operation safely. A tiny deployment with 40 expired rows per night doesn't need a miniature distributed system.
The catch is growth. A single DELETE over millions of rows can hold locks, produce a burst of database work, and compete with the request path that sends actual renewal notices. Queue batches add broker calls and worker runtime, yet buy control over concurrency and blast radius. For a solo founder, that trade is usually justified once the upper bound is unknown, because an unknown maintenance runtime is also an unknown customer-facing latency risk.
Use a decision rule rather than a product label:
| Condition | Execution shape | Main trade-off |
|---|---|---|
| Hard, small row cap and indexed delete | Cron calls HTTP; endpoint deletes inline | Lowest moving-part cost, tighter request deadline |
| Variable backlog or shared hot table | Cron dispatches queue batches | More operations, controlled database pressure |
| CPU-heavy transformation or archive export | Dedicated worker pool with checkpoints | Higher idle capacity, isolated long-running jobs |
| Legal hold or governed retention | No physical cleanup of governed records | More storage, correct policy boundary |
Latency here has two meanings. Cleanup completion latency can be hours without harming the user, while renewal-read latency cannot. Cap worker concurrency to protect the latter. A queue that drains five minutes faster but pushes the primary database over its latency target is losing the wrong race. Cost needs the same honesty. Count queue requests, worker time, database reads, write amplification, and retained storage. Don't optimize only the line item that is easiest to see. Your mileage may vary — especially when deletion triggers indexes, audit records, or replication traffic — so production measurements should decide between a batch of 200 and a batch of 2,000.
Protect live traffic.
Test the race, then operate it
Assume a delivery can be repeated. Each operation should therefore converge on the same state: conditional deletion, deterministic cursors, and a stable sweep identifier. Track examined, deleted, and ineligible counts separately. A sudden rise in ineligible may mean the selection query and delete predicate have drifted, even though no data was lost.
Retries need a ceiling. After a configured attempt count, move a poison message to a dead-letter queue and alert on it; AWS's SQS documentation describes a dead-letter queue as a target for messages that were not processed successfully. Keep enough context to replay the batch, but avoid copying sensitive record content into the failure payload. Redrive is an operator action with an audit trail, not an infinite loop.
If the HTTP dispatcher or a downstream administrative API answers 429 Too Many Requests, read Retry-After when it is present and delay the retry accordingly. MDN documents both the status and that response header. Add jitter so multiple workers don't wake at exactly the same instant. Do not retry authentication failures or malformed messages; those need correction, not more traffic.
The deployment check is concrete. Run the worker first, then enable the endpoint, then enable cron. Exercise duplicate delivery with the same sweepId, a deadline update between selection and deletion, an empty batch, a full batch, and a final partial batch. Observe p95 database latency during a seeded backlog. Pause the schedule once and verify the runbook can launch a dated sweep manually, because a maintenance plan that depends on an assumed backfill is fragile.
Nightly success is not the useful signal. Measure age of the oldest eligible row, batch duration, retries, dead-letter depth, rows removed, and database latency. Alert on backlog age and dead letters; a zero-delete night can be perfectly healthy.
References
Further reading
The two references above are the primary follow-ups for dead-letter handling and standards-based rate-limit responses:
Top comments (0)