DEV Community

BartholomewVance6831
BartholomewVance6831

Posted on

How to Schedule Recurring HTTP Data Cleanup Triggers Without Task Executors (and Why)

#scheduling #http #datacleanup

How to Schedule Recurring HTTP Data Cleanup Triggers Without Task Executors (and Why)

Short answer: use an external recurring HTTP trigger, then make the cleanup endpoint idempotent with a stable run key and a database uniqueness constraint.

The simplest recurring cleanup trigger is an external clock calling a small, authenticated HTTP endpoint. The hard part is not firing the request. It is making a duplicate request harmless when the scheduler retries, the network drops after the provider accepted the work, or two runs overlap. For a nightly payment reconciliation job, I would choose an external scheduler plus a database-backed idempotency record, then keep each invocation bounded and repeatable.

Decision Best fit Main cost
External scheduler -> HTTP endpoint A service that already exposes an authenticated API You must design retries and authentication
Host cron -> local worker One machine, one deployment boundary, low operational ambition The machine becomes scheduler and failure domain
Database lease -> worker loop A team already operating a durable worker process More moving parts and wake-up logic

The recommendation is the first row when there is no internal task executor. It keeps the trigger outside the application while leaving correctness inside the application, where payment state and uniqueness constraints already live.

That boundary is the point.

What method makes recurring HTTP data cleanup triggers safe?

Assume the reconciliation window is 2026-10-24T00:00:00Z through 2026-10-25T00:00:00Z. The trigger sends that window and a stable run key, such as reconcile:2026-10-25. The endpoint authenticates the caller, validates the window, and attempts to insert the run key into a table with a unique constraint.

That insert is the decision point. A new key creates work. An existing key returns the current run state. It does not create a second provider query merely because the scheduler delivered the same request again. The payment provider may also need an idempotency key for writes; application-level deduplication cannot make an external side effect safe by itself.

A response timeout is ambiguous. It means the caller did not receive a response, not that reconciliation did not happen. Treating every timeout as a fresh run is how duplicate settlements and noisy provider traffic begin.

The endpoint should return quickly after recording accepted work if the reconciliation can exceed the request budget. The worker can claim the record, process a bounded page, persist a cursor, and release the claim. The next invocation continues from that cursor. No hidden in-process timer is required.

Short requests are easier to retry.

The small contract I would implement

The request needs fewer fields than most scheduler integrations suggest. A timestamp window, a run key, and an authenticated caller are enough for the trigger. The service owns the provider credentials and the rules for what counts as matched, missing, or ambiguous.

type ReconciliationRequest = {
  runKey: string;
  windowStart: string;
  windowEnd: string;
};

type ReconciliationReply =
  | { status: "accepted"; runKey: string }
  | { status: "already-running"; runKey: string }
  | { status: "complete"; runKey: string; matched: number }
  | { status: "rejected"; reason: string };

async function triggerReconciliation(
  request: ReconciliationRequest,
  accessToken: string,
): Promise<ReconciliationReply> {
  const response = await fetch("https://payments.example.invalid/reconciliation/runs", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${accessToken}`,
      "Content-Type": "application/json",
      "Idempotency-Key": request.runKey,
    },
    body: JSON.stringify(request),
  });

  if (response.status === 429) {
    throw new Error("provider rate limit; retry with backoff");
  }

  if (!response.ok) {
    throw new Error(`reconciliation trigger failed: ${response.status}`);
  }

  return (await response.json()) as ReconciliationReply;
}
Enter fullscreen mode Exit fullscreen mode

The .invalid hostname is deliberate: it marks a contract example, not a vendor setup guide. In production, the endpoint should verify a signed request or equivalent credential, reject windows outside an allowed range, and log the run key without logging payment data.

The key judgment is where to put retries. The scheduler can retry transport failures. The endpoint can retry provider reads with bounded exponential backoff. The database can enforce uniqueness. Each layer has a narrow job. A retry policy scattered across all three becomes impossible to reason about. The trade-off is real: this design adds a run table, a claim state, and request authentication, so it is a poor fit for a one-off script whose only failure consequence is a stale local cache. For payment data, that extra state is the cheaper complexity to operate.

How do you keep retries from becoming a rate-limit storm?

A 429 Too Many Requests response is a server asking the client to slow down. If the response includes Retry-After, use it within a bounded policy; otherwise apply backoff with jitter and a maximum attempt count. Do not make ten concurrent retries because the first request was slow.

function retryDelay(attempt: number, retryAfterSeconds?: number): number {
  const serverDelay = (retryAfterSeconds ?? 0) * 1000;
  const exponential = Math.min(30_000, 500 * 2 ** attempt);
  const jitter = Math.floor(Math.random() * 250);
  return Math.max(serverDelay, exponential) + jitter;
}

async function withRetry<T>(
  operation: () => Promise<T>,
  attempts = 5,
): Promise<T> {
  for (let attempt = 0; attempt < attempts; attempt += 1) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === attempts - 1) throw error;
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(attempt)),
      );
    }
  }

  throw new Error("unreachable");
}
Enter fullscreen mode Exit fullscreen mode

This example keeps the retry decision visible, but production code should preserve the response status and Retry-After value instead of reducing every failure to a generic exception. A 401 is not a backoff problem. A 400 is not a transient provider outage. Classify first.

For reconciliation, observability should answer four questions: which run key was attempted, which window it covered, how many records were read and classified, and whether the last failure is retryable. A duration metric without the run key is decorative. A log containing full card data is a liability.

When is a different trigger the better choice?

Host cron is reasonable for a single, stable machine where losing that machine is an accepted operational risk. It has almost no glue. It also makes deployment, clock configuration, and failover your problem. That trade can be correct for a private utility; it is harder to defend for payment reconciliation. The limitation is clear: an external trigger cannot repair a bad reconciliation state, and an idempotency key cannot undo a provider-side write that was already accepted without an explicit reversal path.

A database lease and worker loop is better when the team already has a durable worker process, a queue, and operational ownership for them. The lease can prevent concurrent claims, but it does not automatically make provider writes idempotent. The same run key still matters.

An external scheduler is the awkward middle: less infrastructure than a worker, more explicit failure handling than cron. That is exactly why I use it for a small recurring HTTP trigger. The trigger stays boring. The service owns the state machine.

Before shipping, test the ugly transitions rather than only the green path: response lost after the provider accepts a request; duplicate delivery during an active run; a worker crash after advancing a page; a 429 with Retry-After; and two different windows that overlap by one minute. If those tests cannot explain the resulting rows, the schedule is not the problem yet.

Sources

Top comments (0)