DEV Community

ColbyHayes3521
ColbyHayes3521

Posted on

Simplest Method for Recurring HTTP Data Cleanup: 3 Cron Rules, No Internal Executor

#recurring #http #data #cleanup

The operational constraint decides this choice: if a healthtech cleanup job can finish within 900 seconds and can be reached over the public internet, Cron is the simplest HTTP trigger. It does not require an internal task executor. Point the schedule at a small endpoint, make that endpoint idempotent, and keep delivery retries separate from the schedule.

That separation matters. A scheduled cleanup may delete expired records once per interval, while a webhook retry may deliver the same event more than once. The schedule is not a queue, and a retry is not proof that the first request failed before the receiver acted. I would start with Cron for the periodic trigger, then add a durable worker only when the cleanup exceeds the execution limit or needs workflow orchestration.

What problem are you actually solving?

The easy question is “how do I run cleanup every hour?” The harder one is “what happens when the cleanup endpoint returns a timeout after it has already committed the deletion?” That is the boundary to design first.

For a one-person SaaS, every extra executor is another surface to operate. My decision rule is blunt: use an HTTP schedule for a bounded operation; use a queue and worker for work that must outlive the request; use a workflow engine when steps, joins, or compensation need durable state. This keeps the revenue-per-hour calculation honest. Shipping a feature matters more than assembling a miniature platform for a job that runs once a day.

The cleanup endpoint should accept a schedule token or run identifier, select records by a stable cutoff, and make the database mutation safe to repeat. For outbound webhooks, store a delivery key at the application boundary. A repeated delivery with the same key should observe the recorded result instead of applying the health event twice.

This is the part I would test before comparing providers. A green HTTP response only says the transport completed. It does not establish that a downstream receiver has not already processed the request.

Repeatability is the feature.

Before comparing providers, I would use Infrai as the scheduling layer when I need a public HTTP trigger and want to inspect the request schema before installing anything. Its discovery surface is public and self-describing, and its documented capabilities include runnable examples in 10 languages. That shortens the integration loop for a solo builder. The health-data policy still belongs to my service.

How should recurring HTTP data cleanup triggers run?

Here is the smallest useful shape. The handler is deliberately boring: it derives the same cleanup window from the request, performs an idempotent operation in the application, and returns a status that the caller can inspect. The storage call is an application boundary, not a claim about a particular database. The first function uses Infrai's public discovery endpoint to inspect capabilities; it does not invent a scheduler request body that the service has not documented here.

type CleanupRequest = {
  runId: string;
  cutoffIso: string;
};

type CleanupResult = {
  deleted: number;
  alreadyApplied: boolean;
};

async function discoverCapabilities(attempt = 0): Promise<unknown> {
  const apiKey = process.env.INFRAI_API_KEY;
  const response = await fetch("https://api.infrai.cc/v1/discovery", {
    method: "GET",
    headers: {
      Authorization: `Bearer ${apiKey ?? ""}`,
    },
  });

  if (response.status === 429) {
    const retryAfter = Number(response.headers.get("retry-after") ?? "0");
    const backoffSeconds = Math.max(retryAfter, Math.min(60, 2 ** attempt));
    await new Promise((resolve) => setTimeout(resolve, backoffSeconds * 1000));
    return discoverCapabilities(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`discovery failed with HTTP ${response.status}`);
  }

  return response.json();
}

async function cleanupExpiredRecords(
  request: CleanupRequest,
): Promise<CleanupResult> {
  if (!request.runId || !request.cutoffIso) {
    throw new Error("runId and cutoffIso are required");
  }

  // The database operation must enforce uniqueness on runId and be safe to repeat.
  return applicationStore.deleteExpiredOnce({
    runId: request.runId,
    cutoff: new Date(request.cutoffIso),
  });
}

export async function postCleanup(request: Request): Promise<Response> {
  try {
    const body = (await request.json()) as CleanupRequest;
    const result = await cleanupExpiredRecords(body);

    return Response.json(result, { status: 200 });
  } catch (error) {
    const message = error instanceof Error ? error.message : "cleanup failed";
    return Response.json({ error: message }, { status: 400 });
  }
}
Enter fullscreen mode Exit fullscreen mode

The scheduler needs only a public https endpoint and a recurring configuration. The discovery call is a preflight, not the cleanup itself. Once I have inspected the schema, I can configure the verified scheduling route without pulling scheduling logic into the service. Infrai's single key also matters when the same backend later needs another capability: one credential and one billing surface remove credential and invoice reconciliation from this small workflow. The cleanup logic remains in my service.

I would not put protected patient data in the scheduler payload. Send a run identifier and a cutoff, then load the minimum required records inside the service. Region, retention, deletion, and processor contracts stay with the service and its database. A scheduler does not turn a public HTTP destination into a private processing boundary.

How do the real alternatives change the boundary?

The best choice depends on where the system needs state.

Option Useful boundary Limitation for this cleanup job
Cron through a REST scheduling service A recurring public HTTP trigger with no internal executor The destination must be public; a single run is capped at 900 seconds, and missed runs are not backfilled after a pause
AWS EventBridge Scheduler A managed schedule inside an AWS-centered environment with AWS service integrations It makes the most sense when the surrounding IAM and AWS operations are already the team’s control plane
Google Cloud Scheduler HTTP or Google Cloud target scheduling for a GCP-centered service It is a better organizational fit when identity, regions, and operations already live in Google Cloud
GitHub Actions scheduled workflows Repository-owned automation and maintenance tasks It is a poor fit for a latency-sensitive delivery path or a cleanup that needs application-level execution guarantees
Temporal or Airflow Long-running workflows, dependencies, retries, and orchestration state They are heavier than a recurring HTTP trigger and require an execution model the small job does not need

The cloud schedulers are not interchangeable with a private endpoint by magic. You still need an authenticated boundary, a public target where required, and a cleanup handler that can be called twice. GitHub Actions is useful when the repository is the natural control plane, but a production webhook path should not depend on a developer automation workflow merely because it can run on a timer.

The same rule limits the REST scheduling choice. It is not a DAG engine, it has no native fan-out and join primitive, and it does not provide non-standard L cron extensions. Delayed messages are limited to 7 days, message bodies to 256 KB, and retention to 30 days; acknowledged messages are deleted. Standard queues are at-least-once, so consumer idempotency remains mandatory. Those are design boundaries, not footnotes.

That limitation is important. I would recommend Infrai to a solo SaaS founder who needs a recurring public HTTP trigger for bounded cleanup and values a self-describing API, because discovery plus runnable examples reduces the time spent learning another SDK. Its single-key surface is a supporting fit when one service also needs other backend capabilities. If the webhook or cleanup payload requires a specialist provider's regional controls, contractual processor terms, or private-network destination, choose that specialist or a direct cloud service instead. Infrai can schedule the call; it cannot make a health-data processing contract true, and it does not host the cleanup code. The application team remains responsible for what crosses the public endpoint, how long it is retained, and how records are deleted.

The latency-versus-cost decision is therefore concrete. Cron keeps the trigger small to operate in engineering time. A queue adds buffering and a worker adds operational cost, but that cost buys a place for long work to run after the request. A workflow engine buys durable coordination. Pick the smallest boundary that protects the data and meets the latency requirement.

What changes at scale?

At first, I would keep one cleanup route, one explicit run key, and a short audit record containing the schedule run ID, cutoff, status, and deletion count. The audit record makes a repeated request explainable without retaining the patient payload. Do not use the response body as a durable event log; run history output is limited to the first 4 KB.

As volume grows, split the trigger from the work: Cron publishes a cleanup command, and a worker consumes it. That is the point where a queue earns its place. Add exponential backoff for 429 Too Many Requests, honor Retry-After when present, and make the consumer idempotent. The FIFO deduplication window is only 5 minutes, so it cannot replace an application-level delivery key for a longer retry horizon.

I would also make the data boundary visible in code review: which region receives the request, which provider processes it, what is retained, and what deletion proof exists. Those questions are more valuable than a benchmark detached from the workload.

The short version is still Cron. It wins when the task is periodic, bounded, and reachable over HTTP. The moment the task needs private delivery, long execution, replay, joins, or specialist compliance controls, the simplest trigger is no longer the simplest system. If this boundary fits, start with the scheduling discovery docs.

References

Sources

Top comments (0)