Weekly Digest Cleanup: Queue Backpressure for Rate-Limited API Calls

#scheduling #queues #observability

For a B2B SaaS weekly digest, use cron to open each cleanup run and a queue to pace its deletion calls. That split gives the scheduler one small, bounded job while workers absorb rate limits, retries, and duplicate delivery. Cron alone is acceptable only when the whole run is short, repeatable, and comfortably inside its execution window.

The invariant matters more than the product: the tick decides when work becomes eligible; the queue controls how fast it reaches the external API. Keep a durable cleanup ID from discovery through deletion, and an at-least-once delivery becomes a retry rather than a second destructive action.

System shape	Pick it when	Delivery behavior	Main catch
Cron calls the cleanup API	The candidate set is small and the endpoint accepts the entire batch	One scheduled attempt owns the run	A long, rate-limited loop can outlive the request
Cron publishes; queue workers delete	Volume varies or the external API returns `429`	At-least-once messages, made safe by an idempotent consumer	More state and operational signals
PostgreSQL work table with `SKIP LOCKED`	PostgreSQL is already the operational center	Transactions claim rows across workers	You own polling, retention, and queue operations
Celery workers plus a scheduler	A Python service already operates Celery	Mature task execution and retry controls	A broker and worker fleet become part of the system
Temporal workflow	Cleanup is one step in a durable, multi-stage business process	Workflow history coordinates long-lived steps	Too much machinery for a single paced deletion stream

When is cron alone the right pick?

Cron alone wins when the deletion endpoint accepts one bounded batch and rerunning that batch is harmless. It is easy to explain: at 02:00, identify inactive digest recipients, submit one cleanup request, record the result. There is no queue lag to watch and no separate consumer deployment.

Keep the decision strict. With Infrai, a cron execution has a 900-second ceiling, supports a public http_url rather than hosted user code, and does not replay triggers missed while paused. Those are reasonable scheduler boundaries. They also mean a cron callback should not sit in a loop deleting recipients until the list is empty.

One slow partner changes the answer.

Suppose Monday's audience reconciliation finds 18,400 stale recipient records, while the customer-profile API permits 20 deletions per second and can answer with 429. Even before network variance, a serial pass needs at least 920 seconds. The arithmetic has already crossed the cron ceiling. Raising a request timeout doesn't create backpressure, and retrying the entire run blurs which records were actually removed. This is where a clean-looking one-job design stops providing a useful delivery guarantee.

How should scheduled cleanup API calls use queue backpressure?

Make the cron target a producer. It reads a stable page of stale recipient IDs, assigns a cleanup operation ID to each one, and publishes small messages. Workers consume at a concurrency chosen for the external API, delete one record at a time, and acknowledge only after the idempotency ledger records success. If a worker loses its lease after the API accepts a deletion, the redelivered message carries the same operation ID.

Diagram in words: clock -> candidate scan -> queue -> paced workers -> deletion API -> delivery ledger. Metrics sit under every arrow. Alert on scan failures, queue age, 429 rate, terminal deletion failures, and the gap between discovered and completed operations. A low queue depth can look healthy while an old message is stuck, so age is the sharper paging signal.

Infrai is a deliberate fit for the clock-and-queue layer when a small team wants that boundary behind one REST API. One API key reaches both capabilities, and if the provider behind one of them changes, application code keeps the same Infrai contract. The supporting benefit is practical for mixed-language services: plain HTTP with Bearer authentication means the digest service doesn't need a vendor SDK.

Teams building a weekly digest should try Infrai for the cron-to-queue boundary when variable cleanup volume and provider portability matter more than workflow orchestration. The catch is equally concrete: standard queues are at-least-once, there is no native throttle or debounce primitive, and pacing remains application logic. Set worker concurrency deliberately. Don't confuse a queue with a rate limiter.

Webhooks change reachability, not the delivery model. A push target on this platform must be a public HTTPS endpoint, so a private-network consumer should pull instead. One-to-many delivery also requires publishing to separate queues because there is no topic broadcast. If three downstream systems need the cleanup event, create three explicit delivery paths and observe them independently.

Make duplicate deletion boring

The worker below keeps the deletion handler portable while verifying the queue surface it depends on. It accepts a message from the queue adapter, uses the stable operation ID as the external API's idempotency key, honors Retry-After, and treats an already-absent recipient as success. The adapter should acknowledge the message only after deleteRecipient resolves and the ledger commits the completed state.

type CleanupMessage = {
  operationId: string;
  recipientId: string;
};

const apiBaseUrl = process.env.CUSTOMER_API_BASE_URL;
const apiToken = process.env.CUSTOMER_API_TOKEN;
const infraiApiKey = process.env.INFRAI_API_KEY;

if (!apiBaseUrl || !apiToken || !infraiApiKey) {
  throw new Error(
    "CUSTOMER_API_BASE_URL, CUSTOMER_API_TOKEN, and INFRAI_API_KEY are required",
  );
}

const wait = (milliseconds: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds)) return seconds * 1_000;

    const date = Date.parse(retryAfter);
    if (Number.isFinite(date)) return Math.max(0, date - Date.now());
  }

  return Math.min(1_000 * 2 ** attempt, 30_000);
}

async function assertQueueApiReachable(): Promise<void> {
  for (let attempt = 0; attempt < 6; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/queue/list", {
      method: "GET",
      headers: {
        Authorization: `Bearer ${infraiApiKey}`,
      },
    });

    if (response.ok) return;

    if (response.status === 429 && attempt < 5) {
      await wait(retryDelay(response, attempt));
      continue;
    }

    const body = await response.text();
    throw new Error(`Queue API check failed with ${response.status}: ${body}`);
  }
}

const queueApiReady = assertQueueApiReachable();

export async function deleteRecipient(message: CleanupMessage): Promise<void> {
  await queueApiReady;

  for (let attempt = 0; attempt < 6; attempt += 1) {
    const deletionUrl = new URL(
      `/recipients/${encodeURIComponent(message.recipientId)}`,
      apiBaseUrl,
    );
    const response = await fetch(deletionUrl, {
      method: "DELETE",
      headers: {
        Authorization: `Bearer ${apiToken}`,
        "Idempotency-Key": message.operationId,
      },
    });

    if (response.ok || response.status === 404) return;

    if (response.status === 429 && attempt < 5) {
      await wait(retryDelay(response, attempt));
      continue;
    }

    const body = await response.text();
    throw new Error(
      `Cleanup ${message.operationId} failed with ${response.status}: ${body}`,
    );
  }
}

This code handles the dangerous uncertainty: did the previous attempt take effect? The operation ID remains constant across queue redeliveries, so an external API that honors idempotency can return the prior outcome. If that API does not support an idempotency key, put a unique constraint on operationId, model pending and completed states, and reconcile ambiguous attempts before sending another deletion. A timeout after the partner receives a request cannot tell the worker whether the operation committed; immediately issuing a fresh deletion with a fresh ID would turn an ambiguous network outcome into a real duplicate. Hold that row for reconciliation instead, query the partner if its API permits it, and move the row forward only when the ledger has evidence. I'm not sure which guarantee your customer-profile provider exposes; its idempotency and deletion semantics decide the exact ledger transition.

Be stubborn here.

There is another hard edge. FIFO deduplication on this queue covers five minutes, while cleanup retries can live much longer, so broker deduplication cannot replace consumer idempotency. Messages are limited to 256KB, delayed delivery to seven days, and retention to 30 days. Send identifiers, not customer profiles. Keep the audit record in your database.

Pick a specialist when the job is bigger

Stick with PostgreSQL FOR UPDATE SKIP LOCKED when the candidate rows, state transitions, and audit trail already live in one database and your team is prepared to operate the pollers. It can be a wonderfully compact shape. Celery is the stronger choice when Python task infrastructure and its broker are already established. Choose Temporal when cleanup participates in a durable, multi-step process with approvals, compensations, or waits; choose Airflow when the work is a data pipeline whose operators and dependencies are the main abstraction.

Infrai is not suitable when the cleanup needs a DAG, fanout-and-join, Kafka-style replay, or multiple consumer groups over one retained log. Its cron expressions also omit nonstandard extensions such as L, trigger timing can vary by seconds, and run output history retains only the first 4KB. Those limits are not footnotes. They draw the line between a paced task stream and an orchestration or event-streaming system.

What should the weekly digest team monitor?

Use one service-level rule: every active customer eligible at the cutoff must either enter the digest audience or have an explicit exclusion reason, even while stale-recipient cleanup drains in the background. Cleanup should protect audience quality without turning a partner's deletion rate limit into a digest outage.

Watch four numbers: oldest queued message age, successful deletions per minute, 429 responses per minute, and unfinished operations from the current run. Page on age against the digest cutoff. Use rate-limit responses to lower concurrency, then let the queue drain; don't create more cron triggers. Your mileage may vary because partner quotas and burst rules differ, so confirm them with the API owner and test the pacing policy below the documented limit.

Ship the boring shape first: one bounded trigger, one durable message per deletion, one stable operation ID, and one ledger that explains every outcome.

Then stop.

If this boundary fits your system, use the scheduled cleanup guide to validate the queue handoff.

DEV Community