TL;DR: Put one authenticated HTTPS endpoint in front of a bounded database cleanup, schedule it at a cadence the session policy can tolerate, and make both cleanup and outbound webhook delivery idempotent. The least complex design is a single cron trigger that claims a small batch, records one stable delivery key per event, and returns before its execution budget is exhausted. Keep aggregate counters and a short-lived sample of failures; do not retain a log line for every successful session.
For an edtech platform, the expensive part is rarely the cron expression. It is the multiplied telemetry around each expired learning session: scheduler request logs, database traces, webhook attempts, label-rich metrics, and retained payloads. Before selecting any scheduler, estimate that multiplication. If a planning model assumes 400,000 expirations per day, three webhook attempts at the retry ceiling, and four 700-byte records per attempt, the upper-bound event volume is 3.36 GB per day before indexing and replicas: 400,000 × 3 × 4 × 700. Those are illustrative inputs, not benchmark results. Replace them with a count and sampled byte size from your own system.
The dominant term is usually attempts multiplied by records per attempt. A cheaper trigger does not change it. Emitting one compact outcome record per logical delivery, plus counters for attempts, status classes, and latency buckets, does.
Can a cheap HTTP cron make expired sessions cleanup reliable?
Treat the endpoint as a trigger, not as the cleanup engine. It authenticates the caller, creates or resumes a bounded run, claims eligible rows, and stops at a fixed limit. A second invocation must be harmless. This matters because cron can overlap, an HTTP client can retry after losing a response, and a 429 Too Many Requests response may include Retry-After, which tells the client how long to wait.
A request can carry a stable key derived from the scheduled time slot. The exact header is an application contract; HTTP does not standardize a general-purpose idempotency header. Here is a reproducible invocation for a hypothetical endpoint:
curl --fail-with-body \
--request POST \
--header "Authorization: Bearer ${CLEANUP_TOKEN}" \
--header "Idempotency-Key: expire-sessions-2026-09-22T01:00Z" \
--header "Content-Type: application/json" \
--data '{"batch_limit":500,"execution_budget_seconds":45}' \
https://jobs.example.eu/expire-learning-sessions
The endpoint should use a transaction or an atomic claim operation so concurrent runs cannot own the same session. Expiration is a state transition guarded by the current state and expiry timestamp, not an unconditional delete. The related webhook event gets a durable event ID, such as the session ID plus transition version. Every retry reuses that ID. A receiver can then reject a duplicate without guessing from timestamps or payload equality.
Duplicates are normal.
Success means the batch was accepted and processed within its stated bound. Authentication failure is permanent. Capacity pressure is temporary and should produce a retryable response with an explicit delay where possible. Do not return success merely because work was queued unless that asynchronous contract is documented and the run can be inspected by ID.
Step 1: Schedule a bounded, replayable trigger
A conventional five-field crontab entry expresses minute, hour, day of month, month, and day of week. Its execution time depends on the cron daemon's configured time zone, so make the intended zone explicit in deployment configuration and verify it after daylight-saving changes. For expired sessions, the cadence should follow the product's tolerance for stale authorization, not the smallest interval a scheduling service offers.
The following command is suitable as the command portion of a cron entry or a scheduler's HTTP action. It fails on HTTP errors, limits connection and total time, and retries transient failures. The idempotency key stays fixed for that scheduled slot across retries.
curl --fail-with-body \
--connect-timeout 5 \
--max-time 55 \
--retry 3 \
--retry-all-errors \
--request POST \
--header "Authorization: Bearer ${CLEANUP_TOKEN}" \
--header "Idempotency-Key: expire-sessions-2026-09-22T01:00Z" \
--header "Content-Type: application/json" \
--data '{"batch_limit":500,"execution_budget_seconds":45}' \
https://jobs.example.eu/expire-learning-sessions
Keep the scheduler timeout longer than the endpoint budget but shorter than the schedule interval. That ordering gives the handler time to stop cleanly and prevents routine overlap. It is a decision rule, not a universal set of numbers: measure the p95 batch duration, then leave headroom for connection setup and the response.
One trap deserves attention. If the scheduler generates a fresh idempotency key on every network retry, the endpoint sees distinct runs. The retry control has then defeated the deduplication control. Generate the key from the logical time slot, and retain the corresponding run record for at least the maximum retry window plus expected clock skew.
Step 2: Separate session expiry from webhook delivery
Do not hold a database transaction open while calling a school, learning-management system, or notification endpoint. In one short transaction, claim expired active sessions, move them to the expired state, and insert an outbox event with a unique delivery identity. A separate worker delivers that event. This ordering prevents a remote timeout from extending locks and preserves the event when the process stops after commit.
The delivery worker sends the same event ID on every attempt:
curl --fail-with-body \
--request POST \
--header "Content-Type: application/json" \
--header "X-Webhook-Event-Id: session_82-expired-v3" \
--data '{"event":"learning_session.expired","session_id":"session_82","version":3}' \
https://partner.example.edu/hooks/session-events
A timeout leaves the outcome unknown. Retry the same event, rather than manufacturing a replacement. The receiver should persist the event ID under a uniqueness constraint before applying side effects, then return the prior outcome for a duplicate. Sender-side idempotency alone cannot prevent a receiver from processing the same request twice.
Classify responses deliberately. A 429 is rate limiting and may include Retry-After; a delayed retry is appropriate. Other status classes need a local policy, because HTTP alone does not tell you whether a particular partner will recover or whether a payload is permanently invalid. Cap attempts, add randomized delay, and move exhausted events to a reviewable terminal state. Do not retry forever.
That separation also makes deployment safer. Ship the schema constraint and outbox write before enabling the schedule, deploy the worker with delivery disabled, exercise a small batch, and then turn on delivery. Rollback can pause new triggers while already committed events remain inspectable.
Step 3: Count cardinality before choosing retention
Start with dimensions that answer operational questions: environment, job name, outcome, response status class, and perhaps partner tier. Session IDs, event IDs, learner IDs, full URLs, and raw error bodies do not belong in metric labels. Each unbounded identifier creates another potential time series; the useful question is not how many metrics exist, but how many distinct label combinations can exist at once.
Use three telemetry layers:
- Counters record sessions claimed, sessions expired, webhook attempts, deduplicated deliveries, terminal failures, and HTTP outcome classes.
- Histograms record batch duration, claimed rows, and delivery latency with deliberately limited buckets.
- Structured event records cover failures and a controlled sample of successes, with sensitive fields removed and event IDs retained only as long as investigation requires.
For retention planning, calculate daily bytes by class: events × average encoded bytes × copies. Then multiply by days retained. Suppose a planning sample finds a compact failure record is 900 bytes, 0.5% of 400,000 logical deliveries enter the retained failure class, and storage keeps two copies for 30 days. That class is about 108 MB: 400,000 × 0.005 × 900 × 2 × 30. Again, this is arithmetic over hypothetical inputs. Measure encoded records, because an in-memory object size is not the stored and indexed size.
Do the same calculation for traces. A trace per attempt can grow with retries, while a trace per sampled logical delivery does not. Keep 100% of terminal failures if their volume is bounded, sample routine successes, and retain aggregate counters longer than high-detail records. This preserves trend detection without paying to store every normal path.
Small labels are not automatically safe. A partner identifier with 50 current values can become 5,000 after onboarding and testing environments are included. Put the exact partner and event ID in sampled event records, not in every metric.
Step 4: Verify failure behavior before enabling the cron
Test duplicates first. Send the same slot key twice, concurrently if the test environment permits, and verify that the second call reports the existing run rather than expiring more rows. Then test a lost response by forcing the client timeout below the handler duration and replaying the same key. The final database state and outbox event count must remain unchanged.
curl --fail-with-body \
--max-time 2 \
--request POST \
--header "Authorization: Bearer ${CLEANUP_TOKEN}" \
--header "Idempotency-Key: expiry-drill-fixed-slot" \
--header "Content-Type: application/json" \
--data '{"batch_limit":10,"execution_budget_seconds":5}' \
https://staging-jobs.example.eu/expire-learning-sessions
Then exercise the receiver's 429 path, its timeout path, and a permanent validation failure. Assert the attempt ceiling and terminal-state transition. Check that logs do not contain bearer tokens, learner attributes, or entire webhook bodies. Finally, compare the observed counts: claimed sessions should reconcile with state transitions, outbox insertions, successful deliveries, pending retries, and terminal failures. A mismatch is more informative than another page of request logs.
Operationally, alert on work that is old, not merely on one failed invocation. A single cron request can fail while the next bounded run drains the backlog. Age of the oldest eligible session and age of the oldest pending outbox event expose user impact directly. Invocation failures remain useful diagnostic counters, but they are not the service objective.
The deliberate loss is detail. After the short failure-retention window expires, an operator may know that 17 deliveries failed in an hour but be unable to reconstruct every response body. That is an acceptable trade when the bodies may contain sensitive educational context and when long retention multiplies storage and indexing cost. Preserve durable business state and delivery outcomes; discard routine payload copies.
A scheduler choice can now be evaluated without a vendor ranking: it must call HTTPS with authentication, preserve or accept a stable idempotency key, expose timeout and retry controls, respect the required region and data-handling boundary, and provide enough execution history to distinguish a missed trigger from a rejected request. Price comes after those gates, because a low trigger fee cannot compensate for duplicate notifications or uncontrolled telemetry.
This design has limits. A public endpoint is not suitable when policy forbids inbound Internet traffic, even with authentication; choose an internal scheduler or a queue consumer inside the same network boundary. The short request model is also a poor fit when one indivisible cleanup action exceeds the HTTP execution budget. In that case, the trigger should enqueue partitions for workers, accepting extra queue state and operational complexity in exchange for controlled execution time. The telemetry policy carries its own trade-off: aggressive sampling lowers retained bytes, but it can hide a rare successful path that later proves relevant to an investigation. Those boundaries should be decided before scheduler price or interface convenience.
Further reading
- crontab(5), Linux manual page: https://man7.org/linux/man-pages/man5/crontab.5.html
- MDN, 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Top comments (0)