Moving scheduled tasks off a cron box and onto serverless functions solves the "one server, one crontab, one single point of failure" problem. It also introduces a quieter one: a job that used to fail loudly in a log you'd eventually read now fails silently, retries in ways you didn't design, and sometimes runs twice. The reliability work doesn't disappear when you delete the crontab — it moves into your function code and your queue configuration, and most teams discover this the first time a nightly billing job double-charges someone.
This is the follow-up to migrating scheduled tasks to serverless: not how to move the jobs, but how to run them without getting paged. The three things that actually matter are delivery semantics (at-least-once, almost always), idempotency (so at-least-once is safe), and observability (so you find out a job didn't run, not just when it errors).
Why does the same job sometimes run twice?
Because nearly every serverless event source is at-least-once, not exactly-once. A scheduled EventBridge rule, an SQS message, a Pub/Sub push, an Azure Queue trigger — all of them can deliver the same event more than once. This isn't a bug you can configure away; it's a property of distributed message delivery. A network blip between the broker and your function acknowledgment means the broker didn't hear "I got it," so it redelivers.
The consequence: any job that mutates state — sending email, charging a card, incrementing a counter, writing a row — must be safe to run twice with the same input. On a single cron box this rarely bit you because there was one process and one clock. In a serverless fabric with retries built in, duplicate execution is the default, and the code has to assume it.
The takeaway: treat every scheduled function as if it will be invoked at least twice with identical input, because eventually it will be.
How do you make a job safe to run twice?
Idempotency. The practical pattern is an idempotency key plus a durable record of what you've already processed. Derive a stable key from the event (not a random UUID generated inside the handler — that changes on every retry), then check-and-set before doing the real work.
import hashlib
def idempotency_key(event: dict) -> str:
# Stable across retries: same event -> same key.
raw = f"{event['job']}:{event['target_date']}:{event['user_id']}"
return hashlib.sha256(raw.encode()).hexdigest()
def handle(event, db):
key = idempotency_key(event)
# Atomic insert; fails if the key already exists.
inserted = db.try_insert_processed(key)
if not inserted:
return {"status": "duplicate_skipped", "key": key}
do_the_actual_work(event)
db.mark_completed(key)
return {"status": "done", "key": key}
Two details make or break this. First, the insert has to be atomic — a INSERT ... ON CONFLICT DO NOTHING in Postgres, a conditional write in DynamoDB with attribute_not_exists. A read-then-write leaves a race window where two concurrent retries both see "not processed." Second, decide what happens if the process dies after try_insert_processed but before mark_completed. Storing an explicit state (pending / completed) rather than mere existence lets a later retry detect a half-finished job and either resume or alert, instead of silently skipping it as a "duplicate."
The takeaway: an idempotency key only helps if the check-and-set is atomic and the record captures completion state, not just "seen."
What happens to a job that keeps failing?
This is where a dead-letter queue (DLQ) earns its keep. Without one, a poison message — malformed input, a downstream dependency that's down, a bug that throws on one specific record — gets retried until the source gives up, then vanishes. You lose the job and the evidence.
The pattern is: bounded retries with backoff, then route the failure to a DLQ instead of dropping it. For an SQS-triggered Lambda, that's a redrive policy on the source queue:
{
"RedrivePolicy": {
"deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:jobs-dlq",
"maxReceiveCount": 5
}
}
After five failed receives, the message lands in jobs-dlq where it sits durably until a human or a redrive job looks at it. The equivalents elsewhere: Google Cloud Pub/Sub has a deadLetterPolicy with maxDeliveryAttempts; Azure Service Bus has built-in dead-lettering with a max delivery count; a plain Postgres-backed queue needs you to add a failure_count column and a WHERE failure_count < 5 filter yourself.
One caution I learned the hard way: a DLQ with nothing watching it is just a slower way to lose data. The queue depth on the DLQ has to be a monitored, alerting metric. ApproximateNumberOfMessagesVisible > 0 on a DLQ should page someone, because it means a job is failing in a way retries won't fix.
The takeaway: a dead-letter queue converts silent data loss into a visible, alertable backlog — but only if something is actually watching its depth.
How do you even know a scheduled job didn't run?
This is the failure mode serverless makes worse, and it's the one teams under-invest in. Error alerting tells you when a function ran and threw. It tells you nothing when the function never fired — a disabled EventBridge rule, a bad IAM permission, an account-level throttle, a deploy that removed the trigger. The nightly report just... doesn't show up, and you find out when a stakeholder asks where it is.
The fix is a heartbeat or dead-man's-switch pattern: the job records a successful completion, and a separate check alerts on the absence of that record within the expected window.
def handle(event, db, clock):
do_the_actual_work(event)
db.record_heartbeat(job="nightly_report", at=clock.now())
# A separate, independently-scheduled watcher:
def check_liveness(db, clock):
last = db.last_heartbeat("nightly_report")
if last is None or (clock.now() - last).total_seconds() > 26 * 3600:
alert("nightly_report has not completed in over 26 hours")
The watcher runs on its own schedule and, ideally, its own infrastructure — if both the job and its monitor depend on the same broken EventBridge rule, neither fires and you're blind. Hosted dead-man's-switch services (the "ping this URL every day or we alert you" kind) exist precisely because the monitor should not share a failure domain with the thing it monitors.
The takeaway: alert on the absence of success, not just the presence of errors — a job that never starts produces no error to catch.
Which reliability mechanism do I actually need?
Not every job needs the full apparatus. Match the mechanism to the job's blast radius.
| Job characteristic | Idempotency | Dead-letter queue | Heartbeat / liveness |
|---|---|---|---|
| Read-only (e.g. cache warmer) | Optional | Nice to have | If business-critical |
| Mutates state (billing, emails) | Required | Required | Required |
| Fan-out over many records | Required (per record) | Required | On the orchestrator |
| Idempotent by nature (full recompute) | Built in | Recommended | If deadline-sensitive |
| One-shot, non-critical | Skip | Skip | Skip |
The mistake I see most is applying all three to a harmless cache-refresh job while a money-touching job has none of them. Reliability effort should follow consequences.
The takeaway: a cache warmer and a payment run deserve different reliability budgets — spend where a duplicate or a miss actually costs something.
What does this cost you in complexity?
Honestly: a fair amount, and it's worth naming the trade-off. A crontab line is one line. The reliable serverless equivalent is a function, an idempotency table, a DLQ, an alarm on the DLQ, a heartbeat table, and a separate liveness monitor. That's real operational surface, and for a genuinely trivial job on a single stable machine, plain cron is still a defensible choice — don't cargo-cult serverless onto a job that a cron line already handles fine.
The reason to pay the complexity cost is when you have many jobs, need them to survive an instance dying, or want per-job scaling and isolation. At that point the crontab-on-one-box model fails in ways that page you at 3AM regardless, and you're just choosing which complexity to own.
The takeaway: serverless reliability is more moving parts than cron — adopt it for scale and fault-tolerance, not because scheduled functions sound modern.
Bottom line
If you've moved scheduled tasks to serverless, assume at-least-once delivery and make every state-changing job idempotent with an atomic check-and-set — this is non-negotiable, not an optimization. Add a dead-letter queue with a monitored depth for anything whose failure you can't afford to lose silently, and add a heartbeat-plus-liveness check for any job where "it never ran" is a real incident. Skip all three only for read-only, non-critical jobs where a duplicate or a miss costs nothing. The goal isn't maximal machinery — it's that no job fails silently at 3AM without something, somewhere, noticing.
Top comments (0)