Any system that calls your endpoint on a schedule, or retries it after a failure, will sooner or later call it twice for the same piece of work. A network timeout after your handler finished, a retry after a slow response, two servers firing the same cron: each of these runs the job again. If your handler sends an email or charges a card, "again" is a real problem. The fix is to make the handler idempotent: running it twice has the same effect as running it once.
Why duplicates are unavoidable
Consider a scheduler calling POST /jobs/send-invoices. Your handler takes 35 seconds. The caller's timeout is 30 seconds, so it records a failure and retries. Your first run finishes anyway. Now the invoices go out twice.
You can't fix this purely on the caller's side. Without the caller knowing exactly what your handler completed, "at least once" delivery is the only safe guarantee, and handling duplicates is the handler's job.
Technique 1: an idempotency key
Give each unit of work a unique key and record it when you process it. If the key has been seen, skip the work.
# Flask + PostgreSQL example
from flask import Flask, request
import psycopg
app = Flask(__name__)
@app.post("/jobs/send-reminder")
def send_reminder():
payload = request.get_json()
key = payload["job_key"] # e.g. "reminder:user42:2026-11-01"
with psycopg.connect(DSN) as conn:
cur = conn.execute(
"INSERT INTO processed_jobs (key) VALUES (%s) ON CONFLICT DO NOTHING",
(key,),
)
if cur.rowcount == 0:
return {"status": "duplicate, skipped"}, 200
send_email(payload["user_id"])
return {"status": "sent"}, 200
CREATE TABLE processed_jobs (
key text PRIMARY KEY,
processed_at timestamptz DEFAULT now()
);
The unique constraint does the hard work: if two copies arrive at the same moment, only one insert succeeds.
Return a 2xx for duplicates. If you return an error, the caller retries again.
Choosing the key
The key should describe the work, not the delivery attempt:
- Good:
invoice-run:2026-11,reminder:user42:trial-day-3 - Bad: a random UUID generated per request, a timestamp of when the call arrived
For recurring jobs, include the period: daily-report:2026-11-01. Then a second call on the same day is skipped, and tomorrow's call runs normally.
Technique 2: make the work itself naturally idempotent
Some operations are safe to repeat by design:
-- Not idempotent
UPDATE accounts SET credits = credits + 100 WHERE id = 42;
-- Idempotent
UPDATE accounts SET plan = 'pro', renewed_until = '2026-12-01' WHERE id = 42;
Setting a value is safe; incrementing it isn't. Upserts, "mark as done" flags and "recalculate from source" jobs are all naturally safe to run again.
Technique 3: claim, then process
For batch jobs, mark each item as claimed in the same step as selecting it:
UPDATE emails
SET status = 'sending', claimed_at = now()
WHERE id IN (
SELECT id FROM emails
WHERE status = 'queued'
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED
)
RETURNING id, recipient;
Two overlapping runs each get different rows, and a re-run picks up only what's left.
External side effects
The idempotency key protects your database, but what about the payment provider or email API you call? Many such APIs accept their own idempotency key header. Pass your job key through so the provider can also drop duplicates:
requests.post(
"https://api.payments.example/charges",
headers={"Idempotency-Key": key},
json={...},
)
If an API has no such option, record "sent" in your database before marking the job complete, and accept that a crash in between could still send twice. Design for the rare duplicate to be harmless.
Respond fast, work later
A long-running handler makes duplicates more likely, because callers time out. Accept the request, store the work, and return 202 Accepted quickly, then process it in a background worker. The idempotency key goes with it.
Test it
The simplest test: call the endpoint twice with the same payload, in parallel.
for i in 1 2; do
curl -s -X POST localhost:5000/jobs/send-reminder \
-H 'content-type: application/json' \
-d '{"job_key":"test-1","user_id":42}' &
done; wait
One response should say "sent" and the other "duplicate".
Where retries come from
A hosted scheduler like delay.run calls your URL on a cron schedule or after a delay and retries failed jobs with backoff, logging each attempt. Retries are what make jobs reliable, and idempotent handlers are what make retries safe.
Related reading
- How to retry failed cron jobs with exponential backoff in Bash and Python
- Heartbeat checks: how to know when a scheduled job didn't run
- How to log cron job output so you can debug it later
- Cron vs systemd timers: which should run your scheduled jobs?
- Dead letter queues for HTTP jobs: what they are and how to use one
Top comments (2)
One trap with the framework-level version of this, in Laravel at least: withoutOverlapping() keeps its lock for 24 hours by default. If a deploy kills the run halfway, the lock stays and the job quietly skips every run until it expires. My inbox sync runs every 15 minutes, so I set its lock to 10.
And that's why the lock alone isn't enough. A run cut off halfway gets picked up by the next one, so the handler still has to be fine doing that work twice.
Write the effect key before the side effect, not after. If the job dies between the send and the record, the retry sends again. A row saying 'sent for run X', checked first, turns the second run into a no-op.