Duplicate Webhooks: Build an Idempotent Inbox in Python

A payment provider sends your server a webhook. Your server is busy, so the provider doesn't hear back in time, and it sends the same webhook again. Now two copies are on their way.

If your code adds a payment for each one, the customer is recorded as paying twice. If it sends a welcome email for each, they get two.

Duplicates aren't a rare bug. They're how most webhook systems are designed to work. This guide explains why, shows the tempting fixes that fail, and builds an inbox that handles duplicates, slow workers and crashes. It uses only Python and SQLite, and all of it is tested.

If webhooks are new to you, start with what a webhook is and how it works.

Why duplicates happen

Most webhook providers promise at least once delivery. They'll keep trying until they hear that you got it, so a message may arrive more than once, but a message is rarely lost.

That's a sensible choice for them. Losing a payment event is worse than sending it twice. But it hands the problem to you. Copies appear when:

• Your server was slow or returned an error, so the provider retried
• The acknowledgement was lost on the way back
• Someone clicked "redeliver" in the provider's dashboard
• An attacker replayed a real delivery

The answer is to make your handling idempotent, which means that doing it twice has the same result as doing it once. Setting a customer's email to a value is idempotent. Adding 25 dollars to their balance isn't.

The same idea appears in APIs, where many services accept a unique "idempotency key" so that a repeated request is ignored. If you're not sure what an API is, here's what an API is and how it works.

Three fixes that don't work

The obvious ideas all have a hole. I tested each one.

1. Remember it in memory. Keep a list of IDs you've handled. It's gone when the program restarts, and every extra process has its own separate list.

2. Check, then act. Look the ID up. If it's new, do the work, then record it. This looks right, and it breaks when two copies arrive at the same moment. Both look, both see "new", and both do the work. Here's the whole failure in a few lines. Save it as naive_check_then_act.py:

import sqlite3
import threading
import time

setup = sqlite3.connect("naive.db")
setup.execute("CREATE TABLE IF NOT EXISTS seen (id TEXT)")
setup.execute("DELETE FROM seen")
setup.commit()
work_done = []


def handle(delivery_id):
    db = sqlite3.connect("naive.db")
    if db.execute("SELECT 1 FROM seen WHERE id = ?", (delivery_id,)).fetchone():
        return  # already handled
    time.sleep(0.2)  # the real work takes a moment
    work_done.append(delivery_id)
    db.execute("INSERT INTO seen VALUES (?)", (delivery_id,))
    db.commit()


copies = [threading.Thread(target=handle, args=("evt_1",)) for _ in range(2)]  # two copies arrive together
for copy in copies:
    copy.start()
for copy in copies:
    copy.join()
print(f"The work was done {len(work_done)} times")

It prints:

The work was done 2 times

Two copies, two runs of the work, even though the code "checks first".

3. Record first, or act first. Order matters, and neither order works:

• If you record the ID before the work and crash in between, the provider's retry looks like a duplicate and is skipped. The event is lost for good.
• If you do the work before recording and crash in between, the retry does the work again.

Each of these failures was reproduced in my tests. The rest of this guide shows a design that avoids all three.

The design: an inbox

Four ideas fit together:

1. Store first, work later. When a delivery arrives, save it to a database and answer 200 straight away. The provider gets a quick answer, and the event is safe, even if the processing is slow or crashes.
2. Let the database decide what's a duplicate. Give each delivery's ID a unique key. The database refuses a second copy, even when two arrive at the same instant. It's the one place that can reliably settle a race.
3. Process separately, with leases. A worker takes a delivery and gets a time-limited lease on it. If the worker crashes, the lease runs out and another worker takes over.
4. Make the work safe to repeat. A crash at the wrong moment can still make a worker repeat something, so the effect itself should do no harm the second time.

What you need

• Python 3
• SQLite, which is built into Python. Nothing to install

Step 1: The inbox

Create inbox.py:

"""A durable inbox for webhook deliveries: store first, process later, and never twice."""
import os
import secrets
import sqlite3
import sys
import time

LEASE = 60        # seconds a worker may hold a delivery before another may take it over
MAX_ATTEMPTS = 5  # after this many tries, a delivery is set aside for a person to look at
RETRY_DELAY = 2   # seconds before the first retry, doubling each time


def connect(path=None):
    conn = sqlite3.connect(path or os.environ.get("INBOX_DB", "inbox.db"), timeout=30, isolation_level=None)  # we run transactions ourselves
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("""CREATE TABLE IF NOT EXISTS deliveries (
        id TEXT PRIMARY KEY, body TEXT, received REAL, status TEXT, attempts INTEGER DEFAULT 0,
        available_at REAL, leased_until REAL, owner TEXT, error TEXT)""")
    return conn


def store(conn, delivery_id, body, now=time.time):
    """Save a delivery. Returns False if this ID has been seen before. The database enforces that, not our code."""
    cursor = conn.execute("INSERT OR IGNORE INTO deliveries (id, body, received, status, available_at) VALUES (?, ?, ?, 'received', ?)",
                          (delivery_id, body, now(), now()))
    return cursor.rowcount == 1


def claim(conn, now=time.time):
    """Take one delivery to work on, or return None. Two workers can never take the same one."""
    conn.execute("BEGIN IMMEDIATE")  # only one worker at a time gets past this line
    try:
        t, owner = now(), secrets.token_hex(4)
        conn.execute("UPDATE deliveries SET status = 'failed', error = 'gave up: the worker kept crashing' "
                     "WHERE status = 'processing' AND leased_until < ? AND attempts >= ?", (t, MAX_ATTEMPTS))
        row = conn.execute("SELECT id, body FROM deliveries WHERE (status = 'received' AND available_at <= ?) "
                           "OR (status = 'processing' AND leased_until < ?) ORDER BY received LIMIT 1", (t, t)).fetchone()  # new work, or work whose worker vanished
        if row:
            conn.execute("UPDATE deliveries SET status = 'processing', attempts = attempts + 1, leased_until = ?, owner = ? WHERE id = ?",
                         (t + LEASE, owner, row[0]))
        conn.execute("COMMIT")
        return (row[0], row[1], owner) if row else None
    except Exception:
        conn.execute("ROLLBACK")
        raise


def finish(conn, delivery_id, owner):
    """Mark a delivery done. Returns False if the lease was lost, meaning someone else took it over."""
    return conn.execute("UPDATE deliveries SET status = 'done', leased_until = NULL, error = NULL "
                        "WHERE id = ? AND owner = ? AND status = 'processing'", (delivery_id, owner)).rowcount == 1


def fail(conn, delivery_id, owner, error, now=time.time):
    """The work failed. Try again later, or set it aside after too many attempts."""
    row = conn.execute("SELECT attempts FROM deliveries WHERE id = ? AND owner = ? AND status = 'processing'", (delivery_id, owner)).fetchone()
    if not row:
        return False
    if row[0] >= MAX_ATTEMPTS:
        conn.execute("UPDATE deliveries SET status = 'failed', error = ?, leased_until = NULL WHERE id = ?", (error, delivery_id))
    else:
        conn.execute("UPDATE deliveries SET status = 'received', error = ?, leased_until = NULL, available_at = ? WHERE id = ?",
                     (error, now() + RETRY_DELAY * 2 ** (row[0] - 1), delivery_id))
    return True


if __name__ == "__main__":
    conn = connect()
    if sys.argv[1:2] == ["status"]:
        for status, count in conn.execute("SELECT status, COUNT(*) FROM deliveries GROUP BY status ORDER BY status"):
            print(f"{status:<11}{count}")
        for delivery_id, attempts, error in conn.execute("SELECT id, attempts, error FROM deliveries WHERE status = 'failed'"):
            print(f"  failed: {delivery_id} after {attempts} attempts: {error}")
    elif sys.argv[1:2] == ["retry"] and len(sys.argv) == 3:  # put a failed delivery back in the queue, once you've fixed the cause
        changed = conn.execute("UPDATE deliveries SET status = 'received', attempts = 0, error = NULL, available_at = ? WHERE id = ? AND status = 'failed'",
                               (time.time(), sys.argv[2])).rowcount
        print("Queued again." if changed else "No failed delivery with that ID.")
    else:
        sys.exit("Usage: python3 inbox.py status | retry <delivery id>")

Here's what the main pieces do.

• store() saves a delivery with INSERT OR IGNORE. The ID is the table's primary key, so a copy is silently refused, and the function reports whether this one was new.
• claim() gives a worker one delivery to handle. The BEGIN IMMEDIATE line takes the database's write lock, so two workers can't both pick the same row. It hands out new deliveries, and also deliveries whose lease has run out, meaning the worker that had them vanished.
• The lease and the owner token. Every claim stamps the delivery with an expiry time and a random token. If a worker is so slow that its lease runs out and someone else takes over, its late finish() or fail() is refused, so it can't overwrite what the new worker did.
• fail() puts a failed delivery back for another try, after a wait that doubles each time: 2 seconds, then 4, then 8. After 5 attempts, the delivery is marked failed and set aside, so one bad delivery can't keep the queue busy forever.
• claim() also gives up on crashes. If a worker keeps dying on the same delivery, the delivery is marked failed after the attempt limit instead of being retried forever.
• The command line. python3 inbox.py status shows how many deliveries are in each state, and lists any failed ones with their errors. python3 inbox.py retry ID puts a failed one back, once you've fixed the cause.

Step 2: The receiver

The receiver does as little as possible. Create inbox_receiver.py:

import logging
import sqlite3
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

from inbox import connect, store

MAX_BODY = 1_000_000


class Receiver(BaseHTTPRequestHandler):
    def reply(self, code, text):
        self.send_response(code)
        self.send_header("Content-Length", str(len(text)))
        self.end_headers()
        self.wfile.write(text.encode())

    def do_POST(self):
        # In real life, check the signature first. The delivery ID comes from the provider,
        # for example GitHub's X-GitHub-Delivery header, or the event "id" inside a Stripe body.
        delivery_id = self.headers.get("X-Delivery-Id")
        length = int(self.headers.get("Content-Length") or 0)
        if not delivery_id:
            return self.reply(400, "missing delivery id")
        if length > MAX_BODY:
            return self.reply(413, "too large")
        body = self.rfile.read(length).decode("utf-8", errors="replace")
        try:
            conn = connect()
            is_new = store(conn, delivery_id, body)
            conn.close()
        except sqlite3.Error as error:
            logging.error("Couldn't store %s: %s", delivery_id, error)
            return self.reply(503, "try again later")  # not saved, so ask the provider to retry
        logging.info("%s %s", "Stored" if is_new else "Ignored a duplicate of", delivery_id)
        self.reply(200, "stored" if is_new else "duplicate")  # either way, the provider is told it's handled

    def log_message(self, *args):
        pass


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(message)s")
    print("Listening on http://127.0.0.1:8000")
    ThreadingHTTPServer(("127.0.0.1", 8000), Receiver).serve_forever()

It stores the delivery and answers right away. A new delivery and a duplicate both get 200, because from the provider's point of view, both are "received". If the database can't save it, the receiver answers 503, so the provider knows to try again. That's the one case where an error is the right answer.

Where does the delivery ID come from? It depends on the provider. GitHub sends one in the X-GitHub-Delivery header, and Stripe puts an event id in the body. The code reads a header called X-Delivery-Id to keep the example simple, so change that line to match your provider. And before you store anything from the internet, check the signature, as I explain in my guide on verifying webhook signatures.

Step 3: The worker

The worker takes deliveries from the inbox and does the actual work. Create worker.py:

import json
import logging
import sys
import time

from inbox import claim, connect, fail, finish

log = logging.getLogger("worker")


def record_payment(conn, delivery_id, event):
    """The example work: record a payment. The delivery ID is the table's key, so doing it twice changes nothing."""
    conn.execute("CREATE TABLE IF NOT EXISTS payments (delivery_id TEXT PRIMARY KEY, customer TEXT, amount REAL)")
    conn.execute("INSERT OR IGNORE INTO payments VALUES (?, ?, ?)", (delivery_id, event["customer"], event["amount"]))


def run(conn, handler, stop_when_empty=False):
    while True:
        job = claim(conn)
        if not job:
            if stop_when_empty:
                return
            time.sleep(1)
            continue
        delivery_id, body, owner = job
        try:
            handler(conn, delivery_id, json.loads(body))
        except Exception as error:
            log.warning("%s failed: %s", delivery_id, error)
            fail(conn, delivery_id, owner, str(error))
        else:
            if finish(conn, delivery_id, owner):
                log.info("%s done", delivery_id)
            else:
                log.warning("%s: the lease was lost while working, so someone else took over", delivery_id)


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(message)s")
    run(connect(), record_payment, stop_when_empty="--once" in sys.argv)

The example work is record_payment(). Look at the table: the delivery ID is its primary key, and the insert is INSERT OR IGNORE. If a crash makes the worker repeat the job, the second attempt changes nothing. That's the fourth idea in the design, in practice.

The handler gets the database connection, the delivery ID and the event. Replace record_payment with your own work, and keep the habit of using the delivery ID to make the work safe to repeat.

Step 4: Try it

Start the receiver in one terminal:

python3 inbox_receiver.py

In another, send some deliveries. This script sends the same delivery three times, as a retrying provider would, plus one other and one broken one. Save it as send_deliveries.py:

import json
import urllib.request


def send(delivery_id, event):
    request = urllib.request.Request("http://127.0.0.1:8000", data=json.dumps(event).encode(), headers={"X-Delivery-Id": delivery_id})
    with urllib.request.urlopen(request) as response:
        print(f"{delivery_id}: {response.read().decode()}")


send("evt_100", {"customer": "Maria Lopez", "amount": 24.99})
send("evt_100", {"customer": "Maria Lopez", "amount": 24.99})  # the provider retried, so the same delivery arrives again
send("evt_101", {"customer": "Sam Okafor", "amount": 12.50})
send("evt_100", {"customer": "Maria Lopez", "amount": 24.99})  # and again
send("evt_102", {"customer": "Eve"})  # a broken one: the amount is missing

Run it with python3 send_deliveries.py:

evt_100: stored
evt_100: duplicate
evt_101: stored
evt_100: duplicate
evt_102: stored

Only the first copy of evt_100 was stored. The receiver's log says the same:

Stored evt_100
Ignored a duplicate of evt_100
Stored evt_101
Ignored a duplicate of evt_100
Stored evt_102

Check what's waiting with python3 inbox.py status:

received   3

Now run the worker once. The --once flag makes it stop when there's nothing left:

python3 worker.py --once

evt_100 done
evt_101 done
evt_102 failed: 'amount'

The first two succeeded. The broken one failed because its amount is missing, and it will be tried again. Check again:

done       2
received   1

Two are done, and one is waiting for its retry. After 5 failed attempts it moves to failed, and status lists it with its error, so you can fix the cause and run retry.

For real use, run python3 worker.py without --once, so it keeps checking for new work, and keep it running with your system's service manager.

What I tested

A design like this is only worth trusting if it survives the awkward cases. These are the ones I tried.

What happens	Result
20 copies of one delivery arrive at the same instant	One row is stored, and every sender gets 200
4 worker processes share 60 deliveries	Each is handled exactly once
A worker is killed halfway through a job	The delivery is stuck for a while, then another worker takes it over
That job's effect uses the delivery ID	The effect happens once
That job's effect doesn't use the delivery ID	The effect happens twice
A worker is so slow that its lease runs out	Its late "done" is refused, and the new worker's result stands
The work keeps failing	Retries with waits of 2, 4, then 8 seconds, then it's set aside
A worker keeps crashing on one delivery	It's set aside after the attempt limit
One broken delivery sits among good ones	The good ones are processed normally
The database can't be opened	The provider gets 503, not 200

The fifth and sixth rows are the important lesson. Even with a perfect inbox, the effect has to be safe to repeat. The inbox makes sure each delivery is taken once at a time. It can't tell whether the previous worker finished the job before it vanished.

"Exactly once" doesn't exist, but "effectively once" does

You can't truly guarantee that something happens exactly once, because a crash can always fall between doing the work and writing it down. What you can build is at least once delivery plus a safe effect, and the result behaves like exactly once.

How you make the effect safe depends on what it is.

• Writing to your own database. Use the delivery ID as a unique key, as record_payment() does. Better still, write the result and mark the delivery done in the same transaction.
• Calling another service's API. Many APIs accept an idempotency key, a unique label that makes them ignore repeats. Pass the delivery ID.
• Sending an email or message. This is the hard one, because you can't take it back. Small gaps remain: a crash between sending and marking done will send it twice. Keep these messages harmless to repeat, or let a person approve them.
• Anything risky. For actions that are expensive or hard to undo, don't let a script act alone. Deciding which tasks need a person is covered in when not to automate.

What to send back to the provider

Situation	Answer	Why
New delivery, saved	200	You have it, and the provider can stop
A copy you already have	200	It's handled, so don't make the provider retry
Bad request, such as a missing ID	400	Retrying won't help
Bad signature	401	It isn't from the provider
You couldn't save it	503	You don't have it, so ask for a retry

The inbox pattern is what makes this simple. You answer once the delivery is saved, not once the work is done, so slow work never causes a timeout and a retry.

Running it for real

Watch the inbox. Run python3 inbox.py status now and then, or have a check that raises an alert when something is failed, or when the oldest received delivery has been waiting too long. A stuck inbox is silent until someone notices a missing payment. I cover how to set up alerts like that in how to monitor your automations when something breaks.

Keep the worker running. If it stops, deliveries pile up safely, but nothing gets processed.

Don't delete IDs too soon. You can remove old done rows to keep the table small, but a duplicate that arrives after you've deleted its ID looks new. Keep IDs for longer than your provider keeps retrying, which can be hours or days. Check their documentation.

One machine or many. SQLite is a good fit for one server. If several servers share the work, use a database they all connect to. Postgres, for example, supports unique keys and a feature for claiming rows without blocking other workers.

What can go wrong

The ID isn't stable. If you use something that changes between retries, like a timestamp, duplicates look new. Use the provider's own delivery or event ID.

Two events about the same thing arrive out of order. Webhooks don't always arrive in the order they happened. Check the event's own timestamp or version before overwriting newer data.

The work isn't safe to repeat. The inbox protects the intake. The effect still needs to be idempotent, or it needs a person.

The lease is too short. If the work can take longer than the lease, a second worker may start the same job. Set the lease longer than your slowest job, and keep jobs short.

Everything runs in the receiver. If you do the slow work before answering, the provider times out and retries, which is how duplicates start in the first place.

Private data in the inbox. The table holds the whole body of each delivery. Protect the file, and prune it.

Quick answers

What does idempotent mean?
Doing it twice has the same result as doing it once.

Why do providers send duplicates on purpose?
Because losing an event is worse than repeating one. They keep trying until they hear back.

Why not just check if the ID exists before processing?
Because two copies can arrive at the same moment, and both will see "not yet". A unique key in the database settles the race.

Is an in-memory set ever enough?
For a throwaway script on one process where a repeat does no harm, maybe. Not for anything that matters.

What if the provider doesn't send an ID?
Build one from the content, such as a hash of the body, though two identical real events would then look the same. Ask the provider about a delivery ID first.

Does this work with GitHub Actions?
The inbox needs somewhere to listen and a database file that stays around. A small server does that, and a GitHub Actions run doesn't.

Can I process in order?
This inbox takes the oldest waiting delivery first, but it doesn't guarantee order when there are several workers, or after retries. If order matters, use a single worker or check timestamps in your handler.

The short version

Treat duplicates as normal. Save each delivery under a unique ID before you do anything else, answer the provider right away, and do the work separately with a lease that expires if the worker disappears. Then make sure the work is safe to repeat.

You can find more practical automation guides on Procwire.