Your FastAPI Webhook Returned 200. Why Did the Order Run Twice?

#webdev #programming #fastapi #python

Your webhook endpoint returned 200 OK.

The provider still delivered the event again.

That is not necessarily a provider bug. Most webhook systems promise at-least-once delivery, not exactly-once delivery. A timeout, a lost response, a process restart, or a manual replay can send the same logical event more than once.

The dangerous implementation looks harmless:

@app.post("/webhooks/provider")
async def receive(event: dict):
    create_order(event)
    return {"ok": True}

If the same event arrives twice, create_order() runs twice. Replace "order" with invoice, email, provisioning job, CRM record, or AI-agent action and the failure becomes expensive quickly.

Here is the small pattern I use to make the HTTP boundary safer.

1. Verify the signature against the raw body

Do not parse and re-serialize the payload before signature verification. Whitespace, key order, and encoding can change the byte sequence.

import hashlib
import hmac

def valid_signature(body: bytes, supplied: str, secret: bytes) -> bool:
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, supplied)

Use hmac.compare_digest() rather than a normal string comparison.

2. Treat the provider event ID as a database key

An in-memory set is not enough. It disappears on restart and does not coordinate multiple workers.

Create a durable inbox table with a unique event ID:

CREATE TABLE webhook_inbox (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  payload TEXT NOT NULL,
  state TEXT NOT NULL DEFAULT 'accepted',
  attempts INTEGER NOT NULL DEFAULT 0,
  last_error TEXT,
  accepted_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The uniqueness constraint is the important part. Two concurrent requests can both pass a separate "does this ID exist?" query. Let the database perform the reservation atomically.

For a minimal SQLite example:

cursor = db.execute(
    """
    INSERT OR IGNORE INTO webhook_inbox(event_id, event_type, payload)
    VALUES (?, ?, ?)
    """,
    (event_id, event_type, body.decode()),
)
db.commit()

duplicate = cursor.rowcount == 0

In PostgreSQL, use INSERT ... ON CONFLICT DO NOTHING.

3. Acknowledge durable acceptance, not completed business work

Keep the HTTP path short:

read the raw body;
verify the signature;
validate the minimum envelope;
persist the inbox record;
return 202 Accepted.

Do not call three downstream APIs before responding. Slow processing increases the chance that the provider times out and retries.

The endpoint can look like this:

from fastapi import FastAPI, Header, HTTPException, Request, status

app = FastAPI()

@app.post("/webhooks/provider", status_code=status.HTTP_202_ACCEPTED)
async def receive_webhook(
    request: Request,
    x_signature: str = Header(default=""),
):
    body = await request.body()
    if not valid_signature(body, x_signature, secret):
        raise HTTPException(status_code=401, detail="Invalid signature")

    event = json.loads(body)
    cursor = db.execute(
        "INSERT OR IGNORE INTO webhook_inbox(event_id,event_type,payload) VALUES(?,?,?)",
        (str(event["id"]), str(event["type"]), body.decode()),
    )
    db.commit()

    return {
        "accepted": True,
        "duplicate": cursor.rowcount == 0,
    }

The worker processes accepted rows separately. A crash after the database commit does not lose the event.

4. Test the duplicate directly

Do not wait for production retries to discover whether idempotency works.

def test_duplicate_delivery_has_one_durable_acceptance():
    event = {"id": "evt_duplicate_1", "type": "order.paid"}

    first = signed_post(event)
    second = signed_post(event)

    assert first.status_code == 202
    assert first.json()["duplicate"] is False
    assert second.status_code == 202
    assert second.json()["duplicate"] is True

Also test:

an invalid signature;
two concurrent deliveries of the same ID;
a worker crash after inbox persistence;
a downstream timeout;
events delivered in reverse order;
a stale event trying to overwrite newer state.

5. Duplicate delivery is not the only problem

Events can arrive out of order. A delayed subscription.updated event may arrive after subscription.cancelled.

Store the provider timestamp or sequence number and define allowed state transitions. An older event should become a no-op rather than silently restoring stale state.

Retries also need a terminal state. Record attempt count, last error, and the next retry time. After the configured limit, move the event to a dead-letter state that can be inspected and replayed intentionally.