Use a transactional outbox, a stable idempotency key, FIFO admission, and a five-minute deduplication window for rate-limited webhook jobs; treat this as a practical exactly-once alternative, not a promise of exactly-once network delivery.
The deciding constraint is the e-commerce order path. A checkout should not wait while a merchant endpoint is slow, yet two workers must not send the same order.paid event concurrently. The design below moves the delivery intent into the order transaction, paces attempts after commit, and makes the receiver's idempotency record the final defense against duplicates.
Short answer: persist intent before publishing, keep one logical key across every retry, and let duplicate work become a cheap lookup rather than an extra shipment notification.
This is an experiment note, so the evaluation target comes first: preserve accepted-order latency while minimizing queue and storage cost, then verify duplicate effects, ordering, retry timing, and backlog age. The tempting implementation is a timer started after checkout commits. It's short. It also leaves a gap between the business commit and enqueue, while a retrying process can create two independently paced jobs. A durable outbox closes that gap; it does not repeal distributed-systems physics.
Why exactly-once delivery is the wrong target
An outbound request crosses at least three durability boundaries: the shop database, the delivery worker, and the merchant's database. The worker can lose its response after the merchant commits. From the worker's point of view, “committed but response lost” and “never committed” look identical, so it must retry or abandon the event. No FIFO queue can infer the remote commit from silence.
Aim for an exactly-once effect instead. The sender provides at-least-once attempts, and the receiver atomically records an idempotency key with the business change. A repeated request returns the stored outcome without applying the change again. This division is less dramatic than an exactly-once label — and much easier to test.
The five-minute window solves a narrower problem. It coalesces accidental duplicate admissions, such as two application paths trying to enqueue the same event close together. It must not define business identity. A retry six minutes later is still the same logical delivery and therefore carries the same permanent key. Otherwise, expiration quietly converts a delayed retry into a new side effect.
For an order event, a useful key has a stable aggregate identifier and event version:
def idempotency_key(order_id: str, event_name: str, version: int) -> str:
return f"order:{order_id}:{event_name}:v{version}"
assert idempotency_key("ord_1042", "paid", 3) == "order:ord_1042:paid:v3"
Don't derive this value from retry count, a timestamp, or a random job identifier. Those describe an attempt, not the effect.
How should a FIFO queue deduplicate five-minute rate-limited jobs with an idempotency key?
Start at the database transaction. Insert the order change and one outbox row together, with a unique constraint on the logical key. A relay later exposes committed rows to workers. This is the transactional outbox pattern: the message exists if and only if the local business transaction commits, although the relay may publish it more than once. That last clause is why consumer idempotency remains mandatory.
FIFO should mean “oldest eligible job first,” not “one slow merchant blocks every merchant.” Partition by destination or tenant, maintain order within that partition, and place a rate limiter at dispatch. If merchant A has exhausted its allowance, merchant B's eligible work should continue. Strict global order is usually an expensive property with no e-commerce meaning.
Here is a compact reference implementation. It uses SQLite so the state transitions are visible and the example runs without a broker or SDK. Production deployments can replace the storage adapter, but should preserve the constraints and transaction boundaries.
from __future__ import annotations
import sqlite3
import time
from dataclasses import dataclass
WINDOW_SECONDS = 5 * 60
@dataclass(frozen=True)
class Delivery:
row_id: int
destination: str
event_key: str
payload: str
def open_db(path: str = ":memory:") -> sqlite3.Connection:
db = sqlite3.connect(path)
db.execute("PRAGMA foreign_keys = ON")
db.executescript(
"""
CREATE TABLE IF NOT EXISTS outbox (
id INTEGER PRIMARY KEY,
destination TEXT NOT NULL,
event_key TEXT NOT NULL UNIQUE,
payload TEXT NOT NULL,
available_at REAL NOT NULL,
created_at REAL NOT NULL,
delivered_at REAL
);
CREATE TABLE IF NOT EXISTS recent_admission (
event_key TEXT PRIMARY KEY,
expires_at REAL NOT NULL
);
"""
)
return db
def admit(
db: sqlite3.Connection,
destination: str,
event_key: str,
payload: str,
now: float,
) -> bool:
"""Return True only when this logical event creates a new outbox row."""
with db:
db.execute("DELETE FROM recent_admission WHERE expires_at <= ?", (now,))
fresh = db.execute(
"""
INSERT INTO recent_admission(event_key, expires_at)
VALUES (?, ?)
ON CONFLICT(event_key) DO NOTHING
""",
(event_key, now + WINDOW_SECONDS),
).rowcount
if not fresh:
return False
inserted = db.execute(
"""
INSERT INTO outbox(destination, event_key, payload, available_at, created_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(event_key) DO NOTHING
""",
(destination, event_key, payload, now, now),
).rowcount
return bool(inserted)
def next_eligible(
db: sqlite3.Connection, destination: str, now: float
) -> Delivery | None:
row = db.execute(
"""
SELECT id, destination, event_key, payload
FROM outbox
WHERE destination = ? AND delivered_at IS NULL AND available_at <= ?
ORDER BY created_at, id
LIMIT 1
""",
(destination, now),
).fetchone()
return Delivery(*row) if row else None
def defer(db: sqlite3.Connection, row_id: int, eligible_at: float) -> None:
with db:
db.execute(
"UPDATE outbox SET available_at = ? WHERE id = ? AND delivered_at IS NULL",
(eligible_at, row_id),
)
def mark_delivered(db: sqlite3.Connection, row_id: int, now: float) -> None:
with db:
db.execute(
"UPDATE outbox SET delivered_at = ? WHERE id = ?",
(now, row_id),
)
if __name__ == "__main__":
db = open_db()
now = time.time()
key = idempotency_key("ord_1042", "paid", 3)
assert admit(db, "merchant_7", key, '{"order_id":"ord_1042"}', now)
assert not admit(db, "merchant_7", key, '{"order_id":"ord_1042"}', now + 1)
assert next_eligible(db, "merchant_7", now).event_key == key
There are two guards on purpose. recent_admission is an inexpensive, expiring optimization; the unique outbox.event_key constraint is the durable correctness rule. Even after the recent entry expires, a second admission cannot invent a second event. Cleanup of delivered outbox rows needs a retention policy, but should happen only after the receiver's replay horizon and audit needs are understood.
The sample does not hold a database transaction open during an HTTP call. A worker claims or reads an eligible row, asks the limiter for permission, sends the request with the stable key in an agreed header, and then marks success or moves available_at. In a multi-worker implementation, claiming must be atomic through the database's locking primitive. The exact SQL varies by database, so pretending one SQLite statement is portable would make the example less honest.
Pace retries without turning the queue into a timer heap
Rate and concurrency are different controls. A token bucket bounds starts per interval; a semaphore bounds requests currently in flight. Keep both per destination when merchants have separate limits. On HTTP 429, honor a valid Retry-After value and reschedule the row. For transport ambiguity, retry with exponential backoff and jitter while retaining the same logical key. Permanent authentication or payload rejection belongs in a terminal state for operator review, not an endless retry loop.
Small detail, big consequence.
The retry time belongs in durable queue state, not in sleep() inside a worker. Sleeping consumes a worker slot, disappears on restart, and makes queue age harder to explain. This pure function can calculate a bounded delay; the caller persists its result with defer.
import random
def retry_delay_seconds(
attempt: int,
retry_after: float | None = None,
rng: random.Random | None = None,
) -> float:
if retry_after is not None and retry_after >= 0:
return min(retry_after, 300.0)
source = rng or random.Random()
ceiling = min(2.0 ** attempt, 300.0)
return source.uniform(0.0, ceiling)
assert retry_delay_seconds(4, retry_after=12.0) == 12.0
seeded = random.Random(7)
assert 0.0 <= retry_delay_seconds(4, rng=seeded) <= 16.0
The 300-second cap is an example policy aligned with this experiment's window, not a universal webhook standard. Your endpoint contract, retry budget, and merchant tolerance decide the real cap. I'm not sure a fixed five-minute maximum is right for a flash-sale backlog; queue-age data and the receiver's published limits would resolve that choice.
Latency versus cost appears here in a useful form. More workers can reduce queue age but cannot raise a merchant's accepted request rate, and idle workers still cost money. A tighter polling interval lowers pickup latency while increasing database reads. Begin with the service-level objective for delivery, then choose the cheapest worker count and poll cadence that hold that objective under the measured arrival burst. Don't optimize from an empty queue benchmark.
What should the eval harness prove before deployment?
I treat the delivery policy like an AI evaluation harness: freeze representative inputs, state the invariants, inject failures, and compare outcomes after each implementation change. Notebook-to-prod is a useful discipline here because a tiny deterministic simulation reveals logic errors before a broker and several dashboards hide them.
Use an event set with repeated order IDs, separate destinations, and attempts on both sides of the five-minute boundary. Then assert that duplicate admission yields one durable row, a retry keeps the original key, FIFO order holds among eligible jobs for the same destination, and a limited destination does not stall another partition. Include the ambiguous case where the receiver commits but the sender observes no response. The next attempt must produce one receiver-side effect because the receiver stores the key and result atomically.
An HTTP 429 at attempt 4 is a valuable fixture — it checks that the job becomes eligible later without losing its place or occupying a worker. Also crash the relay after it reads an outbox row but before it records publication. Duplicate publication is expected; duplicate business impact is not.
Measure p50 and p95 enqueue-to-delivery latency, oldest eligible job age, attempts per successful effect, deduplication hits, terminal failures, and limiter utilization per destination. Record database operations per delivery as the cost proxy before reaching for more infrastructure. For payloads that contain model-generated recommendations or summaries, carry prompt version and token usage alongside the event metadata, but keep them out of the idempotency identity unless a changed prompt is meant to create a new business event.
The catch is operational ownership. A database outbox is not suitable when the existing broker already provides a well-operated delayed-delivery path and the application database cannot absorb relay scans. Stick with that broker when the team has partitioning, dead-letter handling, and replay procedures in place. A dead-letter exchange can route rejected or expired messages for inspection, but it does not make the receiver idempotent. At the other extreme, a low-volume shop may reasonably use one database worker and skip a separate queue; the durable constraints matter more than the number of components.
Before copying this design, run the harness at the actual merchant rate limits and expected burst shape. The decision is ready when the team can explain the maximum queue age, retention horizon, retry budget, and receiver behavior for a repeated key. If any answer is “the queue handles it,” the experiment is not finished.
Top comments (0)