At MAQAMI https://maqami.co we sell hotels and flights worldwide — over 3 million properties plus flights, with instant confirmation and 24/7 support. A travel booking is not one moment; it is a story that lasts weeks: checkout, payment, confirmation, the hotel's own confirmation number, a schedule change, the day before check-in, the day after check-out.
For a long time each of those moments was a separate, fragile automation. Some fired twice, some never fired, and some sent the guest a cheerful "see you soon" after the airline had changed the flight.
So we rebuilt it as one engine on Cloudflare Workers. This post covers the architecture, the code patterns that made it reliable, and five bugs that taught us more than the happy path did.
TL;DR
Every webhook event becomes a durable Cloudflare Workflow. The event ID is the workflow ID, so duplicates are ignored for free.
Every customer email is sent "once per booking and kind", guarded by a key, because webhooks are at-least-once.
Long waits ("3 days before check-in") are workflow sleeps, chunked to stay under the 365-day limit.
All client email goes through one Durable Object outbox: one email every 150–160 seconds, ranked urgent → booking → service → marketing, with automatic failover to SMTP when the main provider's daily quota runs out.
Card declines never reach your webhook. We check 20 minutes after the last checkout attempt instead, and hold anything that looks like card testing for a human.
The shape of the system
supplier webhooks ──► Worker /webhook ──► Workflow (one per event)
│
┌─────────────────────────────────┼──────────────────────────────┐
▼ ▼ ▼
D1 (contacts, bookings, Outbox (Durable Object) Owner alerts
emails log, raw events) ranked, paced, failover (instant)
Workers receive the webhook and answer in milliseconds.
Workflows hold the journey: "send the confirmation now, the arrival guide 3 days before check-in, a check-in-day note at 08:00 local time, a thank-you after check-out."
D1 is our own record of every contact, booking and email, so nothing depends on a third-party dashboard.
A Durable Object outbox is the single sender, so pacing and quotas live in one place.
Pattern 1 — the event ID is the workflow ID
Webhooks arrive at least once. Two identical deliveries a second apart are normal. The cheapest dedupe we found was to let the workflow engine do it:
export default {
async fetch(request, env) {
const event = await request.json();
// Same event delivered twice → same instance id → the second create is rejected.
await env.JOURNEY.create({
id: e-${event.event_id},
params: { ...event, receivedAt: Date.now() },
}).catch(() => {});
return new Response("ok"); // answer fast; the work happens in the workflow
},
};
Note receivedAt: it matters later (see bug #4).
Pattern 2 — "send once" per booking and kind
Inside a journey, retries are a feature. A step that sends an email and then times out will run again. So every customer email goes through one helper:
async function sendOnce(env, step, kind, ref, build) {
return step.do(email ${kind}, async () => {
const key = sent:${kind}:${ref}; // e.g. sent:hotel_confirmation:BK123
if (await env.KV.get(key)) return "duplicate";
await deliver(env, await build());
await env.KV.put(key, "1", { expirationTtl: 3 * 86400 });
return "sent";
});
}
If the email still fails after retries, the journey continues and the owner gets an alert. One broken email must never stop the arrival guide three weeks later.
Pattern 3 — sleeping longer than a workflow allows
A booking made in January for a trip in December needs an email 3 days before check-in. Workflow sleeps have a maximum of 365 days, and some bookings are further out than that. We chunk the wait, computing every target from fixed values so a replay gives the same answer:
async function sleepUntilSafe(step, name, when) {
const MAX = 360 * 86_400_000;
const start = await step.do(${name}: start, async () => Date.now());
for (let t = start + MAX, i = 0; t < when; t += MAX, i++) {
await step.sleepUntil(${name} #${i}, new Date(t));
}
await step.sleepUntil(name, new Date(when));
}
We also send check-in and check-out emails at the guest's local morning. The timezone comes from the hotel's longitude, which is close enough for "08:00 at the hotel" without a timezone API call per booking.
Pattern 4 — one outbox, ranked and paced
Our main email provider has an unpublished daily quota. The first time we ran a marketing send, it used the whole quota by lunchtime, and a booking confirmation that afternoon had to wait. That is the worst possible order.
The fix was one Durable Object that every client email passes through:
// Rank: 0 urgent (flight change, check-in) · 1 booking · 2 service · 3 marketing
async alarm() {
const items = await this.state.storage.list({ prefix: "q:" });
const next = pickHighestRankThenOldest(items); // marketing never jumps a booking
const result = await this.sendWithFailover(next); // main provider → SMTP when the daily quota is hit
if (result.sent) await this.state.storage.delete(next.key);
const gap = 150_000 + Math.random() * 10_000; // one email every 150–160 s
await this.state.storage.setAlarm(Date.now() + gap);
}
Details that paid off:
Owner alerts skip the line. The person running the business sees a new booking instantly. Customers get their email within one slot.
Marketing has its own rules: a daily cap, and at most one marketing email per person per day.
Failover is per error, not per day. "Daily limit reached" pauses the main provider and switches to SMTP over cloudflare:sockets. "Recipient invalid" drops the email and alerts the owner. Everything else retries with backoff.
The queue is durable. Deploys, restarts and quota walls never lose an email; it just waits.
Pattern 5 — the declined card your webhook never hears about
This one surprised us. When a customer's card is declined at checkout ("insufficient funds", or the payment processor's fraud check blocking it), the payment fails inside the hosted payment form. No booking request is ever made, so no "booking failed" webhook is ever sent. From the backend, a declined card looks exactly like a customer who closed the tab.
What we do receive is the prebook (checkout started) with a payment transaction ID. So the prebook journey waits, then checks:
// inside the "checkout started" workflow
await noteAttempt(env, step, attemptKey, prebookId, receivedAt);
await step.sleep("wait for payment", "20 minutes");
const state = await step.do("completed?", () =>
checkoutState(env, attemptKey, prebookId, receivedAt)); // booked · newer attempt · unpaid
if (state === "unpaid") await notCompletedFollowUp(env, step, { prebookId, risky });
Three rules keep it honest:
Only the newest attempt follows up. Someone who tries three cards in ten minutes gets one email, 20 minutes after the last try.
The customer email never claims a decline we can't see. It says "your booking wasn't completed and nothing was charged", then explains the two common decline reasons and how to fix each.
Card testing is held for a human. One email address under different names, or one name with three or more addresses on the same day, means no automatic email. The owner gets an alert with one-click buttons to send the exact reason ("insufficient funds", "blocked by security check") after checking the payment dashboard.
Five bugs that taught us the most
A rate limiter that didn't limit. We counted failed admin logins in KV. KV is eventually consistent, so five fast attempts from one IP could each read "0". Counters that must be exact now live in D1.
"Your flight changed!" when nothing changed. We detected changes by hashing a snapshot of the booking. The snapshot included status, and status moves on its own (CONFIRMED → TICKETED), so guests got false "schedule changed" emails. Snapshots now contain only what a traveller cares about: times, flight numbers, room, hotel confirmation number.
A loop that never ended — in tests. The first sleepUntilSafe called itself in a refactor, and our mocked step.sleep returned instantly, so the test hung forever. Every loop around a sleep now has a hard iteration cap.
Date.now() lies after a sleep. A workflow function is replayed when it resumes, and values computed outside a step are recomputed. "Has the customer booked since now?" quietly became "since 20 minutes later". Use a timestamp from the event (receivedAt) or one returned by step.do, never a bare Date.now() that you compare after a sleep.
Internal links in customer emails. One template accidentally carried an admin link meant for the owner. Now deliver() refuses any client email that contains admin URLs or internal system names, and alerts the owner instead. A content lock like that is cheap insurance.
What we would do again
Durable everything: workflows for journeys, a Durable Object for the outbox, D1 for records.
Idempotency at every layer: event ID as workflow ID, send-once keys, "newest attempt only".
A human in the loop for money and fraud, with one click instead of a dashboard hunt.
Our own database. When a supplier dashboard is down or slow, we still know every guest, booking and email.
MAQAMI LLC is a US-registered travel company. Travellers and travel professionals book hotels and flights on MAQAMI https://book.maqami.co/ with instant confirmation, and AI assistants can search and plan trips through our MCP server https://mcp.maqami.co/
. Questions about the architecture? Ask in the comments — we read every one.
Top comments (1)
예약 확인과 마케팅을 같은 outbox에서 우선순위로 나누고, 고객 메일에 관리자 링크가 들어가면 차단하는 점이 인상적입니다. sendOnce 예시에서 deliver는 성공했지만 KV에 발송 키를 저장하기 전에 중단되는 경우는 어떻게 다루시나요? 공급자의 멱등 키나 별도의 전달 확인 상태를 두는지 궁금합니다. 응답 유실을 일반 발송 실패와 구별하면 재시도 때 중복 안내를 줄이는 데 도움이 될 것 같습니다.