Why Your Booking System Should Assume Every Cancellation Half-Fails

A booking is one direction. Money in, inventory out, everyone's happy. You can write that as a transaction and mostly get away with it.

A cancellation is five things happening across four systems, and at least one of them is going to fail while the others succeed. Not occasionally. Regularly enough that "partial failure" should be your default assumption, not your edge case.

Here's the shape of a hotel cancellation:

Mark the booking cancelled in your database
Release inventory back to sellable
Push availability to connected distribution channels
Issue a refund through the payment provider
Adjust revenue recognition

Steps 1 and 2 are local. Steps 3, 4 and 5 are network calls to systems you don't control, with different latencies, different failure modes, and — critically — no shared transaction.

You cannot make these atomic. Two-phase commit across a payment gateway and a channel manager isn't a thing. So the question isn't how to prevent partial failure. It's what your system does when it happens.

The failure that costs the most

Everyone assumes the refund is the scary one. It isn't, because refund failures are loud — the customer tells you.

The expensive failure is silent: inventory releases locally, the channel push fails, and nobody notices. Your database says the room is sellable. Your OTA still shows it booked. That room sits unsellable through the highest-demand window before check-in, and generates zero signal. No error page, no support ticket, no alert.

You find out at month-end when occupancy doesn't match expectations, if you find out at all.

So the design goal isn't "never fail." It's never fail silently, and always converge.

Step one: make everything idempotent

Before any retry logic, every operation needs to be safely repeatable. If retrying a refund can issue two refunds, you can't retry anything, and if you can't retry you have no recovery story.

python
def issue_refund(booking_id: str, amount_cents: int, reason: str):
# Deterministic key — same cancellation always produces the same key
idempotency_key = f"refund:{booking_id}:{amount_cents}:{reason}"

return payment_client.refunds.create(
    booking_id=booking_id,
    amount=amount_cents,
    idempotency_key=idempotency_key,
)

The key must be derived from the operation, not generated fresh. uuid4() here would defeat the entire purpose — every retry becomes a new refund.

Note the amount_cents in the key. That's deliberate. A partial cancellation followed by a full cancellation are different operations and should not collide.

For your own state transitions, guard at the database level rather than in application code:

sql
UPDATE bookings
SET status = 'cancelled',
cancelled_at = NOW()
WHERE id = $1
AND status = 'confirmed'
RETURNING id;

If this returns zero rows, someone already cancelled it. That's not an error — it's the idempotent path. Handle it as success.

Doing this check with a SELECT followed by an UPDATE is a race condition. Let the database do it in one statement.

Step two: stop making network calls inside your transaction

This is the pattern I see most often, and it's broken in a way that's hard to see:

python

Don't do this

with db.transaction():
booking.status = "cancelled"
booking.save()
release_inventory(booking)
channel_manager.push_availability(booking.property_id, booking.dates) # network
payment_client.refund(booking.id, amount) # network

Two problems.

If push_availability throws, the transaction rolls back — but the refund may have already gone through, or the channel manager may have processed the push and failed on the response. You've now got a booking marked confirmed in your database and a refund issued in the payment system. That's worse than either failure alone.

And if the process dies between the two network calls, you've lost the intent entirely. Nothing records that a refund was supposed to happen.

The fix is the transactional outbox. Commit your local state change and a durable record of what still needs to happen, in the same transaction. Do the network calls afterward.

python
def cancel_booking(booking_id: str, cancelled_nights: list[date]):
with db.transaction():
booking = db.query(
"""
UPDATE bookings SET status = 'cancelled', cancelled_at = NOW()
WHERE id = $1 AND status = 'confirmed'
RETURNING *
""",
booking_id,
)
if not booking:
return AlreadyCancelled()

    release_inventory_local(booking.property_id, cancelled_nights)

    refund_cents = calculate_refund(booking, cancelled_nights)

    # Same transaction — these commit or roll back together
    db.insert_many("outbox", [
        {"booking_id": booking_id, "task": "push_channel_availability",
         "payload": {"property_id": booking.property_id,
                     "dates": cancelled_nights},
         "status": "pending"},
        {"booking_id": booking_id, "task": "issue_refund",
         "payload": {"amount_cents": refund_cents},
         "status": "pending"},
        {"booking_id": booking_id, "task": "adjust_revenue",
         "payload": {"amount_cents": refund_cents},
         "status": "pending"},
    ])

# Transaction committed. Nothing can be lost now.
outbox_worker.wake()

The guarantee this buys you: once the transaction commits, the intent is durable. Your process can die immediately after and a worker will pick the tasks up. No partial failure can leave the system with a cancelled booking and no record that a refund was owed.

Step three: retry with a backoff and a ceiling

The worker is straightforward. What matters is what it does when retries run out.

python
HANDLERS = {
"push_channel_availability": push_channel_availability,
"issue_refund": issue_refund,
"adjust_revenue": adjust_revenue,
}

MAX_ATTEMPTS = 6

def process_task(task):
try:
HANDLERStask.task
mark_complete(task.id)

except TransientError as e:
    if task.attempts + 1 >= MAX_ATTEMPTS:
        escalate(task, reason=str(e))
    else:
        # 2s, 4s, 8s, 16s, 32s with jitter
        delay = (2 ** (task.attempts + 1)) + random.uniform(0, 1)
        reschedule(task.id, delay_seconds=delay)

except PermanentError as e:
    # Malformed request, rejected refund, invalid property — retrying won't help
    escalate(task, reason=str(e))

Two things worth being deliberate about.

Jitter matters. Without it, a channel manager outage produces a thundering herd the moment it recovers — every queued task retrying in lockstep. The random component spreads them out.

Separate transient from permanent. A 503 is worth retrying. A 400 saying the refund amount exceeds the original charge is not. Retrying permanent failures burns your attempt budget and delays the escalation that would have actually fixed it.

And escalate should mean a human sees it. A row in a failed_tasks table that nobody queries is the silent failure you were trying to avoid, relocated.

python
def escalate(task, reason):
mark_failed(task.id, reason)
alerts.page(
severity="high" if task.task == "issue_refund" else "medium",
title=f"Cancellation task failed: {task.task}",
booking_id=task.booking_id,
reason=reason,
)
Step four: reconcile, because the outbox isn't enough

The outbox handles failures you can observe. It doesn't handle the ones you can't.

The channel manager accepts your push, returns 200, and drops it internally. Your task is marked complete. Your database and theirs now disagree, permanently, and nothing in your retry logic will ever discover it.

The only defense is periodically comparing state rather than trusting your own event log.

python
def reconcile_property(property_id: str, window_days: int = 90):
ours = local_availability(property_id, window_days)
theirs = channel_manager.fetch_availability(property_id, window_days)

for date, our_count in ours.items():
    their_count = theirs.get(date)

    if their_count is None:
        log.warning("missing_date", property_id=property_id, date=date)
        continue

    if our_count != their_count:
        metrics.increment("availability.drift",
                          tags={"property": property_id})
        log.error("availability_drift", property_id=property_id,
                  date=date, ours=our_count, theirs=their_count)
        enqueue_repush(property_id, date, our_count)

Run it nightly at minimum. Hourly for the next 14 days of inventory, which is where drift costs the most.

The important part isn't the repush — it's the metrics.increment. Drift count is a health signal. If it's normally 2 and today it's 340, something broke upstream and you want to know before your revenue report tells you.

What this actually gets you

The system still fails. That's not the goal.

What changes is that every failure is either automatically corrected, or visible to someone who can correct it. There's no third state where a room sits unsellable for six days and nobody knows.

Three properties worth holding onto:

Durable intent. Once the cancellation commits, every downstream obligation is recorded. Process crashes don't lose work.
Bounded blast radius. A channel manager outage delays availability pushes. It doesn't block refunds or corrupt booking state.
Convergence. Reconciliation catches what the event path missed, including failures that reported success.

The mental shift is small but load-bearing: stop treating cancellation as a transaction and start treating it as a set of independent obligations with different failure characteristics. The code gets longer. The 3 AM pages get shorter.