Reservation systems look deceptively simple on the surface—check availability, hold a slot, and confirm a booking. But underneath, they're one of the more honest tests of distributed systems thinking: multiple services need to agree on state, failures happen mid-transaction, and "just use a database lock" stops working the moment you scale past a single node.
ED-DRLE (Distributed Reservation Ledger) is my attempt to build that system properly—not as a toy CRUD app with a reservations table, but as a system that survives partial failure, avoids double-booking under load, and is honest about where it still falls short.
This post walks through the architecture, the trade-offs, and—just as importantly—the gaps I found in my own design when I held it up against a real target load.
The Core Problem:
A reservation isn't a single write. It's a sequence: check availability → hold the resource → charge or confirm → finalize. If any step fails partway, you can't just roll back with a database transaction, because the steps span multiple services and, in a real system, multiple data stores.
This is the classic distributed transaction problem, and it's why ED-DRLE is built around the Saga pattern instead of two-phase commit.
Why Saga Over 2PC:
Two-phase commit gives you strong consistency, but it does so by holding locks across services until every participant agrees to commit—which means one slow or failed service blocks everyone else. For a reservation system under real traffic, that's a latency and availability risk I wasn't willing to take on.
Sagas trade strict atomicity for a sequence of local transactions, each with a defined compensating action if a later step fails. If a hold succeeds but the confirmation step fails, the system runs a compensating transaction to release the hold—instead of everyone waiting on a coordinator.
I used AWS Step Functions to orchestrate the saga steps explicitly, rather than hand-rolling a choreography-based saga with event chains. This made the failure paths visible and testable as a state machine, rather than implicit in a web of service-to-service events.
Storage: Redis-to-DynamoDB Tiering
Reservation holds are short-lived and high-frequency—perfect for an in-memory store, but not something you want as your durable source of truth. ED-DRLE uses a tiered persistence model:
Redis holds active reservation locks and short-TTL holds, optimized for fast reads/writes under contention.
DynamoDB persists confirmed reservations durably, once a hold survives the saga's confirmation step.
This keeps the hot path fast without giving up durability for the state that actually matters long-term.
Protecting the System Under Load:
Two mechanisms sit in front of the core reservation logic:
Token-bucket rate limiting, to prevent a burst of requests for the same resource from overwhelming the hold logic
A circuit breaker, so that if a downstream dependency (e.g., the confirmation service) starts failing, the system stops hammering it and fails fast instead of piling up retries
Neither of these is exotic, but both are the difference between a system that degrades gracefully and one that falls over in a cascading failure.
What the Numbers Actually Say:
I load-tested ED-DRLE at 229 RPS, with a p95 latency of 315 ms. I'm stating these numbers plainly rather than rounding them up or extrapolating to a bigger number, because the honest baseline is more useful—to me and to anyone reading this—than an inflated one.
That number matters because I evaluated it against a 10,000 RPS target I set for the system's design, and it's not close yet. That gap is the most useful part of this project, not the part I'd normally put in a portfolio blurb.
Where the System Still Has Gaps:
I analyzed the design against that 10,000 RPS target, deliberately looking for what would break, rather than assuming the architecture was sound because it worked at moderate load. Four gaps stood out:
No expiry reconciliation between Redis and DynamoDB. If a Redis hold expires but the corresponding DynamoDB state isn't reconciled, the two stores can drift out of sync—a real correctness risk, not just a performance one.
Non-idempotent compensating transactions. Saga compensations need to be safely retryable. Right now, a retried compensation isn't guaranteed to be a no-op if it's already been applied, which risks double-compensating (e.g., releasing a hold twice).
No RDS proxy in front of the relational layer. At higher connection volumes, this becomes a real bottleneck—connection exhaustion under a traffic spike is a predictable failure mode without it.
An unbounded Redis set. Without a cap or eviction policy, this is a slow-building memory risk under sustained load—the kind of thing that looks fine in a load test and fails quietly in production weeks later.
None of these are hidden in the codebase—they're the direct output of stress-testing the design against a target an order of magnitude higher than what I've actually measured.
Why I'm Writing This Instead of Just Shipping It
It would be easy to describe ED-DRLE as "a distributed reservation system built with Saga orchestration, Redis, and DynamoDB" and leave it there. But the more useful engineering story is the second half: what happens when you take your own architecture seriously enough to look for where it breaks.
229 RPS at p95 315 ms is a real, measured number—not a projection, not a "should scale to." And the four gaps above are the actual next milestones for this project, not a hidden list I'm keeping to myself.
If you're working on anything with a similar shape—sagas, tiered storage, rate limiting under real load—I'd genuinely like to compare notes on where your design held up and where it didn't.
GitHub-https://github.com/Tejas-h-blitz/Aws-Distributed-Reservation-Ledger
Top comments (0)