DEV Community: Muhammad Sumon Molla Selim

Designing a Flash-Sale Seat Reservation System in AWS (Part 3): Holds, Payments, and the Slow Path

Muhammad Sumon Molla Selim — Fri, 25 Sep 2026 00:56:13 +0000

TL;DR: After admission, a seat is held for 10 minutes while the candidate pays. The central rule for payments is one-sided: many paths can confirm a booking, but only one can release a seat. Three paths can mark a booking PAID: the browser callback, the gateway's signed webhook, and a background reconciler.

The reconciler asks the gateway for the payment status. Each of the three paths checks with the gateway, and each is idempotent (safe to run more than once). Only the reconciler releases seats. It does so in two cases: the gateway reports that the payment failed or was cancelled, or the hold deadline has passed and the gateway has not reported the payment as complete. If the gateway cannot answer, the hold is extended. A seat is never released when the outcome is uncertain. The "cancel" or "failed" status in the redirect itself is ignored, because anyone can fake it. Durable writes happen outside the request path.

An SQS FIFO queue feeds a worker that writes to DynamoDB with conditional puts (writes that succeed only if a condition holds). The worker never touches the seat counters. DynamoDB TTL and a Lambda driven by DynamoDB Streams are only a safety net for expiry. On launch day, 31.6K holds were created for 20.7K seats, and the queue never held a message older than 4 seconds. The post ends with how we tested the design, in layers and many times over, and what that testing found before launch.

This is the last part of a three-part series. Part 1 covers the architecture and the edge. Part 2 covers how seat admission stays correct.

The hold lifecycle

What states does a booking go through?

A booking has three durable (stored) states:

HELD: the seat is counted and a payment exists. The record carries a deadline 10 minutes in the future.
PAID: the gateway has confirmed the payment. The seat is permanent, and a confirmation email is sent.
RELEASED: the hold ended without payment. The seat is back in the pool, the record is deleted, and the candidate can try again.

If payment creation or the enqueue fails, the booking never reaches HELD. The API undoes the admission before it responds.

The 10-minute hold is a trade-off in both directions. A shorter hold returns unpaid seats sooner, but it punishes anyone who needs some time to approve a payment on their phone. A longer hold keeps seats locked by people who have left and will not pay. Ten minutes was comfortably longer than the gateway's checkout took in testing, and it kept resale fast. On launch day, 31.6K holds were created for 20.7K seats. About a third of the holds expired or failed, and their seats were sold again within the hour.

Challenge 1: payment signals you cannot trust

Why can't the payment redirect decide the outcome?

After paying, the gateway redirects the browser to /payment/callback?paymentID=...&status=success|failure|cancel. The obvious design is to trust that status: mark the booking paid on success, and release it on cancel. That design is broken in both directions:

An attacker controls the query string. Anyone can type status=cancel for a payment ID they have seen. That would free a seat that is still being paid for, or one that is already paid.
Redirects can be old or repeated. The browser's Back button, a second tab or a retried request can all deliver the same redirect again, sometimes after the booking has already changed state.
A redirect may never arrive. The user closes the tab, their phone loses signal, or the redirect times out. The payment completed, but nobody told us.

We considered three designs.

Option	Problem
Trust the redirect status	Easy to fake, easy to repeat, and easy to lose, as above.
Confirm every redirect with the gateway's query API before acting	Better, but the gateway's query API only reports a definite status for a completed payment. So a cancel cannot be confirmed one way or the other. Every fake redirect would also cost us a call to a rate-limited API.
Confirm-only redirects, one releaser (chosen)	A redirect can help to confirm a payment. It can never release a seat.

The three confirm paths are:

Browser callback. The API accepts the callback only for a payment ID that it issued, and only with the per-payment signature that the gateway returned when the payment was created. It also refuses to charge once the hold deadline has passed. Otherwise we could take money for a seat that the reconciler has already released. It then calls the gateway's execute API under a lock with an owner token (SET NX with a random token, and only the owner of that token can release the lock). So the payment is executed at most once, even if the redirect arrives twice. If the execute call times out, the API queries the payment once, as the gateway's own integration guide requires. It does that only to recover a completed payment.
Gateway webhook. The gateway also sends a signed notification (standard AWS SNS HTTP delivery) when a payment completes. The handler verifies the SNS signature and looks up the booking by invoice number. It still queries the gateway and compares the transaction ID before it marks the booking paid. It always answers 200 to a correctly signed message, even when it leaves the decision to the reconciler. That way SNS never builds up a flood of redeliveries.
Reconciler query. This path is covered in Challenge 2.

Every confirm path ends in the same conditional update: HELD → PAID, which succeeds only if the record is still HELD with the same payment ID. Whichever path gets there first wins, and the others do nothing. Confirmation emails are sent after that update succeeds, never before.

One race condition shaped the design early. A fast payer can finish before the worker has written the record. The hold is written to DynamoDB asynchronously, so a user who pays within seconds can come back before any HELD record exists. So at admission, the API also stores a copy of the booking record in Redis, keyed by payment ID, with a TTL slightly longer than the hold. If the callback finds no record in the database, it writes the PAID record from that copy.

Challenge 2: exactly one path releases a seat

Who decides that a hold has expired?

The reconciler is a background loop inside the API fleet. A Redis lock with an owner token makes sure that exactly one instance runs it at a time. Every 20 seconds, that instance does a sweep:

Clean up stale pending admissions from Part 2: seats whose request died before it reached the database.
Find HELD records within 6 minutes of their deadline, or past it.
Handle the most overdue records first, 8 at a time, within a 45-second time budget, so the sweep finishes before the next one starts. When many holds expire at the same time, the records that are closest to hurting a user go first.
Ask the gateway for the status of each record:
- Completed: mark it PAID and send the email. This recovers every user whose redirect never arrived.
- Failed or cancelled, or not completed after the deadline: release the seat, using the payment-guarded undo from Part 2, and delete the record.
- Still pending before the deadline: leave it alone. The user may still be paying.
- The query fails three times: extend the hold by 10 minutes, up to three times, then alert a human. We never release a seat only because we could not get an answer, because the payment may have succeeded.

This is the only status query traffic we send to the gateway on the release side. Keeping it in one rate-limited loop keeps our use of the gateway's rate limits predictable.

DynamoDB TTL is not the main timer. TTL (time to live) is set on the record's deadline attribute. But AWS deletes expired items on a best-effort schedule, and the delay can be long. During a sellout, that delay would mean seats that nobody could buy. So the reconciler expires holds itself, on time. TTL, a DynamoDB Stream, and an expiry Lambda form the safety net. The Lambda acts only on deletions made by the TTL service itself (the reconciler's own deletes are ignored), and only if the deleted record was still HELD. It then runs the same payment-guarded undo. If the reconciler already released that seat, the Lambda's undo does nothing.

One bug from QA is worth sharing. Paid records must remove the TTL attribute. One of our paths wrote PAID records with the hold's TTL still set. DynamoDB would have deleted those confirmed bookings 10 minutes later.

Challenge 3: one token, a rate-limited gateway

How do you share an auth token across a fleet?

The gateway's API calls need a short-lived token. The token grant endpoint (the endpoint that issues a new token) is rate-limited to a couple of calls per hour. If every instance fetched its own token, a deploy or a restart would use up the whole allowance in seconds and lock the entire fleet out of payments.

So the token lives in Redis, shared by the whole fleet:

One refresher at a time. An instance that finds no valid token takes a SET NX lock with a 45-second TTL. Only the lock holder calls the grant endpoint. All other instances wait for the new token to appear, while still respecting their own request deadline.
Refresh early. The token is cached until a few minutes before it expires, so a request never uses a token at the moment it expires.
The grant call outlives the request that started it. The grant call runs on its own 30-second deadline, separate from the request that triggered it. A user's 8-second timeout cannot stop a grant halfway and force another one.
Wait after a failure. A failed grant keeps the lock for 60 seconds instead of releasing it. So the fleet does not retry in a loop and use up the hourly allowance.

Challenge 4: durability off the hot path

How does the database keep up without slowing down bookings?

We considered four ways to store each booking durably.

Option	Why not (or why)
Write to DynamoDB inside the request	Adds a network write to every admission. A throttle or a slow partition at T+0 (the moment registration opens) becomes latency the user sees, or an admission that must be undone.
Kinesis Data Streams	Built for ordered, high-volume streams, but shards need capacity planning, and you must build per-record retries and dead-letter handling yourself.
SQS standard	Scales without limit, but it can deliver messages twice and out of order. The worker would have to handle all of that complexity.
SQS FIFO, high-throughput mode (chosen)	Removes duplicates on send and keeps order within each group. With a large number of message groups it has plenty of throughput. The dead-letter queue is built in.

The details that matter:

Message group = the first 4 hex characters of the candidate hash. That gives 65,536 groups. The groups spread the load, and ordering within a group keeps one candidate's messages in sequence. High-throughput FIFO mode (DeduplicationScope=messageGroup, FifoThroughputLimit=perMessageGroupId) removes the old per-queue throughput limit.
Deduplication ID = the candidate hash plus the request timestamp. A retried send is treated as the same message, but a real re-booking is not.
Visibility timeout of 60 seconds and 5 receives before a message moves to the dead-letter queue. That queue has an alarm that fires on even one message.

The worker on each instance long-polls the queue (it keeps a receive request open and waits for messages instead of asking repeatedly). It writes each record with a conditional put (the write succeeds only if the record does not exist yet). When the condition fails, the worker has to find out why, and a guess is not good enough:

PAID: the callback got there first. Skip the message.
HELD with the same payment ID: a repeated delivery of the same message. Delete the message.
HELD with a different payment ID: a stale record from an earlier attempt that was released, after which the candidate booked again. Replace it, on the condition that the old payment ID is still there. We found this case in QA, and it is why the payment ID is part of the check. Without it, the stale record hid the new payment, and the new booking could never be marked paid.
The read failed: leave the message unacknowledged and let SQS deliver it again.

The worker never touches the seat counters. Seats belong to Redis and the admission script. Records belong to the worker. When both could decrease a counter, a redelivered message could free a seat that someone still held. Giving each piece of state exactly one owner made that bug impossible.

DynamoDB itself uses on-demand capacity with one table. The partition key is the candidate hash. A GSI (global secondary index) on payment ID serves the callback's lookup. Point-in-time recovery and deletion protection are both on. In the first five minutes the worker wrote about 37,000 write units' worth of records. The table absorbed that without a single throttle, because the queue had already smoothed out the spike.

Two more stream consumers connect to the table, both with failure handling built in:

The sync Lambda copies every PAID record into a relational database for the back-office app, using upserts (insert or update in one statement).
Both Lambdas use ReportBatchItemFailures, split a batch in half on error, limit the record age, and send anything they give up on to an SQS on-failure destination. Without this, a dropped stream batch would mean a lost release or a missing paid booking.

The live status document

How do thousands of writers keep one JSON file correct?

The availability counts on the page come from status.json in S3 (see Part 1). The API and the expiry Lambda both write it. Our first version read the file, changed one field and wrote it back. With several writers at the same time, two writes overlapped, and an old write reopened a level that had just filled.

The rule that fixed it: every writer builds the whole document from Redis, where the counters and the open/closed flags live, and overwrites the file. Writers that run at the same time then arrive at the same result instead of competing over a copy. Two latches (one-time flags) keep the writes cheap:

A 2-second SET NX throttle turns a burst of admissions into a single S3 write.
The level-full and closed transitions each publish exactly once.

A release clears those latches, so a freed seat reopens the level on the page.

Test, find, fix, repeat

Was this design right the first time?

No. The design in this series is the result of about four months of testing, finding problems, fixing them and testing again. That loop is one of the main reasons launch day was quiet. Every change went through several layers of testing, and each layer answered a different question.

Unit tests for every package: each branch of the admission script, the gateway client, the webhook signature check and the status document.
An end-to-end suite on a local copy of the stack, with a mock payment gateway that can be told to fail or cancel. It runs a booking all the way to PAID. It also runs the failure cases: a gateway error, a duplicate booking, a new booking after a released hold, a forged callback, a callback that arrives twice, and a payment that completes before the worker has written the record.
Full-rate load tests on a stack the same size as production. Each run used 40,000 test candidates, sent 2,000 bookings per second for 25 seconds, and then sent a slower stream of late arrivals and retries. A run passed only if admissions stopped exactly at the cap, DynamoDB reported no throttling, and the counters matched the database afterwards.
The payment gateway's sandbox, for real payment flows: pay, cancel, let a hold expire, and book again.
Reviews and audits of the design, the code and the infrastructure. They looked for signals an attacker could fake, rate limits we could use up, and missing permissions.
Launch rehearsals. We ran the same reset, load-candidates and open steps that we would use on the day, more than a hundred times, from the same pipeline.

Each layer found problems that the others missed. A few examples:

Load tests: the Redis client's default connection pool was sized for normal traffic, not for a burst. We raised it to 300 connections per instance and ran the test again.
Sandbox: the counters drifted after a user cancelled and then booked again. The worker found the old record, treated the new booking as a duplicate, and decremented a counter. This is why the worker never touches the counters, and why it classifies every conflict with a consistent read (Challenge 4).
Testing the undo paths: a rollback that ran twice pushed a level counter below zero. This is why every undo checks and acts in one atomic step (Part 2).
Reviewing paid records: a paid record could still carry the hold's TTL, so DynamoDB would have deleted a paid booking about ten minutes later. A paid record now has no TTL.
Security review: the cancel status in the redirect has no signature, so anyone can fake it. Checking each one with the gateway would also use up the gateway's token limit. This is why a redirect can never release a seat (Challenge 1).
A week before launch: two concurrent writers could leave the status page showing a full level as open. The writers now run in a fixed order, and the page also checks the counts, so a full level can never look open.

None of these problems is unusual. Most of them only show up under a specific timing, order of events or load, and the rest only show up when someone looks for them on purpose. The only reliable way to find them before users do is to test at every layer, many times, and to test again after every fix.

Lessons

What would I tell someone building this next?

Decide the one fact everyone competes for in one atomic step, and keep it off the database. Everything else in the system can be eventually consistent if that one decision is exact.
Make the request path as short as possible. Admit, create the payment, enqueue, and respond. Every durable write that can happen after the response should happen after the response.
Give every piece of state exactly one owner. Redis owns seats, the worker owns records, and the reconciler owns releases. Most of our pre-launch bugs came from two components writing the same state.
Treat every external signal as a hint. Redirects, webhooks and query results can be faked, delayed, duplicated or missing. Confirm with the source, make every state change conditional, and let exactly one path take the action that cannot be undone.
Never release when you are not sure. If you cannot tell whether someone paid, extend the hold and alert a human. Selling a paid seat twice is worse than holding an unpaid one a little longer.
Idempotency must live in the same atomic step as the action. A check followed by a separate action is a race, and under a burst every race is lost sooner or later.
Add capacity before a sudden jump in traffic, and load-test with a mock of every external dependency. The mock gateway let us practise the full booking path at full rate as often as we wanted.
Plan as much time for testing as for building. Test in layers, from unit tests to full-rate load tests and launch rehearsals. When you find a problem, fix it, add a test for it, and run every layer again. Most of what makes this design correct was learned that way.

The result was the launch day in Part 1: 20,700 seats sold, 99.99% of requests served without a server error, and zero double bookings.

Designing a Flash-Sale Seat Reservation System in AWS (Part 2): Never Sell a Seat Twice

Muhammad Sumon Molla Selim — Fri, 25 Sep 2026 00:49:57 +0000

TL;DR: Every booking goes through one Redis Lua script. The script checks, in order, whether the candidate is already admitted, whether their level is full, and whether any seat is left overall. Then it counts the seat and marks the candidate. All of this happens as one atomic step (nothing else can run in between). Lua won over MULTI/WATCH (too many retries when many requests compete) and distributed locks (extra round trips while holding the lock). An in-memory counter can still be wrong in four ways. Each needed its own fix:

Double rollbacks: the undo step has a guard, so running it twice does nothing the second time.
Lost replies: every call carries a nonce (a random value that identifies one request), so a retried call gets its original slot back.
Crashed requests: a list of pending admissions, plus a cleanup job, returns seats that never reached the database.
Failover: a booking counts only after the Redis replica confirms it has the write (WAIT 1 200). Otherwise the booking is undone and the client retries.

This is Part 2 of a three-part series. Part 1 explains why seat admission lives in Redis rather than a database. Part 3 covers what happens after a seat is admitted.

The invariants

What exactly does "never sell a seat twice" mean?

Before writing any code, we wrote down three invariants (rules that must always be true). Everything in this post exists to keep them true.

Caps hold. The number of admitted seats per level never exceeds that level's cap, and the total never exceeds the global cap.
One live booking per candidate. A candidate code is linked to at most one seat that is held or paid.
No leaks. Every admitted seat becomes either paid or released in the end. No seat stays counted with nobody holding it.

Invariants 1 and 2 are what "zero double bookings" means. Invariant 3 stops the event from ending with unsold seats that the system forgot about.

The state

What lives in Redis?

Key	Type	Purpose
`booking:open`	string	The admin's open/close switch
`candidates:valid`	set	Allowlist of eligible candidate codes, loaded before opening
`candidate:email:<code>`	string	The email on file for each code
`adm:seen`	set	Candidate codes that currently hold a seat
`adm:total`	integer	Seats admitted overall
`adm:level:<L>`	integer	Seats admitted per level
`adm:pending`	hash	`code → nonce, level, time, slot` for admissions not yet safely in the database
`hold:*`	strings with a TTL (they expire on their own)	Per-payment stashes (temporary copies): the booking record, the payment URL, and the execute and rollback guards

Two configuration choices matter more than they seem to:

maxmemory-policy noeviction. When memory runs low, Redis must refuse writes with an error. It must never silently delete adm:seen to free space. Losing that set would allow duplicates. A write error, on the other hand, simply becomes a 503 that the client can retry.
Normalised identifiers. Candidate codes have their surrounding spaces removed and are converted to uppercase before any lookup. Otherwise the counter would treat ab123 and AB123 as two different people. A SHA-256 hash of the normalised code is the database key. Its first characters become the queue message group and the payment's invoice number.

Gates, then the script

Why check some things outside the script?

The gates are simple read checks. They reject most invalid traffic before any write happens:

Is booking open? If not, 503 closed.
Is the code on the allowlist? If not, 400 invalid_code. The allowlist is loaded into Redis from the back-office database before opening. There is deliberately no fallback to the database while the system runs. A code that is missing from the allowlist must not turn into a database query at 50,000 requests a minute.
Does the email match the code? If not, 400 email_mismatch.

A hidden honeypot field is checked before all of these. (A honeypot is a form field that real users never see, so only bots fill it in.) A form that fills it in gets a fake success response that looks real, with no admission and no log entry.

The gates can live outside the script because their values change only when an operator changes them, before the event. The checks that compete with each other at the moment of booking are all inside the script:

-- KEYS: adm:seen, adm:total, adm:pending
-- ARGV: code, max_total, level, level_cap, nonce, now_ms
local code, level, nonce = ARGV[1], ARGV[3], ARGV[5]
local level_key = 'adm:level:' .. level

-- 1. Already admitted? A retry of the same request gets its slot back.
if redis.call('SISMEMBER', KEYS[1], code) == 1 then
  local p = redis.call('HGET', KEYS[3], code)
  if p then
    local n, lvl, _, slot = string.match(p, '^([^|]*)|([^|]*)|([^|]*)|([^|]*)$')
    if n == nonce then return {tonumber(slot), lvl} end
  end
  return {-1, ''}                                  -- duplicate
end

-- 2. Level full?
local cap = tonumber(ARGV[4])
if cap > 0 and tonumber(redis.call('GET', level_key) or 0) >= cap then
  return {-3, level}                               -- level full
end

-- 3. Any seat left overall?
if tonumber(redis.call('GET', KEYS[2]) or 0) >= tonumber(ARGV[2]) then
  return {-2, ''}                                  -- closed
end

-- 4. Admit: count, mark, and record as pending.
local slot = redis.call('INCR', KEYS[2])
redis.call('INCR', level_key)
redis.call('SADD', KEYS[1], code)
redis.call('HSET', KEYS[3], code, nonce .. '|' .. level .. '|' .. ARGV[6] .. '|' .. slot)
return {slot, level}

The order matters. The duplicate check comes first, so a candidate who already has a seat is told so, even after their level fills up. The level check comes before the global check, so the user learns the most specific reason. Only the last branch writes anything. A rejection therefore costs a few reads on one thread.

The API loads the script once with SCRIPT LOAD and calls it with EVALSHA. After a restart or failover, the new primary has an empty script cache and answers NOSCRIPT. The client reloads the script and retries once.

Why not the other atomic building blocks in Redis?

Option	Problem under a flash sale
`INCR`, then check, then `DECR` if the counter went too far	The counter briefly goes above the cap, and readers can see that. The duplicate check is a separate step, and another request can run between the steps, so two requests for the same candidate can both pass.
`MULTI/WATCH` (optimistic transactions)	Every booking at the same level watches the same key. In the opening second, nearly all transactions abort and retry. That creates a storm of retries on the busiest key.
Distributed lock (for example Redlock) around a read, change, write sequence	At least two extra round trips while holding the lock, and a lock service that is one more thing that can fail.
Lua script (chosen)	Runs from start to finish on the Redis thread. There is no lock to hold across the network and no retry loop.

Two more design details keep the rest of the system simple:

"Level full" is published once. When a level fills, a SETNX latch (a key that only the first writer can set) makes exactly one API instance rewrite status.json. Thousands of requests cannot compete to publish the same change.
A duplicate can continue. Suppose a candidate's booking is still HELD and they submit again, perhaps because their first request timed out in the browser. The 409 response then carries their existing payment URL. The URL is read from a Redis stash (never from the database), and the page redirects them straight to payment. Without this, a user whose first response was lost could not reach the seat they already hold.

Failure mode 1: undo that runs twice

What happens when rollback runs twice?

A seat must be given back when a booking fails: the payment could not be created, the enqueue failed, or the hold expired without payment. Several paths can trigger that release. Early in load testing, two of them sometimes ran for the same booking, and the counters went negative. The undo step decreased the counters every time it ran, even though removing the candidate from the set only had an effect the first time.

The fix was to make undo a script that only acts if the candidate is still admitted:

if redis.call('SREM', KEYS[1], ARGV[1]) == 0 then return 0 end  -- already undone
redis.call('DECR', KEYS[2])                                     -- total
redis.call('DECR', 'adm:level:' .. ARGV[2])                     -- level
redis.call('HDEL', KEYS[3], ARGV[1])                            -- pending entry
return 1

That fixes repeats, but not a stale (out of date) undo. Suppose a candidate's first payment is released and they book again. Then a late event for the first payment arrives, such as a TTL expiry or a repeated callback. A guard that only asks "is the candidate admitted?" passes, because the candidate is admitted again. The stale event then frees the new seat.

So every release that a payment triggers goes through a second guard, keyed by payment ID. SET hold:rollback:<paymentId> NX EX 86400 and the undo run in the same script. A payment can release a seat at most once, and only the payment that owns the seat can release it.

The rule we took from this: any operation that more than one path can trigger must be idempotent, and the guard that makes it idempotent must live in the same atomic step as the action itself. (Idempotent means it is safe to run more than once, with the same result.) A check followed by a separate action is a race.

Failure mode 2: the reply that never arrives

What if the script succeeds but the API never hears about it?

Networks drop replies. When that happens, the Redis client does the sensible thing and retries the command. For the admission script, that sensible retry is a disaster.

The first call admitted the candidate. The retry finds them in adm:seen and returns "duplicate". The user sees "already registered" with no payment link, and their seat stays counted, with no hold behind it.

The fix is a nonce for each request. The API generates a random nonce for each booking request and passes it to the script. On admission, the script records it in the adm:pending hash (step 4 above). When a call finds the candidate already admitted, it compares nonces. The same nonce means this is the same request, retried, so the script returns the original slot. A different nonce means a real second attempt, so it returns duplicate.

We also run the admission with a context that is separate from the HTTP request's cancellation. If a user closes the tab during the request, that cannot stop the call halfway through its retry.

Failure mode 3: the request that dies halfway

What if the API crashes between admitting and enqueueing?

The hot path is three steps: admit, create payment, enqueue. If the process dies after the first step, the seat is counted, but no record exists anywhere that could expire it. Neither the database nor the queue knows the seat exists. That would break invariant 3.

The pending ledger closes the gap. (The ledger here is simply the list of admissions that are not yet safe in the database.) Every admission writes an adm:pending entry in the same atomic step. /book clears the entry only after the enqueue has succeeded, because from that point on the queued record carries the booking. Any rollback clears the entry atomically too.

On every sweep, the reconciler (the single background runner described in Part 3) looks for pending entries older than 15 minutes:

If a database record exists for that candidate, the booking completed. The reaper (the cleanup step) removes the entry.
If no record exists, the request died between steps. The reaper undoes the admission, with the same nonce check, so it can never undo a newer attempt.

Fifteen minutes is longer than the 10-minute hold plus the queue's worst-case delay. A slow but healthy request is never cleaned up by mistake.

Failure mode 4: the failover that forgets

Can a Redis failover sell a seat twice?

Yes, if you let it. ElastiCache replication is asynchronous. The primary confirms a write to the client before the replica has received it. If the primary dies in that gap, the replica is promoted to primary without the last few admissions.

At the last seat, that gap means one seat sold twice. The same applies to a candidate: the promoted replica does not know they booked, so they can book a second seat.

We compared three options.

Option	Trade-off
Accept the risk	The window is milliseconds long and failover is rare. But it breaks the one promise the client cared about, at exactly the moment when load is highest.
Count seats in the database too	Correct, but it puts the database back on the hot path, which is what Part 1 worked to avoid.
Wait for a replica acknowledgement (chosen)	Send the script and `WAIT 1 200` together in one pipeline: wait up to 200 ms for one replica to confirm it has the write. If the replica confirms, the admission counts. If not, undo it and return a `503` that the client can retry.

Under normal conditions, replication lag between Availability Zones is a few milliseconds, so the wait adds almost nothing to a normal booking. Both commands go out in one pipeline, so there is no extra round trip.

This choice has two consequences we accepted deliberately:

Bookings fail closed. That means: if the replica is down, every booking returns 503 until ElastiCache replaces it. We chose correctness over availability for this one path. The page retries with increasing delays, and replacing a replica takes minutes, not hours.
WAIT is not consensus. It makes a lost admission far less likely. It does not make Redis strongly consistent. With one replica, the node that confirmed the write is the node that gets promoted. That covers the realistic failure: the loss of one node or one Availability Zone.

As a final safety net, every five minutes the reconciler compares the Redis counters with a count of the records in DynamoDB. It raises an alert only if the mismatch is still there on two checks in a row, because records normally lag behind the counter by a few seconds.

How we tested it

How do you convince yourself a counter is right?

Unit tests against an in-memory Redis for every branch of the script: caps, duplicates, nonce replay, idempotent and stale undo, and the pending reaper.
Race tests that send thousands of admissions at the same time against a small cap. They check that the admitted count equals the cap exactly and that no candidate appears twice.
A real primary and replica in Docker for the replica acknowledgement path. In-memory fakes usually report zero replicas, so they cannot test it.
Full-rate load tests with 40,000 test candidates against the real stack. They pass only if admissions stop exactly at the cap and the counters match the database afterwards.

On launch day the invariants held. 20,700 seats were sold, with zero double bookings and zero overbookings.

What's next

An admitted seat is only half a booking. Part 3 follows it through payment. It covers why a payment redirect can confirm a booking but never release one, and how a single runner makes every release decision. It also covers how a rate-limited gateway token is shared across the fleet, and how the queue and DynamoDB keep up without ever slowing down a booking.

Designing a Flash-Sale Seat Reservation System in AWS (Part 1): The Architecture

Muhammad Sumon Molla Selim — Fri, 25 Sep 2026 00:46:21 +0000

TL;DR: A national certification exam had 20,700 seats across five levels. When registration opened, far more people tried to book than there were seats. The previous system crashed under that load. It also sold the same seat twice. The new design follows three rules.

First, one atomic Redis script decides who gets a seat. (Atomic means the whole check-and-count runs as one step that nothing can interrupt.) No database takes part in that decision.

Second, the request path does only three things: admit the booking in Redis, create a payment, and put a record on a queue. Everything else happens later, in the background.

Third, everything that can be a static file is served from the edge (the CDN, close to the user). CloudFront, S3 and an AWS WAF web ACL handled about three out of four requests without touching a server.

We sized the API fleet from load tests and added servers before the event. We did not use autoscaling. In the first hour it served 1.5M+ requests at the origin, peaked at 54.8K requests per minute, kept median latency under 10 ms, returned a server error on fewer than 0.01% of requests, and sold every seat with zero double bookings.

This is Part 1 of a three-part series. It is written as a design reference: the challenges, the options we compared, and why we chose what we chose.

Part 1 (this post): the problem, the architecture, the edge, and capacity.
Part 2: never selling a seat twice, covering the admission script, retries, and failover.
Part 3: holds, payments, and the slow path, covering the hold lifecycle, payment signals we cannot trust, and durable storage that runs in the background.

The problem

What made this hard?

The work itself looks simple: a form with a candidate code, an email address and an exam level, then a payment. The traffic pattern is what makes it hard. Registration opens at an announced minute, so almost everyone arrives in the same few seconds. They keep refreshing the page until they get a seat or the seats run out. The previous system crashed at exactly that moment. It also sold some seats twice. The client's requirement was simple and direct: stay up, and never book a seat twice.

Written out as requirements:

Requirement	Detail
Hard caps	A cap (a maximum) per level, for five levels, and a global cap of 20,700. Never exceeded, not even by one seat.
One booking per candidate	A candidate code can hold at most one live booking.
Eligible candidates only	The code must be on an allowlist loaded in advance, and the email must match the one we have stored for that code.
Pay to keep the seat	A seat is held for 10 minutes while the candidate pays through the payment gateway. If the payment is not completed, the seat returns to the pool of free seats.
Money-safe	Nobody is charged for a seat they did not get, and nobody who paid loses their seat.
Available under the burst	The page must load and answer quickly, even when everyone arrives at once.
First come, first served	No lottery. Whoever is admitted first gets the seat.

"Never exceed the cap" and "one booking per candidate" are the two rules that define correctness. Part 2 is entirely about them. The rest of this post is about surviving the burst while keeping those rules true.

The shape of the load

What does a flash sale look like in the traffic numbers?

Figure 1 shows the real traffic around opening time, grouped into five-minute buckets from CloudWatch.

Three things stand out, and each one shaped the design:

The ramp is a cliff. In other words, traffic does not grow slowly; it jumps almost straight up. Origin traffic went from near zero to about 12K requests per minute in the first five minutes after opening, and to more than 50K per minute within twenty minutes. Anything that reacts to load, such as an Auto Scaling policy, a function with a cold start, or a database that scales its capacity, reacts too late. The moment that matters has already passed.
Most requests are not bookings. People reload the page and check availability long before and long after they submit the form. In the first hour CloudFront saw about 6.2M requests. Only about 1.5M of them reached the servers. Every request the edge answers is one that the booking path never has to handle.
Most of the origin traffic is rejections. Once seats run low, most /book calls end with a fast "no": duplicate, level full, or closed. A rejection must cost almost nothing, because there are far more rejections than admissions.

Challenge 1: where do we decide who gets a seat?

What is the single hardest decision in this system?

Every booking asks the same question: is there still a seat at this level, and has this person already booked one? Thousands of requests ask it at the same instant. The answer must be exactly right for every one of them. Whatever component answers that question is the bottleneck of the whole system, so we chose it first.

We considered four options.

Option	How it works	Why not (or why)
A. Relational database row lock	`SELECT ... FOR UPDATE` on a `seats_left` row, then insert the booking	Correct, but every booking must wait its turn on one row. Lock waits build up, connection pools run empty, and everything times out at the same time. This is the classic failure of older systems of this kind.
B. DynamoDB conditional counters	`UpdateItem` with `seats_left > 0` on a counter item, plus a transaction for the duplicate check	Correct and serverless, but a single counter item is a hot key (one item that every request writes to). One partition limits how many writes per second it can take. Adding a duplicate check means a transaction on every attempt, and even a rejection still costs a write.
C. Virtual waiting room	Put everyone in a queue and let people in at a fixed rate	Protects the backend, but it is a whole product to build or buy. Users wait for minutes, and you still need an atomic counter at the end of the queue.
D. One atomic script in Redis	A Lua script checks for a duplicate, checks the level cap and the global cap, and records the admission, all in one call	Redis runs a script from start to finish before it runs anything else, so no lock is held while waiting on the network. It costs one round trip of microseconds, and a rejection is only a read. The open questions, durability and failover, are answered in Part 2.

We chose D. The deciding argument was this: Redis runs commands on a single thread, and that single thread is the lock. It is held only for the microseconds a script runs, never for a network round trip. The same property makes rejections nearly free. That matters because most of the traffic is rejections.

The cost is that the source of truth for "who has a seat" now lives in memory. That creates three problems that Part 2 has to solve: a reply lost on the network, a process crash between steps, and a failover that loses the last few writes. We accepted those problems with open eyes. The other options had problems we could not solve within a single burst.

We also chose the smallest possible Redis setup: one shard (cluster mode disabled), one primary and one replica in different Availability Zones, with automatic failover. The script touches several keys at once. On one shard, that needs no extra work to keep the keys together (no hash tags). As Figure 4 shows later, a single cache.r6g.large primary never went above 10% engine CPU.

Challenge 2: keep the hot path short

What exactly happens inside a booking request?

The previous system did all its real work inside the request. It wrote the booking to the database, sent the email and updated the counters, all while the user waited. Under a burst, each of those steps becomes a waiting line, and the slowest one sets the latency for everyone.

We split the work into a hot path (the steps the user waits for) and a slow path (the steps that run after the response is sent).

On the hot path, POST /book does exactly three things:

Admit the booking in Redis with the atomic script.
Create a payment with the payment gateway and get back a payment URL.
Enqueue a HELD record, which means send it to an SQS FIFO queue.

It then returns 202 Accepted with the payment URL, and the browser redirects to the gateway. If step 2 or step 3 fails, the API undoes the admission before it responds. A failure never leaks a seat (leaves a seat counted with nobody holding it).

The database is deliberately missing from this list. A worker reads the queue and writes the records into DynamoDB a few seconds later. Payment confirmations, expiry, emails and the back-office sync all run on the slow path. Part 3 covers that path in detail, including why it is safe for the database to be a few seconds behind the seat count.

The payment gateway call is the one dependency on the hot path that we do not control. It is the only reason the hot path is not a pure in-memory operation. It is also the only slow step, and it shows in the slowest requests: p99 latency was 4.1 s in the first five minutes, when every admitted user was creating a payment at the same time. After that it settled to about 0.3 s. The median stayed at 2 to 5 ms the whole time, because most requests were fast rejections that never reached the gateway. Placing the gateway call after admission is what makes this work: rejected requests never wait for it.

Challenge 3: keep traffic off the servers

How do you make three out of four requests never reach your code?

The cheapest request is the one your servers never see. We put everything we could behind CloudFront.

The page itself is a static HTML file in a private S3 bucket, served through CloudFront with Origin Access Control. Page loads never touch a server.
Availability counts (seats left per level, and whether booking is open or closed) are a small status.json document in the same bucket. The API rewrites it as the counts change. The browser reads it once when the page loads. After that, the page reacts to API responses instead of asking the server again and again. A 429 closed disables the form, and a 409 level full disables that level. CloudFront serves the file with caching disabled, so readers always see the latest version. The load still lands on S3, not on our servers.
One origin, one hostname. CloudFront routes /book, /payment/callback, /payment/webhook and /health to the Application Load Balancer, and everything else to S3. The page and the API share one hostname, so the browser sends no CORS preflight (an extra request the browser sends first when the page and the API are on different hosts). Under a burst, skipping the preflight removes a whole extra request from every booking.

In front of all of this sits an AWS WAF web ACL attached to the CloudFront distribution. It checks every request at the edge before CloudFront forwards anything.

Rule	Purpose
Rate limit on `/book`, 2,000 requests per 5 minutes per IP	Stops a single client from flooding the booking endpoint
Rate limit on `/payment/callback`, 100 per 5 minutes per IP	Makes it impractical to guess payment IDs
AWS managed IP reputation list and known-bad-inputs rules	Drops known attackers and exploit payloads at low cost
Body size limit on `/book`	The form is tiny, so a large request body is never legitimate

The per-IP limits are deliberately generous. Mobile carriers in the region use carrier-grade NAT, which puts many subscribers behind a small number of shared public IPs. A strict per-IP limit would have blocked whole neighbourhoods of real candidates. The rate rules exist to stop abuse, not to control traffic. The admission script already rejects excess load cheaply on its own.

The load balancer accepts traffic only from CloudFront. Its security group allows CloudFront's origin-facing IP ranges and nothing else. But that list covers every CloudFront distribution in AWS. An attacker could create their own distribution, point it at our load balancer, and bypass our web ACL. So CloudFront also adds a secret header to every request it sends to the origin. The ALB listener forwards only requests that carry that header. Its default action is a fixed 403. CloudFront talks to the ALB over HTTPS, so the secret never crosses the internet in clear text.

Challenge 4: capacity for a cliff

Why pre-scale instead of autoscale?

We considered three compute options.

Option	Why not (or why)
EC2 Auto Scaling with target tracking	Scaling reacts to metrics that are a minute or more behind, then waits for new instances to boot and pass health checks. By the time new capacity arrives, the jump in Figure 1 is already over.
AWS Lambda behind API Gateway	Scales fast, but the burst hits concurrency ramp limits and cold starts at exactly the wrong moment. Every new execution environment also opens its own Redis connections. A traffic spike then becomes a flood of new connections to the one component that must not stall.
A fixed, pre-scaled fleet	Simple and predictable. Size it from load tests, add instances hours before opening, and remove them afterwards.

We pre-scaled a fixed fleet: three instances, one per Availability Zone, behind an Application Load Balancer. Each instance ran the API and the queue worker as two systemd services, with a fixed Redis connection pool (300 connections per instance, enough that the burst never waits for a connection). The instances live in private subnets with no public IPs. They reach the payment gateway and the email provider through a NAT gateway in each Availability Zone, and they reach AWS APIs through VPC endpoints. Operators use AWS Systems Manager Session Manager, not SSH.

We chose the size from load tests before launch. The setup was a k6 load generator in the same region, an allowlist pre-loaded with 40,000 test candidate codes (about double the real seat count), and a mock payment gateway with adjustable latency. The mock let us practise the full booking path at full rate without hitting the real gateway's limits. The tests also shaped several details you will see in Parts 2 and 3.

On the day, the fleet had far more capacity than it needed.

We sized it that way on purpose. The extra capacity cost a few dollars an hour for a few hours. An outage at opening would have cost the client the whole event. With these numbers, the next run could use two Graviton instances and a smaller Redis node and still use less than a third of their capacity at the peak.

The whole architecture

What does it look like end to end?

A booking moves through it like this:

The browser loads the page and status.json from CloudFront, which serves both from a private S3 bucket. The AWS WAF web ACL on the distribution checks every request.
The browser posts to /book. CloudFront forwards the request over HTTPS, with the origin secret header, to the ALB in the public subnets.
An API instance in a private subnet checks the gates (is booking open, is the code valid) and runs the admission script on ElastiCache for Redis (primary plus replica across two AZs, TLS and AUTH enabled).
If admitted, the API creates a payment with the payment gateway (outbound through the NAT gateway) and enqueues a HELD record on SQS FIFO. It returns the payment URL.
The worker on each instance long-polls the queue. (Long-polling means it keeps a request open and waits for messages instead of asking again and again.) It writes the record to DynamoDB with a conditional put. (A conditional put is a write that succeeds only if a condition is true, here that the record does not exist yet.)
The user pays at the gateway, and the gateway redirects the browser to /payment/callback. The gateway also sends a signed webhook to /payment/webhook. Either one can mark the booking PAID. The API then sends a confirmation email.
A reconciler (a background loop inside the API fleet, with only one instance running it at a time) releases holds that were not paid in time. DynamoDB TTL (time to live: DynamoDB deletes a record some time after its deadline passes) and DynamoDB Streams drive an expiry Lambda that acts as a safety net.
A second stream consumer, the sync Lambda, copies paid bookings into an RDS database for the back-office app. The back-office app also loads the candidate allowlist into Redis before opening.

CloudWatch alarms cover the things that should wake someone up: ALB 5xx errors and unhealthy hosts, Redis CPU, memory, connections and replication lag, queue depth and the dead-letter queue, DynamoDB throttles, Lambda errors and stream iterator age. Application logs go to CloudWatch Logs, so an incident never starts with an SSH session. A few custom metrics sit next to them: admissions per minute, replica-acknowledgement failures (explained in the next post), holds past their deadline, and any difference between the Redis counters and the database.

Launch day

Did it work?

Metric (first hour)	Result
Requests reaching the servers	1.5M+ (about 6.2M at the edge)
Peak throughput at the origin	54.8K requests per minute
Median latency	under 10 ms (2 to 5 ms at the ALB)
Requests served without a server error	99.99%
Seats sold	20,700 of 20,700, about 95% of them within the first hour
Double bookings or overbookings	0

The number I care about most is the last one. It did not happen by accident. It is the result of the decisions in Part 2, and of months of testing, finding problems and fixing them before launch (Part 3 describes how).

What's next

Part 2 looks inside the admission script. It covers what the script checks and in what order. It explains how the script stays correct when a reply is lost and the client library retries, and how the seat from a crashed request is returned to the pool. It also shows how we closed the short time window in which a Redis failover could silently sell the same seat twice.