Kushal Baral

Posted on Aug 22

Millions of Clicks, 200 Tickets: Building an Endpoint That Cannot Oversell

#backend #scalability #reliability #python

Friday morning, 10 AM sharp. Tickets go live.

200 seats. A band people have waited years to see. Within seconds, millions of fans are smashing the Buy button from phones on trains, laptops in offices, that one tablet at the back of a taxi.

Somewhere between request number 199 and request number 201, your database starts lying to you.

Not because it is broken. Because you wrote code that made perfect sense and quietly assumed the world is polite. It is not. This post is the story of every way that assumption breaks, and how to build an endpoint that stays honest no matter how many people click at once.

Every claim below is proven under real load against a working project: k6, autocannon, or plain loops of curl, whatever you prefer. The repo is linked at the end.

The uncomfortable truth about this bug

Here is the part nobody warns you about: the naive solution works fine. Genuinely fine. For a shop selling ten tickets a day, for a demo, for your portfolio site, for everything you have probably ever shipped.

These bugs do not exist at low traffic. They only exist when requests overlap in time, which means no code review catches them, no unit test fails, and your launch demo goes flawlessly. Then the crowd arrives and the lies begin.

You do not need millions of users either. A laptop, one endpoint, and any load tool (k6, autocannon, even a dumb for-loop of curl) reproduce everything below. Scale only changes how fast the lie spreads.

Three quick deaths of a naive counter

The version everyone writes first keeps count in memory and does if count < LIMIT: count += 1. It dies three ways:

The read-write gap: count += 1 is really read, add, write. Two simultaneous requests both read 199, both write 200, ticket 200 gets sold twice.
Multiple workers: run four processes for speed and there are four separate counters in four memory spaces, each selling its own private inventory.
Restarts: deploy at noon, memory resets, this morning's tickets go back on sale.

Shared mutable state under concurrency is dead on arrival, so the counter moves to Postgres. Where it dies again.

The database still lets you race

row = await conn.fetchrow("SELECT sold FROM tickets WHERE id = 1")
if row["sold"] < limit:
    await conn.execute("UPDATE tickets SET sold = sold + 1 WHERE id = 1")

Shared state, survives restarts, multiple workers all see the same number. Ship it?

Still oversells. Run it under load and watch.

The database serializes statements, not intentions. Your check and your write are two separate statements, and between them sits a network round trip where another connection happily runs its own check against the old value. Same movie as the in-memory counter, new costume.

The fix: make the check and the write ONE thing

This is the moment the whole problem collapses:

UPDATE tickets
SET sold = sold + 1
WHERE id = 1 AND sold < total_limit
RETURNING sold;

One statement. Postgres takes the row lock needed for the update and re-evaluates the WHERE condition against the current row version before applying it; if the condition fails you get zero rows back, which means sold out. There is no gap. There is no "between the check and the write" anymore, because they are the same operation.

k6 against this version: 500 concurrent purchases racing for 200 tickets. In repeated runs, exactly 200 succeeded and the remaining 300 were rejected as sold out. And that outcome is not luck or statistics: the check and the write are one atomic statement, so there is no application-level gap between them, regardless of how many workers hammer it.

Rule worth tattooing somewhere: never trust read-then-write across a network round trip. Push conditions INTO the statement.

Retries sell twice

Correctness achieved, so naturally the network ruins the party again.

Mobile networks retry. Users double-click. Payment gateways re-send notifications. Our endpoint charges one ticket per click, so a timeout followed by a retry now buys TWO tickets for one person.

The fix has a fancy name, idempotency, and a simple idea behind it: the client sends a key, and the same key always maps to the same outcome.

INSERT INTO purchases (idempotency_key, status)
VALUES ($1, 'pending')
ON CONFLICT (idempotency_key) DO NOTHING;

First request: the row inserts as pending and the flow moves toward confirmation. Retry with the same key: the insert does nothing, we look up what already happened and replay that stored outcome instead of selling again. One key always maps to its stored outcome, for as long as that record is retained (real systems define a retention window for keys). Stripe built entire SDK features around this concept, which tells you how often networks misbehave in production.

Proving it under fire

Correct when tried manually means nothing, as established. Time to be scientific. Load testing with k6:

export const options = {
  scenarios: {
    rush: {
      executor: 'shared-iterations',
      vus: 500,            // 500 virtual users
      iterations: 500,     // each buys once
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<300'],
    checks: ['rate>0.99'],
  },
};

Two things being verified here. Correctness: never more than 200 confirmed sales. And speed: latency thresholds.

p95 means 95 percent of requests finished faster than that number. Percentiles exist because averages lie: 99 fast requests plus one five-second disaster average out to a polite-looking blur, while p99 points directly at the disaster.

One more production trick: when tickets run out, stop hammering the database. Cache a "sold out" flag in worker memory for a few seconds and reject instantly. The database remains the only source of truth: the flag can cause an early rejection, but it can never cause an oversell. And keep its lifetime short, because inventory can come BACK.

Final boss: payments take seconds

Everything so far assumed buying is instant. Real money is not.

If we mark the ticket SOLD immediately and wait for payment, every declined card permanently eats one ticket. Inventory leaks, slowly, forever. If instead we wait for payment while holding our precious atomic row, the entire system queues behind one human typing their CVV at the speed of a sleepy sloth. Connection pools fill. The site dies. This is Amdahl's law wearing a payment form.

Real systems escape with a state machine that separates GRABBING from OWNING:

                 reserve (fast, atomic)
   AVAILABLE -------------------------> HELD
                                          | verify ok -> CONFIRMED (sold)
                                          | verify fail / TTL expiry
                                          v
                                       RELEASED -> AVAILABLE again

Reserving a hold uses the same atomic trick as before, just counting holds instead of sales:

UPDATE tickets SET held = held + 1
WHERE id = 1 AND sold + held < total_limit
RETURNING id;

Milliseconds, no payment anywhere near the hot path. The customer gets a reservation id and a deadline. Payment gets verified outside the counter: a redirect back from the provider, a webhook, whatever the flow calls for. The state machine does not care who pulls the trigger.

But holds pile up from people who close the tab. Someone has to take the tickets back. Enter the sweeper:

WITH expired AS (
    UPDATE purchases SET status = 'released'
    WHERE status = 'reserved' AND expires_at < now()
    RETURNING id
)
UPDATE tickets
SET held = held - (SELECT count(*) FROM expired)
WHERE id = 1 AND held > 0;

One statement, one transaction, crash safe. Run it opportunistically inside reserve requests plus once per second in the background, and expired holds flow back into availability automatically. Even eight workers sweeping simultaneously cannot corrupt anything: the release only matches rows still sitting in reserved, so any hold transitions to released exactly once, no matter who fires first.

To be clear, that snippet is demonstration code for a single inventory bucket, one tickets row. Production schemas go further: each reservation references the exact inventory it holds, and many skip the separate held counter entirely, deriving availability from live reservations instead. That removes this entire class of bookkeeping from the equation.

Confirming payment is conditional too: WHERE status = 'reserved' AND expires_at > now(). Lose the race against the sweeper and the customer gets a polite "hold expired". One disappointed human. Never two tickets. That trade is the whole philosophy: correctness over comfort, always.

The expiry path proves itself quickly: set a short TTL, reserve a hold, watch /status show held go up, let the deadline pass, watch held drain back to zero while the ledger flips the row from reserved to released. Try paying anyway and you get HTTP 410, which is not an error case but the refund trigger, because money may already have moved. Inventory became self-cleaning.

Choosing the TTL is a business decision

How long should a hold live? Too short and customers lose their ticket mid checkout while typing card details. Rage, support tickets, lost sales. Too long and abandoned tabs lock inventory from real buyers: the event looks sold out when it is not.


TTL too short	customers lose tickets MID-CHECKOUT, rage ensues
TTL too long	dead tabs hold all 200 tickets, event looks sold out

There is no perfect number. Rule of thumb: measure how long your payment flow actually takes, set the TTL slightly above its p95, and for the love of users show a countdown timer. A silent expiry feels like theft. A visible countdown creates urgency. Production systems go further: renewing holds while the user is active on the payment page, grace windows for payments that provably succeeded seconds late, and automatic refunds whenever that "hold expired" response fires.

What the TTL must never be allowed to break is correctness. An expiring hold can cost a sale. It can never create two tickets.

Build it yourself

The whole thing lives as runnable code, one lesson per branch, each with its own README explaining what breaks and why:

Branch	Lesson
`step-1-theory`	the overselling problem, no code yet
`step-2-in-memory`	the naive counter, four workers four truths
`step-3-await-demo`	watching a race condition happen live
`step-4-naive-db`	moving to Postgres, still overselling
`step-5-atomic`	the atomic conditional UPDATE
`step-6-idempotency`	retries, keys, the purchases ledger
`step-7-load-tests`	k6 proof, percentiles, sold-out cache
`step-8-reservation`	holds, TTL sweeper, payments off the hot path