DEV Community: Muhammad Sumon Molla Selim

Designing a URL Shortener in AWS (Part 3): Writes, Clicks, and the Slow Path

Muhammad Sumon Molla Selim — Sun, 04 Oct 2026 00:59:17 +0000

TL;DR: The write path does exactly three things: validate the request without touching the network, mint a code, and write the link and its optional idempotency record in one DynamoDB transaction. A retry with the same Idempotency-Key returns the original link; the same key with a different body returns a conflict. The handler never fetches, resolves or previews the destination, which is the entire defence against being used to reach internal addresses.

Clicks are written as items to a DynamoDB Clicks table after the redirect is written, and the table's stream feeds an aggregation Lambda that folds a batch into one write per code on a separate stats table. The first deployed version used Kinesis. At hobby scale the idle stream and its VPC interface endpoint were most of the bill, so four ways of moving clicks were priced against each other and the one with no fixed cost won. The redirect never writes a counter. Counts are stated to be approximate: edge hits never reach Lambda, and at-least-once delivery can add a retry.

The public API has no accounts in this release, so abuse control is two independent layers: Cloudflare rate-limits link creation per IP, and API Gateway caps aggregate throughput. Takedown is an IAM-authenticated CLI, never an API key.

The whole codebase is split into a provider-neutral core, small port interfaces, and per-provider adapters. Every package tests against in-memory fakes with no cloud account, no network and no emulator, which is how the design was checked before it was ever deployed.

This is the last part of a three-part series. Part 1 covers the architecture and the edge. Part 2 covers how the cache stays correct.

The create path

What happens inside POST /api/v1/links?

The request is a JSON body with one required field, long_url, and two optional ones, alias and expires_in. The handler does the following, in order:

Bound the body at 16 KiB before decoding. A URL is at most 2,048 characters, so a larger body is never legitimate.
Validate the URL without a network call. It must parse, use http or https, have a non-empty host, carry no userinfo and no control characters, and fit in 2,048 characters. Anything else is 400 INVALID_URL. The scheme allowlist is what keeps javascript:, data: and file: out of a Location header.
Validate the alias, if given: 3 to 32 characters from letters, digits, hyphen and underscore, and not api or app, which are routing prefixes at the edge.
Validate the lifetime: 60 seconds to 5 years, with 5 years as the default.
Validate the idempotency key, if the header is present: 1 to 128 visible ASCII characters.
Mint the code: the alias as given, or a leased integer permuted and encoded as described in Part 1.
Write one transaction and map its outcome to a response.

The handler never fetches the target URL. No liveness check, no HEAD request, no preview. A service that never makes an outbound request to a user-supplied URL cannot be tricked into fetching the instance metadata endpoint or an internal address on the caller's behalf, because it never fetches anything at all. Malicious destinations are therefore an asynchronous, post-creation concern (see Abuse below), not a write-path gate. This is also why the API function needs no VPC and no NAT gateway.

Challenge 1: one transaction, three outcomes

How does a retry get the same answer as the original?

Clients retry. A mobile app that times out on create will send the same request again, and without help it will create a second link. The standard answer is an Idempotency-Key header, and the interesting part is what it is scoped to and what it is compared against.

I compared three designs.

Option	Problem
Store the key in the link item and query by it	Needs a secondary index on every link for a field most links do not have, and a race between two concurrent requests with the same key can still create two links.
Write the idempotency record first, then the link	If the process dies between the two writes, the key points at a link that does not exist, and every retry returns a 404 for a link the client thinks it created.
One transaction over both items (chosen)	`TransactWriteItems` puts the link on the condition that the code is absent, and the idempotency record on the condition that the key is absent. Both succeed or neither does. There is no state in which the key exists without the link.

The idempotency record holds the short code, a 24-hour TTL, and a hash of the request. The hash is SHA-256 over the validated {long_url, alias, expires_in} struct with defaults filled in, never over the raw JSON bytes, so whitespace and key order do not matter but a changed body does.

A cancelled transaction has to be classified, and a guess is not good enough:

No idempotency key on the request: the only condition that can fail is the link's. For an alias that is 409 ALIAS_TAKEN. For a generated code it is a collision with an alias someone chose earlier, so the handler takes the next integer from its block and tries again, up to 32 times.
Key given, record found with the same hash: a retry. Read the stored link and return the original 201.
Key given, record found with a different hash: the same key reused for a different request. 409 IDEMPOTENCY_CONFLICT.
Key given, no record: the idempotency put succeeded in some earlier attempt that we never heard about? No: if it had, the record would be there. So the link put is what failed, and the outcome is a collision, handled as above.

The read after the cancel is strongly consistent. An eventually consistent read could miss a record written milliseconds earlier and turn a legitimate retry into a false collision.

In the current release the API is public and has no accounts, so the idempotency scope is effectively global. The record key is built as {owner_id}#{key} with an empty owner, so that when authenticated ownership returns, the scope narrows to one caller without a data migration.

Challenge 2: clicks off the hot path

How do you count 10 billion clicks a month without a counter on the hot row?

Part 1 explained why the click counter is not on the link item. Something has to carry each click from the redirect to a batch writer, and the choice of that something turned out to be the most expensive decision in the deployed system, in the least expected direction.

The first deployed version used the textbook shape: the redirect published each click to a Kinesis Data Streams stream, an aggregation Lambda consumed it, and Firehose archived the raw events to S3. It worked. Then the first month's bill arrived, and the Kinesis line was larger than every other line combined. Not because of traffic, which rounded to zero, but because an on-demand stream bills about $0.04 an hour for existing, and the redirect Lambda, being inside a VPC, needed a Kinesis interface endpoint at about $7.30 per AZ-month just to reach it. About $36 a month of fixed cost to carry a few hundred clicks.

That prompted a proper comparison. Four ways to move a click, priced at the two scales that matter: the deployed hobby scale, where volume is near zero and fixed cost is everything, and the design target of 5 billion clicks a month reaching Lambda, where cost per event is everything.

Option	Fixed cost per month	Cost at 5B clicks a month	What you get, and what you give up
Kinesis Data Streams plus Firehose archive	~$36 (on-demand stream hours plus one interface endpoint)	~$250	Ordered, multi-consumer, built for this volume. Partial-batch failure handling in the event source mapping. Needs an interface endpoint from a VPC function, and bills for existing.
SQS standard	~$7 (interface endpoint; SQS has no gateway endpoint)	~$2,000 (one request per click plus receives)	Simple and near-free per message, but a VPC function still needs an interface endpoint. One consumer only, so no archive.
DynamoDB `Clicks` table plus DynamoDB Streams	$0 (gateway endpoint is free; Lambda reads from Streams are free)	~$6,250 (one on-demand write per click)	Keeps the batch aggregator unchanged. The items are the archive, expiring by TTL. Highest cost per click by far.
Direct `UpdateItem ADD` on `LinkStats` from the redirect	$0	~$6,250 (one write per click)	No pipeline at all. But a per-click write on the hot path, no batching to smooth a viral spike onto one item, no raw events, and no way to add a second consumer later.

I chose the DynamoDB table with Streams. At the scale the system actually runs, the two zero-fixed-cost options cost nothing, and the table wins between them because it keeps everything the aggregator was built to do: batching per code, partial-batch retries, bisecting on error, and the on-failure queue. The raw items also replace the S3 archive, queryable for 90 days and then gone by TTL. The whole change was one adapter, one Lambda entrypoint, and one Terraform module, which is what the port structure described in Challenge 4 is for.

The trade is honest and it is written into the design document: at the design target, one write per click costs about 25 times what Kinesis would. If real traffic ever approaches the envelope, the Kinesis shape is the right one again, and switching back is the same one-adapter change. A design that is right for 5 billion clicks and wrong for 500 is not the right design to deploy for 500.

The details that matter:

The item is small and private. {event_id, short_code, occurred_at, source_ip_hash, ttl}. The source address is HMAC-SHA-256 hashed with a key that changes daily, before it leaves the function, so the same visitor can be counted within a day but never linked across days. No user agent, no referrer.
The write has a 500 ms budget and happens after the 302. A warm call through the VPC endpoint takes about 20 ms. A cold container's first call, TLS handshake included, was observed to exceed 100 ms and drop the click, which is why the budget is 500 ms rather than 100. The redirect decision is already made and written by then; a failed write is logged and lost, never retried inside the request, and never turned into an error for the user.
The stream carries inserts only. The event source mapping filters on eventName = INSERT, so the TTL sweep's REMOVE records, which arrive on the same stream 90 days later, never invoke the function or count anything. The function checks the event name again anyway.
The aggregator folds a batch into one write per code. It reads up to 100 records or 5 seconds' worth, groups them by the short_code in each item's image, and issues UpdateItem ADD clicks :n SET last_click_at once per code. A million clicks on one link in five minutes become a few hundred writes. It returns partial-batch failures so only the failed records are retried, with two retries, bisecting the batch on error, and an encrypted SQS on-failure queue that alarms on a single message.
The event ID is the partition key of the Clicks table, so writes spread evenly and a viral code cannot make one hot partition. Nothing reads the table by code; the stream is the only reader.

Counts are approximate, and the API says so. CloudFront hits never invoke Lambda, so popular links are undercounted. Streams delivery to Lambda is at-least-once, so a retried batch can count a click twice. A per-click deduplication check would double the writes and undo the point of the pipeline. The stats endpoint promises an eventually consistent operational estimate, not a billing-grade count. Reconciling against CloudFront logs is follow-up work.

Challenge 3: abuse controls without accounts

How do you protect a public API that has no callers to identify?

The first design had API keys: a credentials table, hashed tokens, per-owner quotas through API Gateway usage plans, and an owner check on stats and delete. The scaffolding for that is still in the code. The deployed release removed the requirement, because a shortener that needs a key before it will shorten a URL is not useful to a visitor with a link to share. That decision moves every abuse control to layers that do not need an identity.

Layer	What it controls	Why it is there
Cloudflare rate limit	`POST /api/v1/links` per client IP: a handful of requests per 10-second window, then blocked for 10 seconds	The real control on anonymous creation. The zone is on the free plan, which allows one rule with a fixed window, so it is tuned tight, and a blocked request never reaches AWS at all.
Cloudflare WAF	Managed rules on every request	Drops known-bad traffic at the edge before it is billed anywhere else.
API Gateway stage throttle	Total requests per second and burst across all callers	A blunt aggregate ceiling that protects the Lambda concurrency pool and the DynamoDB counter. It is not per-IP.
Strict input validation	Code charset and length on every redirect, body size and URL rules on every create	Bounds what probing and junk can cost before any storage access.
Operator takedown	Soft delete plus tombstone, by an IAM-authenticated CLI with a mandatory reason	The response to a malicious destination. Identity comes from STS, never from a flag or a key.

What is deliberately missing is automated reputation checking. The service does not look up destinations against a blocklist, and it does not fetch them. Both would be asynchronous consumers of a creation event, outside the write path, and both are follow-up work. The first release provides the rate controls, the audit trail and a tested takedown, and states that.

The stats endpoint has one small rule that is easy to miss. A taken-down link returns 404 from stats, not its click history. Confirming that a deleted code once existed, and how popular it was, is information a taken-down link should not give away.

Challenge 4: a codebase that can be tested without a cloud

How was this checked before it was deployed?

The seat reservation system had a launch date, months of load tests, and a mock payment gateway. This project had none of those, and one person. The strategy that replaced them is structural: make every piece of logic testable with no cloud account, and make the cloud-specific parts as thin as possible.

The code is split into three kinds of package:

core holds the pure logic: Base62 encoding, the keyed permutation, validation, click aggregation, and the redirect decision. It imports nothing outside the standard library.
platform declares small interfaces, called ports: the link store, the stats store, the cache, the cache invalidator, the event publisher, the ID allocator, the audit sink, and a clock. Nothing here talks to a cloud.
handlers implements the HTTP contract as plain net/http handlers that depend only on the ports. They never import a Lambda type or an AWS SDK.

Everything AWS-specific lives in adapters that implement the ports, and in two thin Lambda entrypoints that wire adapters into handlers. The same handlers run in a single local process with in-memory adapters, which is what the web UI is developed against.

Each layer answers a different question:

Core tests are table-driven: known Base62 vectors, round-trips over a range of integers, the permutation's bijectivity on a sampled range plus the boundaries 0 and 62^7 - 1, every validation edge, and every redirect branch. A benchmark covers permute-plus-encode, the allocator's hot path.
Handler tests run every endpoint end to end against the in-memory fakes: create with and without an alias, create with a retried key, a conflicting key, a code collision that retries, every cache outcome from Part 2 including a failing store, and a takedown whose invalidation fails and then succeeds.
Adapter tests run against local fakes in-process: a small fake DynamoDB server that returns the same cancellation reasons the real one does, and a fake Redis server for the version script and the tombstone shape. The click store and the stream consumer are tested the same way: the item written, the TTL on it, and a batch with a malformed record, a failing code and a TTL removal in it. Nothing in the default test run needs DynamoDB Local, LocalStack or a network.
Infrastructure checks run on every pull request: formatting, validation of every module and environment root with no backend and no credentials, and a read-only Terraform plan posted as a comment on any pull request that touches the AWS directory. Apply runs only from a version tag, through an OIDC role, never from a merge.
The load envelope is two k6 scripts, one for 15,432 redirects a second and one for 154 creates a second, each with p99 and error-rate thresholds that abort the run. They run only from an operator workstation against a non-production stack. They are the envelope the design was sized for, and the runbook says so; they are not a record of a launch.
An audit of the live stack closed the gaps that only a deployment shows: a Function URL that needed a second IAM permission the documentation did not mention, a KMS key policy that had to name CloudFront before the UI bucket would serve, reserved words in a DynamoDB update expression, and the origin check that has to answer 403 on the bare CloudFront hostname before the domain is switched over.

The deployed system is sized for hobby scale, not for the design targets: no provisioned concurrency, one Redis node, on-demand Kinesis, point-in-time recovery off, and the smallest instance sizes throughout. Every module has the production sizes as documented alternatives, and the runbook lists what to raise, in what order, if traffic ever approaches the envelope.

Lessons

What would I tell someone building this next?

Decide the hot path first, and keep everything else out of it. A redirect is one cache read, one table read on a miss, and one event after the response. Anything that wants to be inside that request has to justify itself against the p99 target, and a click counter never will.
Check the assumptions of the standard answer against your runtime. Snowflake IDs are the textbook choice, and they are wrong on Lambda because Lambda has no stable worker ID. Block leasing with a permutation is less famous and fits.
Give every cache record a version, and compare it in the same atomic step as the write. A cache that can be overwritten by a late writer will eventually serve something you deleted. A tombstone is only as good as the write that protects it.
Serve stale only inside a window you can defend, and never over a positive "no". Availability during a database failure is worth 15 minutes of bounded staleness. It is not worth resurrecting a link that was taken down.
Make the delete finish in the cache. The durable write is not the end of a delete. The operation is done when every tier that could still answer "yes" has been told "no", and until then it should say so.
Never fetch what the user gave you. The simplest defence against server-side request forgery is a write path that makes no outbound requests at all. It also removes a NAT gateway from the bill.
Say what the numbers mean. Click counts here are estimates, and the stats API says so. A design that hides its accuracy bounds gets a reconciliation project later, under pressure.
Structure the code so the cloud is optional. Ports, fakes and pure functions meant every correctness property in this series was tested before an account existed. The live stack still found things, but they were configuration gaps, not logic bugs. The same structure is what made swapping the click pipeline a one-adapter change.
Price the design at the scale you will actually run, not only the scale you designed for. The textbook click pipeline was the cheapest per event and the most expensive per month. Fixed costs dominate a small system the way per-request costs dominate a large one, and the bill is the only test that catches it.

The result is a system whose design targets are 10 billion redirects a month with a p99 under 50 ms, running today at a fraction of that on the smallest sizes of every service, with the same code and the same tests.

Designing a URL Shortener in AWS (Part 2): Never Serve a Dead Link

Muhammad Sumon Molla Selim — Sun, 04 Oct 2026 00:58:41 +0000

TL;DR: The redirect path serves almost every click from cache, so the cache has to be right about two things: a new link must be visible on its first click, and a deleted link must never come back. Both come down to what a cache record holds and how it is written.

Three timestamps, not one TTL. A record is fresh for up to 24 hours, then stale for 15 minutes more, and the Redis key itself never outlives the link's own expiry. Stale means "check DynamoDB first", not "serve it".
Every write compares versions. DynamoDB increments the link's version on every state change. A Redis write is one Lua script that refuses to overwrite a record with a higher version. A slow populate can never undo a tombstone.
Tombstones carry no destination. A deleted link is cached as deleted with its version and a 24-hour lifetime. It answers 404 without touching the table.
Stale is a fallback, not a feature. A stale active record is served only when DynamoDB fails, only within the 15-minute grace, and never over a tombstone.
Takedown finishes in Redis, not in DynamoDB. The durable soft delete happens first. The operation is not reported as done until a tombstone with that version is in the cache. The edge entry ages out on its own within 5 seconds.

This is Part 2 of a three-part series. Part 1 explains the architecture and why the redirect sits behind three cache tiers. Part 3 covers link creation, click analytics, and testing.

The invariants

What exactly does "never serve a dead link" mean?

Before writing the cache, I wrote down three invariants (rules that must always be true). Everything in this post exists to keep them true.

A deleted link stays deleted. Once a delete or takedown is reported as successful, no later click is redirected to that destination, apart from a 302 already cached at the edge for at most 5 seconds.
A new link is visible at once. The first click after creation, including a custom alias created a moment after someone tried it, is served, never hidden behind a cached miss.
An expired link is not served. The link's own expires_at is checked on every path, in the cache and in the table, and never delegated to a background cleanup.

Invariant 1 is the one that a naive cache breaks most easily. Invariant 2 is the reason unknown codes are never cached. Invariant 3 is the reason DynamoDB TTL is treated as cleanup, not as a correctness gate.

The state

What lives in DynamoDB, and what lives in Redis?

DynamoDB is authoritative. Redis is disposable: it can be flushed at any time and rebuilt from the table by ordinary traffic. Neither of those statements is true unless the record formats make them true.

The Links item. One item per code, no sort key, so every redirect is a single-digit-millisecond GetItem on the exact key.

Attribute	Purpose
`short_code`	Partition key. A generated code or a custom alias.
`long_url`	The validated destination. Immutable after creation.
`is_active`	False once deleted.
`version`	Starts at 1. Incremented atomically on every state change. This is what makes cache writes safe.
`expires_at`	The link's logical end. Checked on every read.
`purge_at`	The DynamoDB TTL attribute. Equals `expires_at` for active links, and `deleted_at` plus 30 days for deleted ones.

DynamoDB deletes expired items on a best-effort schedule, sometimes days late. So purge_at is cleanup only. The read path also requires now < expires_at and is_active, every time.

The Redis record. Key url:{code}, a small JSON document:

Field	Purpose
`state`	`active` or `deleted`.
`long_url`	Present only on active records. A tombstone never carries a destination.
`version`	Copied from the DynamoDB item at the time of the write.
`expires_at`	The link's own end, copied from the item.
`fresh_until`	Serve from cache alone until this moment. At most 24 hours after populate, and never past `expires_at`.
`stale_until`	The Redis key's own TTL. 15 minutes after `fresh_until` for active records, 24 hours after `deleted_at` for tombstones.

Why three timestamps instead of one TTL? Because they answer three different questions. expires_at asks whether the link is alive. fresh_until asks whether the cache may answer alone. stale_until asks whether the cache is allowed to hold anything at all. Folding them into one number forces one of two bad trade-offs: either a short TTL that sends most clicks to the table, or a long TTL that serves a deleted link for its whole length.

The lookup

What does the redirect do with each cache outcome?

The handler first checks the code against the alias rule (3 to 32 characters, letters, digits, hyphen, underscore). Anything else is a 404 before any lookup, which bounds what probing can cost. Then it reads Redis and classifies the record against the current time:

Fresh active: return 302 with Cache-Control: public, max-age=5. This is the common case and the only path that costs one round trip.
Deleted: return 404 with no-store. No table read. A tombstone is a positive answer, not a miss.
Stale active: revalidate. Read DynamoDB and apply whatever it says.
Miss, or expired by its own timestamps: read DynamoDB.
Redis error or timeout: treat as a miss. The Redis client has a 150 ms budget and retries disabled, so a failover costs at most 150 ms before the table read starts.

After a DynamoDB read, the outcome is decided once, in one pure function, from the item's is_active, expires_at and the current time:

Active and not expired: populate Redis with an active record and return 302.
Unknown or expired: return 404 and cache nothing. This is invariant 2. A cached miss would hide an alias created a second later. Expired links are not cached either, because a tombstone for a link that simply ran out would be a 24-hour entry for nothing.
Inactive: write a tombstone and return 404. From now on the table is not consulted for this code for 24 hours.

Everything above is deterministic. The one judgement call is what to do when DynamoDB itself fails.

Failure mode 1: the database is down and the cache is old

When is it safe to serve a stale destination?

DynamoDB is very reliable, but the redirect function talks to it through a VPC endpoint with a 1-second budget, and throttles or a slow partition on a viral key are real. When the revalidation read fails, there are three options.

Option	Problem
Return 503	Every click on a link whose cache entry passed `fresh_until` in the last few minutes fails, while a perfectly good destination sits in Redis.
Serve the stale record, whatever it is	A deleted link whose tombstone was never written, or whose cache entry is from before the delete, redirects again. That breaks invariant 1.
Serve the stale record only inside a bounded window (chosen)	The stale destination is served only if the record is `active`, `now < stale_until`, and `now < expires_at`. A tombstone is always a 404. Anything else is 503 with `no-store`.

The window is the 15 minutes between fresh_until and stale_until. Within it, the worst case is that a link deleted in the last 15 minutes, whose tombstone write also failed, is served during a DynamoDB outage. That is a double failure of two independent systems inside a short window, and the alternative is failing every revalidating click for the length of the outage. I accepted it, and the handler logs a warning on every stale serve so the count is visible.

What the window never does is override a tombstone. If Redis says deleted, the answer is 404 no matter what DynamoDB is doing. The tombstone is a positive fact, and the stale path only ever fills in for a missing one.

Failure mode 2: the slow writer

Can a cache populate undo a delete?

Yes, if writes are plain SETs. The redirect function and the invalidation function both write the same key, and they are not coordinated. Consider a click and a takedown that overlap:

The redirect read the item while it was still active. The takedown ran, wrote the DynamoDB update and the Redis tombstone. Then the redirect's populate, carrying the old active record, arrived and overwrote the tombstone. The link is dead in the table and alive in the cache for the next 24 hours. Every click is a cache hit, so nothing ever asks the table again.

The fix is a version on every record and a compare on every write. DynamoDB increments version in the same conditional update that flips is_active. Both the redirect and the invalidation function copy that version into the record they write. The write itself is one Lua script:

local existing = redis.call('GET', KEYS[1])
if existing then
  local ok, decoded = pcall(cjson.decode, existing)
  if ok and type(decoded) == 'table' and decoded['version'] then
    if tonumber(decoded['version']) > tonumber(ARGV[2]) then
      return 0                                  -- stored record is newer: keep it
    end
  end
end
redis.call('SET', KEYS[1], ARGV[1], 'PX', tonumber(ARGV[3]))
return 1

Redis runs a script from start to finish before anything else, so the read and the write cannot be separated by another writer. A separate GET followed by a SET in Go would reopen exactly the race in the figure. Equal versions are allowed to overwrite, so a retried write is idempotent (safe to run more than once, with the same result), and an active populate carrying the same version as the stored active record simply refreshes its timestamps.

The rule I took from this is the same one that decided the seat reservation system's admission script: a guard and the action it guards must live in the same atomic step. A check followed by a separate write is a race, and under load every race is lost sooner or later.

Failure mode 3: the delete that only half happened

What if the table is updated but the cache is not?

A delete has two writes in two systems, and the second one can fail: the invalidation function can time out, Redis can be mid-failover, the invoke can be throttled. If the operation reports success anyway, the link is dead in DynamoDB and alive in Redis for up to 24 hours. That is invariant 1 broken by a partial failure rather than by a race.

The order is deliberate, and each step is idempotent:

Identity comes from AWS, not from a flag. The admin CLI calls GetCallerIdentity and uses the returned ARN as the actor. There is no --actor option to forge. A reason is mandatory.
The durable delete is a conditional update. is_active becomes false and version becomes old + 1, on the condition that the item is still active with the old version. A retry after a partial failure finds the item already inactive and returns the stored deletion, with its version and timestamps, instead of failing. A retry by a different actor is refused.
Invalidation is synchronous. The API role and the operator role are the only principals allowed to invoke the invalidation function, and they invoke it with RequestResponse and wait. The payload is {short_code, version, deleted_at}. Success means the tombstone with that version is in Redis. An older invalidation can never overwrite a newer record, because of the script above.
A failed invalidation is a failed delete. The operation returns 503, records an audit event with outcome invalidate_failed, and raises the alarm that security owns. The caller retries. The retry skips the state change, reissues the same tombstone, and returns success once it lands. The item is never reported deleted while a known positive cache entry may remain.
The edge takes care of itself. CloudFront caches a 302 for 5 seconds and nothing else. There is no invalidation API call, no distribution-wide purge, and no stale-while-revalidate that could extend the window.

The same service runs an owner's own delete when authenticated ownership returns. In the current release, create and stats are public, so there is no owner to authorise, and the only delete is the operator's.

Why not DAX, and why not a simpler Redis

Was all of this necessary?

Part 1 listed the reasons DAX could not replace Redis here: no explicit deleted state, no version comparison, no bounded stale window. It is worth restating them from this side, because each one maps to an invariant.

Invariant	What the cache must support	DAX	Redis with this record
A deleted link stays deleted	A positive "deleted" answer that survives late writers	No: a deleted item is just a miss, and a miss goes to the table	Tombstone plus version compare
A new link is visible at once	No negative caching of misses	Item cache only, so misses are not cached, which is fine	Unknown codes are never written
An expired link is not served	The record carries the link's own expiry and the read checks it	The item carries it, but the cache TTL is separate	`expires_at` in the record, checked on every read, and the key TTL is capped at it

A simpler Redis layout would also have worked for the first two rows, using two keys per code or a plain SET with a short TTL. The third row and Failure mode 2 are what forced the versioned single record. Once every write is one script that compares versions, the tombstone, the stale window and the takedown retry all fall out of the same mechanism.

Two configuration details finish the picture. Redis runs with in-transit and at-rest encryption and an AUTH token held in an encrypted parameter. And eviction is allowed: any record can be evicted under memory pressure, because eviction is just a miss, and a miss goes to DynamoDB. That is the opposite of the seat reservation cache, where eviction would have lost the source of truth. Here the source of truth is the table, and the cache is only ever a copy.

How it is tested

How do you convince yourself a cache is right?

The redirect decision is a pure function with a table test for every branch: active and not expired, active and expired, inactive, unknown.
The handler is tested against in-memory fakes of the store, the cache and the publisher, for every outcome in Figure 2: fresh, stale with a healthy table, stale with a failing table inside and outside the grace window, tombstone with a failing table, miss, and a Redis error.
The Redis adapter is tested against a fake Redis server in-process. The tests cover the version script directly: a lower version is rejected, an equal version is accepted, a tombstone survives a late active write, and the key TTL never exceeds the link's own expiry.
The takedown path is tested for partial failure: a delete whose invalidation fails returns a dependency error, records the failed-invalidation audit event, and succeeds on retry without a second state change.

None of these tests needs a cloud account, a network, or a running emulator. Part 3 explains why the code is structured to make that possible.

What's next

Part 3 follows a link from creation to its click count. It covers how a create request is validated without ever fetching the destination, how an idempotency key and a generated code are written in one transaction, how the click stream is kept off the redirect and aggregated in batches, what the abuse controls are for a public API with no accounts, and how the whole thing is tested in layers without a cloud account.

Designing a URL Shortener in AWS (Part 1): The Architecture

Muhammad Sumon Molla Selim — Sun, 04 Oct 2026 00:58:40 +0000

TL;DR: mol.la is an open-source URL shortener I built in Go on AWS. The design target is 100 million new links a month with a 100:1 read-to-write ratio, a p99 redirect under 50 ms at the edge, and 99.9% availability for the redirect path. The whole design follows three rules.

First, the redirect touches as little as possible. A click is answered by CloudFront if the same code was seen in the last 5 seconds, then by Redis, and only on a double miss by DynamoDB. Click counting happens after the response, never inside it.

Second, short codes come from a leased block of integers, not a hash and not a Snowflake ID. Each Lambda execution environment takes 1,000 integers from one DynamoDB counter with a single atomic ADD, then hides the sequence behind a keyed permutation before Base62 encoding. Lambda has no stable worker ID, which is the one assumption Snowflake needs.

Third, the edge is two layers, and nothing reaches the origin without proof it came through both. Cloudflare terminates TLS and runs the firewall. CloudFront routes by path and caches redirects. A shared secret header, stamped by Cloudflare and checked by a CloudFront function, closes the direct path to the distribution.

The system is deployed and serving traffic, at hobby scale, on the smallest sizes of every managed service. The numbers in this series are the design targets that the code and the load scripts were built for, not measurements from a launch event. Where a number is a model rather than a measurement, the text says so.

This is Part 1 of a three-part series, written as a design reference: the requirements, the options I compared, and why I chose what I chose. The code is on GitHub.

Part 1 (this post): the problem, the short code scheme, the read path, the edge, and the whole architecture.
Part 2: never serving a dead link, covering the cache record, versioned tombstones, stale fallback, and takedown.
Part 3: writes, clicks, and the slow path, covering idempotent creation, the click pipeline, abuse controls, and testing.

The problem

What makes a URL shortener hard?

The product is small. Someone submits a long URL and gets back a seven-character code. Anyone who opens https://mol.la/{code} is sent to the long URL. That is two endpoints and one table.

The traffic shape is what makes it hard. A shortener is written once and read many times, and the reads are not spread evenly. A link posted in a popular place gets thousands of clicks a minute for an hour and then almost none. Every one of those clicks is a request that a human is waiting on, in the middle of opening a page. The redirect must be fast, it must be right, and it must stay right after the owner deletes the link or an operator takes it down.

Written out as requirements:

Requirement	Detail
Write volume	100 million new links a month, about 39 a second on average and 154 a second at peak.
Read volume	100 reads for every write: about 3,900 redirects a second on average, 15,400 at peak.
Redirect latency	p99 under 50 ms measured at the edge.
Availability	99.9% for the redirect path, measured at CloudFront. A region-wide outage is an accepted risk in the first release.
Retention	Links live for 5 years by default, then expire. Callers can choose a shorter lifetime, down to one minute.
Custom aliases	A caller can pick a 3 to 32 character alias instead of a generated code.
Click stats	A count per link, allowed to lag by minutes. Not billing-grade.
Deletion and takedown	A deleted or taken-down link must stop redirecting quickly, and must never come back.
Safety	The service never fetches, previews or probes a destination URL.

Two of these define correctness: a deleted link must stay dead, and a new link must be visible on its first click. Part 2 is about those two. The rest of this post is about the shape of the system that serves the reads cheaply.

The shape of the load

Where do 10 billion monthly redirects get answered?

The design assumes three cache tiers between a click and DynamoDB, and it assumes a hit rate for each. Figure 1 shows the model that every capacity and cost number in this series is built on.

Three things in that picture shaped the design:

Viral links and the long tail behave differently. CloudFront caches a redirect for only 5 seconds. For a link clicked a few times a day, that is always a miss. For a link clicked a thousand times a minute, it absorbs almost everything. The 50% edge hit rate is a blend of the two, and it is the number I trust least. It is also the number that matters least for correctness, because the edge cache is short by design.
The database is the last resort, not the first. With Redis in front of it, DynamoDB sees about one read in forty. A viral link is one partition key, and it cannot be sharded without changing the URL that people have already shared. The mitigation is that most of its traffic never reaches the table.
The per-request cost at the edge dominates the bill. Every one of the 10 billion requests is a CloudFront viewer request, hit or miss. That is the expected shape of a 100:1 read-heavy service, and it is why the edge layer gets so much attention below.

Challenge 1: where does the short code come from?

What is the single decision that everything else depends on?

Every link needs a code that is unique, short, hard to guess, and cheap to mint under concurrent writers. Whatever produces that code is on the write path of every request, so I chose it first.

I compared four options.

Option	How it works	Why not (or why)
A. Hash the long URL	MD5 or SHA-256 of the URL, take the first 7 characters, check for a collision, retry	Collisions become common at billions of rows, and every retry is a database round trip. Two people shortening the same URL also get the same code, which leaks who shortened what.
B. One incrementing counter	A database row holds the next integer; every create increments it	A single hot writer that every request waits on. Codes are also sequential, so anyone can enumerate every link ever created.
C. Snowflake-style IDs	Timestamp, worker ID and a sequence number, packed into an integer and Base62-encoded	Needs a stable, long-lived worker ID per generator. AWS Lambda recycles execution environments on its own schedule and gives them no durable identity. Two environments that mint the same worker ID can mint the same code.
D. Leased blocks of integers, permuted	One DynamoDB counter item, advanced 1,000 at a time with an atomic `ADD`. Each execution environment hands out integers from its block in memory. A keyed permutation hides the sequence before Base62 encoding.	One write to the counter per 1,000 codes, about 0.15 writes a second at peak. No coordination between environments and no worker identity needed. A wasted partial block after a cold start costs nothing but ID space, of which there are 3.5 trillion.

I chose D. The deciding argument was Lambda itself. Snowflake is the usual production recommendation, and it is a good one on a fleet of long-lived servers. On a fleet of ephemeral functions it reintroduces the collision risk it was designed to remove. Block leasing needs only one atomic increment, which DynamoDB's UpdateItem ADD provides without a read-modify-write race.

Three details make this safe and unguessable:

The permutation is a six-round Feistel network over 42 bits, using HMAC-SHA-256 from the Go standard library as the round function, with the key held in an encrypted parameter. Values that land outside the Base62 domain are cycle-walked (permuted again until they land inside). The mapping is a bijection, so two different integers can never produce the same code, and the inverse exists for anyone holding the key.
Seven Base62 characters give 62^7, about 3.5 trillion codes. At 1.2 billion links a year the space lasts about 2,900 years. Base62 rather than Base64 because + and / are not safe in a URL without escaping.
A generated code is written with the same conditional put as a custom alias. If a caller once chose the alias aB3xK9c by hand, the generated code that happens to equal it fails its condition and the allocator takes the next integer. That retry is bounded to the handful of aliases that ever collide, not a search.

The raw integer also reserves 3 high bits for a region ID, so a second region could mint codes without a cross-region coordinator. The first release uses region 0.

Challenge 2: keep the redirect short

What exactly happens inside a click?

The first design I sketched did the obvious thing: look the code up, increment a click counter on the same row, return the redirect. Under a burst, the counter increment makes the hottest row in the table also the most-written row in the table. That is the classic failure of this kind of system, and it is the one the reference literature warns about most loudly.

So the redirect is split into a hot path (what the click waits for) and a slow path (what happens after the response is sent).

On the hot path, GET /{code} does at most three things:

Read Redis for url:{code}. On a fresh active record, that is the whole lookup.
Read DynamoDB with a consistent GetItem, only on a miss or when the cached record is stale. The result populates Redis.
Write one click item to a separate Clicks table, after the 302 has been written, with a 500 ms budget. If the write fails, the click is logged and lost. The redirect is never delayed or changed by it.

The link item is deliberately never written on a click. An aggregation Lambda reads the click items from the table's stream in batches and increments a separate stats table a few seconds later. Part 3 covers that path, and the four ways of moving clicks that were compared before settling on it.

Why 302 and not 301. A 301 tells the browser the redirect is permanent, and browsers cache it indefinitely. That kills click counting, and it makes expiry, deletion and takedown impossible for anyone who has already clicked. A 302 brings the client back on almost every click. The cost of that round trip is paid by the cache tiers, not by weakening the redirect.

Caching a 302 anyway. The redirect carries Cache-Control: public, max-age=5. That looks like a contradiction, but it is not. A 302 tells the browser not to assume the destination is permanent. It does not forbid an intermediary from caching it for exactly as long as the header says. Five seconds at the edge absorbs a viral spike without every click reaching Lambda, and it bounds how long a deleted link can be served after takedown to five seconds. Not-found and error responses carry no-store, so a custom alias created a second after a miss is visible at once.

Challenge 3: three cache tiers, and why Redis over DAX

Why not the managed cache that sits in front of DynamoDB?

DynamoDB Accelerator (DAX) is the obvious cache for a DynamoDB-backed read path. It was rejected because the redirect needs more than a cached row. Part 2 explains the cache record in full, but the short version is:

Need	DAX	Redis
An explicit "deleted" state that carries no destination	No. DAX caches items; a deleted item is a miss, and a miss goes back to the table.	Yes. A tombstone record with its own lifetime.
Version comparison on write, so an old populate cannot overwrite a newer tombstone	No	Yes, in one atomic Lua script.
A bounded stale window for use only when DynamoDB is failing	No	Yes. `fresh_until` and `stale_until` are part of the record.
Provider neutrality	AWS only	Any Redis. The same adapter runs against a local Redis in development.

So the tiers are:

Tier 1, CloudFront. Under 5 ms. Caches a successful 302 for 5 seconds and nothing else. No stale-while-revalidate, because that would extend takedown propagation past the 5-second window.
Tier 2, ElastiCache for Redis. Under 10 ms. Holds active records and deletion tombstones. Sized for the hot 20% of the daily working set, about 1 GB at the design targets. The Redis client has 150 ms dial, read and write timeouts and retries disabled, so a Redis failover degrades to a DynamoDB read instead of a hung invocation.
Tier 3, DynamoDB. Only on a double miss or a revalidation. Reads are strongly consistent, so a newly created link is visible on its first click rather than after DynamoDB's eventual-consistency window.

Unknown codes are cached nowhere. Negative caching was rejected because any valid alias can be created after an earlier miss, and a cached "missing" sentinel would hide it. A Bloom filter was rejected too: at 6 billion codes and a 1% false-positive rate it needs roughly a 7 GB bitset plus a correctness-sensitive update pipeline, to save reads that strict code validation and the edge rate limits already bound.

Challenge 4: the edge

Why two edge providers, and how does nothing get around them?

The redirect endpoint is public and unauthenticated, so it needs a source rate control that costs less than the abuse it stops. The original design put an AWS WAF web ACL on the CloudFront distribution. At the design volume that is 10 billion requests evaluated a month, about $6,000 in WAF request charges alone. For a public shortener that is the wrong trade.

The deployed design puts Cloudflare in front of CloudFront:

Cloudflare holds the DNS zone, terminates TLS in Full (strict) mode against the ACM certificate on CloudFront, runs the firewall, and rate-limits POST /api/v1/links per client IP. It also sets the real client IP in CF-Connecting-IP on every request, which the origin can trust because Cloudflare overwrites any value the client sends.
CloudFront routes by path. /api/* goes to API Gateway with caching disabled. /app/* and the root path go to a private S3 bucket through Origin Access Control and serve the static single-page app. Everything else, which is any bare single-segment path, goes to the redirect origin with the 5-second cache policy. The page and the API share one hostname, so the browser never sends a CORS preflight.
The redirect origin is a Lambda Function URL, not API Gateway. API Gateway would add about $1 per million requests on the read path, roughly $5,000 a month at 5 billion origin requests, for routing and usage plans that the redirect does not use. The Function URL costs nothing beyond the Lambda invocation. It uses IAM authorization, and CloudFront signs every origin request with Origin Access Control, so CloudFront is the only caller that can reach it.

A second edge layer creates a new hole: the *.cloudfront.net hostname is reachable by anyone, and a request sent straight to it never passes Cloudflare's firewall. Figure 4 shows how that hole is closed.

A Cloudflare Transform Rule stamps X-Origin-Verify with a secret on every forwarded request. A CloudFront function runs on every viewer request, compares the header, returns 403 with no-store on a mismatch, and deletes the header before forwarding. The secret is generated by Terraform on the first apply and stored in the state. Both the rule and the function update in one plan, so rotating it is one apply and a few seconds of 403s while the function propagates.

Two more edge details are worth knowing. CloudFront's response headers policy sends a strict Content Security Policy as a header, because browsers ignore frame-ancestors in a meta tag, and it strips the S3 encryption headers that would otherwise reveal the AWS account ID and KMS key ARN to every viewer. And Cloudflare's bot protections are pinned off in Terraform, because the JavaScript detections inject an inline script that the site's CSP blocks.

Challenge 5: compute for a serverless read path

Why Lambda, and what does a VPC-attached function cost?

I compared three compute options for the redirect.

Option	Why not (or why)
A fixed fleet behind a load balancer	Predictable and fast, but it means paying for idle capacity all month for a service whose traffic arrives in spikes, plus the operations of a fleet: patching, scaling policies, and a load balancer with its own hourly charge.
Containers on Fargate	Scales, but slower than Lambda to react to a viral spike, and the load balancer cost is the same.
Lambda (chosen)	Scales to a spike in seconds, costs nothing at idle, and the Go binary on arm64 starts in about 100 to 300 ms cold. The p99 target rules out cold starts during ramps, so the redirect function is built to run with provisioned concurrency, sized from observed concurrency (peak QPS times p50 duration, about 160 at the design targets).

Reaching Redis puts the redirect function inside a VPC, in two private subnets across two Availability Zones. The write function stays outside the VPC, because its normal request path never touches Redis. Consequences:

No NAT gateway and no interface endpoints. No function calls the public internet. Every dependency the redirect function has outside the VPC is DynamoDB, reached through a gateway VPC endpoint, which has no hourly charge. That was not true of the first deployed version, which published clicks to Kinesis through an interface endpoint at about $7.30 per AZ-month. Part 3 explains the change.
One connection per execution environment. The Redis and SDK clients are created once at cold start and reused across invocations. Provisioned concurrency keeps a warm pool of them.
Two roles, two functions. The redirect function can read links, write click items and reach Redis. It cannot create or delete anything. The API function can create links and invoke cache invalidation. It cannot reach Redis directly or write clicks.

Every function is Go, compiled for provided.al2023 on arm64, at 128 MB. The redirect function has a 3-second timeout, and every dependency call inside it has a shorter budget: 150 ms for Redis, 1 second for the DynamoDB link read including the SDK's three attempts, 500 ms for the click write. A hung dependency therefore leaves room to fall back or fail fast, never to time out the whole invocation.

The whole architecture

What does it look like end to end?

A click moves through it like this:

The browser resolves mol.la to Cloudflare, which terminates TLS, runs the firewall, stamps the origin secret, and forwards to CloudFront.
A CloudFront function checks the secret and strips it. On a cache hit, CloudFront answers the 302 itself.
On a miss, CloudFront calls the redirect Lambda through its Function URL, signed with Origin Access Control.
The function reads Redis. A fresh active record answers the click.
On a Redis miss or a stale record, it reads DynamoDB through the gateway endpoint and populates Redis.
After writing the 302, it puts one item into the Clicks table through the same gateway endpoint.

A link is created like this:

POST /api/v1/links goes through CloudFront to API Gateway and the API Lambda, outside the VPC.
The function validates the request, leases an integer from the Counters item if no alias was given, and writes the Links item and the optional Idempotency item in one DynamoDB transaction.

And the slow path:

An aggregation Lambda consumes the Clicks table's DynamoDB Stream in batches and increments LinkStats once per code per batch. Raw click items expire from the table after 90 days.
An operator takedown soft-deletes the item and invokes the invalidation Lambda, which writes a versioned tombstone to Redis. The cached 302 at the edge ages out on its own within 5 seconds.
GitHub Actions deploys by assuming an IAM role through OIDC, on a version tag only. There are no long-lived AWS keys anywhere in the pipeline.

Alarms cover the things that should wake someone up: CloudFront 5xx rate and origin p99 latency against the 50 ms target, API Gateway 5xx and p99 latency, Lambda errors and throttles on every function, Redis engine CPU and evictions, DynamoDB throttles on every table, the aggregation function's iterator age, its dead-letter queue, and any failed cache invalidation, which is owned by security rather than platform because a missing tombstone means a taken-down link is still live. Alarms go to an SNS topic and from there to email. A budget with a monthly limit sits next to them.

What it would cost at the target

What does a 100:1 read-heavy service cost at 10 billion reads a month?

These are list-price arithmetic at the design targets, not a bill. The deployed system runs at a tiny fraction of this.

Line item	Basis	Approx. per month
CloudFront	10B requests, about 5 TB egress	~$7,900
Lambda	~5B redirect invocations plus 100M writes, 128 MB, arm64	~$1,100
Lambda provisioned concurrency	~160 concurrent, redirect function only	~$450
ElastiCache for Redis	cache.r7g.large, primary plus replica	~$400
API Gateway (write path only)	100M requests	~$350
DynamoDB, links	100M writes plus ~250M on-demand reads	~$200
DynamoDB, clicks	5B on-demand writes, one per redirect that reaches Lambda	~$6,250
CloudWatch, Function URL, gateway endpoint		~$100
Cloudflare	Free plan	$0
Total		~$16,750

CloudFront request pricing dominates because every redirect is a viewer request whether it hits or misses. Two lines moved since the original design. AWS WAF at $0.60 per million requests, about $6,000, disappeared when the firewall moved to Cloudflare. And the click line grew from about $250 for Kinesis to about $6,250 for one DynamoDB write per click, because the deployed system chose the pipeline with no fixed monthly cost over the one with the lowest cost per event. At hobby scale that is the right trade by a wide margin, and Part 3 shows the arithmetic in both directions.

What's next

Part 2 looks inside the cache. It covers what a cache record holds and why it has three timestamps, how a versioned tombstone survives a late writer, when a stale record may be served and when it must not, and how a takedown propagates through DynamoDB, Redis and the edge in that order.

Scholia: an AI study partner that answers from course knowledgebase

Muhammad Sumon Molla Selim — Sat, 03 Oct 2026 21:20:01 +0000

This is a submission for the Sanity Challenge: Path One, Ship an Agent That Queries Real Content.

What I Built

Scholia is a study partner for one course. You ask it something, and it answers from that course's own material: the lecture recordings, the slide decks, the textbook, the problem sets and their rubrics, and your own graded work. Every claim comes with a chip you can click. A lecture chip opens the YouTube recording at that second. A slide chip names the deck and slide number. A book chip names the chapter and page.

It has five modes, and each one only works because the content is structured:

Mode	What you get	What in the schema makes it possible
Study	An explanation in the course's own terms, with where to see it again	`lecture.segments[]` carry `startSec`; `slideDeck.slides[]` carry `number`; `bookChapter.sections[]` carry `pageStart`
Assignment	Guidance through a problem set, step by step, under its rubric, without the solution	`assignment.rubric[]` criteria each reference `topics[]`, and assignments reference `relatedLectures[]`
Improve	Where you are losing marks and exactly what to rewatch	`submission.feedback[].criterionKey` points at a rubric criterion, which points at topics, which are referenced by segments, slides and sections
Revise	A revision sheet per topic, every bullet cited	topic references on every segment, slide and book section
Mock exam	Questions one at a time, sampled by exam weight, graded out of 10 with citations	`examScope.topics[]` carry a `weight` and point at topics and learning objectives

When the course does not cover something, Scholia searches a short allowlist of trusted sites (docs.python.org, ocw.mit.edu, Wikipedia and a few universities). Those results never mix with course citations. They appear in an amber "Outside course material" section, and a useful one can be saved back to Sanity as a webReference for course staff to review.

There are three demo courses, all from MIT OpenCourseWare (CC BY-NC-SA 4.0), each paired with an openly licensed textbook:

6.0001 Introduction to Computer Science and Programming in Python, with Allen Downey's Think Python 2e (CC BY-NC 3.0)
6.0002 Introduction to Computational Thinking and Data Science, with Allen Downey's Think Stats 2e (CC BY-NC-SA 4.0)
6.006 Introduction to Algorithms, with Pat Morin's Open Data Structures (CC BY 2.5 Canada)

The graded submissions and their grader feedback are fictional, written so Improve mode has something to work with. The examples below use 6.0001.

What it answers that keyword search could not

"Where am I losing marks, and what should I rewatch?" None of those words appears in the course material. To answer, the agent has to walk a chain: each piece of grader feedback names a rubric criterion by key, each criterion is tagged with topics, and each topic is referenced by specific lecture segments, slides and book sections. Scholia does the walk in GROQ, deterministically, then asks for the exact locators:

Recursion (lost 3 points, 1 major). Base case handling in recursive permutations. Rewatch: understanding base cases and recursive unwinding Lecture 6 · 13:17; slides on recursive multiplication and factorial Lecture 6 · slide 9.

Dictionaries, tuples, lists and mutability (lost 2 points each). Mutating the input hand instead of cloning it. Rewatch: cloning lists to avoid aliasing side effects Lecture 5 · 35:08; list cloning with slicing Lecture 5 · slide 20.

The bars in the side panel (points lost per topic, across all four submissions) come from the same query, not from the model.

"What is the complexity of bisect_search1?" A keyword search finds slide 14 of lecture 11, and that slide says two things. It first multiplies O(log n) calls by O(n) copying per call and writes O(n log n). A few lines later it says "if we are really careful", the copied lengths halve each time, so the total is O(n). Scholia gives both figures, explains why the careful count wins, answers O(n), and cites the slide.

"What is the walrus operator, and does this course cover it?" The course predates Python 3.8. Scholia says so in one sentence, then answers under "Outside course material" from docs.python.org, shown apart from the course sources.

Demo

Live: https://scholia.mol.la (no login). Studio is embedded at https://scholia.mol.la/studio.

Try these:

Study: Why does recursion need a base case?
Improve: Where am I losing marks, and what should I rewatch?
Assignment: Help me start problem set 3 without giving me the answer.
Mock exam: Start a mock exam. Then answer the first question.
Study: What is the walrus operator? (watch it leave the course)

Code

SumonMSelim / scholia

Scholia

A study partner that only works because the course content is structured.

Scholia turns everything a course throws at a student (lecture recordings, slide decks, a textbook assignments, graded submissions) into typed Sanity documents, builds a Sanity Knowledge Base on top, and puts an agent in front of it. Every answer cites the exact lecture second, slide number or book page.

Built for the DEV x Sanity Challenge (Path One). Three demo courses from MIT OpenCourseWare (CC BY-NC-SA 4.0) each paired with an openly licensed textbook:

6.0001 Introduction to Computer Science and Programming in Python, with Think Python 2e (CC BY-NC 3.0)
6.0002 Introduction to Computational Thinking and Data Science, with Think Stats 2e (CC BY-NC-SA 4.0)
6.006 Introduction to Algorithms, with Open Data Structures (CC BY 2.5 CA)

The student's submissions and grader feedback are fictional.

Sanity project dqd1lxzm, dataset production (public)
Live: https://scholia.mol.la

Modes

Mode	What

…

View on GitHub

Next.js 16 with an embedded Sanity Studio 6, the Vercel AI SDK 7 with its MCP client, Amazon Bedrock (Nova 2 Lite), Tavily for the web fallback, and AWS CDK for the deploy. Everything, including the CDK deploy and the headless browser checks, runs in Docker.

How I Used Sanity

Content model

Eighteen schema types. The documents are course, topic, learningObjective, lecture, slideDeck, book, bookChapter, assignment, submission, examScope and webReference. The objects inside them are segment, slide, bookSection, rubricCriterion, feedbackItem, scopedTopic and sourceInfo.

Two decisions shaped everything else.

Locators live inside the documents. A lecture is one document with an array of segments. Each segment is two to four minutes of the caption track, with startSec, endSec, text and topic references. Slides and book sections work the same way with number and pageStart. The Knowledge Base can therefore summarise a whole lecture, while the exact second is still one GROQ projection away.

Topics are the joins. Segments, slides, book sections, rubric criteria and exam scope entries all reference the same twelve topic documents. Feedback references a rubric criterion by its _key. That is what turns "you mutated the input" into "rewatch lecture 5 at 35:08".

The ingest pipeline is plain TypeScript. It parses the OCW caption files (VTT) into time windows, extracts slide and book text per page from the PDFs with unpdf, merges hand-written metadata (topics, rubrics, exam weights, the fictional submissions), and writes each course's documents (77, 74 and 81) in one transaction with deterministic IDs. A course is one seed file of hand-authored structure; adding 6.0002 and 6.006 meant writing two seed files, not changing the pipeline, apart from optional settings for one book's page headers and one lecture without notes. One transaction mattered, because the documents reference each other and Sanity rejects a reference to a document that does not exist yet.

Knowledge Base and Context MCP

One Knowledge Base serves all three courses through one Context MCP endpoint. The plan I am on indexes at most 150 documents per organisation, and the three courses have 232, so the Knowledge Base gets the 106 documents that explain: courses, topics, lectures with their full transcripts, assignments with rubrics, exam scopes and books. Everything else is still one tool away: slide decks and book chapters through lookup_source, graded submissions through weak_topics, and learning objectives through exam_plan, all of which query the dataset directly. Two things keep answers inside the selected course. The agent is told to use only Knowledge Base entries about that course, and every source lookup is filtered by course in GROQ, so a citation can never point into another course.

The agent connects to a Context MCP endpoint in Knowledge Base mode, with an organisation token, through @ai-sdk/mcp. It uses the three tools the endpoint exposes:

initial_context, once per conversation, for the Knowledge Base ID and outline.
knowledge_base_search with two to four keywords.
knowledge_base_read on the best matching entries.

The Knowledge Base explains. It does not reliably give a second or a page number, so I gave the agent four tools of its own next to the MCP ones:

lookup_source runs GROQ over the typed arrays. It takes keywords plus an optional lecture number or topic slug, and returns segments, slides and book sections with ready-made cite strings such as [lecture 6 @ 4:53]. GROQ's match is prefix based, so I stem the keywords ("aliasing" becomes "alias") and rank results by how many keywords each one hits.
weak_topics joins feedback to rubric criteria to topics and totals the points lost. The model reads the totals and does not compute them.
exam_plan samples topics by exam weight, without replacement and with a seed, and attaches the learning objectives.
web_search and save_web_reference handle the fallback. Context MCP is read-only, so saving goes through @sanity/client with a write token and lands as status: pending.

Citations the UI can check

The agent writes citations in a fixed syntax: [lecture 6 @ 12:40], [slides 4 #5], [book ch.11 p.106], [assignment 3: hands], [submission ps3]. The UI parses them and resolves each one against the course's sources (video URL plus &t= seconds, slide deck link, book link). It also collects every cite string the tools returned in that turn. Any citation the model wrote that no tool returned gets an "unverified" badge. A malformed one, such as [lecture 12 @ ?], is not parsed at all and stays plain text, so it never becomes a link to the wrong place.

Deploy

A container Lambda running Next.js behind the Lambda Web Adapter, which streams responses. The Function URL requires IAM auth and only CloudFront can call it, through origin access control. A CloudFront Function returns 403 unless the Host header is scholia.mol.la, so the cloudfront.net and Lambda URLs both refuse direct access. Idle cost is close to zero.

Two things cost me time here. Function URLs created after October 2025 need lambda:InvokeFunction for CloudFront as well as lambda:InvokeFunctionUrl, otherwise every request is a 403. And with origin access control, a POST body has to arrive with its SHA-256 in x-amz-content-sha256, so the chat client hashes each request body in the browser before sending it.

Honest notes

The model is the weak link. My AWS credits only cover first-party models, so the agent runs on Amazon Nova 2 Lite. Nova Pro dropped the citation brackets, and Nova Premier had reached end of life. Early versions invented timestamps, wrote full solutions in Assignment mode, and printed all the exam questions at once. Prompting helped, but the reliable fixes are in code: cite strings come from tools and the UI checks them, Assignment mode folds any code block longer than three lines behind a "try writing this step yourself first" toggle, and points lost and exam sampling are computed in GROQ and TypeScript.
Fitting the plan cost me a conflict check. When slide decks were in the Knowledge Base, its build flagged bisect_search1 as a conflict, and Scholia still answered O(n) without saying the slide also writes O(n log n). With slides out, nothing is flagged any more. Scholia now gives both figures anyway, because lookup_source returns the whole slide, but it cites that one slide on nearly every line.
A citation can point at the right source for the wrong claim. Asked about hash collisions in 6.006, Scholia cited Open Data Structures chapter 5 for how Python's dict probes, which that book does not cover. The UI checks that a citation came from a tool, not that the source says what the sentence says.
A bug I found while writing this post. On the live site, Improve mode once spent its whole step budget on eight lookups and ended without an answer. The agent now gets ten steps, and on the last one it is told to stop looking things up and answer from what it has. Simply removing the tools on that step looked cleaner, but the Bedrock provider then also drops every earlier tool result from the request, so the answer would have lost its sources.
Slide text is lossy. Many OCW slides use two columns, and PDF text extraction interleaves them (see slide 14 above). Segments from the captions are much cleaner, which is one reason lecture timestamps are the most precise citations.

Sanity Project Details

Project ID: dqd1lxzm
Dataset: production (public)
Public query example: https://dqd1lxzm.apicdn.sanity.io/v2026-10-01/data/query/production?query=*%5B_type%3D%3D%22lecture%22%5D%7Bnumber%2Ctitle%2C%22segments%22%3Acount(segments)%7D
Studio: https://scholia.mol.la/studio

Designing a Flash-Sale Seat Reservation System in AWS (Part 3): Holds, Payments, and the Slow Path

Muhammad Sumon Molla Selim — Fri, 25 Sep 2026 00:56:13 +0000

TL;DR: After admission, a seat is held for 10 minutes while the candidate pays. The central rule for payments is one-sided: many paths can confirm a booking, but only one can release a seat. Three paths can mark a booking PAID: the browser callback, the gateway's signed webhook, and a background reconciler.

The reconciler asks the gateway for the payment status. Each of the three paths checks with the gateway, and each is idempotent (safe to run more than once). Only the reconciler releases seats. It does so in two cases: the gateway reports that the payment failed or was cancelled, or the hold deadline has passed and the gateway has not reported the payment as complete. If the gateway cannot answer, the hold is extended. A seat is never released when the outcome is uncertain. The "cancel" or "failed" status in the redirect itself is ignored, because anyone can fake it. Durable writes happen outside the request path.

An SQS FIFO queue feeds a worker that writes to DynamoDB with conditional puts (writes that succeed only if a condition holds). The worker never touches the seat counters. DynamoDB TTL and a Lambda driven by DynamoDB Streams are only a safety net for expiry. On launch day, 31.6K holds were created for 20.7K seats, and the queue never held a message older than 4 seconds. The post ends with how we tested the design, in layers and many times over, and what that testing found before launch.

This is the last part of a three-part series. Part 1 covers the architecture and the edge. Part 2 covers how seat admission stays correct.

The hold lifecycle

What states does a booking go through?

A booking has three durable (stored) states:

HELD: the seat is counted and a payment exists. The record carries a deadline 10 minutes in the future.
PAID: the gateway has confirmed the payment. The seat is permanent, and a confirmation email is sent.
RELEASED: the hold ended without payment. The seat is back in the pool, the record is deleted, and the candidate can try again.

If payment creation or the enqueue fails, the booking never reaches HELD. The API undoes the admission before it responds.

The 10-minute hold is a trade-off in both directions. A shorter hold returns unpaid seats sooner, but it punishes anyone who needs some time to approve a payment on their phone. A longer hold keeps seats locked by people who have left and will not pay. Ten minutes was comfortably longer than the gateway's checkout took in testing, and it kept resale fast. On launch day, 31.6K holds were created for 20.7K seats. About a third of the holds expired or failed, and their seats were sold again within the hour.

Challenge 1: payment signals you cannot trust

Why can't the payment redirect decide the outcome?

After paying, the gateway redirects the browser to /payment/callback?paymentID=...&status=success|failure|cancel. The obvious design is to trust that status: mark the booking paid on success, and release it on cancel. That design is broken in both directions:

An attacker controls the query string. Anyone can type status=cancel for a payment ID they have seen. That would free a seat that is still being paid for, or one that is already paid.
Redirects can be old or repeated. The browser's Back button, a second tab or a retried request can all deliver the same redirect again, sometimes after the booking has already changed state.
A redirect may never arrive. The user closes the tab, their phone loses signal, or the redirect times out. The payment completed, but nobody told us.

We considered three designs.

Option	Problem
Trust the redirect status	Easy to fake, easy to repeat, and easy to lose, as above.
Confirm every redirect with the gateway's query API before acting	Better, but the gateway's query API only reports a definite status for a completed payment. So a cancel cannot be confirmed one way or the other. Every fake redirect would also cost us a call to a rate-limited API.
Confirm-only redirects, one releaser (chosen)	A redirect can help to confirm a payment. It can never release a seat.

The three confirm paths are:

Browser callback. The API accepts the callback only for a payment ID that it issued, and only with the per-payment signature that the gateway returned when the payment was created. It also refuses to charge once the hold deadline has passed. Otherwise we could take money for a seat that the reconciler has already released. It then calls the gateway's execute API under a lock with an owner token (SET NX with a random token, and only the owner of that token can release the lock). So the payment is executed at most once, even if the redirect arrives twice. If the execute call times out, the API queries the payment once, as the gateway's own integration guide requires. It does that only to recover a completed payment.
Gateway webhook. The gateway also sends a signed notification (standard AWS SNS HTTP delivery) when a payment completes. The handler verifies the SNS signature and looks up the booking by invoice number. It still queries the gateway and compares the transaction ID before it marks the booking paid. It always answers 200 to a correctly signed message, even when it leaves the decision to the reconciler. That way SNS never builds up a flood of redeliveries.
Reconciler query. This path is covered in Challenge 2.

Every confirm path ends in the same conditional update: HELD → PAID, which succeeds only if the record is still HELD with the same payment ID. Whichever path gets there first wins, and the others do nothing. Confirmation emails are sent after that update succeeds, never before.

One race condition shaped the design early. A fast payer can finish before the worker has written the record. The hold is written to DynamoDB asynchronously, so a user who pays within seconds can come back before any HELD record exists. So at admission, the API also stores a copy of the booking record in Redis, keyed by payment ID, with a TTL slightly longer than the hold. If the callback finds no record in the database, it writes the PAID record from that copy.

Challenge 2: exactly one path releases a seat

Who decides that a hold has expired?

The reconciler is a background loop inside the API fleet. A Redis lock with an owner token makes sure that exactly one instance runs it at a time. Every 20 seconds, that instance does a sweep:

Clean up stale pending admissions from Part 2: seats whose request died before it reached the database.
Find HELD records within 6 minutes of their deadline, or past it.
Handle the most overdue records first, 8 at a time, within a 45-second time budget, so the sweep finishes before the next one starts. When many holds expire at the same time, the records that are closest to hurting a user go first.
Ask the gateway for the status of each record:
- Completed: mark it PAID and send the email. This recovers every user whose redirect never arrived.
- Failed or cancelled, or not completed after the deadline: release the seat, using the payment-guarded undo from Part 2, and delete the record.
- Still pending before the deadline: leave it alone. The user may still be paying.
- The query fails three times: extend the hold by 10 minutes, up to three times, then alert a human. We never release a seat only because we could not get an answer, because the payment may have succeeded.

This is the only status query traffic we send to the gateway on the release side. Keeping it in one rate-limited loop keeps our use of the gateway's rate limits predictable.

DynamoDB TTL is not the main timer. TTL (time to live) is set on the record's deadline attribute. But AWS deletes expired items on a best-effort schedule, and the delay can be long. During a sellout, that delay would mean seats that nobody could buy. So the reconciler expires holds itself, on time. TTL, a DynamoDB Stream, and an expiry Lambda form the safety net. The Lambda acts only on deletions made by the TTL service itself (the reconciler's own deletes are ignored), and only if the deleted record was still HELD. It then runs the same payment-guarded undo. If the reconciler already released that seat, the Lambda's undo does nothing.

One bug from QA is worth sharing. Paid records must remove the TTL attribute. One of our paths wrote PAID records with the hold's TTL still set. DynamoDB would have deleted those confirmed bookings 10 minutes later.

Challenge 3: one token, a rate-limited gateway

How do you share an auth token across a fleet?

The gateway's API calls need a short-lived token. The token grant endpoint (the endpoint that issues a new token) is rate-limited to a couple of calls per hour. If every instance fetched its own token, a deploy or a restart would use up the whole allowance in seconds and lock the entire fleet out of payments.

So the token lives in Redis, shared by the whole fleet:

One refresher at a time. An instance that finds no valid token takes a SET NX lock with a 45-second TTL. Only the lock holder calls the grant endpoint. All other instances wait for the new token to appear, while still respecting their own request deadline.
Refresh early. The token is cached until a few minutes before it expires, so a request never uses a token at the moment it expires.
The grant call outlives the request that started it. The grant call runs on its own 30-second deadline, separate from the request that triggered it. A user's 8-second timeout cannot stop a grant halfway and force another one.
Wait after a failure. A failed grant keeps the lock for 60 seconds instead of releasing it. So the fleet does not retry in a loop and use up the hourly allowance.

Challenge 4: durability off the hot path

How does the database keep up without slowing down bookings?

We considered four ways to store each booking durably.

Option	Why not (or why)
Write to DynamoDB inside the request	Adds a network write to every admission. A throttle or a slow partition at T+0 (the moment registration opens) becomes latency the user sees, or an admission that must be undone.
Kinesis Data Streams	Built for ordered, high-volume streams, but shards need capacity planning, and you must build per-record retries and dead-letter handling yourself.
SQS standard	Scales without limit, but it can deliver messages twice and out of order. The worker would have to handle all of that complexity.
SQS FIFO, high-throughput mode (chosen)	Removes duplicates on send and keeps order within each group. With a large number of message groups it has plenty of throughput. The dead-letter queue is built in.

The details that matter:

Message group = the first 4 hex characters of the candidate hash. That gives 65,536 groups. The groups spread the load, and ordering within a group keeps one candidate's messages in sequence. High-throughput FIFO mode (DeduplicationScope=messageGroup, FifoThroughputLimit=perMessageGroupId) removes the old per-queue throughput limit.
Deduplication ID = the candidate hash plus the request timestamp. A retried send is treated as the same message, but a real re-booking is not.
Visibility timeout of 60 seconds and 5 receives before a message moves to the dead-letter queue. That queue has an alarm that fires on even one message.

The worker on each instance long-polls the queue (it keeps a receive request open and waits for messages instead of asking repeatedly). It writes each record with a conditional put (the write succeeds only if the record does not exist yet). When the condition fails, the worker has to find out why, and a guess is not good enough:

PAID: the callback got there first. Skip the message.
HELD with the same payment ID: a repeated delivery of the same message. Delete the message.
HELD with a different payment ID: a stale record from an earlier attempt that was released, after which the candidate booked again. Replace it, on the condition that the old payment ID is still there. We found this case in QA, and it is why the payment ID is part of the check. Without it, the stale record hid the new payment, and the new booking could never be marked paid.
The read failed: leave the message unacknowledged and let SQS deliver it again.

The worker never touches the seat counters. Seats belong to Redis and the admission script. Records belong to the worker. When both could decrease a counter, a redelivered message could free a seat that someone still held. Giving each piece of state exactly one owner made that bug impossible.

DynamoDB itself uses on-demand capacity with one table. The partition key is the candidate hash. A GSI (global secondary index) on payment ID serves the callback's lookup. Point-in-time recovery and deletion protection are both on. In the first five minutes the worker wrote about 37,000 write units' worth of records. The table absorbed that without a single throttle, because the queue had already smoothed out the spike.

Two more stream consumers connect to the table, both with failure handling built in:

The sync Lambda copies every PAID record into a relational database for the back-office app, using upserts (insert or update in one statement).
Both Lambdas use ReportBatchItemFailures, split a batch in half on error, limit the record age, and send anything they give up on to an SQS on-failure destination. Without this, a dropped stream batch would mean a lost release or a missing paid booking.

The live status document

How do thousands of writers keep one JSON file correct?

The availability counts on the page come from status.json in S3 (see Part 1). The API and the expiry Lambda both write it. Our first version read the file, changed one field and wrote it back. With several writers at the same time, two writes overlapped, and an old write reopened a level that had just filled.

The rule that fixed it: every writer builds the whole document from Redis, where the counters and the open/closed flags live, and overwrites the file. Writers that run at the same time then arrive at the same result instead of competing over a copy. Two latches (one-time flags) keep the writes cheap:

A 2-second SET NX throttle turns a burst of admissions into a single S3 write.
The level-full and closed transitions each publish exactly once.

A release clears those latches, so a freed seat reopens the level on the page.

Test, find, fix, repeat

Was this design right the first time?

No. The design in this series is the result of about four months of testing, finding problems, fixing them and testing again. That loop is one of the main reasons launch day was quiet. Every change went through several layers of testing, and each layer answered a different question.

Unit tests for every package: each branch of the admission script, the gateway client, the webhook signature check and the status document.
An end-to-end suite on a local copy of the stack, with a mock payment gateway that can be told to fail or cancel. It runs a booking all the way to PAID. It also runs the failure cases: a gateway error, a duplicate booking, a new booking after a released hold, a forged callback, a callback that arrives twice, and a payment that completes before the worker has written the record.
Full-rate load tests on a stack the same size as production. Each run used 40,000 test candidates, sent 2,000 bookings per second for 25 seconds, and then sent a slower stream of late arrivals and retries. A run passed only if admissions stopped exactly at the cap, DynamoDB reported no throttling, and the counters matched the database afterwards.
The payment gateway's sandbox, for real payment flows: pay, cancel, let a hold expire, and book again.
Reviews and audits of the design, the code and the infrastructure. They looked for signals an attacker could fake, rate limits we could use up, and missing permissions.
Launch rehearsals. We ran the same reset, load-candidates and open steps that we would use on the day, more than a hundred times, from the same pipeline.

Each layer found problems that the others missed. A few examples:

Load tests: the Redis client's default connection pool was sized for normal traffic, not for a burst. We raised it to 300 connections per instance and ran the test again.
Sandbox: the counters drifted after a user cancelled and then booked again. The worker found the old record, treated the new booking as a duplicate, and decremented a counter. This is why the worker never touches the counters, and why it classifies every conflict with a consistent read (Challenge 4).
Testing the undo paths: a rollback that ran twice pushed a level counter below zero. This is why every undo checks and acts in one atomic step (Part 2).
Reviewing paid records: a paid record could still carry the hold's TTL, so DynamoDB would have deleted a paid booking about ten minutes later. A paid record now has no TTL.
Security review: the cancel status in the redirect has no signature, so anyone can fake it. Checking each one with the gateway would also use up the gateway's token limit. This is why a redirect can never release a seat (Challenge 1).
A week before launch: two concurrent writers could leave the status page showing a full level as open. The writers now run in a fixed order, and the page also checks the counts, so a full level can never look open.

None of these problems is unusual. Most of them only show up under a specific timing, order of events or load, and the rest only show up when someone looks for them on purpose. The only reliable way to find them before users do is to test at every layer, many times, and to test again after every fix.

Lessons

What would I tell someone building this next?

Decide the one fact everyone competes for in one atomic step, and keep it off the database. Everything else in the system can be eventually consistent if that one decision is exact.
Make the request path as short as possible. Admit, create the payment, enqueue, and respond. Every durable write that can happen after the response should happen after the response.
Give every piece of state exactly one owner. Redis owns seats, the worker owns records, and the reconciler owns releases. Most of our pre-launch bugs came from two components writing the same state.
Treat every external signal as a hint. Redirects, webhooks and query results can be faked, delayed, duplicated or missing. Confirm with the source, make every state change conditional, and let exactly one path take the action that cannot be undone.
Never release when you are not sure. If you cannot tell whether someone paid, extend the hold and alert a human. Selling a paid seat twice is worse than holding an unpaid one a little longer.
Idempotency must live in the same atomic step as the action. A check followed by a separate action is a race, and under a burst every race is lost sooner or later.
Add capacity before a sudden jump in traffic, and load-test with a mock of every external dependency. The mock gateway let us practise the full booking path at full rate as often as we wanted.
Plan as much time for testing as for building. Test in layers, from unit tests to full-rate load tests and launch rehearsals. When you find a problem, fix it, add a test for it, and run every layer again. Most of what makes this design correct was learned that way.

The result was the launch day in Part 1: 20,700 seats sold, 99.99% of requests served without a server error, and zero double bookings.

Designing a Flash-Sale Seat Reservation System in AWS (Part 2): Never Sell a Seat Twice

Muhammad Sumon Molla Selim — Fri, 25 Sep 2026 00:49:57 +0000

TL;DR: Every booking goes through one Redis Lua script. The script checks, in order, whether the candidate is already admitted, whether their level is full, and whether any seat is left overall. Then it counts the seat and marks the candidate. All of this happens as one atomic step (nothing else can run in between). Lua won over MULTI/WATCH (too many retries when many requests compete) and distributed locks (extra round trips while holding the lock). An in-memory counter can still be wrong in four ways. Each needed its own fix:

Double rollbacks: the undo step has a guard, so running it twice does nothing the second time.
Lost replies: every call carries a nonce (a random value that identifies one request), so a retried call gets its original slot back.
Crashed requests: a list of pending admissions, plus a cleanup job, returns seats that never reached the database.
Failover: a booking counts only after the Redis replica confirms it has the write (WAIT 1 200). Otherwise the booking is undone and the client retries.

This is Part 2 of a three-part series. Part 1 explains why seat admission lives in Redis rather than a database. Part 3 covers what happens after a seat is admitted.

The invariants

What exactly does "never sell a seat twice" mean?

Before writing any code, we wrote down three invariants (rules that must always be true). Everything in this post exists to keep them true.

Caps hold. The number of admitted seats per level never exceeds that level's cap, and the total never exceeds the global cap.
One live booking per candidate. A candidate code is linked to at most one seat that is held or paid.
No leaks. Every admitted seat becomes either paid or released in the end. No seat stays counted with nobody holding it.

Invariants 1 and 2 are what "zero double bookings" means. Invariant 3 stops the event from ending with unsold seats that the system forgot about.

The state

What lives in Redis?

Key	Type	Purpose
`booking:open`	string	The admin's open/close switch
`candidates:valid`	set	Allowlist of eligible candidate codes, loaded before opening
`candidate:email:<code>`	string	The email on file for each code
`adm:seen`	set	Candidate codes that currently hold a seat
`adm:total`	integer	Seats admitted overall
`adm:level:<L>`	integer	Seats admitted per level
`adm:pending`	hash	`code → nonce, level, time, slot` for admissions not yet safely in the database
`hold:*`	strings with a TTL (they expire on their own)	Per-payment stashes (temporary copies): the booking record, the payment URL, and the execute and rollback guards

Two configuration choices matter more than they seem to:

maxmemory-policy noeviction. When memory runs low, Redis must refuse writes with an error. It must never silently delete adm:seen to free space. Losing that set would allow duplicates. A write error, on the other hand, simply becomes a 503 that the client can retry.
Normalised identifiers. Candidate codes have their surrounding spaces removed and are converted to uppercase before any lookup. Otherwise the counter would treat ab123 and AB123 as two different people. A SHA-256 hash of the normalised code is the database key. Its first characters become the queue message group and the payment's invoice number.

Gates, then the script

Why check some things outside the script?

The gates are simple read checks. They reject most invalid traffic before any write happens:

Is booking open? If not, 503 closed.
Is the code on the allowlist? If not, 400 invalid_code. The allowlist is loaded into Redis from the back-office database before opening. There is deliberately no fallback to the database while the system runs. A code that is missing from the allowlist must not turn into a database query at 50,000 requests a minute.
Does the email match the code? If not, 400 email_mismatch.

A hidden honeypot field is checked before all of these. (A honeypot is a form field that real users never see, so only bots fill it in.) A form that fills it in gets a fake success response that looks real, with no admission and no log entry.

The gates can live outside the script because their values change only when an operator changes them, before the event. The checks that compete with each other at the moment of booking are all inside the script:

-- KEYS: adm:seen, adm:total, adm:pending
-- ARGV: code, max_total, level, level_cap, nonce, now_ms
local code, level, nonce = ARGV[1], ARGV[3], ARGV[5]
local level_key = 'adm:level:' .. level

-- 1. Already admitted? A retry of the same request gets its slot back.
if redis.call('SISMEMBER', KEYS[1], code) == 1 then
  local p = redis.call('HGET', KEYS[3], code)
  if p then
    local n, lvl, _, slot = string.match(p, '^([^|]*)|([^|]*)|([^|]*)|([^|]*)$')
    if n == nonce then return {tonumber(slot), lvl} end
  end
  return {-1, ''}                                  -- duplicate
end

-- 2. Level full?
local cap = tonumber(ARGV[4])
if cap > 0 and tonumber(redis.call('GET', level_key) or 0) >= cap then
  return {-3, level}                               -- level full
end

-- 3. Any seat left overall?
if tonumber(redis.call('GET', KEYS[2]) or 0) >= tonumber(ARGV[2]) then
  return {-2, ''}                                  -- closed
end

-- 4. Admit: count, mark, and record as pending.
local slot = redis.call('INCR', KEYS[2])
redis.call('INCR', level_key)
redis.call('SADD', KEYS[1], code)
redis.call('HSET', KEYS[3], code, nonce .. '|' .. level .. '|' .. ARGV[6] .. '|' .. slot)
return {slot, level}

The order matters. The duplicate check comes first, so a candidate who already has a seat is told so, even after their level fills up. The level check comes before the global check, so the user learns the most specific reason. Only the last branch writes anything. A rejection therefore costs a few reads on one thread.

The API loads the script once with SCRIPT LOAD and calls it with EVALSHA. After a restart or failover, the new primary has an empty script cache and answers NOSCRIPT. The client reloads the script and retries once.

Why not the other atomic building blocks in Redis?

Option	Problem under a flash sale
`INCR`, then check, then `DECR` if the counter went too far	The counter briefly goes above the cap, and readers can see that. The duplicate check is a separate step, and another request can run between the steps, so two requests for the same candidate can both pass.
`MULTI/WATCH` (optimistic transactions)	Every booking at the same level watches the same key. In the opening second, nearly all transactions abort and retry. That creates a storm of retries on the busiest key.
Distributed lock (for example Redlock) around a read, change, write sequence	At least two extra round trips while holding the lock, and a lock service that is one more thing that can fail.
Lua script (chosen)	Runs from start to finish on the Redis thread. There is no lock to hold across the network and no retry loop.

Two more design details keep the rest of the system simple:

"Level full" is published once. When a level fills, a SETNX latch (a key that only the first writer can set) makes exactly one API instance rewrite status.json. Thousands of requests cannot compete to publish the same change.
A duplicate can continue. Suppose a candidate's booking is still HELD and they submit again, perhaps because their first request timed out in the browser. The 409 response then carries their existing payment URL. The URL is read from a Redis stash (never from the database), and the page redirects them straight to payment. Without this, a user whose first response was lost could not reach the seat they already hold.

Failure mode 1: undo that runs twice

What happens when rollback runs twice?

A seat must be given back when a booking fails: the payment could not be created, the enqueue failed, or the hold expired without payment. Several paths can trigger that release. Early in load testing, two of them sometimes ran for the same booking, and the counters went negative. The undo step decreased the counters every time it ran, even though removing the candidate from the set only had an effect the first time.

The fix was to make undo a script that only acts if the candidate is still admitted:

if redis.call('SREM', KEYS[1], ARGV[1]) == 0 then return 0 end  -- already undone
redis.call('DECR', KEYS[2])                                     -- total
redis.call('DECR', 'adm:level:' .. ARGV[2])                     -- level
redis.call('HDEL', KEYS[3], ARGV[1])                            -- pending entry
return 1

That fixes repeats, but not a stale (out of date) undo. Suppose a candidate's first payment is released and they book again. Then a late event for the first payment arrives, such as a TTL expiry or a repeated callback. A guard that only asks "is the candidate admitted?" passes, because the candidate is admitted again. The stale event then frees the new seat.

So every release that a payment triggers goes through a second guard, keyed by payment ID. SET hold:rollback:<paymentId> NX EX 86400 and the undo run in the same script. A payment can release a seat at most once, and only the payment that owns the seat can release it.

The rule we took from this: any operation that more than one path can trigger must be idempotent, and the guard that makes it idempotent must live in the same atomic step as the action itself. (Idempotent means it is safe to run more than once, with the same result.) A check followed by a separate action is a race.

Failure mode 2: the reply that never arrives

What if the script succeeds but the API never hears about it?

Networks drop replies. When that happens, the Redis client does the sensible thing and retries the command. For the admission script, that sensible retry is a disaster.

The first call admitted the candidate. The retry finds them in adm:seen and returns "duplicate". The user sees "already registered" with no payment link, and their seat stays counted, with no hold behind it.

The fix is a nonce for each request. The API generates a random nonce for each booking request and passes it to the script. On admission, the script records it in the adm:pending hash (step 4 above). When a call finds the candidate already admitted, it compares nonces. The same nonce means this is the same request, retried, so the script returns the original slot. A different nonce means a real second attempt, so it returns duplicate.

We also run the admission with a context that is separate from the HTTP request's cancellation. If a user closes the tab during the request, that cannot stop the call halfway through its retry.

Failure mode 3: the request that dies halfway

What if the API crashes between admitting and enqueueing?

The hot path is three steps: admit, create payment, enqueue. If the process dies after the first step, the seat is counted, but no record exists anywhere that could expire it. Neither the database nor the queue knows the seat exists. That would break invariant 3.

The pending ledger closes the gap. (The ledger here is simply the list of admissions that are not yet safe in the database.) Every admission writes an adm:pending entry in the same atomic step. /book clears the entry only after the enqueue has succeeded, because from that point on the queued record carries the booking. Any rollback clears the entry atomically too.

On every sweep, the reconciler (the single background runner described in Part 3) looks for pending entries older than 15 minutes:

If a database record exists for that candidate, the booking completed. The reaper (the cleanup step) removes the entry.
If no record exists, the request died between steps. The reaper undoes the admission, with the same nonce check, so it can never undo a newer attempt.

Fifteen minutes is longer than the 10-minute hold plus the queue's worst-case delay. A slow but healthy request is never cleaned up by mistake.

Failure mode 4: the failover that forgets

Can a Redis failover sell a seat twice?

Yes, if you let it. ElastiCache replication is asynchronous. The primary confirms a write to the client before the replica has received it. If the primary dies in that gap, the replica is promoted to primary without the last few admissions.

At the last seat, that gap means one seat sold twice. The same applies to a candidate: the promoted replica does not know they booked, so they can book a second seat.

We compared three options.

Option	Trade-off
Accept the risk	The window is milliseconds long and failover is rare. But it breaks the one promise the client cared about, at exactly the moment when load is highest.
Count seats in the database too	Correct, but it puts the database back on the hot path, which is what Part 1 worked to avoid.
Wait for a replica acknowledgement (chosen)	Send the script and `WAIT 1 200` together in one pipeline: wait up to 200 ms for one replica to confirm it has the write. If the replica confirms, the admission counts. If not, undo it and return a `503` that the client can retry.

Under normal conditions, replication lag between Availability Zones is a few milliseconds, so the wait adds almost nothing to a normal booking. Both commands go out in one pipeline, so there is no extra round trip.

This choice has two consequences we accepted deliberately:

Bookings fail closed. That means: if the replica is down, every booking returns 503 until ElastiCache replaces it. We chose correctness over availability for this one path. The page retries with increasing delays, and replacing a replica takes minutes, not hours.
WAIT is not consensus. It makes a lost admission far less likely. It does not make Redis strongly consistent. With one replica, the node that confirmed the write is the node that gets promoted. That covers the realistic failure: the loss of one node or one Availability Zone.

As a final safety net, every five minutes the reconciler compares the Redis counters with a count of the records in DynamoDB. It raises an alert only if the mismatch is still there on two checks in a row, because records normally lag behind the counter by a few seconds.

How we tested it

How do you convince yourself a counter is right?

Unit tests against an in-memory Redis for every branch of the script: caps, duplicates, nonce replay, idempotent and stale undo, and the pending reaper.
Race tests that send thousands of admissions at the same time against a small cap. They check that the admitted count equals the cap exactly and that no candidate appears twice.
A real primary and replica in Docker for the replica acknowledgement path. In-memory fakes usually report zero replicas, so they cannot test it.
Full-rate load tests with 40,000 test candidates against the real stack. They pass only if admissions stop exactly at the cap and the counters match the database afterwards.

On launch day the invariants held. 20,700 seats were sold, with zero double bookings and zero overbookings.

What's next

An admitted seat is only half a booking. Part 3 follows it through payment. It covers why a payment redirect can confirm a booking but never release one, and how a single runner makes every release decision. It also covers how a rate-limited gateway token is shared across the fleet, and how the queue and DynamoDB keep up without ever slowing down a booking.

Designing a Flash-Sale Seat Reservation System in AWS (Part 1): The Architecture

Muhammad Sumon Molla Selim — Fri, 25 Sep 2026 00:46:21 +0000

TL;DR: A language proficiency test certification exam had 20,700 seats across five levels. When registration opened last year, millions of people tried to access and book. The previous system crashed under that load. It also sold the same seat twice.

To make the system scalable, highly-available and reliable for this year's registration, I re-designed the system and new design follows three rules.

First, one atomic Redis script decides who gets a seat. (Atomic means the whole check-and-count runs as one step that nothing can interrupt.) No database takes part in that decision.

Second, the request path does only three things: admit the booking in Redis, create a payment link, and put a record on a queue. Everything else happens later, in the background.

Third, everything that can be a static file is served from the edge (the CDN, close to the user). CloudFront, S3 and an AWS WAF web ACL handled about three out of four requests without touching a server.

We sized the API fleet from load tests and added servers before the event. We did not use autoscaling. In the first hour it served 1.5M+ requests at the origin, peaked at 54.8K requests per minute, kept median latency under 10 ms, returned a server error on fewer than 0.01% of requests, and sold every seat with zero double bookings.

This is Part 1 of a three-part series. It is written as a design reference: the challenges, the options we compared, and why we chose what we chose.

Part 1 (this post): the problem, the architecture, the edge, and capacity.
Part 2: never selling a seat twice, covering the admission script, retries, and failover.
Part 3: holds, payments, and the slow path, covering the hold lifecycle, payment signals we cannot trust, and durable storage that runs in the background.

The problem

What made this hard?

The work itself looks simple: a form with a candidate code, an email address and an exam level, then a payment. The traffic pattern is what makes it hard. Registration opens at an announced minute, so almost everyone arrives in the same few seconds. They keep refreshing the page until they get a seat or the seats run out. The previous system crashed at exactly that moment. It also sold some seats twice. The client's requirement was simple and direct: stay up, and never book a seat twice.

Written out as requirements:

Requirement	Detail
Hard caps	A cap (a maximum) per level, for five levels, and a global cap of 20,700. Never exceeded, not even by one seat.
One booking per candidate	A candidate code can hold at most one live booking.
Eligible candidates only	The code must be on an allowlist loaded in advance, and the email must match the one we have stored for that code.
Pay to keep the seat	A seat is held for 10 minutes while the candidate pays through the payment gateway. If the payment is not completed, the seat returns to the pool of free seats.
Money-safe	Nobody is charged for a seat they did not get, and nobody who paid loses their seat.
Available under the burst	The page must load and answer quickly, even when everyone arrives at once.
First come, first served	No lottery. Whoever is admitted first gets the seat.

"Never exceed the cap" and "one booking per candidate" are the two rules that define correctness. Part 2 is entirely about them. The rest of this post is about surviving the burst while keeping those rules true.

The shape of the load

What does a flash sale look like in the traffic numbers?

Figure 1 shows the real traffic around opening time, grouped into five-minute buckets from CloudWatch.

Three things stand out, and each one shaped the design:

The ramp is a cliff. In other words, traffic does not grow slowly; it jumps almost straight up. Origin traffic went from near zero to about 12K requests per minute in the first five minutes after opening, and to more than 50K per minute within twenty minutes. Anything that reacts to load, such as an Auto Scaling policy, a function with a cold start, or a database that scales its capacity, reacts too late. The moment that matters has already passed.
Most requests are not bookings. People reload the page and check availability long before and long after they submit the form. In the first hour CloudFront saw about 6.2M requests. Only about 1.5M of them reached the servers. Every request the edge answers is one that the booking path never has to handle.
Most of the origin traffic is rejections. Once seats run low, most /book calls end with a fast "no": duplicate, level full, or closed. A rejection must cost almost nothing, because there are far more rejections than admissions.

Challenge 1: where do we decide who gets a seat?

What is the single hardest decision in this system?

Every booking asks the same question: is there still a seat at this level, and has this person already booked one? Thousands of requests ask it at the same instant. The answer must be exactly right for every one of them. Whatever component answers that question is the bottleneck of the whole system, so we chose it first.

We considered four options.

Option	How it works	Why not (or why)
A. Relational database row lock	`SELECT ... FOR UPDATE` on a `seats_left` row, then insert the booking	Correct, but every booking must wait its turn on one row. Lock waits build up, connection pools run empty, and everything times out at the same time. This is the classic failure of older systems of this kind.
B. DynamoDB conditional counters	`UpdateItem` with `seats_left > 0` on a counter item, plus a transaction for the duplicate check	Correct and serverless, but a single counter item is a hot key (one item that every request writes to). One partition limits how many writes per second it can take. Adding a duplicate check means a transaction on every attempt, and even a rejection still costs a write.
C. Virtual waiting room	Put everyone in a queue and let people in at a fixed rate	Protects the backend, but it is a whole product to build or buy. Users wait for minutes, and you still need an atomic counter at the end of the queue.
D. One atomic script in Redis	A Lua script checks for a duplicate, checks the level cap and the global cap, and records the admission, all in one call	Redis runs a script from start to finish before it runs anything else, so no lock is held while waiting on the network. It costs one round trip of microseconds, and a rejection is only a read. The open questions, durability and failover, are answered in Part 2.

We chose D. The deciding argument was this: Redis runs commands on a single thread, and that single thread is the lock. It is held only for the microseconds a script runs, never for a network round trip. The same property makes rejections nearly free. That matters because most of the traffic is rejections.

The cost is that the source of truth for "who has a seat" now lives in memory. That creates three problems that Part 2 has to solve: a reply lost on the network, a process crash between steps, and a failover that loses the last few writes. We accepted those problems with open eyes. The other options had problems we could not solve within a single burst.

We also chose the smallest possible Redis setup: one shard (cluster mode disabled), one primary and one replica in different Availability Zones, with automatic failover. The script touches several keys at once. On one shard, that needs no extra work to keep the keys together (no hash tags). As Figure 4 shows later, a single cache.r6g.large primary never went above 10% engine CPU.

Challenge 2: keep the hot path short

What exactly happens inside a booking request?

The previous system did all its real work inside the request. It wrote the booking to the database, sent the email and updated the counters, all while the user waited. Under a burst, each of those steps becomes a waiting line, and the slowest one sets the latency for everyone.

We split the work into a hot path (the steps the user waits for) and a slow path (the steps that run after the response is sent).

On the hot path, POST /book does exactly three things:

Admit the booking in Redis with the atomic script.
Create a payment with the payment gateway and get back a payment URL.
Enqueue a HELD record, which means send it to an SQS FIFO queue.

It then returns 202 Accepted with the payment URL, and the browser redirects to the gateway. If step 2 or step 3 fails, the API undoes the admission before it responds. A failure never leaks a seat (leaves a seat counted with nobody holding it).

The database is deliberately missing from this list. A worker reads the queue and writes the records into DynamoDB a few seconds later. Payment confirmations, expiry, emails and the back-office sync all run on the slow path. Part 3 covers that path in detail, including why it is safe for the database to be a few seconds behind the seat count.

The payment gateway call is the one dependency on the hot path that we do not control. It is the only reason the hot path is not a pure in-memory operation. It is also the only slow step, and it shows in the slowest requests: p99 latency was 4.1 s in the first five minutes, when every admitted user was creating a payment at the same time. After that it settled to about 0.3 s. The median stayed at 2 to 5 ms the whole time, because most requests were fast rejections that never reached the gateway. Placing the gateway call after admission is what makes this work: rejected requests never wait for it.

Challenge 3: keep traffic off the servers

How do you make three out of four requests never reach your code?

The cheapest request is the one your servers never see. We put everything we could behind CloudFront.

The page itself is a static HTML file in a private S3 bucket, served through CloudFront with Origin Access Control. Page loads never touch a server.
Availability counts (seats left per level, and whether booking is open or closed) are a small status.json document in the same bucket. The API rewrites it as the counts change. The browser reads it once when the page loads. After that, the page reacts to API responses instead of asking the server again and again. A 429 closed disables the form, and a 409 level full disables that level. CloudFront serves the file with caching disabled, so readers always see the latest version. The load still lands on S3, not on our servers.
One origin, one hostname. CloudFront routes /book, /payment/callback, /payment/webhook and /health to the Application Load Balancer, and everything else to S3. The page and the API share one hostname, so the browser sends no CORS preflight (an extra request the browser sends first when the page and the API are on different hosts). Under a burst, skipping the preflight removes a whole extra request from every booking.

In front of all of this sits an AWS WAF web ACL attached to the CloudFront distribution. It checks every request at the edge before CloudFront forwards anything.

Rule	Purpose
Rate limit on `/book`, 2,000 requests per 5 minutes per IP	Stops a single client from flooding the booking endpoint
Rate limit on `/payment/callback`, 100 per 5 minutes per IP	Makes it impractical to guess payment IDs
AWS managed IP reputation list and known-bad-inputs rules	Drops known attackers and exploit payloads at low cost
Body size limit on `/book`	The form is tiny, so a large request body is never legitimate

The per-IP limits are deliberately generous. Mobile carriers in the region use carrier-grade NAT, which puts many subscribers behind a small number of shared public IPs. A strict per-IP limit would have blocked whole neighbourhoods of real candidates. The rate rules exist to stop abuse, not to control traffic. The admission script already rejects excess load cheaply on its own.

The load balancer accepts traffic only from CloudFront. Its security group allows CloudFront's origin-facing IP ranges and nothing else. But that list covers every CloudFront distribution in AWS. An attacker could create their own distribution, point it at our load balancer, and bypass our web ACL. So CloudFront also adds a secret header to every request it sends to the origin. The ALB listener forwards only requests that carry that header. Its default action is a fixed 403. CloudFront talks to the ALB over HTTPS, so the secret never crosses the internet in clear text.

Challenge 4: capacity for a cliff

Why pre-scale instead of autoscale?

We considered three compute options.

Option	Why not (or why)
EC2 Auto Scaling with target tracking	Scaling reacts to metrics that are a minute or more behind, then waits for new instances to boot and pass health checks. By the time new capacity arrives, the jump in Figure 1 is already over.
AWS Lambda behind API Gateway	Scales fast, but the burst hits concurrency ramp limits and cold starts at exactly the wrong moment. Every new execution environment also opens its own Redis connections. A traffic spike then becomes a flood of new connections to the one component that must not stall.
A fixed, pre-scaled fleet	Simple and predictable. Size it from load tests, add instances hours before opening, and remove them afterwards.

We pre-scaled a fixed fleet: three instances, one per Availability Zone, behind an Application Load Balancer. Each instance ran the API and the queue worker as two systemd services, with a fixed Redis connection pool (300 connections per instance, enough that the burst never waits for a connection). The instances live in private subnets with no public IPs. They reach the payment gateway and the email provider through a NAT gateway in each Availability Zone, and they reach AWS APIs through VPC endpoints. Operators use AWS Systems Manager Session Manager, not SSH.

We chose the size from load tests before launch. The setup was a k6 load generator in the same region, an allowlist pre-loaded with 40,000 test candidate codes (about double the real seat count), and a mock payment gateway with adjustable latency. The mock let us practise the full booking path at full rate without hitting the real gateway's limits. The tests also shaped several details you will see in Parts 2 and 3.

On the day, the fleet had far more capacity than it needed.

We sized it that way on purpose. The extra capacity cost a few dollars an hour for a few hours. An outage at opening would have cost the client the whole event. With these numbers, the next run could use two Graviton instances and a smaller Redis node and still use less than a third of their capacity at the peak.

The whole architecture

What does it look like end to end?

A booking moves through it like this:

The browser loads the page and status.json from CloudFront, which serves both from a private S3 bucket. The AWS WAF web ACL on the distribution checks every request.
The browser posts to /book. CloudFront forwards the request over HTTPS, with the origin secret header, to the ALB in the public subnets.
An API instance in a private subnet checks the gates (is booking open, is the code valid) and runs the admission script on ElastiCache for Redis (primary plus replica across two AZs, TLS and AUTH enabled).
If admitted, the API creates a payment with the payment gateway (outbound through the NAT gateway) and enqueues a HELD record on SQS FIFO. It returns the payment URL.
The worker on each instance long-polls the queue. (Long-polling means it keeps a request open and waits for messages instead of asking again and again.) It writes the record to DynamoDB with a conditional put. (A conditional put is a write that succeeds only if a condition is true, here that the record does not exist yet.)
The user pays at the gateway, and the gateway redirects the browser to /payment/callback. The gateway also sends a signed webhook to /payment/webhook. Either one can mark the booking PAID. The API then sends a confirmation email.
A reconciler (a background loop inside the API fleet, with only one instance running it at a time) releases holds that were not paid in time. DynamoDB TTL (time to live: DynamoDB deletes a record some time after its deadline passes) and DynamoDB Streams drive an expiry Lambda that acts as a safety net.
A second stream consumer, the sync Lambda, copies paid bookings into an RDS database for the back-office app. The back-office app also loads the candidate allowlist into Redis before opening.

CloudWatch alarms cover the things that should wake someone up: ALB 5xx errors and unhealthy hosts, Redis CPU, memory, connections and replication lag, queue depth and the dead-letter queue, DynamoDB throttles, Lambda errors and stream iterator age. Application logs go to CloudWatch Logs, so an incident never starts with an SSH session. A few custom metrics sit next to them: admissions per minute, replica-acknowledgement failures (explained in the next post), holds past their deadline, and any difference between the Redis counters and the database.

Launch day

Did it work?

Metric (first hour)	Result
Requests reaching the servers	1.5M+ (about 6.2M at the edge)
Peak throughput at the origin	54.8K requests per minute
Median latency	under 10 ms (2 to 5 ms at the ALB)
Requests served without a server error	99.99%
Seats sold	20,700 of 20,700, about 95% of them within the first hour
Double bookings or overbookings	0

The number I care about most is the last one. It did not happen by accident. It is the result of the decisions in Part 2, and of months of testing, finding problems and fixing them before launch (Part 3 describes how).

What's next

Part 2 looks inside the admission script. It covers what the script checks and in what order. It explains how the script stays correct when a reply is lost and the client library retries, and how the seat from a crashed request is returned to the pool. It also shows how we closed the short time window in which a Redis failover could silently sell the same seat twice.