DEV Community: Hokutoman00

Building on Aurora DSQL multi-region: the setup notes I wish I'd had

Hokutoman00 — Sun, 21 Jun 2026 17:46:09 +0000

I built WorldSeat — a worldwide ticket on-sale that sells one seat to exactly one fan — on Aurora DSQL running active-active across two AWS regions. DSQL gave me the one property the whole app depends on: serializable isolation, so naive read-then-write code stays correct under a simultaneous global stampede. Here are the concrete, load-bearing details that aren't obvious from the headline, written down so the next builder spends those hours elsewhere.

1. A multi-region cluster needs a witness region, and starts life PENDING

You don't just flip a "multi-region" switch. You create one cluster per active region and bind them with a third witness region that holds no endpoint but participates in the quorum:

# region A (active), declaring the witness
aws dsql create-cluster --region us-east-1 \
  --multi-region-properties '{"witnessRegion":"us-west-2"}'
# -> status: PENDING_SETUP

# region B (active), same witness
aws dsql create-cluster --region us-east-2 \
  --multi-region-properties '{"witnessRegion":"us-west-2"}'

Each cluster comes up PENDING_SETUP and stays there until you peer them. They do not become ACTIVE on their own — that surprised me, and it's the first place to look if a cluster seems stuck.

2. Peering is a mutual, per-ARN permission

You update each cluster to add the other as a peer ARN. The gotcha: dsql:AddPeerCluster is authorized per peer ARN, so your IAM policy has to allow the action against the specific cluster ARNs you're peering, not just dsql:* on a wildcard you forgot to include. Once both sides reference each other, both flip to ACTIVE. Connect to each regional endpoint; the witness region has none.

The full IAM set I needed, beyond the default:

dsql:CreateCluster  dsql:GetCluster  dsql:UpdateCluster
dsql:AddPeerCluster dsql:DeleteCluster dsql:ListClusters
dsql:DbConnect      dsql:DbConnectAdmin

3. Auth is an IAM token, not a stored password

DSQL speaks the PostgreSQL wire protocol on port 5432, but you don't manage a password. You mint a short-lived IAM auth token at connect time and hand it to your Postgres driver as the password:

import { Pool } from 'pg';
import { DsqlSigner } from '@aws-sdk/dsql-signer';

const signer = new DsqlSigner({ hostname: host, region, credentials });
const token  = await signer.getDbConnectAdminAuthToken();

const pool = new Pool({
  host, port: 5432, database: 'postgres', user: 'admin',
  password: token,
  ssl: { rejectUnauthorized: false },
  max, idleTimeoutMillis: 5000, connectionTimeoutMillis: 15000,
});

In a serverless deployment (mine runs on Vercel Route Handlers) I mint a fresh pool per request and bound max to the burst size, so a "fire 60 buyers at one seat" test maps to a controlled number of real connections.

4. DDL has no BEGIN wrapper — each statement is its own transaction

Coming from vanilla Postgres I tried to wrap schema creation in a transaction. DSQL runs each DDL statement as its own implicit transaction, so just issue them directly:

await c.query(`CREATE TABLE IF NOT EXISTS seats (
  seat_id TEXT PRIMARY KEY, status TEXT NOT NULL, owner TEXT, sold_at TIMESTAMPTZ)`);
await c.query(`CREATE TABLE IF NOT EXISTS op_log (
  id TEXT PRIMARY KEY, seat_id TEXT, buyer TEXT, outcome TEXT,
  attempts INT, t_start DOUBLE PRECISION, t_end DOUBLE PRECISION)`);

5. The OCC retry loop is the whole point — write it on purpose

DSQL enforces serializability with optimistic concurrency control. When two transactions touch the same row and race to commit, one wins and the other aborts with an OCC error (SQLSTATE 40001/40P01, or an OC### code). This is not a failure to paper over — it's the mechanism doing its job. Catch it, back off, retry; on retry the loser re-reads the now-sold row and correctly rejects:

catch (e) {
  await client.query('ROLLBACK').catch(() => {});
  if (isOcc(e) && attempts < MAX_RETRY) { await sleep(8 * attempts); continue; }
  throw e;
}

isOcc matches 40001, 40P01, codes starting with OC, and /serializ|concurrent|conflict/i in the message. With that loop in place, the naive, unguarded UPDATE seats SET status='sold' — no conditional WHERE, no app-level lock — sells one seat exactly once under 60 concurrent cross-region buyers. The database did the hard part.

What it bought me

Without DSQL, correctness under a worldwide on-sale would have been my code's problem: conditional writes, idempotency keys, a distributed lock, reconciliation jobs. With it, the safety property is a property of the database, and my application code got to stay honest and naive — which is exactly what made the side-by-side demo against DynamoDB Global Tables so stark.

Live: https://worldseat.vercel.app · the OCC retry count is shown right on the scoreboard.

A Jepsen-lite witness: prove "sold once" from committed history, not app hope

Hokutoman00 — Sun, 21 Jun 2026 17:40:31 +0000

When you claim a system is correct under concurrency, the dangerous move is to ask the application whether it was correct. The app is the least trustworthy witness in the building: it reports what it intended, after retries, across a network that drops and reorders. If you count "how many purchase calls returned confirmed", you are grading the system on its own self-report. That's how demos lie without anyone meaning to lie.

WorldSeat — a worldwide ticket on-sale built for AWS's Hack the Zero Stack — answers the correctness question with an external checker instead. I call it the Consistency Witness, and it's deliberately Jepsen-shaped: it ignores what the app said and reconstructs the truth from the committed history in the database.

The invariant, reduced to something checkable

Full linearizability checking is expensive. But the safety property a ticketing system actually needs collapses to one tiny invariant:

For one physical seat, there is at most one successful exclusive acquisition.

That's it. If two different buyers each hold a confirmed claim to seat A1, the history is not linearizable — there is no single global order of operations in which both acquisitions of the same exclusive resource succeed. You don't need a full happens-before graph to detect it; you need to count confirmed acquisitions per seat.

So the witness does exactly that, against committed state:

// DSQL: read the committed op_log directly
SELECT seat_id, count(*)::int AS confirmed
FROM op_log
WHERE outcome = 'confirmed'
GROUP BY seat_id;

// then, for every seat:
doubleSells = Σ max(0, confirmed - 1)
linearizable = (doubleSells === 0)

doubleSells is the number of people who hold a ticket they shouldn't. Zero means every seat was acquired by at most one buyer; anything above zero names the seats with multiple holders.

For the DynamoDB foil the source changes but the logic doesn't: the witness reads the append-only sales ledger with ConsistentRead: true from both regional replicas and unions them, de-duplicated by a globally-unique sale_id, so cross-region replication lag can't make a double-sell disappear by simply not having arrived yet.

Why read the ledger, not the seats table

This is the subtle part, and it's where eventual consistency hides its damage. On the naive DynamoDB backend, the seats table eventually converges to a single owner per seat — last-writer-wins overwrites the others. If you inspected only the seats table, the system would look fine: one seat, one owner. The oversell is invisible there.

The truth lives in the append-only ledger, which records every confirmation as it was issued. Three buyers were each told "you got A1" and each got a confirmation code; LWW later picked one to display, but three people are walking to the same chair. The witness reads the ledger precisely because it's the one place the system can't quietly forget what it promised.

Reconciling two different measurements

WorldSeat shows two numbers that look like they should match but measure different scopes, and reconciling them honestly was a design point:

The Break-it scoreboard measures the single seat the last burst targeted.
The Witness sums double-sells across the whole floor (every seat with history).

If you've run several bursts on different seats, the floor-wide total is legitimately larger than the last seat's count. Rather than paper over that, the witness accepts an optional ?seat= focus so it can report the scope-matched number for the targeted seat and the floor-wide total side by side — and the UI explicitly prints "✓ reconciled" only when scoreboard oversold equals the witness's focus count for that seat. Matching numbers you didn't engineer to match is the difference between a checker and a prop.

The payoff

Because the verdict is computed from committed history every time you ask, it's not a stored boolean you can fake. On Aurora DSQL it reads Linearizable ✓ after sixty concurrent buyers hit one seat across two regions. On DynamoDB Global Tables running the same code it reads Violated ✗ and names the over-sold seats. The checker is the same; only the database's consistency model changed the verdict.

See it live: https://worldseat.vercel.app — press Break it, then read the Witness.

Make database consistency something a judge can break with one click

Hokutoman00 — Sun, 21 Jun 2026 17:40:27 +0000

Most demos of "strong consistency" ask you to take it on faith. A slide says serializable, a diagram has some arrows, and you nod. You never see the property do anything, because when consistency works, nothing happens — that's the whole point. It's invisible plumbing.

WorldSeat makes the plumbing adversarial. It's a worldwide ticket on-sale where one seat must sell to exactly one fan. There's a button labeled Break it. Press it and the app fires dozens of simultaneous buyers at a single seat, from two AWS regions at once, and counts how many people walked away holding a ticket for the same physical chair.

The trick that makes this honest: both backends run the identical naive application code. A read-then-write with no application-level locking, no conditional update, no cleverness:

const r = await client.query('SELECT status FROM seats WHERE seat_id=$1', [seat]);
if (r.rows[0]?.status !== 'available') { rollback(); return 'rejected'; }
await client.query('UPDATE seats SET status=$1, owner=$2 WHERE seat_id=$3', ['sold', buyer, seat]);

That is the canonical race condition. Two buyers both read available, both write sold, both think they won. On any database without serializable isolation, you just sold one seat twice.

The only variable in WorldSeat is which AWS database executes that code:

Aurora DSQL — distributed SQL, multi-region active-active, serializable isolation enforced by optimistic concurrency control (OCC). When two transactions conflict, one commits and the other aborts with OC001/40001. The app catches it, retries, re-reads sold, and correctly rejects the second buyer.
DynamoDB Global Tables — multi-region, eventually consistent, last-writer-wins. Two buyers in two regions each read available from their local replica and each write their own sold. Both succeed. Replication later reconciles to a single owner on the seats table — which quietly hides the damage — but the append-only sales ledger remembers that two different people were each told "you got it."

Here is the live result from the committed stress test (node scripts/stress.mjs), hammering seat A1 at escalating concurrency against the production deployment:

Aurora DSQL (strong / serializable OCC):
  c= 8  confirmed=1  oversold=0
  c=24  confirmed=1  oversold=0   maxRetries=1
  c=48  confirmed=1  oversold=0   maxRetries=1
  c=60  confirmed=1  oversold=0   maxRetries=1

DynamoDB Global Tables (eventual / LWW):
  c= 8  confirmed= 8  oversold= 7
  c=24  confirmed=12  oversold=11
  c=60  confirmed=33  oversold=32

Same code. Same seat. Same concurrency. The database's consistency model is the entire difference between one ticket and thirty-three.

A note on maxRetries, because honesty cuts both ways: it is timing-dependent, and a perfectly correct run can show 0. When a late buyer's transaction begins after the winner has already committed, it simply reads sold and rejects cleanly — no commit conflict, no retry needed. A non-zero maxRetries means two commits genuinely interleaved and OCC aborted the loser. Either path yields oversold = 0 — that's the claim. The retry counter is supporting evidence that a real race sometimes occurs under the hood, not the headline; the headline is the invariant.

Why "oversold" is counted from committed state, not hope

It would be easy — and dishonest — to count oversells from what the app thinks happened (how many requests returned confirmed). Application return values lie under partition and retry. So WorldSeat never trusts them. The oversold number is computed by reading the committed ground truth:

On DSQL, a SELECT count(*) FROM op_log WHERE seat_id=$1 AND outcome='confirmed'.
On DynamoDB, a strongly-consistent scan of the sales ledger across both regional replicas, unioned and de-duplicated by a globally-unique sale_id, so cross-region replication lag can't undercount.

oversold = confirmed_in_committed_ledger − 1. One real seat; everything beyond one is a person who will show up to a venue and find someone already sitting there.

What this teaches that a slide can't

The reason ticketing on-sales melt down — the reason "I had it in my cart and then it was gone" is a universal experience — is precisely this race, at the scale of a stadium and the speed of a fan base refreshing in unison. WorldSeat lets you feel the difference between a database that serializes the stampede for you and one that makes you, the application developer, responsible for inventing correctness on top of eventual consistency.

DSQL's value here isn't "it's fast" or "it's managed." It's that it lets you write the naive, obvious code and still be correct under a worldwide simultaneous on-sale — and you can watch it hold the line, live, at sixty concurrent buyers across two regions, by pressing a button.

Try it: https://worldseat.vercel.app — buy a seat, then break it.