14:07 UTC. PagerDuty fires on
fulfillment-queue-depth, 340% of baseline and climbing. Checkout's on-call pulls up the dashboard expecting a stuck consumer or a slow downstream call. Instead she finds the queue isn't slow, it's just full of orders nobody placed twice on purpose.
the scramble
First theory: the mobile app's retry logic, rewritten three weeks ago to handle flaky connections more aggressively, is double-submitting. Someone pulls the client-side request logs for a sample of the duplicate order IDs. Each one shows exactly one tap, one request, one Idempotency-Key header generated client-side and reused correctly across any retries the client itself issued. The client is behaving. Ruled out in under five minutes.
Second theory, and the one that eats fifteen minutes: API Gateway retrying a Lambda invocation that looked like it timed out, but actually succeeded after the gateway gave up and resent the request. This would produce exactly this symptom, two invocations for one logical request. CloudWatch says otherwise. Every invocation for the affected order IDs completed in 180-240ms, nowhere near the 29-second timeout, and API Gateway's own access logs show one request in, one request out, for every one of them. No retries at that layer either.
Third, and this is the thread that actually goes somewhere: someone notices every duplicate pair shares the same idempotencyKey attribute but has two different order_id values. If the idempotency check were working at all, that shouldn't be possible. So the check itself becomes the suspect.
the hunt
The checkout Lambda's create-order path looks, on paper, like textbook idempotency handling: look up the key, bail out if it exists, write if it doesn't.
checkout-api/createOrder.ts, as it shipped
const existing = await ddb.send(new QueryCommand({
TableName: 'orders',
IndexName: 'idempotency-key-index', // GSI, partition key: idempotencyKey
KeyConditionExpression: 'idempotencyKey = :k',
ExpressionAttributeValues: { ':k': { S: idempotencyKey } },
}));
if (existing.Items && existing.Items.length > 0) {
return { orderId: existing.Items[0].order_id.S, duplicate: true };
}
const orderId = crypto.randomUUID();
await ddb.send(new PutItemCommand({
TableName: 'orders',
Item: { order_id: { S: orderId }, idempotencyKey: { S: idempotencyKey }, /* ... */ },
}));
return { orderId, duplicate: false };
Nothing here looks wrong until you ask one specific question: what consistency guarantee does that Query against a Global Secondary Index actually carry? The answer, in DynamoDB, is none. GSIs are updated asynchronously from the base table and only support eventually consistent reads, full stop, there is no ConsistentRead: true option for a GSI the way there is for the base table. AWS documents this plainly. Nobody on the team had needed to think about it before, because nothing had ever hit the index hard enough, fast enough, for the gap to matter.
14:07 UTC lines up with a known mobile carrier blip in one metro area. Retry logs on the client are clean, as established, but the client's retry backoff for this release is 300ms, and CloudWatch Contributor Insights shows GSI propagation delay for the orders table spiking to 400-600ms during the same window, well above its normal sub-100ms. Two legitimate-looking requests, 300ms apart, both querying a GSI that hadn't caught up from the first request yet.
Reproducing it in staging confirms the mechanism: five identical create-order calls fired 250ms apart, same idempotency key, against a GSI primed to lag by artificially spiking write volume on the table beforehand.
staging repro, 5 calls / 250ms apart / same key
$ for i in 1 2 3 4 5; do
aws lambda invoke --function-name checkout-create-order \
--payload '{"idempotencyKey":"repro-001"}' /tmp/out-$i.json &
sleep 0.25
done; wait
$ grep -h order_id /tmp/out-*.json | sort -u
"order_id":"a1c9..."
"order_id":"a1c9..."
"order_id":"f02e..." # duplicate
"order_id":"a1c9..."
"order_id":"a1c9..."
the find
Invocation 3 queried the GSI before invocation 1's write had propagated to it, saw zero matching items, and concluded, correctly given what it could see, that no order existed yet. It wasn't a bug in the retry logic, the Lambda runtime, or API Gateway. It was a correctness check built on a read path that DynamoDB never promised would be current. The code asked "does this exist yet?" of a view of the data that could lag the thing it was trying to protect against.
the fix
The fix isn't a faster GSI, you can't request strong consistency on one at any price. It's moving the correctness check off the index entirely and onto the one place DynamoDB actually gives you atomicity: a conditional write against the base table (or, cleaner, a dedicated table keyed directly by the idempotency key).
checkout-api/createOrder.ts, after
const orderId = crypto.randomUUID();
try {
await ddb.send(new PutItemCommand({
TableName: 'idempotency-locks',
Item: {
idempotencyKey: { S: idempotencyKey },
order_id: { S: orderId },
ttl: { N: String(Math.floor(Date.now() / 1000) + 86400) },
},
ConditionExpression: 'attribute_not_exists(idempotencyKey)',
}));
} catch (err) {
if (err.name === 'ConditionalCheckFailedException') {
const existing = await ddb.send(new GetItemCommand({
TableName: 'idempotency-locks',
Key: { idempotencyKey: { S: idempotencyKey } },
ConsistentRead: true, // base table, strong reads are available here
}));
return { orderId: existing.Item!.order_id.S, duplicate: true };
}
throw err;
}
await ddb.send(new PutItemCommand({ TableName: 'orders', Item: { order_id: { S: orderId }, /* ... */ } }));
return { orderId, duplicate: false };
A conditional PutItem on a base table is atomic: when two invocations race for the same key, exactly one succeeds and the other gets ConditionalCheckFailedException back, deterministically, every time, regardless of replication lag anywhere else in the system. The loser reads the winner's order_id off the same strongly-consistent base table and returns that instead of minting a new one. The TTL attribute lets DynamoDB expire old lock rows automatically after 24 hours instead of growing the table forever.
Two more changes went in alongside it. A CloudWatch alarm on ConditionalCheckFailedException rate for the idempotency-locks table, so the next retry storm shows up as a metric spike within a minute instead of a downstream queue-depth alert twenty minutes later. And a reconciliation script to walk the affected window and merge or cancel the orders that had already been created in duplicate before the fix shipped.
the aftermath
1,400 duplicate orders created in the 22-minute window
$61,800 fulfillment cost on duplicates not caught before pick-and-pack
1,190 of the 1,400 duplicates were caught by the reconciliation script before fulfillment touched them. The remaining 210 had already been picked, packed, or shipped by the time the script ran, and became the bulk of the $61,800, split between return shipping and goodwill refunds for customers who'd received two of something they ordered once.
- A Global Secondary Index in DynamoDB never supports strongly consistent reads, by design, and no configuration changes that. Querying one as a performance shortcut is fine. Querying one as a correctness check is a race condition waiting for enough concurrency to expose it.
- Idempotency belongs on a conditional write against a base table, where DynamoDB actually gives you atomicity, not on a read-then-write pattern against any index, GSI or otherwise. The read and the write need to be the same operation, or there's a window between them for another invocation to run.
- The race needed concurrency to show up, and the concurrency came from somewhere unrelated to the database: a retry policy tuned for a flaky network, three weeks earlier, by a team that had no reason to think about GSI propagation lag at all.
- Alarming on the symptom (fulfillment queue depth) found this twenty minutes in. Alarming on the mechanism (
ConditionalCheckFailedExceptionrate, which is now expected to show nonzero values during legitimate retries) would have shown the retry storm starting in under a minute.
Nobody had written the original check carelessly. A query that filters on the exact key you're about to insert reads like a correctness guarantee in every language and every database that doesn't make an explicit distinction between "consistent" and "eventually consistent" read paths. DynamoDB does make that distinction, loudly, in its own documentation. The gap wasn't missing diligence, it was a guarantee that existed in the docs and not in the code.
Top comments (0)