How a double-spend during testing taught me why financial systems treat duplicate requests as a first-class problem
VaultPay is a wallet microservice I built on top of AuthShield.
Previous parts:
Part 1 is here: I Built AuthShield and Immediately Knew It Wasn't Enough
Part 2 is here: The Silent Failure I Never Saw Coming: What VaultPay Taught Me About Consistency Under Failure
Part 3 is here: I Started With a Blocklist. That Was the Wrong Instinct and VaultPay Taught Me Why.
Before I implemented idempotency in VaultPay, I ran a test.
A transfer request fired. The network hiccupped mid-response. The client never got confirmation so it retried. The server had already processed the first request. It processed the second one too.
Same sender. Same receiver. Same amount. Two debits.
The auth was perfect. The atomic transaction worked exactly as designed - both transfers completed cleanly, both rolled back cleanly if anything failed. The consistency guarantees from the previous post held up fine.
But money moved twice.
The system had no idea it was seeing the same request again. From VaultPay's perspective, two valid requests arrived. Both passed every check. Both transferred money. Both committed. The fact that they were logically the same operation was invisible to the server.
That's what I wanted to understand. Not just "add idempotency" as a checklist item, but why duplicate requests are a first-class problem in financial systems and how the solution actually works.
Why Duplicates Are Inevitable
The naive assumption is that a client only retries when something went wrong. The server crashed. The network timed out. The response never arrived.
But "response never arrived" is the problem. The client doesn't know whether the server processed the request or not. From the client's perspective, silence looks identical whether the server crashed before processing, crashed after processing, or processed successfully and the response got lost on the way back.
So the client retries. It has to. Not retrying means accepting data loss on network failures. Retrying blindly means accepting duplicate operations on successful-but-unacknowledged requests.
This is a fundamental distributed systems problem. Every financial API in production faces it. The solution isn't to prevent retries - it's to make retries safe.
What Idempotency Actually Means
An operation is idempotent if performing it multiple times produces the same result as performing it once.
In the context of a transfer API, idempotency means: if the client sends the same transfer request ten times, money moves exactly once.
The mechanism is an idempotency key - a unique identifier the client generates and attaches to the request. The server uses this key to recognise duplicate requests and return the cached result instead of re-processing them.
First request: POST /transactions/send { idempotency_key: "123e4567-...", amount: 500 }
→ Server processes transfer, commits, caches result
→ Response: { transaction_ref: "TXN-20260524-ABCDE" }
Retry (same key): POST /transactions/send { idempotency_key: "123e4567-...", amount: 500 }
→ Server finds key in Redis, returns cached result immediately
→ Response: { transaction_ref: "TXN-20260524-ABCDE" }
→ Money did NOT move again
The client gets the same response both times. From its perspective, the retry succeeded. From the server's perspective, the second request never reached the transfer engine.
Where It Lives in the Flow
Idempotency check runs at step 3 in VaultPay's send money flow - immediately after JWT validation, before anything else.
POST /transactions/send
↓
JWT Validation ← step 2: verify identity
↓
Idempotency Check ← step 3: have we seen this key before?
↓
IP Trust Check ← step 4
↓
PIN Verification ← step 5
↓
Wallet + Limit Checks ← step 6
↓
Atomic Transfer ← step 7
↓
Post-Commit Redis Writes ← step 8: cache idempotency key HERE
The early position is intentional. If the key is already cached, there's no point running PIN verification, limit checks, or touching the transfer engine. The response is already known. Return it immediately.
But notice step 8 - the idempotency key only gets written to Redis after the atomic transaction commits. Not before. Never before.
The Redis Key and What Gets Stored
The client generates a UUID and includes it in the request body as idempotency_key. VaultPay stores the result in Redis using a simple key pattern:
idempotency:{client_provided_key}
What gets stored as the value isn't the full response object. It's the transaction reference - the TXN-YYYYMMDD-XXXXX identifier assigned to the completed transfer.
# After successful commit
await redis.set(
f"idempotency:{idempotency_key}",
txn.transaction_ref,
ex=settings.IDEMPOTENCY_TTL # 86400 seconds — 24 hours
)
On a duplicate request, VaultPay checks for the key, finds the transaction reference, and returns it in the same response shape the client would have received originally.
async def check_idempotency(idempotency_key: str, redis: Redis):
existing = await redis.get(f"idempotency:{idempotency_key}")
if existing:
return IdempotentResponse(
transaction_ref=existing,
message="Duplicate request - returning cached result"
)
return None
Storing just the transaction reference keeps the cached value small. The client can use the reference to look up full transaction details from the history endpoint if needed. The idempotency cache is not a substitute for the transaction record - it's a guard that prevents creating a second one.
What Happens When a Transfer Fails
This is the part that matters as much as the happy path.
If a transfer fails - wrong PIN, insufficient balance, limit exceeded, database rollback - the idempotency key does not get cached.
The Redis write only executes after a successful database commit. Any exception raised before that point - at validation, during the atomic transaction, on rollback - exits the function before the caching code runs. The key stays absent from Redis.
try:
async with db.begin():
# ... atomic transfer steps ...
await db.commit()
# Only reaches here on successful commit
await redis.set(
f"idempotency:{idempotency_key}",
txn.transaction_ref,
ex=settings.IDEMPOTENCY_TTL
)
except InsufficientBalanceError:
# Key NOT cached - client can retry with same key after topping up
raise
except Exception:
# Key NOT cached - client can retry with same key
raise
This is the right behaviour. If a transfer failed because the sender had insufficient balance, the client should be able to fix the issue and retry with the same key. Caching a failed result would permanently poison that key for 24 hours - every retry would get back "duplicate request" even though the money never moved.
The idempotency key represents a successful outcome, not an attempted one. It only exists if the transfer actually completed.
The 24-Hour TTL
The idempotency key lives in Redis for 24 hours. After that, it expires automatically.
This means a client that retries more than 24 hours after the original request will be treated as a new request. In practice, a retry that arrives a day later is almost certainly intentional - a user initiating a new transfer - not an accidental duplicate from a network failure.
The 24-hour window covers the realistic retry window for transient failures. A network timeout that triggers a retry happens within seconds or minutes. A client implementation bug that re-sends a request might happen within hours. A day is a reasonable boundary between "retry" and "new request."
For the small number of edge cases where this matters - a client that genuinely needs idempotent behaviour beyond 24 hours - the client should generate a new idempotency key and accept that it's creating a new transfer intent.
What I Actually Learned
Before testing VaultPay without idempotency, I understood the concept abstractly. Idempotency keys prevent duplicate operations. Add them to financial APIs. Got it.
Watching money move twice from the same request made it concrete in a way reading about it didn't.
The client can't know whether the server processed its request. The network doesn't tell you. So from the client's perspective, a retry is always the right thing to do on an ambiguous failure. The server has to be the one that handles this - by remembering what it's already done and returning that result instead of doing it again.
The mechanism is simple. The key is client-generated, the result is cached post-commit, failed transfers don't get cached, the TTL handles cleanup. None of it is complicated to implement. But it has to be built before you discover the problem in production. Because in production, the duplicate isn't a test. It's someone's money moving twice.
Next up: how VaultPay stores identity documents without becoming a security liability - AES-256 encryption, what SHA-256 is doing for duplicate detection, and what actually changes in the system after KYC approval.
Engineering docs + code samples: Vaultpay-Engineering
Top comments (0)