A successful API response does not always mean money moved, and a failed API response does not always mean it didn’t. I've had to learn this while working on features like withdrawals, virtual accounts, and virtual cards for a few fintech companies.
Imagine this scenario:
- A user initiates a ₦50,000 withdrawal.
- Your backend calls the payment gateway.
- The gateway times out midway.
- Your system receives no response.
- But the money actually leaves your account.
Now your app says:
“Withdrawal failed”
…but the user receives the money anyway.
This is one of the classic distributed systems problems in fintech: the system loses visibility into the true state of the transaction.
Here’s how I would design a production-grade system to handle this safely.
1. The Core Principle: Design for Uncertainty
Most junior implementations treat payments like normal CRUD operations:
Request → Gateway → Success/Failure
But real payment systems are not synchronous. They are:
- distributed
- eventually consistent
- network-dependent
- partially unreliable
So instead of designing around “responses,” we design around:
- transaction state
- reconciliation
- eventual correctness
The goal is not:
“respond immediately”
The goal is:
“never lose or duplicate money.”
2. Phase One: Intent and Idempotency
The first challenge is preventing duplicate withdrawals.
Imagine:
- user clicks withdraw twice
- frontend retries due to timeout
- mobile app reconnects and resends request
- reverse proxy retries automatically
Without idempotency, you can easily process the same withdrawal multiple times.
The Solution: Client-Generated Idempotency Keys
The frontend generates a unique UUID per withdrawal action:
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
This key represents:
“this exact withdrawal intent”
The backend stores it alongside the transaction:
withdrawals
- id
- user_id
- amount
- status
- idempotency_key
With a unique constraint:
UNIQUE(user_id, idempotency_key)
Now if the same request is retried:
- the backend does NOT create another withdrawal
- it simply returns the existing transaction
This is extremely important because:
retries are normal in distributed systems.
3. Ledger Safety Before External Calls
Before talking to the payment gateway, I move the funds into a reserved state.
Instead of:
available_balance -= 50000
I use a ledger model:
available_balance -= 50000
reserved_balance += 50000
This prevents:
- double spending
- concurrent withdrawals
- inconsistent balances
At this stage:
- money is internally reserved
- but not yet externally settled
This gives the system a safe rollback point.
4. The Gateway Timeout Problem
Now we send the withdrawal request to the payment provider.
Possible outcomes:
- success response
- explicit failure response
- timeout / unknown response
The third case is the dangerous one.
A timeout does NOT mean:
“payment failed”
It only means:
“we don’t know what happened”
The provider may still process the withdrawal successfully after the timeout.
This is where many systems make catastrophic mistakes by:
- marking transaction failed immediately
- refunding user
- causing duplicate payouts
Instead, I transition the transaction into:
PROCESSING
or
UNKNOWN
This state means:
“The transaction outcome is unresolved.”
5. Queue-Based Reconciliation Instead of Blocking Polling
A common mistake is implementing polling like this:
while(status !== SUCCESS) {
checkProvider()
}
This is bad because:
- it blocks workers
- wastes resources
- scales poorly
- creates stuck processes
Instead, I use a queue system like:
- BullMQ
- Kafka
- RabbitMQ
Important Clarification
The queue itself becomes the polling mechanism.
I do NOT:
- poll repeatedly inside one job
- rely only on retry attempts
Instead:
- each job performs ONE reconciliation attempt
- then schedules another delayed job if needed
Example:
Job 1
- check provider
- status still unknown
- enqueue next check in 30 seconds
Job 2
- check again
- still unresolved
- enqueue next check in 1 minute
Job 3
- resolve successfully
This creates:
- scalable polling
- distributed reconciliation
- non-blocking workers
6. Webhooks + Polling: Why Both?
Another important question:
“Wouldn’t webhooks and polling conflict?”
Yes — if implemented poorly.
The correct design is:
- webhook = primary source of truth
- reconciliation polling = fallback safety mechanism
Both systems update transactions through the SAME state machine.
Example valid transitions:
PENDING → PROCESSING
PROCESSING → SUCCESS
PROCESSING → FAILED
Invalid or stale updates are ignored.
For example:
- webhook resolves SUCCESS
- delayed polling later returns stale FAILED
The system ignores the stale update because:
SUCCESS is already terminal.
This avoids race conditions completely.
7. Reconciliation is the Real Heart of Payments
The most important part of payment systems is not processing.
It is reconciliation.
The system continuously attempts to discover:
“What actually happened financially?”
Reconciliation workers may check:
- provider status endpoints
- webhook logs
- settlement reports
- internal ledger state
This is how the system eventually converges on truth.
8. SLA Does NOT Define Failure
This is another subtle but important point.
Many engineers think:
“If SLA is 10 minutes, mark failed after 10 minutes.”
That is dangerous.
SLA should define:
- retry frequency
- escalation thresholds
- operational expectations
NOT transaction truth.
A transaction should only become FAILED if:
- provider explicitly confirms failure
- reconciliation exhausts all known success paths
- manual review determines failure
Until then:
PROCESSING
is safer than incorrectly refunding money.
Because in payments:
false negatives are often more dangerous than delayed certainty.
9. Final State Resolution
Eventually the transaction reaches a terminal state.
If Successful
reserved_balance -= 50000
ledger_entry = finalized
status = SUCCESS
If Failed
reserved_balance -= 50000
available_balance += 50000
status = FAILED
This guarantees:
- no lost money
- no duplicated money
- auditable transaction history
10. The Real Philosophy Behind Reliable Payment Systems
The biggest mindset shift is this:
Reliable payment systems are NOT built around immediate responses.
They are built around:
- idempotency
- state machines
- queues
- reconciliation
- eventual consistency
The API response is only one signal.
The real truth emerges over time through controlled reconciliation.
And that’s what makes payment engineering fundamentally different from normal CRUD backend development.
Top comments (1)
The sentence that should be printed on the wall is that a failed response does not always mean the money did not move — a timeout is an unknown state, not a negative result, and treating it as a failure is how the app says 'withdrawal failed' while the money leaves. Generating the idempotency key per user action, rather than per request, is the detail that makes the retry safe: the double-click, the reconnect, and the proxy retry all collapse onto the same intent. Reserving in the ledger before the external call completes the picture, because then the uncertain window has a state you can reconcile instead of a gap you have to guess about.