One of the most dangerous backend mistakes is assuming this:
“If the request failed, nothing happened.”
That assumption causes duplicate payments, duplicate orders, duplicate invoices, and a lot of production cleanup.
In real systems, a failed response does not always mean the operation failed.
Sometimes the operation succeeded, but the response never came back.
Where this usually happens
This problem appears a lot in flows like:
- payment processing
- order creation
- invoice generation
- webhook handling
- third-party API integrations
- background job retries
Example:
Your backend sends a payment request to a payment gateway.
The gateway charges the customer.
But before your server receives the response, something fails:
- network timeout
- server restart
- gateway delay
- proxy timeout
- worker crash
Your system thinks:
“Payment failed. Let’s retry.”
But the payment may have already succeeded.
Now the customer gets charged twice.
The real problem
The issue is not the retry itself.
Retries are necessary.
The real problem is retrying without idempotency.
Without idempotency, every retry is treated like a new action.
So this:
Create order
Create order again
Create order again
becomes three different orders.
And this:
Charge customer
Retry charge
Retry charge again
can become multiple charges.
What idempotency means
Idempotency means the same operation can be repeated safely without creating duplicate results.
If the same request is sent twice, the system should return the same result instead of doing the action again.
For example:
Request ID: payment_123
Action: charge customer $100
If the request is received again with the same ID, the system should not charge again.
It should return the original result.
How to fix it
1. Use idempotency keys
Every critical operation should have a unique idempotency key.
Good examples:
order_982_checkout_attempt_1
payment_982_customer_45
invoice_982
Store this key before executing the operation.
If the same key appears again, return the existing result.
2. Store operation status
Do not only store final success or failure.
Track states like:
pending
processing
success
failed
This helps avoid duplicate execution when something is already in progress.
3. Make retries safe
Retries should be controlled.
Avoid blind retries.
A retry system should know:
- what operation is being retried
- whether it already succeeded
- how many times it was retried
- what error caused the retry
- whether it is safe to retry
4. Handle webhooks carefully
Payment gateways often send webhooks after checkout.
But webhooks can arrive:
- late
- multiple times
- out of order
Never assume a webhook is unique.
Always check the external transaction ID before creating or updating records.
5. Use database constraints
Application logic is not enough.
Add database-level protection where possible.
For example:
UNIQUE(transaction_id)
UNIQUE(idempotency_key)
UNIQUE(external_order_reference)
This gives you a final safety layer if application logic fails.
The mistake most teams make
They design for the happy path.
Customer clicks pay.
Payment succeeds.
Order is created.
Invoice is generated.
But production does not only run happy paths.
Production has:
- timeouts
- retries
- duplicate callbacks
- delayed webhooks
- partial failures
- users clicking twice
- workers restarting mid-process
If your backend does not expect these cases, duplicates are only a matter of time.
A better approach
For critical flows, think like this:
Can this operation run twice safely?
If the answer is no, the system is fragile.
Payments, orders, invoices, and stock updates should never depend only on “the request probably ran once”.
They need protection at the design level.
How we handle this at BrainPack
At BrainPack, we design payment and order flows with idempotency from the beginning.
For critical operations, we use idempotency keys, transaction tracking, retry-safe processing, webhook validation, and database-level uniqueness rules.
The goal is simple:
A timeout should not become a duplicate payment.
A retry should not become a duplicate order.
And a webhook should not create business data twice.
Top comments (0)