At two in the morning, the alert group exploded. Customers reported that their bank accounts were charged twice for a single order, but only one order was created. Checking the logs, we found that the same payment callback was consumed twice, inventory was deducted twice, but only one order record was inserted. Worse still, the second deduction had no corresponding order, leading to an unexplained amount during financial reconciliation.
After two days of investigation, the root cause was a classic check-then-act concurrency problem: when processing the payment callback, the system first checked whether the order existed. If not, it created the order and deducted inventory. Two requests arrived almost simultaneously, both found no existing order, and both executed the creation logic. With no unique constraint at the database level and no locking in the business code, the duplicate charge occurred.
This problem is common in e-commerce systems, but it’s trickier in cross-border proxy purchasing scenarios—multiple payment channels (PayPal, Stripe, local payments), unstable callback delays, and inventory synchronization that must interface with domestic Chinese sourcing platforms like 1688 and Taobao. Once concurrency control breaks down, both finances and inventory fall into chaos.
The Core Issue: Non‑Atomic Operations Under Concurrency
When most developers encounter this type of problem, their first instinct is to add a unique database index. But idempotency for payment callbacks cannot rely solely on the database—callbacks may come from different IPs at different times. A unique index can only prevent duplicate inserts; it cannot stop two queries that both return empty followed by two separate inserts (if the transaction isolation level is insufficient). A more robust approach is to introduce a distributed lock, so that callbacks for the same payment are serialized.
Our solution was: Redis distributed lock + inventory pre‑reservation + eventual consistency reconciliation. Below is a breakdown of the key code.
Solution: Redis Distributed Lock + Inventory Pre‑reservation
// Payment callback handler entry point
public function handlePaymentCallback(string $paymentId, float $amount, string $orderNo): void
{
$lockKey = "payment:lock:{$paymentId}";
$lockValue = uniqid('', true); // unique value for lock verification on release
// Attempt to acquire the lock, timeout 3 seconds
$locked = $this->redis->set($lockKey, $lockValue, ['NX', 'EX' => 10]);
if (!$locked) {
// Lock is held by another process, meaning it's already being processed
logger()->warning("Payment callback already processing", ['paymentId' => $paymentId]);
return;
}
try {
// Check if order already exists (idempotency)
$order = $this->orderRepository->findByPaymentId($paymentId);
if ($order !== null) {
logger()->info("Order already exists for payment", ['paymentId' => $paymentId]);
return;
}
// Begin transaction
$this->db->beginTransaction();
try {
// Pre‑reserve inventory (Redis atomic operation)
$stockKey = "stock:preorder:{$orderNo}";
$stockResult = $this->redis->eval(
"local stock = redis.call('GET', KEYS[1])
if not stock or tonumber(stock) < tonumber(ARGV[1]) then
return 0
end
redis.call('DECRBY', KEYS[1], ARGV[1])
return 1",
[$stockKey, $orderNo],
1,
1 // quantity to deduct
);
if ($stockResult === 0) {
throw new \RuntimeException("Insufficient stock for order {$orderNo}");
}
// Create order
$orderId = $this->orderRepository->create([
'payment_id' => $paymentId,
'amount' => $amount,
'status' => 'paid',
'created_at' => date('Y-m-d H:i:s'),
]);
$this->db->commit();
logger()->info("Order created successfully", ['orderId' => $orderId]);
} catch (\Throwable $e) {
$this->db->rollBack();
// Roll back Redis inventory pre‑reservation (lock ensures atomicity)
$this->redis->incrBy($stockKey, 1);
throw $e;
}
} finally {
// Release lock (using Lua script for atomicity)
$this->redis->eval(
"if redis.call('GET', KEYS[1]) == ARGV[1] then
redis.call('DEL', KEYS[1])
end",
[$lockKey, $lockValue],
1,
$lockValue
);
}
}
This code solves three problems:
- Lock mutual exclusion: Only one process can enter the handler for the same paymentId.
- Inventory pre‑reservation: Inventory is deducted atomically via Redis before the order is created, preventing overselling.
- Atomic lock release: A Lua script ensures that only the lock holder can delete the lock, avoiding accidental removal of another process’s lock.
Implicit Knowledge: Lock Expiration and Renewal Traps
In the code above, the lock TTL is set to 10 seconds. But business processing might take longer than 10 seconds (e.g., a timeout retry when calling the 1688 procurement API). If the lock expires, a second request may acquire the lock while the first request is still in progress, causing two requests to handle the same payment concurrently.
The correct approach is a lock renewal mechanism. After acquiring the lock, you can start a background coroutine (or scheduled task) that checks every 3 seconds whether the lock still belongs to you, and if so, extends the TTL. In PHP, native coroutines aren’t available, but you can use Swoole or simply renew the lock manually within the business loop. In our case, because business processing usually took less than 3 seconds, we didn’t add renewal, but it’s a potential risk in high‑latency scenarios.
Another easily overlooked point is the consistency between inventory pre‑reservation and actual deduction. In the code above, if the order is created successfully but the subsequent procurement fails (e.g., a 1688 product is removed from the shelf), you need to roll back the inventory pre‑reservation. We ensure eventual consistency through database transactions and a Redis rollback (incrBy), but the rollback itself could fail. A more robust approach is to use a message queue to asynchronously handle inventory rollback, or to adopt the Redis Redlock algorithm with automatic TTL expiration.
Actual Results
After introducing this solution, duplicate charges never occurred again. With a daily order volume of around 3,000, the inventory pre‑reservation success rate approached 100%, and after‑sales tickets caused by inventory issues dropped by over 60%. More importantly, financial reconciliation no longer required manual investigation of unexplained amounts, saving about two person‑days per month.
Of course, there is no silver bullet. The distributed lock adds dependency and complexity on Redis. If Redis goes down, the entire payment callback pipeline will block. We implemented a fallback: if acquiring the lock fails more than three times, the callback message is sent to a dead‑letter queue for manual intervention.
Summary
The core of e‑commerce concurrency problems is not the technology stack, but an understanding of the business scenario. Idempotency of payment callbacks, atomic inventory deduction, and lock boundary conditions—every detail can be a fuse for an incident. A good solution is one that the user doesn’t even notice—this saying is especially true in concurrency control.
What tricky concurrency problems have you encountered? Did you solve them with database locks, Redis locks, or other methods? Feel free to share your experience.
DESCRIPTION: Starting from a real duplicate charge incident, this article details a Redis distributed lock + inventory pre‑reservation solution for e‑commerce systems, including complete PHP code and implicit knowledge.
About the Author: Building cross‑border e‑commerce solutions — from Taocarts (1688/Taobao daigou system) to AuctionGIt (Japanese auction proxy covering 60+ platforms). Happy to connect.
Top comments (0)