In distributed systems, when I send an API request and receive a "timeout," few things are as frustrating as not knowing if that operation was actually performed on the server side. The TCP connection might have dropped, the load balancer might have returned a 504 Gateway Timeout, but in the background, the database commit might have already gone through. In this moment of uncertainty, if the client sends the request again "just in case," and my system isn't designed according to the principle of idempotency, two identical orders might be created, or the same invoice might be issued twice.
In my 20 years of field experience, the biggest misconception I've encountered is the idea that idempotency can be solved by simply saying, "let's add a check." In reality, it's an architectural decision that spans from the network layer to database transaction management. Last year, while working on a production ERP, when we noticed that raw material stocks were off by 12% due to duplicate production orders coming from operator screens, I realized the problem wasn't "fast-clicked buttons" but rather weak idempotency design. In this post, I'll explain how I solve this problem in modern systems and what trade-offs I consider.
The Network Lies: The Dark Side of Retry Mechanisms
The only way to be sure a network packet reached its destination is to receive an acknowledgment (ACK), but not receiving that ACK doesn't mean the packet didn't go through. I call this the "Network Layer Paradox." Especially in mobile applications or devices operating on weak Wi-Fi networks on a factory floor, the packet reaches the server, is processed, but the connection drops while the response is returning. The client-side "retry" logic, in this scenario, becomes the system's biggest enemy.
While reviewing crash reports from the Play Store for a mobile application
Top comments (1)
Um, where's the rest?