Suppose it is late at night and you decide to order pizza from Zomato. While making the payment, Paytm shows an invalid balance because it is month end.
From your perspective, you created the order, paid, and then it got cancelled.
But in Zomato’s backend, there is an order service connected to SQS that is still processing events.
SQS is a data structure that works like a queue. Standard SQS does not guarantee FIFO ordering. It maintains best-effort ordering and is much cheaper than FIFO queues.
So there are 5 workers processing messages ahead of yours.
Suppose the cancellation event reaches Chef1 first because standard SQS does not guarantee order. Chef1 sees the order was cancelled but finds nothing in the database. Then Chef2 receives the creation event and creates the order in the DB. Now you have a paid order that you never actually paid for. 😅
Then you ordered the same pizza again and enjoyed a movie. 📽️
Comment your favourite movie name below!
Meanwhile the finance team tells engineers to stop vibe coding and fix this mess. You have until Monday. What do you do?
A) Switch to SQS FIFO plus MessageGroupId per order_id and let AWS handle ordering per order.
B) Keep standard SQS, add a sequence number plus reorder buffer in the consumer, and hold out-of-order events until earlier ones arrive.
C) Replace the event stream with a Saga where each step waits for the previous step’s completion signal before firing the next.
D) Add an event version or timestamp and make the state machine idempotent plus order-agnostic by rejecting stale transitions and accepting any arrival order.
All four are patterns I have seen in real production systems. Only one actually fits this problem. One of them is the trap that catches senior engineers on the whiteboard.
Pick one, A, B, C, or D, and tell me why. Full breakdown in the comments.
Drop your answer 👇
Explanation
SQS Standard vs FIFO
Standard SQS is like a normal line where people can sometimes get slightly out of order. It gives best-effort ordering, meaning 99% of the time the sequence is correct but 1% of the time messages may arrive out of order. It is cheap and very fast.
FIFO SQS is a strict line. Whoever comes first gets served first. But it is slower and more expensive.
Consumers and Workers
These are background programs that pick messages from the queue and update the database. In this scenario, 5 workers are processing messages in parallel.
State Machine
This is a strict set of rules. For example, it says “I will only accept a Cancelled state if a Created state already exists.” This prevents data corruption.
Decoding the Options
Now let us fix this problem by looking at each option.
Option A: Switch to SQS FIFO plus MessageGroupId per order_id ❌ (The Senior Engineer Trap)
You replace the standard queue with a FIFO queue. MessageGroupId ensures all messages for one order_id stay in the same line.
AWS will guarantee that Created processes first, then Paid, then Cancelled.
But here is the trap. FIFO queues have much lower throughput than standard queues, around 3000 messages per second per group. If you have 2000 orders per minute with 3 messages each, the FIFO queue becomes a bottleneck. The system slows down. This is a band-aid solution, not a permanent fix.
Option B: Add a sequence number plus reorder buffer in the consumer ⚠️ (Too Complex)
You send a sequence number with every message and keep a buffer in the consumer. If Cancelled arrives first, you store it in the buffer and wait for Created to arrive.
But this is very hard to implement correctly. If the Created message never arrives due to a network issue, the Cancelled message stays stuck in the buffer forever causing memory leaks or deadlocks. Distributed systems avoid such complex buffers.
Option C: Replace the event stream with a Saga ❌ (Wrong Use Case)
Saga pattern is used when you have multiple microservices like Order Service, Payment Service, and Inventory Service and need to manage distributed transactions between them.
But here the problem is queue ordering, not distributed transactions. Saga is overkill for this situation.
Option D: Add event version or timestamp and make the state machine idempotent plus order-agnostic ✅ (The Winner)
You make your state machine smarter and more flexible.
Order-agnostic means the state machine does not care about event arrival order. If Cancelled arrives first, it accepts it and sets the order directly to Cancelled state even if Created comes later.
Idempotent means even if the same event arrives 10 times, the result stays the same.
You fixed the root cause which was a fragile state machine. Now you can use standard SQS for high throughput without worrying about out-of-order messages crashing the system.
The Correct Answer and Logic
Correct Answer: D) Add an event version or timestamp and make the state machine idempotent plus order-agnostic.
The logic is simple. There is a golden rule in distributed systems: Never trust the network. The network can delay messages or deliver them out of order at any time.
If your state machine is so strict that it rejects out-of-order messages, your system will always be fragile.
:))))))))))))))))) SUBSCRIBE :))))))))))))))))))))))))))
Top comments (0)