I read this more as if the client is retrying the requests. A message queue would (to the client) silently retry. Waiting for the response could potentially take a long time.
If the client is implementing this pattern it can keep the user informed of what's going on, instead of having the users wait potentially a few seconds for the request to finish. This also has the benefit of leaving the user experience to the client (long response times in general being detrimental to UX).
I prefer message queues for deferred work where the client doesn't really care about getting an immediate result.
It always depends... whether the user does or does NOT care about the result.
Let's make this more concrete:
Say when a user places an order, he'll put his credit card and pay the money, so he really cares about the response.
Placing an order would require a transaction across multiple tables (to deduct the quota of the product and create an order at the same time).
But but but, if you use a such transaction, your system will be doomed to be a non-scalable one as this might cause a deadlock, a bad one actually!
So yeah, although the user cares about the response, you will really need an MQ to avoid deadlocks and return back the result when it comes even after few seconds.
There are definitely some subtleties in different patterns. One of the reasons I wanted to share a little bit more about patterns is I think too often folks assume shared understanding. Patterns are not algorithms. They are ways we talk about the architecture we are building.
It's really hard to keep descriptions about a single pattern specific and clear but the example with financial transactions you are giving is why I brought up idempotency. I could have gone into greater detail there but it's a delicate balance of info loading and providing enough information at appropriate times. With transactions, they are definitely not idempotent in nature. If you keep applying a -$5 charge for coffee against an account, you'll quickly rack up a lot of credit charges. So it's a balance between retries, fast fails, and complete roll backs.
I'm curious though; why bring up the message queue pattern here? Retry to me is a lower level pattern. How do you communicate to a message queue? What compromises do you make when sending events into the message queue?
You're absolutely right, patterns are just like a way of communication.
why bring up the message queue pattern here?
How do you communicate to a message queue?
What compromises do you make when sending events into the message queue?
The retry pattern is meant for resilience.
Task queues (like Celery) already have the implementation of the retry pattern, and task queues require MQs (or brokers) to do their job.
So, to achieve a greater level or resilience, both a task queue and an MQ are necessary.
Actually, this pattern goes with couple of other design decisions (like microservices and event-driven-architecture).
Because of that, the biggest compromise when using such architecture is that your program would often need a complete rewrite.
Say for the previous example (of using a transaction to place an order), you would instead create an event "order_created".
This will trigger other services like shippingService & chargingService to do the next steps.
Suppose the chargingService failed in the middle, it will retry again (cuz the message of "charging_credit_card" won't leave the queue for couple of retries). If it ultimately fails, it will trigger another event for rolling back across multiple services.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I read this more as if the client is retrying the requests. A message queue would (to the client) silently retry. Waiting for the response could potentially take a long time.
If the client is implementing this pattern it can keep the user informed of what's going on, instead of having the users wait potentially a few seconds for the request to finish. This also has the benefit of leaving the user experience to the client (long response times in general being detrimental to UX).
I prefer message queues for deferred work where the client doesn't really care about getting an immediate result.
It always depends... whether the user does or does NOT care about the result.
Let's make this more concrete:
Say when a user places an order, he'll put his credit card and pay the money, so he really cares about the response.
Placing an order would require a transaction across multiple tables (to deduct the quota of the product and create an order at the same time).
But but but, if you use a such transaction, your system will be doomed to be a non-scalable one as this might cause a deadlock, a bad one actually!
So yeah, although the user cares about the response, you will really need an MQ to avoid deadlocks and return back the result when it comes even after few seconds.
There are definitely some subtleties in different patterns. One of the reasons I wanted to share a little bit more about patterns is I think too often folks assume shared understanding. Patterns are not algorithms. They are ways we talk about the architecture we are building.
It's really hard to keep descriptions about a single pattern specific and clear but the example with financial transactions you are giving is why I brought up idempotency. I could have gone into greater detail there but it's a delicate balance of info loading and providing enough information at appropriate times. With transactions, they are definitely not idempotent in nature. If you keep applying a -$5 charge for coffee against an account, you'll quickly rack up a lot of credit charges. So it's a balance between retries, fast fails, and complete roll backs.
I'm curious though; why bring up the message queue pattern here? Retry to me is a lower level pattern. How do you communicate to a message queue? What compromises do you make when sending events into the message queue?
You're absolutely right, patterns are just like a way of communication.
The retry pattern is meant for resilience.
Task queues (like Celery) already have the implementation of the retry pattern, and task queues require MQs (or brokers) to do their job.
So, to achieve a greater level or resilience, both a task queue and an MQ are necessary.
Actually, this pattern goes with couple of other design decisions (like microservices and event-driven-architecture).
Because of that, the biggest compromise when using such architecture is that your program would often need a complete rewrite.
Say for the previous example (of using a transaction to place an order), you would instead create an event "order_created".
This will trigger other services like shippingService & chargingService to do the next steps.
Suppose the chargingService failed in the middle, it will retry again (cuz the message of "charging_credit_card" won't leave the queue for couple of retries). If it ultimately fails, it will trigger another event for rolling back across multiple services.