Well, it really depends on what you want to achieve, and what your bottlenecks are. Message broker solutions are optimized for the exact purpose of managing/transforming/pushing messages efficiently, while writing your own service with a traditional database and a simple REST server won't be optimized at all for the same purpose.
But I agree with the fact that you leveraged in a previous article of yours : never over-engineer things ! Existing solutions have been developed with a specifc use case in mind (or even trying to solve too many use cases). So it rarely fits well your needs and you usually can create your own microservice for your own usage : it will be cheaper and faster to implement and maintain.
Thank you, and yes of course, that was my exact point. If you need a message broker, by all means use one - However, my experience is that you rarely need one, and another of my experiences is that way too many apps have one ... :/
I have no other explanation for the latter than the "coolness factor", which of course is a serious waste of resources, and quite frankly lack of respect for colleagues and employers, which will have to wait sometimes as much as 10x time before they have solved their business requirements ... :/
I'm not really against anything, I'm just advocating "the middle road", as in avoid making things more complicated than what's required ...
I completely agree with you. Many bad choices are made because something is the "new shit".
Anyway, I want to have your thoughts on something I analysed during the past year (I have not implemented it yet, for some reasons that are not really interested here). I worked on a headless CMS system where I wanted to have a kind of pub/sub mechanism. It is mostly a subscribe service where there is only one publisher: the PostgreSQL server that notifies listeners when something has changed on some tables (this is handled by procedures). I need this so the clients are notified in realtime to improve user experience. And, of course, this would be implemented with "circuit breaker" in mind: this service is just the icing on the cake and it is not needed for the system to work reliably.
Now. What are your thoughts about this ? I'm really interested on those.
For those who read us: the NOTIFY command of PostgreSQL has a time complexity of O(N^2), so it could have performance impact when implementing a LISTEN/NOTIFY system in PostgreSQL without having it in mind (this issue has already been notified to the dev team several years ago and it seems that nothing has changed on the subject yet). Here's a recent thread on the issue.
YAGNI comes to mind, as in "You Ain't Gonna Need It". It seems you've already concluded down that path yourself as you're writing the same in your own comment ...?
It's always simpler to implement request/response type of services in a multi machine/process context. Adding out of process pub/sub to the mix, especially if it's just "the icing on the cake", sounds like something that only adds complexity ...
But, don't take my advice, you know this better than me, and it depends upon the problem at hand ...
I thought some more about this, and in a CMS, headless or not, it's rarely important to have very fresh data - Implying if the data is stale for 5 minutes or not, is almost irrelevant. Cache in these regards would probably solve a lot of your problem, since from the client's point of view, pulling (quite frequently) is a no brainer.
There's only one place in our own stuff that we've got pub/sub stuff, which is during creation of a cloudlet. See screenshot below. And even there it's more for "show", and/or to ensure the client gets feedback, since it's a process that might take a couple of minutes - However, even in these parts we're using "simple web sockets" and no fancy message broker ...
This doesn't add any complexity on the backend since the LISTEN/NOTIFY mechanism is baked by default with first service (which is a simple REST server). But yes, on the frontend it adds its layer of complexity.
Why we need LISTEN/NOTIFY ? Because it is needed to plug other services like a log service, a webhooks service, or anything else. All those services are actually needed in the system, but for simplicity it is better to separate them from the base service since they are not needed to run the first layer.
In relation to the pub/sub service, it is indeed icing on the cake and could be completely removed from the stack. At first, I wanted it to have consistency in cache data hosted by the clients (which come from the REST service). This is needed to support offline connectivity for the mobile application mostly. But, in reality, offline connectivity is really hard to build and, IMHO, offline-first is a chimera. The only way to do it well is by caching the data when it's requested by the user iteratively (and not all at once). Because of that, it is impossible to foresee what data the user will need when its device becomes offline and the only way to have access to the right data is by improving the UX of the application so the user is responsible of what he'll need and must retrieve the relevant data beforehand.
When I first analyse the pub/sub service, I wasn't aware of this. But now that I'm thinking this through it seems this service does not meet any real need. It is just cool to have data automatically updated on clients when someone modifies it. And developing something that is just cool to have but at the price of hundred hours of work, well... Why bother :D
Well, it really depends on what you want to achieve, and what your bottlenecks are. Message broker solutions are optimized for the exact purpose of managing/transforming/pushing messages efficiently, while writing your own service with a traditional database and a simple REST server won't be optimized at all for the same purpose.
But I agree with the fact that you leveraged in a previous article of yours : never over-engineer things ! Existing solutions have been developed with a specifc use case in mind (or even trying to solve too many use cases). So it rarely fits well your needs and you usually can create your own microservice for your own usage : it will be cheaper and faster to implement and maintain.
Thank you, and yes of course, that was my exact point. If you need a message broker, by all means use one - However, my experience is that you rarely need one, and another of my experiences is that way too many apps have one ... :/
I have no other explanation for the latter than the "coolness factor", which of course is a serious waste of resources, and quite frankly lack of respect for colleagues and employers, which will have to wait sometimes as much as 10x time before they have solved their business requirements ... :/
I'm not really against anything, I'm just advocating "the middle road", as in avoid making things more complicated than what's required ...
I completely agree with you. Many bad choices are made because something is the "new shit".
Anyway, I want to have your thoughts on something I analysed during the past year (I have not implemented it yet, for some reasons that are not really interested here). I worked on a headless CMS system where I wanted to have a kind of pub/sub mechanism. It is mostly a subscribe service where there is only one publisher: the PostgreSQL server that notifies listeners when something has changed on some tables (this is handled by procedures). I need this so the clients are notified in realtime to improve user experience. And, of course, this would be implemented with "circuit breaker" in mind: this service is just the icing on the cake and it is not needed for the system to work reliably.
Now. What are your thoughts about this ? I'm really interested on those.
YAGNI comes to mind, as in "You Ain't Gonna Need It". It seems you've already concluded down that path yourself as you're writing the same in your own comment ...?
It's always simpler to implement request/response type of services in a multi machine/process context. Adding out of process pub/sub to the mix, especially if it's just "the icing on the cake", sounds like something that only adds complexity ...
But, don't take my advice, you know this better than me, and it depends upon the problem at hand ...
I thought some more about this, and in a CMS, headless or not, it's rarely important to have very fresh data - Implying if the data is stale for 5 minutes or not, is almost irrelevant. Cache in these regards would probably solve a lot of your problem, since from the client's point of view, pulling (quite frequently) is a no brainer.
There's only one place in our own stuff that we've got pub/sub stuff, which is during creation of a cloudlet. See screenshot below. And even there it's more for "show", and/or to ensure the client gets feedback, since it's a process that might take a couple of minutes - However, even in these parts we're using "simple web sockets" and no fancy message broker ...
You can see it with your own eyes if you create a cloudlet here ...
I'm replying here to your both comments ;)
This doesn't add any complexity on the backend since the LISTEN/NOTIFY mechanism is baked by default with first service (which is a simple REST server). But yes, on the frontend it adds its layer of complexity.
Why we need LISTEN/NOTIFY ? Because it is needed to plug other services like a log service, a webhooks service, or anything else. All those services are actually needed in the system, but for simplicity it is better to separate them from the base service since they are not needed to run the first layer.
In relation to the pub/sub service, it is indeed icing on the cake and could be completely removed from the stack. At first, I wanted it to have consistency in cache data hosted by the clients (which come from the REST service). This is needed to support offline connectivity for the mobile application mostly. But, in reality, offline connectivity is really hard to build and, IMHO, offline-first is a chimera. The only way to do it well is by caching the data when it's requested by the user iteratively (and not all at once). Because of that, it is impossible to foresee what data the user will need when its device becomes offline and the only way to have access to the right data is by improving the UX of the application so the user is responsible of what he'll need and must retrieve the relevant data beforehand.
When I first analyse the pub/sub service, I wasn't aware of this. But now that I'm thinking this through it seems this service does not meet any real need. It is just cool to have data automatically updated on clients when someone modifies it. And developing something that is just cool to have but at the price of hundred hours of work, well... Why bother :D
Hahahahaha, I love the way your brain made its way to a conclusion here ... :D