DEV Community

Discussion on: You should (probably) fire your Kafka developer

 
polterguy profile image
Thomas Hansen

YAGNI comes to mind, as in "You Ain't Gonna Need It". It seems you've already concluded down that path yourself as you're writing the same in your own comment ...?

It's always simpler to implement request/response type of services in a multi machine/process context. Adding out of process pub/sub to the mix, especially if it's just "the icing on the cake", sounds like something that only adds complexity ...

But, don't take my advice, you know this better than me, and it depends upon the problem at hand ...

Thread Thread
 
polterguy profile image
Thomas Hansen

I thought some more about this, and in a CMS, headless or not, it's rarely important to have very fresh data - Implying if the data is stale for 5 minutes or not, is almost irrelevant. Cache in these regards would probably solve a lot of your problem, since from the client's point of view, pulling (quite frequently) is a no brainer.

There's only one place in our own stuff that we've got pub/sub stuff, which is during creation of a cloudlet. See screenshot below. And even there it's more for "show", and/or to ensure the client gets feedback, since it's a process that might take a couple of minutes - However, even in these parts we're using "simple web sockets" and no fancy message broker ...

Creating cloudlet

You can see it with your own eyes if you create a cloudlet here ...

Thread Thread
 
pyrsmk profile image
Aurélien Delogu

I'm replying here to your both comments ;)


This doesn't add any complexity on the backend since the LISTEN/NOTIFY mechanism is baked by default with first service (which is a simple REST server). But yes, on the frontend it adds its layer of complexity.

Why we need LISTEN/NOTIFY ? Because it is needed to plug other services like a log service, a webhooks service, or anything else. All those services are actually needed in the system, but for simplicity it is better to separate them from the base service since they are not needed to run the first layer.

In relation to the pub/sub service, it is indeed icing on the cake and could be completely removed from the stack. At first, I wanted it to have consistency in cache data hosted by the clients (which come from the REST service). This is needed to support offline connectivity for the mobile application mostly. But, in reality, offline connectivity is really hard to build and, IMHO, offline-first is a chimera. The only way to do it well is by caching the data when it's requested by the user iteratively (and not all at once). Because of that, it is impossible to foresee what data the user will need when its device becomes offline and the only way to have access to the right data is by improving the UX of the application so the user is responsible of what he'll need and must retrieve the relevant data beforehand.

When I first analyse the pub/sub service, I wasn't aware of this. But now that I'm thinking this through it seems this service does not meet any real need. It is just cool to have data automatically updated on clients when someone modifies it. And developing something that is just cool to have but at the price of hundred hours of work, well... Why bother :D

Thread Thread
 
polterguy profile image
Thomas Hansen

Hahahahaha, I love the way your brain made its way to a conclusion here ... :D