Why Event Sourcing is a microservice communication anti-pattern

Oliver Libutzki on June 28, 2019

Event-driven architectures in general and Event Sourcing in particular gain traction in the last couple of years. This trend is caused by the fact ... [Read Full]
markdown guide
 

The argument you're making, as I understand it, seems to fall along these lines:

  1. Event Sourcing is a persistence solution
  2. Publishing the Event Sourcing event is akin to publishing your database and therefore an anti-pattern
  3. Instead, if you want to publish events, use alternative X or Y

Based on my experiences with Event Sourcing, I think we may disagree on a few things.

(1) On Event Sourcing being a persistence solution. I don't know if I would agree with this. Instead, I would suggest that Event Sourcing is a set of practices which begins with domain modeling and includes persistence, but is not solely persistence. If you ignore the DDD and modeling elements of Event Sourcing, then I think much of the value is lost.

This really leads to the next issue.

(2) Publishing Event Sourcing is akin to publishing your database. I read the reference article you provided on Domain Events vs Event Sourcing. When I look at the examples the author gives, the Event Sourcing events are named MobileNumberProvided, VerificationCodeGenerated, MobileNumberValidated. While these are technically "events", I would argue this is missing the point of Event Sourcing and are an anti-pattern. Let's call this technique Property Sourcing. Property Sourcing is not Event Sourcing. And if a team is using Property Sourcing, I'll absolutely agree that these events should not be published.

YMMV, from my experience with Event Sourcing, what should be persisted are true domain events like OrderAccepted from your example.

And since these are true domain events, I would expect that from within the BoundedContext, that no other name for OrderAccepted would be used e.g. ubiquitous language. So whether I'm using event sourcing or not, the code within the Bounded Context will be using these events to communicate. However, since I'm already writing these events, combing this with Event Sourcing is often very natural.

Which leads into the last point.

(3) Instead of publishing Event Sourcing events, publish X or Y instead. This may be semantics, but if you accept for a moment that Event Sourcing Events == Domain Events, then are many valid reasons that you may want to publish them as is: (a) analytics teams who want data in the streaming format that event sourcing provides, (b) security teams that may want to perform time critical operations on the data, and perhaps (c) even other micro services that have requirements for a high level of granularity. So saying publishing Domain Events is an anti-pattern probably goes too far. It depends.

Finally, as you point out, there are also many cases where a translation would be appropriate in which case generating Integration Events like those you've described would also be acceptable.

So while I think a number of the points you're making are valid, it sounds to me as if your assumptions are leading you to believe Event Sourcing is an anti pattern when in reality it's quite a useful pattern when applied appropriately.

 

Thanks for your valuable comment. I don't want to discredit Event Sourcing in general. I just want to point out that it's not a silver bullet and should be used with care.

You are absolutely right that Event Sourcing is more than persistence as it utilizes you in thinking about behaviour instead of data. Nevertheless it defines a way how to persist your application state.

In practice it's often required to be able to fix a typo. In an event sourced scenario you need to declare a corresponding event as it's not possible to persist the change otherwise.

The fix might be interesting for other bounded context, so you have to expose the change via an external event, but there is a change that some internal data is fixed and the other contexts don't need to know about it at all.

Anyway, I agree with you that Event Sourcing systems fail because people just use events without reflecting behaviour. They just trabsfer the CRUD mindset into events (OrderCreated, OrderUpdated, OrderDeleted).

 

I completely agree that having to fix already posted events is a pain. However, there are a few techniques that will typically offset this pain:

  1. Use a serialization format that supports schema migration. I most often use protobuf, but avro is another good choice. This would take care of your typo issue without having to modifying any of the existing events.

  2. Ensure that event includes just the facts and not derived information. For example, I might have a domain event Deposit which would includes the property, Amount. This is a nice fact that everyone can agree upon.

I could potentially include a Balance field in the Deposit event to indicate the balance after the deposit has been made, but because Balance is a derived value (the sum of all Deposits and Withdrawals), it's often subject to re-interpretation which might cause me to have to republish the event.

By sticking to facts, I find it greatly reduces my need to have to modify events. And most modifications for me are because my understanding of the domain has improved.

 

Event Sourcing as a top level architecture is bad. It's meant to be used in a bounded context or in a business unit inside a bounded context. This has been said many times by Greg Young and Udi Dahan for almost a decade. udidahan.com/2012/02/10/udi-greg-r...

 

Exactly, the talk by Greg Young is linked at the end of the post. What I say is nothing new, it's just the combination of personal experience and existing references.

 

From the title, I was prepared to disagree with you. Because there is nothing wrong with using Event Sourcing inside a microservice. But I do agree with your conclusion. A service's persistence details should remain private. Anything that is shared publicly should be separate, because it is used for a different purpose and may change for different reasons. This is a core Separation of Concerns.

Also, thanks for the links. I look forward to reviewing some of these I haven't seen.

 

Thanks for your feedback. I adjusted the title to Why Event Sourcing is a microservice communication anti-pattern.

 

This ties in with how I’ve always thought of event sourcing in Microservices. I’ve never really heard of it being used as a communication pattern. We might share the same event store for services that use it, but each service would have its own event stream, which is perfectly fine, like multiple services using separate databases on the same dB server. But each service should definitely be responsible for storing and replaying its own events if event sourcing makes sense for that service.

 

I’ve never really heard of it being used as a communication pattern.

Take a look at 5. Composing services becomes trivial from Event Sourcing: What it is and why it's awesome which is the first hit if you search for Event Sourcing at dev.to. The post has more than 100 likes.

I don't want to discredit the post, but it's an example for mixing the concepts of Event Sourcing and Domain Events.

 

Sorry this is my first time on this site so I never saw that post. I agree that part of that post is off where they seem to be conflating the benefits of an event driven system with the benefits of event sourcing, which is not correct. There can be some overlap, many times we will take a Domain Event that is received by a service and stick that in our event stream, so we can rebuild the state of that service by replaying all of the inputs to the service. But it depends on the nature of the service if that makes sense. And certainly we would not expose that event stream outside of that service.

 

Using a shared event store between bounded contexts is not like using a shared relational database at all.

If you share a relational database between services and then you change some schema of some table other services will blow up but if you share event store between services and you change schema of an event for example from V1 to V2 then nothing bad will happen. sharing an event store is more like sharing a queuing system as greg young mentions in this discussion:

groups.google.com/forum/#!searchin...

Ofcourse the two patterns that you mentioned in this article are perfectly valid but what i want to say is bounded contexts should only have private data therfore there is no antipattern about this, its only a trade off.

Having shared eventstore may have many advantages like having eventstore projections between multiple bounded contexts or simple event tracing via correlation id and causation id inside or between contexts.

There are many intresting talks around this subject by Greg Young , Chris Richardson and David Schmits.

Greg Young: talking about advantage of single event store around bounded contexts in µCon London 2017.
skillsmatter.com/skillscasts/11143...

Chris Richardson: Developing Functional Domain Models with Event Sourcing
youtube.com/watch?v=kQO0WnPo4sw

David Schmitz: Event Sourcing You are doing it wrong
youtube.com/watch?v=GzrZworHpIk

 

There is nothing wrong with a shared event store. I just argue that the events you produce in an event stored entity are rarely suitable for being exposed and shared.

Otherwise you are very limited in the freedom of evolving those events

 

I`m totally agree with you on this, but think about benefits that you may have when you publish all of your internal events to out world, for example simplicity of tracing events is a huge benefit when you have only one event store across all services and services dont have any local event store. If you separate local event store from global event store you may have challenges to trace what local event causes what global event and so on and this is only a simple example of many serious complexities that you may have.

I just want to say having one event store across all microservices without ES per service is not like shared relational database and also anti-pattern, it's only a trade off between simplicity and scalability.

You are right, but because of modularity a lot of challenges arise. Example: At first glance it might be easier and quicker to simply join two tables instead of thinking about an API.

Nevertheless it's better to modularize strictly as the benefits outdo the challenges.

 

Good article I totally agree with what you said, Event Sourcing events are different from Domain Events (one is related to the functioning of aggregates while the other in the integration between bounded contexts) and should be treated as such

 

For me, it boils down to "If you publish all of your events, you can never stop publishing them, because someone else might be relying on them. Even if you don't need them internally any more, having the data in new events!"
So having a translation layer, which at first can just filter events on type, is helpful to prevent this big maintenance problem.

 

Good job at putting all of these togheter Oliver, this is a controversial and confusing topic for a lot of developers out there.

I cheekly identified this as being a naming induced problem, there is a big craze at naming everything events at the moment that most people confuse them as being scoped for out of system use whereas event is a better use of the term data or variable.

 

You've written that event sourcing events shouldn't be propagated outisde the bounded context.
I have a few questions.

  1. Is it possible to use ES events for changing state of aggregates in the same BC by using event handlers or process managers?
  2. How and where should the domain events be created? Should they be created in aggregates after applying ES events? Shall I store them in aggregates and dispatch after ES Events are persisted? Or maybe they should be created and dispatched in application layer after saving the aggregate's state?
 

Regarding 1. Yes, I think so. The scope of those events doesn't have to be restricted to a single aggregate.
Regarding 2. Good question. To be honest: I don't have an opinion on that one. It's a question of responsibility. And one can argue that it's the aggregate's responsibility to publish domain events, but you can also argue that it's sufficient, if the aggregate publishes ES events and another component translates it a domain event (if needed).

 
[deleted]
 

Regardless if the events are local or global they need to have a creation timestamp.

In general it's desirable to rely as less as possible on the order of events. While the events emitted by a single aggregate need to be handled sequentially, you should not make assumptions in which order events of other bounded contexts occur.

 

I think the core underlying problem is that there is no standard for asynchronous communication among services. For example how is operation that can take long time is exposed and invoked. Queuing and event sourcing are attempts to workaround this limitation.

I think the real solution is to expose asynchronous service API explicitly and use an orchestration technology to implement the business transaction using such APIs.

I would recommend looking at Cadence Workflow open source platform that supports such orchestration using natural code.

 

Don't totally agree. If you use something like avro, and make sure the schema's are backwards compatible consumers can use the newer schema when they want/need.

 

See my answer concerning API evolution on Twitter:

 

I hope nobody thinks exposing your event sourcing events 1:1 to an external API is a good idea. Still evolution to the event sourcing objects should be controlled if used by multiple teams.

 

U r right on the point event sourcing should be within the bounded context of the service otherwise it is an anti pattern.

 

Isn't this sometimes differentiated as domain events vs integration events?

 

I don't think so. Domain Events vs. Integration Events in Domain-Driven Design and microservices architectures deals a lot with ACID transactions to clarify the difference between a domain event and an integration event.

I don't make assumptions concerning ACID transactions. A bounded context might work with eventual consistency internally as well.

 

It sure helps if you event source. CQRS/ES is at the heart of information systems. I wrote about it here: EventModeling.org

 
code of conduct - report abuse