Event-sourcing benefits and drawbacks

Quotes from my colleagues.

Benefits of event-sourcing:

• Puts your domain first / DDD I think the repository pattern + all changes happening via a consistent interface within an Aggregate enforce domain boundaries in ways not possible in more typical data modelling.
• Built in structure / layers (don’t have to decide from a million patterns of how to structure your app)
• Easily decomposable into asynchronous components
• Each read projection optimized for access pattern

• Crazy optimized reporting / aggregation projections
  I’d put a conditional in there—projections can be optimised for access patterns. It’s still possible to build inefficient projections with ES. (Also, “crazy optimised” is unsubstantiated—what sort of optimisations are possible because of ES?)
  Not sure I'd agree, you can still make a poor domain model in ES, and in fact it's very hard to get the model right, and the consequences of a poor model are heavy.
  For example, (and I've never taken part in the modelling process so take this with a grain of salt) deciding on where an aggregate boundary lies is crucial. If you get it wrong, how do you change it?

• If we find a problem with the table structure, we can delete and reproject - futureproof

• Always retain history

• Very often want to be able to view things how they looked in the past

• Sometimes you don’t know how you might want to recut old data in the future - how could we ever dream of throwing it away? We’ve gotten used to this strange habit of throwing away data.

• Audit trail

• Incredibly useful for debugging what happened in production
  That’s not guaranteed—if there’s an issue between the ui / transport layer of your code and your event store, the event store won’t tell you anything

• User said “I did X”, we can prove if that was or wasn’t true

• “Highest fidelity” data from which everything else can be derived

• Natively supports event-driven system to system interactions

• Can be done in traditional systems, but tacked on

• In the same way that rails “makes things simple”, our templates do the same for event-sourcing (just write commands, events, aggregates, projectors, reactors)
  This is a trait of our templates, not ES itself.

• Reactors are just another event-processor, removes many situations where you need background jobs
  I’d be careful with this argument. It’s not something you get for “free”—there’s still work involved in architecting a reactor to be truly asynchronous and parallelizable, there’s still infrastructure involved. This feels more like a “how” argument than “why.”

• Sagas give us a way of managing complex state machines where events are spread out over time (provide example)

Great point - I'm not sure what the limitations of materialized views are (can you create any SQL structure?) but doing this means you're adding a new concept / moving part to your traditional system, but in an event-sourced system, it's "just another projection", meaning less cognitive load.
But the types of reports I'm talking about are things like SRI or the reporting in Development Plans that let you filter by date - so the reporting projection builds up a query optimized table that allows filtering by date, something that's not possible in systems that throw away data - you could argue this is more about "not throw away data" or a nice example of why "not throwing away data" is beneficial

Drawbacks of event-sourcing

• New paradigm to learn - knowledge gap
• Eventually consistent UIs
• We have ways to work around this (explain commands returning a sequence number and projections waiting until they’ve caught up to that sequence number)
• Opens up the opportunity to push to the client (Google Firestore etc.)
• You get stuck with your model if you get it wrong or things shift over time
• “Re-partition the event store” is possible, but we’ve never done it - I believe we need to get better at this and embrace mistakes
• This is the eventsourcing version of a schema migration

At the Kafka session a few weeks ago, I think some comments were:
unmanageably large set of events
badly structured events (I think that's your 'stuck with your model'?)
Each service has its own, basically identical, huge, projected store
complicated to manage flow of events - a change that generates a large number of events can choke the normal operation of the service

It might be useful to deblur a bit here. Event sourcing != publishing and consuming an event feed via Kafka. Benefits and drawbacks of that approach probably warrants its own discussion.

I think this one New paradigm to learn - knowledge gap is the one that carries the most weight in my view. There are ways we can address this but it means you need to be really clear that the benefits for your domain outweigh the cost of this one.

I’d love to see us find something scalable to help onboard new engineers who might be working with event sourced systems. I think that would help a lot with hiring.

From what I saw with Develop one of the things that made me uncertain about how Event Sourcing was being done was where you were using the state of a projection to decide whether to commit a command or not. That felt super race-condition-y and unpredictable.
I also didn’t like how it seemed like Event Sourcing created a sense of purity in the backend but that because of the weak guarantees of eventual consistency, the front-end had a much harder time and had to either reproduce the business logic in the front-end, cross its fingers and hope that if it fetched now the data would be up to date, or add boilerplate retrying to every query to work around it.
I got the sense that binding an Event Sourced system directly to an interactive UI perhaps wasn’t a great fit, since when users interact with the system they expect to receive feedback. The fire-and-forget nature makes this really hard.