A good and proper DDD / ES person would further ask the question: What is the scenario? (What problem are you trying to solve?) Because there might be a better alternative.
In an event store, you don't really update data in place if you can help it. You just keep adding new events. Depending on the size of the binary, I might do the traditional thing where you save it to file storage like S3 and then add an event when the file link changes.
Side note: The product EventStore stores the event data as binary. Meaning I actually have to serialize to string and convert to bytes before calling the method to save the event. I think internally it uses ProtoBuf to store events, which I read (but have not browsed the code to verify) handles binary data pretty efficiently.
Lead Developer and Solutions Architect, I specialise in Event Sourcing, DDD and Event Driven systems. PHP and GoLang developer. Enjoys being a smart ass and having a nice whiskey.
Location
Ireland
Education
MSc in Computer Science, Trinity College, Dublin
Work
Lead Developer and Solutions Architect at Contractor
As Kasey suggested, best practice there would be to store the file remotely on S3, and then have an event that references the remote file. This allows you to version binary files, as you just upload a new file and create a new a event.
Eg.
# File A, Version 1FileUploadedfile_id:9f9753b8-6beb-4e36-b7ca-1f6f6bf23702reference:https://link.to.file# File A, Version 2FileUploadedfile_id:9f9753b8-6beb-4e36-b7ca-1f6f6bf23702reference:https://link.to.other-file
file_id is the reference in your system, whereas reference is the remote instance of that version.
Lead Developer and Solutions Architect, I specialise in Event Sourcing, DDD and Event Driven systems. PHP and GoLang developer. Enjoys being a smart ass and having a nice whiskey.
Location
Ireland
Education
MSc in Computer Science, Trinity College, Dublin
Work
Lead Developer and Solutions Architect at Contractor
Yeah, technically. It's just a philosophy though, so there are times when it's not the appropriate philosophy.
In this case it really comes down to practicality. Is it useful to store the full binary file in the event log? Does that give any value? If the answer is no, then there's no point in saving the file in the log, just store a reference.
You can have different sources of truth for each "domain" or specialty. The source of truth for files is the file system... if it is gone from there, it doesn't matter what the database says. :)
It may still be important to (event-sourced) areas of the business to record that something changed and perhaps trigger a further action. You can set this up in a number of ways. The client could issue an API call after successful saving of the file (this is request-driven, probably not the way I would go). Or you could setup event notification on file operations (this is event-driven) -- S3 supports this or just use a file watcher for local apps.
At this point, this is really integration between two systems and no longer event-sourcing. Instead it is Event-Driven Architecture. Event sourcing really only applies inside individual domains, not across different systems. This is probably why you already had an inkling that event-sourcing would not solve the file management problem. By itself, it won't.
We have an audit requirement that it should not be possible to change data without a trace, so storing the file separately was a problem for us until we realized that we just have to store the file sha256 hash and that way we can check if the file is the right one. So we get the best of both worlds.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
how do event stores handle binary data updates efficiently?
A good and proper DDD / ES person would further ask the question: What is the scenario? (What problem are you trying to solve?) Because there might be a better alternative.
In an event store, you don't really update data in place if you can help it. You just keep adding new events. Depending on the size of the binary, I might do the traditional thing where you save it to file storage like S3 and then add an event when the file link changes.
Side note: The product EventStore stores the event data as binary. Meaning I actually have to serialize to string and convert to bytes before calling the method to save the event. I think internally it uses ProtoBuf to store events, which I read (but have not browsed the code to verify) handles binary data pretty efficiently.
The use-case was to allow versioning of files (pdf, avi, mpg, mp3, wav, doc, xml, etc.)
As Kasey suggested, best practice there would be to store the file remotely on S3, and then have an event that references the remote file. This allows you to version binary files, as you just upload a new file and create a new a event.
Eg.
file_idis the reference in your system, whereasreferenceis the remote instance of that version.I see.
Doesn't this work against the "one source of truth" philosophy?
Yeah, technically. It's just a philosophy though, so there are times when it's not the appropriate philosophy.
In this case it really comes down to practicality. Is it useful to store the full binary file in the event log? Does that give any value? If the answer is no, then there's no point in saving the file in the log, just store a reference.
You can have different sources of truth for each "domain" or specialty. The source of truth for files is the file system... if it is gone from there, it doesn't matter what the database says. :)
It may still be important to (event-sourced) areas of the business to record that something changed and perhaps trigger a further action. You can set this up in a number of ways. The client could issue an API call after successful saving of the file (this is request-driven, probably not the way I would go). Or you could setup event notification on file operations (this is event-driven) -- S3 supports this or just use a file watcher for local apps.
At this point, this is really integration between two systems and no longer event-sourcing. Instead it is Event-Driven Architecture. Event sourcing really only applies inside individual domains, not across different systems. This is probably why you already had an inkling that event-sourcing would not solve the file management problem. By itself, it won't.
We have an audit requirement that it should not be possible to change data without a trace, so storing the file separately was a problem for us until we realized that we just have to store the file sha256 hash and that way we can check if the file is the right one. So we get the best of both worlds.