I went further and did some reading on message and streaming data. My main reading source was fundamentals of data engineering book by Joe Reis & Matt Houseley.
According to them, the key things to consider when ingesting event driven data like streaming data are:
-Schema evolution
-Late arrival data
-Ordering & Multiple delivery
-Time to live
-Message size limitations.
-Location & redundancy goals
References:
Fundamentals of data engineering book by Joe Reis & Matt Houseley.
Top comments (0)