Here’s a bunch of interesting links and articles about data that I’ve come across recently.
Martin Kleppmanm’s seminal talk from 2015, Turning the database inside out came up on my feed recently, and is still such an important work.
Going back even further, check out the original SQL paper, from 1974.
Not only do I love the clever title, but The End of the Bronze Age: Rethinking the Medallion Architecture is also a really good explanation of how "shift left" applies in the data world. If you prefer video there’s one of those too.
An interesting interview from a while back with Materialize’s CTO, Nikhil Benesch.
A useful look at the practicalities of Data Products, Data Contracts, and Change Data Capture.
I recently came across an interesting project from the European Union called Big Data Test Infrastructure. Despite the slightyly old-fashioned name, they’re doing some cool stuff with data and public services, such as this one looking at tree health in a town in Germany.
DataDog have their own proprietary event storage system called Husky. They’ve previously shared details of the ingestion process, and have recently posted how data compaction at scale is handled.
Two Apache projects were recently announced as graduating to top-level projects, including Apache StreamPark.
Excellent analysis from Jack Vanlightly looking at Why Snowflake wants streaming (specifically, Redpanda, about whom acquisition rumours are swirling).
What better way to learn the low-level details of Kafka than writing your own broker.
Confluent recently launched a VSCode plugin which now supports Kafka clusters too (not just Confluent Cloud).
A fantastic deep-dive blog on Kafka transactions.
A nicely explained and illustrated guide to windowing in Kafka Streams.
Mickael Maison has now been writing the Kafka Monthly Digest for an impressive seven years!
Hyprstream is a built on Apache Arrow Flight and DuckDB for "real-time data ingestion, windowed aggregation, caching, and serving". Read the associated paper here.
Uber run 2,300 MySQL clusters— this post has details of how they do it.
Top comments (0)