A data lake is cheap object storage full of files. It is great for volume and terrible for correctness. Two jobs writing at once can corrupt a table, a reader can see a half-finished write, and there is no clean way to update or delete rows. The lakehouse idea is to keep the cheap storage but add a transaction layer on top, so a pile of files starts behaving like a real table. Delta Lake is one way to do that.
The core problem
Object storage gives you durable, cheap, effectively infinite space, but it gives you almost no guarantees about a collection of files as a unit. There is no atomic "replace these ten files with these twelve". A writer that fails halfway leaves the table in a broken state. A reader scanning the directory during a write sees an inconsistent mix. Appends are easy, but updates and deletes mean rewriting files, and doing that safely while others read is the hard part.
Classic warehouses solved this decades ago with a transaction manager, but they store data in proprietary formats on managed storage and cost more. The question is whether you can get the same ACID guarantees while keeping open file formats on plain object storage.
Key design decisions
A transaction log is the source of truth, not the file listing. Delta keeps an ordered log of commits alongside the data files. Each commit records which files were added and which were removed. The current state of the table is not "whatever files are in the directory" but "the result of replaying the log". A reader consults the log to learn exactly which files make up the table right now, so it never sees a partial write.
Writes are atomic through the log. A writer stages new data files, then makes one atomic commit that appends an entry to the log. Until that entry lands, readers see the old version. The moment it lands, they see the new one. There is no in-between, which is what turns a set of file operations into a single transaction.
Optimistic concurrency for multiple writers. Delta assumes conflicts are rare. Each writer reads the current log version, does its work, and tries to commit at the next version number. If someone else committed first, the loser detects the collision, checks whether the changes actually conflict, and retries against the new version. This avoids heavy locking while still keeping writers from clobbering each other.
Updates and deletes without breaking readers. Because the log tracks add and remove actions, an update rewrites only the affected files and commits a swap: remove the old files, add the new ones, atomically. Old files stick around for a retention window, which gives you time travel and rollback, since you can replay the log to an earlier version.
The trade-offs
The log is powerful but it is also a bottleneck you have to manage. Every commit appends to it, and over time it grows, so the system periodically checkpoints the log into a compact snapshot to keep reads fast. Small frequent writes create many small files and many log entries, which hurts read performance until a compaction job merges them. You trade write simplicity for a background maintenance burden.
Optimistic concurrency is cheap when writers rarely touch the same data and expensive when they do. Under heavy contention on the same files, writers keep retrying and throughput drops. If your workload is many writers hammering the same partition, a lock-based store may serve you better.
You also inherit object storage's latency and its consistency behavior. Metadata operations that a warehouse does in memory become log reads and file operations, so single-row lookups are not the point. The lakehouse wins on large scans over open formats, not on transactional point queries.
How the real system does it
The real pattern stores data as open columnar files on object storage, with an ordered transaction log that defines table state, atomic log commits for isolation, optimistic concurrency with conflict detection for multiple writers, and periodic checkpointing and compaction to keep the log and files healthy. Time travel falls out naturally from retaining old file versions the log still references.
The broad lesson: you do not need a new storage engine to get transactions. A well-designed log over immutable files can give you atomicity, isolation, and versioning on top of the cheapest storage you have.
I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-databricks
Top comments (0)