Most Delta Lake explanations start from the feature list - ACID, time travel, schema enforcement - and ask you to take them on trust. Go the other way, start from what is physically on disk, and every one of those features turns out to be the same mechanism wearing a different hat.
A table is a directory
There is no database server involved, and no hidden proprietary store. A Delta table is a directory on cloud storage.
The data is ordinary Parquet
Every write drops one or more Parquet files into that directory. Parquet is an open columnar format, and crucially, nothing inside a Parquet file knows it belongs to a table. On its own, that directory is just a pile of files.
The log is the table
Beside the data files sits a directory named _delta_log. Every successful write appends one JSON file to it, named by version number and zero-padded to 20 digits - so the first commit is 00000000000000000000.json. One commit file is one version of the table.
Inside a commit
Each commit holds a list of actions. The two that do the work:
-
add- this Parquet file is now part of the table. It carriespath,partitionValues,size,modificationTime,dataChange, and optionallystats(per-column statistics). -
remove- this file is no longer part of the table.
Alongside them, metaData carries the schema and protocol carries the reader and writer versions a client needs to support. The spec defines others too - commitInfo, txn, cdc, domainMetadata, sidecar, checkpointMetadata - but add and remove are the ones that decide what you read.
How a read actually works
A reader never just lists the directory and reads what it finds. It reads the log first, replays the add and remove actions to work out which Parquet files are current, and only then reads those files.
That indirection is the whole design. Membership in a table is a log decision, not a filesystem fact.
Removed does not mean deleted
A file named in a remove action is still sitting on storage. It is simply no longer in the current set.
This is exactly why time travel works: replay the log only up to an older version, and you get that version's file set, still physically present. VACUUM is what eventually deletes them, and its default retention threshold is 7 days - it removes only files no longer referenced by the table, never files the current version depends on.
It also explains the warning people run into: vacuum aggressively enough and you delete the files older versions still point at, so time travel stops working for those versions.
Checkpoints
Replaying thousands of JSON commits on every read would be slow. So Delta periodically writes a checkpoint: a Parquet file in the log directory holding the whole table state at that version. A reader starts from the newest checkpoint and replays only the commits after it. A _last_checkpoint file points at the most recent one, so the reader does not even need a directory listing to find it.
Everything else follows
- ACID - a commit is one atomic file write. It either lands or it does not.
- Time travel - old versions are still described by the log, and their files are still there.
- Schema enforcement - the schema lives in the log, so a write that does not match can be rejected before anything lands.
The one-line answer
A Delta table is Parquet data files plus a transaction log, and every Delta feature is just something that log records.
Watch it drawn step by step
The folder opened up one layer at a time, 2:29: https://www.youtube.com/watch?v=nVNhgjdbYN4
Episode 5 of a data engineering interview prep series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4
Top comments (0)