DEV Community: dino david

What a Delta table actually is: Parquet files plus a transaction log

dino david — Wed, 23 Sep 2026 15:53:35 +0000

Most Delta Lake explanations start from the feature list - ACID, time travel, schema enforcement - and ask you to take them on trust. Go the other way, start from what is physically on disk, and every one of those features turns out to be the same mechanism wearing a different hat.

A table is a directory

There is no database server involved, and no hidden proprietary store. A Delta table is a directory on cloud storage.

The data is ordinary Parquet

Every write drops one or more Parquet files into that directory. Parquet is an open columnar format, and crucially, nothing inside a Parquet file knows it belongs to a table. On its own, that directory is just a pile of files.

The log is the table

Beside the data files sits a directory named _delta_log. Every successful write appends one JSON file to it, named by version number and zero-padded to 20 digits - so the first commit is 00000000000000000000.json. One commit file is one version of the table.

Inside a commit

Each commit holds a list of actions. The two that do the work:

add - this Parquet file is now part of the table. It carries path, partitionValues, size, modificationTime, dataChange, and optionally stats (per-column statistics).
remove - this file is no longer part of the table.

Alongside them, metaData carries the schema and protocol carries the reader and writer versions a client needs to support. The spec defines others too - commitInfo, txn, cdc, domainMetadata, sidecar, checkpointMetadata - but add and remove are the ones that decide what you read.

How a read actually works

A reader never just lists the directory and reads what it finds. It reads the log first, replays the add and remove actions to work out which Parquet files are current, and only then reads those files.

That indirection is the whole design. Membership in a table is a log decision, not a filesystem fact.

Removed does not mean deleted

A file named in a remove action is still sitting on storage. It is simply no longer in the current set.

This is exactly why time travel works: replay the log only up to an older version, and you get that version's file set, still physically present. VACUUM is what eventually deletes them, and its default retention threshold is 7 days - it removes only files no longer referenced by the table, never files the current version depends on.

It also explains the warning people run into: vacuum aggressively enough and you delete the files older versions still point at, so time travel stops working for those versions.

Checkpoints

Replaying thousands of JSON commits on every read would be slow. So Delta periodically writes a checkpoint: a Parquet file in the log directory holding the whole table state at that version. A reader starts from the newest checkpoint and replays only the commits after it. A _last_checkpoint file points at the most recent one, so the reader does not even need a directory listing to find it.

Everything else follows

ACID - a commit is one atomic file write. It either lands or it does not.
Time travel - old versions are still described by the log, and their files are still there.
Schema enforcement - the schema lives in the log, so a write that does not match can be rejected before anything lands.

The one-line answer

A Delta table is Parquet data files plus a transaction log, and every Delta feature is just something that log records.

Watch it drawn step by step

The folder opened up one layer at a time, 2:29: https://www.youtube.com/watch?v=nVNhgjdbYN4

Episode 5 of a data engineering interview prep series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

Databricks classic vs serverless compute: the limitation list decides it

dino david — Mon, 21 Sep 2026 12:49:33 +0000

Pick "classic" or "serverless" in Databricks and you are choosing where the machines live. Almost everything else follows from that.

The same request, twice

You press run on a notebook or a SQL query. Databricks can serve that request two ways.

Classic. The control plane asks your cloud provider for machines, and a cluster of VMs is created inside your own account and network. It reads your tables from your own cloud storage. You choose the instance types, you own autoscaling and networking, and the bill has two parts: the cloud provider for the VMs, and Databricks for the DBUs on top.

Serverless. The documentation is direct about it: serverless compute resources run in "the serverless compute plane, which is managed by Databricks", and you run workloads "without provisioning any compute resources in your cloud account". Databricks manages, patches and scales that fleet.

Your data does not move

This is the part that reassures people: in both cases the tables stay in your own cloud storage. It is the compute that changes sides, not the data.

The list that actually decides it

Startup time is the headline, but the decision usually comes down to whether your workload uses something serverless does not support. From the limitations page:

R is not supported.
Compute-scoped libraries and compute-scoped init scripts are not supported. Notebook-scoped libraries are the alternative.
Only Spark Connect APIs are available, and the Spark RDD APIs are not supported.
Streaming triggers: Trigger.AvailableNow() is supported; Trigger.Continuous(interval) and Trigger.ProcessingTime(interval) are not.
Maximum runtime is 7 days. A run that exceeds it is terminated and not retried.

So which one

Serverless suits SQL warehouses, ad hoc exploration and spiky jobs, where waiting for a cluster is the real cost and nothing on that list applies.

Classic is the answer when you need something specific: a particular instance type or GPUs, libraries installed at cluster level, R, RDD code, a continuous stream, or your own network setup.

The one-line answer

Classic compute runs in your cloud account and you manage the machines; serverless runs in a compute plane managed by Databricks and you manage nothing, while your data stays in your own storage either way.

Watch it drawn step by step

Both paths drawn side by side, 2:36: https://youtu.be/s8452E2Y6WM

Episode 4 of a data engineering interview prep series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

Databricks accounts, workspaces and metastores: which layer owns what

dino david — Sun, 20 Sep 2026 17:18:29 +0000

Three words that get used interchangeably until the day you have to design a platform with them. Here is the hierarchy, top down.

The account

The account is the container for your whole organisation, and you normally have one per cloud provider. You manage it in the account console, where billing and account admins live. Identities live here too: users, groups and service principals.

The workspace

A workspace is a single deployment of the Databricks UI, with its own notebooks, clusters, jobs and dashboards. Most teams run separate workspaces for development, test and production.

The one-line difference: the account is where you manage identities, billing and metastores for the organisation. A workspace is where people actually do the work.

The Unity Catalog metastore

The metastore is the top-level container for data objects and their permissions, and you create it at account level. The documentation is direct about the count: you must have one metastore for each region in which your organisation operates, and each regional metastore can be linked to any number of workspaces in that region.

So dev, test and prod in the same region attach to the same metastore, rather than each carrying its own legacy Hive metastore.

The three-level namespace

Inside the metastore, objects follow a three-level namespace: a catalog holds schemas, and a schema holds tables, views and volumes. You address a table as catalog.schema.table, for example prod.sales.orders.

Identity federation

With identity federation you configure users, service principals and groups once in the account console rather than repeating it in every workspace, then assign them to the workspaces that need them. One group, granted permissions on data once, applies everywhere it works. Workspace-local groups still exist in non-federated workspaces, but they cannot be granted Unity Catalog permissions.

Where the data actually sits

Metastore-level managed storage is optional, and managed tables are written there when it is configured. Data you already have in a bucket or container is registered as an external location, backed by a storage credential, and you grant access to that.

The one-line answer

The account holds identities and billing, workspaces hold the tools and the compute, and the metastore holds the data and who can see it, shared across every workspace in the region.

Watch it drawn step by step

I drew this as a 2:32 animated diagram: https://youtu.be/2yy9-BL_RDo

It is episode 3 of a data engineering interview prep series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

Lakehouse vs Data Warehouse vs Data Lake: The Difference in One Picture

dino david — Fri, 18 Sep 2026 12:58:26 +0000

If you work with data, you will be asked this sooner or later: what is a lakehouse, and why not just use a warehouse? Here is the answer I use, with a short animated diagram at the end.

The data warehouse

A data warehouse (Snowflake, Redshift, BigQuery) stores structured tables in its own optimised format. It is very good at three things:

ACID transactions
enforced schemas and governance
fast BI queries

The trade-off is cost and reach. Storage is proprietary and expensive at scale, and it was never designed for raw files, images or logs, so machine learning teams struggle to work in it.

The data lake

A data lake is cheap object storage, such as S3 or ADLS, holding open file formats. It takes any data type, costs very little, and Python and ML tools can read it directly.

But a plain lake has no transactions. Two writers can corrupt a table, nothing enforces a schema, and without a catalog it slowly becomes a data swamp nobody trusts.

Why most companies ran both

So teams loaded everything into the lake, then copied the modelled part into the warehouse. That means two copies of the data, two pipelines to maintain, and two sets of numbers that drift apart.

The lakehouse

A lakehouse is a data lake with an open table format, such as Delta Lake or Apache Iceberg, on top. That table format gives you warehouse behaviour directly on cheap object storage.

From the warehouse it takes:

ACID transactions on the table files
schema enforcement and a catalog for governance
fast BI, because the engine keeps statistics on those same files and skips what a query does not need

From the lake it keeps:

the same cheap, open storage
every data type, structured or not
direct access for Python and ML, with no export step

The one-line answer

One copy of the data, in open formats, serving BI and machine learning at the same time.

Watch it drawn step by step

This is part of a free series that builds Databricks up one layer at a time. Watch it in order from episode 1: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

Databricks Architecture Explained: Control Plane, Compute Plane, Delta Lake and Unity Catalog

dino david — Wed, 16 Sep 2026 21:52:00 +0000

"Explain the Databricks architecture" is one of the most common data engineering questions. Here is the answer as a picture, built up layer by layer.

1. It starts with your cloud account

Databricks runs on Azure, AWS or Google Cloud, and your cloud account is the foundation everything else sits on.

2. The control plane

Databricks runs as two halves. The control plane is hosted and managed by Databricks. It is where the web application, the notebooks, the job scheduler and the cluster manager live.

3. The compute plane

The compute plane (you will also hear "data plane") is where the actual compute runs. In the classic setup it sits inside your own cloud account, in your own virtual network. With serverless compute, it runs in the Databricks account instead.

4. Your data lake

At the bottom is your data lake: ADLS Gen2 on Azure, S3 on AWS. Your raw files live here, and they never leave your account.

5. Delta Lake

On top of the lake sits Delta Lake, a storage layer that adds ACID transactions, schema enforcement and time travel to plain Parquet files. This is what turns a data lake into a lakehouse.

6. Bronze, silver, gold

Inside Delta, data is usually organised in three zones:

Bronze: raw ingested data
Silver: cleaned and joined data
Gold: business-level aggregates, ready for reporting

7. Unity Catalog

Across all of it sits Unity Catalog, the governance layer: permissions, lineage, and a single catalog across every workspace.

8. What happens when you run a notebook

The control plane sends the instruction. A cluster spins up in the compute plane, reads from Delta and writes results back. Only results and metadata go back to the control plane. The data stays put.

9. The consumers

SQL warehouses for BI, machine learning workloads and streaming pipelines all read from the same gold layer.

Watch it drawn step by step

This is episode 1 of a free series that builds Databricks up one layer at a time. Watch it in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4