Databricks classic vs serverless compute: the limitation list decides it

Pick "classic" or "serverless" in Databricks and you are choosing where the machines live. Almost everything else follows from that.

The same request, twice

You press run on a notebook or a SQL query. Databricks can serve that request two ways.

Classic. The control plane asks your cloud provider for machines, and a cluster of VMs is created inside your own account and network. It reads your tables from your own cloud storage. You choose the instance types, you own autoscaling and networking, and the bill has two parts: the cloud provider for the VMs, and Databricks for the DBUs on top.

Serverless. The documentation is direct about it: serverless compute resources run in "the serverless compute plane, which is managed by Databricks", and you run workloads "without provisioning any compute resources in your cloud account". Databricks manages, patches and scales that fleet.

Your data does not move

This is the part that reassures people: in both cases the tables stay in your own cloud storage. It is the compute that changes sides, not the data.

The list that actually decides it

Startup time is the headline, but the decision usually comes down to whether your workload uses something serverless does not support. From the limitations page:

• R is not supported.
• Compute-scoped libraries and compute-scoped init scripts are not supported. Notebook-scoped libraries are the alternative.
• Only Spark Connect APIs are available, and the Spark RDD APIs are not supported.
• Streaming triggers: Trigger.AvailableNow() is supported; Trigger.Continuous(interval) and Trigger.ProcessingTime(interval) are not.
• Maximum runtime is 7 days. A run that exceeds it is terminated and not retried.

So which one

Serverless suits SQL warehouses, ad hoc exploration and spiky jobs, where waiting for a cluster is the real cost and nothing on that list applies.

Classic is the answer when you need something specific: a particular instance type or GPUs, libraries installed at cluster level, R, RDD code, a continuous stream, or your own network setup.

The one-line answer

Classic compute runs in your cloud account and you manage the machines; serverless runs in a compute plane managed by Databricks and you manage nothing, while your data stays in your own storage either way.

Watch it drawn step by step

Both paths drawn side by side, 2:36: https://youtu.be/s8452E2Y6WM

Episode 4 of a data engineering interview prep series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4