The latest articles on DEV Community by dino david (@dino_david_c5d8f55f119b7f). https://dev.to/dino_david_c5d8f55f119b7f https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4128735%2F1e87cc10-230b-4353-986c-19b0e9bff94a.png https://dev.to/dino_david_c5d8f55f119b7f en dino david Wed, 16 Sep 2026 21:52:00 +0000 https://dev.to/dino_david_c5d8f55f119b7f/databricks-architecture-explained-control-plane-compute-plane-delta-lake-and-unity-catalog-1167 https://dev.to/dino_david_c5d8f55f119b7f/databricks-architecture-explained-control-plane-compute-plane-delta-lake-and-unity-catalog-1167 <p>"Explain the Databricks architecture" is one of the most common data engineering questions. Here is the answer as a picture, built up layer by layer.</p> <h2> 1. It starts with your cloud account </h2> <p>Databricks runs on Azure, AWS or Google Cloud, and your cloud account is the foundation everything else sits on.</p> <h2> 2. The control plane </h2> <p>Databricks runs as two halves. The <strong>control plane</strong> is hosted and managed by Databricks. It is where the web application, the notebooks, the job scheduler and the cluster manager live.</p> <h2> 3. The compute plane </h2> <p>The <strong>compute plane</strong> (you will also hear "data plane") is where the actual compute runs. In the classic setup it sits inside <strong>your own cloud account</strong>, in your own virtual network. With serverless compute, it runs in the Databricks account instead.</p> <h2> 4. Your data lake </h2> <p>At the bottom is your data lake: <strong>ADLS Gen2</strong> on Azure, <strong>S3</strong> on AWS. Your raw files live here, and they never leave your account.</p> <h2> 5. Delta Lake </h2> <p>On top of the lake sits <strong>Delta Lake</strong>, a storage layer that adds ACID transactions, schema enforcement and time travel to plain Parquet files. This is what turns a data lake into a lakehouse.</p> <h2> 6. Bronze, silver, gold </h2> <p>Inside Delta, data is usually organised in three zones:</p> <ul> <li> <strong>Bronze:</strong> raw ingested data</li> <li> <strong>Silver:</strong> cleaned and joined data</li> <li> <strong>Gold:</strong> business-level aggregates, ready for reporting</li> </ul> <h2> 7. Unity Catalog </h2> <p>Across all of it sits <strong>Unity Catalog</strong>, the governance layer: permissions, lineage, and a single catalog across every workspace.</p> <h2> 8. What happens when you run a notebook </h2> <p>The control plane sends the instruction. A cluster spins up in the compute plane, reads from Delta and writes results back. <strong>Only results and metadata go back to the control plane. The data stays put.</strong></p> <h2> 9. The consumers </h2> <p>SQL warehouses for BI, machine learning workloads and streaming pipelines all read from the same gold layer.</p> <h2> Watch it drawn step by step </h2> <p> <iframe src="https://www.youtube.com/embed/B5iHmoYgnqY" width="710" height="399"> </iframe> </p> <p>This is episode 1 of a free series that builds Databricks up one layer at a time. Watch it in order: <a href="https://www.youtube.com/watch?v=B5iHmoYgnqY&amp;list=PLDB5WDkDOYF4" rel="noopener noreferrer">https://www.youtube.com/watch?v=B5iHmoYgnqY&amp;list=PLDB5WDkDOYF4</a></p> dataengineering databricks azure aws