Databricks Asset Bundles: CI/CD & Infra-as-Code for Jobs, Pipelines & ML in One Repo

databricks asset bundles are the deploy/promote/rollback unit that decides whether your lakehouse is a reproducible, version-controlled system or a pile of hand-edited notebooks that only one person knows how to redeploy. Every workflow your team ships — a nightly ingestion job, a Delta Live Tables pipeline, an MLflow model and the endpoint that serves it — has to travel from a developer's laptop to a dev workspace, then to staging, then to production, without someone clicking through the Jobs UI at 2 a.m., without the prod schedule silently pointing at last quarter's notebook, and without a rollback plan that amounts to "hope we remember what changed." The hard part of running Databricks at scale was never writing the transformation; it was getting the same job, pipeline, and model definition to deploy identically across three environments and prove, in a diff, exactly what shipped.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "how do you promote a Databricks job from dev to prod without click-ops?", or "what does a databricks.yml actually declare and how do targets work?", or "walk me through a databricks ci/cd pipeline that deploys with a service principal." It covers why notebooks-in-prod stops scaling and what a bundle fixes, the anatomy of the databricks.yml manifest (bundle, variables, artifacts, resources, targets) and how it declares jobs, pipelines, and models as code, the dev/staging/prod target model with mode: development versus mode: production, per-target variables, lookups and presets, the full CI/CD story — bundle validate/deploy/run, GitHub Actions and Azure DevOps, service principals, and gated promotion — and finally the ML and advanced patterns (serving endpoints, multi-workspace, monorepo vs polyrepo, and when NOT to reach for DABs). Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse system topology on the design practice library →, and tighten the transform layer with the data-transformation practice library →.

---

On this page

• Why notebooks-in-prod doesn't scale and what DABs fix
• Bundle anatomy — databricks.yml and resources
• Targets and environments — dev, staging, prod
• CI/CD — validate, deploy, run, and gated promotion
• ML resources, advanced patterns, and interview signals
• Cheat sheet — Databricks Asset Bundle recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why notebooks-in-prod doesn't scale and what DABs fix

Click-ops has no diff, no rollback, and no promotion path — a bundle turns your whole workspace into version-controlled infrastructure as code

The one-sentence invariant: a Databricks Asset Bundle is a directory of source code plus a databricks.yml manifest that declares your jobs, pipelines, and ML assets as code, so the entire unit can be validated, deployed, promoted across environments, and rolled back from a git commit — instead of being assembled by hand in the Jobs UI where there is no diff, no review, and no reproducible path from dev to prod. The failure mode DABs exist to kill is click-ops: someone builds a job in the UI, points it at a notebook in their personal folder, wires a schedule, and now production depends on manual steps that live in exactly one person's memory. There is no pull request, no git blame, no way to answer "what changed between the working run and the broken one," and no atomic rollback beyond re-clicking the old configuration you hopefully wrote down.

The click-ops failure modes senior engineers have all lived through.

• No diff. A job edited in the UI leaves no reviewable artifact. You cannot ask "show me the change" because the change never existed as text. Every incident postmortem that starts with "someone must have edited the schedule" is a click-ops artifact.
• No rollback. Reverting a UI change means remembering the previous cluster spec, notebook path, and parameters by hand. Bundles make rollback a git revert plus a redeploy — the previous state is fully described in the manifest.
• No promotion. Dev and prod drift apart because they were configured independently. The dev job points at 15.4.x, prod is stuck on 13.3.x, and nobody noticed until a UDF behaved differently. A bundle deploys the same definition to every target with only the intended differences.
• Notebook-path coupling. Prod jobs that reference notebooks in /Users/[e-mail]/... break the day Alice leaves. Bundles deploy source into a bundle-managed root path owned by the deploy identity, not a personal folder.
• No identity discipline. UI jobs run as whoever created them. When that human's token is rotated or their account is deprovisioned, the job dies. Bundles let production run as a service principal with a stable, auditable identity.

What a bundle actually is — the mental model.

• The manifest. A single databricks.yml at the repo root (plus optional included YAML files) declaring bundle metadata, variables, artifacts to build, resources (jobs, pipelines, models…), and targets (environments).
• The source. Your notebooks, Python modules, SQL, and wheel-building pyproject.toml/setup.py — the actual code the resources point at.
• The deploy unit. databricks bundle deploy -t <target> uploads the source to a bundle root path in the target workspace and creates/updates every declared resource. databricks bundle destroy tears it all down. The bundle is the atom of deployment.
• Terraform underneath. DABs are implemented on top of the Databricks Terraform provider. You write friendly YAML; the CLI generates and applies Terraform, keeping state in the workspace under the bundle root. You get IaC semantics (plan/apply, state, drift detection) without hand-writing HCL.

The four axes interviewers actually probe.

• Reproducibility. Can you recreate production from git alone? With a bundle, git clone && databricks bundle deploy -t prod rebuilds every job, pipeline, and model. Click-ops cannot make this claim.
• Promotion path. How does a change travel dev → staging → prod? A bundle promotes by deploying the same code to a different target, with per-target overrides for hosts, catalogs, and identity — not by re-clicking.
• Identity (run_as). Who runs prod? A named human (fragile) or a service principal (stable, auditable, least-privilege)? Bundles make run_as an explicit, reviewable field.
• Blast radius. What happens when a deploy is wrong? Bundles give you validate before deploy, a --dry-run/plan step from Terraform, mode: development sandboxing, and git revert rollback — bounded blast radius at every step.

What "jobs as code" buys you.

• Reviewable change. Every schedule change, cluster resize, and new task is a pull-request diff a colleague can approve.
• Environment parity. One definition, many targets. The only differences between dev and prod are the ones you declared — a host, a catalog, an identity — not accidental drift.
• Automated deployment. CI runs bundle validate on every PR and bundle deploy -t prod on merge to main or on a release tag. Humans stop touching production directly.
• Auditability. Git history is your deployment log. "What shipped on the 3rd?" is git log, not a Slack archaeology dig.

Worked example — the click-ops vs bundle scorecard

Detailed explanation. The single most useful artifact for a DAB interview is a memorised scorecard comparing click-ops, ad-hoc scripting (Databricks CLI/REST called by hand), and a full Asset Bundle across the four axes. Every senior conversation about "why bundles" converges on this table; having it in your head separates a fluent answer from a hand-wave. Walk through building it for a team that runs one ingestion job, one DLT pipeline, and one nightly model-scoring job.

• Team. Four data engineers, one Databricks workspace per environment (dev, staging, prod).
• Assets. ingest_orders job, orders_dlt pipeline, score_churn job that loads a registered model.
• Goal. Ship a change to the ingestion cluster size and have it appear identically in all three environments with an approval gate before prod.
• Constraint. Auditors need to answer "what changed and who approved it" from artifacts, not memory.

Question. Build the three-way comparison and pick the approach that satisfies the audit constraint.

Input.

Capability	Click-ops (UI)	Ad-hoc CLI/REST scripts	Asset Bundle
Reviewable diff	no	partial (scripts, not state)	yes (YAML in git)
Atomic rollback	no	manual	git revert + redeploy
Dev/prod parity	drifts	script-dependent	declared per target
Prod identity	job creator	script runner	service principal
Reproduce from git	no	partial	yes

Code.

# databricks.yml — the whole team's deployable surface, in one reviewable file
bundle:
  name: orders_platform

include:
  - resources/*.yml            # jobs, pipelines, models live in split files

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dbc-dev.cloud.databricks.com
  staging:
    mode: production
    workspace:
      host: https://dbc-staging.cloud.databricks.com
  prod:
    mode: production
    workspace:
      host: https://dbc-prod.cloud.databricks.com
    run_as:
      service_principal_name: sp-orders-platform

Step-by-step explanation.

• Bundle name and includes. bundle.name scopes the deployment; include splits resources into resources/*.yml so jobs, pipelines, and models are each reviewable in isolation. The whole surface is text.
• Three targets, one file. dev, staging, and prod are declared once. Promoting a change means deploy -t staging then deploy -t prod — the definition is shared, only the host and identity differ.
• Mode encodes intent. dev uses mode: development (sandboxed, name-prefixed, schedules paused); staging and prod use mode: production (strict, no prefix, live schedules).
• run_as on prod. Production runs as sp-orders-platform, a service principal — not whoever merged the PR. That single field satisfies "who runs prod" for the auditor.
• Audit falls out of git. The cluster-size change is a one-line YAML diff in a PR; the approval is the PR review; the deploy is a CI log. Every audit question maps to a git artifact.

Output.

Audit question	Click-ops answer	Bundle answer
What changed?	"someone edited it"	PR diff on resources/ingest.yml
Who approved it?	unknown	PR reviewer + required check
When did it ship?	Slack archaeology	CI deploy log / git tag
Can we revert?	rebuild by hand	git revert + redeploy
Who runs prod?	job creator	sp-orders-platform

Rule of thumb. If you cannot recreate production by cloning a git repo and running one deploy command, you are running click-ops, and every incident will start with "someone must have changed something." A bundle makes the workspace a function of the repo.

Worked example — what a bundle deploy actually does under the hood

Detailed explanation. Interviewers love to probe whether you understand that databricks bundle deploy is not "upload my notebooks" — it is a Terraform apply that reconciles declared state against the workspace. Knowing the deploy lifecycle lets you reason about idempotency, drift, and rollback. Walk through the five phases the CLI runs for databricks bundle deploy -t prod.

• Config load + merge. The CLI reads databricks.yml and included files, merges the selected target's overrides on top of the base, and resolves ${...} substitutions.
• Validate. It checks the merged config against the bundle JSON schema and the workspace (does the cluster policy exist, is the warehouse reachable).
• Artifact build + upload. Declared artifacts (e.g. a Python wheel) are built locally, then source files and artifacts are synced to the bundle root path in the workspace.
• Terraform apply. The CLI translates resources into Terraform, computes a plan against stored state, and applies — creating, updating, or deleting jobs/pipelines/models to match the manifest.
• State write-back. Updated Terraform state is written back into the bundle root, so the next deploy knows exactly what exists.

Question. Trace what happens when you change a job's num_workers from 2 to 4 and redeploy — and explain why the operation is idempotent.

Input.

Phase	Input	Effect
Load + merge	edited resources/ingest.yml	merged config has num_workers: 4
Validate	merged config + workspace	schema OK; policy exists
Sync	source + wheel	uploaded to bundle root
Terraform plan	prior state (num_workers=2)	diff: update in place
Apply	plan	job updated to 4 workers

Code.

# Inspect first, then deploy — never deploy blind
databricks bundle validate -t prod         # schema + workspace checks
databricks bundle deploy   -t prod          # sync + terraform apply
databricks bundle summary  -t prod          # show deployed resource URLs + IDs

# The generated plan/state lives under the workspace bundle root, e.g.
#   /Workspace/Users/<deploy-identity>/.bundle/orders_platform/prod/
# containing terraform state + the synced source snapshot.

Step-by-step explanation.

• Merge produces one config. The target override (num_workers: 4 if set per-target, else the base value) is merged so the CLI sees a single fully-resolved job definition. There is no ambiguity about which value wins — the target layers on top of the base.
• Validate is cheap insurance. It catches a missing cluster policy or a typo'd notebook path before any workspace mutation. Running validate in CI on every PR is the first quality gate.
• Sync is content-addressed-ish. Only changed source files are uploaded; the wheel is rebuilt if its inputs changed. The bundle root is the single source of truth for deployed code.
• Terraform makes it idempotent. The plan compares desired state (num_workers: 4) with stored state (2) and emits exactly one in-place update. Re-running the deploy with no changes produces a no-op plan — that is the idempotency guarantee.
• State write-back closes the loop. Because state is persisted in the bundle root, a second engineer deploying from the same repo sees the same state and does not double-create resources. Drift (someone hand-edited the job in the UI) shows up as a plan diff on the next deploy.

Output.

Deploy	Prior state	Desired	Plan	Result
1 (initial)	none	2 workers	create	job created, 2 workers
2 (resize)	2 workers	4 workers	update in place	job now 4 workers
3 (no change)	4 workers	4 workers	no-op	nothing happens
4 (after UI edit to 8)	4 workers (state)	4 workers	update back to 4	drift corrected

Rule of thumb. Treat bundle deploy as terraform apply, not scp. It reconciles declared state; it is idempotent; and it will undo out-of-band UI edits on the next run, which is exactly why you stop editing prod in the UI.

Senior interview question on migrating off click-ops

A senior interviewer often opens with: "You inherit a Databricks workspace with 40 jobs and 6 DLT pipelines, all built in the UI, all pointing at notebooks in personal user folders, all running as the humans who created them. Two of those humans have left. Walk me through how you'd migrate this to Asset Bundles, adopt the existing resources without recreating them, and establish a dev → prod promotion path."

Solution Using bundle generate plus deployment bind to adopt existing resources

# Step 1 — scaffold a bundle from the default template
databricks bundle init default-python       # or an org template repo
cd orders_platform

# Step 2 — generate YAML from EXISTING UI jobs/pipelines (reverse-engineer)
databricks bundle generate job      --existing-job-id 620...     # writes resources/*.yml + source
databricks bundle generate pipeline --existing-pipeline-id 9f... # DLT pipeline as code

# Step 3 — the generated resources/ingest_orders.yml (trimmed), now in git
resources:
  jobs:
    ingest_orders:
      name: ingest_orders
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ../src/ingest_orders.py   # moved out of a personal folder
          job_cluster_key: main
      job_clusters:
        - job_cluster_key: main
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: i3.xlarge
            num_workers: 4
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: UTC

# Step 4 — BIND the generated resource to the live job so deploy updates it in place
#          (instead of creating a duplicate)
databricks bundle deployment bind ingest_orders 620... -t prod

# Step 5 — first managed deploy: from now on the job is bundle-owned
databricks bundle validate -t prod
databricks bundle deploy   -t prod

# Step 6 — repeat generate+bind for all 40 jobs / 6 pipelines, then flip run_as
#          to the service principal in the prod target and redeploy.

Step-by-step trace.

Step	Command	State before	State after
1	bundle init	no repo	scaffolded bundle skeleton
2	bundle generate job	UI job, no YAML	resources/*.yml + source in git
3	edit YAML	notebook in personal folder	notebook path under src/
4	deployment bind	job unlinked from bundle	job linked to bundle state
5	bundle deploy	UI-owned job	bundle-owned job (in place)
6	flip run_as + redeploy	runs as departed human	runs as service principal

After the migration, all 40 jobs and 6 pipelines are described in YAML in one repo, each bound to its live resource so deploys update in place (no duplicates), the notebook paths live under a repo-managed src/ tree instead of personal folders, and production runs as a service principal that survives any individual leaving the team.

Output:

Metric	Before (click-ops)	After (bundle)
Jobs described in git	0 of 40	40 of 40
Rollback mechanism	none	git revert + redeploy
Prod run identity	departed humans	service principal
Duplicate resources on adopt	risk (recreate)	none (deployment bind)
Reproduce prod from git	impossible	one deploy command

Why this works — concept by concept:

• bundle generate — reverse-engineers an existing UI job or pipeline into resources/*.yml plus its source, so you adopt what already runs instead of rewriting 40 definitions from scratch. It is the on-ramp from click-ops to code.
• deployment bind — links a generated resource to the live resource ID in the workspace so the next deploy updates it in place. Without bind, the deploy would create a second, duplicate job and you would have two schedules firing.
• Bundle-managed source paths — moving notebooks out of /Users/[e-mail]/... into a repo src/ tree and letting the bundle sync them to a managed root path removes the human-folder coupling that kills jobs when people leave.
• run_as service principal — flipping production identity to a service principal decouples job execution from any individual's account lifecycle; the identity is stable, auditable, and least-privilege.
• Cost — the migration is O(number of resources) of generate + bind calls, done once, plus a one-time run_as flip. The payoff is O(1) reproducibility forever after: every future change is a reviewable diff and every environment is rebuildable from git. Compared to the unbounded, recurring cost of click-ops incidents, the migration pays for itself on the first avoided 2 a.m. page.

ETL
Topic — etl
ETL problems on reproducible pipeline deployment

Practice →

Design Topic — design Design problems on infrastructure-as-code workflows

Practice →

---

2. Bundle anatomy — databricks.yml and resources

One databricks.yml declares metadata, variables, artifacts, resources, and targets — and the resources block is where jobs, pipelines, and ML models become code

The mental model in one line: the databricks.yml manifest is a layered YAML document with a small set of top-level keys — bundle, include, variables, artifacts, resources, targets, and a few workspace-level knobs — where resources maps one-to-one onto Databricks REST API objects (jobs, pipelines, experiments, models, serving endpoints, schemas, volumes) so that "creating a job" becomes "declaring a resources.jobs.<key> block" that the CLI reconciles on deploy. Learn the seven keys and you can read any bundle; learn the resources shapes and you can author one.

The seven top-level keys you must know.

• bundle. Identity and global settings — name (scopes the deployment root), optional git metadata, cluster_id/compute_id for a default compute, and databricks_cli_version pinning. The bundle name plus the target name form the workspace root path.
• include. A list of glob paths (e.g. resources/*.yml) whose contents are merged into the root config. This is how large bundles stay readable — one file per job or domain.
• variables. Named inputs with a description, a default, an optional type (including complex for maps/lists), or a lookup that resolves an ID at deploy time (warehouse, cluster policy, instance pool, metastore). Referenced as ${var.name}.
• artifacts. Buildable outputs — most commonly type: whl with a build command — that the CLI compiles locally and uploads so tasks can install them as libraries.
• resources. The heart of the bundle: jobs, pipelines, experiments, models, registered_models, model_serving_endpoints, schemas, volumes, quality_monitors, clusters, dashboards, apps. Each key maps onto an API object.
• targets. Named environments (dev/staging/prod) that override the base config — hosts, variables, mode, run_as, permissions, presets. Exactly one target can be default: true.
• Workspace-level keys. workspace (host, root_path, profile), sync (include/exclude file globs), run_as (default identity), permissions (who can manage the deployed resources), and presets (bundle-wide behaviour toggles).

How resources maps to real objects.

• resources.jobs.<key>. A Jobs API object — name, tasks, job_clusters, schedule/trigger, parameters, email_notifications, permissions. This is "jobs as code."
• resources.pipelines.<key>. A Delta Live Tables / Lakeflow Declarative Pipeline — name, catalog, target/schema, libraries (the notebooks or files that define the pipeline), configuration, development/continuous flags.
• resources.experiments / models / registered_models. MLflow experiments and models, including Unity Catalog registered models (catalog.schema.name).
• resources.model_serving_endpoints.<key>. A serving endpoint that references a registered model version — the "serve" side of an ML deployment, as code.
• resources.schemas / volumes. Unity Catalog schemas and volumes the bundle owns, so storage layout is version-controlled alongside compute.

Substitution and references — the glue.

• ${var.name}. Injects a variable value.
• ${workspace.current_user.userName} and ${workspace.root_path}. Workspace context — useful for dev name-spacing.
• ${bundle.name}, ${bundle.target}. The active bundle and target names — the canonical way to name resources per environment (daily_etl_${bundle.target}).
• ${resources.jobs.foo.id}. Cross-reference another resource's runtime ID, e.g. a run_job_task that triggers another job in the same bundle. This is how you wire multi-job DAGs without hard-coded IDs.

Worked example — a minimal, runnable databricks.yml

Detailed explanation. The smallest useful bundle is a single job that runs a single notebook on a job cluster, deployable to a dev target. Building it from scratch cements what each key does. Walk through the minimum viable manifest and the commands that deploy and run it.

• Goal. Deploy one job, hello_etl, that runs src/etl.py on a 2-worker cluster.
• Target. A single dev target, mode: development, marked default so -t dev is implicit.
• Deploy. bundle validate then bundle deploy; run with bundle run.

Question. Author the minimal databricks.yml and the deploy/run commands, and explain what mode: development changes about the deployed job.

Input.

Element	Value
Bundle name	hello_bundle
Job key	hello_etl
Source	src/etl.py
Cluster	15.4 LTS, i3.xlarge, 2 workers
Target	dev (development, default)

Code.

# databricks.yml — the smallest bundle that deploys a real job
bundle:
  name: hello_bundle

resources:
  jobs:
    hello_etl:
      name: hello_etl
      tasks:
        - task_key: run_etl
          notebook_task:
            notebook_path: ./src/etl.py
          job_cluster_key: main
      job_clusters:
        - job_cluster_key: main
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: i3.xlarge
            num_workers: 2

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dbc-dev.cloud.databricks.com

databricks bundle validate      # schema + workspace checks (uses default target: dev)
databricks bundle deploy        # sync source + create the job
databricks bundle run hello_etl # trigger the job now, stream the run to your terminal
databricks bundle summary       # print the job URL + resource IDs

Step-by-step explanation.

• bundle.name sets the root. Deployed source lands under a per-user .bundle/hello_bundle/dev/ root path in the workspace — no personal-folder coupling.
• The job is one task on one cluster. notebook_task points at repo-relative ./src/etl.py; job_cluster_key binds the task to the inline main cluster so the job is fully self-contained.
• mode: development sandboxes it. On deploy, the job name is prefixed to [dev your.name] hello_etl, its schedule (if any) is force-paused, development-tagged, and concurrent runs are allowed — so many engineers can deploy their own copy without colliding.
• default: true makes -t optional. With one default target, validate/deploy/run need no -t flag. Add more targets later and the flag becomes required for the non-default ones.
• bundle run closes the loop. It triggers the deployed job and streams output, so the whole author → deploy → run cycle is three commands with no UI.

Output.

Command	Effect	Result
validate	schema + host check	"Validation OK"
deploy	sync + create	job [dev your.name] hello_etl created
run hello_etl	trigger	run starts, logs stream to terminal
summary	introspect	prints job URL + numeric job ID

Rule of thumb. Start every new bundle with bundle init or this minimal skeleton, get one job deploying to a dev target, then grow outward. A bundle that deploys nothing is impossible to debug; a bundle that deploys one job is a foundation.

Worked example — a DLT pipeline plus a wheel artifact

Detailed explanation. Real bundles build a Python wheel from src/, install it on job clusters, and declare a DLT pipeline whose transformations import that wheel. This is the pattern that turns a bundle from "runs a notebook" into "ships a versioned library plus a pipeline." Walk through the artifacts block and a pipelines resource.

• Artifact. Build dist/*.whl from the project with poetry build (or python -m build).
• Pipeline. A DLT pipeline named orders_dlt, writing to Unity Catalog catalog.schema, defined by a notebook under src/.
• Library wiring. The job/pipeline references the built wheel so import orders_lib works on the cluster.

Question. Declare a wheel artifact and a DLT pipeline that uses it, and explain how the wheel reaches the cluster.

Input.

Element	Value
Artifact	Python wheel, built by poetry build
Pipeline	orders_dlt (DLT), catalog=main, schema=orders
Definition	src/pipelines/orders_dlt.py
Library	the built wheel, installed on the pipeline cluster

Code.

# databricks.yml (excerpt) — build a wheel and declare a DLT pipeline
artifacts:
  orders_lib:
    type: whl
    build: poetry build          # runs in the project dir; emits dist/*.whl
    path: .

resources:
  pipelines:
    orders_dlt:
      name: orders_dlt
      catalog: main               # Unity Catalog target catalog
      schema: orders              # target schema for the materialized tables
      serverless: true
      libraries:
        - notebook:
            path: ./src/pipelines/orders_dlt.py
      configuration:
        source_path: /Volumes/main/landing/orders

  jobs:
    refresh_orders:
      name: refresh_orders
      tasks:
        - task_key: run_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.orders_dlt.id}   # cross-reference

Step-by-step explanation.

• artifacts.orders_lib builds locally. On deploy, the CLI runs poetry build in the project directory, producing a wheel under dist/, then uploads it to the bundle root so clusters can install it.
• The pipeline is declared, not clicked. resources.pipelines.orders_dlt sets the UC catalog and schema, marks it serverless, and points libraries at the transformation notebook — the pipeline definition now lives in git.
• configuration passes runtime settings. Key/value pairs (like source_path) are injected into the pipeline's Spark config and read by the transformation code, so the same code parametrizes cleanly per target.
• The job triggers the pipeline by reference. pipeline_task.pipeline_id: ${resources.pipelines.orders_dlt.id} wires the refresh_orders job to the pipeline by substitution, so there is no hard-coded ID that breaks across environments.
• The wheel reaches the cluster via the bundle root. Because the artifact is uploaded under the deployment root, the pipeline/job cluster installs it from a stable workspace path, and every environment gets the exact wheel that was built from that commit.

Output.

Deployed object	Source	Wiring
Wheel orders_lib-*.whl	poetry build	uploaded to bundle root
DLT pipeline orders_dlt	src/pipelines/orders_dlt.py	writes to main.orders
Job refresh_orders	manifest	triggers pipeline by ${...id}
Cluster libraries	built wheel	installed from bundle root

Rule of thumb. Put reusable logic in a wheel declared under artifacts, keep thin notebooks as entry points, and cross-reference resource IDs with ${resources...id} instead of pasting numeric IDs. That combination is what makes a bundle portable across every target.

Senior interview question on structuring a large bundle

A senior interviewer might ask: "Your platform has 30 jobs across three domains — ingestion, transformation, and ML — plus shared cluster and warehouse configuration. How do you structure the databricks.yml, the include files, and the resources so the bundle stays readable, avoids duplication, and lets each domain team own its slice without merge-conflicting on one giant file?"

Solution Using include-split resources with shared variables and a per-domain layout

# Repo layout — one root manifest, split resource files per domain
orders_platform/
├── databricks.yml               # bundle + variables + artifacts + targets + include
├── resources/
│   ├── ingestion.yml            # resources.jobs.ingest_*        (ingestion team)
│   ├── transform.yml            # resources.jobs + pipelines     (transform team)
│   └── ml.yml                   # resources.experiments/models/endpoints (ML team)
└── src/                         # shared source; wheel built from here

# databricks.yml — the root that ties it together
bundle:
  name: orders_platform

include:
  - resources/*.yml              # each domain owns its own file; no giant manifest

variables:
  catalog:
    description: Unity Catalog catalog for this environment
    default: dev
  warehouse_id:
    description: SQL warehouse for downstream tasks
    lookup:
      warehouse: shared-serverless    # resolve the ID by name at deploy time
  default_node_type:
    default: i3.xlarge

artifacts:
  orders_lib:
    type: whl
    build: poetry build

targets:
  dev:   { mode: development, default: true, workspace: { host: https://dbc-dev.cloud.databricks.com } }
  prod:
    mode: production
    workspace: { host: https://dbc-prod.cloud.databricks.com }
    variables:
      catalog: main               # prod overrides the catalog variable
    run_as: { service_principal_name: sp-orders-platform }

# resources/ingestion.yml — owned by the ingestion team, references shared vars
resources:
  jobs:
    ingest_orders:
      name: ingest_orders
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ../src/ingest_orders.py
            base_parameters:
              catalog: ${var.catalog}          # shared variable, per-target value
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: ${var.default_node_type}
            num_workers: 4

Step-by-step trace.

Layer	Owner	Contents	Conflict surface
databricks.yml	platform team	bundle, variables, artifacts, targets	small, rarely edited
resources/ingestion.yml	ingestion team	ingestion jobs	isolated per team
resources/transform.yml	transform team	pipelines + transform jobs	isolated per team
resources/ml.yml	ML team	experiments/models/endpoints	isolated per team
src/	shared	wheel source	reviewed cross-team

After this restructuring, each domain team edits only its own resources/*.yml, so merge conflicts on the manifest nearly vanish; shared knobs (catalog, warehouse_id, default_node_type) live once in variables and resolve per target; and the platform team owns only the small root file with targets and run_as, which changes rarely and is reviewed carefully.

Output:

Concern	Before (one file)	After (include-split)
Merge conflicts	constant on one manifest	rare, per-domain
Ownership	ambiguous	file-level, per team
Shared config duplication	copy-pasted per job	one variables block
Per-env differences	scattered	one targets override block
Readability	1500-line YAML	small focused files

Why this works — concept by concept:

• include globs — include: [resources/*.yml] merges many small files into one logical config, so team ownership maps onto files and the root manifest stays tiny. Git conflicts localize to the file a team actually edits.
• Shared variables with lookups — declaring catalog, warehouse_id, and default_node_type once, and resolving IDs by name via lookup, removes copy-pasted values and the "prod points at the dev warehouse ID" class of bug.
• Per-target variable overrides — the prod target sets catalog: main in one place; every job that references ${var.catalog} gets the right value automatically. One override, global effect.
• Cross-file resource references — because all included files merge into one resources namespace, a job in transform.yml can reference a pipeline in the same namespace with ${resources.pipelines...id} regardless of which file declared it.
• Cost — the structure costs a one-time layout decision and a naming convention. It buys O(teams) parallel ownership with near-zero manifest merge conflicts, versus the O(1)-file bottleneck of a monolithic manifest where every change touches the same 1500 lines. At 30 jobs and three teams, the split is the difference between shipping daily and serializing behind one file.

Design
Topic — design
Design problems on declarative resource modeling

Practice →

Data Transformation Topic — data-transformation Transformation-pipeline definition problems

Practice →

---

3. Targets and environments — dev, staging, prod

Targets are the promotion primitive — one code base, many environments, differing only by host, variables, identity, and mode

The mental model in one line: a target is a named environment overlay on top of the base bundle config — it sets the workspace host and root_path, overrides variables, chooses an operating mode (development or production), assigns a run_as identity, grants permissions, and applies presets — so that promoting a change from dev to prod means deploying the same resources with a different target, never editing the resources themselves. Targets are why "one repo, three environments" is a config decision, not three parallel code bases.

What a target can override.

• workspace.host / root_path. Which workspace and which deployment root the target deploys into. Dev, staging, and prod are usually separate workspaces (or at least separate root paths).
• variables. Per-environment values — catalog: dev versus catalog: main, a smaller warehouse in dev, a bigger cluster in prod. Every ${var.x} reference picks up the target's value.
• mode. development or production — a bundle of behavioural defaults (below) that encode "this is a sandbox" versus "this is real."
• run_as. The identity that runs deployed jobs — a human in dev, a service principal in prod.
• permissions. Who can manage the deployed resources (a group like data-eng with CAN_MANAGE), applied uniformly to everything the target deploys.
• presets. Bundle-wide toggles like name_prefix, trigger_pause_status, jobs_max_concurrent_runs, pipelines_development, and tags, letting you tune behaviour without editing each resource.

mode: development versus mode: production.

• mode: development. Prefixes every resource name with [dev <username>], force-pauses schedules and triggers, tags resources dev, sets pipelines to development mode, allows concurrent runs, and deploys under a per-user path — so many engineers can each deploy an isolated copy that never fires on a schedule or clobbers a colleague.
• mode: production. Applies strict validation, forbids the dev name prefix, keeps schedules live, and (best practice) requires a run_as that is not an interactive user and a shared root_path. It is the "this deploy is real, treat it carefully" mode.
• Why it matters in interviews. Naming the exact behavioural differences — name prefix, paused schedules, per-user path — is the senior signal. "Dev is a sandbox, prod is strict" is the weak version; the specifics are the strong one.

Variables, lookups, and precedence.

• Declaration. A variable has a description and either a default, a type (string, complex), or a lookup.
• Lookups. lookup: { warehouse: "shared-serverless" } resolves the warehouse's ID at deploy time by its name — so you never hard-code an ID that differs per workspace. Lookups exist for warehouses, cluster policies, instance pools, metastores, service principals, and more.
• Override precedence (highest wins). Command line --var > DATABRICKS_BUNDLE_VAR_* env var > target-level variables > base variables.default. Knowing this order is a common interview probe.
• Complex variables. type: complex lets a variable hold a whole map or list (e.g. a reusable cluster spec) that resources reference with ${var.cluster_spec} — DRY across many jobs.

Worked example — a three-target bundle with per-env variables

Detailed explanation. The canonical setup is one bundle with dev, staging, and prod targets, each pointing at its own workspace and overriding the catalog and cluster size, with dev in development mode and staging/prod in production mode. Walk through the full targets block.

• Dev. Development mode, default, small cluster, catalog: dev, runs as the developer.
• Staging. Production mode, catalog: staging, service principal, gated in CI.
• Prod. Production mode, catalog: main, service principal, bigger cluster, restricted permissions.

Question. Author the three-target block and show how one job's cluster size and catalog differ across environments from a single resource definition.

Input.

Target	Mode	Catalog	Workers	run_as
dev	development	dev	2	developer
staging	production	staging	2	sp-staging
prod	production	main	8	sp-prod

Code.

variables:
  catalog:  { default: dev }
  workers:  { default: 2 }

resources:
  jobs:
    transform_orders:
      name: transform_orders
      tasks:
        - task_key: transform
          notebook_task:
            notebook_path: ../src/transform.py
            base_parameters: { catalog: ${var.catalog} }
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: i3.xlarge
            num_workers: ${var.workers}

targets:
  dev:
    mode: development
    default: true
    workspace: { host: https://dbc-dev.cloud.databricks.com }
  staging:
    mode: production
    workspace: { host: https://dbc-staging.cloud.databricks.com }
    variables: { catalog: staging }
    run_as: { service_principal_name: sp-staging }
  prod:
    mode: production
    workspace: { host: https://dbc-prod.cloud.databricks.com }
    variables: { catalog: main, workers: 8 }
    run_as: { service_principal_name: sp-prod }
    permissions:
      - level: CAN_MANAGE
        group_name: data-eng

Step-by-step explanation.

• One job, three behaviours. transform_orders is declared once. The catalog and num_workers come from variables, so each target reshapes the same job without duplicating it.
• Dev is a sandbox. mode: development prefixes the job [dev you] transform_orders, pauses any schedule, and deploys under your user path — safe to iterate on.
• Staging overrides one variable. Only catalog: staging changes; workers stay at the default 2. Staging is a production-mode dress rehearsal on smaller compute.
• Prod overrides two variables and locks permissions. catalog: main, workers: 8, a sp-prod identity, and a CAN_MANAGE grant to data-eng. The difference between staging and prod is fully described in these few lines.
• Deploy per target. deploy -t dev, deploy -t staging, deploy -t prod ship the identical resource to three workspaces with only the declared deltas.

Output.

Deploy	Job name	Catalog param	Workers	Runs as
-t dev	[dev you] transform_orders	dev	2	you
-t staging	transform_orders	staging	2	sp-staging
-t prod	transform_orders	main	8	sp-prod

Rule of thumb. Put everything that differs between environments into variables and override it per target. If you find yourself editing a resources block to change behaviour between dev and prod, you have missed a variable — the resource should be environment-agnostic and the target should carry the differences.

Worked example — variable lookups and complex variables

Detailed explanation. Hard-coding a warehouse ID or a cluster policy ID breaks the moment you deploy to a second workspace, because those IDs differ per workspace. Lookups resolve IDs by name at deploy time; complex variables let you define a cluster spec once and reuse it. Walk through both.

• Lookup. Resolve the SQL warehouse ID by its name shared-serverless, per workspace.
• Complex variable. Define one job_cluster spec and reference it from every job.
• Payoff. No numeric IDs in git; one cluster spec instead of N copies.

Question. Replace a hard-coded warehouse ID and duplicated cluster specs with a lookup and a complex variable.

Input.

Anti-pattern	Fix
warehouse_id: 862f... (per-workspace)	lookup: { warehouse: shared-serverless }
cluster spec copy-pasted in 10 jobs	one type: complex variable
prod points at dev's warehouse ID	lookup resolves per target automatically

Code.

variables:
  # Resolve the warehouse ID by NAME at deploy time — different ID per workspace
  warehouse_id:
    description: Serverless SQL warehouse
    lookup:
      warehouse: shared-serverless

  # One cluster spec, reused everywhere via ${var.small_cluster}
  small_cluster:
    type: complex
    default:
      spark_version: 15.4.x-scala2.12
      node_type_id: i3.xlarge
      num_workers: 2

resources:
  jobs:
    report_daily:
      name: report_daily
      tasks:
        - task_key: sql_step
          sql_task:
            warehouse_id: ${var.warehouse_id}      # resolved per workspace
            query: { query_id: ${var.report_query_id} }
        - task_key: py_step
          notebook_task: { notebook_path: ../src/report.py }
          new_cluster: ${var.small_cluster}          # complex var expands here

Step-by-step explanation.

• The lookup removes per-workspace IDs. lookup.warehouse: shared-serverless makes the CLI query each target workspace for a warehouse named shared-serverless and inject its ID. Dev and prod resolve to different IDs automatically — the "prod uses dev's warehouse" bug is impossible.
• The complex variable is reused by reference. small_cluster holds a full cluster map; new_cluster: ${var.small_cluster} expands it. Change the Spark version once and every job that references it updates.
• Targets can still override. A prod target can set variables.small_cluster to a larger spec, so the same reference yields bigger compute in prod without touching any job.
• Everything stays in git as names, not IDs. The manifest references human-readable names (shared-serverless) and variable keys, so a reviewer can reason about it without decoding opaque IDs.
• Deploy-time resolution is validated. If shared-serverless does not exist in the target workspace, validate fails early with a clear error rather than deploying a broken job.

Output.

Reference	dev resolves to	prod resolves to
${var.warehouse_id}	dev warehouse ID	prod warehouse ID
${var.small_cluster}	2-worker i3.xlarge	overridable to 8-worker
numeric IDs in git	none	none

Rule of thumb. Never commit a resource ID that varies per workspace — warehouses, policies, pools, metastores. Use a lookup to resolve by name, and hoist any spec you copy-paste more than twice into a type: complex variable. IDs in git are a portability bug waiting for its second environment.

Senior interview question on safe dev-to-prod promotion

A senior interviewer might ask: "Explain exactly what changes between deploying your bundle with mode: development versus mode: production, why a developer's deploy never triggers a scheduled prod run, and how you'd let five engineers each work on the same bundle in one shared dev workspace without stepping on each other. Then show the target configuration that enforces it."

Solution Using mode-driven presets and per-user development isolation

# One bundle, isolation by mode + presets
targets:
  dev:
    mode: development            # per-user prefix, paused schedules, dev tags, user path
    default: true
    workspace:
      host: https://dbc-dev.cloud.databricks.com
    presets:
      name_prefix: "[dev ${workspace.current_user.short_name}] "  # explicit isolation
      trigger_pause_status: PAUSED     # belt-and-braces: no schedule fires in dev
      jobs_max_concurrent_runs: 5      # allow parallel dev iterations

  prod:
    mode: production             # strict validation, no prefix, live schedules
    workspace:
      host: https://dbc-prod.cloud.databricks.com
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: sp-orders-prod   # never an interactive user
    permissions:
      - level: CAN_MANAGE
        group_name: data-platform
    presets:
      trigger_pause_status: UNPAUSED   # prod schedules are live
      tags:
        env: prod
        managed_by: dab

# Five engineers, one shared dev workspace — each deploy is isolated by username
databricks bundle deploy -t dev     # engineer A -> [dev a.lee] jobs under /Users/a.lee/.bundle/...
databricks bundle deploy -t dev     # engineer B -> [dev b.ng]  jobs under /Users/b.ng/.bundle/...
# No collision: names are prefixed by user, paths are per-user, schedules are paused.

# Prod is deployed only by CI as the service principal, never from a laptop:
databricks bundle deploy -t prod    # (runs in CI; auth = sp-orders-prod via OAuth M2M)

Step-by-step trace.

Behaviour	mode: development	mode: production
Resource name	[dev <user>] name	name (prefix forbidden)
Schedules/triggers	force-paused	live (UNPAUSED)
Deploy root path	per-user /Users/<user>/.bundle/...	shared /Workspace/Shared/.bundle/...
Default run_as	the developer	must be non-interactive (SP)
Concurrent dev copies	isolated per user	single shared deployment
Validation strictness	relaxed	strict (fails on prod-unsafe config)

When five engineers deploy to the shared dev workspace, each gets [dev their-name]-prefixed resources under their own user path with schedules paused, so no one's iteration triggers a run or overwrites another's copy. Production, by contrast, deploys once to a shared path, runs as sp-orders-prod, keeps schedules live, and is only ever deployed by CI — so a laptop deploy can never accidentally mutate prod.

Output:

Scenario	Result
Dev A + Dev B deploy to dev	two isolated, prefixed, paused copies — no collision
Dev deploy accidentally targets a schedule	schedule is PAUSED in dev — no fire
Laptop tries deploy -t prod	blocked by CI-only SP auth + reviewed pipeline
Prod deploy from CI	one shared deployment, live schedules, SP identity
Name clash across engineers	impossible — names are user-prefixed

Why this works — concept by concept:

• mode: development — bundles the per-user name prefix, forced schedule pause, dev tagging, and per-user deploy path into one setting, which is exactly the isolation you need for many engineers sharing one workspace. It makes "safe by default" the default.
• mode: production — turns on strict validation, forbids the dev prefix, and expects a non-interactive run_as, encoding "this is real" so the tooling refuses configurations that are unsafe for prod.
• presets — name_prefix, trigger_pause_status, tags, and jobs_max_concurrent_runs let you fine-tune the mode defaults without editing individual resources, so an org convention lives in one target block.
• Per-user root_path in dev vs shared root_path in prod — isolates dev deploys by user path while giving prod a single shared, service-principal-owned root, so dev iterations never touch prod state and prod has exactly one authoritative deployment.
• Cost — the isolation is free: it is configuration, not extra infrastructure. It buys O(engineers) safe parallel development in one workspace and an O(1), CI-only, service-principal-gated prod, versus the click-ops alternative where every shared-workspace edit risks clobbering a colleague or firing a real schedule. The blast radius of a mistake shrinks from "prod incident" to "my own dev copy."

ETL
Topic — etl
ETL problems on multi-environment promotion

Practice →

Optimization Topic — optimization Optimization problems on environment-scoped compute sizing

Practice →

---

4. CI/CD — validate, deploy, run, and gated promotion

The CLI verbs plus a service-principal identity turn a bundle into a pipeline — validate on every PR, deploy to prod only through a gated, automated job

The mental model in one line: databricks ci/cd with bundles is a small set of CLI verbs — validate, deploy, run, summary, destroy, generate, deployment bind — wired into a pipeline runner (GitHub Actions, Azure DevOps, GitLab) that authenticates as a service principal via OAuth machine-to-machine, runs bundle validate on every pull request, deploys to dev/staging automatically, and deploys to prod only behind a manual-approval gate — so humans review changes and machines apply them. The bundle is the artifact; the pipeline is the promotion machine; the service principal is the identity that makes automated prod deploys auditable and least-privilege.

The CLI verbs that matter in CI.

• bundle validate. Schema + workspace validation. The universal PR check — fast, read-only, catches typos and missing resources before any deploy.
• bundle deploy -t <target>. Sync source + Terraform apply. Runs on merge (dev/staging) or on a tag/approval (prod).
• bundle run <resource> -t <target>. Trigger a job/pipeline and stream its result — used for post-deploy smoke tests ("deploy, then run the health-check job").
• bundle summary -t <target>. Emit deployed resource URLs/IDs — handy for pipeline logs and for humans verifying what shipped.
• bundle destroy -t <target>. Tear down everything the bundle deployed — used for ephemeral PR environments and cleanup.
• bundle deployment bind / unbind. Adopt or release existing resources — used once during migration, not on every run.

Service-principal identity for automation.

• Why not a personal token. A personal access token dies when the human is deprovisioned and carries that human's full permissions. A service principal is a stable, non-interactive identity you can scope narrowly and audit independently.
• OAuth M2M. CI authenticates with a service principal client_id + client_secret (OAuth machine-to-machine). The CLI reads DATABRICKS_HOST, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET from the environment — no interactive login.
• Least privilege. The prod service principal gets exactly the workspace permissions it needs to deploy the bundle's resources — CAN_MANAGE on the bundle root, catalog grants for the schemas it writes — and nothing else.
• run_as alignment. The service principal that deploys is usually the same identity that runs prod jobs (run_as.service_principal_name), so both deploy and execution are attributable to one auditable actor.

The gated-promotion shape.

• On PR. bundle validate -t staging (and optionally deploy to an ephemeral PR environment). No prod access.
• On merge to main. bundle deploy -t staging automatically, then run staging smoke tests.
• To prod. Either a git tag (release) or a protected environment with a required reviewer triggers bundle deploy -t prod. The approval is the human gate; the deploy is automated.
• Rollback. git revert the offending commit and re-run the prod deploy — the previous declared state is reapplied.

Worked example — a GitHub Actions PR-validate and prod-deploy workflow

Detailed explanation. The canonical GitHub Actions setup runs bundle validate on every pull request and bundle deploy -t prod only when a version tag is pushed, authenticating as a service principal. Walk through the two workflows.

• PR job. On pull_request, install the CLI, bundle validate -t staging. Read-only, fast, required check.
• Prod job. On push of a v* tag, bundle deploy -t prod as the prod service principal, gated by a protected production environment that requires approval.
• Auth. Databricks host + service-principal client ID/secret from GitHub secrets.

Question. Write the GitHub Actions workflow that validates PRs and deploys to prod on a tag, and explain where the approval gate lives.

Input.

Trigger	Job	Target	Auth
pull_request	validate	staging	staging SP
push tag v*	deploy	prod	prod SP + approval

Code.

# .github/workflows/bundle.yml
name: databricks-bundle

on:
  pull_request:
    branches: [main]
  push:
    tags: ["v*"]

jobs:
  validate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    env:
      DATABRICKS_HOST:          ${{ secrets.STAGING_HOST }}
      DATABRICKS_CLIENT_ID:     ${{ secrets.STAGING_SP_CLIENT_ID }}
      DATABRICKS_CLIENT_SECRET: ${{ secrets.STAGING_SP_CLIENT_SECRET }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main          # installs the databricks CLI
      - run: databricks bundle validate -t staging

  deploy-prod:
    if: startsWith(github.ref, 'refs/tags/v')
    runs-on: ubuntu-latest
    environment: production                       # <-- protected env = approval gate
    env:
      DATABRICKS_HOST:          ${{ secrets.PROD_HOST }}
      DATABRICKS_CLIENT_ID:     ${{ secrets.PROD_SP_CLIENT_ID }}
      DATABRICKS_CLIENT_SECRET: ${{ secrets.PROD_SP_CLIENT_SECRET }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle deploy -t prod
      - run: databricks bundle run health_check -t prod   # post-deploy smoke test

Step-by-step explanation.

• PR validation is the first gate. Every pull request runs bundle validate -t staging with a staging service principal that has no prod access — a broken manifest fails the required check before merge.
• The CLI is installed by an official action. databricks/setup-cli@main puts the databricks binary on the runner, so no manual download or version drift.
• Auth is environment variables, not a login step. Setting DATABRICKS_HOST / DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET from secrets makes the CLI authenticate as the service principal via OAuth M2M automatically.
• The approval gate is the environment: production. GitHub protected environments require a named reviewer to approve before the deploy-prod job runs — the human gate sits between "tag pushed" and "prod mutated."
• Post-deploy smoke test. bundle run health_check -t prod triggers a lightweight verification job after deploy; if it fails, the pipeline fails loudly and you git revert the tag's commit.

Output.

Event	Job that runs	Prod touched?
Open PR	validate (staging SP)	no
Merge to main	(optional staging deploy)	no
Push v1.4.0	deploy-prod, after approval	yes, gated
health_check fails	pipeline red	revert + redeploy

Rule of thumb. Validate with a low-privilege identity on every PR; deploy to prod only from a protected, approval-gated job as a scoped service principal; and always end a prod deploy with a smoke-test run. The pipeline should make "accidentally deploy to prod" structurally impossible.

Worked example — an Azure DevOps pipeline with a service connection

Detailed explanation. Teams on Azure DevOps get the same shape with a YAML pipeline: a validate stage on PRs and a deploy stage gated by an environment approval, authenticating the service principal via pipeline variables (ideally from a variable group backed by Key Vault). Walk through the pipeline.

• Validate stage. Runs on PR; installs the CLI; bundle validate -t staging.
• Deploy stage. Runs on main; targets a prod Azure DevOps environment with a required approval check; bundle deploy -t prod.
• Secrets. Service-principal client ID/secret from a linked variable group.

Question. Write the Azure DevOps pipeline and explain how the environment approval enforces the gate.

Input.

Stage	Condition	Target	Gate
Validate	PR to main	staging	required check
DeployProd	main branch	prod	environment approval

Code.

# azure-pipelines.yml
trigger:
  branches: { include: [main] }

variables:
  - group: databricks-sp                # DATABRICKS_CLIENT_ID / _SECRET from Key Vault

stages:
  - stage: Validate
    jobs:
      - job: validate
        pool: { vmImage: ubuntu-latest }
        steps:
          - script: curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
            displayName: Install Databricks CLI
          - script: databricks bundle validate -t staging
            displayName: Validate bundle
            env:
              DATABRICKS_HOST:          $(STAGING_HOST)
              DATABRICKS_CLIENT_ID:     $(DATABRICKS_CLIENT_ID)
              DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)

  - stage: DeployProd
    dependsOn: Validate
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: deploy_prod
        environment: production          # <-- approval check configured on this env
        pool: { vmImage: ubuntu-latest }
        strategy:
          runOnce:
            deploy:
              steps:
                - script: curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
                - script: databricks bundle deploy -t prod
                  env:
                    DATABRICKS_HOST:          $(PROD_HOST)
                    DATABRICKS_CLIENT_ID:     $(DATABRICKS_CLIENT_ID)
                    DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)

Step-by-step explanation.

• The variable group centralizes secrets. group: databricks-sp links a Key Vault-backed variable group, so the service-principal credentials live in one governed place, not in the YAML.
• Validate is a normal job. It installs the CLI and runs bundle validate -t staging; a failed validate blocks the PR via branch policy.
• Deploy is a deployment job bound to an environment. Azure DevOps environment: production is where you attach the approval and checks — a required reviewer, a business-hours window, or a change-ticket gate.
• The condition scopes prod to main. eq(Build.SourceBranch, 'refs/heads/main') ensures the deploy stage only ever runs off the protected branch, never a feature branch.
• Same identity model as GitHub. The CLI authenticates via DATABRICKS_CLIENT_ID/_SECRET env vars as the service principal — identical OAuth M2M semantics, different runner.

Output.

Event	Stage	Approval	Prod deploy
PR to main	Validate	n/a	no
Merge to main	DeployProd (queued)	required	after approval
Approver approves	deploy_prod runs	granted	yes
Approver rejects	stage cancelled	denied	no

Rule of thumb. Whatever the runner — GitHub, Azure DevOps, GitLab — the pattern is identical: install the CLI, authenticate as a scoped service principal from a secret store, validate cheaply and early, and put the human approval on the deploy environment, not in the script. The tool changes; the shape does not.

Senior interview question on designing the CI/CD promotion flow

A senior interviewer might ask: "Design the full CI/CD flow for a bundle across dev, staging, and prod. Cover which identity runs each stage, what happens on a pull request versus a merge versus a release, how you gate prod, how you smoke-test after deploy, and how you roll back a bad prod deploy. Then show the identity and permission model that makes it least-privilege."

Solution Using scoped service principals, gated environments, and revert-based rollback

# Three service principals, three scopes — least privilege per stage
# (configured in the workspace; referenced by the pipeline + the bundle targets)
#
#   sp-ci-dev      -> CAN_MANAGE on dev bundle root only
#   sp-ci-staging  -> CAN_MANAGE on staging bundle root + staging catalog grants
#   sp-ci-prod     -> CAN_MANAGE on prod bundle root  + prod catalog grants
#
targets:
  dev:     { mode: development, workspace: { host: https://dbc-dev...    } }
  staging:
    mode: production
    workspace: { host: https://dbc-staging... }
    run_as: { service_principal_name: sp-ci-staging }
  prod:
    mode: production
    workspace: { host: https://dbc-prod... }
    run_as: { service_principal_name: sp-ci-prod }
    permissions:
      - { level: CAN_MANAGE, group_name: data-platform }
      - { level: CAN_VIEW,   group_name: data-eng }

# The promotion flow, stage by stage
# 1. Pull request  -> validate only, no deploy, staging SP (read scope)
databricks bundle validate -t staging

# 2. Merge to main -> auto-deploy staging, then smoke test
databricks bundle deploy -t staging
databricks bundle run smoke_suite -t staging      # fail the pipeline if red

# 3. Release tag   -> gated prod deploy (approval on the protected environment)
databricks bundle deploy -t prod
databricks bundle run smoke_suite -t prod

# 4. Bad prod deploy -> revert the commit, redeploy the previous declared state
git revert <bad-sha> && git push
databricks bundle deploy -t prod                  # reapplies prior state (idempotent)

Step-by-step trace.

Stage	Trigger	Identity	Action	Gate
Validate	pull_request	sp-ci-staging	bundle validate	required check
Staging	merge to main	sp-ci-staging	deploy + smoke	branch policy
Prod	release tag	sp-ci-prod	deploy + smoke	env approval
Rollback	incident	sp-ci-prod	git revert + redeploy	same approval

The flow guarantees that no human ever holds prod deploy credentials directly — only sp-ci-prod can mutate prod, and only from the protected pipeline stage. Each pull request is validated cheaply; each merge lands in staging and is smoke-tested; each release is a gated, approved, automated prod deploy followed by a smoke test; and any bad deploy is undone by reverting the commit and reapplying the prior declared state, which is idempotent because the deploy reconciles against stored Terraform state.

Output:

Property	Value
Humans with direct prod creds	none (SP only)
Prod mutation path	protected pipeline stage only
Blast radius of a bad PR	fails validate; never reaches prod
Rollback mechanism	git revert + idempotent redeploy
Audit trail	git history + pipeline logs + SP identity

Why this works — concept by concept:

• Scoped service principals per stage — sp-ci-dev/staging/prod each hold only the permissions their stage needs, so a compromised staging credential cannot touch prod. Least privilege is enforced by identity, not by convention.
• Gated environments — the approval lives on the protected production environment, so "tag pushed" and "prod mutated" are separated by a required human review that the pipeline cannot skip.
• Validate-then-deploy ordering — running bundle validate on every PR and smoke tests after every deploy means broken config is caught before merge and broken deploys are caught before they page anyone.
• Revert-based rollback — because a deploy is an idempotent reconciliation against declared state, git revert + redeploy deterministically restores the previous configuration; there is no "remember what prod looked like" step.
• Cost — the model costs three service principals and a couple of pipeline stages to set up once. It buys an auditable, least-privilege, human-gated promotion path where the blast radius of any mistake is bounded to the stage it happened in — O(1) rollback time versus the O(?) archaeology of undoing a click-ops prod edit. The credential surface shrinks from "every engineer" to "three scoped machine identities."

Design
Topic — design
Design problems on CI/CD promotion topologies

Practice →

ETL Topic — etl ETL problems on automated deployment pipelines

Practice →

---

5. ML resources, advanced patterns, and interview signals

Bundles ship ML like they ship jobs — experiments, registered models, and serving endpoints as code — and the senior signal is knowing the monorepo/polyrepo trade-off and when NOT to use DABs at all

The mental model in one line: an Asset Bundle deploys the whole ML lifecycle as code — an MLflow experiment to track runs, a Unity Catalog registered_model to version the artifact, a training job to produce it, and a model_serving_endpoint to serve it — under the same targets, variables, and CI/CD you use for data jobs, which is why "MLOps on Databricks" is increasingly just "a bundle with ML resources," and why the interview differentiator is architectural judgment: monorepo vs polyrepo, multi-workspace, and the workloads where a bundle is the wrong tool. DABs unify data and ML deployment under one manifest; the senior value is knowing where that unification stops.

The ML resources a bundle can declare.

• resources.experiments.<key>. An MLflow experiment (a name under a workspace path) that training runs log to — so experiment tracking is provisioned, not created by hand.
• resources.registered_models.<key>. A Unity Catalog registered model (catalog.schema.name) that versions the trained artifact with governance and lineage.
• The training job. A normal resources.jobs.<key> whose task trains, evaluates, and calls mlflow.register_model(...) into the UC registered model — the "produce the artifact" step, as code.
• resources.model_serving_endpoints.<key>. A serving endpoint referencing a model version, so the "serve" side (real-time or batch inference) is declared and promoted with the same targets.
• resources.quality_monitors.<key>. Lakehouse Monitoring on a table or inference log, so drift/quality monitoring ships with the model instead of being bolted on later.

Advanced patterns senior engineers reach for.

• Monorepo. One repo, one (or a few) bundles, all domains and ML together. Wins on shared code (one wheel), atomic cross-cutting changes, and a single CI config. Costs a bigger blast radius per change and coarser access control.
• Polyrepo. One repo (and bundle) per team/domain. Wins on ownership isolation, independent release cadence, and fine-grained access. Costs code duplication (shared libs must be published) and cross-repo coordination for changes that span domains.
• Multi-workspace. Targets pointing at different workspaces (per region, per environment, per business unit). One bundle, N targets, each with its own host and identity — the same promotion primitive scaled horizontally.
• Complex variables for reusable specs. A type: complex variable holding a cluster or endpoint spec, shared across many resources and overridden per target — the DRY primitive for ML and data alike.
• deployment bind for adoption. Bring existing UI-created experiments, models, or endpoints under bundle management without recreating them — the same adoption on-ramp as for jobs.

When NOT to use DABs.

• Ad-hoc exploration. Interactive notebook exploration, one-off analyses, and scratch work do not belong in a bundle. Bundles are for things you deploy repeatedly; a notebook you run twice is not that.
• Workspace-level and account-level infrastructure. Creating the workspace itself, metastores, account-level networking, or IAM belongs to Terraform (the Databricks Terraform provider) or account APIs — DABs deploy into a workspace, they do not provision the workspace.
• Non-Databricks resources. Cloud buckets, VPCs, and external services are Terraform/CloudFormation territory. Use DABs for Databricks resources and a general IaC tool for the surrounding cloud.
• A single, truly one-off notebook. If it will never be promoted, scheduled, or reproduced, the ceremony of a bundle is overhead. Know the envelope: DABs shine for repeatable, promoted, multi-environment workloads.

Worked example — train-register-serve as one bundle

Detailed explanation. The end-to-end ML bundle declares an experiment, a UC registered model, a training job that logs and registers a version, and a serving endpoint that serves the latest version — all promotable dev → prod. Walk through the resources.

• Experiment. churn_exp under the team's ML path.
• Registered model. main.ml.churn in Unity Catalog.
• Training job. Runs src/train.py, which logs to the experiment and registers a version.
• Serving endpoint. churn-serving serving the registered model, sized per target.

Question. Declare the four ML resources in one bundle and explain how a new model version reaches production.

Input.

Resource	Value
Experiment	churn_exp
Registered model	main.ml.churn (UC)
Training job	src/train.py
Serving endpoint	churn-serving

Code.

resources:
  experiments:
    churn_exp:
      name: /Shared/ml/churn_exp

  registered_models:
    churn_model:
      catalog_name: ${var.catalog}       # main in prod, dev in dev
      schema_name: ml
      name: churn

  jobs:
    train_churn:
      name: train_churn
      tasks:
        - task_key: train
          notebook_task:
            notebook_path: ../src/train.py
            base_parameters:
              experiment: ${resources.experiments.churn_exp.name}
              model: ${var.catalog}.ml.churn
          new_cluster: ${var.small_cluster}

  model_serving_endpoints:
    churn_serving:
      name: churn-serving
      config:
        served_entities:
          - entity_name: ${var.catalog}.ml.churn
            entity_version: "latest"
            workload_size: ${var.serving_size}   # Small in dev, Medium in prod
            scale_to_zero_enabled: true

Step-by-step explanation.

• The experiment and model are provisioned as code. Deploying the bundle creates the MLflow experiment and the UC registered model if they do not exist — no manual "create experiment" click.
• The training job wires itself by reference. base_parameters inject the experiment name and the fully-qualified model name via ${...}, so train.py logs to the right experiment and registers into the right UC model per target.
• ${var.catalog} makes it environment-aware. In dev the model is dev.ml.churn; in prod it is main.ml.churn. The same training code registers to the correct governed location for each target.
• The serving endpoint is declarative. churn-serving serves the latest version of the registered model, with workload_size and scale_to_zero driven by variables — small and scale-to-zero in dev, larger and warm in prod.
• Promotion is a deploy. deploy -t prod provisions the prod experiment/model/endpoint; the prod training job produces a prod-registered version; the prod endpoint serves it. The lifecycle promotes like any other bundle resource.

Output.

Deploy	Model registered to	Endpoint size	Scale-to-zero
-t dev	dev.ml.churn	Small	on
-t prod	main.ml.churn	Medium	off
new version (retrain)	new version in UC	endpoint serves latest	per target

Rule of thumb. Declare experiment, model, training job, and serving endpoint in one bundle so the entire ML lifecycle promotes together. If your training runs in a bundle but your endpoint is clicked together by hand, you have re-introduced click-ops on the highest-stakes half of the system.

Worked example — monorepo vs polyrepo decision

Detailed explanation. Choosing monorepo or polyrepo for bundles is a recurring senior interview question because it trades shared-code convenience against ownership isolation. Walk through the decision for a platform with three domain teams and a shared feature library.

• Shared code. A feature_lib wheel used by ingestion, transform, and ML.
• Cadence. Ingestion ships daily; ML ships weekly.
• Access control. ML team should not be able to deploy ingestion jobs.

Question. Decide monorepo vs polyrepo for this platform and justify it against the three constraints.

Input.

Factor	Monorepo	Polyrepo
Shared wheel	one build, imported directly	published + versioned per repo
Release cadence	coupled by default	independent per repo
Access control	repo-wide	per-repo, fine-grained
Cross-cutting change	atomic PR	coordinated multi-PR

Code.

# Monorepo layout — one repo, one bundle, shared wheel
platform/
├── databricks.yml            # include: resources/*.yml
├── libs/feature_lib/         # shared wheel, imported by all domains
├── resources/{ingestion,transform,ml}.yml
└── src/{ingestion,transform,ml}/

# Polyrepo layout — one repo + bundle per team; shared lib is PUBLISHED
platform-ingestion/  databricks.yml  (depends on feature_lib==1.4.2 from artifact registry)
platform-transform/  databricks.yml  (depends on feature_lib==1.4.2)
platform-ml/         databricks.yml  (depends on feature_lib==1.5.0)   # own cadence

Step-by-step explanation.

• Monorepo makes shared code trivial. feature_lib is built once and imported directly; a change to it and its consumers is one atomic PR. This is the strongest reason to pick monorepo when code sharing is heavy.
• Monorepo couples cadence by default. Everything in the repo tends to release together unless you add path-based CI filters; the ML team's weekly cadence and ingestion's daily cadence fight over one pipeline unless carefully split.
• Polyrepo isolates ownership and cadence. Each team's repo has its own bundle, CI, and permissions; ML ships weekly without touching ingestion. The cost is that feature_lib must be published to an artifact registry and pinned per repo.
• Access control favors polyrepo when strict. Repo-level permissions cleanly prevent the ML team from deploying ingestion. In a monorepo you approximate this with CODEOWNERS and path protections, which is softer.
• The pragmatic middle. Many platforms run a monorepo with path-scoped CI (validate/deploy only the domains a PR touches) to get shared code plus mostly-independent cadence — the best of both until access-control requirements force a hard split.

Output.

Constraint	Winner	Reason
Heavy shared code	monorepo	direct import, atomic changes
Independent cadence	polyrepo	per-repo release
Strict access control	polyrepo	repo-level permissions
This platform (balanced)	monorepo + path-scoped CI	shared lib + mostly-independent deploys

Rule of thumb. Default to a monorepo with path-scoped CI when teams share code and trust each other; split to polyrepo when release cadence or access control must be hard-isolated. State the trade-off out loud in an interview — the wrong answer is picking one without naming what you gave up.

Senior interview question on the full MLOps bundle plus its limits

A senior interviewer might ask: "Design a bundle that trains a model nightly, registers it to Unity Catalog, promotes it to a serving endpoint only if it beats the current champion, and monitors it in production — across dev and prod. Then tell me what parts of this system you would deliberately NOT put in the bundle, and why."

Solution Using a gated train-eval-promote bundle with an explicit anti-pattern boundary

resources:
  jobs:
    ml_lifecycle:
      name: ml_lifecycle
      tasks:
        - task_key: train
          notebook_task: { notebook_path: ../src/train.py }
          new_cluster: ${var.small_cluster}
        - task_key: evaluate                       # champion/challenger gate
          depends_on: [{ task_key: train }]
          notebook_task:
            notebook_path: ../src/evaluate.py
            base_parameters: { model: ${var.catalog}.ml.churn, min_auc: "0.82" }
          new_cluster: ${var.small_cluster}
        - task_key: promote                        # only runs if evaluate passes
          depends_on: [{ task_key: evaluate }]
          condition_task:                          # gate on the eval task's output
            op: EQUAL_TO
            left: "{{tasks.evaluate.values.beats_champion}}"
            right: "true"
        - task_key: deploy_endpoint
          depends_on: [{ task_key: promote }]
          notebook_task: { notebook_path: ../src/deploy_endpoint.py }
          new_cluster: ${var.small_cluster}
      schedule: { quartz_cron_expression: "0 0 3 * * ?", timezone_id: UTC }

  registered_models:
    churn_model: { catalog_name: ${var.catalog}, schema_name: ml, name: churn }

  quality_monitors:
    churn_monitor:
      table_name: ${var.catalog}.ml.churn_inference_log
      inference_log:
        problem_type: PROBLEM_TYPE_CLASSIFICATION
        prediction_col: prediction
        timestamp_col: ts
        model_id_col: model_version
        granularities: ["1 day"]

# What is DELIBERATELY NOT in the bundle — documented as an ADR next to databricks.yml
# 1. The workspace, metastore, and network      -> Terraform (databricks provider)
# 2. The S3/ADLS landing buckets + IAM roles     -> Terraform / cloud IaC
# 3. Interactive feature exploration notebooks    -> a personal/dev folder, never deployed
# 4. Very-high-QPS custom serving infra           -> dedicated infra if endpoint SLOs don't fit

Step-by-step trace.

Task	Depends on	Gate	Effect
train	—	schedule 03:00	logs run + registers candidate
evaluate	train	AUC ≥ 0.82	sets beats_champion value
promote	evaluate	condition_task	proceeds only if challenger wins
deploy_endpoint	promote	—	points endpoint at new version
churn_monitor	(always)	Lakehouse Monitoring	drift/quality on inference log

The bundle trains nightly, evaluates the challenger against the champion, and uses a condition_task to gate promotion so a worse model never reaches the endpoint; if it wins, deploy_endpoint repoints the serving endpoint at the new version; and quality_monitors watches the production inference log for drift. Crucially, the answer names what stays out of the bundle — the workspace, metastore, buckets, and IAM belong to Terraform; ad-hoc exploration belongs in a dev folder; extreme-SLO serving may need dedicated infra — which is the senior differentiator.

Output:

Concern	In the bundle?	Owner
Train / evaluate / promote / serve	yes	this bundle
Champion/challenger gate	yes (condition_task)	this bundle
Production monitoring	yes (quality_monitors)	this bundle
Workspace / metastore / network	no	Terraform
Landing buckets + IAM	no	cloud IaC
Interactive exploration	no	dev folder

Why this works — concept by concept:

• ML resources as code — declaring the experiment, registered model, training job, and serving endpoint in one manifest means the entire lifecycle promotes with the same targets and CI as data jobs; there is no separate, click-ops MLOps path.
• condition_task gate — a champion/challenger check expressed as a task condition ensures a model is promoted only when it beats the incumbent, encoding the safety gate in the deployable artifact rather than a human's memory.
• quality_monitors — shipping Lakehouse Monitoring in the bundle means drift detection is provisioned with the model, so "we forgot to set up monitoring" cannot happen.
• Explicit anti-pattern boundary — naming the workspace, metastore, buckets, IAM, and ad-hoc exploration as out of scope for DABs shows you know the tool's envelope: bundles deploy Databricks resources into a workspace; they do not provision the workspace or the cloud around it.
• Cost — the bundle costs one manifest and a scheduled multi-task job to express the whole train-eval-promote-monitor loop, promotable across environments for free via targets. The judgment to keep workspace/cloud infra in Terraform keeps each tool in its lane, avoiding the O(pain) of trying to bend DABs into an account-level IaC tool it was never meant to be. The result is O(1) reproducibility for the ML lifecycle and a clean seam to the surrounding infrastructure.

Optimization
Topic — optimization
Optimization problems on model-serving compute sizing

Practice →

Data Transformation
Topic — data-transformation
Transformation problems on feature and inference pipelines

Practice →

---

Cheat sheet — Databricks Asset Bundle recipes

• databricks.yml skeleton. bundle: { name } → include: [resources/*.yml] → variables: (with default, type: complex, or lookup) → artifacts: (type: whl, build: poetry build) → resources: (jobs / pipelines / experiments / registered_models / model_serving_endpoints / schemas / volumes / quality_monitors) → targets: (dev/staging/prod). Exactly one target is default: true. Everything that differs per environment lives in variables + targets, never in resources.
• Core CLI verbs. databricks bundle init (scaffold), validate (schema + workspace check — the PR gate), deploy -t <target> (sync + terraform apply, idempotent), run <resource> -t <target> (trigger + stream, for smoke tests), summary -t <target> (URLs + IDs), destroy -t <target> (tear down), generate job|pipeline (reverse-engineer existing UI resources), deployment bind|unbind (adopt/release live resources). Treat deploy as terraform apply, not scp.
• Target/mode matrix. mode: development = per-user [dev <user>] name prefix + force-paused schedules + dev tags + per-user root path + concurrent runs — isolation for a shared dev workspace. mode: production = strict validation + no name prefix + live schedules + shared root path + non-interactive run_as (service principal). Name the exact differences in interviews; "sandbox vs strict" is the weak answer.
• Variable + lookup template. variables: { catalog: { default: dev }, warehouse_id: { lookup: { warehouse: shared-serverless } }, small_cluster: { type: complex, default: {…} } }. Reference with ${var.name}. Override precedence (highest wins): --var CLI flag > DATABRICKS_BUNDLE_VAR_* env > target variables: > base default. Never commit an ID that varies per workspace — resolve it with a lookup.
• Substitutions worth memorizing. ${bundle.name}, ${bundle.target} (name resources per env: job_${bundle.target}), ${workspace.current_user.userName} / .short_name (dev isolation), ${workspace.root_path}, ${var.x}, and ${resources.jobs.foo.id} / ${resources.pipelines.bar.id} (wire multi-resource DAGs by reference, never by pasted numeric ID).
• CI service-principal auth. CI authenticates as a service principal via OAuth M2M — set DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET from the secret store; install the CLI with databricks/setup-cli@main (GitHub) or the install script (Azure DevOps/GitLab). Never use a personal access token in CI — it dies with the human and carries their full permissions. Scope one service principal per stage (dev/staging/prod) for least privilege.
• Gated promotion shape. PR → bundle validate -t staging (read-only, required check). Merge to main → bundle deploy -t staging + smoke bundle run. Release tag or protected environment → gated, approved bundle deploy -t prod + smoke. Put the human approval on the deploy environment, not in the script. Rollback = git revert <sha> + redeploy (idempotent reconcile of prior state).
• ML resource block. Declare experiments (tracking), registered_models (UC catalog.schema.name), a training job that logs + registers a version, and model_serving_endpoints (served entity + workload_size + scale_to_zero) so the whole train → register → serve lifecycle promotes with the same targets. Gate promotion with a condition_task (champion/challenger) and ship quality_monitors for drift so monitoring is never bolted on later.
• Adopting existing resources. databricks bundle generate job --existing-job-id <id> (or pipeline) reverse-engineers UI-built resources into resources/*.yml + source; databricks bundle deployment bind <key> <id> -t <target> links the YAML to the live resource so the next deploy updates it in place instead of creating a duplicate. This is the click-ops → code on-ramp — do it once per resource, then flip run_as to a service principal.
• Monorepo vs polyrepo. Monorepo = one repo/bundle, shared wheel imported directly, atomic cross-cutting changes, coarser access control; add path-scoped CI so only touched domains deploy. Polyrepo = one repo/bundle per team, published + pinned shared lib, independent cadence, repo-level access control, cross-repo coordination cost. Default monorepo + path-scoped CI; split to polyrepo when cadence or access control must be hard-isolated.
• When NOT to use DABs. Not for ad-hoc/interactive exploration (a notebook you run twice is not a deploy unit). Not for workspace/metastore/account-level or non-Databricks cloud infra — that is Terraform (databricks provider) + cloud IaC territory; DABs deploy into a workspace, they do not create it. Not for a truly one-off notebook that will never be promoted. Knowing the envelope and naming it is the senior signal.
• Failure-mode cheat sheet. Duplicate resource after adopting → you skipped deployment bind. Prod points at dev's warehouse/policy → hard-coded ID instead of a lookup. Dev deploy fired a real schedule → target wasn't mode: development (or preset trigger_pause_status wrong). Wrong variable value in prod → check override precedence (--var > env > target > default). Out-of-band UI edit reverted on deploy → expected: deploy reconciles declared state; stop editing prod in the UI. CI can't auth → personal token used instead of service-principal OAuth M2M env vars.
• First-minute interview framing. "A Databricks Asset Bundle is a directory of source plus a databricks.yml that declares jobs, pipelines, and ML resources as code, deployed via the Databricks CLI on top of Terraform. Targets give me dev/staging/prod from one code base with per-env variables, mode: development vs production, and a run_as service principal for prod. CI runs bundle validate on every PR and gated bundle deploy -t prod behind an approval, so humans review and machines apply. I keep workspace-level and cloud infra in Terraform — DABs deploy into a workspace, they don't provision it." That is the whole senior framing in one paragraph.

Frequently asked questions

What are Databricks Asset Bundles in one sentence?

databricks asset bundles (DABs) are Databricks' native infrastructure-as-code format: a project directory containing your source code plus a databricks.yml manifest that declares jobs, Delta Live Tables / Lakeflow pipelines, MLflow experiments, Unity Catalog registered models, and serving endpoints as code, which the Databricks CLI validates and deploys — on top of the Databricks Terraform provider — so an entire workspace's worth of jobs, pipelines, and ML assets can be version-controlled, reviewed as diffs, promoted across dev/staging/prod targets, and rolled back from git. The mental model is "Terraform for the things inside a Databricks workspace, with friendly YAML instead of HCL." Every senior data-engineering interview probes DABs now because they are the standard answer to "how do you get Databricks workloads out of click-ops and into CI/CD," replacing both hand-built UI jobs and the older dbx tool that DABs superseded.

DABs vs Terraform vs dbx — which do I use for what?

Use Databricks Asset Bundles for the resources inside a workspace — jobs, pipelines, models, endpoints, schemas, volumes — because DABs give you friendly YAML, mode: development/production semantics, per-user dev isolation, and bundle run for smoke tests, all purpose-built for the Databricks workload lifecycle. Use Terraform (the databricks provider plus your cloud provider) for workspace-level and account-level infrastructure — creating the workspace itself, metastores, Unity Catalog catalogs at the account level, networking, and cloud IAM/buckets — which DABs deliberately do not manage. The two are complementary and DABs actually run on the Terraform provider under the hood, so they share concepts (declared state, plan/apply, idempotency). dbx was the previous-generation open-source deployment tool for Databricks jobs; it is now legacy and Databricks recommends migrating dbx projects to Asset Bundles — treat dbx as deprecated for new work. The clean division of labor: Terraform builds the workspace and cloud around it; bundles deploy the workloads into it.

How do the dev and prod targets actually differ?

They differ by the target's mode plus its per-target overrides. mode: development prefixes every deployed resource name with [dev <your-username>], force-pauses all schedules and triggers, tags resources as dev, sets pipelines to development mode, allows concurrent runs, and deploys under a per-user root path — so many engineers can each deploy an isolated, non-firing copy into one shared dev workspace without colliding. mode: production applies strict validation, forbids the dev name prefix, keeps schedules live, uses a shared root path, and (as best practice) requires a non-interactive run_as service principal. Beyond mode, each target typically overrides the workspace host, the catalog and cluster-size variables, the run_as identity, and permissions. The critical property is that the resources themselves are identical across targets — only the target overlay changes — so promotion is "deploy the same code to a different target," never "edit the job for prod."

How do service principals fit into DAB CI/CD?

A service principal is the non-interactive identity your CI pipeline uses to deploy, and (usually) the identity production jobs run_as. In CI, the pipeline authenticates via OAuth machine-to-machine by setting DATABRICKS_HOST, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET (the service principal's OAuth credentials) as environment variables from the runner's secret store — the CLI then acts as that service principal with no interactive login. This matters for three reasons: stability (a service principal survives any individual leaving the team, unlike a personal access token), least privilege (you scope one service principal per stage — sp-ci-dev/staging/prod — so a compromised staging credential cannot touch prod), and auditability (every deploy and every prod job run is attributable to a named machine identity, not "whoever happened to click deploy"). The best-practice model is: no human holds prod deploy credentials directly; only the prod service principal can mutate prod, and only from a protected, approval-gated pipeline stage.

Can DABs deploy DLT pipelines and ML models, not just jobs?

Yes — that is the whole point of "one repo for jobs, pipelines, and ML." Under resources, a bundle can declare pipelines (Delta Live Tables / Lakeflow Declarative Pipelines, with catalog, schema, libraries, serverless, and configuration), experiments (MLflow tracking), registered_models (Unity Catalog catalog.schema.name model versioning), model_serving_endpoints (real-time or batch serving of a model version with a workload_size and scale_to_zero), and quality_monitors (Lakehouse Monitoring for drift) — alongside jobs, schemas, volumes, clusters, dashboards, and apps. Because all of these live under the same targets, variables, and CI, the entire lifecycle — a training job that registers a model, a serving endpoint that serves it, a monitor that watches it — promotes together across dev and prod. You cross-reference them by substitution (${resources.pipelines.orders_dlt.id}) so a job can trigger a pipeline, or a serving endpoint can point at a registered model, without any hard-coded IDs that break between environments.

When should I NOT use Databricks Asset Bundles?

Do not use DABs for three categories of work. First, ad-hoc and interactive exploration — a scratch notebook, a one-off analysis, or exploratory feature work you run a couple of times is not a deployment unit; the ceremony of a manifest and targets is pure overhead there, so keep it in a dev folder. Second, workspace-level and account-level infrastructure — creating the Databricks workspace itself, metastores, account networking, or cloud IAM roles and storage buckets belongs to Terraform (the databricks provider) and general cloud IaC; DABs deploy resources into an existing workspace, they do not provision the workspace or the cloud around it. Third, non-Databricks resources — anything outside a Databricks workspace (external services, VPCs, DNS) is another tool's job. The senior signal in a DAB interview is naming this envelope unprompted: bundles are the right tool for repeatable, promoted, multi-environment Databricks workloads, and the wrong tool for one-off exploration and the infrastructure layer beneath the workspace. Every technology has a workload envelope; knowing DABs' envelope is what separates "I've used bundles" from "I know where bundles stop."

Practice on PipeCode

• Drill the ETL practice library → for the reproducible-deployment, incremental-ingestion, and multi-environment-promotion problems that Asset Bundle workflows are built to standardize.
• Rehearse on the design practice library → for the infrastructure-as-code, CI/CD topology, service-principal identity, and monorepo-vs-polyrepo questions senior interviewers open with when DABs are on the table.
• Sharpen the transform layer with the data-transformation practice library → for the DLT pipeline, feature, and inference-pipeline patterns a bundle declares as code.
• Tune the cost axis with the optimization practice library → for the per-target cluster-sizing and serving-endpoint workload-size decisions that make dev cheap and prod fast.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the target/mode/variable decision matrix against real graded inputs.

Lock in Databricks Asset Bundle muscle memory

Docs explain the YAML. PipeCode drills explain the decision — when `mode: development` isolates a shared workspace, when a `lookup` beats a hard-coded ID, when a service principal must own prod, when a bundle is the wrong tool and Terraform owns the layer beneath it. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face across jobs, pipelines, and ML in one repo.

Practice ETL problems →
Practice design problems →