DEV Community: Jin

How to Connect Azure SQL Database in Azure Databricks with PySpark

Jin — Wed, 12 Aug 2026 06:59:55 +0000

In this article, I will show how to connect Azure SQL Database from Azure Databricks, read a SQL table or query result, and write a Spark DataFrame back to Azure SQL Database.

We will use PySpark because Azure Databricks is built for distributed data processing. PySpark can read data from external databases into Spark DataFrames, transform the data at scale, and write the result back to Azure SQL Database. Microsoft also provides Spark connector support for SQL Server and Azure SQL as input data sources and output sinks for Spark jobs.

Connect the Azure SQK Database

define the Azure SQL Database connection settings.

Read data from Azure SQL Database

Read a full table and Read a SQL query
To read a full table, use the dbtable option. To run a SQL query against Azure SQL Database and load the result into Databricks, use the query option.

Databricks also supports JDBC-style reads and writes with Spark Data Source APIs, including reading from a table or pushing down a source SQL query.

Write Data to Azure SQL Database

In this example, we first create a small pandas DataFrame and convert it to a Spark DataFrame.

import pandas as pd

data = {
    "Product": [
        "Laptop", "Mouse", "Keyboard", "Monitor", "Headset",
        "Webcam", "USB Hub", "Laptop", "Mouse", "Monitor"
    ],
    "Region": [
        "North", "South", "East", "West", "North",
        "East", "South", "West", "North", "East"
    ],
    "Units_Sold": [15, 42, 30, 8, 25, 17, 60, 22, 38, 10],
    "Revenue": [22500, 1260, 2400, 3200, 1875, 850, 1800, 33000, 1140, 4000]
}

pdf = pd.DataFrame(data)

df = spark.createDataFrame(pdf)

display(df)

In Azure Databricks notebooks, spark is usually available automatically. If you run the code outside Databricks, you may need to create a SparkSession manually.

Use .write.format("sqlserver") and provide the connection options.

df.write
  .format("sqlserver")
  .options(**SQL_OPTIONS)
  .option("dbtable", "dbo.test_table")
  .mode("overwrite") #or "append"
  .save()

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Databricks Asset Bundles, Git, and ALM Explained and with Real Exam Questions

Jin — Tue, 11 Aug 2026 08:00:13 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand how Azure Databricks projects are deployed and managed across environments.

This article focuses on:

Declarative Automation Bundles, also known in many exam-style questions as Databricks Asset Bundles
databricks.yml
bundle targets
Databricks CLI deployment
Git folders
Git merge
merge conflicts
application lifecycle management, ALM

The related questions in your DP-750 question bank are Q12, Q62, Q66, Q67, Q73, Q80, and Q81.

1. Why ALM matters in Azure Databricks

In real data engineering projects, you do not want to manually copy notebooks, jobs, pipelines, and configuration from development to production.

You need a repeatable process for:

source control
deployment
environment-specific configuration
promotion from development to production
CI/CD automation
rollback
team collaboration

This is called application lifecycle management, or ALM.

For DP-750, the common ALM pattern is:

Use Git for source control.
Use Declarative Automation Bundles for deployable Databricks resources.
Use Databricks CLI to validate, deploy, and run bundles.
Use targets to parameterize environments such as dev, test, and prod.

Microsoft documentation describes Declarative Automation Bundles as the recommended approach to CI/CD on Databricks. Bundles describe resources such as jobs and pipelines as source files, can be source controlled, and can be deployed through external CI/CD automation.

2. What are Declarative Automation Bundles?

Declarative Automation Bundles are a way to define a Databricks project as files.

They were formerly known as Databricks Asset Bundles, and many exam questions still use the older name.

A bundle can include:

jobs
pipelines
apps
notebooks
Python files
SQL files
configuration
environment-specific targets

A bundle is usually defined with a YAML file named:

databricks.yml

Microsoft documentation says bundle metadata is defined using YAML files that specify the artifacts, resources, and configuration of a Databricks project. The Databricks CLI can then validate, deploy, and run bundles using these YAML files.

For DP-750, remember:

Declarative Automation Bundles = repeatable deployment of Databricks resources

3. Why use bundles instead of manual workspace export/import?

Manual export and import can work for simple cases, but it is not a strong ALM process.

It is weak because:

it is manual
it is hard to repeat consistently
environment parameters are difficult to manage
promotion between dev/test/prod is error-prone
source control is weaker
CI/CD integration is harder

Bundles are better because the project is defined as code.

For example, a bundle can define:

Job A
Job B
Pipeline C
App D
dev target
prod target

Then the same project can be deployed repeatedly.

For DP-750, if the requirement says:

Deploy notebooks, workflows, and jobs in a consistent and repeatable way.
Parameterize the target deployment environment.
Provide source control.

the answer should point to:

Declarative Automation Bundles in Git

4. `databricks.yml`

The main configuration file for a bundle is usually:

databricks.yml

A very small example looks like this:

bundle:
  name: my_bundle

targets:
  dev:
    default: true

Microsoft documentation says the simplest databricks.yml defines the bundle name and a target deployment.

In real projects, databricks.yml often contains:

bundle:
  name: dp750_project

resources:
  jobs:
    ingestion_job:
      name: ingestion_job
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ./src/ingest.py

targets:
  dev:
    default: true
    workspace:
      host: https://adb-dev.azuredatabricks.net

  prod:
    mode: production
    workspace:
      host: https://adb-prod.azuredatabricks.net

For DP-750, you usually do not need to memorize every YAML property. You need to know what each major node is for.

5. Important bundle nodes

The most important nodes for DP-750 are:

Node	Purpose
`bundle`	Defines bundle metadata, such as the bundle name
`resources`	Defines deployable resources, such as jobs, pipelines, and apps
`targets`	Defines deployment environments such as dev and prod
`variables`	Defines reusable parameters
`workspace`	Defines target workspace settings
`artifacts`	Defines build artifacts such as Python packages

For exam questions, the most important one is usually:

targets

6. What are targets?

A target is an environment-specific deployment configuration.

Common examples:

dev
test
prod

Targets let you deploy the same bundle to different environments with different settings.

For example:

targets:
  dev:
    default: true
    workspace:
      host: https://adb-dev.azuredatabricks.net

  prod:
    mode: production
    workspace:
      host: https://adb-prod.azuredatabricks.net

Microsoft documentation explains that target workspaces are defined in the targets mapping of the bundle’s databricks.yml.

For DP-750:

Deploy same bundle to dev and prod = use targets

7. Targets for selective deployment

Some exam questions ask something like:

You have two jobs and an app.
You need to deploy the app to both environments.
You need to deploy only one job to development.
Minimize administrative effort.

This points to target-specific configuration.

The answer is:

Use a targets node in databricks.yml.

Why?

Because targets lets you define different deployment behavior for different environments.

A simplified example:

resources:
  jobs:
    job_a:
      name: job_a

    job_b:
      name: job_b

  apps:
    my_app:
      name: my_app

targets:
  dev:
    resources:
      jobs:
        job_a:
          name: job_a_dev

  prod:
    resources:
      jobs:
        job_a:
          name: job_a_prod
        job_b:
          name: job_b_prod
      apps:
        my_app:
          name: my_app_prod

The exact syntax depends on the project, but the concept is:

targets control environment-specific deployment behavior.

8. Databricks CLI and bundles

The Databricks CLI is the standard tool used to work with bundles.

Common commands include:

databricks bundle validate

databricks bundle deploy --target dev

databricks bundle run <job-name> --target dev

Microsoft documentation says you use Databricks CLI bundle commands to create, validate, deploy, run, and destroy bundles.

For DP-750:

Deploy Declarative Automation Bundles = Databricks CLI

not:

Jobs UI
Git folders only
Databricks SDK for Python
Azure CLI

Those tools may be useful in other scenarios, but the standard bundle deployment tool is the Databricks CLI.

9. `databricks bundle deploy --target dev`

One common exam question asks how to complete the command for deploying a bundle to the dev environment.

The correct command is:

databricks bundle deploy --target dev

The Databricks CLI bundle command documentation says that to deploy a bundle to a specific target, you can use the -t or --target option.

For DP-750:

Requirement	Command
Deploy bundle to dev	`databricks bundle deploy --target dev`
Validate bundle	`databricks bundle validate`
Run bundle workflow	`databricks bundle run`

10. Git folders in Azure Databricks

Azure Databricks Git folders integrate notebooks and source files with Git repositories.

They allow teams to:

clone repositories
create branches
commit changes
push changes
pull changes
merge branches
resolve conflicts

Microsoft documentation describes Git folders as supporting repository cloning, branch management, commits, merge conflict resolution, and Git CLI commands.

For DP-750, Git folders are often tested through basic Git operations:

merge
pull
push
rebase
conflict resolution

11. Merge vs rebase vs pull vs push

A common DP-750 question asks:

From the main branch, you create Branch1 and commit changes to Branch1.
You need to incorporate the changes from Branch1 into main.
The solution must preserve commit history.
Which Git operation should you use?

The correct answer is:

merge

Microsoft documentation says the merge function in Azure Databricks Git folders uses git merge to combine the commit history from one branch into another. It also recommends merge instead of rebase for Git beginners because merge does not rewrite commit history.

For DP-750:

Git operation	Meaning
`merge`	Combine changes from one branch into another while preserving history
`rebase`	Replay commits on top of another branch; can rewrite history
`pull`	Fetch and integrate remote changes into current branch
`push`	Send local commits to remote repository

If the question says:

preserve commit history

choose:

merge

12. Merge conflicts

A merge conflict happens when Git cannot automatically combine changes.

For example:

You changed Notebook1 in Branch1.
Another user changed the same part of Notebook1 in main.
You try to merge Branch1 into main.
Git cannot decide which version to keep.

In this case, you must resolve the conflict manually.

Databricks documentation says that to manually resolve conflicts, you edit the file contents, select the lines you want to preserve, remove merge conflict markers, and mark the conflict as resolved.

For DP-750, the best answer pattern is:

Apply the main branch changes to Branch1 and resolve the conflicts.

This lets you include all changes from both branches before merging.

13. Why not clone a new repository?

Some wrong answers suggest cloning the main branch or feature branch as a new repository.

That does not solve the merge conflict.

A conflict is about combining changes in the same repository history.

Creating a new clone may give you another working copy, but it does not automatically integrate both sets of changes into main.

For DP-750:

Merge conflict = resolve conflict

not:

clone a new repository

14. Git folders vs Declarative Automation Bundles

Git folders and bundles are related, but they are not the same.

Feature	Purpose
Git folders	Source control and collaboration
Declarative Automation Bundles	Define and deploy Databricks resources
Databricks CLI	Execute bundle commands such as validate/deploy/run
Targets	Environment-specific deployment configuration

A strong production ALM pattern usually combines them:

Code and bundle files stored in Git.
Bundle defines jobs, pipelines, apps, and configuration.
Databricks CLI deploys the bundle to dev/test/prod.
CI/CD system automates the deployment.

For DP-750:

Git alone is not enough for repeatable environment deployment.
Bundles alone without Git are weaker for source control.
Bundles in Git is the best ALM pattern.

15. Declarative Automation Bundles vs Terraform

Terraform can manage Databricks resources, and it can be useful for infrastructure-as-code scenarios.

However, DP-750 questions about notebooks, workflows, jobs, target environment parameterization, and repeatable project deployment often expect:

Declarative Automation Bundles in Git

This is because bundles are specifically designed for Databricks project deployment and CI/CD.

For DP-750:

Requirement	Best answer pattern
Deploy notebooks, workflows, and jobs as a project	Declarative Automation Bundles
Source control project files	Git
Parameterize target environment	Bundle targets
Deploy bundle	Databricks CLI
Manage lower-level cloud infrastructure	Terraform may be relevant, but not usually the DP-750 bundle answer

16. DP-750 decision table for bundles, Git, and ALM

Scenario in the question	Best answer pattern
Repeatable deployment of notebooks, workflows, and jobs	Declarative Automation Bundles
Source control for Databricks project	Git
Parameterize dev/test/prod	`targets` in `databricks.yml`
Deploy bundle to dev	`databricks bundle deploy --target dev`
Tool used to deploy bundles	Databricks CLI
Bundle contains two jobs and an app with environment-specific deployment	`targets` node
Preserve commit history when combining branches	Merge
Merge fails because of conflicts	Apply main changes to branch and resolve conflicts
Clone new repository to solve conflict	Wrong
Manual workspace export/import for ALM	Weak / not preferred

Real Exam Questions

Question 12

You have an Azure Databricks solution that was developed by multiple engineers and deployed to development, test, and production environments.

You need to implement an application lifecycle management, ALM, process that meets the following requirements:

Deploys and provides source control for Databricks notebooks, workflows, and jobs in a consistent and repeatable way.
Parameterizes the target deployment environment.

What should you include in the solution?

A. the export and import of Databricks workspaces

B. Databricks resources managed by using the Databricks Terraform provider

C. Git folders and manual promotion

D. Declarative Automation Bundles in Git ✅ Correct Answer

Question 62

You use Databricks Asset Bundles to manage two jobs and an app.

You need to deploy the bundle to development and production environments.

The solution must meet the following requirements:

Deploy the app to both environments.
Deploy only one job to development.
Minimize administrative effort.

What should you use?

A. separate databricks.yml files for each environment

B. a variables node in a databricks.yml file

C. a resources node in a databricks.yml file

D. a targets node in a databricks.yml file ✅ Correct Answer

Question 66

You have an Azure Databricks workspace that contains a Git folder and uses an Azure Repos Git repository.

From the main branch, you create a branch named Branch1 and commit changes to Branch1.

You need to incorporate the changes from Branch1 into main. The solution must preserve the commit history in the repository.

Which Git operation should you use?

A. rebase

B. pull

C. merge ✅ Correct Answer

D. push

Question 67

You have an Azure Databricks workspace named Workspace1 that uses a Git repository. The repository contains a Databricks notebook named Notebook1.

From the main branch, you create a feature branch named Branch1 and commit changes to Notebook1. Another user commits changes to Notebook1 in main.

When you attempt to merge Branch1 into main, the merge fails due to conflicts.

You need to merge Branch1 into the main branch. The solution must ensure that Notebook1 includes all the changes from both the branches.

What should you do?

A. From Workspace1, clone the main branch as a new repository.

B. Apply the main branch changes to Branch1 and resolve the conflicts. ✅ Correct Answer

C. From Workspace1, clone Branch1 as a new repository.

D. Apply the changes directly to the main branch.

Question 73

You need to deploy Databricks Asset Bundles to a development environment. The solution must support automated and repeatable deployments across environments.

What should you use?

A. Git folders

B. the Databricks CLI ✅ Correct Answer

C. the Databricks SDK for Python

D. the Jobs UI

Question 80

You have a Declarative Automation Bundle.

You plan to use the Databricks CLI to deploy the bundle to an environment named dev.

How should you complete the CLI command?

Area	Answer
Dropdown 1	`databricks bundle` ✅ Correct Answer
Dropdown 2	`deploy --target dev` ✅ Correct Answer

Completed command:

databricks bundle deploy --target dev

Question 81

You need to deploy Declarative Automation Bundles to a development environment. The solution must support automated and repeatable deployments across environments.

What should you use?

A. the Databricks CLI ✅ Correct Answer

B. Git folders

C. the Azure Developer CLI, azd

D. the Azure Command-Line Interface, CLI

Key takeaways

For DP-750, Databricks Asset Bundles, Git, and ALM questions usually test whether you know how to deploy Databricks projects repeatably across environments.

Remember these patterns:

Declarative Automation Bundles are the modern Databricks project deployment mechanism.
Many exam questions still call them Databricks Asset Bundles.
Bundle configuration is usually stored in databricks.yml.
resources define deployable objects such as jobs, pipelines, and apps.
targets define environment-specific deployment settings.
The Databricks CLI is used to validate, deploy, and run bundles.
To deploy to dev, use databricks bundle deploy --target dev.
Git folders provide source control and collaboration.
Use merge to incorporate one branch into another while preserving commit history.
If a merge conflict happens, apply the latest main branch changes to the feature branch and resolve the conflict.
Git folders alone are not the same as a full ALM deployment strategy.
Bundles in Git provide source-controlled, repeatable, environment-aware Databricks deployment.

If you can separate source control, deployment configuration, environment targets, and Git branch operations, these DP-750 ALM questions become much easier.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

Azure Databricks: Fix “DataFrame Object Has No Attribute write”

Jin — Tue, 11 Aug 2026 08:00:07 +0000

While writing data from an Azure Databricks notebook to SQL Server, I encountered the following error:

AttributeError: 'DataFrame' object has no attribute 'write'

The notebook was running on Azure Databricks Serverless Compute.

The error occurred because the DataFrame was created with pandas, but the code attempted to use the PySpark .write API.

The Problem

Suppose we have a Pandas DataFrame:

import pandas as pd

data = {
    "Product": [
        "Laptop", "Mouse", "Keyboard", "Monitor", "Headset",
        "Webcam", "USB Hub", "Laptop", "Mouse", "Monitor"
    ],
    "Region": [
        "North", "South", "East", "West", "North",
        "East", "South", "West", "North", "East"
    ],
    "Units_Sold": [15, 42, 30, 8, 25, 17, 60, 22, 38, 10],
    "Revenue": [
        22500, 1260, 2400, 3200, 1875,
        850, 1800, 33000, 1140, 4000
    ]
}

df = pd.DataFrame(data)

We then try to write the DataFrame to SQL Server:

df.write \
    .format("sqlserver") \
    .options(**SQL_OPTIONS) \
    .option("dbtable", "test_table") \
    .mode("overwrite") \
    .save()

The operation fails with:

Why This Error Happens

The variable df is a Pandas DataFrame, not a PySpark DataFrame.

You can confirm this by checking its type:

print(type(df))

The result will be similar to:

<class 'pandas.core.frame.DataFrame'>

Pandas and PySpark both use the name DataFrame, but they are different Python classes with different APIs.

A Pandas DataFrame provides writing methods such as:

df.to_csv()
df.to_excel()
df.to_parquet()
df.to_sql()

It does not provide the Spark .write property.

The following syntax belongs to the PySpark DataFrame API:

df.write.format(...).options(...).mode(...).save()

For a PySpark DataFrame, .write returns a DataFrameWriter, which is responsible for writing data to external storage systems and databases.

The Solution

Convert the Pandas DataFrame to a PySpark DataFrame before calling .write:

df = spark.createDataFrame(df)

After the conversion, the write operation works:

df.write \
    .format("sqlserver") \
    .options(**SQL_OPTIONS) \
    .option("dbtable", "test_table") \
    .mode("overwrite") \
    .save()

spark.createDataFrame() can create a PySpark DataFrame directly from a Pandas DataFrame. Spark converts the local Pandas data into a Spark DataFrame and infers the column schema when no schema is provided.

Complete Working Example

Using different variable names makes it clear which DataFrame type is being used:

Summary

The error is caused by using a PySpark method on a Pandas DataFrame.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Slowly Changing Dimensions and Data Quality Expectations Explained and with Real Exam Questions

Jin — Mon, 10 Aug 2026 12:02:13 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand two related but different topics:

Slowly Changing Dimensions, SCD
Data quality expectations in Lakeflow Spark Declarative Pipelines

SCD is about how to model changes in dimension data over time.

Data quality expectations are about validating records as they flow through a pipeline.

The related questions in your DP-750 question bank are Q36, Q55, Q57, Q76, Q77, and Q78.

1. What is a Slowly Changing Dimension?

A Slowly Changing Dimension, or SCD, is a data modeling pattern used when dimension records change over time.

A dimension table usually stores descriptive business data, such as:

customer profile
equipment metadata
product information
employee information
supplier information
IoT sensor ownership

Example:

equipment_id = E001
equipment_name = Turbine A
manufacturer = Contoso Energy
owner = Team West

Over time, some values may change:

owner = Team Central

The question is:

Should we overwrite the old value, or should we keep the full history?

That is where SCD types are used.

2. SCD Type 1

SCD Type 1 overwrites existing values.

It does not preserve historical versions.

Example before update:

equipment_id	equipment_name	manufacturer
E001	Turbine A	Contoso Energy

Example after correction:

equipment_id	equipment_name	manufacturer
E001	Turbine Alpha	Contoso Energy

The old value, Turbine A, is replaced.

Use SCD Type 1 when:

the old value is not needed
the change is a correction
the previous value was a data-entry mistake
only the latest value matters

Databricks documentation describes SCD Type 1 as history tracking off: outdated records are overwritten when they are updated or deleted in the source.

For DP-750:

Historical values are NOT required = SCD Type 1
Data-entry mistake correction = SCD Type 1
Overwrite existing values = SCD Type 1

3. SCD Type 2

SCD Type 2 keeps history.

Instead of overwriting the old row, it creates a new row and marks which version is current.

Example before ownership change:

sensor_id	owner	valid_from	valid_to	is_current
S001	Team West	2025-01-01	null	true

Example after ownership change:

sensor_id	owner	valid_from	valid_to	is_current
S001	Team West	2025-01-01	2026-03-01	false
S001	Team Central	2026-03-01	null	true

Use SCD Type 2 when:

business users need historical tracking
analysts need to know what was true at a previous time
ownership changes over time
full history must be preserved

Databricks documentation describes SCD Type 2 as history tracking on: the ingestion pipeline keeps the old row, adds the update as a new row, and marks the old row as inactive.

For DP-750:

Track full history = SCD Type 2
Ownership changes over time = SCD Type 2
Keep old row and add new row = SCD Type 2

4. SCD Type 1 vs SCD Type 2

Requirement	SCD type
Correct data-entry mistakes	Type 1
Historical values are not required	Type 1
Only latest value matters	Type 1
Overwrite existing values	Type 1
Track ownership changes over time	Type 2
Keep full history	Type 2
Add a new row for each change	Type 2
Support historical analysis	Type 2

This is one of the most important DP-750 decision patterns.

If the question says:

Historical values are NOT required.

choose Type 1.

If the question says:

Analysts must track the full history.

choose Type 2.

5. SCD Type 2 and Lakeflow

Lakeflow Spark Declarative Pipelines can support CDC and SCD patterns.

Databricks documentation shows that Lakeflow SDP can create a Type 2 SCD table to track all changes, and uses AUTO CDC ... INTO to apply changes into the final table.

In Lakeflow SQL, SCD Type 2 can be expressed with:

STORED AS SCD TYPE 2

In Python, it can be expressed with:

stored_as_scd_type="2"

For DP-750, you usually do not need to memorize all syntax details. The most important concept is:

SCD Type 2 = persistent record-level history

6. Delta Lake time travel vs SCD Type 2

Delta Lake time travel and SCD Type 2 are related, but they are not the same thing.

Delta Lake time travel lets you query earlier table versions by timestamp or version number. Databricks documentation says time travel supports querying previous table versions based on timestamp or table version recorded in the transaction log.

Example:

SELECT *
FROM customer_profiles VERSION AS OF 10;

SCD Type 2 stores business history directly in the table as rows.

Example:

customer_id	status	valid_from	valid_to	is_current
C001	Silver	2025-01-01	2026-01-01	false
C001	Gold	2026-01-01	null	true

For DP-750:

Requirement	Best answer
Maintain persistent historical record changes	SCD Type 2
Query earlier Delta table versions	Delta Lake time travel
Correct wrong values without keeping history	SCD Type 1

7. What are data quality expectations?

In Lakeflow Spark Declarative Pipelines, expectations are data quality constraints.

They validate records as data flows through the pipeline.

An expectation is usually written as a Boolean condition, such as:

amount > 0

or:

transaction_id IS NOT NULL

Databricks documentation says expectations apply quality constraints that validate data as it flows through ETL pipelines. They can provide metrics, drop invalid records, or fail updates when invalid records are detected.

For DP-750, expectations usually appear in questions about:

validating records
dropping invalid records
failing pipeline updates
collecting data quality metrics
preventing bad records from being written
quarantining invalid records

8. Expectation actions

There are three main expectation behaviors.

Behavior	Meaning
Retain invalid records	Keep records, but collect metrics
Drop invalid records	Do not write invalid records to the target
Fail on invalid records	Stop the update when invalid data is detected

In current Lakeflow documentation, expectation decorators are in the pyspark.pipelines module, commonly imported as dp. The exam questions may still use the older dlt naming, but the behavior is the important part. Databricks documentation lists decorators such as expect, expect_or_drop, and expect_or_fail.

9. `expect`

expect validates data and records metrics, but it does not drop or fail invalid records.

Example:

@dlt.expect("valid_amount", "amount > 0")

Behavior:

Valid records are written.
Invalid records are also written.
Metrics are collected.
Pipeline continues.

Use this when:

you want to monitor data quality
invalid records can remain in the target
you only need metrics

For DP-750:

Collect metrics only = expect

10. `expect_or_drop`

expect_or_drop drops invalid records before they are written to the target.

Example:

@dlt.expect_or_drop("valid_amount", "amount > 0")

Behavior:

Valid records are written.
Invalid records are dropped.
Metrics are collected.
Pipeline continues.

Databricks documentation says expect_or_drop prevents further processing of invalid records, and records that violate the expectation are dropped from the target dataset.

Use this when:

invalid records must not be written
the pipeline should continue
you want expectation metrics
you do not need to preserve invalid records in the same target table

For DP-750:

Invalid records must NOT be written, but pipeline continues = expect_or_drop

11. `expect_or_fail`

expect_or_fail fails the update when invalid records are detected.

Example:

@dlt.expect_or_fail("rule1", "amount > 0")

Behavior:

If invalid records are detected, the update fails.
The transaction is rolled back.
Invalid data is not committed.

Databricks documentation says expect_or_fail stops execution immediately when a record fails validation. If the operation is a table update, the system atomically rolls back the transaction.

Use this when:

invalid records are unacceptable
the target table must not be updated
the update must not be committed if invalid data exists

For DP-750:

Prevent update from being committed = expect_or_fail

12. Expectation metrics

Lakeflow expectations can produce data quality metrics.

Databricks documentation says metrics for warn or drop actions can be viewed from the pipeline UI, and the Data quality tab is available in the pipeline UI for pipeline datasets with expectations. It also says expectation metrics can be queried from the Lakeflow Spark Declarative Pipelines event log.

For DP-750:

Requirement	Answer pattern
Review expectation metrics with minimal effort	Pipeline Data Quality tab
Query metrics programmatically	Pipeline event log
Invalid records fail update	Metrics may not be recorded because update fails

13. Quarantine invalid records

A quarantine pattern means invalid records are preserved somewhere for review.

For example:

valid_records      -> target table
invalid_records    -> quarantine table

This is different from simply dropping invalid records.

If the requirement says:

Invalid records must NOT be written to Table1.
Invalid records must be preserved for review.
Pipeline must continue.

then expect_or_drop alone is not enough, because it drops invalid records and does not preserve them as reviewable records.

The answer is usually to implement a quarantine pattern, such as writing invalid records to a separate quarantine table.

For DP-750:

Drop invalid records from target + preserve invalid records = quarantine logic

14. Expectations vs table constraints

Expectations and table constraints both validate data, but they are used differently.

Feature	Typical use
Lakeflow expectations	Validate records during pipeline processing
`NOT NULL` constraint	Enforce non-null values at table write time
`CHECK` constraint	Enforce Boolean rule at table write time
Quarantine logic	Preserve invalid records for review

For example:

amount > 0

could be implemented as:

an expectation in a Lakeflow pipeline
a CHECK constraint on a Delta table

But the exam context matters.

If the question says:

Lakeflow Spark Declarative Pipelines pipeline
rule1
expectation

then use expectation decorators.

If the question says:

table-level data quality enforcement
invalid records must be rejected when written to the table

then use table constraints.

15. DP-750 decision table for SCD and expectations

Scenario in the question	Best answer pattern
Correct data-entry mistake	SCD Type 1
Historical values are not required	SCD Type 1
Overwrite name and address values	SCD Type 1
Track ownership changes over time	SCD Type 2
Keep full history of profile changes	SCD Type 2
Query earlier Delta table versions	Delta Lake time travel
Monitor data quality only	`expect`
Drop invalid records and continue	`expect_or_drop`
Fail update when invalid data exists	`expect_or_fail`
Prevent table update from being committed	`expect_or_fail`
Preserve invalid records for review	Quarantine logic
View expectation metrics with minimal effort	Pipeline Data Quality tab
Query expectation metrics	Pipeline event log

Real Exam Questions

Question 36

Useful case information

Contoso identifies the following data modeling and optimization requirements:

Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.

Contoso also has the following governance issue:

Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership.
Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.

Which SCD type should you use to support the planned data modeling changes?

Issue	Answer
Data-entry mistakes by the equipment manufacturers	Type 1 ✅ Correct Answer
Changes to IoT Sensor ownership	Type 2 ✅ Correct Answer

Question 55

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a managed Delta table named Table1.

Table1 stores customer profile data.

Business users must analyze how customer profile records change over time. They must also be able to query earlier versions of the table.

You need to implement a solution that:

Maintains persistent historical versions of customer profile records for long-term analysis.
Allows users to query earlier versions of the Delta table.
Minimizes maintenance effort.

What should you do?

Area	Answer
To record historical changes	Implement a Type 2 slowly changing dimension, SCD ✅ Correct Answer
To support temporal analysis	Use Delta Lake time travel ✅ Correct Answer

Question 57

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Table1. Table1 is used as a dimension table and contains the following columns:

id
name
address

You need to apply a Type 1 slowly changing dimension, SCD, approach when updates are processed in the name and address columns.

What will occur when an update is processed?

A. A new row will be added for each change to a name and address.

B. The value of the id column will be updated.

C. The existing name and address values will be overwritten. ✅ Correct Answer

D. Historical versions of the row and its effective dates will be retained.

Question 76

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You have a Lakeflow Spark Declarative Pipelines, SDP, pipeline that writes numerical data to a table named Table1 by using a data quality validation rule named rule1.

You need to modify rule1 to meet the following requirements:

Ensure that amount is always greater than 0.
Prevent an update to Table1 from being committed when data that violates rule1 is detected.

Which statement should you execute?

A. @dlt.expect_or_fail("rule1", "amount > 0") ✅ Correct Answer

B. @dlt.expect("rule1", "amount > 0")

C. @dlt.expect_all_or_drop({"rule1": "amount > 0"})

D. @dlt.expect_or_drop("rule1", "amount > 0")

Question 77

You have a Lakeflow Spark Declarative Pipelines, SDP, pipeline in Azure Databricks. The pipeline ingests transaction data into a table named Table1.

You need to ensure that in the event of an invalid record, the pipeline continues to run.

The solution must meet the following requirements:

Invalid records must NOT be written to Table1.
Invalid records must be preserved for review.
Minimize development effort.

What should you do?

A. Add a CHECK constraint to Table1.

B. Run WHERE clauses in downstream queries to filter out invalid records.

C. Define a pipeline expectation.

D. Implement advanced logic to quarantine the invalid records. ✅ Correct Answer

Question 78

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You have a Lakeflow Spark Declarative Pipelines, SDP, pipeline that writes records to a Delta table named Table1 by using a data quality rule named rule1.

You need to meet the following requirements:

Records that violate rule1 must NOT be written to Table1, but the pipeline must continue processing valid records.
Data engineers must be able to review expectation metrics by using minimal development effort.

What should you do?

Area	Answer
Set action for rule1 to	`dlt.expect_or_drop` ✅ Correct Answer
View expectation metrics in	The Data Quality tab of the pipeline ✅ Correct Answer

Key takeaways

For DP-750, SCD and expectation questions are mainly decision-pattern questions.

Remember these patterns:

SCD Type 1 overwrites existing values.
SCD Type 1 is used when historical values are not required.
SCD Type 1 is good for correcting data-entry mistakes.
SCD Type 2 keeps historical versions.
SCD Type 2 is used when analysts must track full history.
Delta Lake time travel lets users query earlier table versions.
expect keeps invalid records and records metrics.
expect_or_drop drops invalid records and continues processing.
expect_or_fail fails the update and prevents the transaction from being committed.
If invalid records must be preserved for review, use a quarantine pattern.
Expectation metrics can be reviewed in the pipeline Data Quality tab or queried from the event log.
In current Lakeflow documentation, the decorators use dp, but many exam-style questions still show dlt.

If you can separate overwrite vs history, and monitor vs drop vs fail vs quarantine, these DP-750 questions become much easier.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Delta Lake Performance Optimization Explained and with Real Exam Questions

Jin — Thu, 06 Aug 2026 06:59:55 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand how to keep Delta table performance stable as data grows.

This article focuses on three common DP-750 performance topics:

liquid clustering
deletion vectors
partitioning decisions

The related questions in your DP-750 question bank are Q9 and Q58.

1. Why Delta Lake performance optimization matters

Delta tables often start small, but production tables can grow quickly.

For example:

100 GB
500 GB
1 TB
10 TB
100 TB

As the table grows, query performance can become unstable if data is not organized well.

Common symptoms include:

queries become slower over time
filters scan too much data
update and delete operations take too long
partitioning becomes difficult to manage
too many small partitions appear
table maintenance becomes expensive

For DP-750, the key question is usually:

Which Delta table optimization feature should you use for this workload?

2. What is data layout?

Data layout means how data is physically organized in storage.

Even if users query a table logically like this:

SELECT *
FROM events
WHERE event_time BETWEEN '2026-01-01' AND '2026-01-31';

the query engine still needs to find the relevant files in storage.

If the table is organized well, Databricks can skip irrelevant files and scan less data.

If the table is organized poorly, Databricks may scan too many files, which increases:

query duration
compute cost
latency
storage I/O

For DP-750, data layout is mainly tested through:

liquid clustering
partitioning
Z-ORDER, mostly as background knowledge

3. What is liquid clustering?

Liquid clustering is a Delta Lake data layout optimization technique.

Azure Databricks documentation describes liquid clustering as a data layout optimization technique that replaces table partitioning and ZORDER, simplifying table management and improving query performance. It is designed for tables that grow over time and need flexible layout optimization.

In simple terms:

Liquid clustering organizes data based on clustering columns so that Databricks can skip more irrelevant data during queries.

Example:

CREATE TABLE events (
    event_id STRING,
    event_time TIMESTAMP,
    event_type STRING
)
CLUSTER BY (event_time);

Or for an existing table:

ALTER TABLE events
CLUSTER BY (event_time);

For DP-750, if a large table is frequently filtered by a column such as event_time, and the question wants to avoid over-partitioning, the answer is usually:

Liquid clustering

4. Why liquid clustering is preferred over fixed partitioning

Traditional partitioning physically separates data into directory-like groups.

Example:

/event_date=2026-01-01/
/event_date=2026-01-02/
/event_date=2026-01-03/

Partitioning can be useful, but it can also create problems:

too many small partitions
uneven partition sizes
high metadata overhead
hard-to-change layout
poor performance if query patterns change

Azure Databricks documentation now recommends liquid clustering for all new Delta tables and managed Iceberg tables, and the SQL CLUSTER BY documentation recommends automatic liquid clustering and predictive optimization for Unity Catalog managed tables.

For DP-750, this is very important:

Large growing table + filter columns + avoid over-partitioning = liquid clustering

5. Liquid clustering vs partitioning

Feature	Liquid clustering	Traditional partitioning
Main purpose	Flexible data layout optimization	Directory-level data organization
Good for growing tables	Yes	Sometimes
Avoids over-partitioning	Yes	No
Easy to adapt to query patterns	Better	Harder
Replaces ZORDER for many use cases	Yes	No
Recommended for new Delta tables	Yes	Not usually the first choice

For DP-750, partitioning is usually wrong when the question says:

avoid over-partitioning
remain efficient as data volume grows

This wording strongly points to liquid clustering.

6. Choosing clustering columns

A clustering column should usually be a column frequently used in filters.

Examples:

WHERE event_time BETWEEN ...

WHERE CustomerId = ...

WHERE EventDate >= ...

If a table is frequently queried by filtering on:

CustomerId
EventDate

then those are strong candidates for liquid clustering.

If a table is frequently queried by filtering on:

event_time

then event_time is a strong clustering column.

For DP-750, the exact syntax is less important than the decision pattern:

Choose the layout strategy that matches common query filters.

7. What are deletion vectors?

Deletion vectors are a Delta Lake storage optimization feature that accelerates row-level table modifications.

Azure Databricks documentation says deletion vectors accelerate DELETE, UPDATE, and MERGE operations. Without deletion vectors, deleting a single row requires rewriting the entire Parquet file that contains the record. With deletion vectors enabled, Databricks can mark rows as modified without immediately rewriting the whole Parquet file.

This matters when a table has frequent:

deletes
updates
merges
compliance-related removal requests
row-level changes

For DP-750, if the question says:

Rows are frequently updated and deleted.
The solution must minimize update and deletion effort.

then the answer is usually:

Enable deletion vectors

8. Why deletion vectors help with updates and deletes

Without deletion vectors, a small update or delete can be expensive.

Example:

One Parquet file contains 1,000,000 rows.
You delete 10 rows.

Without deletion vectors, the system may need to rewrite the file without those 10 rows.

With deletion vectors, Databricks can record which rows are deleted or changed without immediately rewriting the full file.

This reduces the cost of row-level operations.

For DP-750:

Requirement	Feature
Improve filtering performance as table grows	Liquid clustering
Minimize update/delete effort	Deletion vectors
Frequent compliance deletes	Deletion vectors
Avoid over-partitioning	Liquid clustering

9. Deletion vectors and table properties

For Delta Lake tables, deletion vectors can be enabled with the table property:

ALTER TABLE table_name
SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);

Databricks documentation notes that deletion vectors are not automatically enabled for every Delta table in every situation, and that they can be enabled using the enableDeletionVectors table property. Enabling deletion vectors upgrades the table protocol, so client compatibility should be considered.

For DP-750, you usually do not need to worry about protocol details unless the question explicitly mentions compatibility with older clients.

The exam pattern is simpler:

Frequent UPDATE / DELETE / MERGE = enable deletion vectors

10. Liquid clustering and deletion vectors together

Liquid clustering and deletion vectors solve different problems.

Problem	Best feature
Query performance with filters	Liquid clustering
Growing table performance	Liquid clustering
Avoid over-partitioning	Liquid clustering
Frequent row updates	Deletion vectors
Frequent deletes	Deletion vectors
Compliance delete requests	Deletion vectors

In Q9, both issues appear:

The table is queried frequently by filtering on CustomerId and EventDate.
The table is frequently updated and deleted for compliance.

So the solution includes both:

Liquid clustering
Deletion vectors

11. What about OPTIMIZE?

OPTIMIZE rewrites data files to improve file layout. Azure Databricks documentation says the OPTIMIZE command rewrites data files to improve layout for Delta Lake and Apache Iceberg tables. For liquid clustered tables, OPTIMIZE clusters the data according to the clustering columns.

For DP-750, OPTIMIZE is more often tested in table maintenance or retention questions.

In the performance questions from this article, the correct answers focus on:

liquid clustering
deletion vectors

not manual partitioning or fixed partition counts.

12. What about ZORDER?

ZORDER was historically used to colocate related data and improve data skipping.

However, current Azure Databricks documentation describes liquid clustering as replacing table partitioning and ZORDER for Delta table layout optimization.

For DP-750, if the answer options include liquid clustering, and the question asks for a modern data layout strategy for a growing table, liquid clustering is usually the stronger answer.

13. Common DP-750 traps

Trap 1: Partition by event_time

If the table is huge and queries filter on ranges in event_time, it may seem natural to partition by event_time.

But if event_time has high cardinality or fine granularity, partitioning can create too many partitions.

If the question says:

avoids over-partitioning
remains efficient as data volume grows

choose liquid clustering.

Trap 2: Fixed number of partitions

A fixed number of partitions based on table size sounds controlled, but it is not a query-aware layout strategy.

It does not directly optimize filtering on event_time, CustomerId, or EventDate.

For DP-750, if the requirement is query performance for specific filter columns, use liquid clustering.

Trap 3: Partition by low-value column

If a table has only a few values in event_type, partitioning by event_type may not help range filters on event_time.

For example:

WHERE event_time BETWEEN '2026-01-01' AND '2026-01-31'

Partitioning by event_type does not directly align with this filter.

For DP-750, match the layout to the query filter columns.

Trap 4: Use only deletion vectors for query filters

Deletion vectors help with row-level modifications, not primary data layout for filtering.

If the problem is:

frequent queries filter by CustomerId and EventDate

you need liquid clustering.

If the problem is:

frequent updates and deletes

you need deletion vectors.

If the question has both, use both.

14. DP-750 decision table for Delta performance

Scenario in the question	Best answer pattern
Large table grows over time	Liquid clustering
Frequent filters on one or more columns	Liquid clustering
Avoid over-partitioning	Liquid clustering
Range filters on `event_time`	Liquid clustering on `event_time`
Frequent filters on `CustomerId` and `EventDate`	Liquid clustering
Frequent updates and deletes	Enable deletion vectors
Compliance deletes	Enable deletion vectors
Minimize update/delete effort	Enable deletion vectors
Fixed partitions by total table size	Usually wrong
Partition by high-cardinality timestamp	Risk of over-partitioning
Partition by unrelated column	Does not match filter pattern

Real Exam Questions

Question 9

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a managed Delta table named Table1.

Table1 is written by batch jobs every hour and is queried frequently by filtering two columns named CustomerId and EventDate.

You expect Table1 to grow significantly over time.

The rows in Table1 are frequently updated and deleted to support compliance requests.

You need to keep query performance consistent as Table1 grows. The solution must minimize update and deletion effort.

What should you include in the solution? To answer, select the appropriate options in the answer area.

Area	Answer
Data layout strategy	Liquid clustering ✅ Correct Answer
Update and deletion optimization	Enable deletion vectors ✅ Correct Answer

Question 58

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Table1.

Table1 contains approximately 1.5 TB of data and the following columns:

event_time
event_type

Queries against Table1 frequently filter on ranges in event_time.

You need to implement a data layout strategy that improves query performance for filters on event_time, avoids over-partitioning, and remains efficient as the data volume grows over time.

What should you do?

A. Implement liquid clustering on Table1. ✅ Correct Answer

B. Create a fixed number of partitions based on the total table size.

C. Partition Table1 by using the event_type column.

D. Partition Table1 by using the event_time column.

Key takeaways

For DP-750, Delta Lake performance optimization questions usually test whether you can identify the right feature for the right performance problem.

Remember these patterns:

Liquid clustering improves data layout for query filtering.
Liquid clustering is preferred for growing Delta tables.
Liquid clustering helps avoid over-partitioning.
Deletion vectors optimize row-level updates, deletes, and merges.
Frequent compliance deletes point to deletion vectors.
Range filters on a large timestamp-based table point to liquid clustering.
Do not choose fixed partitions just because the table is large.
Do not partition by a column that does not match the query filter.
If the question mentions both query filters and frequent deletes, use both liquid clustering and deletion vectors.

If you can separate query layout optimization from row-level modification optimization, these DP-750 questions become much easier.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Delta Lake Table Operations Explained and with Real Exam Questions

Jin — Wed, 05 Aug 2026 04:59:56 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand common Delta Lake and Spark table operations.

This article focuses on operations that appear frequently in DP-750 questions:

INSERT INTO
INSERT OVERWRITE
table constraints
NOT NULL
CHECK
INTERSECT
append writes
PySpark null filtering
dropna
fillna

The related questions in your DP-750 question bank are Q40, Q41, Q42, Q45, Q49–52, and Q53.

1. Why table operations matter in DP-750

Many DP-750 questions are not about advanced Spark optimization. They test whether you understand the basic behavior of table operations.

For example:

INSERT INTO table2
SELECT * FROM table1;

This preserves existing rows and adds new rows.

But:

INSERT OVERWRITE table2
SELECT * FROM table1;

replaces existing data.

Similarly, in PySpark:

df.dropna(subset=["order_amount"])

removes rows where order_amount is null, while:

df.fillna(0, subset=["order_amount"])

does not remove rows. It replaces null values with 0.

These details are simple, but they are very common exam traps.

2. `INSERT INTO`

INSERT INTO inserts new rows into a table. Azure Databricks documentation describes INSERT as a statement that inserts new rows into a table, using either value expressions or the result of a query.

Example:

INSERT INTO sales.schema1.table2
SELECT * FROM sales.schema1.table1;

This means:

Take rows from table1 and append them to table2.

It does not delete existing rows from table2.

Use INSERT INTO when the requirement says:

preserve existing data
add new rows
append new records
load current-year data into historical table
daily file contains new records only
minimize processing effort

For DP-750:

Preserve existing data + add new rows = INSERT INTO

3. `INSERT OVERWRITE`

INSERT OVERWRITE replaces existing data in the target table or partition.

This is different from INSERT INTO.

Example:

INSERT OVERWRITE sales.schema1.table2
SELECT * FROM sales.schema1.table1;

This means:

Replace the existing data in table2 with the query result.

In DP-750, if the question says:

Preserve any existing data

then INSERT OVERWRITE is usually wrong.

Use INSERT OVERWRITE only when the requirement is to replace existing data.

4. `CREATE TABLE AS SELECT` vs `CREATE OR REPLACE TABLE AS SELECT`

These commands are also common traps.

CREATE TABLE sales.schema1.table2 AS
SELECT * FROM sales.schema1.table1;

creates a new table from query results.

CREATE OR REPLACE TABLE sales.schema1.table2 AS
SELECT * FROM sales.schema1.table1;

creates a table or replaces the existing table.

For DP-750:

Requirement	Correct pattern
Add rows to an existing table	`INSERT INTO`
Replace existing table definition and data	`CREATE OR REPLACE TABLE AS SELECT`
Create a new table from query result	`CREATE TABLE AS SELECT`
Preserve existing data	Avoid `INSERT OVERWRITE` and `CREATE OR REPLACE`

If the target table already exists and you need to preserve existing data, INSERT INTO is the safest answer.

5. Table constraints

Azure Databricks supports table constraints for data quality. Microsoft documentation says constraints can define and enforce data quality rules, including NOT NULL and CHECK constraints. NOT NULL means values in specific columns cannot be null, and CHECK means a specified Boolean expression must be true for each row.

Common examples:

transaction_id STRING NOT NULL

CHECK (amount > 0)

Use constraints when the requirement says:

table-level data quality enforcement
invalid records must be rejected during writes
a column must never be null
a numeric value must be greater than zero
enforce the rule when data is written

For DP-750:

Reject invalid records at table write time = table constraints

6. `NOT NULL` constraint

A NOT NULL constraint prevents a column from containing null values.

Example:

CREATE TABLE Sales (
    transaction_id STRING NOT NULL,
    transaction_date DATE,
    amount DECIMAL(10,2)
);

If a write tries to insert a row where transaction_id is null, the write fails.

For DP-750:

transaction_id must never be null

means:

Add a NOT NULL constraint to transaction_id

7. `CHECK` constraint

A CHECK constraint requires a Boolean expression to be true.

Example:

ALTER TABLE Sales
ADD CONSTRAINT valid_amount CHECK (amount > 0);

If a write tries to insert a row where amount <= 0, the write fails.

For DP-750:

amount must be greater than 0

means:

Add a CHECK constraint to amount

Do not use a view or a WHERE clause if the requirement is to reject invalid records during table writes. A view only changes what users see when querying. It does not enforce the rule at write time.

8. `INTERSECT`

INTERSECT returns rows that appear in both query results. Azure Databricks documentation describes INTERSECT as one of the supported set operators used to combine query results. Both subqueries must have the same number of columns and compatible types.

Example:

SELECT Column1
FROM Table1
INTERSECT
SELECT Column2
FROM Table2;

This returns values that appear in both result sets.

Now look at this pattern:

SELECT Column1
FROM Table1
GROUP BY Column1
HAVING COUNT(*) > 1

INTERSECT

SELECT Column2
FROM Table2
GROUP BY Column2
HAVING COUNT(*) > 1;

The first query returns values that appear more than once in Table1.

The second query returns values that appear more than once in Table2.

INTERSECT returns only values that appear in both results.

So the final result is:

Values that appear more than once in both tables.

9. PySpark null filtering

In PySpark, null handling is a common exam trap.

To keep only rows where a column is not null, use:

df.filter(df.order_amount.isNotNull())

Spark documentation says Column.isNotNull() returns true if the current expression is not null.

You can also use:

df.dropna(subset=["order_amount"])

Spark documentation says DataFrame.dropna() returns a new DataFrame omitting rows with null or NaN values.

For DP-750:

Exclude rows where order_amount is null = isNotNull() or dropna(subset=[...])

10. `fillna` does not remove rows

fillna replaces null values. It does not remove rows.

Example:

df.fillna(0, subset=["order_amount"])

If a row has:

order_amount = null

then after fillna, it becomes:

order_amount = 0

Spark documentation says DataFrame.fillna() returns a new DataFrame where null values are filled with a new value.

For DP-750:

Requirement: exclude rows where order_amount is null

fillna(0) = wrong

Because replacing null with 0 is not the same as excluding the row.

11. Why `df.order_amount != None` is wrong

This is another common Spark trap.

In normal Python, you might write:

x != None

But Spark DataFrame columns are not normal Python values. They are distributed column expressions. Null comparison in Spark SQL follows SQL null semantics, and comparisons with null do not behave like normal Boolean comparisons.

So this expression is not the correct exam answer:

df.filter(df.order_amount != None)

Use:

df.filter(df.order_amount.isNotNull())

or:

df.dropna(subset=["order_amount"])

For DP-750:

Null filtering in Spark = use isNotNull() or dropna()

12. Writing DataFrames to Delta

A common PySpark write pattern is:

df.write \
    .format("delta") \
    .mode("append") \
    .save(path)

This means:

Code	Meaning
`.format("delta")`	Write data in Delta format
`.mode("append")`	Add rows to existing data
`.save(path)`	Save to the target path

For DP-750, if the source is CSV and the target is an existing Delta table or Delta path, the pattern is:

spark.read.format("csv")
df.write.format("delta").mode("append")

Use append when the requirement says:

each new CSV file must be added
existing Delta table must be preserved
new rows only
no updates to existing rows

13. Append vs overwrite

This is the same idea as INSERT INTO vs INSERT OVERWRITE.

SQL	PySpark write mode	Meaning
`INSERT INTO`	`.mode("append")`	Add new rows
`INSERT OVERWRITE`	`.mode("overwrite")`	Replace existing data

For DP-750:

Preserve existing data = append
Replace existing data = overwrite

14. DP-750 decision table for table operations

Scenario in the question	Best answer pattern
Preserve existing data and add rows	`INSERT INTO`
Daily file contains new records only	`INSERT INTO` or append mode
Replace existing data	`INSERT OVERWRITE` or overwrite mode
Create a new table from query result	`CREATE TABLE AS SELECT`
Recreate existing table from query result	`CREATE OR REPLACE TABLE AS SELECT`
`transaction_id` must never be null	`NOT NULL` constraint
`amount` must be greater than 0	`CHECK` constraint
Invalid records must be rejected on write	Table constraints
Values appear more than once in both tables	`GROUP BY` + `HAVING COUNT(*) > 1` + `INTERSECT`
Exclude rows where column is null	`isNotNull()` or `dropna(subset=[...])`
Replace nulls with zero	`fillna(0)`
Add new CSV data to Delta path	`.format("delta").mode("append")`

Real Exam Questions

Question 40

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a managed Delta table named Sales. Sales stores transaction data and contains the following columns:

transaction_id, string
transaction_date, date
amount, decimal

You need to implement the following data quality requirements by using table-level data quality enforcement:

amount must be greater than 0.
transaction_id must never be null.
Invalid records must be rejected when data is written to the Sales table.

What should you do?

A. Use a SELECT statement with WHERE conditions to validate the data before querying.

B. Create a view that filters out rows where transaction_id is null or amount is less than or equal to 0.

C. Add a NOT NULL constraint to transaction_id and a CHECK constraint to amount. ✅ Correct Answer

D. Configure row-level security, RLS, where transaction_id is null or amount is less than or equal to 0.

Question 41

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains two Delta tables named Table1 and Table2 of the same data type.

Table1 contains a column named Column1. Table2 contains a column named Column2.

You run the following query.

SELECT Column1
FROM Table1
GROUP BY Column1
HAVING COUNT(*) > 1

INTERSECT

SELECT Column2
FROM Table2
GROUP BY Column2
HAVING COUNT(*) > 1;

What occurs when you run the query?

A. Values appear in both tables more than once. ✅ Correct Answer

B. Values appear in either table more than once.

C. Values appear in Table2 but NOT Table1.

D. Values appear in Table1 more than once.

Question 42

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains two managed Delta tables named sales.schema1.table1 and sales.schema1.table2.

sales.schema1.table1 contains sales data from the current year. sales.schema1.table2 contains historical data.

You need to load all the rows from sales.schema1.table1 into sales.schema1.table2.

The solution must preserve any existing data in sales.schema1.table2 and minimize processing effort.

Which command should you run?

A. INSERT OVERWRITE sales.schema1.table2 SELECT * FROM sales.schema1.table1;

B. CREATE TABLE sales.schema1.table2 AS SELECT * FROM sales.schema1.table1;

C. INSERT INTO sales.schema1.table2 SELECT * FROM sales.schema1.table1; ✅ Correct Answer

D. CREATE OR REPLACE TABLE sales.schema1.table2 AS SELECT * FROM sales.schema1.table1;

Question 45

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Sales_orders.

Sales_orders stores historical sales data.

You receive a daily CSV file daily that contains new sales records only. The file does NOT contain updates to existing rows.

You need to load the daily data into Sales_orders. The solution must meet the following requirements:

Preserve the existing data.
Add only the new records.
Minimize processing effort.

Which command should include in the loading strategy?

A. UPDATE

B. INSERT OVERWRITE

C. INSERT INTO ✅ Correct Answer

Question 49

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Orders.

You load the Orders table into an Apache Spark DataFrame named df.

You need to create a DataFrame that excludes rows where the order amount is null.

Solution: You run the following expression.

df.fillna(0, subset=["order_amount"])

Does this meet the goal?

A. Yes

B. No ✅ Correct Answer

Question 50

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Orders.

You load the Orders table into an Apache Spark DataFrame named df.

You need to create a DataFrame that excludes rows where the order amount is null.

Solution: You run the following expression.

df.filter(df.order_amount.isNotNull())

Does this meet the goal?

A. Yes ✅ Correct Answer

B. No

Question 51

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Orders.

You load the Orders table into an Apache Spark DataFrame named df.

You need to create a DataFrame that excludes rows where the order amount is null.

Solution: You run the following expression.

df.dropna(subset=["order_amount"])

Does this meet the goal?

A. Yes ✅ Correct Answer

B. No

Question 52

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Orders.

You load the Orders table into an Apache Spark DataFrame named df.

You need to create a DataFrame that excludes rows where the order amount is null.

Solution: You run the following expression.

df.filter(df.order_amount != None)

Does this meet the goal?

A. Yes

B. No ✅ Correct Answer

Question 53

You have an Azure Databricks workspace.

You have an Azure Data Lake Storage Gen2 account named account1.

You need to use a Databricks notebook to read CSV files from account1. The data from each new CSV file must be added to an existing Delta table named customer.

How should you complete the PySpark code segment?

basePath = "abfss://data@storage1.dfs.core.windows.net/"

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .load(basePath)
)

df.write
    .format("delta")
    .mode("append")
    .save(basePath + "delta/customers")

✅ Correct Answer

Key takeaways

For DP-750, Delta Lake table operation questions usually test basic but important behavior.

Remember these patterns:

INSERT INTO adds rows and preserves existing data.
INSERT OVERWRITE replaces existing data.
.mode("append") adds data.
.mode("overwrite") replaces data.
NOT NULL prevents null values in a column.
CHECK enforces a Boolean condition such as amount > 0.
Views and WHERE clauses do not enforce table-level write rules.
INTERSECT returns values that appear in both query results.
fillna() replaces nulls; it does not remove rows.
isNotNull() keeps only non-null rows.
dropna(subset=[...]) removes rows with nulls in the specified column.
df.order_amount != None is not the correct Spark null-filtering pattern.

If you understand the difference between append vs overwrite, query filtering vs write enforcement, and null replacement vs null exclusion, these DP-750 table operation questions become much easier.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Structured Streaming, Checkpointing, Schema Evolution, and Change Data Feed Explained and with Real Exam Questions

Jin — Tue, 04 Aug 2026 04:59:55 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand how Azure Databricks handles streaming and incremental data processing.

This topic includes several important concepts:

Apache Spark Structured Streaming
checkpointing
exactly-once processing
availableNow trigger
schema drift and schema evolution
Auto Loader schema evolution modes
Delta table streaming reads and writes
Change Data Feed, CDF

The related questions in your DP-750 question bank are Q8, Q43, Q47, Q48, Q54, Q59, and Q71.

1. What is Structured Streaming?

Apache Spark Structured Streaming is Spark’s stream processing engine. In Azure Databricks, it is commonly used to process data continuously or incrementally from sources such as:

cloud storage files
Delta tables
Event Hubs
Kafka
Auto Loader
streaming tables

In DP-750, Structured Streaming often appears in questions about:

avoiding duplicate processing after failure
writing to Delta tables
checkpoint locations
schema changes
Event Hubs ingestion
streaming recovery
change data capture

For exam purposes, the most important idea is:

Structured Streaming needs checkpointing to remember progress and recover after failures.

2. What is checkpointing?

A checkpoint stores streaming query progress and metadata.

For example, if a streaming job has already processed files or events, the checkpoint helps the job remember what has already been processed.

A checkpoint location is usually configured like this:

.option("checkpointLocation", checkpoint_path)

Azure Databricks documentation states that checkpointLocation is required for fault tolerance and exactly-once processing guarantees, and each streaming query must use a unique checkpoint location.

For DP-750, the key pattern is:

Streaming job reprocesses old data after restart = missing or wrong checkpoint location.

So if the question says:

After the cluster restarts, the streaming job reprocesses previously ingested data.

the answer is:

Configure a checkpoint location for the streaming query.

3. Exactly-once processing

Exactly-once processing means each record is processed into the target exactly once, even if there is a failure and restart.

In Databricks, exactly-once behavior is usually achieved through:

Structured Streaming
checkpointing
Delta Lake transaction log
idempotent writes
unique checkpoint location per query

Databricks documentation says Delta Lake is deeply integrated with Structured Streaming and that the Delta transaction log guarantees exactly-once processing when writing data into a Delta table using Structured Streaming.

For DP-750, remember:

Streaming + Delta target + checkpointLocation = fault tolerance / no reprocessing

4. Checkpoint location vs schema location

When using Auto Loader, you often see both:

.option("cloudFiles.schemaLocation", "/cloud/schema")
.option("checkpointLocation", "/data/checkpoint")

They are not the same thing.

Option	Purpose
`checkpointLocation`	Stores streaming progress and state
`cloudFiles.schemaLocation`	Stores inferred schema information for Auto Loader

Auto Loader documentation says specifying cloudFiles.schemaLocation enables schema inference and evolution. The same directory can be used for checkpointLocation, but the two options serve different purposes.

For DP-750:

If the issue is reprocessing after restart, choose checkpointLocation.
If the issue is schema inference or schema evolution, choose cloudFiles.schemaLocation or schema evolution settings.

5. Schema drift and schema evolution

Schema drift means the source data structure changes over time.

Example:

{
  "sensor_id": "S001",
  "temperature": 21.5
}

Later, the source sends a new column:

{
  "sensor_id": "S001",
  "temperature": 21.5,
  "humidity": 62
}

If the pipeline cannot handle the new column, it may fail.

Schema evolution allows the target schema to evolve when new columns appear.

Databricks documentation says Auto Loader supports column changes and type widening, and schema evolution can be configured with cloudFiles.schemaEvolutionMode.

For DP-750:

New columns are added to the source and pipeline writes fail = enable schema evolution.

Do not choose row filters. Row filters control which rows users can see; they do not solve schema drift.

6. `mergeSchema`

When writing to Delta, mergeSchema allows additive schema changes, such as adding new columns.

Example:

.option("mergeSchema", "true")

This is commonly used when incoming data may contain new columns that should be added to the target Delta table.

In the DP-750 question bank, Q43 uses:

.option("mergeSchema", "true")

This means new columns from incoming JSON payloads can be added automatically to the target Delta table.

7. Auto Loader schema evolution mode: `failOnNewColumns`

Auto Loader has several schema evolution modes.

One of them is:

.option("cloudFiles.schemaEvolutionMode", "failOnNewColumns")

This mode can be confusing.

With failOnNewColumns, when Auto Loader detects new columns, the stream fails so that the schema can evolve. After restart, the evolved schema is used. Databricks documentation describes this pattern: new columns fail the stream and evolve the schema, and the stream can then be restarted with the updated inferred schema.

For DP-750, if a question asks:

When a new element is found in the source data, will the element be written successfully?

and the code uses:

.option("cloudFiles.schemaEvolutionMode", "failOnNewColumns")

then the answer is:

No

Because the stream fails when it detects the new column.

8. `availableNow` trigger

The availableNow trigger processes all available data as an incremental batch and then stops.

It is useful when you want streaming-style incremental processing without keeping the stream running forever.

Example:

.trigger(availableNow=True)

For DP-750, availableNow is commonly used with:

Auto Loader
file ingestion
incremental batch processing
structured streaming jobs that should process currently available data and then stop

In the exam, if the code uses:

.trigger(availableNow=True)
.outputMode("append")

then it does not mean each batch overwrites the target table. With append mode, new records are appended.

9. Delta table as streaming target

A common Structured Streaming write pattern is:

df.writeStream \
    .format("delta") \
    .option("checkpointLocation", checkpoint_path) \
    .option("mergeSchema", "true") \
    .trigger(availableNow=True) \
    .table(raw_events)

For DP-750, remember:

Requirement	Code option
Write to Delta table	`.format("delta")`
Resume after failure without reprocessing	`.option("checkpointLocation", checkpoint_path)`
Add new columns automatically	`.option("mergeSchema", "true")`
Process available data and stop	`.trigger(availableNow=True)`

10. Change Data Feed, CDF

Change Data Feed, or CDF, lets you track row-level changes from a Delta table.

It is useful when downstream pipelines need to consume:

inserts
updates
deletes

Databricks documentation describes Change Data Feed as a feature used to track row-level changes for Delta Lake and Apache Iceberg v3 tables.

A streaming read from a Delta table’s change feed can look like this:

spark.readStream \
    .format("delta") \
    .option("readChangeFeed", "true") \
    .table("db1.sales_orders")

For DP-750, if the requirement says:

Ingest all changes from a Delta table, including inserts, updates, and deletes.

then the key option is:

readChangeFeed = true

11. `skipChangeCommits`

When streaming from Delta tables, some options control how to handle source table updates and deletes.

One important option is:

.option("skipChangeCommits", "false")

For the DP-750 question in your bank, the correct dropdown combination is:

readChangeFeed = true
skipChangeCommits = false

This means the stream reads the change feed rather than skipping commits that contain changes.

For exam purposes:

Requirement	Option
Read CDF changes	`readChangeFeed`
Do not skip change commits	`skipChangeCommits = false`

12. Checkpointing vs watermarking vs CDF

These options solve different problems.

Feature	What it solves
Checkpointing	Recovery and progress tracking
Watermarking	Late data handling in stateful aggregations
Change Data Feed	Read inserts, updates, and deletes from Delta table
Trigger interval	How often micro-batches run
Schema evolution	Handles new columns or schema changes

For DP-750:

Reprocessing after restart → checkpointing
New columns causing failure → schema evolution
Need inserts, updates, deletes → CDF
Late arriving event-time data → watermarking

13. DP-750 decision table for streaming

Scenario in the question	Best answer pattern
Streaming job reprocesses data after restart	Configure checkpoint location
Job must resume from failure point	Implement checkpointing
Need exactly-once write to Delta	Structured Streaming + Delta + checkpoint
New source columns cause write failure	Enable schema evolution
Auto Loader code uses `failOnNewColumns`	New column fails the stream initially
Code uses `mergeSchema=true`	New columns can be added to target
Code uses `outputMode("append")`	New data is appended, not overwritten
Need to ingest inserts, updates, deletes from Delta	Read Change Data Feed
CDF streaming option	`readChangeFeed = true`
Write stream to Delta	`.format("delta")`
Failure recovery option	`checkpointLocation`
Process available data then stop	`availableNow=True`

Real Exam Questions

Question 8

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You have an Apache Spark Structured Streaming job that writes data to a Delta table.

After the cluster restarts, the streaming job reprocesses previously ingested data.

You need to prevent the streaming job from reprocessing the data after the cluster restarts.

What should you do?

A. Configure a checkpoint location for the streaming query. ✅ Correct Answer

B. Increase the trigger interval of the streaming query.

C. Enable change data feed, CDF, for the target table.

D. Configure a watermark for the streaming query.

Question 43

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You plan to run the following PySpark code.

(
    df.writeStream
        .format("delta")
        .option("checkpointLocation", CHECKPOINT_LOCATION)
        .outputMode("append")
        .option("mergeSchema", "true")
        .trigger(availableNow=True)
        .toTable(BRONZE_TABLE)
)

For each of the following statements, select Yes if the statement is true. Otherwise, select No.

Statement	Answer
New columns from incoming JSON payload will be added automatically to the target Delta table.	Yes ✅ Correct Answer
Previously processed data will not be reprocessed if the pipeline fails.	Yes ✅ Correct Answer
Each new batch of telemetry data will overwrite the existing data in the target Delta table.	No ✅ Correct Answer

Question 47

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named db1.sales_orders.

db1.sales_orders is updated nightly and has change data feed, CDF, enabled.

You need to ingest all the changes from the db1.sales_orders table, including inserts, updates, and deletes, into a downstream pipeline.

How should you complete the PySpark code segment?

spark.readStream \
    .format("delta") \
    .option(______, "true") \
    .option(______, "false") \
    .table("db1.sales_orders")

Area	Answer
First dropdown	`"readChangeFeed"` ✅ Correct Answer
Second dropdown	`"skipChangeCommits"` ✅ Correct Answer

Question 48

You have an Azure Databricks workspace that uses Unity Catalog.

You have a Lakeflow Spark Declarative Pipelines, SDP, pipeline that ingests data into a managed Delta table named Table1. Table1 is used for analytics.

New columns are added to the source data, causing pipeline failures during writes to Table1.

You need to prevent the pipeline failures. The solution must ensure that schema changes are detected and handled.

What should you do?

A. Disable schema enforcement for Table1.

B. Use row filters to exclude records that have new columns.

C. Enable schema evolution. ✅ Correct Answer

D. Create a separate table for each schema version.

Question 54

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You plan to run the following PySpark code.

(
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .option("cloudFiles.schemaLocation", "/cloud/schema")
        .option("cloudFiles.schemaEvolutionMode", "failOnNewColumns")
        .load("/data/source")
        .writeStream
        .option("checkpointLocation", "/data/checkpoint")
        .toTable("table1")
)

For each of the following statements, select Yes if the statement is true. Otherwise, select No.

Statement	Answer
When a new element is found in the source data, the element will be written to table1 successfully.	No ✅ Correct Answer
The streaming process will resume from a failure without data loss.	Yes ✅ Correct Answer
Data will be written to a JSON file.	No ✅ Correct Answer

Question 59

Useful case information

Contoso identifies the following data ingestion and processing requirements:

Handle schema drift for the maintenance and telemetry data.
Store all the ingested data in a format that supports incremental processing.
Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics.
Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events.
The pipelines must support schema drift and resume processing after failures without reprocessing the data.

You need to complete the PySpark code for the Spark Structured Streaming pipelines. The solution must meet the data ingestion and processing requirements.

How should you complete the code segment?

telemetry_path = "abfss://telemetry@contosodata.dfs.core.windows.net/"
checkpoint_path = "abfss://telemetry@contosodata.dfs.core.windows.net/checkpoints/raw_events"
raw_events = "analytics.telemetry.raw_events"

df = (
    spark.readStream
        .format("json")
        .load(telemetry_path)
)

df.writeStream
    .format( ______ )
    .option( ______ )
    .option("mergeSchema", "true")
    .trigger(availableNow=True)
    .table(raw_events)

Area	Answer
First dropdown	`"delta"` ✅ Correct Answer
Second dropdown	`("checkpointLocation", checkpoint_path)` ✅ Correct Answer

Question 71

You have an Azure Databricks workspace.

You have an Apache Spark Structured Streaming job named Job1 that processes data continuously and fails periodically due to transient errors.

You need to ensure that Job1 meets the following requirements:

Resumes processing from the point that Job1 failed
Minimizes how long it takes to restart Job1
Minimizes the costs to restart Job1

What should you do?

A. Decrease the retry interval.

B. Increase the minimum number of nodes in the cluster.

C. Add an alert and manually restart Job1.

D. Implement checkpointing. ✅ Correct Answer

Key takeaways

For DP-750, Structured Streaming questions are usually about recovery, incremental processing, and schema changes.

Remember these patterns:

Checkpointing prevents reprocessing after restart.
checkpointLocation is required for fault tolerance and exactly-once guarantees.
Each streaming query should use a unique checkpoint location.
Delta tables are the normal target for reliable streaming writes.
mergeSchema=true allows new incoming columns to be added.
Schema evolution handles new source columns.
failOnNewColumns causes the stream to fail when new columns appear, so the schema can evolve.
availableNow=True processes available data and then stops.
outputMode("append") appends data; it does not overwrite existing records.
Change Data Feed is used to ingest inserts, updates, and deletes from a Delta table.
For CDF streaming reads, use readChangeFeed=true.

If you can identify whether the issue is failure recovery, schema drift, Delta change capture, or output behavior, you can usually choose the correct DP-750 streaming answer quickly.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Data Ingestion Patterns Explained and with Real Exam Questions

Jin — Mon, 03 Aug 2026 06:59:56 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to know how to choose the right ingestion pattern for different data sources.

This is a common exam topic because Azure Databricks supports several ingestion options:

Auto Loader
COPY INTO
Lakeflow Connect
Apache Spark Structured Streaming
Databricks notebooks
Lakeflow Jobs triggers

The related questions in your DP-750 question bank are Q37, Q44, Q46, and Q56.

1. Why data ingestion patterns matter in DP-750

Data ingestion means loading data from source systems into the lakehouse.

In DP-750, the source system is often one of these:

Source type	Typical ingestion pattern
Files in cloud storage	Auto Loader or COPY INTO
Frequently arriving files	Auto Loader
Simple SQL-based file loading	COPY INTO
SaaS applications or databases	Lakeflow Connect
Azure Event Hubs	Spark Structured Streaming
REST API	Databricks notebook or custom code

The exam usually gives you requirements such as:

minimize custom code
handle schema drift
process new files incrementally
support near-real-time ingestion
ingest from operational databases
read streaming events
preserve existing table data
use Unity Catalog governance

Your job is to match the requirement to the right ingestion tool.

2. Auto Loader

Auto Loader is one of the most important ingestion features for DP-750.

Auto Loader incrementally discovers and processes new files as they arrive in cloud object storage. Databricks documentation describes Auto Loader as a way to incrementally and efficiently process new data files as they arrive in cloud storage. It can load files into Delta tables and supports both Python and SQL.

Auto Loader is commonly used with:

spark.readStream.format("cloudFiles")

Example:

df = (
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/path/to/source")
)

Use Auto Loader when the question says:

new files arrive frequently
data must be available in near-real time
minimize custom code
handle large-scale file ingestion
support schema drift or schema evolution
ingest JSON, CSV, Parquet, or other file-based data

3. Auto Loader and schema drift

Schema drift means the source data structure changes over time.

For example, today’s JSON files contain:

{
  "sensor_id": "S001",
  "temperature": 21.5
}

Tomorrow’s JSON files contain a new field:

{
  "sensor_id": "S001",
  "temperature": 21.5,
  "humidity": 62
}

This is common in IoT telemetry and operational logs.

Auto Loader supports schema inference and schema evolution. Databricks documentation says Auto Loader can automatically detect the schema of loaded data and supports schema evolution options such as cloudFiles.schemaEvolutionMode.

For DP-750:

File-based ingestion + schema drift + minimal operational effort = Auto Loader

4. COPY INTO

COPY INTO is a SQL command for loading files from cloud storage into a Delta table.

Example:

COPY INTO main.sales.orders
FROM 'abfss://raw@storage1.dfs.core.windows.net/orders/'
FILEFORMAT = CSV;

Databricks documentation says COPY INTO can load data incrementally and idempotently from cloud storage into a Delta table. It is often used for incremental or bulk loading from cloud object storage.

Use COPY INTO when the question says:

load files into an existing table
use SQL
load incrementally
minimize processing effort
source is cloud object storage
source files are CSV, JSON, Parquet, Avro, ORC, text, or similar

For DP-750:

SQL-based incremental file loading = COPY INTO

5. COPY INTO vs Auto Loader

Both Auto Loader and COPY INTO can load files from cloud storage, but they are used in slightly different situations.

Requirement	Better choice
Near-real-time file ingestion	Auto Loader
Frequently arriving files	Auto Loader
Large-scale file discovery	Auto Loader
Schema drift handling	Auto Loader
SQL-based incremental loading	COPY INTO
Simple one-command file ingestion	COPY INTO
Thousands of files	COPY INTO can be suitable
Millions of files or advanced ingestion	Auto Loader is usually better

Databricks documentation says COPY INTO works well for data sources that contain thousands of files, while Auto Loader is recommended for loading millions of files and advanced use cases.

For DP-750, if the question says:

New rows are appended frequently and must be available near-real-time.

choose Auto Loader.

If the question says:

Complete the SQL statement to incrementally load CSV files.

choose COPY INTO.

6. Lakeflow Connect

Lakeflow Connect is used to ingest data from systems such as SaaS applications, databases, cloud storage, and message buses.

Databricks documentation describes Lakeflow Connect as a set of connectors for ingesting data from local files, enterprise applications, databases, cloud storage, message buses, and more. Managed connectors are governed by Unity Catalog, powered by serverless compute and Lakeflow Spark Declarative Pipelines, and support efficient incremental reads and writes.

Use Lakeflow Connect when the question says:

ingest from an operational database
ingest from SQL Server
ingest from Salesforce, ServiceNow, Google Analytics, or supported applications
minimize custom ingestion code
use a managed connector
load data into the lakehouse

For DP-750:

Structured operational database ingestion + managed connector = Lakeflow Connect

7. Lakeflow Connect vs foreign catalog

This is an important distinction.

Requirement	Correct choice
Ingest data from external system into Databricks	Lakeflow Connect
Query external database tables without copying data	Foreign catalog
Data should appear in Unity Catalog but remain in source system	Foreign catalog
Data should be loaded into lakehouse tables	Lakeflow Connect

In the previous article, we used foreign catalog for read-only federation.

In this article, Lakeflow Connect is about ingestion.

For DP-750:

If data is copied or ingested into Databricks, think Lakeflow Connect.

If data is not copied and only queried externally, think foreign catalog.

8. Azure Event Hubs ingestion

Azure Event Hubs is used for streaming event data.

In Azure Databricks, Event Hubs data can be read with Spark Structured Streaming. A common pattern is:

events = (
    spark.readStream
        .format("eventhubs")
        .option("eventhubs.connectionString", "<auth_value>")
        .load()
)

Event Hubs event payloads are stored in the body field, so you commonly convert the body to a string:

payload = events.selectExpr("CAST(body AS STRING) AS payload")

Microsoft documentation for Azure Event Hubs with Databricks describes using Structured Streaming to ingest Event Hubs data into raw landing tables. Databricks also documents Event Hubs as a pipeline data source for Lakeflow pipelines.

For DP-750:

Event Hubs + streaming ingestion = Spark Structured Streaming

And for the PySpark code:

Connection option = "eventhubs.connectionString"
Payload expression = "CAST(body AS STRING) AS payload"

9. REST API ingestion

Some data sources do not have a managed connector.

For example:

external weather API
custom business API
third-party REST endpoint
internal HTTP service

In these cases, a Databricks notebook is often the simplest answer.

The notebook can:

call the REST API
parse the JSON response
convert it into a DataFrame
write it to a Delta table
run on a schedule through Lakeflow Jobs

For DP-750, if the source is an external REST API and no managed connector option is available, the answer is often:

A Databricks notebook

This is especially true when the question says the external data is retrieved from a REST API and written to cloud storage or Delta tables.

10. Incremental processing and Delta tables

DP-750 often includes the requirement:

Store all ingested data in a format that supports incremental processing.

The answer is usually Delta Lake.

Delta tables support:

ACID transactions
scalable metadata
time travel
streaming reads and writes
change data feed
incremental processing patterns

For ingestion questions, this means the target should usually be a Delta table, especially a managed Delta table when Unity Catalog is enabled.

11. DP-750 decision table for ingestion patterns

Scenario in the question	Best answer pattern
Ingest new files from cloud storage with minimal effort	Auto Loader
Files arrive frequently and near-real-time is required	Auto Loader
File-based telemetry with schema drift	Auto Loader
Maintenance logs with unstructured files	Auto Loader
Simple SQL command to load CSV files incrementally	COPY INTO
Complete SQL statement: `[?] table [?] path FILEFORMAT = CSV`	`COPY INTO table FROM path`
Operational database ingestion	Lakeflow Connect managed connector
SQL Server ingestion into lakehouse	Lakeflow Connect
External database query without copying data	Foreign catalog
External REST API	Databricks notebook
Azure Event Hubs streaming	Spark Structured Streaming
Event Hubs authentication option	`"eventhubs.connectionString"`
Event Hubs payload column	`"CAST(body AS STRING) AS payload"`

Real Exam Questions

Question 37

Useful case information

Contoso ingests the following operational and business data:

Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.
Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.

Contoso identifies the following data ingestion and processing requirements:

Auto-scale ingestion pipelines to handle bursty workloads.
Handle schema drift for the maintenance and telemetry data.
Ingest file-based telemetry data by using minimal operational effort.
Store all the ingested data in a format that supports incremental processing.
Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server.

Which ingestion option should you recommend for each data source?

Data source	Answer
Telemetry Data	Auto Loader ✅ Correct Answer
Operational maintenance data	Lakeflow Connect managed connector ✅ Correct Answer
Maintenance logs	Auto Loader ✅ Correct Answer
External weather data	A Databricks notebook ✅ Correct Answer

Question 44

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You plan to ingest data from CSV files stored in Azure Data Lake Storage Gen2. New rows are appended frequently.

You need to implement a data ingestion solution that meets the following requirements:

New data must be available in near-real-time, NRT.
The data must be stored in managed Delta tables.
The solution must minimize custom code and maintenance effort.

What should you include in the solution?

A. scheduled Apache Spark batch jobs
B. an Azure Data Factory pipeline
C. Auto Loader ✅ Correct Answer
D. an external table that references the CSV files

Question 46

You have an Azure Databricks workspace.

You need to ingest streaming data from Azure Event Hubs by using Apache Spark Structured Streaming. The solution must authenticate to Event Hubs and read the event payload.

How should you complete the PySpark code segment?

events = (
    spark.readStream
        .format("eventhubs")
        .option(______, "<auth_value>")
        .load()
)

payload = events.selectExpr(______ )

Area	Answer
First dropdown	`"eventhubs.connectionString"` ✅ Correct Answer
Second dropdown	`"CAST(body AS STRING) AS payload"` ✅ Correct Answer

Question 56

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You need to ingest data from the CSV files stored in an external Azure Data Lake Storage Gen2 container into a managed Delta table. The solution must meet the following requirements:

Load data incrementally as new files arrive.
Minimize custom code and operational effort.

How should you complete the SQL statement?

[Dropdown 1] main.sales.orders

[Dropdown 2] 'abfss://raw@storage1.dfs.core.windows.net/orders/'

FILEFORMAT = CSV

Area	Answer
Dropdown 1	COPY INTO ✅ Correct Answer
Dropdown 2	FROM ✅ Correct Answer

Completed statement:

COPY INTO main.sales.orders
FROM 'abfss://raw@storage1.dfs.core.windows.net/orders/'
FILEFORMAT = CSV

Key takeaways

For DP-750, ingestion questions are mostly about matching the source and requirement to the right tool.

Remember these patterns:

Auto Loader is best for incremental cloud file ingestion, especially near-real-time ingestion and schema drift.
COPY INTO is a SQL-based way to load files incrementally into Delta tables.
Lakeflow Connect is used for managed ingestion from databases and enterprise applications.
Foreign catalog is for querying external databases without copying data, not ingestion.
Databricks notebooks are useful for custom ingestion, especially REST APIs.
Azure Event Hubs streaming ingestion uses Spark Structured Streaming.
Event Hubs payload is in the body column.
Use "eventhubs.connectionString" for the Event Hubs connection option.
Use "CAST(body AS STRING) AS payload" to extract the event payload.

If you can identify whether the source is cloud files, operational database, Event Hubs, or REST API, you can usually choose the correct DP-750 ingestion pattern quickly.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: External Data Access Explained and with Real Exam Questions

Jin — Sun, 02 Aug 2026 19:59:56 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand how Azure Databricks accesses data outside managed Delta tables.

This topic includes several important Unity Catalog concepts:

storage credentials
external locations
external volumes
Databricks access connector
managed identity
Key Vault-backed secret scopes
foreign catalogs
COPY INTO

The related questions in your DP-750 question bank are Q16, Q18, Q25, Q26–29, and Q32.

1. Why external data access matters in DP-750

Not all data is stored directly inside Databricks-managed tables.

In real projects, data may live in:

Azure Data Lake Storage Gen2
Azure Blob Storage
SQL Server
Azure SQL Database
PostgreSQL
external file systems
operational databases
third-party systems

Unity Catalog provides governed ways to access this data. For DP-750, the exam often asks you to choose the right access pattern while following least privilege and avoiding unnecessary data movement.

The most important distinction is:

Cloud storage files       -> storage credential + external location / volume
External databases        -> connection + foreign catalog
Secrets in Azure Key Vault -> Key Vault-backed secret scope

2. Storage credential

A storage credential is a Unity Catalog object that represents the identity used to access cloud storage.

For Azure Databricks on Azure, this identity is often based on a managed identity through a Databricks access connector.

Microsoft documentation explains that Unity Catalog can use Azure managed identities to access Azure storage, and that managed identities remove the need to maintain credentials or rotate secrets. To use this pattern, you create an access connector for Azure Databricks, grant its managed identity access to the storage account, and use the access connector when creating a Unity Catalog storage credential.

For DP-750, remember:

Access connector + managed identity = preferred governed access to Azure storage without storing secrets in Databricks.

3. Databricks access connector

A Databricks access connector is an Azure resource that provides a managed identity for Azure Databricks.

It is commonly used with Unity Catalog to access Azure Data Lake Storage Gen2.

The common setup flow is:

1. Create a Databricks access connector.
2. Assign the access connector’s managed identity permissions on the storage account.
3. Register the access connector as a Unity Catalog storage credential.
4. Use the storage credential in an external location.

For least privilege, you should assign only the required Azure role to the access connector. In many Unity Catalog external-location scenarios, this is Storage Blob Data Contributor on the relevant storage scope, not a broad subscription-level role. Microsoft’s managed identity guidance describes creating the access connector, granting the managed identity access to the storage account, and then using that access connector for the storage credential.

For DP-750, the access connector is used for storage access. It is not used to expose SQL Server tables as Unity Catalog objects.

4. External location

An external location is a Unity Catalog securable object that combines:

cloud storage path + storage credential

Databricks documentation defines an external location as a securable object that combines a storage path with a storage credential that authorizes access to that path.

Example:

CREATE EXTERNAL LOCATION raw_orders
URL 'abfss://raw@storage1.dfs.core.windows.net/orders/'
WITH (STORAGE CREDENTIAL my_storage_credential);

External locations are used when Databricks needs governed access to files in cloud storage.

For DP-750, if a question says:

A SQL statement such as COPY INTO must access files in an ADLS Gen2 container.

the expected answer is often:

Configure an external location that uses a storage credential.

5. External location vs external table vs volume

These three concepts are related but different.

Concept	Meaning
Storage credential	Identity used to access cloud storage
External location	Storage path plus storage credential
External table	Table whose data lives in an external storage path
Volume	Unity Catalog object for governing file access

An external table is still a table. It is useful when you want to query structured data as a table.

A volume is used for file access, including structured, semi-structured, and unstructured files. Databricks documentation describes volumes as Unity Catalog objects that represent logical storage in cloud object storage and provide governance over non-tabular datasets.

For DP-750, if the question is about users accessing files directly, think about volumes and READ VOLUME / WRITE VOLUME.

6. External volume

A Unity Catalog volume is used to govern access to files.

Volumes can be:

managed volumes
external volumes

An external volume points to a storage location outside the Unity Catalog managed storage location. Databricks documentation says external volumes must be registered against a directory within an external location, and volumes can only be created in Unity Catalog schemas.

Example:

CREATE EXTERNAL VOLUME main.raw.files
LOCATION 'abfss://raw@storage1.dfs.core.windows.net/files/';

For DP-750, if the question says:

expose files from Azure Storage
authentication should not require storing credentials in Databricks
users can access files but not modify files
follow least privilege

then the answer pattern is:

Authentication type: Databricks access connector
Permission: READ VOLUME

READ VOLUME allows read access to files in the volume. WRITE VOLUME would be too broad if users must not modify files.

7. Key Vault-backed secret scope

A secret scope allows notebooks and jobs to retrieve secrets securely at runtime.

Azure Databricks supports Azure Key Vault-backed secret scopes. Microsoft documentation explains that a Key Vault-backed scope references secrets stored in Azure Key Vault, and secrets must be managed in Azure Key Vault.

The notebook retrieves the value with:

dbutils.secrets.get(scope="<scope-name>", key="<secret-name>")

Microsoft’s secret workflow documentation says to use the dbutils.secrets utility to access secrets in an Azure Databricks notebook.

For DP-750, if the question says:

the secret value is managed by a security team in Azure Key Vault
Databricks must always retrieve the latest value
least privilege is required
notebook must retrieve the secret at runtime

then the answer is:

Expose the secret with a Key Vault-backed secret scope.
Retrieve it with dbutils.secrets.get.

Do not use:

spark.conf.get
dbutils.secret.list
Databricks-backed secret scope

because those do not meet the requirement as directly.

8. Foreign catalog

A foreign catalog is used for Lakehouse Federation.

It mirrors an external database in Unity Catalog so that users can query external tables from Databricks without copying the data into Databricks-managed storage.

Databricks documentation says a foreign catalog mirrors a database in an external data system and enables read-only queries on that data system from an Azure Databricks workspace.

For DP-750, this is very important:

External database schemas and tables should appear alongside Unity Catalog objects, but data must not be copied into Databricks-managed storage = foreign catalog.

This is different from Lakeflow Connect.

9. Foreign catalog vs Lakeflow Connect

Lakehouse Federation and Lakeflow Connect solve different problems.

Requirement	Best fit
Query external database tables without copying data	Foreign catalog
Ingest data from an external system into the lakehouse	Lakeflow Connect
Mirror external database metadata into Unity Catalog	Foreign catalog
Build a managed ingestion pipeline	Lakeflow Connect

Databricks documentation describes a Unity Catalog connection as a securable object that stores the path and credentials for accessing an external database system, and CREATE FOREIGN CATALOG allows creating a read-only mirror of a database from that external data source.

So if the requirement says:

The data is NOT copied into Databricks-managed storage.

then a Lakeflow Connect pipeline is not the best answer, because Lakeflow Connect is an ingestion approach. You should choose a foreign catalog.

10. COPY INTO and external locations

COPY INTO is used to load files from cloud object storage into a table.

For Unity Catalog, Databricks documentation describes using COPY INTO to load data from an Azure Data Lake Storage container into a table in Databricks SQL, and recommends using Unity Catalog volumes to access files as part of ingestion.

A typical SQL pattern is:

COPY INTO main.sales.orders
FROM 'abfss://raw@storage1.dfs.core.windows.net/orders/'
FILEFORMAT = CSV;

If this statement needs to access the external ADLS Gen2 path, Databricks must have governed access to the path.

For DP-750, the answer is usually:

external location + storage credential

not:

external table
Lakeflow ingestion gateway
volume only

The key point is that the path must be authorized through Unity Catalog.

11. DP-750 decision table for external data access

Scenario in the question	Best answer pattern
Access ADLS Gen2 by managed identity	Databricks access connector
Avoid storing storage credentials in Databricks	Managed identity / access connector
Register cloud storage access in Unity Catalog	Storage credential
Authorize a cloud storage path	External location
`COPY INTO` must access files in ADLS Gen2	External location that uses a storage credential
Expose files from Azure Storage as a governed object	External volume
Users can read files but not modify them	`READ VOLUME`
Notebook retrieves Key Vault secret at runtime	Key Vault-backed secret scope + `dbutils.secrets.get`
Query external SQL Server schemas/tables without copying data	Foreign catalog
Ingest external database data into Databricks	Lakeflow Connect
Access connector proposed for SQL Server metadata federation	No

Real Exam Questions

Question 16

You have an Azure Databricks workspace named Workspace1 that is attached to a Unity Catalog metastore named metastore1.

You need to register an Azure Storage account named account1 that has a hierarchical namespace enabled as an external location.

The external location must use a managed identity to authenticate to account1 and the solution must follow the principle of least privilege.

Which three actions should you perform in sequence?

Actions:

Assign the Storage Blob Data Owner role to the access connector.
Assign Workspace1 the Storage Blob Data Contributor role for account1.
Register the access connector as a storage credential in metastore1.
Create a Databricks access connector.
Assign the Storage Blob Data Contributor role to the access connector.
Register the access connector as a storage credential in Workspace1.

Order	Answer
Step 1	4. Create a Databricks access connector ✅ Correct Answer
Step 2	5. Assign the Storage Blob Data Contributor role to the access connector ✅ Correct Answer
Step 3	3. Register the access connector as a storage credential in metastore1 ✅ Correct Answer

Question 18

You have an Azure Databricks workspace.

You have an Azure key vault named kv-secure that stores a secret named storageKey. The value of storageKey is managed and updated by the cloud security team at your company.

You need to enable a Databricks notebook named Notebook1 to retrieve the value of storageKey securely at runtime. The solution must follow the principle of least privilege and always retrieve the latest value.

What should you do?

Area	Answer
Expose storageKey to Databricks by using	A Key Vault-backed secret scope ✅ Correct Answer
Retrieve storageKey from within Notebook1 by using	dbutils.secret.get ✅ Correct Answer

Question 25

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You need to create an external volume named Volume1 in an existing schema. Volume1 must expose files from an Azure Storage container.

The solution must meet the following requirements:

Ensure that authentication does NOT require storing credentials in Databricks.
Ensure that users can access the files, but NOT modify the files.
Follow the principle of least privilege.

Which type of authentication should you configure, and which permission should you grant to the users?

Area	Answer
Authentication type	A Databricks access connector ✅ Correct Answer
Permission	READ VOLUME ✅ Correct Answer

Question 26

You have an Azure Databricks workspace named Workspace1 that contains a lakehouse and is enabled for Unity Catalog.

You have a connection to a Microsoft SQL Server database named DB1.

You need to expose the schemas and tables of DB1 to meet the following requirements:

The schemas and tables can be queried in Databricks.
The schemas and tables appear alongside other Unity Catalog objects.
The data is NOT copied into Databricks-managed storage.

Solution: You create a Lakeflow Connect pipeline and connect it to DB1.

Does this meet the goal?

A. Yes

B. No ✅ Correct Answer

Question 27

You have an Azure Databricks workspace named Workspace1 that contains a lakehouse and is enabled for Unity Catalog.

You have a connection to a Microsoft SQL Server database named DB1.

You need to expose the schemas and tables of DB1 to meet the following requirements:

The schemas and tables can be queried in Databricks.
The schemas and tables appear alongside other Unity Catalog objects.
The data is NOT copied into Databricks-managed storage.

Solution: You create a Databricks access connector.

Does this meet the goal?

A. Yes

B. No ✅ Correct Answer

Question 28

You have an Azure Databricks workspace named Workspace1 that contains a lakehouse and is enabled for Unity Catalog.

You have a connection to a Microsoft SQL Server database named DB1.

You need to expose the schemas and tables of DB1 to meet the following requirements:

The schemas and tables can be queried in Databricks.
The schemas and tables appear alongside other Unity Catalog objects.
The data is NOT copied into Databricks-managed storage.

Solution: You create a foreign catalog in Catalog Explorer.

Does this meet the goal?

A. Yes ✅ Correct Answer

B. No

Question 29

You have an Azure Databricks workspace named Workspace1 that contains a lakehouse and is enabled for Unity Catalog.

You have a connection to a Microsoft SQL Server database named DB1.

You need to expose the schemas and tables of DB1 to meet the following requirements:

The schemas and tables can be queried in Databricks.
The schemas and tables appear alongside other Unity Catalog objects.
The data is NOT copied into Databricks-managed storage.

Solution: You create a new native catalog in Unity Catalog.

Does this meet the goal?

A. Yes

B. No ✅ Correct Answer

Question 32

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You have a CSV file stored in an Azure Data Lake Storage Gen2 container.

You plan to ingest the data into an existing table by running the following SQL statement.

COPY INTO Customer
FROM 'abfss://container1@storageaccount.dfs.core.windows.net/data/customer'
FILEFORMAT = CSV

You need to ensure that the statement can access the data in the container.

What should you configure?

A. an external table named Customer in the CSV file format

B. a Lakeflow ingestion gateway

C. an external location that uses a storage credential ✅ Correct Answer

D. a volume in a catalog

Key takeaways

For DP-750, external data access questions usually test whether you can choose the correct Unity Catalog object for the access pattern.

Remember these patterns:

Storage credential stores or represents the identity used to access cloud storage.
Databricks access connector provides managed identity-based access to Azure storage.
External location combines a storage path with a storage credential.
External volume governs file access in Unity Catalog.
READ VOLUME allows users to read files without modifying them.
Key Vault-backed secret scope lets Databricks reference secrets stored in Azure Key Vault.
dbutils.secrets.get retrieves secrets securely in notebooks.
Foreign catalog exposes external database schemas and tables without copying data.
Lakeflow Connect is for ingestion, not read-only federation.
COPY INTO needs governed access to the source path, usually through an external location and storage credential.

If you can identify whether the source is cloud storage, external files, Key Vault secrets, or an external database, you can usually choose the correct DP-750 external access answer quickly.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

# DP-750: Unity Catalog Governance Explained and with Real Exam Questions

Jin — Thu, 30 Jul 2026 05:01:32 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, Unity Catalog governance is one of the most important areas to understand.

In previous articles, we covered:

Unity Catalog object model: catalog, schema, table
Unity Catalog permissions: USE CATALOG, USE SCHEMA, SELECT, MODIFY

This article focuses on higher-level governance features:

AI/BI Genie Spaces
Unity Catalog data lineage
attribute-based access control, ABAC
governed tags
column masks
audit logs

The related questions in your DP-750 question bank are Q14, Q17, Q19, Q23, and Q35.

1. What does governance mean in Unity Catalog?

In Azure Databricks, governance is not only about granting table permissions.

Governance also includes:

controlling access to sensitive data
masking columns
tracking data lineage
auditing data access
organizing metadata consistently
helping users find and understand governed data
applying policies centrally across many objects

Unity Catalog is the central governance layer for data and AI assets in Azure Databricks. It provides a unified model for permissions, lineage, discovery, and policy enforcement across Databricks workspaces.

For DP-750, you need to understand which governance feature solves which problem.

Requirement	Governance feature
Users ask natural-language questions about data	AI/BI Genie Space
Business terms should map to the right table	Genie instructions / knowledge store
Track where table data came from	Data lineage
Review lineage visually	Catalog Explorer
Tag data and apply dynamic policies	Governed tags + ABAC
Hide part of a sensitive column	Column mask
Capture data access events	Audit logs
Analyze logs quickly in Azure	Log Analytics

2. AI/BI Genie Space

An AI/BI Genie Space, now also referred to in current documentation as a Genie Agent, allows business users to interact with data by using natural language. Domain experts configure Genie with datasets, sample queries, and text guidelines so that business questions can be translated into analytical queries.

For example, users might ask:

Show me total sales by month.

or:

How many invoices were created last quarter?

Genie then tries to generate a SQL query based on the configured data, metadata, descriptions, and instructions.

For DP-750, the key idea is:

Genie is scoped to a Genie Space.

If a question says:

The instruction should affect only Space1 and no other Genie spaces.

then the instruction belongs on Space1, not on the table, schema, catalog, or notebook.

3. Genie instructions and business synonyms

A common governance problem is that business users use different terms for the same concept.

For example:

sale
transaction
event
order
invoice

Technically, these words might not all be the same. But in a specific business domain, users may use them to refer to the same sales table.

Genie quality can be improved by adding examples, instructions, and knowledge store snippets. Databricks documentation says you can add example SQL queries, Unity Catalog functions, plain-text instructions, and knowledge store snippets such as table descriptions, join relationships, and SQL expressions for business semantics.

So if users in Space1 use the words:

transaction
event
order
invoice

to refer to a sale, then you should add the instruction to Space1.

Do not add the instruction to:

the catalog
the schema
the notebook
all Genie spaces globally

because the requirement says other Genie spaces must remain unaffected.

4. Unity Catalog data lineage

Data lineage tracks how data flows through tables, notebooks, jobs, dashboards, and queries.

For example, if a pipeline reads from:

bronze.raw_events

then writes to:

silver.cleaned_events

then writes to:

gold.telemetry_summary

lineage helps you understand that relationship.

Unity Catalog data lineage can be used for impact analysis, root-cause investigation, and tracking sensitive data flows. Databricks documentation says lineage is captured for queries that use Spark DataFrame or Databricks SQL interfaces such as notebooks and SQL query editor, and that tables must be registered in a Unity Catalog metastore.

For DP-750, the key exam pattern is:

To capture lineage, use Unity Catalog-enabled compute.

To review lineage, use Catalog Explorer.

5. Why Catalog Explorer for lineage?

Catalog Explorer is the main UI for browsing Unity Catalog objects such as catalogs, schemas, tables, views, and lineage.

If a DP-750 question asks:

Where should you review data lineage?

the answer is usually:

Catalog Explorer

not:

Log Analytics
Azure Data Explorer
DAG

A DAG may show job task dependencies, but it is not the Unity Catalog lineage review tool.

Log Analytics is useful for logs, not for browsing table lineage.

6. Attribute-based access control, ABAC

Attribute-based access control, or ABAC, controls access based on attributes.

In Unity Catalog, ABAC uses:

governed tags
access policies
user-defined functions, UDFs

Databricks documentation says ABAC in Unity Catalog uses governed tags, policies, and UDFs to enforce dynamic, attribute-based access control.

A simple example:

Table1 is tagged as sensitivity = confidential.
Only users in an approved group can see confidential data.

Instead of manually creating separate views for every access pattern, you can use governed tags and Unity Catalog access policies to apply consistent rules.

For DP-750, the key pattern is:

ABAC = governed tag + Unity Catalog access policy

7. Governed tags

A governed tag is an account-level tag with rules.

Databricks documentation describes governed tags as account-level tags with built-in rules for consistency and control. A tag policy defines how the tag can be used, including allowed values and who can assign them.

Governed tags can be used to classify objects, for example:

sensitivity = public
sensitivity = internal
sensitivity = confidential
sensitivity = restricted

or:

data_domain = finance
data_domain = sales
data_domain = hr

For DP-750, if the question says:

Access must be controlled by using ABAC.

then the correct object-level input is usually:

A governed tag

not:

A table privilege
A workspace permission

A table privilege such as SELECT controls direct access, but it is not ABAC. A workspace permission controls workspace objects, not dynamic data access rules.

8. Unity Catalog access policy

A Unity Catalog access policy defines how access should be enforced based on attributes such as governed tags.

Databricks documentation says ABAC policies can be used for row filtering and column masking and can be created, edited, viewed, and deleted in Catalog Explorer, SQL, or REST APIs.

For DP-750, the pattern is simple:

Apply governed tag to table.
Create Unity Catalog access policy.

This is different from manually granting users SELECT.

SELECT allows users to query data, but it does not define an ABAC rule.

9. Column masks

A column mask changes what users see when they query a sensitive column.

For example, a table contains:

email_address
credit_card_number

The business requirement says:

analysts can query all rows
analysts cannot see the full email address
analysts can see only the domain after @
analysts can see only the last four digits of the credit card number

This is a column-level data protection problem.

The right solution is:

Grant SELECT on the table.
Apply column masks to sensitive columns.

Databricks documentation describes row filters and column masks as Unity Catalog access controls that restrict the rows and column values a user can see at query time. It also says column masks are used for redacting sensitive column data based on identity.

For DP-750:

Requirement	Correct feature
Hide rows	Row filter
Hide or transform values in a column	Column mask
Hide full email but keep domain	Column mask
Hide full card number but keep last four digits	Column mask
Create a separate read-only object	Dynamic view
Encrypt values so users cannot query normally	Not the best answer here

10. Column masks vs row filters

Column masks and row filters are often tested together.

Feature	What it controls	Example
Row filter	Which rows users can see	Analyst sees only rows for their region
Column mask	What values users see in a column	Analyst sees only last four digits of card number

If the requirement says:

Business analysts can query all the rows.

then do not choose row-level filters.

The problem is not which rows they can access. The problem is which column values they can see.

So the correct answer is column masks.

11. Dynamic views vs column masks

A dynamic view can also be used to apply filtering or masking logic, but it creates a separate view object.

Databricks documentation explains that dynamic views, row filters, and column masks can all apply filtering or transformation logic at query time, but row filters and column masks apply directly to tables and keep the table name unchanged.

For DP-750, if the question says:

analysts must query the table without errors
analysts should see all rows
only certain column values should be hidden
follow least privilege

then the best answer is usually:

Grant SELECT on the table and apply column masks.

not:

Create a dynamic view.

12. Audit logs

Audit logs capture important security and access events.

In Azure Databricks, audit logs help answer questions such as:

Who accessed this table?
Who changed permissions?
Which workspace generated this event?
Which user ran this command?
Which service principal performed this operation?

Databricks documentation provides an audit log reference for services and events, and notes that audit log availability depends on how you access logs.

For DP-750, if the question says:

Unity Catalog table data access events do not appear in the logs.

then you should think about audit or diagnostic logging for the Azure Databricks workspace.

13. Diagnostic log delivery and Log Analytics

Azure Databricks diagnostic log delivery is configured for workspaces through Azure diagnostic settings. Databricks documentation says to open the Azure Databricks service resource in the Azure portal, go to Diagnostic settings, turn on diagnostics, choose the services to log, and save the configuration.

Azure Monitor diagnostic settings can send resource logs to several destinations, including a Log Analytics workspace. Microsoft documentation describes Log Analytics as a destination where you can retrieve data by using log queries and workbooks and use log alerts proactively.

For DP-750, if the requirement says:

The log data must be available for analysis as quickly as possible.

then Log Analytics is usually the best destination.

Storage account is useful for archiving. Event Hubs is useful for streaming to external systems. Azure Monitor metrics is not the same as detailed audit logs.

14. DP-750 decision table for governance

Scenario in the question	Best answer pattern
Business users ask natural-language questions	Genie Space
Business synonyms should affect only one Genie space	Add instructions to that Genie Space
Capture table lineage for notebooks and jobs	Unity Catalog-enabled compute
Review lineage visually	Catalog Explorer
Control access with ABAC	Governed tag + Unity Catalog access policy
Mask email and credit card columns	Column masks
Analysts must query all rows	Do not use row filters
Data access events missing from logs	Enable workspace audit / diagnostic logging
Logs must be quickly analyzable	Send to Log Analytics

Real Exam Questions

Question 14

You have an Azure Databricks workspace that contains the objects shown in the following table.

Name	Type
Catalog 1	Catalog
Schema 1	Schema
Sales1	Table
Notebook1	Notebook
Space1	AI/BI Genie Space

Users often use the following words to refer to a sale: transaction, event, order, and invoice.

You need to create a knowledge store. The solution must ensure that when the users use any of the words in Space1, Genie queries the Sales1 table. Any other Genie spaces must remain unaffected.

To which object should you add the instructions?

A. Sales1
B. Schema1
C. Space1 ✅ Correct Answer
D. Catalog1
E. Notebook1

Question 17

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You need to ensure that data lineage is captured and can be reviewed for tables accessed by Databricks notebooks and jobs. The solution must minimize administrative effort.

Which compute configuration should you use to capture the data lineage and what should you use to review the data lineage?

Area	Answer
Capture	A Unity Catalog-enabled cluster ✅ Correct Answer
Review	Catalog Explorer ✅ Correct Answer

Question 19

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named Catalog1. Catalog1 contains a schema named Schema1 and a table named Table1.

You need to ensure that access to the data in Table1 is controlled by using attribute-based access control, ABAC.

What should you apply to Table1, and how should you control access for users?

Area	Answer
Apply to Table1	A governed tag ✅ Correct Answer
To control user access	Create a Unity Catalog access policy ✅ Correct Answer

Question 23

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named Catalog1.

Catalog1 contains a table named Transactions.

Transactions contains the following columns:

transaction_id
customer_name
email_address
credit_card_number
transaction_amount

You need to ensure that business analysts can query all the rows in the Transactions table. The solution must meet the following requirements:

Prevent the analysts from seeing the full values in the email_address and credit_card_number columns.
Ensure that the analysts can see only the values after the @ character in each email address.
Ensure that the analysts can see only the last four digits of each credit card number.
Enable the analysts to query the table without errors.
Follow the principle of least privilege.

What should you do?

A. Grant the analysts the SELECT permission for the Transactions table and apply column-level encryption.
B. Grant the analysts the SELECT permission for columns that do NOT contain sensitive data.
C. Grant the analysts the SELECT permission for the Transactions table and implement row-level filters.
D. Grant the analysts the SELECT permission for the Transactions table and apply column masks to email_address and credit_card_number. ✅ Correct Answer

Question 35

You have an Azure Databricks account that contains a single workspace named Workspace1. Workspace1 is enabled for Unity Catalog.

You discover that data access events for Unity Catalog tables fail to appear in the logs.

You need to ensure that all the data access events are captured centrally for auditing purposes. The log data must be available for analysis as quickly as possible.

What should you do?

Area	Answer
Enable audit logging for	Workspace1 ✅ Correct Answer
Send audit logs to	Log Analytics ✅ Correct Answer

Key takeaways

For DP-750, Unity Catalog governance questions usually ask you to choose the correct governance feature, not just the correct SQL privilege.

Remember these patterns:

Genie Space is used for natural-language business questions.
If synonyms or instructions must affect only one Genie Space, add them to that Genie Space.
Unity Catalog lineage can capture lineage for tables used by notebooks and jobs.
Catalog Explorer is used to review lineage.
ABAC uses governed tags and Unity Catalog access policies.
Governed tags classify objects for policy-based access control.
Column masks hide or transform sensitive column values.
Row filters restrict rows, not column values.
Audit logs capture access and security events.
Log Analytics is the best option when logs must be available quickly for analysis.

If you can identify whether the requirement is about natural-language semantics, lineage, tag-based policy, column-level masking, or logging, you can usually choose the right DP-750 governance answer quickly.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Unity Catalog Permissions and Least Privilege Access Explained and with Real Exam Questions

Jin — Wed, 29 Jul 2026 20:00:50 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you must understand how permissions work in Azure Databricks.

Many DP-750 questions are not asking only:

Which permission allows a user to read a table?

They are really testing whether you understand least privilege access across different layers:

catalog
  └── schema
        └── table

In Unity Catalog, a user usually needs permissions on multiple objects before they can actually access data. For example, to read a table, the user needs access to the table itself and usage privileges on the parent schema and catalog.

1. What is least privilege?

Least privilege means giving a user, group, or service principal only the permissions required to complete a task, and nothing more.

For DP-750, this means you should avoid answers such as:

assign workspace admin
grant MANAGE when only SELECT is needed
grant permissions at a broad catalog level when schema-level access is enough
grant CREATE TABLE when the user only needs to update existing data
grant ownership when the user only needs to query or modify data

A good least-privilege answer usually gives permissions at the lowest reasonable level.

For example:

Requirement	Least privilege pattern
Read one table	`USE CATALOG` + `USE SCHEMA` + `SELECT` on the table
Update one table	`USE CATALOG` + `USE SCHEMA` + `SELECT` + `MODIFY` on the table
Create tables in one schema	`USE CATALOG` + `USE SCHEMA` + `CREATE TABLE` on the schema
Run one job	`CAN RUN` on the job
Create unrestricted clusters	Allow unrestricted cluster creation entitlement

2. Unity Catalog permission model

Unity Catalog uses a hierarchical permission model. Data and metadata live in a metastore, and data objects are organized in a three-level namespace: catalog.schema.table. Every object in this hierarchy is a securable object, and access is controlled by privileges such as SELECT, MODIFY, and USE SCHEMA.

The main hierarchy is:

metastore
  └── catalog
        └── schema
              └── table / view / volume / function / model

The important exam point is:

Permissions on a child object are not enough if the user does not also have usage permission on the parent objects.

For example, even if a user has SELECT on a table, the user still needs:

USE CATALOG on the parent catalog
USE SCHEMA on the parent schema
SELECT on the table

Databricks documentation explicitly says that to read from a table, a user needs SELECT on the table, USE CATALOG on the parent catalog, and USE SCHEMA on the parent schema.

3. Usage privileges are not data privileges

This is one of the most important DP-750 exam traps.

USE CATALOG and USE SCHEMA are usage privileges. They allow a user to interact with objects inside the catalog or schema, but they do not grant access to data by themselves.

For example:

GRANT USE CATALOG ON CATALOG sales TO `analysts`;
GRANT USE SCHEMA ON SCHEMA sales.customers TO `analysts`;

This does not allow analysts to query tables yet.

They still need:

GRANT SELECT ON TABLE sales.customers.customer_details TO `analysts`;

So the full read pattern is:

GRANT USE CATALOG ON CATALOG sales TO `analysts`;
GRANT USE SCHEMA ON SCHEMA sales.customers TO `analysts`;
GRANT SELECT ON TABLE sales.customers.customer_details TO `analysts`;

For DP-750, remember:

USE CATALOG + USE SCHEMA = can reach the object

SELECT = can read the data

Databricks documentation says USE CATALOG does not grant access to the catalog itself or to any specific objects within it, and USE SCHEMA does not grant access to the schema itself or to any specific objects within it.

4. Reading data: `SELECT`

The SELECT privilege allows a user to select from a table, view, or materialized view. However, the user must also have USE CATALOG on the parent catalog and USE SCHEMA on the parent schema.

For a table named:

catalog1.schema1.table1

the minimum read permissions are:

GRANT USE CATALOG ON CATALOG catalog1 TO `group1`;
GRANT USE SCHEMA ON SCHEMA catalog1.schema1 TO `group1`;
GRANT SELECT ON TABLE catalog1.schema1.table1 TO `group1`;

In DP-750, when the question says:

The user or service principal must query data.

you should look for:

USE CATALOG + USE SCHEMA + SELECT

5. Modifying data: `MODIFY`

The MODIFY privilege allows a user to insert, update, and delete data in a table. Databricks documentation says that when MODIFY is applied to a table, the user must also have SELECT on the table, USE SCHEMA on the parent schema, and USE CATALOG on the parent catalog.

For example:

GRANT USE CATALOG ON CATALOG sales TO `data_engineers`;
GRANT USE SCHEMA ON SCHEMA sales.customers TO `data_engineers`;
GRANT SELECT ON TABLE sales.customers.customer_details TO `data_engineers`;
GRANT MODIFY ON TABLE sales.customers.customer_details TO `data_engineers`;

For DP-750, when the question says:

User1 can update the data in a table but cannot create new tables.

the answer should include:

USE CATALOG on catalog
USE SCHEMA on schema
MODIFY on table

Do not choose CREATE TABLE, because that would allow creating new tables.

Do not choose MANAGE, because that is broader than data modification and can involve managing privileges or ownership-like operations.

6. Creating tables: `CREATE TABLE`

The CREATE TABLE privilege allows a user to create a table or view in a schema. Databricks recommends granting CREATE TABLE at the schema level for least privilege. The user must also have USE CATALOG on the parent catalog and USE SCHEMA on the parent schema.

For example:

GRANT USE CATALOG ON CATALOG catalog1 TO `group1`;
GRANT USE SCHEMA ON SCHEMA catalog1.schema1 TO `group1`;
GRANT CREATE TABLE ON SCHEMA catalog1.schema1 TO `group1`;

In DP-750, when the question says:

group1 can create tables in schema1

group1 must not be able to grant permissions

you should choose:

USE SCHEMA + CREATE TABLE

not:

MANAGE

because MANAGE would be too broad.

7. `MANAGE` vs data permissions

The MANAGE privilege allows a user to manage privileges on an object, transfer ownership, and delete an object without being the owner. Databricks documentation also says users with MANAGE are not automatically granted all privileges, but they can explicitly grant themselves privileges.

For DP-750, this matters because:

MANAGE is not the same as SELECT.
MANAGE is not the same as MODIFY.
MANAGE is broader than normal data access.

If the requirement says:

Cannot grant permissions for the schema and its objects.

then do not choose MANAGE.

If the requirement says:

Can update table data.

then choose MODIFY, not MANAGE.

8. Privilege inheritance

Unity Catalog supports privilege inheritance. When a privilege is granted on a parent object, it applies to current and future child objects. For example, granting a privilege on a schema can apply to all current and future tables in that schema.

Example:

GRANT SELECT ON SCHEMA sales.customers TO `analysts`;

This can grant SELECT on all current and future tables and views in that schema.

This is useful when a group needs access to many tables in the same schema.

But for least privilege, be careful:

Grant level	Access scope
Table level	Narrowest
Schema level	All current and future objects in one schema
Catalog level	All current and future objects in one catalog

For DP-750, if the question says all data in Catalog1 and Catalog2, granting at the catalog level might be reasonable. If the question says one table only, grant table-level access.

9. Service principals and SQL warehouses

A service principal is often used by external applications, automation, or jobs.

If an external application connects to a SQL warehouse and needs to query Unity Catalog data, the service principal needs the same Unity Catalog data privileges as a normal user.

For example, if service principal SP1 must query tables in Catalog1 and Catalog2, it needs:

USE CATALOG
USE SCHEMA
SELECT

For DP-750, remember:

Authentication is not enough.

The service principal also needs Unity Catalog privileges.

10. Job permissions vs Run as permissions

Lakeflow Jobs use two permission concepts:

Job privileges: who can view, run, or manage the job.
Run as privileges: what identity the job uses to access data and resources during execution.

Databricks documentation says job privileges are evaluated when a user performs an action on the job, such as editing or running it, while Run as privileges are evaluated during the job run.

For DP-750, this means:

Requirement	Permission type
User can run a job	Job permission, such as `CAN RUN`
Job can read/write data	Run as user or service principal needs data privileges
User can edit job settings	Higher job permission, such as manage permission
User only needs to trigger a job	`CAN RUN` is usually enough

In exam questions, if the requirement says:

User1 can run Databricks jobs.

the least-privilege answer is:

Grant CAN RUN permissions for the jobs.

Do not assign workspace admin.

11. Workspace entitlements vs Unity Catalog privileges

Not every permission in Databricks is a Unity Catalog privilege.

Some permissions are workspace entitlements.

For example, Databricks documentation says the Allow unrestricted cluster creation entitlement gives users or service principals permission to create unrestricted clusters. Non-admin users are not granted this entitlement unless explicitly assigned.

This is different from Unity Catalog data access.

Requirement	Permission category
Query table data	Unity Catalog privileges
Update table data	Unity Catalog privileges
Create tables in a schema	Unity Catalog privileges
Run a Databricks job	Job permission
Create unrestricted clusters	Workspace entitlement

For DP-750, do not confuse:

USE CATALOG / USE SCHEMA / SELECT / MODIFY

with:

CAN RUN
Allow unrestricted cluster creation
Workspace admin

They solve different problems.

12. Basic GRANT syntax

Unity Catalog privileges are usually granted with SQL.

Common examples:

GRANT USE CATALOG ON CATALOG catalog1 TO `group1`;

GRANT USE SCHEMA ON SCHEMA catalog1.schema1 TO `group1`;

GRANT SELECT ON TABLE catalog1.schema1.table1 TO `group1`;

GRANT MODIFY ON TABLE catalog1.schema1.table1 TO `group1`;

GRANT CREATE TABLE ON SCHEMA catalog1.schema1 TO `group1`;

In some older exam-style options, you may see USAGE. In current Databricks documentation, the Unity Catalog terminology is generally USE CATALOG and USE SCHEMA.

13. DP-750 decision table for permissions

Scenario in the question	Best answer pattern
Query table data	`USE CATALOG` + `USE SCHEMA` + `SELECT`
Query many tables in a schema	`USE CATALOG` + `USE SCHEMA` + `SELECT` on schema
Query many tables in a catalog	`USE CATALOG` + `USE SCHEMA` + `SELECT` at catalog/schema scope
Update table data	`USE CATALOG` + `USE SCHEMA` + `MODIFY`
Create tables in one schema	`USE CATALOG` + `USE SCHEMA` + `CREATE TABLE`
Cannot grant permissions	Do not grant `MANAGE` or ownership
Run Databricks jobs	`CAN RUN` on the job
Provision clusters of any size	Allow unrestricted cluster creation entitlement
Least privilege	Avoid workspace admin, owner, broad catalog grants if narrower grants work

Real Exam Questions

Question 22

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains two catalogs named Catalog1 and Catalog2.

An external application uses a service principal named SP1 to connect to a SQL warehouse.

You need to ensure that SP1 can query the data in Catalog1 and Catalog2. The solution must follow the principle of least privilege.

Which permissions should you grant to SP1 for the catalogs?

A. USE SCHEMA and SELECT
B. USE CATALOG and SELECT
C. USE CATALOG, USE SCHEMA, and SELECT ✅ Correct Answer
D. USE CATALOG and USE SCHEMA

Question 24

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named catalog1.

You have a group named group1.

You plan to create a schema named schema1 in catalog1.

You need to ensure that group1 meets the following requirements:

Can create tables in schema1
Can modify and query tables
Cannot grant permissions for the schema and its objects

How should you complete the SQL statements?

CREATE SCHEMA catalog1.schema1;

GRANT {} ON SCHEMA catalog1.schema1 {};

Area	Answer
Privileges	USAGE, CREATE TABLE ✅ Correct Answer
Principal	TO ROLE group1 ✅ Correct Answer

Question 31

You have an Azure Databricks workspace named Workspace1.
You have a user named User1 that is a non-admin user for Workspace1.
You need to ensure that User1 can perform the following tasks:

Provision clusters of any size.
Run Databricks jobs.

The solution must follow the principle of least privilege.

What should you do?

Options:

Assign the workspace admins role
Assign the Consumer access entitlement
Grant the CAN RUN permissions for the jobs
Assign the Contributor role for a resource group
Assign the Allow unrestricted cluster creation entitlement

Requirement	Answer
To ensure that User1 can provision clusters	5. Assign the Allow unrestricted cluster creation entitlement ✅ Correct Answer
To ensure that User1 can run jobs	3. Grant the CAN RUN permissions for the jobs ✅ Correct Answer

Question 33

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named catalog1.

You have a group named group1. Group1 already has the USE CATALOG privilege on catalog1.

You create a schema named schema1 in catalog1.

You need to ensure that group1 can create tables in schema1. Group1 must not be able to grant permissions on the schema or its objects. The solution must follow the principle of least privilege.

How should you complete the SQL statement?

GRANT {} ON SCHEMA catalog1.schema1 {};

Area	Answer
Privileges	CREATE TABLE, USE SCHEMA ✅ Correct Answer
Principal	TO group1 ✅ Correct Answer

Question 34

You have an Azure Databricks workspace that is attached to a Unity Catalog metastore named metastore1.

metastore1 contains:

A catalog named Sales
A schema named Customers in the Sales catalog
A table named Customer_details in the Customers schema

You need to ensure that a user named User1 can update the data in Customer_details.

The solution must meet the following requirements:

Ensure that User1 cannot create new tables.
Follow the principle of least privilege.

Which permission should you grant to User1 for each object?

Object	Answer
The Sales catalog	USE CATALOG ✅ Correct Answer
The Customers schema	USE SCHEMA ✅ Correct Answer
The Customer_details table	MODIFY ✅ Correct Answer

Key takeaways

For DP-750, Unity Catalog permission questions usually test whether you understand the full access path.

Remember these patterns:

To read data: USE CATALOG + USE SCHEMA + SELECT
To update data: USE CATALOG + USE SCHEMA + MODIFY
To create tables: USE CATALOG + USE SCHEMA + CREATE TABLE
USE CATALOG does not mean SELECT.
USE SCHEMA does not mean SELECT.
MODIFY is for data changes.
CREATE TABLE is for creating new tables.
MANAGE is broader and can allow permission management.
CAN RUN is for job execution, not data access.
Allow unrestricted cluster creation is a workspace entitlement, not a Unity Catalog privilege.

If you can identify whether the question is about data access, object creation, job execution, or cluster provisioning, you can usually find the correct DP-750 permission answer quickly.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DP-750: Unity Catalog Object Model Explained and with Real Exam Questions

Jin — Tue, 28 Jul 2026 15:59:56 +0000

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you must understand how Unity Catalog organizes data.

Unity Catalog is not only a permission system. It is the central governance layer for data and AI assets in Azure Databricks. It provides a structured object model for organizing data, applying access control, managing metadata, and supporting governance across workspaces.

For DP-750, the most important Unity Catalog structure is:

metastore
  └── catalog
        └── schema
              └── table / view / volume / function / model

Azure Databricks documentation describes catalogs as the first layer in Unity Catalog’s three-level namespace: catalog.schema.table-etc. A catalog contains schemas, and schemas contain tables, views, volumes, models, and functions.

1. What is a Unity Catalog metastore?

A metastore is the top-level container for Unity Catalog metadata.

It contains securable objects such as:

catalogs
schemas
tables
views
volumes
functions
storage credentials
external locations
connections
foreign catalogs

In practice, you usually do not create a new metastore for every team or department. A metastore is often shared across a Databricks account or region, while catalogs and schemas are used to organize and isolate business domains, environments, or data products.

For DP-750, when a question says:

The workspace is attached to a Unity Catalog metastore.

it means the workspace can use Unity Catalog objects such as catalogs, schemas, managed tables, external locations, and governed permissions.

2. What is a catalog?

A catalog is the first level of the Unity Catalog namespace.

Example:

sales.transactions.orders

In this name:

sales        = catalog
transactions = schema
orders       = table

A catalog is the primary unit of data organization and logical isolation in Unity Catalog. Microsoft documentation explains that catalogs often mirror organizational units or software development lifecycle scopes, such as business units, domains, or environments.

Common catalog design patterns include:

dev
test
prod

or:

sales
finance
hr

or:

bronze
silver
gold

However, for governance and isolation, catalogs are often better used for business domains or environments, while schemas are used for more granular organization.

3. What is a schema?

A schema is the second level of the Unity Catalog namespace.

Example:

finance.procurement.assets

In this name:

finance     = catalog
procurement = schema
assets      = table

A schema organizes data and AI assets into a more granular category than a catalog. Microsoft documentation says schemas typically represent a single use case, project, or team sandbox, and they help with access control and data discovery.

For DP-750, schemas are often used to organize data by:

team
project
department area
data layer
sandbox
curated domain

Example:

finance.default
finance.procurement
finance.reporting

4. What is a table?

A table is a data object inside a schema.

Example:

finance.procurement.assets

This is a fully qualified table name.

For Unity Catalog, managed tables are usually the recommended default table type. Microsoft documentation says Unity Catalog managed tables are the default and recommended table type in Azure Databricks for Delta Lake and Apache Iceberg, and Unity Catalog manages read, write, storage, and optimization responsibilities for them.

For DP-750, if a question asks you to create a table in a specific catalog and schema, you can either:

use the full three-part name:

CREATE TABLE finance.procurement.assets (
    asset_id INT,
    asset_name STRING,
    asset_type STRING
);

or:

set the current catalog and schema first:

USE CATALOG finance;
USE SCHEMA procurement;

CREATE TABLE assets (
    asset_id INT,
    asset_name STRING,
    asset_type STRING
);

Both approaches identify the same table location in the Unity Catalog namespace.

5. Three-level namespace

Unity Catalog uses a three-level namespace:

catalog.schema.object

For tables, this means:

catalog.schema.table

Example:

finance.procurement.assets

This structure matters in DP-750 because many exam questions test whether you know where an object belongs.

Object	Level
Metastore	Top-level metadata container
Catalog	First namespace level
Schema	Second namespace level
Table	Third namespace level
View	Third namespace level
Volume	Third namespace level
Function	Third namespace level
Model	Third namespace level

A common exam trap is confusing catalog and schema.

For example:

CREATE CATALOG schema2

is wrong if the requirement is to create a schema named schema2 inside an existing catalog.

The correct pattern is:

CREATE SCHEMA catalog1.schema2;

6. Managed location

A managed location is a cloud storage path used by Unity Catalog to store managed objects such as managed tables and managed volumes.

Managed storage can be configured at different levels:

metastore
catalog
schema

Microsoft documentation explains that managed tables and volumes are stored in managed storage locations, and that managed storage can be set at the metastore, catalog, or schema level. Data is stored at the lowest available location in the hierarchy.

This hierarchy is important:

Schema managed location
    overrides catalog managed location

Catalog managed location
    overrides metastore managed location

Metastore managed location
    fallback default

So if a schema has its own managed location, managed tables in that schema use the schema’s managed location.

7. MANAGED LOCATION vs LOCATION

This is a very important DP-750 exam detail.

For Unity Catalog schemas, if you want to specify the storage location, you should use:

MANAGED LOCATION

not:

LOCATION

Microsoft documentation for CREATE SCHEMA states that LOCATION is not supported in Unity Catalog, and if you want to specify a storage location for a schema in Unity Catalog, you must use MANAGED LOCATION.

Correct example:

CREATE SCHEMA catalog1.schema2
MANAGED LOCATION 'abfss://container@storageaccount.dfs.core.windows.net/data';

Incorrect example:

CREATE SCHEMA catalog1.schema2
LOCATION 'abfss://container@storageaccount.dfs.core.windows.net/data';

For DP-750, remember:

Unity Catalog schema storage path = MANAGED LOCATION

8. Organizing catalogs and schemas

Unity Catalog design is about balancing:

governance
isolation
discoverability
naming consistency
access control
operational simplicity

Microsoft’s setup guidance says catalogs are the primary unit of data isolation in Unity Catalog, and all schemas, tables, volumes, views, and functions live in catalogs. It also recommends creating catalogs around logical boundaries such as business units, environments, or major data products.

For example, if your company has departments such as:

sales
finance
hr

and each department must have independently managed access and logically isolated assets, then each department can have its own catalog:

sales
finance
hr

Each catalog can then contain schemas:

sales.raw
sales.cleansed
sales.curated

finance.raw
finance.cleansed
finance.curated

hr.raw
hr.cleansed
hr.curated

This supports centralized governance while keeping access boundaries clear.

9. Development area organization

Another common DP-750 pattern is about organizing development areas.

If the requirement says:

isolated development areas
standard naming conventions
centralized governance
consistent structure

then do not let every team create random objects in the default catalog and schema.

A better pattern is:

development catalog
  ├── team_a_schema
  ├── team_b_schema
  └── team_c_schema

For example:

dev.team_sales
dev.team_finance
dev.team_iot

This keeps development isolated while still enforcing a shared naming standard.

In DP-750, if the case study says that naming conventions are inconsistent and development teams need isolated areas, the answer is usually:

shared development catalog + standardized naming convention + separate schema per team

10. DP-750 decision table for Unity Catalog object model

Scenario in the question	Best answer pattern
Need to create a schema inside catalog1	`CREATE SCHEMA catalog1.schema_name`
Need schema-specific managed storage	`MANAGED LOCATION`
Need to create a table in a specific schema	`USE CATALOG`, `USE SCHEMA`, then `CREATE TABLE`
Need logical isolation for departments	Separate catalog per department
Need isolated development areas with standard naming	Shared development catalog + schema per team
Need consistent raw, cleansed, curated organization	Standard catalog/schema structure
Need centralized governance	Unity Catalog
Need object name with catalog, schema, table	Three-level namespace

Real Exam Questions

Question 4

You have an Azure Databricks workspace that is attached to a Unity Catalog metastore named metastore1, metastore1 contains a catalog named catalog1.

You need to create a new schema named schema2 that meets the following requirements:

Is contained in catalog1
Uses abfss://container@storageaccount.dfs.core.windows.net/data as the managed location

Which SQL statement should you execute?

A. CREATE SCHEMA catalog1.schema2

LOCATION ‘abfss://container@storageaccount.dfs.core.windows.net/data’;

B. CREATE SCHEMA catalog1.schema2

MANAGED LOCATION ‘abfss://container@storageaccount.dfs.core.windows.net/data’; ✅ Correct Answer

C. CREATE CATALOG schema2

MANAGED LOCATION ‘abfss://container@storageaccount.dfs.core.windows.net/data’;

D. CREATE SCHEMA catalog1.schema2

WITH DBPROPERTIES (LOCATION-’abfss://container@storageaccount.dfs.core.windows.net/data’);

Question 6

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named finance, finance contains two schemas named default and procurement.

You need to create a table named assets in the procurement schema, assets must contain the following columns:

asset_id
asset_type
asset_name

How should you complete the SQL statement?

USE CATALOG finance;
USE SCHEMA procurement;
CREATE TABLE assets (
    asset_id INT,
    asset_name STRING,
    asset_type STRING
);

✅ Correct Answer

Question 15

Useful case information

Contoso identifies the following governance requirements:

Centralize the metadata catalog.
Provide isolated development areas that follow standard naming conventions.
Establish a consistent structure for organizing raw, cleansed, and curated data.

You need to organize Unity Catalog. The solution must meet the governance requirements.

What should you do?

A. Use a single shared schema for all the development teams and rely on table-level permissions for isolation.

B. Enable the development teams to create objects directly in the default catalog and schema.

C. Create a separate catalog for each development team and enable each team to choose its own schema names.

D. Create a shared development catalog, enforce a standardized naming convention, and assign each development team its own schema. ✅ Correct Answer

Question 30

Your company has sales, finance, and HR departments.

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You need to implement Unity Catalog to meet the following requirements:

Access to Unity Catalog for each department must be managed independently from that of the other departments.
The data assets of each department must be isolated logically from those of the other departments.
The solution must support centralized governance.

What should you do for each department?

A. Create a separate Azure Data Lake Storage Gen2 account.

B. Create dynamic views.

C. Create a new workspace.

D. Create a new catalog. ✅ Correct Answer

Key takeaways

For DP-750, Unity Catalog object model questions are usually about where objects belong and how to organize them for governance.

Remember these patterns:

Unity Catalog uses the namespace: catalog.schema.object
A catalog is the first level of data organization and logical isolation.
A schema is the second level and often represents a project, use case, or team sandbox.
Tables, views, volumes, functions, and models live inside schemas.
To create a schema inside a catalog, use CREATE SCHEMA catalog.schema.
To specify a Unity Catalog schema storage path, use MANAGED LOCATION.
For independent department-level access, create separate catalogs.
For isolated development areas with standard naming, use one shared development catalog and one schema per team.

If you understand the object model, many Unity Catalog questions become much easier because you can identify whether the answer should be at the catalog level, schema level, or table level.

Explore more

JinFollow

Hello there! 👋 I'm Jin, a Business Intelligence Developer with a passion for all things data. Proficient in Python, SQL, Power BI, Tableau

Thank you for taking the time to explore data-related insights with me. I appreciate your engagement.

Connect with me on LinkedIn

Connect with me on X

DEV Community: Jin

How to Connect Azure SQL Database in Azure Databricks with PySpark

Connect the Azure SQK Database

Read data from Azure SQL Database

Write Data to Azure SQL Database

Explore more

JinFollow

DP-750: Databricks Asset Bundles, Git, and ALM Explained and with Real Exam Questions

1. Why ALM matters in Azure Databricks

2. What are Declarative Automation Bundles?

3. Why use bundles instead of manual workspace export/import?

4. databricks.yml

5. Important bundle nodes

6. What are targets?

7. Targets for selective deployment

8. Databricks CLI and bundles

9. databricks bundle deploy --target dev

10. Git folders in Azure Databricks

11. Merge vs rebase vs pull vs push

12. Merge conflicts

13. Why not clone a new repository?

14. Git folders vs Declarative Automation Bundles

15. Declarative Automation Bundles vs Terraform

16. DP-750 decision table for bundles, Git, and ALM

Real Exam Questions

Question 12

Question 62

Question 66

Question 67

Question 73

Question 80

Question 81

Key takeaways

Explore more

JinFollow

Azure Databricks: Fix “DataFrame Object Has No Attribute write”

The Problem

Why This Error Happens

The Solution

Complete Working Example

Summary

Explore more

JinFollow

DP-750: Slowly Changing Dimensions and Data Quality Expectations Explained and with Real Exam Questions

1. What is a Slowly Changing Dimension?

2. SCD Type 1

3. SCD Type 2

4. SCD Type 1 vs SCD Type 2

5. SCD Type 2 and Lakeflow

6. Delta Lake time travel vs SCD Type 2

7. What are data quality expectations?

8. Expectation actions

9. expect

10. expect_or_drop

11. expect_or_fail

12. Expectation metrics

13. Quarantine invalid records

14. Expectations vs table constraints

15. DP-750 decision table for SCD and expectations

Real Exam Questions

Question 36

Useful case information

Question 55

Question 57

Question 76

Question 77

Question 78

Key takeaways

Explore more

JinFollow

DP-750: Delta Lake Performance Optimization Explained and with Real Exam Questions

The related questions in your DP-750 question bank are Q9 and Q58.

1. Why Delta Lake performance optimization matters

2. What is data layout?

3. What is liquid clustering?

4. Why liquid clustering is preferred over fixed partitioning

5. Liquid clustering vs partitioning

6. Choosing clustering columns

7. What are deletion vectors?

8. Why deletion vectors help with updates and deletes

4. `databricks.yml`

9. `databricks bundle deploy --target dev`

9. `expect`

10. `expect_or_drop`

11. `expect_or_fail`

2. `INSERT INTO`

3. `INSERT OVERWRITE`

4. `CREATE TABLE AS SELECT` vs `CREATE OR REPLACE TABLE AS SELECT`

6. `NOT NULL` constraint

7. `CHECK` constraint

8. `INTERSECT`

10. `fillna` does not remove rows

11. Why `df.order_amount != None` is wrong

6. `mergeSchema`

7. Auto Loader schema evolution mode: `failOnNewColumns`

8. `availableNow` trigger

11. `skipChangeCommits`