Databricks interview questions for data engineers, and what each tests

Originally published on four-leaf.ai.

Databricks interview questions for data engineers test whether you understand what the platform does underneath the notebook. Databricks' own documentation states that "Delta Lake is the default format for all operations on Databricks", so a large share of the technical questions in these loops come back to how a Delta table stores, versions and rewrites data.

This post covers the Databricks platform questions a data engineer meets when interviewing at any company that runs on it. If you are interviewing for a job at Databricks itself, the stages and timeline are a different subject, covered in the Databricks interview process. The platform behavior described below is taken from the Databricks and Apache Spark documentation, and each source is named in the section that uses it.

What do Databricks interview questions for data engineers test?

Databricks interview questions for data engineers test judgment about tradeoffs. Most interviewers assume you can write a PySpark job. They want to know whether you can explain why a job got slow, why a table grew, or why a pipeline wrote duplicates.

Expect questions in five areas. Delta Lake internals come first, because Delta is the default table format on Databricks. Then pipeline design, usually framed around the medallion architecture. Then Spark execution, where the shuffle is the center of gravity. Then writes and layout, meaning MERGE, clustering and data skipping. Ingestion and governance close the list, covering Auto Loader and Unity Catalog.

Read the posting to see which areas the employer weights. A posting that names streaming, Kafka or Auto Loader is telling you the ingestion round carries weight. A posting that mentions cost, cluster sizing or performance tuning will push hard on the Spark and layout questions. A posting that mentions governance, lineage or access control expects Unity Catalog fluency.

What is Delta Lake, and what does the transaction log do?

Delta Lake is a storage layer that adds a transaction log to Parquet files. The Databricks documentation describes it as open source software that "extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling." The log is what makes a directory of files behave like a table.

A typical question is "A teammate ran VACUUM and now time travel to last week fails. Why?" A strong answer separates two retention settings. The Databricks VACUUM reference says files are deleted according to when they were logically removed from the transaction log plus the retention period, and "the default threshold is 7 days." The Databricks table history page says logRetentionDuration defaults to 30 days and warns that "Time travel and table history are controlled by different retention thresholds." It follows that history can still list a version whose data files VACUUM already deleted.

Interviewers follow up by asking what you would change. Name the tradeoff first, which is storage cost against how far back you need to restore, and only then the setting that expresses it.

How do you explain the medallion architecture without reciting it?

The medallion architecture organizes a lakehouse into bronze, silver and gold layers of increasing quality. The Databricks medallion page labels them raw, validated and enriched. Interviewers want to hear what you would do differently because of those layers.

Two details from the Databricks medallion page make better answers. In bronze, "Databricks recommends storing most fields as string, VARIANT, or binary to protect against unexpected schema changes." The point is that bronze should never reject a record because a source changed shape. For silver, "Databricks does not recommend writing to silver tables directly from ingestion," because schema changes and corrupt records in sources then break the validated layer.

The same page says "Following the medallion architecture is a recommended best practice but not a requirement." That sentence is useful in an interview. When asked to design a pipeline, explain which layers your use case needs and why, instead of drawing three boxes because the pattern has three.

What Spark questions come up, and how deep do they go?

Spark questions for Databricks data engineers usually go one level below the API. Expect to explain lazy evaluation, the shuffle and how a skewed join behaves.

The Apache Spark programming guide states that "All transformations in Spark are lazy," computed only when an action needs a result. That explains a classic puzzle where a job seems to fail on a count() when the bug is in a transformation written ten lines earlier. The same guide calls the shuffle the mechanism for redistributing data across partitions, which involves "copying data across executors and machines, making the shuffle a complex and costly operation." The guide lists repartition, groupByKey and join operations among the operations that can cause one. Databricks' adaptive query execution page adds that AQE can change a sort merge join into a broadcast hash join at runtime.

Skew is the usual follow-up. Databricks' adaptive query execution page says AQE can split skewed tasks in sort merge and shuffle hash joins into roughly evenly sized ones at runtime, and "AQE is enabled by default." A good answer to "one task runs for an hour while the rest finish in a minute" checks the query plan first to see whether AQE already acted. Salting keys by hand comes after that check.

How do you answer a MERGE or upsert question?

MERGE questions test whether you have run an upsert in production. The prompt is often "Your daily MERGE started failing after a source change. What happened?"

The Databricks merge documentation gives a likely answer. "A merge operation can fail if multiple rows of the source dataset match and the merge attempts to update the same rows of the target Delta Lake table." The fix is deduplicating the source before the merge, usually by keeping the latest record per key.

The opposite failure shows up in insert-only merges used for deduplication. The same documentation notes that MERGE matches new data against the table, "but if there is duplicate data within the new dataset, it is inserted." A candidate who mentions both cases shows they understand that MERGE checks each source row against the target and leaves duplicates inside the source batch for you to remove. Interviewers may extend this into slowly changing dimensions. Explain which columns you track and how you close out an old row before you write any SQL.

How do layout and performance questions work?

Layout questions ask how data is physically arranged so queries read less of it. The modern Databricks answer is liquid clustering, which the Databricks documentation describes as a technique that "replaces table partitioning and ZORDER." The same page says "you can redefine clustering keys without rewriting existing data," and also that the default behavior does not apply clustering to previously written data. Changing keys is cheap to declare, and old files keep their old layout until you run OPTIMIZE with FULL, which does rewrite them.

Explain the mechanism underneath. The Databricks data skipping page says the engine keeps per-file statistics ("minimum and maximum values, null counts, and total records") and uses them at query time to skip files. Good layout on a filtered column means more files can be ruled out from those statistics alone.

Photon questions test whether you oversell. Photon is the Databricks vectorized engine, and its documentation says queries "that normally complete in under two seconds don't see meaningful improvement" because planning and scheduling dominate. An answer that names when Photon will not help is stronger than one that promises it always will.

What do ingestion and governance questions ask?

Ingestion questions usually center on Auto Loader, the Databricks source for picking up new files from cloud storage. The question is often "How does Auto Loader avoid processing a file twice after a failure?" The Databricks Auto Loader page says discovered file metadata lives in a RocksDB key-value store in the checkpoint location, and "This key-value store ensures that data is processed exactly once." Follow-ups probe what happens if someone deletes or moves the checkpoint, so know that the checkpoint is the state.

Governance questions center on Unity Catalog. The Unity Catalog overview says data and AI assets, including tables, views, volumes, functions and models, follow a three-level namespace, catalog.schema.object. Expect to design permissions for a team, such as analysts who read gold tables while engineers write to silver, and to explain at which level of that namespace you would grant each one.

What is overrated

Memorizing feature names is overrated. An interviewer who asks about liquid clustering will follow up with "when would you not use it?", and a list of features gives you nothing to say.

Reciting bronze, silver and gold is overrated for the same reason. The Databricks medallion page itself calls the pattern a recommendation, so a design that skips a layer for a good reason will usually score better than one that copies the diagram.

Leading with tuning flags is overrated too. Most performance answers start with reading the query plan and the file layout. Configuration comes last.

Prepare for a Databricks data engineering interview this week

1. Build one small scratch Delta table, update it a few times and run DESCRIBE HISTORY. A default VACUUM on it deletes nothing for 7 days, so to see time travel fail you have to shorten the table's delta.deletedFileRetentionDuration and turn off the safety check with spark.databricks.delta.retentionDurationCheck.enabled set to false. Never do that on a real table. Watching it happen once teaches the VACUUM and time travel question better than reading about it.
2. Write a MERGE that fails on duplicate source keys, then fix it. Be ready to explain both the failure and the fix out loud.
3. Practice SQL until it is automatic. SQL was used by 58.6% of respondents in the Stack Overflow 2025 Developer Survey. The SQL interview guide covers the patterns.
4. Prepare one pipeline design story from bronze to gold, including a layer you would skip and why. The system design interview questions guide covers how to structure that conversation.
5. Rehearse out loud. Knowing why a MERGE fails and explaining it under follow-up questions are different skills. In Four-Leaf's mock interviews you can name the round yourself, such as "Delta Lake and MERGE", and answer follow-up questions built on what you actually said.

Where this is heading

Databricks keeps moving the defaults. Liquid clustering replaced partitioning advice, and AQE handles skew that engineers used to fix by hand. Check the documentation date on any answer you memorize.

The questions underneath those answers last longer. Every one of them asks whether you know what a table is doing to your data, and can say so clearly when someone pushes back. That is the skill to practice.

About Four-Leaf

Four-Leaf is the all-in-one AI job search assistant covering every stage from application to signed offer: voice mock interviews across 25 roles, resume tailoring, cover letters, salary negotiation, email drafting, AI job search, and LinkedIn optimization. 3-day free trial, no credit card. Pricing: $20/mo or $5 / 5 Day Pass.

• Website: https://four-leaf.ai
• Pricing: https://four-leaf.ai/pricing
• Contact: [e-mail]