Delta Lake Schema Evolution and Time Travel on Azure Databricks: mergeSchema, overwriteSchema, Column Mapping, and RESTORE

Sooner or later a source system adds a column, renames one, or someone runs a bad overwrite on a production table. How a Delta table reacts in those moments is a favourite interview topic, because it separates people who have used Delta Lake from people who have only read about it. This post covers schema enforcement, the two schema evolution options, column mapping for renames and drops, and how to use time travel and RESTORE to get out of trouble on Azure Databricks.

Schema enforcement: Delta says no by default

Delta checks every write against the table's schema. If you append a DataFrame with a column the table doesn't have, the write fails instead of silently dropping or adding data.

from pyspark.sql import functions as F

orders = spark.createDataFrame(
    [(1, "C001", 120.50), (2, "C002", 80.00)],
    "order_id INT, customer_id STRING, amount DOUBLE",
)
orders.write.format("delta").mode("overwrite").saveAsTable("demo.orders")

# The source now sends a new column
new_orders = spark.createDataFrame(
    [(3, "C003", 42.00, "web")],
    "order_id INT, customer_id STRING, amount DOUBLE, channel STRING",
)

new_orders.write.format("delta").mode("append").saveAsTable("demo.orders")
# AnalysisException: A schema mismatch detected when writing to the Delta table...

The same check catches type mismatches. Writing a string into an INT column fails rather than quietly casting it to null. That is the behaviour you want in Silver and Gold tables. A missing column is fine, though: if the incoming DataFrame lacks a nullable column, Delta writes null for it.

Option 1: mergeSchema adds columns

When the new column is expected, opt in to evolution for that one write:

(new_orders.write
    .format("delta")
    .mode("append")
    .option("mergeSchema", "true")
    .saveAsTable("demo.orders"))

spark.table("demo.orders").printSchema()
# order_id, customer_id, amount, channel  (old rows have channel = null)

mergeSchema adds new columns to the end of the schema and can make some compatible changes, such as letting a NOT NULL column accept nulls. It does not let you change amount from DOUBLE to STRING. Type changes need a rewrite (or, on recent runtimes, the type widening table feature for safe widenings like INT to BIGINT).

You can also turn evolution on for the whole session with spark.databricks.delta.schema.autoMerge.enabled = true. I'd avoid that in shared jobs. It applies to every write in the session, so an upstream bug that adds a junk column lands in your table without anyone noticing.

Schema evolution in MERGE

MERGE is where people get caught. A per-write mergeSchema option doesn't apply to it. On recent Databricks runtimes you can ask for evolution in the statement itself:

MERGE WITH SCHEMA EVOLUTION INTO demo.orders AS t
USING updates AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

On older runtimes you set the autoMerge session config just before the merge. Either way, evolution only happens with UPDATE SET * and INSERT * (or with explicit assignments to the new column); a merge that lists only existing columns won't add anything.

Option 2: overwriteSchema replaces everything

Sometimes you really want a new schema, for example after a redesign of a Gold table:

redesigned = (spark.table("demo.orders")
    .withColumn("amount", F.col("amount").cast("decimal(12,2)"))
    .drop("channel"))

(redesigned.write
    .format("delta")
    .mode("overwrite")
    .option("overwriteSchema", "true")
    .saveAsTable("demo.orders"))

This rewrites the data and replaces the schema, including types and partitioning. It's the right tool for intentional redesigns and the wrong one for daily jobs. Any downstream query that depended on channel breaks on its next run. In SQL, CREATE OR REPLACE TABLE ... AS SELECT does the same job. Both keep the table's history, so you can still time travel to before the change, which is why they beat DROP TABLE followed by a fresh CREATE.

Renaming and dropping columns: column mapping

Delta originally matched columns to Parquet files by physical name, so renaming a column meant rewriting the table. Column mapping fixes that by giving each column a stable internal ID:

ALTER TABLE demo.orders SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name'
);

ALTER TABLE demo.orders RENAME COLUMN customer_id TO customer_key;
ALTER TABLE demo.orders DROP COLUMN channel;

Both are metadata-only operations, so they're instant even on huge tables. Two things to mention in an interview. First, enabling column mapping upgrades the table protocol, so old readers and some external tools that read the Parquet files directly may no longer read it correctly. Second, DROP COLUMN doesn't remove the data from the files until they are rewritten, so it's not how you remove sensitive data.

Time travel: reading the past

Every commit to a Delta table creates a version in the _delta_log. You can look at them:

DESCRIBE HISTORY demo.orders;

The output shows the version, timestamp, operation (WRITE, MERGE, DELETE...), the user, and operationMetrics such as rows and files written. Then query any version that hasn't been cleaned up:

SELECT * FROM demo.orders VERSION AS OF 3;
SELECT * FROM demo.orders TIMESTAMP AS OF '2026-10-26 08:00:00';

In PySpark:

v3 = (spark.read.format("delta")
      .option("versionAsOf", 3)
      .table("demo.orders"))

A handy pattern is diffing two versions to see what a job actually changed:

before = spark.read.option("versionAsOf", 5).table("demo.orders")
after  = spark.read.option("versionAsOf", 6).table("demo.orders")

removed = before.exceptAll(after)
added   = after.exceptAll(before)
print(removed.count(), added.count())

Time travel also makes reproducible reads easy. Pin a version for a model training run or a month-end report and you'll get the same data even after later loads.

RESTORE: undoing a bad write

Someone ran an overwrite with the wrong filter. Find the last good version in DESCRIBE HISTORY, then:

RESTORE TABLE demo.orders TO VERSION AS OF 5;
-- or
RESTORE TABLE demo.orders TO TIMESTAMP AS OF '2026-10-27 06:00:00';

RESTORE doesn't delete history. It writes a new commit that makes the table look like version 5, so you can even restore the restore if you picked the wrong version. It reports how many files were removed and restored. Restoring the table's data doesn't undo side effects elsewhere, though. If downstream jobs already read the bad version, they need to be rerun.

Why time travel isn't a backup

Time travel only works while both the log entries and the data files exist:

• delta.logRetentionDuration (30 days by default) controls how long commit history is kept.
• delta.deletedFileRetentionDuration (7 days by default) controls how old a removed file must be before VACUUM can delete it.

Once VACUUM runs, versions older than the file retention window usually can't be read anymore, because their files are gone. So by default you can count on about a week of time travel, not a month. If you need longer, raise both properties together and accept the storage cost:

ALTER TABLE demo.orders SET TBLPROPERTIES (
  'delta.logRetentionDuration' = 'interval 60 days',
  'delta.deletedFileRetentionDuration' = 'interval 30 days'
);

Time travel also lives in the same storage account as the table. If the container is deleted, history goes with it. For real recovery, use DEEP CLONE to a separate location or storage-level backups.

Interview questions to practise

1. What happens when you append a DataFrame with an extra column to a Delta table, and what are two ways to make it succeed?
2. Why is enabling autoMerge for a whole session risky?
3. How do you rename a column on a 5 TB Delta table without rewriting it, and what's the trade-off?
4. A table was overwritten by mistake yesterday. Walk through recovering it.
5. Why might VERSION AS OF fail for a version from 10 days ago?

If you can answer those with DESCRIBE HISTORY output in front of you, you're in good shape for most Delta Lake questions.

---

If you want more practice like this, there's a free 10-question sample of Azure Databricks and PySpark interview questions with worked answers: free sample. The full set is in the Azure Databricks & PySpark Interview Pack.